LLM-based business travel service management method and computer equipment

Abstract

The business travel service management method based on the LLM is applied to a service system constructed based on the LLM. Services provided by the service system for users comprise a plurality of service fields; the service system manages a plurality of service models in one-to-one correspondence with a plurality of service fields; comprising the following steps: receiving a user question related to business travel service input by a user; retrieving target structured knowledge from a target sub-knowledge base related to the user questions in a plurality of sub-knowledge bases in one-to-one correspondence with the plurality of service fields, and acquiring historical context information from historical user questions input by the user; generating a cue word based on the user question, the target structured knowledge and the historical context information, and inputting the cue word to an LLM, so that the LLM generates structured intention description information based on the cue word; and analyzing the intention description information, determining a target service field corresponding to the user intention, and distributing the user question to a target service model corresponding to the target service field.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of artificial intelligence, and in particular to a business travel service management method based on LLM and computer equipment. BACKGROUND

[0002] On some enterprise-level service platforms, in order to better manage the daily business travel activities of employees, the service platform can provide the user with a business travel service page to provide the user with a full-process business travel service. The user can complete online services such as reservation services related to the user's business travel (such as flight reservations, hotel reservations, train ticket reservations, and car services related to reservations), reimbursement services, and the like through the business travel service page.

[0003] With the continuous development of LLM (Large Language Model) technology, a business travel service system based on LLM has gradually evolved into a mainstream service mode to provide intelligent business travel services to users in the form of question and answer.

[0004] However, in the related art, the business travel service system based on LLM usually adopts an end-to-end solution based on a single LLM. This technical route has significant limitations when dealing with user questions in highly complex or professional service fields. SUMMARY

[0005] The present specification proposes a business travel service management method based on LLM, applied to a service system based on LLM; wherein the business travel services provided by the service system to users include multiple service fields; the external knowledge base accessed by the service system contains multiple sub-knowledge bases corresponding one-to-one to the multiple service fields; the service system manages multiple service models corresponding one-to-one to the multiple service fields; the service model includes a service model based on LLM; the method comprises: receiving a user question input by a user related to the business travel service; determining a target sub-knowledge base related to the user question from the multiple sub-knowledge bases, retrieving target structured knowledge similar in semantics to the user question from the target sub-knowledge base, and obtaining historical context information from historical user questions input by the user related to the business travel service; generate a prompt word for guiding the LLM to judge the user intention of the user based on the user question, the target structured knowledge and the historical context information, and input the prompt word into the LLM to perform inference calculation based on the prompt word by the LLM to generate structured intention description information; wherein the intention description information is used to describe the user intention of the user; analyze the intention description information, determine a target service field corresponding to the user intention of the user, and distribute the user question to a target service model corresponding to the target service field, so that the target service model performs service processing corresponding to the business travel service on the user question.

[0006] Optionally, the target sub-knowledge base related to the user question is determined from the plurality of sub-knowledge bases, comprising: extracting a question feature from the user question; generating a feature vector based on the question feature, and calculating a vector distance between the feature vector and a description vector of the plurality of sub-knowledge bases; wherein the description vector comprises a vector generated based on the description information of the plurality of sub-knowledge bases; the target sub-knowledge base with the smallest vector distance is determined as the sub-knowledge base related to the user question; or, extracting a question feature from the user question; inputting the question feature into a pre-trained multi-classification model accessed by the service system to perform classification processing on the question feature by the multi-classification model; wherein the classification result output by the multi-classification model represents the sub-knowledge base related to the user question; based on the classification result of the question feature output by the multi-classification model, determining the target sub-knowledge base related to the user question from the plurality of sub-knowledge bases.

[0007] Optionally, the sub-knowledge base comprises a vector database composed of vectorized structured knowledge; retrieving target structured knowledge similar in semantics to the user question from the target sub-knowledge base, comprising: generating a question vector corresponding to the user question; running an approximate nearest neighbor algorithm to filter a candidate vector set related to the question vector from the vector database; further calculating vector distances between the user question and the vectorized structured knowledge in the candidate vector set; at least one vectorized structured knowledge in the candidate vector set with the smallest vector distance from the question vector is taken as the target structured knowledge similar in semantics to the user question.

[0008] Optionally, the historical user questions input by the user and related to the business travel service include a historical user question sequence composed of historical user questions corresponding to a plurality of historical sessions; The historical context information includes: historical user questions corresponding to historical sessions in the plurality of historical sessions with a session round less than a first threshold; and a user question summary of historical user questions corresponding to historical sessions in the plurality of historical sessions with a session round greater than the first threshold and less than a second threshold; Optionally, obtaining historical context information from the historical user questions input by the user and related to the business travel service includes: Obtaining historical user questions corresponding to historical sessions with a session round less than or equal to a first threshold from the historical user question sequence; and, Obtaining historical user questions corresponding to historical sessions with a session round greater than the first threshold and less than a second threshold from the historical user question sequence, and calculating a user question summary of the historical user questions.

[0009] Optionally, the service system accesses a processing model for standardizing the user question; Before generating the prompt word for guiding the LLM to judge the user's user intent based on the user question, the target structured knowledge, and the historical context information, the method further includes: Standardizing the user question based on the accessed processing model to process the user question into a standardized user question; The standardization processing performed on the user question includes one or more combinations of the following: Correcting errors in the user question; Correcting colloquial expressions in the user question to standardized expressions; Rewriting the user question into a user question with standardized semantics; Disassembling the user question into a plurality of sub-user questions.

[0010] Optionally, the service system supports generating a prompt word based on a maintained structured prompt word template; wherein the structured prompt word template contains fields for maintaining the user question, the target structured knowledge, and the historical context information, respectively; Generating the prompt word for guiding the LLM to judge the user's user intent based on the user question, the target structured knowledge, and the historical context information includes: filling the user question, the target structured knowledge and the historical context information into corresponding fields in the structured prompt template respectively to generate a prompt for guiding the LLM to determine the user intent of the user.

[0011] Optionally, the prompt template comprises a CO-STAR prompt template; wherein the CO-STAR prompt template comprises a first extension field for maintaining the user question, and a second extension field for maintaining the historical context information. generating a prompt for guiding the LLM to determine the user intent of the user based on the user question, the target structured knowledge and the historical context information, comprising: filling the user question into the first extension field included in the CO-STAR prompt template; filling the historical context information into the second extension field included in the CO-STAR prompt template; filling the target structured knowledge into a target base field specified from base fields included in the CO-STAR prompt template to generate a prompt for guiding the LLM to determine the user intent of the user.

[0012] Optionally, the service model comprises an artificial intelligence service program constructed based on the LLM. The plurality of service domains comprises a reservation service domain related to the business travel service, a business travel data analysis service domain, and a business travel policy inquiry service domain; and the plurality of service models comprises a first artificial intelligence service program for executing the reservation service related to the business travel service, a second artificial intelligence service program for executing the business travel data analysis service related to the business travel service, and a third artificial intelligence service program for executing the business travel policy inquiry service related to the business travel service.

[0013] In the above embodiments, in the service system constructed based on the LLM, by utilizing the reasoning capability of the LLM, the user question input by the user, the structured knowledge retrieved from the knowledge base related to the user question input by the user, and the historical context information related to the historical user question of the user and other external evidence are used as reasoning context for reasoning calculation to determine the user intent of the user, and a target service domain corresponding to the user intent of the user is determined, and then the user question is distributed to a target service model corresponding to the target domain among a plurality of service models managed by the service system for service processing, so that when the user question input by the user involves service demand in a professional service domain, the user question can be accurately distributed to a service model related to the service domain, and the service model in the domain is used to provide services to the user.

[0014] Moreover, by employing multiple service models respectively corresponding to different service fields to provide services for users, the services provided for users can no longer be provided by employing only one single service model, so that the accuracy, reliability and professionalism of the services provided for users can be improved, and the problems of low service accuracy, poor reliability and professionalism when using a single universal service model to provide services for users in multiple service fields can be avoided. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present specification, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained according to these drawings without creative labor.

[0016] Figure 1 is a schematic diagram of an architecture of an Internet service system in an embodiment of the present specification; Figure 2 is a flowchart of a business travel service management method based on LLM in an embodiment of the present specification; Figure 3 is an architecture diagram of a service system constructed based on LLM in an embodiment of the present specification; Figure 4 is a schematic structural diagram of an electronic device in an embodiment of the present specification; Figure 5 is a block diagram of a business travel service management device based on LLM in an embodiment of the present specification. DETAILED DESCRIPTION

[0017] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is with reference to the drawings, in which like numerals refer to like elements throughout. The embodiments described in the following exemplary embodiments do not represent all of the implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples consistent with some aspects of one or more embodiments of the present specification as detailed in the appended claims.

[0018] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in the specification in other embodiments. In some other embodiments, the steps included in the method thereof can be more or less than described in the specification. In addition, a single step described in the specification may, in other embodiments, be broken down into multiple steps for description; and multiple steps described in the specification may, in other embodiments, be combined into a single step for description.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0020] The service system constructed based on a single LLM, although compared with the traditional service system (such as the service system based on rules or keyword matching), can utilize the reasoning ability of LLM itself to output accurate service results for user's questions when providing services to users, but for itself relatively complex services, especially composite services containing multiple service fields, there will still be certain limitations.

[0021] For example, the user services provided by the service system to users usually contain multiple service fields, different service fields usually have high heterogeneity and professionalism, and different service fields usually may involve different knowledge bases. In this case, if the service system based on a single LLM is used to provide services of all service fields to users at the same time, the LLM must encode the knowledge and logic of all service fields in its limited parameter space, which will lead to the following problems: First, the problem of semantic confusion and performance degradation: Due to the intersection and even conflict of the semantic spaces of different service fields, forcing an LLM to integrate the semantic spaces of multiple service fields may cause the decision accuracy of the LLM to decrease significantly when providing services of a specific field to users. For example, the LLM may misunderstand a user's question in an A service field as a service demand in a B service field.

[0022] Second, the problem of "catastrophic forgetting" of knowledge: When fine-tuning training is performed on the LLM in a certain specific service field among the above-mentioned multiple service fields, the LLM can lose its general service capability in other service fields.

[0023] Thirdly, the problem of Lack of Factual Consistency and Reliability: Some service fields can involve some highly professional and difficult-to-obtain structured knowledge (such as internal rules and regulations of an enterprise, and other highly private structured knowledge), which can lead to a lack of relevant training data when fine-tuning training is performed on the LLM in this service field. In this case, the fine-tuned LLM can easily produce hallucination and provide false or fabricated information.

[0024] As can be seen, when a single general LLM is used to provide users with complex services involving multiple service fields, there can be problems of low service accuracy, poor reliability, and poor professionalism.

[0025] Based on this, the present specification proposes a technical solution of introducing multiple service models based on LLMs corresponding to multiple service fields as domain expert models in a service system, and using the reasoning ability of the LLM to perform reasoning calculation on multiple external evidence as reasoning context to determine the user's user intent, and distributing the user's question to the domain expert model corresponding to the service field related to the user's intent for service processing based on the user's intent.

[0026] Figure 1 Fig. 1 is an exemplary embodiment of an architecture schematic diagram of an Internet service system. As shown in Fig. 1, the system can include a server 11, a network 12, and several electronic devices such as a PC (Personal Computer) 13, a mobile phone 14, etc. Figure 1

[0027] The server 11 can be a physical server containing a standalone host, or the server 11 can be a virtual server carried by a host cluster. In the running process, the server 11 can run the server-side program of a certain service program (such as an artificial intelligence service program based on LLM). For example, when the server 11 runs the server-side program of the service program, it can be implemented as a server of the service program.

[0028] ​PC 13, mobile phone 14 are only part of the types of electronic devices that users can use. In fact, users can obviously also use electronic devices of the following types: tablet devices, notebook computers, personal digital assistants (PDAs), wearable devices (such as smart glasses, smart watches, etc.), and the like, and one or more embodiments of the present specification do not limit this. During operation, the electronic device can run a client-side program of a certain service program to implement the relevant functions of the application, for example, when the electronic device runs a user-side program of the service program, it can be implemented as a client of the service program.

[0029] Among them, the client program of the above-mentioned service program can be started and run on the electronic device. The client-side program can be a native application installed on the electronic device, or the client-side program can be an applet, a fast application, or other similar forms.

[0030] Of course, when using web technologies such as HTML5 or the like, the relevant functions can be implemented through a page displayed by a browser, which can be a standalone browser application or a browser module embedded in some application.

[0031] As for the network 12 between the electronic devices such as PC 13, mobile phone 14 and the server 11, it can be implemented based on the communication mode supported by the corresponding electronic device, and the present specification does not limit whether it is implemented through a wired or wireless network. For example, the PC 13 can support wired and wireless communication at the same time, so wired or wireless communication can be used as needed, while the mobile phone 14 usually only supports wireless communication, so wireless communication can be used.

[0032] It should be noted that the services provided by the above-mentioned service program can include any service implemented on the Internet; for example, the services provided by the above-mentioned service program can specifically include business travel services provided to employees on the service platform of an enterprise.

[0033] The technical solutions of the present specification will be described in detail below with reference to the accompanying drawings.

[0034] Please refer to Figure 2 , Figure 2 A flowchart of a business travel service management method based on LLM shown in the present specification includes the following execution process: Step 202: receiving a user inputted user question related to the business travel service; The execution subject of the above-mentioned method can specifically include a service system constructed based on LLM.

[0035] In actual application, the service system can specifically include a server program and a client program, and the server program and the client program cooperate with each other to provide the user with the user service related to the itinerary planning.

[0036] For example, in some embodiments, the execution of the flow of the above method can specifically be the server program of the above service system, or the client program of the above service system; or, part of the steps in the above method can be run on the server program, and part of the steps can be run on the client program, and the above method corresponding flow is executed by the mutual cooperation of the server program and the client program.

[0037] The service provided by the above service system to the user can specifically include any form of service that can be implemented on the Internet, which is not limited in the specification. For example, in some embodiments, the service provided by the above service system to the user can specifically include business travel services.

[0038] In some embodiments, the service provided by the above service system to the user can include multiple service fields; the above service system can manage multiple service models corresponding to the above multiple service fields as domain expert models; the above service model can also be a service model constructed based on LLM.

[0039] For example, in some embodiments, the above service system can be an enterprise-level service platform, and the above multiple service fields can specifically include a reservation service field related to business travel services (such as ticket reservation, hotel reservation, train ticket reservation, and car service related to reservation), business travel data analysis service field, and business travel policy inquiry service field, etc. Correspondingly, the multiple service models based on LLM corresponding to the above multiple service fields managed by the service system can specifically include a first service model for executing a reservation service related to business travel services, a second service model for executing a business travel data analysis service related to business travel services, and a third service model for executing a business travel policy inquiry service related to business travel services, etc.

[0040] It should be noted that the service model based on LLM can specifically include any form of service model that takes LLM as the core computing engine and uses the LLM to provide natural language understanding, generation, reasoning, and decision-making capabilities.

[0041] In some embodiments, the above service model based on LLM can specifically include an artificial intelligence service program based on LLM; for example, it can include an agent (Agent) based on LLM. In this case, the above service model can be an AI assistant for business travel management based on LLM.

[0042] In some embodiments, in addition to being capable of managing a plurality of service models corresponding to a plurality of service domains described above, due to the fact that these service domains can involve some highly specialized structured knowledge, the external knowledge bases accessed by the service system described above can specifically further include a plurality of sub-knowledge bases corresponding to the plurality of service domains. In these sub-knowledge bases, structured knowledge with specialization in the relevant service domain can be specifically stored.

[0043] For example, due to the fact that the service domain involved in the business travel service can require data such as internal regulations of the enterprise regarding business travel as structured knowledge, in this scenario, data such as internal regulations of the enterprise regarding business travel can be processed into structured knowledge and stored in the sub-knowledge base corresponding to each service domain.

[0044] In this way, the structured knowledge stored in these sub-knowledge bases can be used by the subsequent LLM as reasoning context to accurately provide business travel services for the user in accordance with the internal regulations of the enterprise regarding business travel.

[0045] In this specification, when the user has a service demand, the user can input the user's query through the visual interface provided by the service system described above, and the service system described above can receive the user's query input by the user through the visual interface, and further input the user's query to the LLM accessed by the service system for further processing.

[0046] It should be noted that the LLM accessed by the service system can be a general LLM, and the LLM used to build the service model described above can be a specialized LLM obtained by further fine-tuning the general LLM in the relevant service domain. In actual applications, in order to ensure the isolation of services, the LLM accessed by the service system and the LLM used to build each of the plurality of service models described above can be different LLMs.

[0047] Please refer to Figure 3 , Figure 3 for an architecture diagram of a service system based on LLM constructed by the present specification.

[0048] In some embodiments, as Figure 3 shown, the service system described above can implement a user interface module on the user side, which can be used to receive the user's query based on natural language input. When the user has a service demand, the user can input the user's query through the visual interface provided by the service system described above, and the user interface module described above can specifically receive the user's query input by the user through the visual interface, and further submit the user's query to the service system for further processing.

[0049] Step 204: determining a target sub-knowledge base related to the user question from the plurality of sub-knowledge bases, retrieving a target structured knowledge similar in semantics to the user question from the target sub-knowledge base, and obtaining historical context information from historical user questions related to the business travel service input by the user; In some embodiments, the service system described above can also access a processing model for standardizing the user question input by the user; after the service system obtains the user question input by the user, the processing model accessed can be used to standardize the user question, so as to process the user question into a standardized user question.

[0050] It should be noted that in actual application, the user may input the user question based on natural language, which can be diverse and highly ambiguous; for example, the user can use colloquial language, abbreviations, misspelled words, or express his service demand in a brand-new way. The LLM can usually only rely on its internal parameters for probabilistic guessing, which leads to a natural bottleneck in the recognition of ambiguous or unseen user demands, so that the LLM can also have difficulty in accurately identifying the intent when facing novel expressions beyond its knowledge boundary. Therefore, the processing model accessed by the service system described above can customize diversified standardization processing modes based on the characteristics of the user question input by the user, so as to convert the user question input by the user into a standardized user question suitable for processing by the LLM.

[0051] Among them, the diversified standardization processing mode customized for the processing model can be flexibly defined based on specific needs in actual application, which will not be particularly limited in this specification.

[0052] For example, in some embodiments, the standardization processing of the user question input by the user can specifically include a combination of one or more of the following processing: correcting misspelled words contained in the user question; correcting colloquial expressions contained in the user question into standardized expressions; rewriting the user question into a user question with standardized semantics; for example, assuming that the user question input by the user is “book a ticket to Shanghai tomorrow”, it can be corrected and rewritten into the standard question “book a ticket to Shanghai tomorrow”.

[0053] dissolving the user question into multiple sub-user questions; for example, the user intent involved in the user question can be a relatively complex intent, and the complex intent of the user can be dissolved in the form of multiple sub-user questions with coherent semantics.

[0054] Of course, in addition to the processing model that can be introduced to automatically standardize the user's questions input by the user, in actual application, it can also be processed manually by the relevant user.

[0055] In some embodiments, please continue to refer to Figure 3 The service system described above can also implement a standardization processing module on the service side, which can be used to call the processing model to standardize the user's questions input by the user.

[0056] When the user interface module receives the user's questions input by the user through the visual interface, it can further submit the user's questions to the standardization processing module, which further calls the processing model to standardize the user's questions to process the user's questions into standardized user's questions.

[0057] It should be noted that the specific type of the processing model is not particularly limited in this specification, and can be selected flexibly based on specific needs in actual application. For example, in some embodiments, the processing model can be a lightweight LLM, or other forms of special models based on neural networks.

[0058] In this specification, after the service system obtains the user's questions input by the user, on the one hand, the service system can determine the target sub-knowledge base related to the user's questions from the managed multiple sub-knowledge bases, and retrieve the target structured knowledge similar in semantics to the user's questions from the determined target sub-knowledge base.

[0059] In this way, the search range when searching for structured knowledge can be narrowed down; for example, structured knowledge can be searched only in the sub-knowledge base related to the user's questions.

[0060] Among them, the target sub-knowledge base can include one or more sub-knowledge bases related to the user's questions; the service system can use different ways to measure the relevance between the service field of the user's questions and the sub-knowledge base when determining the target sub-knowledge base related to the user's questions from the multiple sub-knowledge bases.

[0061] In one embodiment shown, the service system can first extract question features from the user's questions when determining the target sub-knowledge base related to the user's questions from the multiple sub-knowledge bases; the way of extracting question features from the user's questions and the specific type of the extracted question features are not limited in this specification; for example, the keywords in the user's questions that can express the user's potential needs or the service field to which the user's questions belong can be extracted as question features.

[0062] When the question features are extracted from the user question, a feature vector can be generated based on the extracted question features, and a vector distance between the feature vector and a plurality of description vectors of the sub-knowledge bases can be calculated. The description vectors can include vectors generated based on the description information of the plurality of sub-knowledge bases. The vector distance can be used to describe the similarity between the feature vector and the description vector. In actual applications, the measurement method of the vector distance can be flexibly selected. For example, the cosine distance or other forms of vector distance measurement methods can be used.

[0063] Then, the target sub-knowledge base with the smallest vector distance among the plurality of sub-knowledge bases can be determined as the sub-knowledge base related to the user question.

[0064] In an embodiment, when the service system determines the target sub-knowledge base related to the user question from the plurality of sub-knowledge bases, the service system can first extract the question features from the user question and input the question features into a pre-trained multi-classification model connected to the service system, so that the multi-classification model classifies the question features to determine the sub-knowledge base related to the user question.

[0065] In actual applications, the classification result output by the multi-classification model can represent the sub-knowledge base related to the user question. For example, the classification result output by the multi-classification model can be an identifier of a sub-knowledge base, and the label of the sub-knowledge base can be used as the classification label of the multi-classification model.

[0066] Then, the service system can determine the target sub-knowledge base related to the user question from the plurality of sub-knowledge bases based on the classification result of the question features output by the multi-classification model.

[0067] When the service system retrieves the target structured knowledge with similar semantics to the user question from the determined target sub-knowledge base, different methods can be used to query the structured knowledge.

[0068] In an embodiment, the target sub-knowledge base can include a vector database composed of vectorized structured knowledge. That is, the structured knowledge can be processed into a vector form to create a vector database as a sub-knowledge base. Accordingly, the service system can use vector retrieval to retrieve the target structured knowledge with similar semantics to the user question from the vector database.

[0069] To improve the efficiency of vector retrieval, ANN (Approximate Nearest Neighbor) can be introduced to reduce the query range of vector retrieval in actual applications.

[0070] In this case, the service system can generate a query vector corresponding to the user query when performing vector retrieval, and then run the ANN algorithm to screen a candidate vector set related to the query vector from the vector database corresponding to the target sub-knowledge base. It should be noted that the specific type of the ANN algorithm will not be specifically limited in the present specification, and can be flexibly selected based on specific needs in actual applications. For example, in one example, the HNSW (Hierarchical Navigable Small World) algorithm can be selected as the ANN algorithm. Correspondingly, the specific implementation process of running the ANN algorithm to screen a candidate vector set related to the query vector from the vector database corresponding to the target sub-knowledge base is usually also dependent on the specific type of the selected ANN algorithm, and will not be described in detail in the present specification. Those skilled in the art can refer to the description in the related art.

[0071] After running the ANN algorithm to screen a candidate vector set related to the query vector, the vector distance between the user query and the vectorized structured knowledge in the candidate vector set can be further calculated, and then at least one vectorized structured knowledge in the candidate vector set with the smallest vector distance from the query vector can be selected as the target structured knowledge similar in semantics to the user query.

[0072] In some embodiments, please continue to refer to Figure 3 The service system can further implement a retrieval enhancement module based on RAG (Retrieval-Augmented Generation) technology on the server side. The retrieval enhancement module can be specifically used to determine a target sub-knowledge base related to the user query from the plurality of sub-knowledge bases, and retrieve a target structured knowledge similar in semantics to the user query from the target sub-knowledge base.

[0073] After the standardized query processing module processes the user query into a standardized user query, the standardized user query can be further submitted to the retrieval enhancement module, the retrieval enhancement module generates a query vector corresponding to the user query, and runs an approximate nearest neighbor algorithm to screen a candidate vector set related to the query vector from the vector database, and further retrieves at least one vectorized structured knowledge with the smallest vector distance from the query vector from the candidate vector set as the target structured knowledge similar in semantics to the user query.

[0074] In another aspect, the service system can determine a target sub-knowledge base related to the user question from the plurality of sub-knowledge bases managed, and retrieve a target structured knowledge similar in semantics to the user question from the determined target sub-knowledge base, in addition to obtaining historical context information from historical user questions related to the business travel service input by the user.

[0075] In some embodiments, referring back to Figure 3 The service system can further implement a context memory module on the server side, which can be configured to store and manage historical context information of multiple rounds of historical conversations. In actual applications, the context memory module can receive user questions of each round of conversation input by the user from the user interface module, and store and manage these user questions in a sequence.

[0076] After the standardized processing module processes the user question input by the user into a standardized user question, in addition to generating a question vector corresponding to the user question through the retrieval enhancement module, running an approximate nearest neighbor algorithm to filter a candidate vector set related to the question vector from the vector database, and further retrieving at least one vectorized structured knowledge with the smallest vector distance to the question vector from the candidate vector set as a target structured knowledge similar in semantics to the user question, the service system can further obtain historical context information from historical user questions related to the business travel service maintained in the context memory module.

[0077] In some embodiments, the historical user questions related to the business travel service input by the user can be in the form of a historical user question sequence, which can include historical user questions corresponding to multiple rounds of historical conversations.

[0078] Since the historical conversation rounds included in the user historical question sequence can be relatively large, in order to control the amount of data of the obtained historical context information, the service system can adopt a strategy of retaining details of multiple rounds of historical conversations of recent interactions, and retaining question summaries of multiple rounds of historical conversations of long-term interactions, to obtain historical context information.

[0079] Under this strategy, the finally collected historical context information can include historical user questions corresponding to historical conversations with conversation rounds less than a first threshold in the plurality of rounds of historical conversations, and user question summaries of historical user questions corresponding to historical conversations with conversation rounds greater than the first threshold and less than a second threshold in the plurality of rounds of historical conversations.

[0080] For example, assuming that the first threshold value is denoted as N, and the second threshold value is denoted as M, the finally collected historical context information can specifically include the original content of historical user questions corresponding to the last N rounds of historical sessions; and the user question summaries of historical user questions corresponding to the historical sessions from the Nth round to the Mth round.

[0081] In this case, the first threshold value and the second threshold value can be flexibly set based on requirements. For example, in one example, assuming that N takes the value of 5, and M takes the value of 10, the finally collected historical context information can specifically include the original content of historical user questions corresponding to the last 5 rounds of historical sessions; and the user question summaries of historical user questions corresponding to the historical sessions from the 5th round to the 10th round.

[0082] At this time, when the service system acquires historical context information from historical user questions input by the user and related to the business travel service, it can specifically first acquire the original content of historical user questions corresponding to historical sessions with a session round less than or equal to the first threshold value from the historical user question sequence, and then acquire historical user questions corresponding to historical sessions with a session round greater than the first threshold value and less than the second threshold value from the historical user question sequence, and calculate the user question summary of the historical user questions.

[0083] The user question summary can specifically include any form of summary that can describe the core content of the historical user questions. For example, it can be a more concise user question rewritten based on the key fragments in the historical user questions.

[0084] In some embodiments, for any historical session with a session round greater than the first threshold value and less than the second threshold value, in order to make the finally generated user question summary corresponding to the historical session with the session round semantically coherent and not deviate from the user summary information corresponding to the historical session with the previous session round, the historical user question corresponding to the session round and the user question summary of the historical user question corresponding to the historical session with the previous session round of the session round can be jointly used as input data to calculate the user question summary corresponding to the historical session with the session round.

[0085] In this case, when the service system calculates the user question summary of the historical user question corresponding to the session round, it can first acquire the historical user question corresponding to the session round from the historical user question sequence, and acquire the user question summary of the historical user question corresponding to the historical session with the previous session round of the session round; then, based on the historical user question corresponding to the session round and the user question summary of the historical user question corresponding to the historical session with the previous session round of the session round, the user question summary corresponding to the historical user question of the session round can be further calculated.

[0086] The specific manner in which the summary of the user question is calculated is not specifically shown in the specification.

[0087] For example, in some embodiments, the summary of the user question can be generated by introducing a recurrent neural network capable of processing sequence data in the service system.

[0088] In this case, when calculating the summary of the historical user question corresponding to the session round, the service system can specifically further input the historical user question corresponding to the session round and the summary of the historical user question corresponding to the previous session round of the session round to the pre-trained recurrent neural network, to further generate the summary of the historical user question corresponding to the session round based on the historical user question and the summary of the user question by the recurrent neural network.

[0089] It should be noted that the specific type of the recurrent neural network will not be specifically limited in the specification, and in actual applications, it can be selected flexibly based on specific needs; for example, in some embodiments, the recurrent neural network can specifically include a GRU (Gated Recurrent Unit, Gated Recurrent Unit) network.

[0090] Step 206: based on the user question, the target structured knowledge and the historical context information, generate a prompt word for guiding the LLM to judge the user's user intent, and input the prompt word into the LLM to generate structured intent description information by the LLM based on the prompt word; wherein the intent description information is used to describe the user's user intent; In the specification, after the service system obtains the user input user question, the target structured knowledge and the historical context information, it can further generate a prompt word (prompt) as an inference context; the prompt word can be specifically used to guide the LLM to judge the user's user intent based on the inference context contained in the prompt word.

[0091] In some embodiments, the service system can specifically support generating a prompt word based on a maintained structured prompt word template; wherein the structured prompt word template can specifically include fields for maintaining the user input user question, the obtained target structured knowledge and the historical context information, respectively; In this case, the service system can specifically fill the user question, the target structured knowledge and the historical context information into the corresponding fields of the structured prompt template to generate a prompt for guiding the LLM to determine the user's user intent when generating the prompt.

[0092] In some embodiments, referring to Figure 3 the service system can also implement a prompt construction module on the server side; the prompt construction module is specifically configured to fill the standardized user question output by the standardization processing module, the historical context information obtained from the context memory module and the target structured knowledge retrieved by the retrieval enhancement module into the corresponding fields of the structured prompt template to generate a prompt for guiding the LLM to determine the user's user intent.

[0093] Among them, the structured prompt template can be customized based on experience in actual application, or a general prompt framework can also be used.

[0094] In some embodiments, the prompt template can specifically use a general CO-STAR prompt template.

[0095] CO-STAR is a structured prompt framework designed specifically for LLMs, aiming to help users interact with LLMs more effectively, thereby obtaining higher quality and more expected output. This prompt framework is particularly suitable for scenarios that require precise and professional output, and is an effective method to improve collaboration efficiency with AI. The name of the framework is an acronym, representing the seven key elements that should be included in the prompt.

[0096] The key elements in the prompt included in the CO-STAR framework are: C-Context (background) field: provides background information for the task. This element aims to help the LLM understand the current context, history or related circumstances, so that its output is more targeted.

[0097] O-Objective (objective) field: clearly states the specific goal you hope to achieve through this interaction. This element can provide a clear goal for the LLM, helping it focus on key issues and avoid deviating from the topic.

[0098] S-Style (style) field: specifies the language style you expect the LLM to output.

[0099] T-Tone (tone) field: specifies the tone you expect the LLM to use when responding. For example: professional, objective expectations, etc.

[0100] The A-Audience field specifies the audience for the LLM output, that is, who the output is ultimately intended for. This element helps LLM adjust the depth of language and explanation by clearly defining the audience.

[0101] The R-Response field specifies the output format or structure you expect from the LLM.

[0102] In some embodiments, in order to allow the prompt words built based on the CO-STAR framework to also accommodate user-inputted questions and the aforementioned historical context information, the existing fields in the CO-STAR framework can be extended. A first extended field for maintaining the aforementioned user-inputted questions and a second extended field for maintaining the aforementioned historical context information can be added to the CO-STAR prompt word template.

[0103] In this scenario, when generating prompts based on the CO-STAR prompt template, the service system can populate the aforementioned user questions into the first extended field included in the CO-STAR prompt template, and the aforementioned historical context information into the second extended field included in the CO-STAR prompt template. For the aforementioned target structured knowledge, it can be populated into the target base field specified from the base fields included in the CO-STAR prompt template. After completing the above population, a structured prompt can be constructed based on the CO-STAR framework to guide the LLM in determining the user's intent.

[0104] The aforementioned target base field can be any one of the seven fields already available in the CO-STAR framework.

[0105] For example, in some embodiments, the aforementioned target base field may specifically include the Context field contained in the CO-STAR prompt template. In this case, the prompt built based on the CO-STAR framework can adopt the following structure: The Context field explicitly informs the LLM that its role is an intent-based expert, and maintains the retrieved structured knowledge of the aforementioned target in this field; The Objective field clearly defines the task as determining the user's intent based on the user's question, the structured knowledge of the objectives mentioned above, and the historical context information mentioned above. Style field: Requires the output to be rigorous and precise; Tone field: Requires professional and objective output; Audience field: the audience of the output result can be an intent distributor implemented in the service system; the intent distributor can be specifically used for parsing the user intent of the LLM output, and distributing the user query to a correct service model; Response field: the LLM is forced to output the intent in a structured data format, which is used for describing the intent description information of the user intent judged by the LLM; for example, output in JSON format; User Query field: the field is the first extended field described above, and is used for maintaining the user query input by the user; for example, the standardized user query described above; History field: the field is the second extended field described above, and is used for maintaining the historical context information described above.

[0106] In the present specification, after the service system generates the prompt word based on the user query input by the user, the target structured knowledge and the historical context information described above, the service system can further input the prompt word to the LLM accessed by the service system, so that the LLM judges the user intent based on the reasoning context contained in the prompt word by performing reasoning calculation, and generates structured intent description information based on the judged user intent; wherein the intent description information can be specifically used for describing the user intent judged by the LLM.

[0107] For example, in some embodiments, the LLM can be required to output the intent description information in JSON format in the prompt word.

[0108] In some embodiments, referring to Figure 3 , the service system can also implement an LLM reasoning engine on the server based on the accessed LLM; the reasoning engine is specifically used for performing reasoning calculation based on the reasoning context contained in the prompt word generated by the prompt word construction module, judging the user intent of the user, and generating structured intent description information based on the judged user intent.

[0109] Step 208: analyzing the intent description information, determining a target service field corresponding to the user intent of the user, and distributing the user query to a target service model corresponding to the target service field, so that the target service model performs service processing corresponding to the business travel service on the user query.

[0110] In the present specification, when the LLM outputs the structured intent description information, the service system can analyze the structured intent description information, determine a target service field corresponding to the user intent of the user, and distribute the user query input by the user to a target service model corresponding to the target service field, so that the target service model performs service processing on the user query.

[0111] It should be noted that the specific content contained in the above structured intent description information is not specifically limited in the present specification, and in actual application, any form of information related to the user's intent can be contained.

[0112] In some embodiments, the above structured intent description information can specifically include: an intent field for maintaining the user intent judged by the LLM; and an entity field for maintaining at least one service entity related to the user's intent extracted from the user's question; For example, taking the above structured intent description information as JSON format intent description information as an example, at this time the intent description information can be the following data structure: {"intent":"XXX", "entities":"XXX"}; wherein intent represents the above intent field; entities represent the above entity field.

[0113] In this case, the service system can parse the above structured intent description information, obtain the user intent maintained in the above intent field, and obtain at least one service entity maintained in the above at least one entity field; then, determine the target service field corresponding to the obtained user intent, and distribute the above user question and the above at least one service entity to the target service model corresponding to the target service field, so that the target service model performs service processing on the above user question and the above at least one service entity.

[0114] In some embodiments, the above prompt words can also be used to guide the LLM to judge the user's user intent, and determine the target service field corresponding to the user's user intent; For example, still taking the CO-STAR framework to build the prompt word as an example, at this time the Objective field contained in the built prompt word can no longer define the task as judging the user's user intent according to the user's question, the above target structured knowledge and the above historical context information; but is defined as judging the user's user intent according to the user's question, the above target structured knowledge and the above historical context information, and determining the target service field corresponding to the user's user intent.

[0115] In this case, the function of the intent field included in the structured intent description information described above will also change, and at this time the intent field is specifically used to maintain a target service field corresponding to the user intent of the user; when the service system parses the structured intent description information described above, it can directly obtain the target service field corresponding to the user intent of the user maintained in the intent field, and obtain at least one service entity maintained in the at least one entity field, and then the user question and the at least one service entity can be directly distributed to the target service model corresponding to the target service field obtained, so that the target service model performs service processing on the user question and the at least one service entity.

[0116] In some embodiments, please continue to refer to Figure 3 The service system described above can also implement an intent distributor on the service side; the intent distributor can be specifically used to parse the user intent output by the LLM and distribute the user question to the correct service model. At this time, the plurality of service models can be Agents constructed based on the LLM, and the service system can maintain an Agent pool on the service side to manage a plurality of Agents corresponding one by one to the plurality of service fields through the Agent pool.

[0117] When the LLM outputs the structured intent description information, the intent distributor can parse the structured intent description information, obtain the user intent maintained in the intent field or the target service field corresponding to the user intent, and obtain at least one service entity maintained in the at least one entity field; then, a target Agent corresponding to the target service field is determined from the Agent pool, and the user question and the at least one service entity are distributed to the target Agent corresponding to the target service field for service processing.

[0118] When the target service model performs service processing on the user question and the at least one service entity, the service result can be output and displayed to the user.

[0119] In some embodiments, please continue to refer to Figure 3 The user interface module implemented in the service system can be used to receive the user question of the user based on the natural language input, and can also be used to output and display the service result output by the target service model to the user.

[0120] It should be noted that Figure 3 Each module shown can also be an independent subsystem or other form of execution unit in actual application, and is not specifically limited in the present specification.

[0121] In the above technical solution, in the service system constructed based on the LLM, the inference ability of the LLM is utilized to take the user question input by the user, the structured knowledge retrieved from the knowledge base related to the user question input by the user, and the historical context information related to the historical user question of the user and other external evidence as inference context to perform inference calculation, to determine the user intention of the user, and determine the target service field corresponding to the user intention of the user, and then distribute the user question to the target service model corresponding to the target field among the plurality of service models managed by the service system for service processing, so that when the user question input by the user involves service demand in a professional service field, the user question can be accurately distributed to the service model related to the service field, and the service model in the field is utilized to provide services to the user.

[0122] For example, the present specification creatively redefines the distribution problem of the user question as an "inference decision problem based on multiple types of inference context", and by combining multiple inference contexts, the inference ability of the LLM can be utilized to generate structured intention description information that can represent the user intention, so that by analyzing the structured intention description information, the user question can be accurately distributed to the field expert model corresponding to the professional service field required by the user for processing.

[0123] Moreover, by using multiple service models corresponding to different service fields to provide services to the user, it is no longer necessary to use only a single service model to provide services to the user, so that the accuracy, reliability and professionalism of the services provided to the user can be improved, and the problems of low service accuracy, poor reliability and poor professionalism that exist when a single universal service model is used to provide services to the user in multiple service fields can be avoided.

[0124] Corresponding to the embodiments of the foregoing method, the present specification also provides embodiments of an apparatus, an electronic device and a storage medium.

[0125] Figure 4 FIG. 1 is a schematic structural diagram of an electronic device provided by an example embodiment. Please refer to Figure 4At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and can also include other required hardware. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 910 into the memory 408 by the processor 402 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0126] As shown in Figure 5 , Figure 5 is a block diagram of an LLM-based business travel service management apparatus according to an exemplary embodiment of the present specification, which can be applied to an electronic device as shown in Figure 4 to implement the technical solutions of the present specification. Wherein, the service system provides business travel services for users, including multiple service fields; the external knowledge base accessed by the service system contains multiple sub-knowledge bases corresponding to the multiple service fields; the service system manages multiple service models corresponding to the multiple service fields; the service model includes a service model constructed based on LLM; the apparatus includes: The receiving module 501 receives the user input of the user question related to the business travel service; The input module 502 determines the target sub-knowledge base related to the user question from the multiple sub-knowledge bases, retrieves the target structured knowledge similar in semantics to the user question from the target sub-knowledge base, and obtains historical context information from historical user questions related to the business travel service input by the user; The generation module 503 generates a prompt word for guiding the LLM to judge the user's user intent based on the user question, the target structured knowledge, and the historical context information, and inputs the prompt word into the LLM to generate structured intent description information by the LLM based on the prompt word. Inference calculation; wherein the intent description information is used to describe the user's user intent; The distribution module 504 analyzes the intent description information, determines the target service field corresponding to the user's user intent, and distributes the user question to the target service model corresponding to the target service field to process the user question by the target service model. Service processing.

[0127] Correspondingly, the specification also provides an electronic device, comprising a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps in all the method flows described previously.

[0128] Correspondingly, the specification also provides a computer-readable storage medium having stored thereon executable computer program instructions; wherein the instructions, when executed by a processor, implement the steps in all the method flows described previously.

[0129] Correspondingly, the specification also provides a computer program product having stored thereon executable computer program instructions; wherein the computer program instructions, when executed by a processor, implement the steps in all the method flows described previously.

[0130] The system, apparatus, module or unit illustrated in the above embodiments can be implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a server system. Of course, it does not exclude that with the development of future computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an electronic mail device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0131] Although the specification one or more embodiments provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps can be included based on conventional or non-inventive means. The order of steps listed in the embodiments is only one of the many step execution orders, and does not represent the only execution order. In actual device or terminal product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, parallel processor or multi-threaded environment, or even distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, product or device. Without more limitations, it does not exclude that the process, method, product or device including the described elements also has other same or equivalent elements. For example, if the first, second, etc. terms are used to represent names, they do not represent any particular order.

[0132] For ease of description, the above apparatus is described in various modules with functions respectively. Of course, functions of the modules can be implemented in one or more software and / or hardware in implementing one or more of the present specification, and the modules with the same function can be implemented by a combination of a plurality of sub-modules or sub-units, etc. The apparatus embodiments described above are merely schematic, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, and can be electrical, mechanical or other forms.

[0133] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0134] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0135] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0136] In a typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0137] Memory can include, without limitation, non-removable memory, random access memory (RAM), and / or non-volatile memory, e.g., read-only memory (ROM), or flash memory. Memory is an example of computer-readable media.

[0138] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, without limitation, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, graphene storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0139] Those skilled in the art will appreciate that one or more embodiments of the specification can be provided as methods, systems or computer program products. Therefore, one or more embodiments of the specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, without limitation, magnetic disk storage, CD-ROM, optical storage and the like) containing computer-usable program code.

[0140] One or more embodiments of the specification can be described in the general context of computer-executable instructions, such as program modules, being executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0141] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the system embodiments are described simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments. In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the specification. The illustrative description of the above terms in the specification does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0142] The above only describes the embodiments of one or more embodiments of the specification and does not limit the one or more embodiments of the specification. The one or more embodiments of the specification can have various changes and variations for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the specification shall be included in the scope of claims.

Claims

1. A business travel service management method based on LLM, applied to a service system constructed based on LLM; wherein, The service system provides business travel services to users, including multiple service fields; and an external knowledge base accessed by the service system includes multiple sub-knowledge bases corresponding to the multiple service fields; The service system manages multiple service models corresponding to the multiple service fields; The service model includes a service model based on an LLM; and the method includes: receiving a user question input by a user and related to the business travel service; determining a target sub-knowledge base related to the user question from the multiple sub-knowledge bases, retrieving target structured knowledge similar in semantics to the user question from the target sub-knowledge base, and obtaining historical context information from historical user questions related to the business travel service input by the user; generating a prompt word for guiding the LLM to determine a user intent of the user based on the user question, the target structured knowledge, and the historical context information, and inputting the prompt word into the LLM to generate structured intent description information by the LLM based on the prompt word; wherein the intent description information is used to describe the user intent of the user; parsing the intent description information, determining a target service field corresponding to the user intent of the user, and distributing the user question to a target service model corresponding to the target service field, so that the target service model performs service processing corresponding to the business travel service on the user question.

2. The method of claim 1, wherein determining a target sub-knowledge base related to the user question from the multiple sub-knowledge bases includes: extracting question features from the user question; generating a feature vector based on the question features and calculating a vector distance between the feature vector and a description vector of the multiple sub-knowledge bases; wherein the description vector includes a vector generated based on description information of the multiple sub-knowledge bases; and determining a target sub-knowledge base with the smallest vector distance as the sub-knowledge base related to the user question; or extracting question features from the user question; inputting the question features into a pre-trained multi-classification model accessed by the service system to perform classification processing on the question features by the multi-classification model; wherein a classification result output by the multi-classification model represents a sub-knowledge base related to the user question; and determining a target sub-knowledge base related to the user question from the multiple sub-knowledge bases based on the classification result of the question features output by the multi-classification model.

3. The method of claim 1, wherein the sub-knowledge base includes a vector database composed of vectorized structured knowledge; retrieving target structured knowledge similar in semantics to the user question from the target sub-knowledge base includes: generating a question vector corresponding to the user question; running an approximate nearest neighbor algorithm to filter a candidate vector set related to the question vector from the vector database; further calculating vector distances between the user question and vectorized structured knowledge in the candidate vector set; and at least one vectorized structured knowledge in the candidate vector set with the smallest vector distance to the question vector as target structured knowledge similar in semantics to the user question.

4. The method of claim 1, the user-entered historical user query related to the business travel service comprising: a historical user question sequence composed of historical user questions corresponding to the multi-round historical conversation; the historical context information includes: historical user questions corresponding to historical conversations with a conversation round less than a first threshold in the multi-round historical conversation; and a user question summary of historical user questions corresponding to historical conversations with a conversation round greater than the first threshold and less than a second threshold in the multi-round historical conversation; the historical context information is obtained from the historical user questions input by the user related to the business travel service, including: historical user questions corresponding to historical conversations with a conversation round less than or equal to a first threshold are obtained from the historical user question sequence; and, historical user questions corresponding to historical conversations with a conversation round greater than the first threshold and less than a second threshold are obtained from the historical user question sequence, and a user question summary of the historical user questions is calculated.

5. The method of claim 1, wherein the service system accesses a processing model for standardizing the user question; before generating the prompt word for guiding the LLM to determine the user intent of the user based on the user question, the target structured knowledge, and the historical context information, further comprising: standardizing the user question based on the accessed processing model to process the user question into a standardized user question; wherein the standardization processing performed on the user question includes one or more combinations of the following: correcting misspelled words contained in the user question; correcting colloquial expressions contained in the user question to standardized expressions; rewriting the user question into a user question with standardized semantics; disassembling the user question into multiple sub-user questions.

6. The method of claim 1, the service system supports generating a prompt word based on a structured prompt word template maintained; wherein, the structured prompt word template contains fields for maintaining the user question, the target structured knowledge, and the historical context information, respectively; generating the prompt word for guiding the LLM to determine the user intent of the user based on the user question, the target structured knowledge, and the historical context information, including: filling the user question, the target structured knowledge, and the historical context information into the corresponding fields in the structured prompt word template, respectively, to generate the prompt word for guiding the LLM to determine the user intent of the user.

7. The method of claim 6, the prompt word template comprising a CO-STAR prompt word template; wherein, the CO-STAR prompt word template contains a first extended field for maintaining the user question and a second extended field for maintaining the historical context information; generating the prompt word for guiding the LLM to determine the user intent of the user based on the user question, the target structured knowledge, and the historical context information, including: filling the user question into the first extended field contained in the CO-STAR prompt word template; filling the historical context information into the second extended field contained in the CO-STAR prompt word template; filling the target structured knowledge into a target base field specified from a base field contained in the CO-STAR prompt template to generate a prompt word for guiding the LLM to judge a user intention of the user. 8.The method of claim 1, wherein the service model comprises an artificial intelligence service program constructed based on the LLM. The plurality of service domains comprises a reservation service domain related to the business travel service, a business travel data analysis service domain, and a business travel policy inquiry service domain; and the plurality of service models comprises a first artificial intelligence service program for performing a reservation service related to the business travel service, a second artificial intelligence service program for performing a business travel data analysis service related to the business travel service, and a third artificial intelligence service program for performing a business travel policy inquiry service related to the business travel service.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 8. 10.A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any one of claims 1 to 8.

Images