A Business Travel Service Management Method and Computer Equipment Based on LLM

Abstract

A business travel service management method based on LLM is applied to a service system built on LLM. The service system provides services to users that include multiple service domains. The service system manages multiple service models that correspond one-to-one with the multiple service domains. The method includes: receiving user input related to business travel services; retrieving target structured knowledge from the target sub-knowledge base related to the user's question from multiple sub-knowledge bases that correspond one-to-one with the multiple service domains, and obtaining historical context information from the user's historical input questions; generating prompt words based on the user's question, target structured knowledge, and historical context information, and inputting the prompt words into the LLM so that the LLM can generate structured intent description information based on the prompt words; parsing the intent description information, determining the target service domain corresponding to the user's intent, and distributing the user's question to the target service model corresponding to the target service domain.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a business travel service management method and computer equipment based on LLM. Background Technology

[0002] On some enterprise-level service platforms, to better manage employees' daily business travel activities, the platform can provide users with a business travel service page offering end-to-end business travel services. Through this page, users can complete online tasks such as booking services related to their business travel (e.g., flight bookings, hotel bookings, train ticket bookings, and car rentals), expense reimbursement, and more.

[0003] With the continuous development of LLM (Large Language Model) technology, building a business travel service system based on LLM to provide intelligent business travel services to users in the form of question and answer has gradually evolved into the mainstream service model.

[0004] However, in related technologies, business travel service systems built on LLM typically employ end-to-end solutions based on a single LLM. This approach has significant limitations when dealing with highly complex user inquiries or those involving highly specialized service areas. Summary of the Invention

[0005] This specification proposes an LLM-based business travel service management method, applied to a service system built on LLM; wherein, the business travel services provided to users by the service system include multiple service domains; the external knowledge base accessed by the service system contains multiple sub-knowledge bases corresponding one-to-one with the multiple service domains; the service system manages multiple service models corresponding one-to-one with the multiple service domains; the service models include service models built on LLM; the method includes:

[0006] Receive user inquiries related to the business travel services;

[0007] The target sub-knowledge base related to the user's question is determined from the plurality of sub-knowledge bases, target structured knowledge that is semantically similar to the user's question is retrieved from the target sub-knowledge base, and historical context information is obtained from the user's historical user questions related to the business travel service.

[0008] Based on the user's question, the target structured knowledge, and the historical context information, prompt words are generated to guide the LLM in determining the user's intent. These prompt words are then input into the LLM, which performs inference calculations based on them to generate structured intent description information. The intent description information describes the user's intent.

[0009] The intent description information is parsed to determine the target service domain corresponding to the user's intent, and the user's question is distributed to the target service model corresponding to the target service domain, so that the target service model can perform service processing on the user's question corresponding to the business travel service.

[0010] Optionally, a target sub-knowledge base related to the user's question is determined from the plurality of sub-knowledge bases, including:

[0011] Extract question features from the user's question; generate feature vectors based on the question features, and calculate the vector distance between the feature vectors and the description vectors of the multiple sub-knowledge bases; wherein the description vectors include vectors generated based on the description information of the multiple sub-knowledge bases; determine the target sub-knowledge base with the smallest vector distance as the sub-knowledge base related to the user's question; or,

[0012] Extract question features from the user's question; input the question features into a pre-trained multi-classification model accessed by the service system, so that the multi-classification model can classify the question features; wherein, the classification result output by the multi-classification model represents a sub-knowledge base related to the user's question; based on the classification result of the question features output by the multi-classification model, determine a target sub-knowledge base related to the user's question from the multiple sub-knowledge bases.

[0013] Optionally, the sub-knowledge base includes a vector database composed of vectorized structured knowledge;

[0014] Retrieving semantically similar target structured knowledge from the target sub-knowledge base that is similar to the user's question includes:

[0015] Generate a question vector corresponding to the user's question;

[0016] Run the approximate nearest neighbor algorithm to filter a set of candidate vectors related to the query vector from the vector database;

[0017] Further calculate the vector distance between the user's question and the vectorized structured knowledge in the candidate vector set;

[0018] The structured knowledge vector that has the smallest vector distance to the question vector from the candidate vector set is taken as the target structured knowledge that is semantically similar to the user's question.

[0019] Optionally, the historical user questions related to the business travel service input by the user include: a sequence of historical user questions consisting of historical user questions corresponding to multiple rounds of historical conversations;

[0020] The historical context information includes: historical user questions corresponding to historical sessions in which the number of session rounds in the multi-round historical sessions is less than a first threshold; and user question summaries of historical user questions corresponding to historical sessions in which the number of session rounds in the multi-round historical sessions is greater than the first threshold and less than a second threshold.

[0021] Optionally, historical context information can be obtained from historical user queries related to the business travel service input by the user, including:

[0022] Extract historical user questions from the historical user question sequence corresponding to historical sessions with session rounds less than or equal to a first threshold; and...

[0023] From the historical user question sequence, obtain the historical user questions corresponding to historical sessions with session rounds greater than a first threshold and less than a second threshold, and calculate the user question summary of the historical user questions.

[0024] Optionally, the service system incorporates a processing model for standardizing the user's query.

[0025] Before generating prompts to guide the LLM in determining the user's intent based on the user's question, the target structured knowledge, and the historical context information, the method further includes:

[0026] The user question is standardized based on the access processing model to transform it into a standardized user question.

[0027] The standardization process performed on the user's question includes one or more of the following:

[0028] Correct the typos contained in the user's question;

[0029] The colloquial expressions in the user questions were corrected to standardized expressions;

[0030] Rewrite the user question into a user question with standardized semantics;

[0031] The user's question is broken down into multiple sub-user questions.

[0032] Optionally, the service system supports generating prompts based on a maintained structured prompt template; wherein the structured prompt template includes fields for maintaining the user's question, the target structured knowledge, and the historical context information;

[0033] Based on the user's question, the target structured knowledge, and the historical context information, prompts are generated to guide the LLM in determining the user's intent, including:

[0034] The user's question, the target structured knowledge, and the historical context information are respectively filled into the corresponding fields in the structured prompt word template to generate prompt words used to guide the LLM to determine the user's intent.

[0035] Optionally, the prompt template includes a CO-STAR prompt template; wherein the CO-STAR prompt template includes a first extended field for maintaining the user's question and a second extended field for maintaining the historical context information;

[0036] Based on the user's question, the target structured knowledge, and the historical context information, prompts are generated to guide the LLM in determining the user's intent, including:

[0037] The user's question is populated into the first extended field contained in the CO-STAR prompt template;

[0038] The historical context information is populated into the second extended field contained in the CO-STAR prompt template;

[0039] The target structured knowledge is populated into the target base fields specified from the base fields included in the CO-STAR prompt template to generate prompts for guiding the LLM to determine the user's intent.

[0040] Optionally, the service model includes an artificial intelligence service program built on an LLM basis;

[0041] The multiple service areas include the booking service area, the business travel data analysis service area, and the business travel policy inquiry service area related to the business travel service; the multiple service models include a first artificial intelligence service program for performing the booking service related to the business travel service, a second artificial intelligence service program for performing the business travel data analysis service related to the business travel service, and a third artificial intelligence service program for performing the business travel policy inquiry service related to the business travel service.

[0042] In the above embodiments, in a service system built on LLM, the inference capabilities of LLM are utilized to perform inference calculations using various external evidence, such as user-inputted questions, structured knowledge retrieved from a knowledge base related to the user-inputted questions, and historical context information related to the user's past questions, to determine the user's intent and identify the target service domain corresponding to the user's intent. The user question is then distributed to the target service model corresponding to the target domain among multiple service models managed by the service system for service processing. This allows for accurate distribution of user questions to service models related to specific service domains when the user-inputted questions involve service needs within those domains, enabling targeted service provision to the user using those domain-specific service models.

[0043] Moreover, by employing multiple service models corresponding to different service areas to provide services to users, it is no longer necessary to use only a single service model to provide services to users. This can improve the accuracy, reliability, and professionalism of the services provided to users, and avoid the problems of low service accuracy, poor reliability, and poor professionalism that occur when using a single general service model to provide services to users in multiple service areas at the same time. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the architecture of an Internet service system shown in one embodiment of this specification;

[0046] Figure 2 This is a flowchart illustrating an LLM-based business travel service management method in one embodiment of this specification;

[0047] Figure 3 This is an architecture diagram of a service system built on an LLM-based architecture, as shown in one embodiment of this specification.

[0048] Figure 4 This is a schematic structural diagram of an electronic device shown in one embodiment of this specification;

[0049] Figure 5 This is a block diagram of an LLM-based business travel service management device shown in one embodiment of this specification. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0051] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0052] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0053] While service systems built on a single LLM can provide more accurate service results for users' queries by leveraging the LLM's reasoning capabilities compared to traditional service systems (such as rule-based or keyword-matching systems), they still have certain limitations for more complex services, especially composite services that involve multiple service domains.

[0054] For example, user services provided by a service system may typically encompass multiple service domains. These service domains are often highly heterogeneous and specialized, and may involve different knowledge bases. In this case, if a service system built on a single LLM simultaneously provides services from all service domains to users, the LLM must encode the knowledge and logic of all service domains within its limited parameter space. This leads to the following problems:

[0055] Firstly, there is the issue of semantic confusion and performance degradation:

[0056] Because the semantic spaces of different service domains overlap or even conflict, forcibly merging the semantic spaces of multiple service domains into a single LLM may lead to a significant decrease in the accuracy of the LLM's decision-making when providing services to users in a specific domain. For example, an LLM might misinterpret a user's question in service domain A as a service request in service domain B.

[0057] Secondly, the problem of "catastrophic forgetting" of knowledge:

[0058] When fine-tuning the LLM for a specific service area among the above multiple service areas, it may cause the LLM to lose its general service capabilities in other service areas.

[0059] Third, the problem of lack of factual consistency and reliability:

[0060] Some service areas may involve highly specialized and difficult-to-obtain structured knowledge (such as highly privatized structured knowledge like internal company rules and regulations). This may result in a lack of relevant training data when fine-tuning the LLM in this service area. In such cases, the fine-tuned LLM may be prone to "hallucination," providing incorrect or fabricated information.

[0061] It is evident that relying on a single, general-purpose LLM to provide users with highly specialized, multi-service-area composite services may result in low service accuracy, poor reliability, and a lack of professionalism.

[0062] Based on this, this specification proposes a technical solution that introduces multiple LLM-based service models, each corresponding to a specific service domain, into a service system as domain expert models. By leveraging the reasoning capabilities of LLM, various external evidence is used as reasoning contexts for inference calculations to determine the user's intent. Based on the user's intent, the user's query is accurately distributed to the domain expert model corresponding to the service domain related to the user's intent for service processing.

[0063] Figure 1 This is a schematic diagram of the architecture of an Internet service system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices, such as a PC (Personal Computer) 13, a mobile phone 14, etc.

[0064] Server 11 can be a physical server containing an independent host, or server 11 can be a virtual server hosted in a host cluster. During operation, server 11 can run a server-side program of a certain service program (such as an artificial intelligence service program built based on LLM); for example, when server 11 runs the server-side program of the service program, it can act as the server of the service program.

[0065] PC13 and mobile phone 14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can run a client-side program of a service program to implement the relevant functions of the application. For example, when the electronic device runs a user-side program of the service program, it can act as a client of the service program.

[0066] The client-side program of the aforementioned service program can be launched and run on an electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form.

[0067] Of course, when using web technologies such as HTML5 or similar, the relevant functions can be achieved through the page displayed by the browser. The browser here can be a standalone browser application or a browser module embedded in some applications.

[0068] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0069] It should be noted that the services provided by the above service program can include any service implemented on the Internet; for example, the services provided by the above service program can specifically include business travel services provided to employees on the enterprise's service platform.

[0070] The technical solution of this specification will be described in detail below with reference to the accompanying drawings.

[0071] Please see Figure 2 , Figure 2The flowchart illustrating an LLM-based business travel service management method in this specification includes the following execution process:

[0072] Step 202: Receive user questions related to the business travel service;

[0073] The entity that executes the above method may specifically include a service system built on LLM.

[0074] In practical applications, this service system can specifically include server-side programs and client-side programs. The server-side programs and client-side programs work together to provide users with trip planning-related user services.

[0075] For example, in some embodiments, the execution of the above method process can be specifically the server program of the above service system or the client program of the above service system; or, some steps in the above method can run on the server program and some steps can run on the client program, and the process corresponding to the above method can be jointly executed through the mutual cooperation of the server program and the client program.

[0076] The services provided to users by the aforementioned service system can specifically include any form of service that can be implemented on the Internet, and will not be specifically limited in this specification. For example, in some embodiments, the services provided to users by the aforementioned service system may specifically include business travel services.

[0077] In some embodiments, the services provided to users by the service system may include multiple service domains; the service system may manage multiple service models corresponding one-to-one with the multiple service domains as domain expert models; the service models may specifically be service models built based on LLM.

[0078] For example, in some embodiments, the aforementioned service system may specifically be an enterprise-level service platform. The multiple service areas may specifically include reservation services related to business travel (such as flight bookings, hotel bookings, train ticket bookings, and car rental services), business travel data analysis services, and business travel policy inquiry services, etc. Correspondingly, the service system manages multiple LLM-based service models that correspond one-to-one with the aforementioned service areas. These may specifically include a first service model for performing reservation services related to business travel, a second service model for performing business travel data analysis services related to business travel, a third service model for performing business travel policy inquiry services related to business travel, etc.

[0079] It should be noted that the service model built on LLM can specifically include any form of service model that uses LLM as the core computing engine and is responsible for providing natural language understanding, generation, reasoning and decision-making capabilities.

[0080] In some embodiments, the service model built on LLM may specifically include an AI service program built on LLM; for example, it may include an agent built on LLM. In this case, the service model may specifically be an AI assistant for business travel management built on LLM.

[0081] In some embodiments, in addition to managing multiple service models corresponding one-to-one with the aforementioned multiple service domains, the service system may also include multiple sub-knowledge bases corresponding one-to-one with the multiple service domains in the external knowledge base accessed by the service system, since these service domains may involve some highly specialized structured knowledge. These sub-knowledge bases can specifically store specialized structured knowledge related to the relevant service domains.

[0082] For example, since the business travel service involves various service areas, it may be necessary to use internal company rules and regulations regarding business travel as structured knowledge. Therefore, in this scenario, the internal company rules and regulations regarding business travel can be processed into structured knowledge and stored in sub-knowledge bases corresponding to each service area.

[0083] In this way, the LLM can then use the structured knowledge stored in these sub-knowledge bases as reasoning context to provide users with precise business travel services in accordance with the company's internal rules and regulations regarding business travel.

[0084] In this specification, when a user has a service request, they can input their question through the visual interface provided by the aforementioned service system. The service system can then receive the user's question through this visual interface and further input it into the LLM accessed by the service system for further processing.

[0085] It is important to emphasize that the LLM used for service system access can be a general-purpose LLM, while the LLM used to construct the aforementioned service models can be a specialized LLM obtained by further fine-tuning and training the general-purpose LLM in the relevant service domain. In practical applications, to ensure service isolation, the LLM used for service system access and the LLM used to construct the various service models mentioned above can be different LLMs.

[0086] Please see Figure 3 , Figure 3This document presents an architecture diagram of a service system built on an LLM model.

[0087] In some embodiments, such as Figure 3 As shown, the aforementioned service system can implement a user interface module on the user's end. This user interface module can be used to receive user queries based on natural language input. When a user has a service request, they can input their query through the visual interface provided by the service system. The user interface module can then receive the user's query through this visual interface and submit it to the service system for further processing.

[0088] Step 204: Determine the target sub-knowledge base related to the user's question from the plurality of sub-knowledge bases, retrieve the target structured knowledge that is semantically similar to the user's question from the target sub-knowledge base, and obtain historical context information from the historical user questions related to the business travel service input by the user;

[0089] In some embodiments, the service system may also access a processing model for standardizing user-input questions; after the service system obtains the user-input question, it can standardize the user-input question based on the accessed processing model to process the user-input question into a standardized user-input question.

[0090] It's important to note that in practical applications, user-inputted questions based on natural language can be diverse and highly ambiguous. For example, users might use colloquialisms, abbreviations, misspellings, or express their service needs in entirely new ways. Since LLMs typically rely on probabilistic guesses based on their internal parameters, they inherently face limitations in recognizing ambiguous or unfamiliar user requests. This makes it difficult for LLMs to accurately identify intent when faced with novel expressions outside their knowledge boundaries. Therefore, the processing model integrated into the aforementioned service system can be tailored to these characteristics of user-input questions, allowing for the customization of diverse standardized processing methods. This transforms user-input questions into standardized user queries suitable for LLM processing.

[0091] The various standardized processing methods customized for the processing model can be flexibly defined based on specific needs in practical applications, and will not be subject to special restrictions in this specification.

[0092] For example, in some embodiments, the standardization processing of user-inputted queries may specifically include a combination of one or more of the following processes:

[0093] Correcting typos in user questions;

[0094] Correct the colloquial expressions in user questions to standardized expressions;

[0095] Rewrite user queries into standardized semantic user queries; for example, if a user enters the query "Book me a flight to Shanghai tomorrow", it can be corrected and rewritten as the standard question "Book a flight to Shanghai tomorrow".

[0096] Break down user questions into multiple sub-user questions; for example, if a user question involves a complex intent, it can be broken down into multiple semantically coherent sub-user questions to deconstruct the user's complex intent.

[0097] Of course, in addition to automatically standardizing user-input queries through the introduced processing model, in practical applications, they can also be handled manually by relevant users.

[0098] In some embodiments, please continue to see Figure 3 The aforementioned service system can also implement a standardized processing module on the server side. This standardized processing module can be used to call the aforementioned processing model to perform standardized processing on user-input user queries.

[0099] After the user interface module receives the user's question through the visual interface, it can further submit the question to the standardization processing module. The standardization processing module then calls the processing model to standardize the question, thus transforming it into a standardized question.

[0100] It should be noted that the specific type of the processing model described above is not specifically limited in this specification. In practical applications, it can be flexibly selected based on specific needs. For example, in some embodiments, the processing model described above can be a lightweight LLM, or it can be a special-purpose model based on neural networks other than an LLM.

[0101] In this specification, after the service system receives a user's question, it can determine the target sub-knowledge base related to the user's question from the multiple sub-knowledge bases it manages, and retrieve target structured knowledge that is semantically similar to the user's question from the determined target sub-knowledge base.

[0102] In this way, the search scope can be narrowed when retrieving structured knowledge; for example, structured knowledge can be retrieved only in sub-knowledge bases related to user questions.

[0103] Specifically, the aforementioned target sub-knowledge base may include one or more sub-knowledge bases related to the user's question; when the service system determines the target sub-knowledge base related to the user's question from the aforementioned multiple sub-knowledge bases, it may use different methods to measure the relevance between the service domain of the user's question and the sub-knowledge base.

[0104] In one embodiment shown, when the service system determines the target sub-knowledge base related to the user's question from the above-mentioned multiple sub-knowledge bases, it can first extract question features from the user's question. The method of extracting question features from the user's question and the specific types of the extracted question features are not specifically limited in this specification. For example, keywords that can express the user's potential needs or the service domain to which the user's question belongs can be extracted as question features.

[0105] After extracting question features from user queries, a feature vector can be generated based on these features, and the vector distance between this feature vector and the description vectors of multiple sub-knowledge bases can be calculated. Specifically, the description vectors can include vectors generated based on the descriptive information from the aforementioned sub-knowledge bases. This vector distance can be used to describe the similarity between the feature vector and the description vector. In practical applications, the method of measuring the vector distance can be flexibly chosen; for example, cosine distance or other forms of vector distance measurement can be used.

[0106] Then, the target sub-knowledge base with the smallest vector distance among the above multiple sub-knowledge bases can be identified as the sub-knowledge base related to the user's question.

[0107] In one embodiment shown, when the service system determines the target sub-knowledge base related to the user's question from the above-mentioned multiple sub-knowledge bases, it can also first extract the question features from the user's question and input the question features into a pre-trained multi-classification model accessed by the service system, so that the multi-classification model can classify the question features to determine the sub-knowledge base related to the user's question.

[0108] In practical applications, the classification results output by the above multi-classification model can specifically represent sub-knowledge bases related to user questions; for example, the classification results output by the above multi-classification model can specifically be the identifier of a sub-knowledge base, and the tags of the sub-knowledge base can be used as the classification tags of the multi-classification model.

[0109] Then, based on the classification results of the question features output by the multi-classification model, the service system can determine the target sub-knowledge base related to the user's question from the multiple sub-knowledge bases.

[0110] When the service system retrieves target structured knowledge that is semantically similar to the user's question from the identified target sub-knowledge base, different methods can be used to query the structured knowledge.

[0111] In one embodiment shown, the aforementioned target sub-knowledge base may specifically include a vector database composed of vectorized structured knowledge; that is, the structured knowledge can be processed into vector form to create a vector database as a sub-knowledge base. Accordingly, the service system can use vector retrieval to retrieve target structured knowledge that is semantically similar to the user's question from this vector database.

[0112] In order to improve the efficiency of vector retrieval, in practical applications, ANN (Approximate Nearest Neighbor) can be introduced to narrow the query range of vector retrieval.

[0113] In this scenario, when performing vector retrieval, the service system can generate a question vector corresponding to the user's question, and then run the ANN algorithm to filter a set of candidate vectors related to the question vector from the vector database corresponding to the target sub-knowledge base.

[0114] It should be noted that the specific type of ANN algorithm mentioned above will not be specifically limited in this specification. In practical applications, it can be flexibly selected based on specific needs. For example, in one example, the HNSW (Hierarchical Navigable Small World) algorithm can be selected as the ANN algorithm. Correspondingly, the specific implementation process of running the ANN algorithm to filter the candidate vector set related to the question vector from the vector database corresponding to the target sub-knowledge base usually depends on the specific type of ANN algorithm selected. It will not be detailed in this specification. Those skilled in the art can refer to the records in related technologies.

[0115] After running the ANN algorithm to filter out the candidate vector set related to the question vector, the vector distance between the user's question and the vectorized structured knowledge in the candidate vector set can be further calculated. Then, at least one vectorized structured knowledge in the candidate vector set with the smallest vector distance to the question vector can be used as the target structured knowledge that is semantically similar to the user's question.

[0116] In some embodiments, please continue to see Figure 3The aforementioned service system can also implement a retrieval enhancement module based on RAG (Retrieval-Augmented Generation) technology on the server side; specifically, this retrieval enhancement module can be used to determine the target sub-knowledge base related to the user's question from the above multiple sub-knowledge bases, and retrieve target structured knowledge that is semantically similar to the user's question from the target sub-knowledge base.

[0117] After the standardization processing module processes the user's question into a standardized user question, it can further submit the standardized user question to the retrieval enhancement module. The retrieval enhancement module generates a question vector corresponding to the user's question and runs an approximate nearest neighbor algorithm to filter a set of candidate vectors related to the question vector from the vector database. Then, it retrieves at least one vectorized structured knowledge with the smallest vector distance to the question vector from the candidate vector set as the target structured knowledge that is semantically similar to the user's question.

[0118] On the other hand, in addition to identifying the target sub-knowledge base related to the user's question from the multiple sub-knowledge bases managed above, and retrieving target structured knowledge that is semantically similar to the user's question from the identified target sub-knowledge base, the service system can also obtain historical context information from the user's historical questions related to the aforementioned business travel services.

[0119] In some embodiments, please continue to see Figure 3 The aforementioned service system can also implement a context memory module on the server side. This context memory module is specifically responsible for storing and managing the historical context information of multiple rounds of historical dialogue. In practical applications, this context memory module can receive user questions from each round of the conversation from the user interface module, process these user questions into a sequence for storage and management.

[0120] After the standardization processing module processes the user's input question into a standardized question, in addition to generating a question vector corresponding to the question through the retrieval enhancement module and running an approximate nearest neighbor algorithm to filter a set of candidate vectors related to the question vector from the vector database, and further retrieving at least one vectorized structured knowledge with the smallest vector distance to the question vector from the candidate vector set as the target structured knowledge that is semantically similar to the question, historical context information can also be obtained from historical user questions related to the business travel service maintained in the context memory module.

[0121] In some embodiments, the historical user questions related to the business travel service entered by the user can specifically be in the form of a sequence of historical user questions, which may include historical user questions corresponding to multiple rounds of historical conversations.

[0122] Since the user's historical question sequence may contain a large number of historical conversation rounds, in order to control the amount of historical context information obtained, a strategy can be adopted to retain the details of recent multi-round historical conversations and retain the question summaries of long-term multi-round historical conversations to obtain historical context information.

[0123] Under this strategy, the historical context information finally collected can specifically include historical user questions corresponding to historical sessions with a session number less than the first threshold in the aforementioned multi-round historical sessions; and user question summaries corresponding to historical user questions corresponding to historical sessions with a session number greater than the first threshold and less than the second threshold in the aforementioned multi-round historical sessions.

[0124] For example, assuming the first threshold is denoted as N and the second threshold is denoted as M, the final collected historical context information may specifically include the original content of historical user questions corresponding to the most recent N rounds of historical sessions; and the user question summaries of historical user questions corresponding to the Nth to Mth rounds of historical sessions.

[0125] The first and second thresholds mentioned above can be flexibly set based on requirements;

[0126] For example, in one case, assuming N is 5 and M is 10, the final collected historical context information may include the original content of historical user questions corresponding to the most recent 5 rounds of historical conversations; and user question summaries of historical user questions corresponding to the 5th to 10th rounds of historical conversations respectively.

[0127] At this time, when the service system obtains historical context information from the historical user questions related to the business travel service input by the user, it can first obtain the original content of the historical user questions corresponding to the historical sessions with a session round number less than or equal to the first threshold from the above historical user question sequence, and then obtain the historical user questions corresponding to the historical sessions with a session round number greater than the first threshold and less than the second threshold from the above historical user question sequence, and calculate the user question summary of the historical user questions.

[0128] Specifically, the aforementioned user question summary can include any form of summary that describes the core content of a user's historical user questions. For example, it could be a shorter user question rewritten semantically based on key fragments from historical user questions.

[0129] In some embodiments, for any historical session with a session round greater than a first threshold and less than a second threshold, in order to ensure that the user question summary corresponding to the historical session of the current session round and the user question summary information corresponding to the historical session of the previous session round are semantically coherent and do not deviate, the historical user questions corresponding to the current session round and the user question summaries corresponding to the historical user questions of the previous session round can be used together as input data to calculate the user question summary corresponding to the historical session of the current session round.

[0130] In this scenario, when the service system calculates the user question summary for historical user questions corresponding to the current session round, it can first obtain the historical user questions corresponding to the current session round from the aforementioned historical user question sequence, and then obtain the user question summary for the historical user questions corresponding to the previous session round. Then, based on the historical user questions corresponding to the current session round and the user question summaries for the historical user questions corresponding to the previous session round, it can further calculate the user question summary corresponding to the historical user questions of the current session round.

[0131] The specific methods used to calculate the user query summary will not be shown in detail in this specification.

[0132] For example, in some embodiments, user query summaries can be generated by introducing a recurrent neural network suitable for processing sequence data into the service system.

[0133] In this scenario, when the service system calculates the user question summary of historical user questions corresponding to the current session round, it can further input the user question summaries of historical user questions corresponding to the current session round and the previous session round into a pre-trained recurrent neural network. The recurrent neural network can then generate a user question summary corresponding to the historical user questions of the current session round based on the historical user questions and the user question summary.

[0134] It should be noted that the specific type of recurrent neural network described above will not be specifically limited in this specification. In practical applications, it can be flexibly selected based on specific needs. For example, in some embodiments, the recurrent neural network described above may specifically include a GRU (Gated Recurrent Unit) network.

[0135] Step 206: Based on the user's question, the target structured knowledge, and the historical context information, generate prompt words to guide the LLM in determining the user's intent, and input the prompt words into the LLM so that the LLM can perform inference calculations based on the prompt words to generate structured intent description information; wherein, the intent description information is used to describe the user's intent;

[0136] In this specification, after the service system obtains the user's input question, the aforementioned target structured knowledge, and the aforementioned historical context information, it can further use this information as inference context to generate a prompt. Specifically, the prompt can be used to guide the LLM to determine the user's intent based on the inference context contained in the prompt.

[0137] In some embodiments, the above-mentioned service system may specifically support the generation of prompt words based on a maintained structured prompt word template; wherein, the structured prompt word template may specifically include fields for maintaining user-inputted questions, the acquired target structured knowledge, and the historical context information;

[0138] In this case, when generating the aforementioned prompt words, the service system can specifically fill the user's question, the target structured knowledge, and the historical context information into the corresponding fields of the structured prompt word template to generate prompt words used to guide the LLM in judging the user's intent.

[0139] In some embodiments, please continue to see Figure 3 The aforementioned service system can also implement a prompt word construction module on the server side. Specifically, this prompt word construction module is used to fill the standardized user questions output by the aforementioned standardization processing module, the aforementioned historical context information obtained from the context memory module, and the aforementioned target structured knowledge retrieved by the aforementioned retrieval enhancement module into the corresponding fields of the structured prompt word template, so as to generate prompt words used to guide the LLM to judge the user's intent.

[0140] In practical applications, the aforementioned structured prompt word templates can be customized based on experience, or existing general prompt word frameworks can be adopted.

[0141] In some embodiments, the above-mentioned prompt word template may specifically adopt the general CO-STAR prompt word template.

[0142] CO-STAR is a structured cue word framework designed specifically for LLM (Low-Level Writers). It aims to help users interact with LLM more effectively, resulting in higher-quality, more expected output. This cue word framework is particularly suitable for scenarios requiring precise, professional output and is an effective way to improve the efficiency of collaboration with AI. The framework's name is an abbreviation representing the seven key elements that should be included in the cue words.

[0143] The key elements of the prompts included in the CO-STAR framework are:

[0144] The C-Context field provides background information about the task. This element is designed to help LLMs understand the current context, history, or relevant situations, making their output more targeted.

[0145] The O-Objective field clearly states the specific goal you hope to achieve through this interaction. This element provides LLMs with clear objectives, helping them focus on key issues and avoid straying from the topic.

[0146] The S-Style field specifies the language style you want the LLM output to have.

[0147] The T-Tone field specifies the tone you want the LLM to use in its response. For example, professional, objective expectations, etc.

[0148] The A-Audience field specifies the audience for the LLM output, that is, who the output is ultimately intended for. This element helps LLM adjust the depth of language and explanation by clearly defining the audience.

[0149] The R-Response field specifies the output format or structure you expect from the LLM.

[0150] In some embodiments, in order to allow the prompt words built based on the CO-STAR framework to also accommodate user-inputted questions and the aforementioned historical context information, the existing fields in the CO-STAR framework can be extended. A first extended field for maintaining the aforementioned user-inputted questions and a second extended field for maintaining the aforementioned historical context information can be added to the CO-STAR prompt word template.

[0151] In this scenario, when generating prompts based on the CO-STAR prompt template, the service system can populate the aforementioned user questions into the first extended field included in the CO-STAR prompt template, and the aforementioned historical context information into the second extended field included in the CO-STAR prompt template. For the aforementioned target structured knowledge, it can be populated into the target base field specified from the base fields included in the CO-STAR prompt template. After completing the above population, a structured prompt can be constructed based on the CO-STAR framework to guide the LLM in determining the user's intent.

[0152] The aforementioned target base field can be any one of the seven fields already available in the CO-STAR framework.

[0153] For example, in some embodiments, the aforementioned target base field may specifically include the Context field contained in the CO-STAR prompt template. In this case, the prompt built based on the CO-STAR framework can adopt the following structure:

[0154] The Context field explicitly informs the LLM that its role is an intent-based expert, and maintains the retrieved structured knowledge of the aforementioned target in this field;

[0155] The Objective field clearly defines the task as determining the user's intent based on the user's question, the structured knowledge of the objectives mentioned above, and the historical context information mentioned above.

[0156] Style field: Requires the output to be rigorous and precise;

[0157] Tone field: Requires professional and objective output;

[0158] Audience field: The audience of the output result can be an intent dispatcher implemented in the service system; specifically, the intent dispatcher can be used to parse the user intent output by the LLM and distribute the user's question to the correct service model;

[0159] Response field: Requires the LLM to output intent description information in a structured data format, describing the user intent determined by the LLM; for example, outputting in JSON format.

[0160] User Query field: This field is the first extended field mentioned above and is used to maintain user-inputted queries; for example, the standardized user queries mentioned above.

[0161] History field: This field is the second extended field mentioned above and is used to maintain the historical context information.

[0162] In this specification, after the service system generates prompt words based on the user's input question, the aforementioned target structured knowledge, and the aforementioned historical context information, the prompt words can be further input into the LLM accessed by the service system. The LLM then performs inference calculations based on the reasoning context contained in the prompt words to determine the user's intent and generates structured intent description information based on the determined user intent. Specifically, this intent description information can be used to describe the user intent determined by the LLM.

[0163] For example, in some embodiments, the prompt may require the LLM to output the aforementioned intent description information in JSON format.

[0164] In some embodiments, please continue to see Figure 3 The aforementioned service system can also implement an LLM inference engine on the server side based on the accessed LLM; this inference engine is specifically used to perform inference calculations based on the inference context contained in the prompt words generated by the prompt word construction module to determine the user's intent, and generate structured intent description information based on the determined user intent.

[0165] Step 208: Parse the intent description information, determine the target service domain corresponding to the user's intent, and distribute the user's question to the target service model corresponding to the target service domain, so that the target service model can perform service processing on the user's question corresponding to the business travel service.

[0166] In this specification, after the LLM outputs structured intent description information, the service system can parse the structured intent description information, determine the target service domain corresponding to the user's intent, and distribute the user's input query to the target service model corresponding to the target service domain, so that the target service model can perform service processing on the user's query.

[0167] It should be noted that the specific content included in the above-mentioned structured intent description information is not specifically limited in this specification. In practical applications, it can include any form of information related to user intent.

[0168] In some embodiments, the structured intent description information described above may specifically include: an intent field for maintaining the user intent determined by the LLM; and an entity field for maintaining the service entity related to the user intent extracted from the user's query.

[0169] For example, taking the above structured intent description information as JSON format as an example, the intent description information can be the following data structure: {"intent":"XXX", "entities":"XXX"}; where intent represents the intent field and entities represents the entity field.

[0170] In this case, the service system can parse the structured intent description information, obtain the user intent maintained in the intent field, and obtain at least one service entity maintained in the at least one entity field; then, determine the target service domain corresponding to the obtained user intent, and distribute the user question and the at least one service entity to the target service model corresponding to the target service domain, so that the target service model can perform service processing on the user question and the at least one service entity.

[0171] In some embodiments, the above prompts can also be used to guide the LLM to determine the user's intent and identify the target service area corresponding to the user's intent;

[0172] For example, taking the construction of prompt words based on the CO-STAR framework as an example, the Objective field contained in the constructed prompt words can no longer be defined as "determine the user's intent based on the user's question, the above-mentioned target structured knowledge, and the above-mentioned historical context information"; instead, it can be defined as "determine the user's intent based on the user's question, the above-mentioned target structured knowledge, and the above-mentioned historical context information, and determine the target service domain corresponding to the user's intent".

[0173] In this case, the function of the intent field contained in the structured intent description information will also change. At this time, the intent field is specifically used to maintain the target service domain corresponding to the user's intent. When the service system parses the structured intent description information, it can directly obtain the target service domain corresponding to the user's intent maintained in the intent field, and obtain at least one service entity maintained in the at least one entity field. Then, it can directly distribute the user question and the at least one service entity to the target service model corresponding to the obtained target service domain, so that the target service model can perform service processing on the user question and the at least one service entity.

[0174] In some embodiments, please continue to see Figure 3The aforementioned service system can also implement an intent dispatcher on the server side. This intent dispatcher can be used to parse the user intent output by the LLM and distribute the user's query to the correct service model. In this case, the aforementioned multiple service models can specifically be Agents built based on the LLM. The service system can maintain an Agent pool on the server side to manage multiple Agents that correspond one-to-one with the aforementioned multiple service domains.

[0175] When the LLM outputs structured intent description information, the intent dispatcher can parse the structured intent description information, obtain the user intent maintained in the intent field or the target service domain corresponding to the user intent, and obtain at least one service entity maintained in at least one entity field; then, it determines the target agent corresponding to the target service domain from the agent pool, and then distributes the user question and the at least one service entity to the target agent corresponding to the target service domain for service processing.

[0176] Once the target service model has completed the service processing for the user's question and at least one of the service entities, the service results can be output and displayed to the user.

[0177] In some embodiments, please continue to see Figure 3 The user interface module implemented in the above service system can not only receive user questions based on natural language input, but also further display the service results output by the target service model to the user.

[0178] It should be noted that, Figure 3 The modules shown can also be independent subsystems or other forms of execution units in practical applications, and no specific restrictions are made in this specification.

[0179] In the above technical solution, within the service system built on LLM, the inference capabilities of LLM are utilized to perform inference calculations using various external evidence, including user-inputted questions, structured knowledge retrieved from a knowledge base related to the user-inputted questions, and historical context information related to the user's past questions, to determine the user's intent and identify the target service domain corresponding to the user's intent. The user question is then distributed to the target service model corresponding to the target domain among multiple service models managed by the service system for service processing. This allows for accurate distribution of user questions to relevant service models when they involve service needs within a specific service domain, enabling targeted service delivery to the user using those models.

[0180] For example, this specification creatively redefines the user question distribution problem as a "reasoning decision problem based on multiple types of reasoning contexts". By combining multiple reasoning contexts, the reasoning capabilities of LLM can be used to generate structured intent description information that can represent user intent. By parsing this structured intent description information, the user question can be accurately distributed to the domain expert model corresponding to the professional service domain that the user needs for processing.

[0181] Moreover, by employing multiple service models corresponding to different service areas to provide services to users, it is no longer necessary to use only a single service model to provide services to users. This can improve the accuracy, reliability, and professionalism of the services provided to users, and avoid the problems of low service accuracy, poor reliability, and poor professionalism that occur when using a single general service model to provide services to users in multiple service areas at the same time.

[0182] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of apparatus, electronic devices, and storage media.

[0183] Figure 4 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0184] like Figure 5 As shown, Figure 5 This specification is a block diagram illustrating an LLM-based business travel service management device according to an exemplary embodiment. This device can be applied to applications such as... Figure 4 The electronic device shown implements the technical solution of this specification. The service system provides business travel services to users, including multiple service areas; the external knowledge base accessed by the service system contains multiple sub-knowledge bases corresponding one-to-one with the multiple service areas; the service system manages multiple service models corresponding one-to-one with the multiple service areas; the service models include service models built based on LLM; the device includes:

[0185] The receiving module 501 receives user questions related to the business travel service input by the user;

[0186] The input module 502 determines a target sub-knowledge base related to the user's question from the plurality of sub-knowledge bases, retrieves target structured knowledge that is semantically similar to the user's question from the target sub-knowledge base, and obtains historical context information from the user's historical user questions related to the business travel service.

[0187] The generation module 503 generates prompt words based on the user's question, the target structured knowledge, and the historical context information to guide the LLM in judging the user's intent. The prompt words are then input into the LLM, which performs inference calculations based on the prompt words to generate structured intent description information. The intent description information describes the user's intent.

[0188] The distribution module 504 parses the intent description information, determines the target service domain corresponding to the user's intent, and distributes the user's question to the target service model corresponding to the target service domain, so that the target service model can perform service processing on the user's question.

[0189] Accordingly, this specification also provides an electronic device including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement all the steps in the previously described method flow.

[0190] Accordingly, this specification also provides a computer-readable storage medium having stored thereon executable computer program instructions; wherein, when executed by a processor, the instructions implement all the steps in the previously described method flow.

[0191] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, they implement all the steps in the previously described method flow.

[0192] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, it is not excluded that with the future development of computer technology, the computer implementing the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0193] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, it is not excluded that the process, method, product, or apparatus that includes the elements may also have other identical or equivalent elements. For example, the use of terms such as "first," "second," etc., is used to indicate names and does not indicate any particular order.

[0194] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0195] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0196] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0197] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0198] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0199] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0200] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0201] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0203] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0204] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A business travel service management method based on LLM, applied to a service system constructed based on LLM; wherein, The service system provides business travel services to users, including multiple service fields; and an external knowledge base accessed by the service system includes multiple sub-knowledge bases corresponding to the multiple service fields; The service system manages multiple service models corresponding to the multiple service fields; The service model includes a service model based on an LLM; and the method includes: receiving a user question input by a user about the business travel service; determining a target sub-knowledge base related to the user question from the multiple sub-knowledge bases, retrieving target structured knowledge similar in semantics to the user question from the target sub-knowledge base, and obtaining historical context information from historical user questions input by the user about the business travel service; wherein the service system supports generating a prompt based on a maintained CO-STAR prompt template; the CO-STAR prompt template includes a first extension field for maintaining the user question and a second extension field for maintaining the historical context information; filling the user question into the first extension field included in the CO-STAR prompt template; filling the historical context information into the second extension field included in the CO-STAR prompt template; filling the target structured knowledge into a target base field specified from a base field included in the CO-STAR prompt template, to generate a prompt for guiding an LLM to determine a user intent of the user, and inputting the prompt into the LLM to perform inference calculation based on the prompt by the LLM, to generate structured intent description information; wherein the intent description information is used to describe the user intent of the user; parsing the intent description information, determining a target service field corresponding to the user intent of the user, and distributing the user question to a target service model corresponding to the target service field, to perform service processing corresponding to the business travel service on the user question by the target service model.

2. The method of claim 1, wherein determining a target sub-knowledge base related to the user question from the multiple sub-knowledge bases includes: extracting a question feature from the user question; generating a feature vector based on the question feature, and calculating a vector distance between the feature vector and a description vector of the multiple sub-knowledge bases; wherein the description vector includes a vector generated based on description information of the multiple sub-knowledge bases; and determining a target sub-knowledge base with the smallest vector distance as the sub-knowledge base related to the user question; or extracting a question feature from the user question; inputting the question feature into a pre-trained multi-classification model accessed by the service system, to perform classification processing on the question feature by the multi-classification model; wherein a classification result output by the multi-classification model represents a sub-knowledge base related to the user question; and determining a target sub-knowledge base related to the user question from the multiple sub-knowledge bases based on the classification result of the question feature output by the multi-classification model.

3. The method of claim 1, wherein the sub-knowledge base comprises a vector database consisting of vectorized structured knowledge; retrieving target structured knowledge similar in semantics to the user question from the target sub-knowledge base, comprising: generating a question vector corresponding to the user question; running an approximate nearest neighbor algorithm to filter a candidate vector set related to the question vector from the vector database; further calculating vector distances between the user question and the vectorized structured knowledge in the candidate vector set; selecting at least one vectorized structured knowledge in the candidate vector set with the smallest vector distance to the question vector as the target structured knowledge similar in semantics to the user question.

4. The method of claim 1, the user-entered historical user query related to the business travel service comprising: a historical user question sequence consisting of historical user questions corresponding to historical sessions of the multi-round historical session; the historical context information comprises: historical user questions corresponding to historical sessions with session turns less than a first threshold in the multi-round historical session; and a user question summary of historical user questions corresponding to historical sessions with session turns greater than the first threshold and less than a second threshold in the multi-round historical session; obtaining historical context information from the historical user questions related to the business travel service input by the user, comprising: obtaining historical user questions corresponding to historical sessions with session turns less than or equal to the first threshold from the historical user question sequence; and, obtaining historical user questions corresponding to historical sessions with session turns greater than the first threshold and less than the second threshold from the historical user question sequence, and calculating a user question summary of the historical user questions.

5. The method of claim 1, wherein the service system accesses a processing model for standardizing the user question; before generating the prompt word for guiding the LLM to determine the user intent of the user based on the user question, the target structured knowledge, and the historical context information, further comprising: standardizing the user question based on the accessed processing model to process the user question into a standardized user question; wherein the standardization of the user question comprises one or more of the following combinations: correcting misspelled words contained in the user question; correcting colloquial expressions contained in the user question into standardized expressions; rewriting the user question into a standardized semantic user question; disassembling the user question into multiple sub-user questions.

6. The method of claim 1, wherein the service model comprises an artificial intelligence service program constructed based on the LLM; the plurality of service domains comprises a reservation service domain related to the business travel service, a business travel data analysis service domain, and a business travel policy inquiry service domain; and the plurality of service models comprises a first artificial intelligence service program for performing a reservation service related to the business travel service, a second artificial intelligence service program for performing a business travel data analysis service related to the business travel service, and a third artificial intelligence service program for performing a business travel policy inquiry service related to the business travel service.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 6.

8. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images