Port intelligent customer service method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of shipping ports, and discloses a port intelligent customer service method and device, computer equipment and a storage medium, and the method comprises the steps: receiving multi-modal question data sent by a target user through a large language model agent, and converting the multi-modal question data into to-be-analyzed question data in a preset data format; performing semantic analysis processing on the to-be-analyzed question data to obtain question field data of the target user; performing intention understanding according to the question field data and the question data to be analyzed by using the large language model agent to obtain intention prediction data of the target user; retrieving a pre-constructed port multi-modal knowledge base according to the intention prediction data, generating answer data of at least one modal by using a retrieval result, and returning the answer data to the target user; through the above mode, the user data service capability of the port intelligent customer and the communication efficiency of the port intelligent customer service are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of port customer service, and in particular to a port intelligent customer service method, device, computer equipment and storage medium. Background Art

[0002] In the current intelligent customer service management of shipping ports, artificial intelligence technology has been integrated into the customer service environment to fully upgrade the port customer service system. However, on the one hand, the current intelligent port customer service technology solution can only process text-type data, and cannot process other types of data sent by users to the port intelligent customer service, such as voice, video, pictures, etc. On the other hand, when the current port intelligent customer service communicates with users, it can only provide text-type answer data content, and the answer data content sometimes lacks professionalism in terms of port professional knowledge. Summary of the invention

[0003] The embodiments of the present invention provide a port intelligent customer service method, device, computer equipment and storage medium method and device, which can improve the user data service capabilities of the port intelligent customer and the communication efficiency of the port intelligent customer service.

[0004] In a first aspect, an embodiment of the present application provides a port intelligent customer service method, comprising: receiving multimodal question data sent by a target user through a large language model agent, and converting the multimodal question data into question data to be analyzed in a preset data format; performing semantic analysis on the question data to be analyzed to obtain question domain data of the target user; using the large language model agent to understand the intention based on the question domain data and the question data to be analyzed to obtain the intention prediction data of the target user; searching a pre-constructed port multimodal knowledge base based on the intention prediction data, generating answer data of at least one modality using the search results, and returning the answer data to the target user.

[0005] Optionally, after the step of performing semantic analysis on the question data to be analyzed to obtain the question domain data of the target user, it also includes: if the question domain data does not include the port domain, forwarding the question data to be analyzed to manual customer service; if the question domain data includes the port domain, and the number of port domains is 1, forwarding the question domain data to the large language model agent corresponding to the port domain; if the question domain data includes the port domain, and the number of port domains is greater than 1, conducting a domain clarification dialogue with the target user through the large language model agent, and continuing to obtain the port domain according to the context of the domain clarification dialogue, until the number of port domains obtained is 1 or forwarding the question data to be analyzed and the domain clarification dialogue to manual customer service.

[0006] Optionally, the step of retrieving a pre-constructed port multimodal knowledge base according to the intention prediction data and using the retrieval results to generate answer data of at least one modality includes: inputting the intention prediction data and the multimodal question data into a question and answer engine corresponding to each modal data; performing multimodal understanding processing by the question and answer engine using a model corresponding to each modal data in a preset model warehouse to obtain a multimodal understanding result; performing data matching from a preset multimodal knowledge base according to the intention prediction data to obtain a matching knowledge base list; using the multimodal understanding result to perform a mixed search in the matching knowledge bases included in the matching knowledge base list to obtain knowledge retrieval results corresponding to different modalities; and performing multimodal generation using the knowledge retrieval results through the models corresponding to each modal data to obtain the answer data corresponding to each modality.

[0007] Optionally, after the step of using the multimodal understanding result to perform a mixed search in the matching knowledge bases included in the matching knowledge base list to obtain knowledge retrieval results corresponding to different modalities, the step further includes: obtaining the in-library rankings of different data blocks in the knowledge retrieval results in the corresponding different matching knowledge bases; and calculating the comprehensive score of each of the data blocks by a preset reciprocal ranking fusion formula, wherein the preset reciprocal ranking fusion formula is as follows:

[0008]

[0009] Wherein, N represents the number of matching knowledge bases, c represents the data block, and r i (c) represents the ranking of the data block in the i-th matching knowledge base, and k represents a constant used to control the ranking; the knowledge retrieval results of the different modalities are optimized according to the comprehensive score of the data block.

[0010] Optionally, the steps of constructing the port multimodal knowledge base include: collecting candidate knowledge data in the port field and port-related fields from preset collection channels; performing high-frequency knowledge extraction on the candidate knowledge data through preset data mining tools and preset knowledge extraction rules to obtain high-frequency knowledge extraction results; matching the high-frequency knowledge extraction results with preset question and answer engines corresponding to each modal data to obtain a knowledge engine matching relationship; based on the knowledge engine matching relationship, classifying and arranging the high-frequency knowledge extraction results according to preset categories and organizational methods to obtain knowledge organization results corresponding to each question and answer engine.

[0011] Optionally, after the step of optimizing the knowledge retrieval results of different modalities according to the comprehensive scores of the data blocks, the step further includes: vectorizing the data blocks using a preset vector model, performing dimensionality reduction on the vectorized database, and soft clustering the data blocks after dimensionality reduction using a Gaussian mixture model to obtain data block clustering results; setting data blocks of different categories in the data block clustering results as leaf nodes corresponding to different tree structures, generating corresponding different first summary data according to the data blocks of different categories through a preset large language model, and setting the first summary data as the parent node of the leaf node; replacing the data block with the first summary data, and recursively executing the steps from vectorizing the data block using a preset vector model to setting the first summary data as the parent node of the leaf node, until the number of parent nodes is within the limit range of the number of context windows of the large language model agent.

[0012] Optionally, after the step of using the retrieval results to generate answer data of at least one modality and returning the answer data to the target user, it also includes: obtaining the target user's evaluation feedback data on the answer data; performing data analysis on the evaluation feedback data, and determining whether to send the evaluation feedback data to manual customer service based on the data analysis results.

[0013] In a second aspect, an embodiment of the present application provides a port intelligent customer service device, comprising: a question data module for analysis, which is used to receive multimodal question data sent by a target user through a large language model agent, and convert the multimodal question data into question data for analysis in a preset data format; a question domain data module, which is used to perform semantic analysis on the question data for analysis to obtain question domain data of the target user; an intention prediction data module, which is used to use the large language model agent to understand the intention based on the question domain data and the question data for analysis to obtain the intention prediction data of the target user; a customer service answer data module, which is used to retrieve a pre-constructed port multimodal knowledge base based on the intention prediction data, generate answer data of at least one modality using the search results, and return the answer data to the target user.

[0014] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the port intelligent customer service method as described above when executing the program stored in the memory.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the port intelligent customer service method as described above are implemented.

[0016] The embodiments of the present application provide a port intelligent customer service method, apparatus, computer equipment and storage medium, which receive multimodal question data sent by a target user through a large language model agent, and convert the multimodal question data into question data to be analyzed in a preset data format; perform semantic analysis on the question data to be analyzed to obtain question domain data of the target user; use the large language model agent to understand the intention based on the question domain data and the question data to be analyzed to obtain the intention prediction data of the target user; retrieve a pre-constructed port multimodal knowledge base based on the intention prediction data, use the retrieval results to generate answer data of at least one modality, and return the answer data to the target user; through the above method, the user data service capabilities of the port intelligent customer and the communication efficiency of the port intelligent customer service are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0018] Figure 1 This is a schematic diagram of the application environment of a port intelligent customer service method provided by the present application.

[0019] Figure 2 It is a schematic diagram of the execution flow of the port intelligent customer service method provided in an embodiment of the present application.

[0020] Figure 3 It is a schematic diagram of the execution flow of port domain judgment in the port intelligent customer service method provided in an embodiment of the present application.

[0021] Figure 4 This is an example diagram of the execution of domain routing in the port intelligent customer service method provided in an embodiment of the present application.

[0022] Figure 5 It is a schematic diagram of the execution flow of obtaining answer data in the port intelligent customer service method provided in the embodiment of the present application.

[0023] Figure 6 This is an execution example diagram of obtaining multimodal answer data in the port intelligent customer service method provided in an embodiment of the present application.

[0024] Figure 7It is a schematic diagram of the execution flow of organizing port knowledge in the port intelligent customer service method provided in an embodiment of the present application.

[0025] Figure 8 It is an execution example diagram of constructing port multimodal knowledge assets in the port intelligent customer service method provided in the embodiment of the present application.

[0026] Fig. 9 It is a schematic diagram of the execution flow of optimizing knowledge retrieval results in the port intelligent customer service method provided in an embodiment of the present application.

[0027] Fig.10 It is an execution example diagram of constructing port multimodal knowledge assets in the port intelligent customer service method provided in the embodiment of the present application.

[0028] Fig.11 It is a schematic diagram of the system structure of the port intelligent customer service device provided in an embodiment of the present application.

[0029] Fig.12 A schematic diagram of the structure of a computer device according to an embodiment of the present application.

[0030] Fig.13 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0032] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0033] In the embodiments of the present application, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0034] Moreover, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0035] In the description of the embodiments of the present application, words such as "example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "example" or "for example" in the embodiments of the present application is not to be interpreted as being more preferred or having more advantages than another embodiment or design. The use of words such as "example" or "for example" is intended to present relative concepts in a clear manner.

[0036] In addition, the "plurality" in the embodiments of the present application refers to two or more than two. In view of this, in the embodiments of the present application, "plurality" can also be understood as "at least two". "At least one" can be understood as one or more, for example, one, two or more. For example, including at least one means including one, two or more, and there is no limit on which ones are included. For example, including at least one of A, B and C, then A, B, C, A and B, A and C, B and C, or A, B and C can be included.

[0037] See also Figure 1 , Figure 1 This is a schematic diagram of the application environment of a port intelligent customer service method provided by the present application. Figure 1 As shown, the execution subject of the port intelligent customer service method provided in the embodiment of the present application is generally an electronic device with certain computing capabilities, such as a computer device. In some possible implementations, the port intelligent customer service method can be implemented by a processor calling a computer-readable instruction stored in a memory. Figure 1 The computer device in the term can be a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. It can be understood that Figure 1 The number of computer devices in the figure is only for reference and can be expanded to any number according to actual needs.

[0038] An embodiment of the present application provides a port intelligent customer service method, Figure 2It is a schematic diagram of the execution flow of the port intelligent customer service method provided in the embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order shown in the embodiment or the accompanying drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the port intelligent customer service method may include the following steps S1 to S4.

[0039] S1. Receive multimodal question data sent by a target user through a large language model agent, and convert the multimodal question data into question data to be analyzed in a preset data format.

[0040] Among them, multimodal data refers to a data set containing multiple data types or sources, which can include text, images, videos, audio and other forms. In multimodal data, different types of data can be interrelated or complementary to each other, thereby providing more comprehensive and rich information. In the embodiment of the present application, the multimodal question data sent by the target user can be text data, video data, voice data, picture data, etc., or file data in other formats, such as PDF files, Word files, Excel spreadsheet files, PPT files, etc. The aforementioned various data and files can all be about issues within the scope of port business. For example, a user sends a picture of a container ship, and the port intelligent customer service can parse the picture to obtain the ship name, ship parameters, and the associated berthing time, berthing berth, berthing reasons, etc. By presetting various processing tools or processing corresponding to different modal data in the port customer service intelligent weight, the user data service capability of the port intelligent customer service can be further improved.

[0041] S2. Perform semantic analysis on the question data to be analyzed to obtain the question domain data of the target user.

[0042] See also Figure 3 , Figure 3 : is a schematic diagram of the execution flow of port domain judgment in the port intelligent customer service method provided in the embodiment of the present application, such as Figure 3 As shown, the following steps S21 to S23 may be included.

[0043] S21. If the question field data does not include the port field, forward the question data to be analyzed to the manual customer service.

[0044] Among them, given the limitations of large language models in reasoning ability, accuracy and transparency at this stage, it is unrealistic to hope to rely entirely on artificial intelligence customer service to handle complex port customer service support work. Standardized consultation, FAQs and other tasks that artificial intelligence can effectively handle at this stage can be handed over to artificial intelligence customer service. For tasks involving judgment and flexibility, artificial intelligence customer service can provide preliminary suggestions or solutions, which will still need to be verified or adjusted by manual customer service. For extremely complex tasks or tasks that require highly personalized services (such as problems in unknown fields that artificial intelligence customer service cannot understand), artificial intelligence customer service should be abandoned and transferred to senior customer service personnel.

[0045] S22. If the question domain data includes the port domain, and the number of port domains is 1, the question domain data is forwarded to the large language model agent corresponding to the port domain.

[0046] For example, according to different business types in the port field, port customer service agents can be set up to handle different types of port business issues, such as product consultation port customer service agents, technical support port customer service agents, etc.

[0047] S23. If the question domain data contains port domains, and the number of port domains is greater than 1, a domain clarification dialogue is conducted with the target user through the large language model agent, and the port domains are continued to be obtained according to the context of the domain clarification dialogue, until the number of port domains obtained is 1 or the question data to be analyzed and the domain clarification dialogue are forwarded to manual customer service.

[0048] Among them, domain clarification can be achieved in many ways, asking and clarifying questions: The agent can better understand the user's intention by asking details or questions that need clarification. For example, asking the meaning of keywords, requiring more contextual information, etc. Confirmatory response: When the agent is not sure about the meaning of the user's question, it can use a confirmatory response to clarify. For example, "Are you asking about port berths?". Prompt the user to provide more information: The agent can request the user to provide more information or examples to better understand the problem. Guide the user to limit the scope: When the question is vague or broad, the agent can guide the user to narrow the scope of the question in order to better answer the question.

[0049] See also Figure 4 , Figure 4It is an execution example diagram of domain routing in the port intelligent customer service method provided by the embodiment of the present application. As shown in the figure, the user question data is first processed by domain routing. If the question obtained from the user question data is a question of unknown domain, such as a professional question of ship navigation rather than a port question, the user will be asked to re-describe the question. The specific method is to continue to ask the user to obtain a more accurate description of the question. If the user's question is still not understood for 2 consecutive times, that is, the question obtained is a question of unknown domain, then the user's question is transferred to the manual seat for processing. Secondly, if the user's question obtained is the port domain A, the contextual intention judgment is performed for the port domain A. If the user's intention is to chat, it is transferred to the preset chat agent to continue the conversation with the user, and the chat agent continues to ask questions to the user, or the user takes the initiative to exit. If the user's intention is a question-and-answer type intention, the user's question is processed by the question-and-answer engine, or the user takes the initiative to exit. Finally, when the number of domains asked by the user is greater than 1, multiple rounds of dialogue with the user are required to determine the unique port domain.

[0050] S3. Use a large language model agent to understand the intent based on the question domain data and the question data to be analyzed, and obtain the target user's intention prediction data.

[0051] S4. Retrieve a pre-built port multimodal knowledge base based on the intention prediction data, use the retrieval results to generate answer data of at least one modality, and return the answer data to the target user.

[0052] See also Figure 5 , Figure 5 is a schematic diagram of the execution flow of obtaining answer data in the port intelligent customer service method provided in the embodiment of the present application, such as Figure 5 As shown, the following steps S41 to S45 may be included.

[0053] S41. Input the intention prediction data and multimodal question data into the question-answering engine corresponding to each modal data.

[0054] Among them, in the embodiments of the present application, various forms of question-and-answer engines are constructed in combination with the characteristics of the port business. For example, the common question-and-answer engine: by configuring high-frequency question-and-answer knowledge, the port intelligent customer service can solve most of the high-frequency user problems. Data question-and-answer: You can get high-quality question-and-answer results by simply connecting to the database or importing table data. This engine supports advanced question-and-answer functions such as complex reasoning, multi-table queries, multi-round interactions, and human-like generation. Table question-and-answer engine: specially designed for structured two-dimensional table structure knowledge, it can quickly extract table information based on the two-dimensional data table to generate questions and answers, and supports question-and-answers with conditional constraints and aggregate function calculations. Graph question-and-answer engine: focuses on the question-and-answer of structured and systematic knowledge, relying on the knowledge graph Schema to realize fast question-and-answer of triple knowledge with certain structural relationships, and supports multi-hop, calculation and reasoning capabilities.

[0055] S42. The question-answering engine uses the models corresponding to the modal data in the preset model warehouse to perform multimodal understanding processing to obtain a multimodal understanding result.

[0056] For example, a text recognition model can be used for pictures with text in user questions, a speech recognition model can be used for audio data in user questions, and a video processing model can be used for video data in user questions. It should be noted that the aforementioned speech recognition model and video processing model can not only extract corresponding information feature data (audio or video), but also perform corresponding understanding and prediction based on the extracted information feature data to obtain multimodal understanding results.

[0057] S43. Perform data matching from a preset multimodal knowledge base according to the intention prediction data to obtain a matching knowledge base list.

[0058] S44. Use the multimodal understanding result to perform a mixed search in the matching knowledge bases included in the matching knowledge base list to obtain knowledge retrieval results corresponding to different modalities.

[0059] As an implementation method, after the above step S44, the in-library rankings of different data blocks in the knowledge retrieval results in the corresponding different matching knowledge bases can also be obtained. Then, the comprehensive score of each data block is calculated by a preset reciprocal ranking fusion formula, and the preset reciprocal ranking fusion formula is as follows:

[0060]

[0061] Where N represents the number of matching knowledge bases, c represents the data block, and r i (c) represents the ranking of the data block in the i-th matching knowledge base, and k represents a constant used to control the ranking. Finally, the knowledge retrieval results of different modalities are optimized according to the comprehensive scores of the data blocks.

[0062] S45. Use knowledge retrieval to perform multi-modal generation through the model corresponding to each modal data to obtain the answer data corresponding to each modality.

[0063] See also Figure 6 , Figure 6 1 is an example diagram of the execution of obtaining multimodal answer data in the port intelligent customer service method provided in the embodiment of the present application. Figure 6As shown in the figure, after determining that the question asked by the port customer is a question-and-answer type of intention, the multimodal input data sent by the user's question is first forwarded to different types of question-and-answer engines for processing, such as a common question-and-answer engine, a task question-and-answer engine, a table question-and-answer engine, or an image question-and-answer engine, and the text recognition, video understanding, speech recognition, or image-and-text understanding are performed through the processing models corresponding to various modes in the model warehouse. Then, based on the results of multimodal understanding, a search is performed in the multimodal knowledge base to obtain the corresponding search results. Finally, data outputs of different modes are generated based on the search results, which can be text-type search results to generate text, generate pictures, generate videos, generate speech, etc.

[0064] See also Figure 7 , Figure 7 : is a schematic diagram of the execution flow of organizing port knowledge in the port intelligent customer service method provided in the embodiment of the present application, such as Figure 7 As shown, the following steps S51 to S54 may be included.

[0065] S51. Collect candidate knowledge data in the port field and port-related fields from preset collection channels.

[0066] Among them, the knowledge sources of intelligent customer service are extensive, including historical logs, historical knowledge, document articles, SOP processes, table data, etc. This knowledge may come from various formats, forms and channels. When building an intelligent customer service knowledge system, candidate knowledge can be collected based on the knowledge source, and the knowledge candidate set can be determined by comprehensively considering business coverage and data format.

[0067] S52. Perform high-frequency knowledge extraction on the candidate knowledge data through a preset data mining tool and preset knowledge extraction rules to obtain a high-frequency knowledge extraction result.

[0068] Among them, we conduct in-depth mining of the collected knowledge to determine the final core knowledge scope. For massive knowledge data, we draw on historical operation experience, follow the Pareto principle, 80 / 20 principle, etc. to extract high-frequency knowledge, and screen out knowledge that is close to the business, easy to process by the system, and easy to operate.

[0069] S53. Match the high-frequency knowledge extraction results with the preset question-answering engines corresponding to each modal data to obtain a knowledge engine matching relationship.

[0070] Among them, the knowledge is accurately matched to the most appropriate engine. Intelligent customer service usually contains multiple algorithm engines. Different knowledge plays different roles in different engines, which helps to improve the efficiency of system operation. In the intelligent customer service system, different knowledge is adapted to different algorithm engines. An efficient intelligent customer service system is usually composed of multiple engines. The refined and sorted knowledge needs to be matched with the corresponding engine to achieve the best use effect.

[0071] S54. Based on the knowledge engine matching relationship, the high-frequency knowledge extraction results are classified and sorted according to the preset categories and organization methods to obtain the knowledge sorting results corresponding to each question-and-answer engine.

[0072] See also Figure 8 , Figure 8 1 is an example diagram of the execution of constructing multimodal knowledge assets of a port in the method for intelligent customer service of a port provided in an embodiment of the present application. Figure 8 As shown, firstly, the knowledge related to the port field is collected, which can be the existing knowledge base, historical conversation logs, historical work orders, articles and policies in the port field, and other channels of knowledge related to the port field. Then, the collected knowledge is subjected to high-frequency knowledge extraction to obtain three categories of port knowledge, namely factual knowledge, procedural knowledge, and conceptual knowledge. Then, the obtained three categories of port knowledge are split, and for example, irregular specific knowledge, structured knowledge, intentional fuzzy knowledge, table type knowledge, and multi-dimensional computing reasoning knowledge can be obtained. Then, the split knowledge is selected to different question-and-answer engines, and sorted and summarized by the question-and-answer engines to obtain the final knowledge sorting results, which can be used as reference data for the subsequent answers of the port intelligent customer service. For example, irregular specific knowledge is sorted into high-frequency QA list data through the common question-and-answer engine, structured knowledge and intentional fuzzy knowledge are sorted into multi-round mind map data through the task question-and-answer engine, table type knowledge is sorted into knowledge two-dimensional table through the table question-and-answer engine, and multi-dimensional computing reasoning knowledge is sorted into the graph question-and-answer engine through the graph question-and-answer engine.

[0073] See also Fig. 9 , Fig. 9 is a schematic diagram of the execution flow of optimizing the knowledge retrieval results in the port intelligent customer service method provided in the embodiment of the present application, such as Fig. 9 As shown, the following steps S451 to S453 may be included.

[0074] S451. Use a preset vector model to vectorize the data block, perform dimensionality reduction on the vectorized database, and use a Gaussian mixture model to soft cluster the data block after dimensionality reduction to obtain a data block clustering result.

[0075] For example, the data block can be vectorized using a small vector model, and then the uniform manifold approximation and projection technology (UMAP) is used for dimensionality reduction, and finally the Gaussian mixture model is used to soft cluster the reduced data. Unlike traditional hard clustering, soft clustering allows data blocks to belong to multiple clusters, which increases flexibility. The embodiment of the present application also uses the Bayesian Information Criterion (BIC) to determine the optimal number of clusters. For a given Gaussian distribution, BIC not only penalizes model complexity, but also rewards goodness of fit.

[0076] S452, setting data blocks of different categories in the data block clustering results as leaf nodes corresponding to different tree structures, generating corresponding different first summary data according to data blocks of different categories through a preset large language model, and setting the first summary data as the parent node of the leaf node.

[0077] S453, replace the data block with the first summary data, and recursively perform steps from vectorizing the data block using a preset vector model to setting the first summary data as the parent node of the leaf node, until the number of parent nodes is within the limit of the number of context windows of the large language model agent. The purpose of step S453 is to effectively organize the knowledge fragments to form a tree structure with different levels, so as to provide more accurate and comprehensive context information during retrieval.

[0078] See also Fig.10 , Fig.10 1 is an example diagram of the execution of constructing multimodal knowledge assets of a port in the method for intelligent customer service of a port provided in an embodiment of the present application. Fig.10 As shown in the figure, firstly, the data blocks are embedded, dimensionally reduced and clustered to obtain data block clusters of different categories and generate corresponding leaf nodes. Then, the corresponding data of each cluster database is summarized to obtain the parent node of the leaf node. Finally, the parent node is embedded and clustered again to obtain the parent node of the next level, and the subsequent processing is similar.

[0079] As an implementation method, after the above step S4, the target user's evaluation feedback data on the answer data can also be obtained. Then, the evaluation feedback data is analyzed, and it is determined whether to send the evaluation feedback data to the manual customer service according to the data analysis results. Among them, effective opinions are collected, followed up manually, and the customer information database is recorded and updated. When necessary, artificial intelligence can work with customer service representatives to solve problems, improve the speed of solving problems, and flexibly adjust solutions according to user feedback to meet personalized needs.

[0080] The port customer service method provided in the embodiment of the present application develops a port customer service agent based on a large language model, meets the mixed scenario of knowledge question and answer and manual seats, supports multi-modal question and answer output, realizes the ability to automatically generate question and answer pairs and similar questions from documents, and improves the efficiency of maintaining the question and answer knowledge base. In the port intelligent customer service scenario, the powerful multi-modal understanding and generation capabilities of the large model are used to understand the questions raised by users, generate replies that match user needs, and even predict user behavior and provide personalized recommendations, which not only greatly improves the efficiency and quality of customer service, but also saves a lot of manpower and time costs, and improves the processing rate and resolution rate.

[0081] Please continue reading Fig.11 , an embodiment of the present application provides a port intelligent customer service device, Fig.11 : is a schematic diagram of the system structure of the port intelligent customer service device provided in the embodiment of the present application, such as Fig.11 As shown, the port intelligent customer service device 80 includes: a question data module 81 to be analyzed, a question domain data module 82, an intention prediction data module 83 and a customer service answer data module 84. Among them, the question data module to be analyzed is used to receive the multimodal question data sent by the target user through the large language model intelligent agent, and convert the multimodal question data into question data to be analyzed in a preset data format; the question domain data module is used to perform semantic analysis on the question data to be analyzed to obtain the question domain data of the target user; the intention prediction data module is used to use the large language model intelligent agent to understand the intention according to the question domain data and the question data to be analyzed, and obtain the intention prediction data of the target user; the customer service answer data module is used to retrieve the pre-constructed port multimodal knowledge base according to the intention prediction data, use the search results to generate at least one modality of answer data, and return the answer data to the target user.

[0082] As an implementation method, the question domain data module 82 is also specifically used to forward the question data to be analyzed to manual customer service if the question domain data does not include the port domain; if the question domain data includes the port domain, and the number of port domains is 1, then forward the question domain data to the large language model agent corresponding to the port domain; if the question domain data includes the port domain, and the number of port domains is greater than 1, then conduct a domain clarification dialogue with the target user through the large language model agent, and continue to obtain the port domain according to the context of the domain clarification dialogue, until the number of port domains obtained is 1 or forward the question data to be analyzed and the domain clarification dialogue to manual customer service.

[0083] As an implementation method, the customer service answer data module 84 is also specifically used to input intention prediction data and multimodal question data into the question and answer engine corresponding to each modal data; the question and answer engine uses the models corresponding to each modal data in the preset model warehouse to perform multimodal understanding processing to obtain a multimodal understanding result; data matching is performed from a preset multimodal knowledge base according to the intention prediction data to obtain a matching knowledge base list; the multimodal understanding result is used to perform a mixed search in the matching knowledge base included in the matching knowledge base list to obtain knowledge retrieval results corresponding to different modalities; the knowledge retrieval method is used to perform multimodal generation through the models corresponding to each modal data to obtain answer data corresponding to each modality.

[0084] As an implementation method, the customer service answer data module 84 is also specifically used to obtain the in-library rankings of different data blocks in the knowledge retrieval results in the corresponding different matching knowledge bases; the comprehensive score of each data block is calculated by a preset reciprocal ranking fusion formula, and the preset reciprocal ranking fusion formula is as follows:

[0085]

[0086] Where N represents the number of matching knowledge bases, c represents the data block, and r i (c) represents the ranking of the data block in the i-th matching knowledge base, and k represents the constant used to control the ranking; the knowledge retrieval results of different modalities are optimized according to the comprehensive score of the data block.

[0087] As an implementation method, the port intelligent customer service device 80 also includes a knowledge base construction module 85, which is specifically used to collect candidate knowledge data in the port field and port-related fields from preset collection channels; perform high-frequency knowledge extraction on the candidate knowledge data through preset data mining tools and preset knowledge extraction rules to obtain high-frequency knowledge extraction results; match the high-frequency knowledge extraction results with the preset question and answer engines corresponding to each modal data to obtain a knowledge engine matching relationship; based on the knowledge engine matching relationship, classify and organize the high-frequency knowledge extraction results according to preset categories and organizational methods to obtain knowledge organization results corresponding to each question and answer engine.

[0088] As an implementation mode, the customer service answer data module 84 is also specifically used to use a preset vector model to vectorize the data blocks, perform dimensionality reduction on the vectorized database, and use a Gaussian mixture model to soft cluster the data blocks after dimensionality reduction to obtain data block clustering results; set different categories of data blocks in the data block clustering results as leaf nodes corresponding to different tree structures, generate different first summary data corresponding to different categories of data blocks through a preset large language model, and set the first summary data as the parent node of the leaf node; replace the data block with the first summary data, and recursively execute the steps from vectorizing the data block using the preset vector model to setting the first summary data as the parent node of the leaf node, until the number of parent nodes is within the limit of the number of context windows of the large language model agent.

[0089] As an implementation method, the port intelligent customer service device 80 also includes an evaluation feedback data module 86, which is specifically used to obtain the target user's evaluation feedback data on the answer data; perform data analysis on the evaluation feedback data, and determine whether to send the evaluation feedback data to manual customer service based on the data analysis results.

[0090] An embodiment of the present application provides a computer device, see Fig.12 As shown, the computer device 90 includes a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92 and the memory 93 communicate with each other via the communication bus 94; the memory 93 is used to store computer programs; the processor 91 is used to implement the steps of the above-mentioned port intelligent customer service method when executing the program stored in the memory 93.

[0091] The processor 91 may also be a CPU (Central Processing Unit). The processor 91 may also be an integrated circuit chip having signal processing capabilities. The processor 91 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0092] The memory 93 may be, for example, a read-only memory (ROM), and the processor 91 and the memory 93 are connected to each other via a communication bus 94. The communication interface 92 is also connected to the communication bus 94.

[0093] See also Fig.13 , Fig.13It is a structural diagram of a computer-readable storage medium of an embodiment of the present application. The computer-readable storage medium 100 of the embodiment of the present application stores a computer program 101 that can implement all the above methods. The computer program 101 is a computer executable instruction or a computer program, wherein the computer program 101 can be stored in the above computer-readable storage medium in the form of a software product, and can include several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The computer-readable storage medium 100 can be a non-volatile storage medium or a volatile storage medium. The aforementioned computer-readable storage medium 100 can include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0094] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0095] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

[0096] The above is only an implementation method of the present application. It should be pointed out that a person skilled in the art can make improvements without departing from the inventive concept of the present application, but these improvements are within the scope of protection of the present application.

Claims

1. A port intelligent customer service method, characterized in that: include: Receiving multimodal question data sent by a target user through a large language model agent, and converting the multimodal question data into question data to be analyzed in a preset data format; Performing semantic analysis on the question data to be analyzed to obtain question domain data of the target user; Using the large language model agent, understanding the intention according to the question domain data and the question data to be analyzed, and obtaining the intention prediction data of the target user; A pre-constructed port multimodal knowledge base is retrieved according to the intention prediction data, answer data of at least one modality is generated using the retrieval results, and the answer data is returned to the target user.

2. The port intelligent customer service method according to claim 1, characterized in that: After the step of performing semantic analysis on the question data to be analyzed to obtain the question domain data of the target user, the method further includes: If the question field data does not include the port field, forward the question data to be analyzed to manual customer service; If the question domain data includes the port domain, and the number of port domains is 1, forwarding the question domain data to the large language model agent corresponding to the port domain; If the question domain data includes port domains, and the number of port domains is greater than 1, a domain clarification dialogue is conducted between the large language model agent and the target user, and port domains are continuously acquired according to the context of the domain clarification dialogue until the number of port domains acquired is 1 or the question data to be analyzed and the domain clarification dialogue are forwarded to manual customer service.

3. The port intelligent customer service method according to claim 1, characterized in that: The step of retrieving a pre-built port multimodal knowledge base according to the intention prediction data and generating answer data of at least one modality using the retrieval result comprises: Inputting the intention prediction data and the multimodal question data into a question-answering engine corresponding to each modal data; The question-answering engine uses the models corresponding to each modality data in the preset model warehouse to perform multimodal understanding processing to obtain a multimodal understanding result; According to the intention prediction data, data matching is performed from a preset multimodal knowledge base to obtain a matching knowledge base list; Using the multimodal understanding result, a mixed search is performed in the matching knowledge bases included in the matching knowledge base list to obtain knowledge retrieval results corresponding to different modalities; The model corresponding to each modal data is used to perform multimodal generation using the knowledge retrieval method to obtain the answer data corresponding to each modality.

4. The port intelligent customer service method according to claim 3, characterized in that: After the step of using the multimodal understanding result to perform a mixed search in the matching knowledge bases included in the matching knowledge base list to obtain knowledge retrieval results corresponding to different modalities, the method further includes: Obtaining the in-library rankings of different data blocks in the knowledge retrieval results in corresponding different matching knowledge bases; The comprehensive score of each data block is calculated by a preset reciprocal ranking fusion formula, and the preset reciprocal ranking fusion formula is as follows: Wherein, N represents the number of matching knowledge bases, c represents the data block, and r i (c) represents the ranking of the data block in the i-th matching knowledge base, and k represents a constant used to control the ranking; The knowledge retrieval results of the different modalities are optimized according to the comprehensive scores of the data blocks.

5. The port intelligent customer service method according to claim 1, characterized in that: The steps of constructing the port multimodal knowledge base include: Collect candidate knowledge data in the port field and port-related fields from preset collection channels; Performing high-frequency knowledge extraction on the candidate knowledge data by using a preset data mining tool and a preset knowledge extraction rule to obtain a high-frequency knowledge extraction result; Matching the high-frequency knowledge extraction result with the preset question-answering engine corresponding to each modal data to obtain a knowledge engine matching relationship; According to the knowledge engine matching relationship, the high-frequency knowledge extraction results are classified and sorted according to preset categories and organizational methods to obtain knowledge sorting results corresponding to each question and answer engine.

6. The port intelligent customer service method according to claim 4, characterized in that: After the step of optimizing the knowledge retrieval results of different modalities according to the comprehensive scores of the data blocks, the method further includes: Performing vectorization processing on the data blocks using a preset vector model, performing dimensionality reduction processing on the vectorized database, and performing soft clustering on the dimensionality reduced data blocks using a Gaussian mixture model to obtain a data block clustering result; Setting data blocks of different categories in the data block clustering result as leaf nodes corresponding to different tree structures, generating corresponding different first summary data according to the data blocks of different categories by using a preset large language model, and setting the first summary data as the parent node of the leaf node; The data block is replaced with the first summary data, and the steps of vectorizing the data block using a preset vector model to setting the first summary data as the parent node of the leaf node are recursively executed until the number of parent nodes is within the limit of the number of context windows of the large language model agent.

7. The port intelligent customer service method according to claim 1, characterized in that: After the step of using the search results to generate answer data of at least one modality and returning the answer data to the target user, the method further includes: Obtaining evaluation feedback data of the target user on the answer data; Perform data analysis on the evaluation feedback data, and determine whether to send the evaluation feedback data to manual customer service based on the data analysis results.

8. A port intelligent customer service device, characterized in that: include: The question data module to be analyzed is used to receive multimodal question data sent by the target user through the large language model agent, and convert the multimodal question data into question data to be analyzed in a preset data format; A question domain data module is used to perform semantic analysis on the question data to be analyzed to obtain the question domain data of the target user; An intention prediction data module is used to use the large language model agent to understand the intention according to the question domain data and the question data to be analyzed, so as to obtain the intention prediction data of the target user; The customer service answer data module is used to retrieve a pre-built port multimodal knowledge base according to the intention prediction data, use the search results to generate answer data of at least one modality, and return the answer data to the target user.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the port intelligent customer service method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the port intelligent customer service method according to any one of claims 1 to 7 are implemented.

Images