Man-machine interaction method and system of multi-model collaborative architecture

Abstract

The invention provides a man-machine interaction method of a multi-model collaborative architecture, which comprises the following steps: step 1, acquiring a multi-modal input content of a user, preprocessing the input content by a multi-modal recognition model, and sending the preprocessed input content to a large language model; step 2, the big language model accurately understands the information identified by the multi-modal identification model according to a set business rule prompt, and summarizes the information into a plurality of keywords to be output to the dialogue strategy model; and step 3, importing the dialogue strategy model into a user database, performing dialogue skill matching on the keywords according to the formulated dialogue rule, and organizing reply content of the man-machine dialogue. According to the method, a multi-model architecture in which a large language model and a professional small model are combined for use is constructed, and the super-strong generation capability of the large language model is utilized, so that the professional small model of a dialogue strategy is ensured to accurately control a man-machine interaction output result; the dialogue strategy model calls the user database, the user database is isolated from the large language model, and user privacy is prevented from being leaked by the large language model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a human-computer interaction method and system with a multi-model collaborative architecture. Background Technology

[0002] Existing generative large language models possess powerful language understanding and generation capabilities, but their performance in understanding and outputting natural language is limited by the model's capabilities. Sometimes, misunderstandings and incorrect outputs occur, leading to serious adverse consequences in practical applications. For example, the model's output may be inaccurate or unreliable, misleading users into making incorrect judgments. While large language models have strong computational power, they lack intuitive understanding and common sense, easily leading to laughable conclusions. Large language models may also generate offensive or harmful content, resulting in a poor user experience, and in severe cases, violating public order and good morals, and breaking laws. Due to the non-public nature of training data, especially in highly specialized fields, if large language models fail to correctly understand professional skills or private data, they are prone to "talking nonsense with a straight face," becoming artificial stupidity rather than artificial intelligence.

[0003] In addition to the above, existing generative large language models also pose a risk of leaking training data, namely, the leakage of user privacy during the question-and-answer process. Summary of the Invention

[0004] In view of this, the present invention proposes a human-computer interaction method and system with a multi-model collaborative architecture to solve the problems of uncontrolled and inaccurate output results of generative large language models in the prior art.

[0005] The specific technical solution of this invention is as follows:

[0006] A human-computer interaction method based on a multi-model collaborative architecture includes:

[0007] Step 1: Obtain the user's multimodal input content. After the multimodal recognition model preprocesses the input content, it sends it to the large language model.

[0008] Step 2: The large language model, based on the set business rule prompt, accurately understands the information identified by the multimodal recognition model and summarizes it into one or more keywords, which are then output to the dialogue strategy model.

[0009] Step 3: Dialogue strategy model. Import the user database and, based on the established dialogue rules, match keywords with relevant phrases to organize the response content for the human-computer dialogue.

[0010] Specifically, in step 1, the multimodal input content can be text, images, videos, or audio.

[0011] Specifically, the human-computer interaction method also includes: Step 4, the response content is packaged and processed by the multimodal transformation model to form the required modal content and then output.

[0012] Specifically, in step 4, the output modal content is text, images, videos, or audio.

[0013] Specifically, the user database consists of both professional knowledge and private databases.

[0014] Specifically, the large language model is responsible for the NLU part of the traditional NLP model, while the dialogue strategy model is responsible for the DM part of the traditional NLP model.

[0015] A human-computer interaction system with a multi-model collaborative architecture includes: a large language model robot and multiple specialized small model robots; the multiple specialized small model robots include: a multimodal recognition model robot, a dialogue strategy model robot, and a multimodal transformation model robot;

[0016] Multimodal recognition model robots are used to preprocess questions or instructions input by users in the system and output the processing results to large language model robots;

[0017] After accurately understanding the pre-processed questions or instructions, the large language model robot summarizes one or more keywords and outputs them to the dialogue strategy model robot.

[0018] The dialogue strategy model robot organizes dialogue responses after matching keywords according to the set matching rules.

[0019] The beneficial effects of this invention are as follows:

[0020] This application constructs a multi-model architecture that combines a large language model with specialized small models. This architecture efficiently utilizes the powerful generative capabilities of the large language model while ensuring precise control of the human-computer interaction output by the specialized small models of the dialogue strategy. The two models communicate semantically and connect functionally through predetermined rules, achieving precise dialogue functions in specialized fields. This results in intelligent understanding, security and controllability, no misleading messages, and human-centered intelligent human-computer communication. The dialogue strategy model can access the user database, and by isolating the user database from the large language model, it effectively protects user privacy data and prevents user privacy from being leaked to the public network by the large language model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating an embodiment of the human-computer interaction method of the multi-model collaborative architecture of the present invention. Detailed Implementation

[0023] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0024] This application addresses the problems existing in the prior art by proposing a multi-model collaborative architecture for human-computer interaction. Drawing on the workflow of early NLP models, it splits NLP into two parts: NLU (Natural Language Understanding) and DM (Dialogue Management). A large language model is responsible for the NLU part, while specialized small models are responsible for the DM part. The functions of the large language model and the specialized small models are integrated and used together. First, the large language model generates the NLU results. Based on the NLU results, the specialized small models organize the content of the human-computer dialogue according to pre-defined dialogue rules. This makes the output of the large language model completely controllable, avoids misleading users, and significantly improves the accuracy of responses.

[0025] The specific implementation steps of this application are as follows: Figure 1 As shown, it includes:

[0026] Step 1, the first professional small model - the multimodal recognition model, preprocesses the user's input questions or instructions, that is, converts the multimodal content into text content, and then inputs it into the business platform. Users can input questions or instructions in multimodal forms such as text, audio, images, and video.

[0027] Step 2: The large language model in the business platform uses pre-defined business rules (prompt) to understand pre-processed user questions or instructions (i.e., NLU), summarizes them into one or more keywords, and outputs them to the dialogue management backend. Specifically, this includes:

[0028] Step 2.1, understanding language using a large language model;

[0029] Step 2.2: The large language model outputs keywords according to the pre-set prompt requirements;

[0030] Step 3: The second specialized mini-model in the dialogue management backend—the dialogue strategy model—retrieves user databases—both professional knowledge and private databases—and matches keywords with dialogue scripts according to pre-defined dialogue rules to organize the human-computer dialogue response content (i.e., DM). The scripts are pre-designed to ensure that the response content is controlled within a certain range, preventing the robot from replying arbitrarily and thus improving the accuracy of the response. Private data (such as user medical conditions, medical records, etc.) is isolated from the large language model, thus protecting user privacy.

[0031] Step 4: After the response content is packaged and processed by the third professional small model - the multimodal transformation model, it is transformed into the required modal content and output. The modal content here can be text, audio, images, video, etc.

[0032] The following are specific business use cases:

[0033] First embodiment:

[0034] Hospitals are deploying robots to automatically answer patients' pre-visit inquiries. These responses include preliminary diagnoses of diseases or symptoms. The robots need to provide some advice but avoid making overly aggressive diagnoses to prevent adverse effects. The medical field is highly specialized and covers a vast range of knowledge; extensive training would be very costly and the results uncertain. Furthermore, errors could have serious consequences. Therefore, the ideal situation is for the robots to respond within safe limits.

[0035] Pre-defined business rules: "Now let's play a role-playing game. You will play the role of a triage nurse in a hospital. You are familiar with the symptoms of basic illnesses and each department in the hospital, knowing the types of patients each department can handle. I will play the role of a patient seeking treatment, and I will tell you my symptoms. If you understand my symptoms correctly, please summarize them. There can be one or more symptoms, and tell me which clinic I should go to. There can be one or more clinics. Your answer format is: {palpitations, chest tightness, cardiology}. No other information is required. If my description of the symptoms is unclear or you do not understand..." If you don't know the answer, please reply directly with "{I don't know}". If you simply don't know which department to recommend, please reply with "{palpitations, chest tightness, unable to recommend}". Your answer should be limited to the following symptoms: fever, sore throat, cough, headache, limb pain, vomiting, diarrhea, palpitations, difficulty breathing, insomnia, drowsiness, edema, muscle stiffness, chest tightness, arrhythmia, abdominal pain, high blood pressure, high blood sugar. Your answer should be limited to the following departments: Gastroenterology, Cardiology, Endocrinology, Pediatrics, Otolaryngology, Obstetrics and Gynecology, Rheumatology and Immunology, Hepatobiliary and Laparoscopic Surgery, Infectious Diseases, Orthopedic Surgery, Neurology.

[0036] Keyword and dialogue matching rules:

[0037] (1) Palpitation - A; Chest tightness - B; Irregular heartbeat - C; Insomnia - D; Limb pain - E; Headache - F...

[0038] (2) Keywords A, AB, AC, and ABC all match the phrase "1+a1", keywords AD, ABD, ACD, and ABCD all match the phrase "2+a2", and if there is no corresponding match, a fallback phrase is used...

[0039] (3) Script 1: Don't be nervous. There are many reasons for heart discomfort. Strenuous exercise or sudden stimulation of the nervous system can cause unstable heartbeat. It may also be due to some pathological reasons, such as premature ventricular contractions or myocardial ischemia. I suggest you come to the hospital for a 24-hour electrocardiogram so that an accurate diagnosis can be made.

[0040] Script 2: Hello, your insomnia, palpitations, and anxiety are mostly related to autonomic nervous system dysfunction. It would be better if you came to the hospital for a specific examination.

[0041] Script A1: Would you like me to schedule an appointment with the cardiology department? The cardiology department is open from XX, and the earliest you can come for your checkup is XX month XX day at XX hour, is that alright?

[0042] Script A2: Would you like me to schedule an appointment for you with a neurology specialist?

[0043] A fallback approach: Hello, your symptoms are difficult to diagnose directly. You need to come to the hospital for a detailed diagnosis before we can give you specific medical advice. Shall I schedule an appointment for you to have a check-up?

[0044] ...

[0045]

[0046]

[0047] Second embodiment:

[0048] The pre-defined business rules are as follows: "From now on, you will play the role of a recruiter for an insurance company, responsible for recruiting insurance salespeople. I will play the role of an applicant, consulting with you. You are familiar with the insurance company and its business. You need to summarize my various questions into a keyword. These keywords have been pre-defined and match the meaning of the questions. You only need to match my questions with the meaning of the keywords and then output that keyword. Some of these keywords are variables, and the others are fixed words. For the variables, you need to extract the variable values ​​separately. For example, if I say 'Let's schedule an interview tomorrow afternoon,' you need to output {schedule an interview time, time = tomorrow}, which is {keyword, variable name = variable}." The range of values ​​for variables will be defined by me, and you must assign values ​​according to my specifications. For other non-variables, simply output the keywords themselves, such as: {questioning fraud} or {lack of business confidence}. No other content is required. If you understand my question and can make a match, answer according to the above rules. If my question is not within the keywords I have specified, or if you do not understand it, simply answer {don't know}. The pre-defined keywords are: scheduled interview time, age, questioning fraud, inquiring about salary, inquiring about working hours, inquiring about working location, and lack of business confidence. The variables are: scheduled interview time (variable name: time, value range is time), and age (variable name: age, value range is number).

[0049]

[0050] Third Implementation Example (Order Inquiry):

[0051] Step 1: The user can input the question "When will my order arrive?" in a multimodal format. The multimodal recognition model converts the multimodal content into text content and then inputs it into the large language model.

[0052] Step 2, the large language model, using the pre-defined business rule prompt, understands the pre-processed user question and summarizes it into one or more keywords, such as "{asking about order arrival time}", which are then output to the dialogue strategy model.

[0053] Step 3, Dialogue Strategy Model: Based on the previously obtained user identity information "phone number, user ID", the system queries the order number in the user database and checks the order progress on the logistics platform using the order number. The query results are then used to organize the dialogue response using preset language rules: "Mr. Zhang, your order has just arrived in Shenzhen. Our courier will deliver the package to your door as soon as possible. If you have any other questions, please feel free to ask me."

[0054] Step 4: The response content is packaged and processed by the multimodal model to form a "text + voice-over" output.

[0055] This application also proposes a multi-model collaborative architecture for human-computer interaction, including a large language model robot and multiple specialized small model robots. The multiple specialized small model robots include a multimodal recognition model robot, a dialogue strategy model robot, and a multimodal transformation model robot. The multimodal recognition model robot preprocesses the questions or instructions input by the user in the system and outputs the processing results to the large language model robot. The large language model robot accurately understands the preprocessed questions or instructions, summarizes one or more keywords, and outputs them to the dialogue strategy model robot. The dialogue strategy model robot performs phrasal matching on the keywords according to pre-set matching rules and organizes the dialogue response content as the final output of the human-computer interaction system.

[0056] The beneficial effects of this application are as follows:

[0057] (1) This application constructs a multi-model architecture that combines a large language model and a professional small model. It not only makes efficient use of the powerful generation capability of the large language model, but also ensures the precise control of the human-computer interaction output results by the professional small model of the dialogue strategy. The two models transmit semantics and connect functions through predetermined rules to achieve precise dialogue function in professional fields. It achieves intelligent understanding, security and controllability, no misleading, and intelligent human-computer communication with humanistic care.

[0058] (2) The dialogue strategy model can call the user database. By isolating the user database from the large language model, the privacy data of users is effectively protected and the privacy of users is prevented from being leaked to the public network by the large language model.

[0059] (3) This application splits the human-computer dialogue processing flow, splits the original NLP model into NLU part and DM part, and lets the large language model only handle the NLU part, while the dialogue strategy model is responsible for the DM part. Thus, the large language model can be controlled to diverge arbitrarily from the output level, and has the ability to conduct accurate human-computer dialogue.

[0060] (4) This application integrates multimodal recognition technology and multimodal conversion technology, enabling human-computer interaction to be applied in multiple modes, including text, audio, images, video, etc. When the amount of data reaches a certain scale, the more complete the modal types, the better the effect of the multimodal model.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human-computer interaction method with a multi-model collaborative architecture, characterized in that, include: Step 1: Obtain the user's multimodal input content. After the multimodal recognition model preprocesses the input content, it sends it to the large language model. Step 2: The large language model, according to the set business rule prompt, accurately understands the information identified by the multimodal recognition model and summarizes it into one or more keywords, which are then output to the dialogue strategy model. Step 3: The dialogue strategy model is imported into the user database, and according to the established dialogue rules, the keywords are matched with the corresponding dialogue messages to organize the response content for the human-computer dialogue.

2. The human-computer interaction method as described in claim 1, characterized in that, In step 1, the multimodal input content is text, images, videos, or audio.

3. The human-computer interaction method as described in claim 1, characterized in that, The human-computer interaction method further includes: step 4, where the response content is packaged and processed by a multimodal transformation model to form the required modal content before being output.

4. The human-computer interaction method as described in claim 3, characterized in that, In step 4, the output modal content is text, images, video, or audio.

5. The human-computer interaction method as described in claim 1, characterized in that, The user database is a professional knowledge and private database.

6. The human-computer interaction method as described in claim 1, characterized in that, The large language model is responsible for the NLU part of the traditional NLP model, and the dialogue strategy model is responsible for the DM part of the traditional NLP model.

7. A human-computer interaction system with a multi-model collaborative architecture, characterized in that, include: Large language model robot, and several specialized small model robots; Several specialized miniature robot models include: multimodal recognition model robot, dialogue strategy model robot, and multimodal transformation model robot; The multimodal recognition model robot is used to preprocess the questions or instructions input by the user in the system and output the processing results to the large language model robot; The large language model robot accurately understands the preprocessed question or instruction, summarizes one or more keywords, and outputs them to the dialogue strategy model robot. The dialogue strategy model robot organizes dialogue response content after matching the keywords according to the set matching rules.

Images