Interface system and method based on interaction of natural language question and answer interface and large model

Abstract

The invention discloses an interface system and method for interaction between a natural language question and answer interface and a large model, and the method comprises the following steps: firstly, carrying out the multi-modal information interaction, and carrying out the natural language information interaction through employing a plurality of modal modes of text, sound, image, video and sensing; wherein the text is interacted in a webpage / APP, input method and file / media mode, the sound is interacted in a webpage / APP and microphone mode, the image is interacted in a screen and handwriting pen mode, the video is interacted in a screen and camera mode, and the sensing is interacted in a sensor mode; therefore, a multi-modal large model system is understood and generated. The large model adopts a modal interaction mode, so that not only can natural language questions and answers be supported, but also images, actions and characters can be fed back. Therefore, the intention of the user can be better understood, complex situation understanding and reaction close to human beings are realized, and the interaction process is more humanized.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language question answering and large model interaction, and specifically to an interface system and method based on the interaction between a natural language question answering interface and a large model. Background Art

[0002] An interactive large model transcends the limitations of a single modality, supports question and answer in various natural languages, can provide real-time, "human-like" understanding and feedback, and can communicate with people seamlessly.

[0003] The core of the interactive large model is to generate natural, fluent and emotional voice output according to the user's question and answer needs, making the feedback more realistic. At the same time, it imitates human emotional states such as joy, sadness or anger, and then expresses the corresponding emotions by adjusting the tone, rhythm and volume, providing users with more intuitive information expression, thereby achieving a more efficient and rich communication experience.

[0004] However, the current large-scale model interaction has a strong ability to recognize natural language, so questions and answers generated in natural language can generally provide efficient and accurate feedback. However, when faced with images, actions, and text, it is unable to capture and parse the details in complex images and videos, and it is impossible for users to interact with the terminal in the form of text, images, sounds, videos, and sensor data. Not only does it result in a single form of interaction between users and terminals, limiting the use of some disabled people who cannot make sounds or write text, but it also cannot generate various cross-media content to provide users with more intuitive information expression, reducing the efficiency of the interaction between users and models in question and answer.

[0005] Therefore, an interface system and method based on natural language question-answering interface and large model interaction are proposed to solve this problem. Summary of the invention

[0006] The purpose of the present invention is to provide an interface system and method based on a natural language question-answering interface interacting with a large model, which solves the problem.

[0007] To achieve the above purpose, the present invention provides the following technical solution: an interface system and method based on natural language question-answering interface and large model interaction, comprising the following steps:

[0008] Step 1: Multimodal information interaction: natural language information interaction is carried out through multiple modal methods such as text, sound, image, video, and sensor. Text is interacted through web pages / APPs, input methods, and files / media; sound is interacted through web pages / APPs and microphones; images are interacted through screens and styluses; videos are interacted through screens and cameras; and sensors are interacted through sensors. This allows understanding and generation of a multimodal large model system.

[0009] Step 2: Identify the intention of question and answer: The multimodal model system processes multiple natural languages ​​such as text, sound, image, video, and sensor to analyze the user's intention and needs. At the same time, the user's questions are matched by thresholds. If the matching degree is lower than the threshold, the multimodal model system cannot determine the intention and determines that the recognition intention has failed. On the contrary, if the matching degree is higher than the threshold, the recognition is successful, and then the conversation question and answer process is entered;

[0010] Step 3: Communication through words: Through the multimodal model system, users can communicate with the terminal in various data types such as text, image, sound, video, and sensor, so as to better understand the user's intention. By identifying the user's question and answer words, the multimodal model system can convert the user's multimodal input into structured data, so as to accurately judge the user's core needs and select the appropriate response strategy to generate a reply;

[0011] Step 4: Knowledge base matching reply words: The knowledge base can be a structured database or a complex database based on a knowledge graph. To ensure accurate responses, the multimodal model system assigns a matching score to each candidate answer and selects the answer with the highest score. The multimodal model system also selects the best answer based on the matching score to ensure the relevance and accuracy of the reply words. This enables the multimodal model system to generate a variety of cross-media reply words;

[0012] Step 5: Multi-round dialogue: To improve user experience, the multimodal model system can support multi-round dialogue so that it can continue to follow up and provide accurate answers when users have follow-up questions. Based on the dialogue state management technology, a state machine or a neural network-based dialogue management model (such as Transformer) is used to maintain the dialogue context. At the same time, the context is tracked to record the user's previous questions and the answers of the multimodal model system, so as to maintain consistency in the response words in multiple rounds of dialogue and avoid repeated questions.

[0013] Preferably, in step 1, the medium based on information transmission can not only be text, sound, image, video, and sensing, but also radar, infrared, GPS, and accelerometer sensors can be used as modal interaction data.

[0014] Preferably, in step 1, structured text and unstructured text are generated according to natural language for input, wherein classification is performed through language synchronization, images can be input as layout structures or document charts, and videos can be input as animations, real videos, and conference recordings.

[0015] Preferably, in step 2, the multimodal model system first needs to perform basic NLP processing on the user's input, including word segmentation, part-of-speech tagging, named entity recognition, and image and speech error correction, such as removing stop words, processing spelling and text errors, etc., to ensure the correctness of the image and speech data.

[0016] Preferably, in step 2, the multimodal model system will collect unidentified questions and help the model learn new expressions through manual or automatic labeling. At the same time, it will use incremental training or transfer learning and other methods to update the system model to ensure that the multimodal model system can adapt to the ever-changing user needs.

[0017] Preferably, in step 3, for questions that the system cannot recognize, the system enters the model training module and regenerates the corresponding question-and-answer script to continuously improve recognition accuracy and reduce unrecognized situations.

[0018] Preferably, in step 4, the structured database is more suitable for processing FAQ-type queries, while the knowledge graph is suitable for processing highly correlated content. Commonly used algorithms include TF-IDFBM25 and matching models based on deep learning (such as Sentence-BERT).

[0019] Preferably, in step 4, by combining MLOps (machine learning operations) practices, new data is regularly collected, the model system is retrained, and the new model is automatically deployed to the production environment, so that the feedback is more realistic, thereby providing personalized services.

[0020] Preferably, in step 5, the multimodal model system will collect unrecognized questions and help learn new expressions through manual or automatic annotation, so as to generate suitable answers and feedback to the user, making the interaction process more humane.

[0021] Preferably, in step 5, the answering process in different situations is designed by defining a dialogue strategy, for example, continuing to provide detailed information in depth, or ending the dialogue, so as to improve the user experience.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The invention patent enables the large model to adopt a multi-modal interaction method of text, sound, image, video, and sensor, so that it can not only support natural language questions and answers, but also provide feedback on images, actions, and text. It can better understand user intentions, achieve complex situational understanding and response close to humans, and has strong understanding and generation capabilities. It can integrate various types of natural languages ​​for unified semantics, situational analysis, and contextual association, making the interaction process more humane;

[0024] 2. The patent of this invention supports multi-round dialogues by making the multimodal model system support multi-round dialogues, so that when users have follow-up questions, they can continue to follow up and provide accurate answers. For questions that the model system cannot recognize, it will enter the model training module and re-recognize and train to help learn new expressions to continuously improve recognition accuracy and reduce unrecognized situations. At the same time, ensure the relevance and accuracy of the reply words, so as to maintain the consistency of the reply words in multi-round dialogues and improve the efficiency of question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the question-answering process of the present invention;

[0026] Figure 2 Schematic diagram of multimodal interaction of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be described in more detail below by way of examples, which are merely illustrative and do not in any way limit the scope of the present invention.

[0028] The present invention provides a technical solution: an interface system and method based on a natural language question-answering interface interacting with a large model, comprising the following steps:

[0029] Step 1: Multimodal information interaction: natural language information interaction is carried out through multiple modal methods such as text, sound, image, video, and sensor. Text is interacted through web pages / APPs, input methods, and files / media; sound is interacted through web pages / APPs and microphones; images are interacted through screens and styluses; videos are interacted through screens and cameras; and sensors are interacted through sensors. This allows understanding and generation of a multimodal large model system.

[0030] Step 2: Identify the intention of question and answer: The multimodal model system processes multiple natural languages ​​such as text, sound, image, video, and sensor to analyze the user's intention and needs. At the same time, the user's questions are matched by thresholds. If the matching degree is lower than the threshold, the multimodal model system cannot determine the intention and determines that the recognition intention has failed. On the contrary, if the matching degree is higher than the threshold, the recognition is successful, and then the conversation question and answer process is entered;

[0031] Step 3: Communication through words: Through the multimodal model system, users can communicate with the terminal in various data types such as text, image, sound, video, and sensor, so as to better understand the user's intention. By identifying the user's question and answer words, the multimodal model system can convert the user's multimodal input into structured data, so as to accurately judge the user's core needs and select the appropriate response strategy to generate a reply;

[0032] Step 4: Knowledge base matching reply words: The knowledge base can be a structured database or a complex database based on a knowledge graph. To ensure accurate responses, the multimodal model system assigns a matching score to each candidate answer and selects the answer with the highest score. The multimodal model system also selects the best answer based on the matching score to ensure the relevance and accuracy of the reply words. This enables the multimodal model system to generate a variety of cross-media reply words;

[0033] Step 5: Multi-round dialogue: To improve user experience, the multimodal model system can support multi-round dialogue so that it can continue to follow up and provide accurate answers when users have follow-up questions. Based on the dialogue state management technology, a state machine or a neural network-based dialogue management model (such as Transformer) is used to maintain the dialogue context. At the same time, the context is tracked to record the user's previous questions and the answers of the multimodal model system, so as to maintain consistency in the response words in multiple rounds of dialogue and avoid repeated questions.

[0034] Embodiment 1:

[0035] First, multimodal information interaction is carried out, and natural language information interaction is carried out through multiple modal methods such as text, sound, image, video, and sensor. Among them, text is interacted through web pages / APPs, input methods, files / media, sound is interacted through web pages / APPs, microphones, images are interacted through screens and styluses, videos are interacted through screens and cameras, and sensors are interacted through sensors. In this way, a multimodal large model system is understood and generated. Then, the question-answering intention is identified, and multiple natural languages ​​such as text, sound, image, video, and sensor are processed through the multimodal model system to analyze the user's intention and needs. At the same time, the user's question is matched through a threshold. If the matching degree is lower than the threshold, the multimodal model If the system cannot determine the intent, it will determine that the recognition of the intent has failed. Otherwise, if the matching degree is higher than the threshold, the recognition is successful, and then the process of question and answer dialogue is entered; then the dialogue communication is carried out, and the multimodal model system allows users to communicate with the terminal in various data types such as text, images, sounds, videos, and sensors, so as to better understand the user's intention, and by identifying the user's question and answer dialogue, the multimodal model system converts the user's multimodal input into structured data, so as to accurately judge the user's core needs and select the appropriate response strategy to generate a reply; then the reply dialogue is matched through the knowledge base. The knowledge base can be a structured database or a complex database based on a knowledge graph. In order to ensure the accuracy of the response, the multimodal model system will assign a matching score to each candidate answer and select the answer with the highest score. In addition, the multimodal model system will select the best answer based on the matching score to ensure the relevance and accuracy of the reply dialogue. This enables the multimodal model system to generate various cross-media response scripts; finally, multiple rounds of dialogue are carried out. In order to improve the user experience, the multimodal model system can support multiple rounds of dialogue so that it can continue to follow up and provide accurate answers when the user has follow-up questions. Based on the technology of dialogue state management, a state machine or a neural network-based dialogue management model (such as Transformer) is used to maintain the dialogue context. At the same time, the context is tracked to record the user's previous questions and the answers of the multimodal model system, so as to maintain the consistency of the response script in multiple rounds of dialogue and avoid repeated questions.

[0036] Embodiment 2:

[0037] In the first embodiment, the following steps are added:

[0038] In step 1, the media based on information transmission can not only be text, sound, image, video, and sensor methods, but also radar, infrared, GPS, and accelerometer sensors can be used as modal interaction data, and structured text and unstructured text can be generated according to natural language for input, where classification is performed through language synchronization, images can be input as layout structures or document charts, and videos can be input as animations, real videos, and conference recordings.

[0039] First, multimodal information interaction is carried out, and natural language information interaction is carried out through multiple modal methods such as text, sound, image, video, and sensor. Among them, text is interacted by means of web pages / APPs, input methods, and files / media, sound is interacted by means of web pages / APPs and microphones, images are interacted by means of screens and styluses, videos are interacted by means of screens and cameras, and sensors are interacted by means of sensors. In this way, a multimodal large model system is understood and generated. The media based on information transmission can not only be text, sound, image, video, and sensor methods, but also radar, infrared, GPS, and accelerometer sensors can be used as modal interaction data, and structured text and unstructured text are generated according to natural language for input. Classification is carried out through language synchronization, images can be input into layout structures or document charts, and videos can be input into animations, real videos, and conference recordings. Then, the question-answering intention is identified, and the multimodal model system processes multiple natural languages ​​such as text, sound, image, video, and sensor to analyze the user's intention and needs. At the same time, the user's questions are matched through the threshold. If the matching degree is lower than the threshold, the multimodal model system cannot determine the intention and determines that the recognition intention fails. On the contrary, if the matching degree is higher than the threshold, the recognition is successful, and then the speech question-answering process is entered; then the speech communication is carried out, and the multimodal model system allows users to communicate with the terminal in multiple data types such as text, image, sound, video, and sensor to better understand the user's intention, and by identifying the user's question-answering speech, the multimodal model system converts the user's multimodal input into structured data, so as to accurately judge the user's core needs and select the appropriate response strategy to generate a reply; then the reply speech is matched through the knowledge base. The knowledge base can be a structured database or a complex database based on the knowledge graph. In order to ensure the accuracy of the response, the multimodal model system will assign a matching score to each candidate answer and select the answer with the highest score. In addition, the multimodal model system will select the best answer based on the matching score to ensure the relevance and accuracy of the reply speech. This enables the multimodal model system to generate various cross-media response scripts; finally, multiple rounds of dialogue are carried out. In order to improve the user experience, the multimodal model system can support multiple rounds of dialogue so that it can continue to follow up and provide accurate answers when the user has follow-up questions. Based on the technology of dialogue state management, a state machine or a neural network-based dialogue management model (such as Transformer) is used to maintain the dialogue context. At the same time, the context is tracked to record the user's previous questions and the answers of the multimodal model system, so as to maintain the consistency of the response script in multiple rounds of dialogue and avoid repeated questions.

[0040] Embodiment three:

[0041] In the second embodiment, the following steps are added:

[0042] In step 2, the multimodal model system first needs to perform basic NLP processing on the user's input, including word segmentation, part-of-speech tagging, named entity recognition, and image and voice error correction, such as removing stop words, processing spelling and text errors, etc., to ensure the correctness of image and voice data. The multimodal model system will collect unrecognized problems and help the model learn new expressions through manual or automatic annotation. At the same time, it uses incremental training or transfer learning to update the system model to ensure that the multimodal model system can adapt to the ever-changing user needs.

[0043] In step 3, for questions that the system cannot recognize, the system enters the model training module and regenerates the corresponding question-and-answer script to continuously improve recognition accuracy and reduce unrecognized situations.

[0044] First, multimodal information interaction is carried out, and natural language information interaction is carried out through multiple modal methods such as text, sound, image, video, and sensor. Among them, text is interacted by means of web pages / APPs, input methods, and files / media, sound is interacted by means of web pages / APPs and microphones, images are interacted by means of screens and styluses, videos are interacted by means of screens and cameras, and sensors are interacted by means of sensors. In this way, a multimodal large model system is understood and generated. The media based on information transmission can not only be text, sound, image, video, and sensor methods, but also radar, infrared, GPS, and accelerometer sensors can be used as modal interaction data, and structured text and unstructured text are generated according to natural language for input. Classification is carried out through language synchronization, images can be input into layout structures or document charts, and videos can be input into animations, real videos, and conference recordings. Then, the question-and-answer intention is identified, and the multimodal model system is used to process multiple natural languages ​​such as text, sound, image, video, and sensor to analyze the user's intention and needs. At the same time, the user's questions are matched through the threshold. If the matching degree is lower than the threshold, the multimodal model system cannot determine the intention and determines that the recognition intention has failed. On the contrary, if the matching degree is higher than the threshold, the recognition is successful and the conversation question-and-answer process is entered. The multimodal model system first needs to perform basic NLP processing on the user's input, including word segmentation, part-of-speech tagging, named entity recognition, and image and voice error correction, such as removing stop words, processing spelling and text errors, etc., to ensure the correctness of image and voice data. The multimodal model system will collect unrecognized questions and help the model learn new expressions through manual or automatic labeling. At the same time, incremental training or transfer learning and other methods are used to update the system model to ensure that the multimodal model system can adapt to the ever-changing user needs. After that, the communication is carried out in the form of words. Through the multimodal model system, users can communicate with the terminal in various data types such as text, images, sound, video, and sensors, so as to better understand the user's intentions. By identifying the user's question and answer words, the multimodal model system converts the user's multimodal input into structured data, so as to accurately judge the user's core needs and select the appropriate response strategy to generate a reply. For questions that the system cannot recognize, the system enters the model training module and regenerates the corresponding question and answer words to continuously improve the recognition accuracy and reduce the number of unrecognized situations. Then, the reply words are matched through the knowledge base. The knowledge base can be a structured database or a complex database based on a knowledge graph. In order to ensure the accuracy of the response, the multimodal model system will assign a matching score to each candidate answer and select the answer with the highest score. In addition, the multimodal model system will select the best answer based on the matching score to ensure the relevance and accuracy of the reply words.This enables the multimodal model system to generate various cross-media response scripts; finally, multiple rounds of dialogue are carried out. In order to improve the user experience, the multimodal model system can support multiple rounds of dialogue so that it can continue to follow up and provide accurate answers when the user has follow-up questions. Based on the technology of dialogue state management, a state machine or a neural network-based dialogue management model (such as Transformer) is used to maintain the dialogue context. At the same time, the context is tracked to record the user's previous questions and the answers of the multimodal model system, so as to maintain the consistency of the response script in multiple rounds of dialogue and avoid repeated questions.

[0045] Embodiment 4:

[0046] In the third embodiment, the following steps are added:

[0047] In step 4, structured databases are more suitable for processing FAQ-type queries, while knowledge graphs are suitable for processing highly correlated content. Commonly used algorithms include TF-IDFBM25 and deep learning-based matching models (such as Sentence-BERT). By combining MLOps (machine learning operations) practices, new data is collected regularly, the model system is retrained, and the new model is automatically deployed to the production environment, making the feedback more realistic and providing personalized services.

[0048] In step 5, the multimodal model system will collect unrecognized questions and help learn new expressions through manual or automatic annotation, so as to generate appropriate answers and feedback to users, making the interaction process more humane, and designing the answer process in different situations by defining dialogue strategies. For example, continue to provide detailed information in depth, or end the dialogue, so as to improve the user experience.

[0049] First, multimodal information interaction is carried out, and natural language information interaction is carried out through multiple modal methods such as text, sound, image, video, and sensor. Among them, text is interacted by means of web pages / APPs, input methods, and files / media, sound is interacted by means of web pages / APPs and microphones, images are interacted by means of screens and styluses, videos are interacted by means of screens and cameras, and sensors are interacted by means of sensors. In this way, a multimodal large model system is understood and generated. The media based on information transmission can not only be text, sound, image, video, and sensor methods, but also radar, infrared, GPS, and accelerometer sensors can be used as modal interaction data, and structured text and unstructured text are generated according to natural language for input. Classification is carried out through language synchronization, images can be input into layout structures or document charts, and videos can be input into animations, real videos, and conference recordings. Then, the question-and-answer intention is identified, and the multimodal model system is used to process multiple natural languages ​​such as text, sound, image, video, and sensor to analyze the user's intention and needs. At the same time, the user's questions are matched through the threshold. If the matching degree is lower than the threshold, the multimodal model system cannot determine the intention and determines that the recognition intention has failed. On the contrary, if the matching degree is higher than the threshold, the recognition is successful and the conversation question-and-answer process is entered. The multimodal model system first needs to perform basic NLP processing on the user's input, including word segmentation, part-of-speech tagging, named entity recognition, and image and voice error correction, such as removing stop words, processing spelling and text errors, etc., to ensure the correctness of image and voice data. The multimodal model system will collect unrecognized questions and help the model learn new expressions through manual or automatic labeling. At the same time, incremental training or transfer learning and other methods are used to update the system model to ensure that the multimodal model system can adapt to the ever-changing user needs. After that, the communication is carried out in the form of words. Through the multimodal model system, users can communicate with the terminal in various data types such as text, images, sound, video, and sensors, so as to better understand the user's intentions. By identifying the user's question and answer words, the multimodal model system converts the user's multimodal input into structured data, so as to accurately judge the user's core needs and select the appropriate response strategy to generate a reply. For questions that the system cannot recognize, the system enters the model training module and regenerates the corresponding question and answer words to continuously improve the recognition accuracy and reduce the number of unrecognized situations. Then, the reply words are matched through the knowledge base. The knowledge base can be a structured database or a complex database based on a knowledge graph. In order to ensure the accuracy of the response, the multimodal model system will assign a matching score to each candidate answer and select the answer with the highest score. In addition, the multimodal model system will select the best answer based on the matching score to ensure the relevance and accuracy of the reply words.This enables the multimodal model system to generate a variety of cross-media response scripts; structured databases are more suitable for processing FAQ-type queries, while knowledge graphs are suitable for processing highly correlated content. Commonly used algorithms include TF-IDFBM25 and deep learning-based matching models (such as Sentence-BERT). By combining MLOps (machine learning operation and maintenance) practices, new data is collected regularly, the model system is retrained, and the new model is automatically deployed to the production environment, making the feedback more realistic and providing personalized services. Finally, multiple rounds of dialogue are conducted. In order to improve the user experience, the multimodal model system can support multiple rounds of dialogue so that it can continue to follow up and provide accurate answers when the user has follow-up questions. Based on the dialogue state management technology, a state machine or a neural network-based dialogue management model (such as Transformer) is used to maintain the dialogue context, and the context is tracked to record the user's previous questions and the answers of the multimodal model system, so as to maintain the consistency of the reply words in multiple rounds of dialogue and avoid repeated questions; the multimodal model system will collect unrecognized questions and help learn new expressions through manual or automatic annotation, so as to generate suitable answers and feedback to the user, making the interaction process more humane, and designing the answer process in different situations by defining dialogue strategies. For example, continue to provide specific information in depth, or end the dialogue, so as to improve the user experience.

[0050] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An interface system and method based on natural language question-answering interface and large model interaction, characterized in that: The following steps are involved: Step 1: Multimodal information interaction: natural language information interaction is carried out through multiple modal methods such as text, sound, image, video, and sensor; Text is interacted through web pages / APPs, input methods, files / media, sound is interacted through web pages / APPs and microphones, images are interacted through screens and styluses, videos are interacted through screens and cameras, and sensing is interacted through sensors; thus understanding and generating a multimodal large model system; Step 2: Identify the intention of question and answer: The multimodal model system processes multiple natural languages ​​such as text, sound, image, video, and sensor to analyze the user's intention and needs. At the same time, the user's questions are matched by thresholds. If the matching degree is lower than the threshold, the multimodal model system cannot determine the intention and determines that the recognition intention has failed. On the contrary, if the matching degree is higher than the threshold, the recognition is successful, and then the conversation question and answer process is entered; Step 3: Communication through words: Through the multimodal model system, users can communicate with the terminal in various data types such as text, image, sound, video, and sensor, so as to better understand the user's intention. By identifying the user's question and answer words, the multimodal model system can convert the user's multimodal input into structured data, so as to accurately judge the user's core needs and select the appropriate response strategy to generate a reply; Step 4: Knowledge base matching reply words: The knowledge base can be a structured database or a complex database based on a knowledge graph. To ensure accurate responses, the multimodal model system assigns a matching score to each candidate answer and selects the answer with the highest score. The multimodal model system also selects the best answer based on the matching score to ensure the relevance and accuracy of the reply words. This enables the multimodal model system to generate a variety of cross-media reply words; Step 5: Multi-round dialogue: To improve user experience, the multimodal model system can support multi-round dialogue so that it can continue to follow up and provide accurate answers when users have follow-up questions. Based on the dialogue state management technology, a state machine or a neural network-based dialogue management model (such as Transformer) is used to maintain the dialogue context. At the same time, the context is tracked to record the user's previous questions and the answers of the multimodal model system, so as to maintain consistency in the response words in multiple rounds of dialogue and avoid repeated questions.

2. According to claim 1, an interface system and method for interacting with a large model based on a natural language question-answering interface, characterized in that: In step 1, the medium based on information transmission can not only be text, sound, image, video, and sensor, but also radar, infrared, GPS, and accelerometer sensors can be used as modal interaction data.

3. According to claim 1, an interface system and method based on natural language question-answering interface and large model interaction, characterized in that: In step 1, structured text and unstructured text are generated according to natural language for input, wherein classification is performed through language synchronization, images can be input as layout structures or document charts, and videos can be input as animations, real videos, and conference recordings.

4. According to claim 1, an interface system and method based on natural language question-answering interface and large model interaction, characterized in that: In step 2, the multimodal model system first needs to perform basic NLP processing on the user's input, including word segmentation, part-of-speech tagging, named entity recognition, and image and speech error correction, such as removing stop words, processing spelling and text errors, etc., to ensure the correctness of the image and speech data.

5. According to claim 1, an interface system and method based on natural language question-answering interface and large model interaction, characterized in that: In step 2, the multimodal model system will collect unidentified questions and help the model learn new expressions through manual or automatic labeling. At the same time, it will use incremental training or transfer learning to update the system model to ensure that the multimodal model system can adapt to the ever-changing user needs.

6. The interface system and method based on natural language question-answering interface and large model interaction according to claim 1, characterized in that: In step 3, for questions that the system cannot recognize, the system enters the model training module and regenerates the corresponding question-and-answer script to continuously improve recognition accuracy and reduce unrecognized situations.

7. The interface system and method based on natural language question-answering interface and large model interaction according to claim 1, characterized in that: In step 4, structured databases are more suitable for processing FAQ-type queries, while knowledge graphs are suitable for processing highly correlated content. Commonly used algorithms include TF-IDFBM25 and matching models based on deep learning (such as Sentence-BERT).

8. The interface system and method based on natural language question-answering interface and large model interaction according to claim 1, characterized in that: In step 4, by combining MLOps (machine learning operations) practices, new data is collected regularly, the model system is retrained, and the new model is automatically deployed to the production environment, making the feedback more realistic and providing personalized services.

9. The interface system and method based on natural language question-answering interface and large model interaction according to claim 1, characterized in that: In step 5, the multimodal model system will collect unrecognized questions and help learn new expressions through manual or automatic annotation, so as to generate appropriate answers and feedback to users, making the interaction process more humane.

10. The interface system and method based on natural language question-answering interface and large model interaction according to claim 1, characterized in that: In step 5, the answering process in different situations is designed by defining the dialogue strategy, for example, continuing to provide detailed information in depth, or ending the dialogue, so as to improve the user experience.

Images