Man-machine interaction behavior prediction method and system based on large language model

Abstract

The invention provides a man-machine interaction behavior prediction method and system based on a large language model, and relates to the technical field of artificial intelligence. By analyzing the multi-modal instruction of the user to extract the semantic tag, integrating information and constructing the knowledge base, and combining vector retrieval and graph retrieval to obtain related knowledge and generate optimized cue words, efficient and accurate user behavior prediction and scene adjustment are realized, and the intelligent level of man-machine interaction is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for predicting human-computer interaction behavior based on a large language model. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and big data technologies, especially the emergence of deep learning and large language models (LLM), human-computer interaction systems have been significantly improved. Early human-computer interaction primarily relied on rule-based systems, which struggled to understand complex user commands and execution environments. However, with the introduction of multimodal technologies, systems can now accept information input through various forms such as voice, images, and text, greatly enhancing the user experience. Simultaneously, advancements in semantic understanding and natural language processing technologies have enabled machines to more accurately interpret user intent, leading to more effective interactions.

[0003] Currently, an increasing number of studies focus on using large-scale language models for predicting human-computer interaction behavior. The trends in related technologies are mainly reflected in the following aspects: First, the scale and complexity of models are constantly increasing, enabling them to handle larger and more complex input data, thereby improving the accuracy of semantic understanding. Second, the application of hybrid RAG (Retrieval Augmented Generation) technology can combine information retrieval with content generation, enhancing the system's ability to handle complex instructions. Furthermore, the increasing prevalence of multimodal input methods promotes the synergistic development between natural language processing, computer vision, and sensor fusion, making human-computer interaction systems more intelligent and user-friendly.

[0004] Despite significant progress in human-computer interaction (HCI) technology, several shortcomings remain. First, existing knowledge bases often face limitations related to locality and data security when processing domain-specific knowledge, resulting in insufficient knowledge coverage. Second, large language models may still produce "illusion" responses that do not accurately reflect reality, affecting the accuracy of final behavior predictions. Finally, the high dependence on input data makes the system vulnerable to unknown or changing situations, hindering efficient and accurate real-time responses. These shortcomings restrict the further application and development of HCI systems. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for predicting human-computer interaction behavior based on a large language model. By combining the reasoning ability of a large language model with hybrid retrieval technology, it achieves accurate prediction of human-computer interaction behavior, significantly improving the intelligence level and response efficiency of human-computer interaction.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for predicting human-computer interaction behavior based on a large language model includes: Parse the received multimodal commands from the user to extract command semantic tags and scene semantic tags; Integrate the instruction semantic tags and the scene semantic tags to establish an association model between instruction tags and scene tags, so as to form comprehensive semantic information; A local knowledge base is constructed to store initial knowledge fragments related to user commands and scenarios; the local knowledge base includes local knowledge text and knowledge graph. The local knowledge text and the knowledge graph are retrieved by vector retrieval and graph retrieval respectively to obtain retrieval knowledge fragments related to the comprehensive semantic information; The retrieved knowledge fragments are formatted to obtain standardized knowledge fragments with a standardized structure for processing by a large language model. Suitable prompt words are generated based on the standardized knowledge fragments and the user's instructions, so as to optimize the input data using the prompt words. The optimized input data is fed into a large language model to obtain prediction results about user behavior and scenario tasks; Based on the prediction results, control commands are sent to the corresponding devices to achieve user behavior feedback and scene adjustment.

[0007] Preferably, the multimodal commands include: voice commands received by the user through a microphone, gesture commands received by the user through a camera, and text commands input through a text input interface.

[0008] Preferably, the instruction semantic tags and the scene semantic tags are integrated to establish an association model between instruction tags and scene tags, thereby forming comprehensive semantic information, including: Machine learning algorithms are used to analyze historical interaction data to identify potential relationships between instructions and scene semantics; An association model is constructed based on the potential relationships to enhance the fusion effect of instruction semantic tags and scene semantic tags.

[0009] Preferably, a local knowledge base is constructed, including: Collect documentation related to user instructions and scenarios; such documentation includes operation manuals, equipment specifications, and frequently asked questions. Based on the aforementioned documents, a knowledge graph is constructed using data mining techniques; the knowledge graph is used to describe the relationships and attributes between people, equipment, and tasks.

[0010] Preferably, the local knowledge text and the knowledge graph are retrieved using vector retrieval and graph retrieval respectively to obtain retrieval knowledge fragments related to the comprehensive semantic information, including: The local knowledge text is vectorized, and based on the vector similarity algorithm, the text knowledge fragments related to the current instruction and scenario are quickly located. Based on the node and edge relationships in the graph structure, graph retrieval is performed on the knowledge graph to obtain relevant graph knowledge fragments; The text knowledge fragment and the graph knowledge fragment are identified as the retrieval knowledge fragment.

[0011] Preferably, the retrieved knowledge fragments are formatted to obtain standardized knowledge fragments with a standardized structure for processing by a large language model, including: The retrieved knowledge fragments are converted into a unified standard format to obtain standard knowledge fragments; The standard knowledge fragments are evaluated for quality, and the standard knowledge fragments are selected based on the evaluation results.

[0012] Preferably, for the standard knowledge fragment and the user's instruction, suitable prompt words are generated to optimize the input data using the prompt words, including: Generate corresponding prompt words based on the content of the standard knowledge fragment; The structure and tone of the prompt words are adjusted and optimized based on the semantic tags of the user's instructions to enhance the expression of the user's intent.

[0013] Preferably, control commands are sent to the corresponding devices based on the prediction results to achieve user behavior feedback and scene adjustment, including: The prediction results are converted into control signals for use in adjusting specific equipment; The control signals are dynamically adjusted based on real-time user feedback to adapt to changes in user needs.

[0014] A human-computer interaction behavior prediction system based on a large language model includes: The tag extraction unit is used to parse the received multimodal instructions from the user in order to extract instruction semantic tags and scene semantic tags; The semantic determination unit is used to integrate the instruction semantic tags and the scene semantic tags, and establish an association model between the instruction tags and the scene tags to form comprehensive semantic information; The knowledge base construction unit is used to build a local knowledge base to store initial knowledge fragments related to user commands and scenarios; the local knowledge base includes local knowledge text and knowledge graph. The knowledge retrieval unit is used to retrieve the local knowledge text and the knowledge graph through vector retrieval and graph retrieval respectively, to obtain retrieval knowledge fragments related to the comprehensive semantic information; The prompt word generation unit is used to format the retrieved knowledge fragment to obtain a standardized knowledge fragment with a standardized structure for processing by a large language model, and to generate suitable prompt words for the standard knowledge fragment and the user's instructions, so as to optimize the input data using the prompt words; The prediction result generation unit is used to feed the optimized input data into the large language model to obtain prediction results about user behavior and scenario tasks; The feedback adjustment unit is used to send control commands to the corresponding devices based on the prediction results, so as to realize user behavior feedback and scene adjustment.

[0015] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention provides a method and system for predicting human-computer interaction behavior based on a large language model, comprising: parsing received multimodal user commands to extract command semantic tags and scene semantic tags; integrating the command semantic tags and the scene semantic tags to establish an association model between command tags and scene tags to form comprehensive semantic information; constructing a local knowledge base to store initial knowledge fragments related to user commands and scenes; the local knowledge base includes local knowledge text and a knowledge graph; retrieving the local knowledge text and the knowledge graph through vector retrieval and graph retrieval respectively to obtain retrieved knowledge fragments related to the comprehensive semantic information; formatting the retrieved knowledge fragments to obtain standardized knowledge fragments with a standardized structure for processing by the large language model, and generating suitable prompt words for the standardized knowledge fragments and user commands to optimize the input data using the prompt words; sending the optimized input data into the large language model to obtain prediction results about user behavior and scene tasks; and sending control commands to corresponding devices according to the prediction results to achieve user behavior feedback and scene adjustment. This invention combines the reasoning capabilities of large-scale language models with hybrid retrieval technology to achieve accurate prediction of human-computer interaction behavior, significantly improving the intelligence level and response efficiency of human-computer interaction. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of the method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the technical route provided for an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The purpose of this invention is to provide a method and system for predicting human-computer interaction behavior based on a large language model. By combining the reasoning ability of a large language model with hybrid retrieval technology, it achieves accurate prediction of human-computer interaction behavior, significantly improving the intelligence level and response efficiency of human-computer interaction.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a method for predicting human-computer interaction behavior based on a large language model, including: Step 100: Parse the received multimodal instructions from the user to extract instruction semantic tags and scene semantic tags; Step 200: Integrate instruction semantic tags and scene semantic tags, and establish a correlation model between instruction tags and scene tags to form comprehensive semantic information; Step 300: Build a local knowledge base to store initial knowledge fragments related to user commands and scenarios; the local knowledge base includes local knowledge text and knowledge graph; Step 400: Retrieve local knowledge text and knowledge graph through vector retrieval and graph retrieval respectively to obtain retrieval knowledge fragments related to comprehensive semantic information; Step 500: Format the retrieved knowledge fragments to obtain standardized knowledge fragments with a standardized structure for processing by the large language model, and generate suitable prompt words based on the standard knowledge fragments and user instructions, so as to optimize the input data using the prompt words; Step 600: Feed the optimized input data into the large language model to obtain prediction results about user behavior and scenario tasks; Step 700: Send control commands to the corresponding devices based on the prediction results to achieve user behavior feedback and scene adjustment.

[0022] The core of this invention is to use the reasoning capabilities of a large language model (LLM) to predict behavior based on multimodal perceived instruction inputs. This process involves multiple key steps, which work closely together to form a complete technical solution.

[0023] 1. Integrate semantic tags In complex multimodal perception command input environments, the fusion of command semantic labels and scene semantic labels forms the cornerstone of this research. Command semantic labels convey the user's explicit operational intent, while scene semantic labels depict the specific environmental context in which the command is executed. Taking a smart home environment as an example, when the command semantic label is "turn it on," combined with the scene semantic label "the living room is dimly lit, and people are watching TV," the user's intent can be more accurately interpreted as turning on the living room lights. By analyzing massive amounts of data and applying machine learning algorithms, this invention has successfully constructed a correlation model between commands and scene labels. This model can accurately extract comprehensive semantic information based on different combinations of commands and scenes, thus providing a rich and accurate semantic analysis foundation for behavior prediction.

[0024] 2. Enhanced retrieval of local knowledge base using hybrid RAG The local knowledge base plays a core role in the system's operational framework; however, it faces challenges such as limitations of locality, illusion problems, and data security. To overcome these difficulties, this invention adopts a hybrid RAG (Reranking Answer Generation) strategy. This mechanism, on the one hand, uses a hybrid vector and graph retrieval strategy to perform precise retrieval of local knowledge text and local knowledge graphs respectively. For knowledge text, this invention vectorizes it and, relying on vector similarity algorithms, quickly locates text knowledge fragments closely related to the current instruction and scenario. For the knowledge graph, this invention can fully utilize the unique properties of the graph structure, deeply mining rich semantic relationships along the connection methods of nodes and edges. This hybrid retrieval method allows this invention to maximize the advantages of both data structures, thereby broadening the coverage of knowledge retrieval and improving retrieval accuracy. On the other hand, the RAG mechanism further strengthens the connection between the knowledge base and the generated content, preventing the model from generating content that does not match the current knowledge base, while also ensuring data security and preventing the possibility of data leakage and malicious use.

[0025] 3. Prompt prompt word project After the acquired knowledge fragments undergo careful engineering processing using carefully crafted prompts, their application value will be further enhanced. First, the knowledge fragments should be formatted to conform to specific structures and specifications. For example, they can be graded and categorized based on multiple dimensions such as importance and relevance. Then, depending on the specific application environment and task nature, a single-learning or few-learning paradigm should be selected. In single-learning, this invention provides the model with an example, enabling it to quickly learn and understand how to make predictions based on current instructions and knowledge. In few-learning, this invention provides a small number of examples to further enhance the model's understanding and mastery of complex semantics and behavioral patterns. Similarly, carefully crafting appropriate prompts is crucial. Prompts need to be concise and intuitive, accurately conveying the intent of the instruction and the focus of knowledge. For example, in predicting a user's behavior in a smart office environment, the prompt could be set as "Based on the current office environment and user instructions, predict the user's upcoming actions, focusing on knowledge related to document processing and meeting arrangements." This design guides the LLM to focus on key information, optimize input information, and further improve the accuracy of predictions.

[0026] 4. Large model processing stage The optimized input data is then processed by LLM. Leveraging its powerful natural language processing capabilities, LLM performs in-depth analysis of the comprehensive semantic information, standardized knowledge fragments, and carefully designed prompts in the input. Internally, through a multi-layered neural network architecture and sophisticated algorithms, it uncovers hidden patterns and relationships within the data. For example, for inputs incorporating multimodal perceptual commands, scene information, and local knowledge, LLM can identify the logical connections between different insight elements, thereby generating accurate predictions about the scene, task, and human behavior. This prediction mechanism, built upon deep learning from massive amounts of text data and accurate analysis of the current input information, provides valuable decision support for various application scenarios such as smart homes, smart offices, and intelligent transportation. This not only promotes the intelligence of human-computer interaction but also significantly improves interaction efficiency.

[0027] Corresponding to the above methods, such as Figure 2 As shown, this embodiment also provides a human-computer interaction behavior prediction system based on a large language model, including: The tag extraction unit is used to parse the received multimodal instructions from the user in order to extract instruction semantic tags and scene semantic tags; The semantic determination unit is used to integrate the instruction semantic tags and the scene semantic tags, and establish an association model between the instruction tags and the scene tags to form comprehensive semantic information; The knowledge base construction unit is used to build a local knowledge base to store initial knowledge fragments related to user commands and scenarios; the local knowledge base includes local knowledge text and knowledge graph. The knowledge retrieval unit is used to retrieve the local knowledge text and the knowledge graph through vector retrieval and graph retrieval respectively, to obtain retrieval knowledge fragments related to the comprehensive semantic information; The prompt word generation unit is used to format the retrieved knowledge fragment to obtain a standardized knowledge fragment with a standardized structure for processing by a large language model, and to generate suitable prompt words for the standard knowledge fragment and the user's instructions, so as to optimize the input data using the prompt words; The prediction result generation unit is used to feed the optimized input data into the large language model to obtain prediction results about user behavior and scenario tasks; The feedback adjustment unit is used to send control commands to the corresponding devices based on the prediction results, so as to realize user behavior feedback and scene adjustment.

[0028] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0029] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for predicting human-computer interaction behavior based on a large language model, characterized in that, include: Parse the received multimodal commands from the user to extract command semantic tags and scene semantic tags; Integrate the instruction semantic tags and the scene semantic tags to establish an association model between instruction tags and scene tags, so as to form comprehensive semantic information; A local knowledge base is constructed to store initial knowledge fragments related to user commands and scenarios; the local knowledge base includes local knowledge text and knowledge graph. The local knowledge text and the knowledge graph are retrieved by vector retrieval and graph retrieval respectively to obtain retrieval knowledge fragments related to the comprehensive semantic information; The retrieved knowledge fragments are formatted to obtain standardized knowledge fragments with a standardized structure for processing by a large language model. Suitable prompt words are generated based on the standardized knowledge fragments and the user's instructions, so as to optimize the input data using the prompt words. The optimized input data is fed into a large language model to obtain prediction results about user behavior and scenario tasks; Based on the prediction results, control commands are sent to the corresponding devices to achieve user behavior feedback and scene adjustment.

2. The human-computer interaction behavior prediction method based on a large language model according to claim 1, characterized in that, The multimodal commands include: voice commands received from the user via the microphone, gesture commands received from the user via the camera, and text commands entered via the text input interface.

3. The human-computer interaction behavior prediction method based on a large language model according to claim 1, characterized in that, Integrate the instruction semantic tags and the scene semantic tags to establish a correlation model between instruction tags and scene tags, thereby forming comprehensive semantic information, including: Machine learning algorithms are used to analyze historical interaction data to identify potential relationships between instructions and scene semantics; An association model is constructed based on the potential relationships to enhance the fusion effect of instruction semantic tags and scene semantic tags.

4. The human-computer interaction behavior prediction method based on a large language model according to claim 1, characterized in that, Building a local knowledge base includes: Collect documentation related to user instructions and scenarios; such documentation includes operation manuals, equipment specifications, and frequently asked questions. Based on the aforementioned documents, a knowledge graph is constructed using data mining techniques; the knowledge graph is used to describe the relationships and attributes between people, equipment, and tasks.

5. The human-computer interaction behavior prediction method based on a large language model according to claim 1, characterized in that, The local knowledge text and the knowledge graph are retrieved using vector retrieval and graph retrieval respectively to obtain retrieval knowledge fragments related to the comprehensive semantic information, including: The local knowledge text is vectorized, and based on the vector similarity algorithm, the text knowledge fragments related to the current instruction and scenario are quickly located; Based on the node and edge relationships in the graph structure, graph retrieval is performed on the knowledge graph to obtain relevant graph knowledge fragments; The text knowledge fragment and the graph knowledge fragment are identified as the retrieval knowledge fragment.

6. The human-computer interaction behavior prediction method based on a large language model according to claim 1, characterized in that, The retrieved knowledge fragments are formatted to obtain standardized knowledge fragments with a standardized structure for processing by large language models, including: The retrieved knowledge fragments are converted into a unified standard format to obtain standard knowledge fragments; The standard knowledge fragments are subjected to quality assessment, and the standard knowledge fragments are selected based on the quality assessment results.

7. The human-computer interaction behavior prediction method based on a large language model according to claim 1, characterized in that, Based on the aforementioned standard knowledge fragments and user instructions, generate suitable prompt words to optimize the input data, including: Generate corresponding prompt words based on the content of the standard knowledge fragment; The structure and tone of the prompt words are adjusted and optimized based on the semantic tags of the user's instructions to enhance the expression of the user's intent.

8. The human-computer interaction behavior prediction method based on a large language model according to claim 1, characterized in that, Based on the prediction results, control commands are sent to the corresponding devices to achieve user behavior feedback and scene adjustment, including: The prediction results are converted into control signals for use in adjusting specific equipment; The control signals are dynamically adjusted based on real-time user feedback to adapt to changes in user needs.

9. A human-computer interaction behavior prediction system based on a large language model, characterized in that, include: The tag extraction unit is used to parse the received multimodal instructions from the user in order to extract instruction semantic tags and scene semantic tags; The semantic determination unit is used to integrate the instruction semantic tags and the scene semantic tags, and establish an association model between the instruction tags and the scene tags to form comprehensive semantic information; The knowledge base construction unit is used to build a local knowledge base to store initial knowledge fragments related to user commands and scenarios; the local knowledge base includes local knowledge text and knowledge graph. The knowledge retrieval unit is used to retrieve the local knowledge text and the knowledge graph through vector retrieval and graph retrieval respectively, to obtain retrieval knowledge fragments related to the comprehensive semantic information; The prompt word generation unit is used to format the retrieved knowledge fragment to obtain a standardized knowledge fragment with a standardized structure for processing by a large language model, and to generate suitable prompt words for the standard knowledge fragment and the user's instructions, so as to optimize the input data using the prompt words; The prediction result generation unit is used to feed the optimized input data into the large language model to obtain prediction results about user behavior and scenario tasks; The feedback adjustment unit is used to send control commands to the corresponding devices based on the prediction results, so as to realize user behavior feedback and scene adjustment.

Images