Intelligent assistant system application interaction method based on personal knowledge base

Abstract

The invention discloses an intelligent assistant system application interaction method based on a personal knowledge base, and relates to the technical field of large models, and the method comprises the steps: capturing multi-dimensional data of an application scene through multimedia information, so as to construct a more comprehensive prompt word engineering template; secondly, optimizing the cue word engineering template by accessing and utilizing personalized information closely related to the user in a pre-constructed personal knowledge base, and further performing scene optimization, entity optimization and multimedia recognition optimization, so that the cue words generated by the system can be ensured to better meet the actual requirements and context of the user; according to the method, more accurate and effective input is provided for subsequent large model processing, so that the intention of the user can be more accurately understood, diversified requirements of different users can be better met, response is made according to personalized information of the user, and the accuracy and efficiency of application interaction are improved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to an intelligent assistant system application interaction method based on a personal knowledge base. Background Art

[0002] With the rapid development of big model technology, from text to text, to text to image, and then to text to video, the capabilities of big models are constantly expanding, and the application of building intelligent agent platforms by combining big model technology with RAG (retrieval-augmented generation) technology is also gradually being promoted. These intelligent agent platforms use the ultra-large-scale parameters and powerful computing power of big models, combined with the external knowledge retrieval capabilities of RAG technology, to achieve more accurate and rich content generation and response, meeting the interactive use needs of users.

[0003] When existing intelligent agent platforms receive text requirements input by users, they generally first use RAG technology to retrieve relevant information from external knowledge bases to obtain context information, and then input the text requirements and context information into the large model to generate content and output responses. Although RAG technology provides additional context information for the large model through the retrieval model, which helps to make the generated content more relevant and accurate, this process is still highly dependent on the user's active input. Users need to clearly express their needs or questions before the intelligent agent platform can retrieve and generate based on this information. This reliance on user active input limits the intelligent agent platform's ability to understand user intentions and provide proactive services. This defect limits the intelligent agent platform's ability to provide comprehensive and intelligent services, affecting the performance and intelligence of the intelligent agent platform. Summary of the invention

[0004] In response to the above problems and technical requirements, this application proposes an intelligent assistant system application interaction method based on a personal knowledge base. The technical solution of this application is as follows:

[0005] An intelligent assistant system application interaction method based on a personal knowledge base, the intelligent assistant system application interaction method comprising:

[0006] Extracting features of the collected multimedia information in the application scenario to obtain content information, and constructing a prompt word engineering template according to the content information in the application scenario;

[0007] Optimizing the prompt word engineering template according to the user personalized information in the personal knowledge base, wherein the user personalized information in the personal knowledge base includes the user's personal information, historical interaction data, and preference setting data;

[0008] Performing scene optimization, entity optimization, and multimedia recognition optimization on the prompt word engineering template that has completed prompt word optimization to obtain a prompt word engineering template with content optimization;

[0009] The big model is used to conduct deep learning and analysis on the prompt word engineering template after content optimization to obtain the interactive results and display them.

[0010] A further technical solution is to optimize the prompt word engineering template according to the user personalized information in the personal knowledge base, including:

[0011] Determine the user's preference according to the user's personal information and preference setting data, increase the priority of the prompt words that meet the user's preference in the prompt word engineering template, and generate the prompt words that meet the user's preference and add them to the prompt word engineering template;

[0012] Also, conduct behavioral pattern analysis on user historical interaction data to mine user implicit preferences, and generate prompt words that meet the user's implicit preferences and add them to the prompt word engineering template;

[0013] Furthermore, the user's personal information is feature-encoded and added to the prompt word engineering template as a new prompt word.

[0014] Its further technical solution is to analyze the behavior patterns of users' historical interaction data to mine users' implicit preferences, including:

[0015] Analyze the time-varying sequence of attention paid to different technical fields and interaction behaviors in the user's historical interaction data, and take the technical fields with the highest attention and the most frequent interaction behaviors in a predetermined time period as the mined user implicit preferences.

[0016] Its further technical solution is to analyze the behavior patterns of the user's historical interaction data to mine the user's implicit preference technology field and generate prompt words, which also includes:

[0017] The query relevance of different technical fields in the user's historical interaction data is analyzed, and cross-field compound prompt words are generated for multiple technical fields whose query relevance reaches the relevance threshold.

[0018] A further technical solution is that the collected multimedia information in the application scenario includes audio information and video streams, and multimedia recognition optimization is performed on the prompt word engineering template that has completed prompt word optimization, including:

[0019] Introducing a human feedback mechanism to adjust and optimize prompt words through human-computer interaction, and using automatic prompt engineering to automatically search and optimize prompt words to achieve speech recognition output optimization;

[0020] Image recognition perturbations are added to the prompt word engineering template to achieve adversarial robustness optimization, and the prompt words are fine-tuned using image annotation data to achieve prompt learning optimization to achieve image recognition optimization.

[0021] A further technical solution is to optimize the prompt word engineering template after the prompt word optimization, including:

[0022] Add scenario descriptions and examples to the prompt word project template to achieve contextual learning optimization, and decompose the task into multiple sub-steps and present them one by one in the prompt word to achieve chain thinking optimization.

[0023] A further technical solution is to perform entity optimization on the prompt word engineering template after prompt word optimization, including:

[0024] Entity extraction is optimized during the knowledge graph construction process, and multi-attribute optimization is performed by combining language models and prompt engineering.

[0025] Its further technical solution is to extract features from the collected multimedia information in the application scenario to obtain content information including:

[0026] Extract features from the collected audio information in the application scenario and map it to text output to obtain voice content;

[0027] Image recognition is performed on the collected video stream in the application scenario to extract scene information and character information. The character information includes face recognition results, character facial expressions and character gestures.

[0028] Its further technical solution is to extract features from the audio information in the application scenario and map it into text output to obtain speech content including:

[0029] After preprocessing the audio information, acoustic features are extracted to obtain feature vectors, and a pre-trained acoustic model is used to map the feature vectors into text and output it, and the output text is spell-corrected and format-converted to obtain speech content; wherein the acoustic feature extraction method used includes at least one of Mel-frequency cepstral coefficients and linear predictive coding, and the acoustic model used is trained based on a deep neural network or a long short-term memory network.

[0030] Its further technical solution is that the intelligent assistant system application interaction method also includes: collecting multimedia information in the application scenario at predetermined time intervals and updating the content information obtained by feature extraction into the personal knowledge base.

[0031] The beneficial technical effects of this application are:

[0032] The present application discloses an intelligent assistant system application interaction method based on a personal knowledge base, which constructs and utilizes a personal knowledge base containing user personal information, historical behavior and preferences. By accessing and utilizing personalized information closely related to the user in the personal knowledge base to optimize the prompt word engineering template, it can ensure that the prompt words generated by the system are more in line with the actual needs and context of the user, and provide more accurate and effective input for subsequent large model processing. The large model usually has powerful processing and generalization capabilities, but often lacks personalized processing for specific users. The present invention combines the personal knowledge base with the large model so that the large model can fully consider the personalized needs of the user when processing user requests. This combination not only improves the intelligence level of the system, but also enables the system to better adapt to the diverse needs of different users, so that it can more accurately understand the user's intentions and respond according to the user's personalized information. The interaction between the user and the intelligent assistant becomes smoother and more natural, improving the accuracy and efficiency of application interaction. Users do not need to repeat or restate their needs, which reduces the time required for interaction and improves the user experience. As users continue to use it, the system will accumulate more personalized data to expand the personal knowledge base, and continuously optimize its performance through machine learning algorithms. In the long run, the intelligent assistant will better adapt to the changing needs and preferences of individual users, and achieve continuous personalized evolution and service optimization.

[0033] This method uses multimedia information to capture the user's voice, facial expressions, gestures and other information in real time, and integrates it for analysis and processing to build a prompt word engineering template. This integration of multimodal information provides the system with more comprehensive and detailed input data. Compared with existing intelligent assistants with a single mode (such as voice or text only), it can more accurately capture the user's true intentions and needs. Combined with the personal knowledge base, it can improve the ability to understand user intentions, especially in complex or ambiguous instructions. It can more accurately identify user needs, reduce the possibility of misunderstanding and incorrect responses, and improve the naturalness and efficiency of interaction.

[0034] In addition, thanks to the combined effect of all the above technical effects, the present application can effectively serve users in a variety of different environments and scenarios. Whether at home, in the office or on the move, whether performing simple queries or complex task processing, the present invention can provide accurate, timely and personalized intelligent assistant services. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a method flow chart of an intelligent assistant system application interaction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0037] This application discloses an intelligent assistant system application interaction method based on a personal knowledge base, please refer to Figure 1 As shown in the flowchart, the intelligent assistant system application interaction method includes the following steps:

[0038] Step 1: Collect multimedia information in the application scenario, which includes audio information and video streams in the application scenario. This can be achieved by using audio acquisition devices and video acquisition devices respectively, or by using terminal devices with integrated audio acquisition and video acquisition functions, such as various mobile terminals such as mobile phones and tablets.

[0039] Step 2: extract features from the collected multimedia information in the application scenario to obtain content information, and construct a prompt word engineering template based on the content information in the application scenario.

[0040] The collected audio information in the application scenario is feature extracted and mapped to text output to obtain the voice content, including:

[0041] (1) First, the audio information is preprocessed. The preprocessing includes noise reduction and enhancement of the audio information through an adaptive filter, thereby removing background noise and improving the signal-to-noise ratio of the effective voice signal, implementing noise suppression and echo cancellation technology, and ensuring that the user's voice commands can still be accurately captured in a noisy environment.

[0042] (2) Then, acoustic feature extraction is performed on the preprocessed audio information to obtain a feature vector, and the acoustic feature extraction method used includes at least one of Mel Frequency Cepstral Coefficient (MFCC) and Linear Predictive Coding (LPC).

[0043] (3) A pre-trained acoustic model is further used to map the feature vector into text and output it. The acoustic model used is trained based on a deep neural network (DNN) or a long short-term memory network (LSTM).

[0044] (4) Finally, the output text is spell-corrected and formatted to obtain the speech content, ensuring the accuracy and readability of the speech content in the final text format.

[0045] Image recognition is performed on the collected video stream in the application scenario to extract scene information and character information. The character information includes face recognition results, character facial expressions and character gestures.

[0046] The content information of the application scenario generated based on the multimedia information includes voice content, scene information, face recognition results, character facial expressions and character gestures, and then prompt engineering can be used to construct a preliminary prompt engineering template based on the content information.

[0047] Step 3: Optimize the prompt words of the prompt word engineering template according to the user personalized information in the personal knowledge base.

[0048] Unlike traditional intelligent platforms, this application does not use external databases to retrieve contexts, but instead uses user personalized information in a personal knowledge base to retrieve for prompt word optimization. The user personalized information in the personal knowledge base can characterize the user's personal needs and preferences. The user personalized information includes the user's personal information, historical interaction data, and preference setting data, where: the user's personal information includes at least the user's age, gender, and occupation. The user's historical interaction data includes the prompt word engineering template constructed by the user during the historical application interaction process and the final interaction results. The user's preference setting data includes the preferred technical field and the user's preference for the form of the interaction results. By searching the personal knowledge base, contextual information closely related to the user can be obtained, thereby obtaining a prompt word engineering template that better meets the user's personal needs and preferences. The optimization methods include the following categories:

[0049] (1) The user's personal information and directly set preference data can directly reflect the user's preferences. Therefore, the user's preferences are determined based on the user's personal information and preference data, and the priority of the prompt words that meet the user's preferences in the prompt word engineering template is increased, and the prompt words that meet the user's preferences are generated and added to the prompt word engineering template.

[0050] For example, for the user's personal information. If the user's profession is a programmer, this information may imply a high degree of familiarity with specific programming-related concepts or technologies. When optimizing the prompt word engineering template, the priority and weight of prompt words related to the programming field are increased. The user's age may affect the degree of exposure to certain technology trends. Younger users may prefer prompt words related to emerging technology fields, while older users may prefer prompt words related to traditional technology fields. The same is true for the user's gender, so the field relevance of the prompt word can be adjusted based on the user's personal information.

[0051] For another example, when the user directly sets the preference data for the form of the interaction result as a concise answer, instructions related to concise expression can be added as prompt words in the prompt word engineering template, such as "concise explanation", "key point summary", etc.

[0052] For example, when the preferred technology field in the preference setting data directly set by the user is the field of artificial intelligence technology, concepts and terms related to the field of artificial intelligence technology can be added to the prompt word engineering template to guide the model to generate content that is more in line with the user's preferences.

[0053] (2) Conduct behavioral pattern analysis on user historical interaction data to mine user implicit preferences, and generate prompt words that meet the user's implicit preferences and add them to the prompt word engineering template. In addition to the user's personal information and the user preferences reflected by the directly set preference data, the user's implicit preferences can also be inferred from the user's historical interaction data, including: analyzing the attention paid to different technical fields and the sequence of interaction behaviors over time in the user's historical interaction data, and taking the technical field with the highest attention and the most frequent interaction behaviors in a predetermined time period as the mined user implicit preferences. For example, if the user's historical interaction data shows that the user often queries content about machine learning algorithms, then when optimizing the prompt word engineering template, increase the priority of prompt words related to machine learning algorithms, and observe the time series information in the user's historical interaction data, such as the changes in the user's attention to certain technical topics in a specific time period. For another example, if the user often clicks on the interaction results containing code examples, although the preference setting data is not explicitly set, it can be inferred that the user has a preference for specific technical implementation examples. Then, prompt words such as "contains code examples" can be added to the prompt word engineering template to optimize the model output.

[0054] In addition to mining the user's implicit preferences for technical fields and forms of interaction results, in another embodiment, the query relevance for different technical fields in the user's historical interaction data is analyzed, and cross-domain composite prompt words are generated for multiple technical fields whose query relevance reaches the relevance threshold. That is, by analyzing the association between different queries in the user's historical interaction data, the user's potential knowledge structure is discovered. For example, if the user often queries database management and data mining related content, it means that these two fields have a certain association in the user's knowledge system. When optimizing the prompt word engineering template, some composite prompt words involving both fields are created to better meet the user's potential needs.

[0055] (3) After the user's personal information is feature-encoded, it is added to the prompt word engineering template as a new prompt word, thereby converting the personal information into a feature vector that can be used for model calculation. For example, gender is encoded as 0 (male) or 1 (female), and combined with other personal information encodings to form a multi-dimensional feature vector. This feature vector is added to the prompt word engineering template as an input part of the model to influence the optimization direction of the prompt word template.

[0056] Step 4: Perform scene optimization, entity optimization and multimedia recognition optimization on the prompt word engineering template that has completed prompt word optimization to obtain a content-optimized prompt word engineering template.

[0057] In addition to the above-mentioned prompt word optimization based on the personal knowledge base, this application will also perform a series of content optimization on the prompt word engineering template, including automatic, semi-automatic or manual scene optimization, entity optimization, multimedia recognition optimization, etc. These optimization steps are based on advanced image processing technology and natural language processing technology, aiming to improve the quality and accuracy of output content. Based on multimedia information including audio information and video streams, multimedia recognition optimization includes image recognition optimization and speech recognition output optimization. The specific optimization objects and execution optimization methods include:

[0058] (1) Speech recognition output optimization

[0059] Speech recognition output optimization focuses on prompt words and their impact on speech signal processing, with the goal of improving the accuracy and robustness of speech recognition. The implementation methods include:

[0060] Interactive Optimization: Introducing a human feedback mechanism to continuously adjust and optimize the prompt words through human-computer interaction to achieve the best speech recognition effect. For example, the iPrOp method uses a human-in-the-loop approach to adjust the prompt words in real time to improve the recognition accuracy of the model.

[0061] Automatic Prompt Engineering (APE): Automatic prompt engineering is used to automatically search and optimize prompt words to optimize speech recognition output to adapt to specific speech recognition tasks. Through a large number of experiments and evaluations, the optimal prompt word configuration is found to improve the performance of the model on specific tasks.

[0062] (2) Image recognition optimization

[0063] Adversarial Robustness Optimization: Add image recognition perturbations to the prompt word engineering template and introduce adversarial training to enhance the model's defense against adversarial attacks, so that the model can still maintain high accuracy and robustness when facing adversarial samples, thereby achieving adversarial robustness optimization.

[0064] Prompt Learning: Fine-tuning the prompt words using image annotation data significantly improves the performance of the model on specific tasks. For example, in image classification tasks, by optimizing the prompt words, the accuracy and robustness of the model are significantly improved.

[0065] (3) Scene Optimization

[0066] Scene optimization mainly targets prompt words in specific application scenarios to ensure that they can efficiently serve specific task requirements, such as 3D object generation, virtual reality (VR) scene generation, and English oral teaching. Implementation methods include:

[0067] In-Context Learning: Adding scenario descriptions and examples to the prompt engineering template enables the model to respond more reasonably in specific scenarios. For example, in the scenario of generating 3D objects, detailed scenario descriptions and step-by-step instructions can enable the model to generate more accurate and complex 3D models.

[0068] Chain-of-Thought (CoT): The task is broken down into multiple sub-steps and presented one by one in the prompt words to guide the model to reason step by step. This technique is particularly suitable for tasks that require multi-step reasoning to complete, such as the construction of virtual reality scenes and oral English teaching.

[0069] (4) Entity Optimization

[0070] Entity optimization focuses on the representation and extraction of specific entities, especially in areas such as knowledge graph construction and multi-attribute molecular optimization. Implementation methods include:

[0071] Entity Extraction: Use prompt words to guide the model to extract specific entities in the text, and improve the accuracy and recall of entity recognition by optimizing the structure and content of prompt words. For example, in the process of building a knowledge graph, through carefully designed prompt words, the required entities can be efficiently extracted from a large amount of text.

[0072] Multi-property Optimization: Combining language models and prompt engineering, multi-objective optimization of molecular structures is performed. By designing specific prompt words, the model can be guided to optimize other properties while maintaining certain chemical properties, thereby obtaining a more ideal molecular structure.

[0073] Step 5: Use the big model to conduct deep learning and analysis on the content-optimized prompt word engineering template to obtain the interactive results and display them. The interactive results can be displayed to users through terminal devices. These contents include scenes, entities, and voices, etc., aiming to provide users with comprehensive intelligent assistant services. In this process, the system can manually or automatically specify the output version to the big model to meet the needs of different scenarios. For example:

[0074] Automatic intelligent customer service scenario. When a customer asks for detailed information about a product, such as "What is the pixel of the camera on this phone?", the big model analyzes the entity content such as the product database input into the system and automatically and accurately answers "The pixel of the rear camera on this phone is 50 million." When a customer reports an order problem, such as "Why hasn't my order been shipped yet?" the big model can query the entity content related to the order system and give the answer "Due to the recent large order volume, your order will be shipped within 24 hours. Please wait patiently."

[0075] In manual office scenarios. In daily office meetings, the system automatically summarizes and analyzes the meeting content, meeting background, and emotional state of on-site personnel. It helps users accurately analyze key content and output multiple versions of speech copy. At the same time, users can manually select the output content format, and the large model automatically fills the content into the format according to the content format to complete the final content output.

[0076] In addition, in order to maintain the continuous operation and optimization of the system, the system will set a scheduled task to collect multimedia information in the application scenario at predetermined time intervals and update the content information obtained by feature extraction to the personal knowledge base, so as to regularly retrieve and update relevant information to optimize the output scenes, entities, voice and other content. In this way, the system can continuously learn and improve to adapt to the ever-changing user needs and environmental changes. In addition, based on the construction of a personal knowledge base, this method can also be compatible with scenarios in which users ask questions to personal intelligent assistants in video scenarios and expect answers.

[0077] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. An intelligent assistant system application interaction method based on a personal knowledge base, characterized in that: The intelligent assistant system application interaction method includes: Extracting features of the collected multimedia information in the application scenario to obtain content information, and constructing a prompt word engineering template according to the content information in the application scenario; Optimizing the prompt word engineering template according to the user personalized information in the personal knowledge base, wherein the user personalized information in the personal knowledge base includes the user's personal information, historical interaction data, and preference setting data; Performing scene optimization, entity optimization, and multimedia recognition optimization on the prompt word engineering template that has completed prompt word optimization to obtain a prompt word engineering template with content optimization; The big model is used to conduct deep learning and analysis on the prompt word engineering template after content optimization to obtain the interactive results and display them.

2. The intelligent assistant system application interaction method according to claim 1, characterized in that: Optimizing the prompt word project template according to the user personalized information in the personal knowledge base includes: Determine the user's preference according to the user's personal information and preference setting data, increase the priority of the prompt words that meet the user's preference in the prompt word engineering template, and generate the prompt words that meet the user's preference and add them to the prompt word engineering template; Also, conduct behavioral pattern analysis on user historical interaction data to mine user implicit preferences, and generate prompt words that meet the user's implicit preferences and add them to the prompt word engineering template; Furthermore, the user's personal information is feature-encoded and added to the prompt word engineering template as a new prompt word.

3. The intelligent assistant system application interaction method according to claim 2, characterized in that: Behavioral pattern analysis of user historical interaction data to mine user implicit preferences includes: Analyze the time-varying sequence of attention paid to different technical fields and interaction behaviors in the user's historical interaction data, and take the technical fields with the highest attention and the most frequent interaction behaviors in a predetermined time period as the mined user implicit preferences.

4. The intelligent assistant system application interaction method according to claim 2, characterized in that: Analyze the behavior patterns of historical user interaction data to mine users' implicit preference technology areas and generate prompt words, including: The query relevance of different technical fields in the user's historical interaction data is analyzed, and cross-field compound prompt words are generated for multiple technical fields whose query relevance reaches the relevance threshold.

5. The intelligent assistant system application interaction method according to claim 1, characterized in that: The collected multimedia information in the application scenario includes audio information and video streams. The multimedia recognition optimization of the prompt word engineering template after prompt word optimization includes: Introducing a human feedback mechanism to adjust and optimize prompt words through human-computer interaction, and using automatic prompt engineering to automatically search and optimize prompt words to achieve speech recognition output optimization; Image recognition perturbations are added to the prompt word engineering template to achieve adversarial robustness optimization, and the prompt words are fine-tuned using image annotation data to achieve prompt learning optimization to achieve image recognition optimization.

6. The intelligent assistant system application interaction method according to claim 1, characterized in that: The scenario optimization of the prompt word engineering template after prompt word optimization includes: Add scenario descriptions and examples to the prompt word project template to achieve contextual learning optimization, and decompose the task into multiple sub-steps and present them one by one in the prompt word to achieve chain thinking optimization.

7. The intelligent assistant system application interaction method according to claim 1, characterized in that: The entity optimization of the prompt word engineering template after prompt word optimization includes: Entity extraction is optimized during the knowledge graph construction process, and multi-attribute optimization is performed by combining language models and prompt engineering.

8. The intelligent assistant system application interaction method according to claim 5, characterized in that: Feature extraction of the collected multimedia information in the application scenario to obtain content information includes: Extract features from the collected audio information in the application scenario and map it to text output to obtain voice content; Image recognition is performed on the collected video stream in the application scenario to extract scene information and character information. The character information includes face recognition results, character facial expressions and character gestures.

9. The intelligent assistant system application interaction method according to claim 8, characterized in that: The audio information in the application scenario is feature extracted and mapped to text output to obtain speech content including: After preprocessing the audio information, acoustic features are extracted to obtain feature vectors, and a pre-trained acoustic model is used to map the feature vectors into text and output it, and the output text is spell-corrected and format-converted to obtain speech content; wherein the acoustic feature extraction method used includes at least one of Mel-frequency cepstral coefficients and linear predictive coding, and the acoustic model used is trained based on a deep neural network or a long short-term memory network.

10. The intelligent assistant system application interaction method according to claim 1, characterized in that: The intelligent assistant system application interaction method also includes: collecting multimedia information in the application scenario at predetermined time intervals and updating the content information obtained by feature extraction into the personal knowledge base.

Images