Large language model interaction method based on expression information

Abstract

The invention discloses an intelligent robot based on a large language model, and relates to the field of large language models. The intelligent robot comprises the following steps: collecting a diversified English corpus, performing text clearness on the English corpus to remove irrelevant contents, making a data set, obtaining expression information by combining a camera module, obtaining voice information by an audio adapter plate, and forming a multi-modal training model; and the data set is imported into the training model for training and testing, so that the generalization ability of the model and the efficient and smooth voice interaction ability in various scenes are improved. And a main control and a coding motor are combined to provide motion ability, so that the robot adapts to dynamic changes in various actual scenes, an accurate and smooth human body tracking function is provided, and the interactive experience of the robot is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of large language models, and more particularly to the secondary development of large language models in the field of computer vision technology. Background Technology

[0002] Natural Language Processing (NLP) technology encompasses all levels of language, from lexical and syntactic to semantic, with the goal of enabling computers to understand, generate, and process human language. Traditional NLP methods rely on statistics, rules, and manually annotated corpora, which, due to their limitations, struggle to handle complex and diverse language expressions.

[0003] Sentiment analysis is a crucial application of natural language processing (NLP) technology. It determines the emotional tone of text by analyzing elements such as vocabulary, sentence structure, and context. Traditional sentiment analysis methods rely on sentiment dictionaries and simple machine learning algorithms, while modern sentiment analysis leverages deep learning and large language models to more accurately capture complex emotional information. With the development of multimodal technologies, sentiment recognition has gradually expanded from single-text analysis to multimodal information processing including speech and images. By comprehensively analyzing user facial expressions, speech tone, and text content, more comprehensive emotional information can be obtained. Summary of the Invention

[0004] This invention proposes a novel interactive method for large language models to address the issue that current large language models may generate responses that are not context-appropriate based solely on text input. This improves the relevance and adaptability of the feedback from large language models.

[0005] To achieve the above objectives, this invention provides a large language model interaction method based on facial expression information, comprising:

[0006] Image data containing rich facial expression information covering a variety of emotions is acquired. Face detection algorithms are used to locate facial regions in the images, which are then cropped and aligned to ensure consistency in the position, size, and angle of the faces within the images. After normalization processing, feature vectors representing facial expression information are extracted from key facial points such as the corners of the eyes and mouth.

[0007] Collect a diverse corpus of English language data, including dialogues, speeches, and articles, ensuring the data covers different contexts, accents, and language styles as a language dataset. Utilize natural language processing techniques to obtain the text data.

[0008] The text data is segmented to identify word boundaries and each word is labeled with a part-of-speech tag based on context. Common but low-information-content conjunctions and stop words are removed to increase the proportion of sentiment information in the dataset. Based on existing sentiment dictionaries and sentiment classification models, words and sentences in the text are sentiment-annotated to obtain sentiment categories and intensities. Timestamps are introduced into the text data, and corresponding facial expression information is paired into a format of facial expression information + linguistic information to complete feature fusion.

[0009] The text data is parsed by converting it into vectors using a pre-trained word vector model. This vectors are then combined with facial expression vectors, which serve as pre-inputs for the speech information.

[0010] Visual and linguistic features are mapped to the same vector space and incorporated into the database through a multimodal embedding layer.

[0011] Furthermore, a model suitable for the generation task is selected based on the Transformer architecture. To address the need for multimodal information fusion, a model architecture combining visual and text input is chosen, employing facial expression information as pre-input before speech information to combine the two types of information.

[0012] Furthermore, the multimodal resources are correlated and logically organized with relevant text fragments, and an emotion recognition and management module is constructed at the emotion interaction layer to provide an emotional response to the emotions contained in facial expressions.

[0013] Furthermore, within the collaborative enhancement framework, RAG and LLM technologies are integrated. First, the user's original input text is broken down into multiple parts, and queries are performed on each part within a specific domain of the knowledge base. Then, leveraging the generative capabilities of LLM, the retrieved knowledge is organically combined with the partially generated LLM responses to ultimately form a complete answer.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention significantly enhances the interactive capabilities of the large language model by integrating facial expression and voice input. By combining facial expression recognition and voice analysis, the large language model can more accurately understand the user's emotional state and context, thus providing more natural and situation-appropriate responses. This multimodal information fusion effectively solves the problem of inaccurate contextual understanding in traditional voice interaction systems, making the large language model more intelligent and human-like while improving user comfort and interaction coherence.

[0015] 2. By collecting image and text data from various contexts, accents, and emotional expressions, and removing words with low information content, this invention enables the large language model of this invention to have excellent emotional understanding and expression capabilities in diverse situations, and to flexibly adapt to different language styles and emotional scenarios. Attached Figure Description

[0016] Figure 1 Flowchart of this large language model

[0017] Figure 2 Flowchart for processing facial expression information

[0018] Figure 3 Flowchart of voice information processing Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the specific embodiments of this invention will be further described in detail below with reference to the accompanying drawings and examples. This invention provides a large language model interaction based on facial expression information. By integrating rich facial expression information and linguistic emotional information, deep emotion recognition is used to assist the large language model in generating responses that are more relevant to the current context.

[0020] This embodiment ensures high recognition accuracy in diverse emotional scenarios by acquiring rich facial expression data covering a variety of emotional expressions. Facial image data containing different user emotions is collected using camera equipment. Image data sources include daily communication and social media images to ensure the images cover a wide range of emotional expressions. Then, a face detection algorithm is used to locate the face region in the image. After the algorithm identifies the facial contour, the region is cropped and aligned to ensure that the position, size, and angle of the face region in the image are consistent. The alignment operation uses a standardized facial coordinate system, allowing subsequent feature extraction to be based on a unified facial reference point. In the aligned face image, a feature detection model identifies key facial points such as the corners of the eyes and mouth. The positional information, shape features, and surrounding texture information of the key points are extracted and converted into vector representations. These feature vectors effectively represent the expression information in the image, providing accurate input for the sentiment analysis model.

[0021] A diverse English corpus was collected, including data from everyday conversations, public speeches, and written articles, ensuring that the selected data covered various contexts, accents, and language styles. The collected text data underwent word segmentation to identify word boundaries; and part-of-speech tags were added to each word using natural language processing tools. This step allowed for fine-grained extraction of sentiment information based on specific contexts. Connectives and stop words, which have low information content, were removed to increase the proportion of sentiment information in the dataset. Based on a sentiment dictionary and sentiment classification model, important words and sentences in the text were sentiment-annotated to obtain sentiment category and intensity information. Timestamp information was introduced into the annotated text data and paired with facial expression information for the corresponding time period, so that each record was saved in a format of facial expression information + linguistic information. The completed multimodal information was then fed into a feature fusion module to support subsequent cross-modal sentiment analysis.

[0022] The segmented text data is input into a pre-trained word vector model and converted into a vector representation, serving as the basis for subsequent fusion. Combining facial expression vectors, text and visual information are embedded into a multimodal embedding layer, allowing both types of information to be processed in the same vector space. By using facial expression information as pre-input for speech information, a unified multimodal database is formed, further improving the accuracy of emotion recognition.

[0023] Based on the Transformer architecture, a suitable model for the generation task is selected, and a network structure suitable for multimodal information fusion is designed. Firstly, in the input layer of the emotion generation model, facial expression information is used as pre-input before speech information and combined with language data to enhance the weight of facial expressions in the generation process. Visual and text inputs are processed in parallel, effectively combining the two types of emotional information. Then, an emotion recognition and management module is constructed in the emotion interaction layer to identify and manage the emotions contained in facial expression information. Emotional responses are based on factors such as emotion intensity and emotion category, providing comprehensive emotion analysis results.

[0024] In the emotion interaction and generation stages, a collaborative enhancement framework integrating RAG and LLM technologies is introduced to further improve the model's professionalism and generation performance. Under this framework, the user's original input text is broken down into multiple parts, each of which is queried in different domains of the professional knowledge base based on its contextual meaning, yielding more professional answers. Leveraging the generative capabilities of the Large Language Model (LLM), the retrieved professional knowledge is organically combined with the model-generated partial answers, and further integrated with the emotional information output from the emotion recognition and management module, forming a complete feedback that aligns with the current emotional context.

Claims

1. An interaction method based on a large language model of facial expression information, comprising the following steps: Step 1: Collect facial image data covering a variety of emotional expressions, use face detection algorithms to locate the face regions in the images, and crop and align the face regions to ensure that the position, size and angle of the faces in the images are consistent; Step 2: Normalize the aligned face image and extract facial features at key facial points such as the corners of the eyes and mouth to generate feature vectors of facial expression information. Step 3: Collect a diverse English corpus, including text data such as dialogues, speeches, and articles, to cover different contexts, accents, and language styles; Step 4: Perform word segmentation on the text data and attach part-of-speech tags to each word according to the context; remove conjunctions and stop words with low information content, and perform sentiment annotation on words and sentences to obtain sentiment categories and intensities; Step 5: Introduce timestamps into the text data, pair facial expression information with language information, and map facial expression feature vectors and language feature vectors to the same vector space through a multimodal embedding layer to complete the fusion of visual and language information; Step 6: In the multimodal model, the model based on the Transformer architecture uses facial expression information as pre-input of speech information, and combines facial expression and speech information to generate comprehensive feedback.

2. The interaction method based on a large language model with facial expression information according to claim 1, characterized in that, In the facial expression feature extraction step, the extracted facial expression feature vectors are incorporated into the sentiment analysis model to provide input for emotion recognition of facial expressions.

3. The interaction method based on a large language model with facial expression information according to claim 1, characterized in that, In the feature fusion step, an emotion recognition and management module is constructed based on the emotion interaction layer to provide an emotional response to facial expression information.

4. The interaction method based on a large language model with facial expression information according to claim 1, characterized in that, In the emotion recognition and generation step, RAG and LLM technologies are integrated through a collaborative enhancement framework. The system queries specific domains of the professional knowledge base for each part of the user input text and combines the responses generated by the large language model to form a complete emotional feedback.

5. The interaction method based on a large language model with facial expression information according to claim 4, characterized in that, The collaborative enhancement framework breaks down the user-input text into multiple parts, queries the knowledge base for each part, and utilizes the generative capabilities of a large language model to integrate the retrieved professional knowledge with the generated partial answers, thereby achieving a combined output of emotion and professional knowledge.

Images