Multi-modal large model expression detection and analysis method for wearing visual assistance and application of multi-modal large model expression detection and analysis method

Abstract

The invention discloses a multi-modal large model expression detection analysis method for wearing visual assistance and application thereof. The method comprises the steps of detecting an image to be detected by applying a face detection algorithm to obtain face frame information; the face frame information is input into an expression recognition model based on a multi-modal large model to obtain expression information, the expression recognition model comprises a ViT feature extraction module, an adapter operator module, a large language model module and a classifier module, the input of the feature extraction module is the face frame information, and the output of the feature extraction module is used as the input of the operator module and the input of the large language model module; the output of the operator module is used as the input of the large language model module, the output of the large language model module is used as the input of the classifier module, and the output of the classifier module is expression information; and performing overall planning and analysis on the expression recognition result in the current time period, and generating an analysis report. The method is high in expression recognition precision, high in feedback speed, capable of achieving real-time interaction and good in application prospect.

Description

Technical Field

[0001] The present invention relates to the fields of human-computer interaction, computer vision and large model technology, and relates to a multimodal large model expression detection and analysis method for wearable visual aids and applications thereof, and in particular to a method for realizing multimodal expression detection and analysis by specifically using large model technology and deep learning methods and applications thereof. Background Art

[0002] Facial expressions are one of the important ways for humans to express their emotions. Facial expression recognition technology can better understand human emotional states and enhance human-computer interactions. By analyzing facial expressions, it is possible to assess an individual's mental state, which is helpful for psychological counseling and treatment. This is of great significance in mental health management and diagnosis of emotional disorders. At the same time, facial expression recognition also has broad application value in medical care, education and other fields.

[0003] Most of the existing electronic glasses wearable devices focus on object recognition and route navigation. Few devices provide users with functions for face and expression detection to effectively help users quickly understand the communication objects. The reason is that, on the one hand, the expression recognition technology in these visual aids mostly adopts CNN-based static recognition algorithms, which have large jitters and low recognition accuracy in actual application scenarios, especially in multi-person interaction scenarios; on the other hand, the device outputs a single classification label, which cannot perform reasonable and coordinated analysis of multi-person recognition information in the scene in real time and intelligently in complex multi-user-computer interactions.

[0004] These problems make expression recognition technology less practical and less generalizable in practical applications, especially in human-computer interaction, resulting in many restrictions on its application.

[0005] Therefore, it is of great practical significance to develop an expression recognition method for wearable visual assistance devices that has high recognition accuracy and can be used for real-time interaction. Summary of the invention

[0006] Due to the above-mentioned defects in the prior art, the present invention provides an expression recognition method for a wearable visual assistive device with high recognition accuracy and capable of being used for real-time interaction, specifically a multimodal large-model expression detection and analysis method for a wearable visual assistive device and its application. Specifically, by designing a reasonable large-model adapter, the accuracy of the expression recognition result is improved, while the general capabilities of the large model are fully utilized. The algorithm recognition results can be used for wearable visual assistive devices to enhance multi-person interaction, and analysis reports of multiple interactive objects are generated in real time, assisting users to better judge and evaluate effective interactions in multi-person scenarios, thereby overcoming the defects of existing expression recognition methods with low recognition accuracy and inability to meet the needs of real-time interaction.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A multi-modal large-model expression detection and analysis method for wearable visual assistance comprises the following steps:

[0009] (1) Apply a lightweight face detection algorithm to detect the image to be detected and obtain all face frame information in the image to be detected;

[0010] (2) inputting the face frame information obtained in step (1) into an expression recognition model based on a multimodal large model to obtain expression information of all face frame information (i.e., the categories corresponding to the expressions and their components), wherein the expression recognition model based on the multimodal large model comprises a vision transformer (ViT) feature extraction module, an adapter operator module, a large language model module, and a classifier module, wherein the input of the vision transformer feature extraction module is the face frame information, and the output is preliminary feature extraction information, and the outputs thereof are respectively used as inputs of the adapter operator module and the large language model module, and the output of the adapter operator module is used as input of the large language model module, and the output of the large language model module is comprehensive information with temporal modeling information and integrated with local detail features and global features, which is used as input of the classifier module, and the output of the classifier module is expression information;

[0011] (3) The expression recognition results of the current time period are coordinated and analyzed to generate an analysis report. The analysis refers to analyzing the changes in the expression information of a target in the current time period. If the change in the expression information exceeds a set threshold, it is determined that the target has a negative emotion at that moment, otherwise it is a positive emotion.

[0012] The multimodal large model expression detection and analysis method for wearable visual assistance of the present invention constructs an expression recognition model based on the multimodal large model. First, a vision transformer is used to build an image feature extraction module to preliminarily obtain class tokens covering global information and all image features. Then, a large language model is combined to capture global and local detail feature information by designing a specific fine-tuning operator adapter, and time series information is modeled. Expressions are classified by a classifier to achieve accurate and rapid expression recognition, and the expression of an interactive object can be recognized in real time based on a video stream. At the same time, an expression information change threshold is introduced, and the expression information change is evaluated by the expression information change threshold, so that the evaluation of the expression change of the interactive object is realized, and more accurate recognition of the expression of the interactive object is realized, and the application prospect is good.

[0013] As the preferred technical solution:

[0014] In the above-mentioned multimodal large-model expression detection and analysis method for wearable visual assistance, the lightweight face detection algorithm is the Ultra-Light-Fast-Generic-Face-Detector algorithm. The lightweight face detection model is a lightweight face detection model designed for edge computing devices. It uses widerface training at different input resolutions and can be applied to different application scenarios. It can quickly and accurately obtain image data at different input resolutions and select face images in different application scenarios.

[0015] In the multimodal large-model expression detection and analysis method for wearable visual assistance as described above, step (1) specifically comprises: locating the facial image of the image to be detected and then selecting all the facial images within the detection range.

[0016] A multimodal large-model expression detection and analysis method for wearable visual assistance as described above, wherein the vision transformer feature extraction module is established based on the CLIP large model;

[0017] The adapter operator module includes a learnable query token and a cross-attention submodule. The query token input is the preliminary feature extraction information, and its output is used as the input of the cross-attention submodule. The output of the cross-attention submodule is the local detail feature with time series modeling information.

[0018] The large language model is the llama-7B large language model.

[0019] Specifically, the present invention builds a basic multimodal large model backbone network based on the vision transformer and the llama-7B large language model containing 7 billion parameters. The overall network is mainly based on self-attention and multihead self-attention (MHSA) mechanisms:

[0020]

[0021] Where Q = W Q X is the query, K = W K X is the index, V = W V X as content;

[0022] The multi-head attention mechanism is:

[0023]

[0024] MHSA(X)=Concat(head1,…,head j ,…,head h )W O

[0025] The self-attention mechanism of module stacking establishes global dependencies and expands the image receptive field. Compared with convolutional neural networks, it can obtain more contextual information.

[0026] The network part based on vision transformer is mainly built from the CLIP large model. The text part uses the large language model to generate detailed descriptions of each expression category according to different expression categories. For example, the description of the happy category includes: "accompanied by upward or wide-open mouth corners, raised cheeks, curved or wrinkled eyes and arched eyebrows";

[0027] The feature extraction module built based on the vision transformer needs to first set the network to video frame input, freeze the text encoder, and use the stacked self-attention mechanism module to extract the preliminary feature extraction information of the first stage;

[0028] Combined with the large language model, a specific fine-tuning operator adapter is designed to capture global and local detail feature information and model time series information, specifically:

[0029] 1) Different from algorithms based on static images, video-based work is more suitable for practical application scenarios because it focuses on the change information at different moments. Therefore, the time series branch is modeled to model temporal facial features;

[0030] 2) Combine the preliminary feature extraction information from the first stage and the time series branch, iteratively use cross-attention, and obtain local and global interaction information containing time information through query operators;

[0031] It branches in the time dimension:

[0032] After the preliminary feature extraction information of the image encoder, N learnable query tokens are set to query the image details through cross-attention.

[0033]

[0034] MHCS time =Concat(head1,…,head h )W O

[0035] Here X is the image encoder output and Y is the learnable query token;

[0036] By obtaining different input sequences, different tokens can be associated to establish local relationships;

[0037] In the images branch:

[0038]

[0039] MHCA image =Concat(head1,…,head j ,…,head h )W O

[0040] Here X is the image token, and Y is the output of the image encoder as the query token;

[0041] Integrate the two branches and fuse all local and global information:

[0042] MHCA out =MHCA time +MHCA image

[0043] 3) After classification by the classifier, output labels (i.e. expression information):

[0044]

[0045] In the above-mentioned multi-modal large-model expression detection and analysis method for wearable visual assistance, the image to be detected is obtained by performing image preprocessing on image data acquired by a camera in real time and intercepting video frames within a time period;

[0046] The expression recognition model based on the multimodal large model is trained using a dynamic expression recognition training data set. A set of data in the dynamic expression recognition training data set includes image data and expression information corresponding to the image data. The image data is obtained by image preprocessing and intercepting video frames. The training phase of the expression recognition model based on the multimodal large model also requires image preprocessing. The data sets used for training are three large-scale dynamic expression recognition data sets DFEW, FERV39k and MAFW publicly available in the field of expression recognition. Most of the data sets used are taken from a large number of film and television works. Different clips contain different numbers of video frames. Here, the acquired image sequence is mainly segmented, 16 frames are selected for each clip, and then a series of image enhancement operations are performed.

[0047] In the above-mentioned multimodal large-model expression detection and analysis method for wearable visual assistance, step (3) stores the relevant data in a database after generating an analysis report and makes the data into a visualization interface. The visualization interface can be displayed on a wearable visual assistance device, and the analysis results in the database are read by the wearable visual assistance device, and the images collected by the camera are unified in the visualization interface to form a real-time interface including a real-time video stream input interface, expression changes of the interactive object at different times, the emotional state of the interactive subject, and an analysis report within the time period in which the overall recognition results are generated. The visualization interface can include real-time image visualization and data visualization. The real-time image visualization is responsible for rendering the display of the on-site image transmission, and the data visualization mainly includes emotional component data, emotional change curves, overall emotional real-time classification bar charts, and specific analysis reports of the interactive objects at the time.

[0048] The multimodal large-model expression detection and analysis method for wearable visual assistance as described above is run on a cloud server.

[0049] The overall expression recognition architecture is installed on the cloud server. The multimodal large model is fine-tuned using the designed and trained adapter operator. The acquired facial image features are initially extracted in the first stage and further refined in the second stage to obtain comprehensive information with temporal modeling information. The information is then sent to the classifier for classification. The soft and hard labels of the classification results are calculated. While outputting the current expression category, the components of the current expression are highlighted to facilitate the evaluation of the expression.

[0050] Considering that the expression of the interactive object will change when responding to a question or scene, and this change can often reflect the psychological reaction of the subject, in order to highlight this change, the result of the expression recognition network is differentially calculated based on the emotional component, and the formula is:

[0051] score=S i -S i-1

[0052] Among them, S j ∈R 1×7 , S i represents the current moment, S i-1 Indicates the previous moment;

[0053] According to the difference result, the expression change process is reflected, and the seven expression categories are divided into "neutral" expression, and the expression threshold is set, that is, the threshold setting mentioned above, and the expression change process is combined to divide into two major categories: positive and negative (for example, anger, disgust, sadness, fear, and fluctuations in the expression change process exceeding the threshold are classified as negative emotions, while happiness, calmness, surprise, and fluctuations in the expression change process less than the threshold are classified as positive emotions);

[0054] By combining the recognition results, the facial expression data records at different times, and the facial expression change process, a specific analysis report is generated through the large model.

[0055] The present invention also provides a computer device, the computer device comprising:

[0056] at least one processor; and,

[0057] a memory communicatively connected to the at least one processor; wherein,

[0058] The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the multimodal large model expression detection and analysis method for wearable visual assistance as described above is implemented.

[0059] In addition, the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the multimodal large model expression detection and analysis method for wearable visual assistance as described above is implemented.

[0060] The above technical solution is only a feasible technical solution of the present invention, and the protection scope of the present invention is not limited thereto. Those skilled in the art can reasonably adjust the specific design according to actual needs.

[0061] The above invention has the following advantages or beneficial effects:

[0062] (1) The multimodal large model expression detection and analysis method for wearable visual assistance of the present invention constructs an expression recognition model based on a multimodal large model. First, a vision transformer is used to build an image feature extraction module to preliminarily obtain class tokens covering global information and all image features. Then, a specific fine-tuning operator adapter is designed in combination with a large language model to capture global and local detail feature information, and time series information is modeled. Expressions are classified by a classifier to achieve accurate and rapid expression recognition, and the expression of the interactive object can be recognized in real time based on the video stream.

[0063] (2) The multimodal large-model expression detection and analysis method for wearable visual assistance of the present invention introduces an expression information change threshold, which is used to evaluate the expression information change, thereby realizing the evaluation of the expression change of the interactive object and realizing more accurate recognition of the expression of the interactive object;

[0064] (3) The multimodal large-model expression detection and analysis method for wearable visual assistance of the present invention can display the analysis report and related detection data in the form of a visual interface, which can greatly provide convenience for users and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention and its features, appearance and advantages will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings. The same reference numerals indicate the same parts in all the drawings. The drawings are not drawn to scale, and emphasis is placed on illustrating the subject matter of the present invention.

[0066] Figure 1 It is a flow chart of the multi-modal large-model expression detection and analysis method for wearable visual aid of the present invention;

[0067] Figure 2 This is a network structure diagram of the expression recognition model based on the multimodal large model of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0069] Example 1

[0070] A multimodal large model expression detection and analysis method for wearable visual assistance (running on a cloud server), the steps are as follows (such as Figure 1 shown):

[0071] (1) Obtain image data in real time through the camera;

[0072] (2) After image preprocessing the image data obtained in the previous step, the video frames within the time period are intercepted;

[0073] (3) Applying a lightweight face detection algorithm (i.e., Ultra-Light-Fast-Generic-Face-Detector algorithm) to detect the image to be detected (specifically, locating the facial image of the image to be detected and then selecting all facial images within the detection range) to obtain all facial frame information in the image to be detected;

[0074] (4) inputting the face frame information obtained in step (3) into an expression recognition model based on a multimodal large model to obtain expression information of all face frame information (i.e., the category corresponding to the expression and its components);

[0075] Facial expression recognition models based on multimodal large models such as Figure 2 As shown, it includes a vision transformer feature extraction module, an adapter operator module, a large language model module, and a classifier module;

[0076] The input of the vision transformer feature extraction module is the face frame information, and the output is the preliminary feature extraction information. Its output is used as the input of the adapter operator module and the large language model module respectively. The output of the adapter operator module is used as the input of the large language model module. The output of the large language model module is comprehensive information with temporal modeling information and integrated with local detail features and global features, which is used as the input of the classifier module. The output of the classifier module is expression information.

[0077] The vision transformer feature extraction module is built based on the CLIP large model;

[0078] The adapter operator module includes learnable query token and cross-attention submodules. The input of query token is preliminary feature extraction information, and its output is used as the input of cross-attention submodule. The output of cross-attention submodule is local detail features with temporal modeling information.

[0079] The large language model is the llama-7B large language model;

[0080] The expression recognition model based on the multimodal large model is trained using a dynamic expression recognition training data set. A set of data in the dynamic expression recognition training data set (the data comes from three large-scale dynamic expression recognition data sets DFEW, FERV39k and MAFW publicly available in the field of expression recognition) includes image data and expression information corresponding to the image data (i.e., the category corresponding to the expression and its components). The image data is obtained by image preprocessing and intercepting video frames.

[0081] (5) Coordinate and analyze the expression recognition results of the current time period, generate an analysis report, store the relevant data in a database and make the data into a visualization interface. The analysis refers to analyzing the changes in the expression information of a target in the current time period. If the change in expression information exceeds a set threshold, the target is considered to have a negative emotion at that moment, otherwise it is considered to have a positive emotion;

[0082] Specifically, considering that the expression of the interactive object will change when responding to a question or scene, and this change can often reflect the psychological reaction of the subject, in order to highlight this change, the result of the expression recognition network is differentially calculated based on the emotional component, and the formula is:

[0083] score=S i -S i-1

[0084] Among them, S j ∈R 1×7 , S i represents the current moment, S i-1 Indicates the previous moment;

[0085] According to the difference result, the expression change process is reflected, and the seven expression categories are divided into "neutral" expressions and the expression threshold is set (i.e. Figure 1 The threshold value described in , and compare it with the difference result - Figure 1 The emotion threshold described in the above is used to classify the emotion into two categories: positive and negative. For example, anger, disgust, sadness, fear, and fluctuations in the expression process exceeding the threshold are classified as negative emotions, while happiness, calmness, surprise, and fluctuations in the expression process below the threshold are classified as positive emotions.

[0086] Example 2

[0087] A computer device, comprising: at least one processor and a memory communicatively connected to the at least one processor;

[0088] Among them, the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the multi-modal large model expression detection and analysis method for wearable visual assistance as described in Example 1.

[0089] Example 3

[0090] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the multimodal large model expression detection and analysis method for wearable visual assistance as described in Example 1.

[0091] It has been verified that the multimodal large-model expression detection and analysis method for wearable visual assistance of the present invention constructs an expression recognition model based on a multimodal large model. First, a vision transformer is used to build an image feature extraction module to preliminarily obtain class tokens covering global information and all image features. Then, a large language model is combined to capture global and local detail feature information by designing a specific fine-tuning operator adapter, and time series information is modeled. Expressions are classified by a classifier to achieve accurate and rapid expression recognition, and the expression of an interactive object can be recognized in real time based on a video stream. An expression information change threshold is introduced, and the expression information change is evaluated by the expression information change threshold, thereby realizing the evaluation of the expression change of the interactive object and realizing a more accurate recognition of the expression of the interactive object. The analysis report and related detection data can be displayed in the form of a visual interface, which can greatly provide convenience for users and has a good application prospect.

[0092] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0093] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the devices and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A multi-modal large-model expression detection and analysis method for wearable visual assistance, characterized in that: The following steps are involved: (1) Apply a lightweight face detection algorithm to detect the image to be detected and obtain all face frame information in the image to be detected; (2) inputting the face frame information obtained in step (1) into an expression recognition model based on a multimodal large model to obtain expression information of all face frame information, wherein the expression recognition model based on a multimodal large model comprises a vision transformer feature extraction module, an adapter operator module, a large language model module and a classifier module, wherein the input of the vision transformer feature extraction module is the face frame information, and the output is preliminary feature extraction information, and the outputs thereof are respectively used as inputs of the adapter operator module and the large language model module, and the output of the adapter operator module is used as input of the large language model module, and the output of the large language model module is comprehensive information with temporal modeling information and integrated with local detail features and global features, which is used as input of the classifier module, and the output of the classifier module is expression information; (3) The expression recognition results of the current time period are coordinated and analyzed to generate an analysis report. The analysis refers to analyzing the changes in the expression information of a target in the current time period. If the change in the expression information exceeds a set threshold, it is determined that the target has a negative emotion at that moment, otherwise it is a positive emotion.

2. The multimodal large model expression detection and analysis method for wearable visual assistance according to claim 1 is characterized in that: The lightweight face detection algorithm is Ultra-Light-Fast-Generic-Face-Detector algorithm.

3. The multimodal large model expression detection and analysis method for wearable visual assistance according to claim 1 is characterized in that: Step (1) specifically includes: locating the facial image of the image to be detected and then selecting all the facial images within the detection range.

4. The multimodal large model expression detection and analysis method for wearable visual assistance according to claim 1, characterized in that: The vision transformer feature extraction module is built based on the CLIP large model; The adapter operator module includes a learnable query token and a cross-attention submodule. The query token input is the preliminary feature extraction information, and its output is used as the input of the cross-attention submodule. The output of the cross-attention submodule is the local detail feature with time series modeling information. The large language model is the llama-7B large language model.

5. The multimodal large model expression detection and analysis method for wearable visual assistance according to claim 1 is characterized in that: The image to be detected is obtained by performing image preprocessing on the image data acquired by the camera in real time and intercepting the video frames within the time period; The expression recognition model based on the multimodal large model is trained using a dynamic expression recognition training data set, wherein a set of data in the dynamic expression recognition training data set includes image data and expression information corresponding to the image data, and the image data is obtained by image preprocessing and capturing video frames.

6. The multimodal large model expression detection and analysis method for wearable visual assistance according to claim 1, characterized in that: Step (3) After generating the analysis report, the relevant data is stored in the database and the data is made into a visual interface.

7. The multimodal large model expression detection and analysis method for wearable visual assistance according to claim 1, characterized in that: It is characterized in that The method is run on the cloud server side.

8. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the multi-modal large model expression detection and analysis method for wearable visual assistance as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the multi-modal large model expression detection and analysis method for wearable visual assistance as described in any one of claims 1 to 7 is implemented.

Images