Multi-granularity layer spectrum fusion occasion cognition method

Abstract

The invention discloses an occasion cognition method for multi-granularity layer spectrum fusion, which comprises the following steps of: acquiring video images and audio data under different occasions, and marking a character face, a skeleton, occasion description, character interaction detection, occasion background, occasion category and emotion category on the video images and the audio data; constructing a facial feature extraction network, a skeleton feature extraction network, a text feature extraction network, a figure interaction feature extraction network, a background feature extraction network and an audio feature extraction network based on deep learning, respectively extracting features of different granularities and different layer spectrums in an occasion, and performing feature fusion by using a graph neural network; and finally, a classifier is constructed to realize occasion recognition and emotion recognition, so that occasion cognition is completed. According to the method, the objective and continuous cognitive basis is provided for occasion cognition by utilizing the occasion characteristics of multi-layer spectrum and multi-granularity, so that more accurate occasion cognition is realized.

Description

Technical Field

[0001] The invention relates to a multi-granularity layer spectrum fusion occasion recognition method, belonging to the field of computer vision and pattern recognition. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, occasion recognition technology has received extensive attention and research in various application scenarios. Occasion refers to events and activities that occur at a specific time and space. Occasion recognition refers to the process by which a computer system understands and recognizes the current occasion by analyzing various information in a specific time and space, such as images, audio, sensor data, etc. This technology has important application value in the fields of intelligent monitoring, autonomous driving, smart home, virtual reality, etc.

[0003] Current technologies mainly focus on scene cognition, that is, the understanding and analysis of the physical environment and spatial layout. However, there is a lack of occasion cognition, that is, the recognition and understanding of specific events, activities and their social and cultural contexts and emotions. This lack limits the performance of the system in the complex and dynamic real world. The importance of occasion cognition lies in that it enables the system to understand not only "where" but also "what happened" and "what emotions were expressed", thereby providing more intelligent and humane services in areas such as smart assistants, social robots and smart homes. Therefore, the development of occasion cognition technology is crucial to improving the adaptability and interactivity of artificial intelligence systems. In addition, in a changeable and complex actual environment, single modality data cannot fully capture the diverse information in complex occasions, which will lead to insufficient accuracy and robustness of occasion cognition. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a multi-granularity layer spectrum fusion occasion recognition method.

[0005] Technical solution: The multi-granularity layer spectrum fusion occasion recognition method of the present invention comprises the following steps:

[0006] (1) Collect video images and audio data from different occasions;

[0007] (2) performing character face annotation, human skeleton annotation, scene text description annotation, character interaction detection annotation, scene background annotation, scene category annotation, and emotion category annotation on the video image and audio data obtained in step (1) to obtain a scene dataset;

[0008] (3) constructing a feature extraction network based on deep learning, and using the scene data set obtained in step (2) to train the feature extraction network, wherein the trained feature extraction network is used to extract features of different granularities and spectral levels in the scene, wherein the features of different granularities and spectral levels include facial features, human three-dimensional skeleton features, text features, character interaction behavior features, background features, and audio features, and the feature extraction network includes a facial feature extraction network, a skeleton feature extraction network, a text feature extraction network, a character interaction feature extraction network, a background feature extraction network, and an audio feature extraction network;

[0009] (4) constructing a multi-granularity layer-spectrum fusion model through a graph convolutional neural network, wherein the multi-granularity layer-spectrum fusion model uses the features of different granularities and layer spectra obtained in step (3) to obtain multi-granularity layer-spectrum fusion features;

[0010] (5) Constructing an occasion recognition classifier and an emotion recognition classifier, wherein the occasion recognition classifier and the emotion recognition classifier complete occasion recognition and emotion recognition based on the multi-granularity layer spectrum fusion features obtained in step (4), thereby completing occasion recognition.

[0011] Furthermore, the occasions in step (1) include social occasions, cultural occasions, public occasions, educational occasions and entertainment occasions, wherein the social occasions include family gatherings, business meetings and weddings, the cultural occasions include cultural celebrations, religious ceremonies and ethnic activities, the public occasions include public speeches and community activities, the educational occasions include classroom teaching, academic lectures and graduation ceremonies, and the entertainment occasions include concerts and sports games.

[0012] Furthermore, in step (2), the facial annotation of the characters uses the face detection algorithm Retinaface to perform face detection on the video image, obtains the position and size of the face, and manually corrects the inaccurate detection results; the human skeleton annotation uses the 3D human pose estimation algorithm VideoPose3D to estimate the human skeleton of the characters in the video image, obtains the coordinates of the three-dimensional skeleton of the characters, and manually corrects them; the occasion text description annotation uses the image description model BLIP to generate a text description for the collected video image, and manually corrects the inaccurate text description; the character interaction detection annotation uses the target detection algorithm DETR to perform position detection and object detection on the video image, and manually corrects the detection results; the occasion background annotation uses the segmentation algorithm SAM to segment and annotate the occasion background in the video image, and manually corrects the inaccurate segmentation results; finally, the video image is annotated with occasion categories and corresponding emotion categories.

[0013] Furthermore, in step (2), the occasion categories include social occasions, cultural occasions, public occasions, educational occasions and entertainment occasions; and the emotion categories include happy, sad, nervous, excited, surprised, angry, embarrassed and neutral.

[0014] Furthermore, the facial feature extraction network in step (1) is a facial feature extraction network based on Resnet18, the skeleton feature extraction network is a skeleton feature extraction network based on ST-GCN, the text feature extraction network is a text feature extraction network based on CLIP text encoder, the character interaction feature extraction network is a character interaction feature extraction network based on the backbone network of CenterNet, the background feature extraction network is a background feature extraction network based on ViT, and the audio feature extraction network is an audio feature extraction network based on ERes2Net.

[0015] Furthermore, in step (3), the feature extraction network is trained using the scene data set obtained in step (2), including the following steps:

[0016] Step 31: pre-training the feature extraction network using a public general data set;

[0017] Step 32: fine-tune the feature extraction network using the scenario data set to obtain a trained feature extraction network.

[0018] Furthermore, in step 32, fine-tuning the feature extraction network using the scenario data set is specifically as follows: freezing all network parameters except the last layer in the feature extraction network, and training the last layer of the feature extraction network using the scenario data set.

[0019] Furthermore, the multi-granularity layer spectrum fusion model in step (4) is expressed by the following formula:

[0020]

[0021] F=concatenate(α1·F1,α2·F2,α3·F3,α4·F4,α5·F5,α6·F6)

[0022] In the formula, F f represents multi-granularity layer-spectrum fusion features, F1, F2, F3, F4, F5 and F6 represent facial features, human 3D skeleton features, text features, character interaction behavior features, background features and audio features respectively, α1, α2, α3, α4, α5 and α6 are learnable parameters, concatenate(*) represents feature concatenation, W represents the weight matrix of graph convolutional neural network, represents the normalized adjacency matrix, and σ(*) represents the nonlinear activation function.

[0023] The multi-granularity layer spectrum fusion model performs feature splicing through different weights, and then further fuses the spliced ​​features using the graph convolutional network to obtain fused features.

[0024] Furthermore, the occasion recognition classifier in step (5) includes a first fully connected layer and a first softmax layer, and the emotion recognition classifier includes a second fully connected layer and a second softmax layer. The inputs of the occasion recognition classifier and the emotion recognition classifier are both the multi-granularity layer-spectrum fusion features obtained in step 4, and the outputs are the probability distribution of occasion categories and the probability distribution of emotion categories, respectively.

[0025] The occasion recognition classifier and emotion recognition classifier are composed of a fully connected layer and a softmax layer respectively. The input is the fusion feature of the previous stage, and the output is the probability distribution of the occasion category and the emotion category.

[0026] Furthermore, the occasion recognition classifier is represented by the following formula:

[0027]

[0028] In the formula, represents the output of the multi-granularity layer spectrum fusion feature after passing through the first fully connected layer, C O The number of categories representing occasions, p O,j Indicates the probability distribution of output corresponding to different occasions;

[0029] The emotion recognition classifier is expressed as follows:

[0030]

[0031] In the formula, represents the output of the multi-granularity layer spectrum fusion feature after passing through the second fully connected layer, C E The number of categories representing emotions, p E,i Represents the probability distribution of the output corresponding to different emotions.

[0032] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The multi-granularity layer spectrum fusion situation recognition method of the present invention captures the complex relationships and dependencies in multi-modal data by extracting data features of different modalities and using graph neural networks for feature fusion, improving the quality of feature fusion, and achieving effective fusion of multi-scale information by propagating and aggregating information in the graph structure. By fusing data from different modalities at multiple levels and granularity, accurate cognition of complex situations can be achieved, allowing the system to understand "what happened" and "what emotions were expressed." BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the occasion recognition method of multi-granularity layer spectrum fusion of the present invention;

[0034] Figure 2 This is a schematic diagram of the structure of the facial feature extraction network based on Resnet18;

[0035] Figure 3 It is a schematic diagram of the structure of the skeleton feature extraction network based on ST-GCN;

[0036] Figure 4 It is a structural diagram of the text feature extraction network based on CLIP text encoder;

[0037] Figure 5 This is a schematic diagram of the structure of the character interaction feature extraction network based on the CenterNet backbone network;

[0038] Figure 6 It is a schematic diagram of the structure of the background feature extraction network based on ViT;

[0039] Figure 7 This is a structural diagram of the audio feature extraction network based on ERes2Net. DETAILED DESCRIPTION

[0040] Example 1

[0041] like Figure 1 As shown, the multi-granularity layer spectrum fusion occasion recognition method of the present invention includes the following steps:

[0042] Step S1: Collect video images and audio data of different occasions. The occasions involved in the video images include social occasions, cultural occasions, public occasions, educational occasions and entertainment occasions, among which social occasions include family gatherings, business meetings and weddings, cultural occasions include cultural celebrations, religious ceremonies and ethnic activities, public occasions include public speeches and community activities, educational occasions include classroom teaching, academic lectures and graduation ceremonies, entertainment occasions include concerts and sports competitions, and audio data include various audios existing in various occasions, such as voices of people communicating, background music, etc.

[0043] Step S2: Label the face of the person, the human skeleton, the occasion text description, the interaction behavior and position of the person, the occasion background, the occasion category and the emotion in the video image. Among them, the face labeling of the person uses the face detection algorithm Retinaface to detect the face in the video, obtain the position and size of the face, and correct the inaccurate detection results. The human skeleton labeling uses the 3D human posture estimation algorithm VideoPose3D to estimate the human skeleton of the person in the video, obtain the coordinates of the three-dimensional skeleton of the person in the video, and make manual corrections. The occasion text description labeling uses the image description model BLIP to generate different text descriptions for the collected video images, and manually corrects the inaccurate text descriptions to fit the actual data. The character interaction detection labeling uses the target detection algorithm DETR to perform the position and object detection of the person and object in the video, and manually corrects the detection results. The occasion background labeling uses the segmentation algorithm SAM to perform corresponding segmentation labeling on the occasion background in the video, and manually corrects the inaccurate segmentation results. Finally, all the collected video images are labeled with the category of the occasion and the corresponding emotion.

[0044] Step S3: Please refer to Figure 2 As shown in the figure, a facial feature extraction network based on Resnet18 is constructed, the last layer of which is the fully connected layer. The last layer of the network is trained using the collected data set, and the pre-trained network weights of the model are fine-tuned to extract facial features. Figure 3 As shown in the figure, a skeleton feature extraction network based on ST-GCN is constructed, the last layer of which is a fully connected layer. The last layer of the network is trained using the collected data set, and the pre-trained network weights of the model are fine-tuned to extract the 3D skeleton features of the human body. Figure 4 As shown in the figure, a text feature extraction network based on the CLIP text encoder is further constructed, the last layer of which is a fully connected layer. The last layer of the network is trained using the collected data set, and the pre-trained network weights of the model are fine-tuned to extract text features. Figure 5 As shown in the figure, a character interaction feature extraction network based on the CenterNet backbone network is constructed. The last layer is a fully connected layer. The last layer of the network is trained using the collected data set, and the pre-trained network weights of the model are fine-tuned to extract character interaction features. Figure 6 As shown in the figure, a background feature extraction network based on ViT is constructed, the last layer of which is a fully connected layer. The last layer of the network is trained using the collected data set, and the pre-trained network weights of the model are fine-tuned to extract background features. Figure 7As shown, an audio feature extraction network based on ERes2Net is constructed, the last layer of which is a fully connected layer. The last layer of the network is trained with the collected data set, and the pre-trained network weights of the model are fine-tuned to extract audio features.

[0045] Step S4: Align the features of different granularity spectra through the convolution layer to obtain features of the same size. Then construct adaptive learnable parameters (α1, α2, α3, α4, α5, α6) to parameterize the feature importance of different granularity layer spectra, and concatenate the features of each granularity layer spectra, which can be specifically expressed as:

[0046] F=concatenate(α1·F1,α2·F2,α3·F3,α4·F4,α5·F5,α6·F6)

[0047] Among them, F1, F2, F3, F4, F5, and F6 represent the extracted facial features, skeleton features, text features, character interaction features, background features, and audio features, respectively, and concatenate means feature concatenation. Finally, the multi-granularity layer spectral features are fused based on the graph neural network, which can be specifically expressed as:

[0048]

[0049] Among them, W refers to the weight matrix of the graph convolutional neural network, is the normalized adjacency matrix, and σ is the nonlinear activation function.

[0050] Step S5: Based on the fully connected layer and softmax, the occasion recognition classifier and the emotion recognition classifier are constructed, and different occasions and emotions are recognized by inputting the fused layer spectrum features of different granularities, thereby completing the occasion recognition. The occasion recognition classifier can be specifically expressed as:

[0051]

[0052] in, represents the output of the multi-granularity layer spectrum fusion feature after passing through the fully connected layer, C O The number of categories representing occasions, p O,j It represents the probability distribution of the output corresponding to different occasions. The emotion recognition classifier can be specifically expressed as:

[0053]

[0054] in, represents the output of the multi-granularity layer spectrum fusion feature after passing through the fully connected layer, C E The number of categories representing emotions, p E,i Represents the probability distribution of the output corresponding to different emotions.

[0055] The present invention introduces occasion cognition for the first time, allowing the system to understand not only "where" but also "what happened" and "what emotions were expressed". This can make the system perform better in a complex and dynamic real world. It expands the existing technology that only focuses on understanding and analyzing the physical environment and spatial layout. It expands the application areas, such as smart assistants, social robots, and smart home systems, so that these systems can provide more personalized and context-related services based on specific events or activities. The present invention uses graph neural networks to fuse data from multi-modality at different granularity levels through multi-granularity feature extraction and layer-spectrum fusion mechanisms. This multi-level information integration enables the system to more accurately identify and understand occasions in complex environments.

Claims

1. A multi-granularity layer spectrum fusion occasion recognition method, characterized in that: The steps include: (1) Collect video images and audio data from different occasions; (2) performing character face annotation, human skeleton annotation, scene text description annotation, character interaction detection annotation, scene background annotation, scene category annotation, and emotion category annotation on the video image and audio data obtained in step (1) to obtain a scene dataset; (3) constructing a feature extraction network based on deep learning, and using the scene data set obtained in step (2) to train the feature extraction network, wherein the trained feature extraction network is used to extract features of different granularities and spectral levels in the scene, wherein the features of different granularities and spectral levels include facial features, human three-dimensional skeleton features, text features, character interaction behavior features, background features, and audio features, and the feature extraction network includes a facial feature extraction network, a skeleton feature extraction network, a text feature extraction network, a character interaction feature extraction network, a background feature extraction network, and an audio feature extraction network; (4) constructing a multi-granularity layer-spectrum fusion model through a graph convolutional neural network, wherein the multi-granularity layer-spectrum fusion model uses the features of different granularities and layer spectra obtained in step (3) to obtain multi-granularity layer-spectrum fusion features; (5) Constructing an occasion recognition classifier and an emotion recognition classifier, wherein the occasion recognition classifier and the emotion recognition classifier complete occasion recognition and emotion recognition based on the multi-granularity layer spectrum fusion features obtained in step (4), thereby completing occasion recognition.

2. The multi-granularity layer spectrum fusion scenario recognition method according to claim 1, characterized in that: In step (1), the occasions include social occasions, cultural occasions, public occasions, educational occasions and entertainment occasions. The social occasions include family gatherings, business meetings and weddings. The cultural occasions include cultural celebrations, religious ceremonies and ethnic activities. The public occasions include public speeches and community activities. The educational occasions include classroom teaching, academic lectures and graduation ceremonies. The entertainment occasions include concerts and sports games.

3. The multi-granularity layer spectrum fusion scenario recognition method as claimed in claim 1, characterized in that: In step (2), the character face annotation uses the face detection algorithm Retinaface to perform face detection on the video image, obtains the position and size of the face, and manually corrects inaccurate detection results; the human skeleton annotation uses the 3D human pose estimation algorithm VideoPose3D to estimate the human skeleton of the character in the video image, obtains the coordinates of the three-dimensional skeleton of the character, and manually corrects; the occasion text description annotation uses the image description model BLIP to generate a text description for the collected video image, and manually corrects the inaccurate text description; the character interaction detection annotation uses the target detection algorithm DETR to perform position detection and object detection on the video image, and manually corrects the detection results; the occasion background annotation uses the segmentation algorithm SAM to segment and annotate the occasion background in the video image, and manually corrects the inaccurate segmentation results; finally, the video image is annotated with occasion categories and corresponding emotion categories.

4. The multi-granularity layer spectrum fusion scenario recognition method according to claim 1, characterized in that: In step (2), the occasion categories include social occasions, cultural occasions, public occasions, educational occasions, and entertainment occasions; and the emotion categories include happy, sad, nervous, excited, surprised, angry, embarrassed, and neutral.

5. The multi-granularity layer spectrum fusion scenario recognition method as claimed in claim 1, characterized in that: In step (2), the facial feature extraction network is a facial feature extraction network based on Resnet18, the skeleton feature extraction network is a skeleton feature extraction network based on ST-GCN, the text feature extraction network is a text feature extraction network based on CLIP text encoder, the character interaction feature extraction network is a character interaction feature extraction network based on the backbone network of CenterNet, the background feature extraction network is a background feature extraction network based on ViT, and the audio feature extraction network is an audio feature extraction network based on ERes2Net.

6. The multi-granularity layer spectrum fusion scenario recognition method according to claim 1, characterized in that: In step (3), the feature extraction network is trained using the scene data set obtained in step (2), including the following steps: Step 31: pre-training the feature extraction network using a public general data set; Step 32: fine-tune the feature extraction network using the scenario data set to obtain a trained feature extraction network.

7. The multi-granularity layer spectrum fusion scenario recognition method according to claim 6, characterized in that: In step 32, fine-tuning the feature extraction network using the scenario data set is specifically as follows: freezing all network parameters except the last layer in the feature extraction network, and training the last layer of the feature extraction network using the scenario data set.

8. The multi-granularity layer spectrum fusion scenario recognition method as claimed in claim 1, characterized in that: In step (4), the multi-granularity layer spectrum fusion model is expressed by the following formula: F=concatenate(α1·F1,α2·F2,α3·F3,α4·F4,α5·F5,α6·F6) In the formula, F f represents multi-granularity layer-spectrum fusion features, F1, F2, F3, F4, F5 and F6 represent facial features, human 3D skeleton features, text features, character interaction behavior features, background features and audio features respectively, α1, α2, α3, α4, α5 and α6 are learnable parameters, concatenate(*) represents feature concatenation, W represents the weight matrix of graph convolutional neural network, represents the normalized adjacency matrix, and σ(*) represents the nonlinear activation function.

9. The multi-granularity layer spectrum fusion scenario recognition method as claimed in claim 1, characterized in that: In step (5), the occasion recognition classifier includes a first fully connected layer and a first softmax layer, and the emotion recognition classifier includes a second fully connected layer and a second softmax layer. The inputs of the occasion recognition classifier and the emotion recognition classifier are both the multi-granularity layer-spectrum fusion features obtained in step 4, and the outputs are the probability distribution of the occasion category and the probability distribution of the emotion category, respectively.

10. The multi-granularity layer spectrum fusion scenario recognition method according to claim 9, characterized in that: The occasion recognition classifier is represented by the following formula: In the formula, represents the output of the multi-granularity layer spectrum fusion feature after passing through the first fully connected layer, C O Indicates the number of categories of occasions, p O,j Indicates the probability distribution of output corresponding to different occasions; The emotion recognition classifier is expressed as follows: In the formula, represents the output of the multi-granularity layer spectrum fusion feature after passing through the second fully connected layer, C E The number of categories representing emotions, p E,i Represents the probability distribution of the output corresponding to different emotions.

Images