A multi-granularity layer spectrum fusion occasion cognition method

Abstract

The application discloses a multi-granularity layer spectrum fusion occasion cognition method, comprising the following steps: collecting video images and audio data under different occasions, marking the video images and audio data for character face, skeleton, occasion description, character interaction detection, occasion background, occasion category and emotion category, constructing a face feature extraction network, a skeleton feature extraction network, a text feature extraction network, a character interaction feature extraction network, a background feature extraction network and an audio feature extraction network based on deep learning, respectively extracting features of different granularities and different layer spectrums in the occasion, and utilizing a graph neural network for feature fusion to obtain multi-granularity layer spectrum fusion features, and finally constructing a classifier to realize occasion recognition and emotion recognition, so as to complete the occasion cognition. The application utilizes multi-layer spectrum multi-granularity occasion features to provide objective and continuous cognition basis for the occasion cognition, so that more accurate occasion cognition is realized.

Description

Technical Field

[0001] This invention relates to a multi-granularity layer fusion method for situational cognition, belonging to the fields of computer vision and pattern recognition. Background Technology

[0002] With the rapid development of artificial intelligence and machine learning technologies, occasion recognition technology has received widespread attention and research in various application scenarios. Occasion refers to events and activities that occur in a specific time and space. Occasion recognition refers to the process by which computer systems understand and identify the current occasion by analyzing various information in a specific time and space, such as images, audio, and sensor data. This technology has significant application value in fields such as intelligent surveillance, autonomous driving, smart homes, and virtual reality.

[0003] Current technologies primarily focus on scene cognition, namely the understanding and analysis of the physical environment and spatial layout. However, they lack context cognition, namely the recognition and understanding of specific events, activities, their social and cultural contexts, and emotions. This deficiency limits the system's performance in the complex and dynamic real world. The importance of context cognition lies in its ability to enable systems not only to understand "where" but also "what happened" and "what emotions were expressed," thereby providing more intelligent and human-like services in fields such as intelligent assistants, social robots, and smart homes. Therefore, developing context cognition technology is crucial for improving the adaptability and interactivity of artificial intelligence systems. Furthermore, in changing and complex real-world environments, single-modal data cannot comprehensively capture the diverse information in complex situations, leading to insufficient accuracy and robustness of context cognition. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a contextual cognition method based on multi-granularity layer fusion.

[0005] Technical solution: The multi-granularity tomographic fusion method for situational cognition described in this invention includes the following steps:

[0006] (1) Collect video images and audio data from different occasions;

[0007] (2) Perform character face annotation, human skeleton annotation, occasion text description annotation, character interaction detection annotation, occasion background annotation, occasion category annotation and emotion category annotation on the video images and audio data obtained in step (1) to obtain the occasion dataset;

[0008] (3) Construct a feature extraction network based on deep learning. Use the occasion dataset obtained in step (2) to train the feature extraction network. The trained feature extraction network is used to extract features of different granularities and hierarchies in the occasion. Features of different granularities and hierarchies include facial features, human three-dimensional skeleton features, text features, human interaction behavior features, background features and audio features. The feature extraction network includes a facial feature extraction network, a skeleton feature extraction network, a text feature extraction network, a human interaction feature extraction network, a background feature extraction network and an audio feature extraction network.

[0009] (4) A multi-granularity layer spectrum fusion model is constructed by graph convolutional neural network. The multi-granularity layer spectrum fusion model uses the features of different granularities and layers obtained in step (3) to obtain multi-granularity layer spectrum fusion features.

[0010] (5) Construct an occasion recognition classifier and an emotion recognition classifier. The occasion recognition classifier and the emotion recognition classifier complete the occasion recognition and emotion recognition based on the multi-granularity layer fusion features obtained in step (4), thus completing occasion cognition.

[0011] Furthermore, the occasions in step (1) include social occasions, cultural occasions, public occasions, educational occasions, and entertainment occasions. The social occasions include family gatherings, business meetings, and weddings. The cultural occasions include cultural celebrations, religious ceremonies, and ethnic activities. The public occasions include public speeches and community activities. The educational occasions include classroom teaching, academic lectures, and graduation ceremonies. The entertainment occasions include concerts and sports competitions.

[0012] Furthermore, in step (2), the facial annotation of the person is performed using the Retinaface face detection algorithm to detect faces in the video image, obtain the position and size of the face, and manually correct any inaccurate detection results; the human skeleton annotation is performed using the VideoPose3D 3D human pose estimation algorithm to estimate the human skeleton of the person in the video image, obtain the coordinates of the person's three-dimensional skeleton, and manually correct any inaccurate text descriptions; the context text annotation is performed using the BLIP image description model to generate text descriptions for the acquired video image, and manually correct any inaccurate text descriptions; the person interaction detection annotation is performed using the DETR object detection algorithm to detect the position of people and objects in the video image, and manually correct any detection results; the context background annotation is performed using the SAM segmentation algorithm to segment and annotate the context background in the video image, and manually correct any inaccurate segmentation results; finally, the video image is annotated with context category and corresponding emotion category.

[0013] Furthermore, in step (2), the occasion categories include social occasions, cultural occasions, public occasions, educational occasions, and entertainment occasions; the emotion categories include happiness, sadness, tension, excitement, surprise, anger, embarrassment, and neutrality.

[0014] Furthermore, in step (1), the facial feature extraction network is a ResNet18-based facial feature extraction network, the skeleton feature extraction network is an ST-GCN-based skeleton feature extraction network, the text feature extraction network is a CLIP text encoder-based text feature extraction network, the character interaction feature extraction network is a CenterNet-based backbone network-based character interaction feature extraction network, the background feature extraction network is a ViT-based background feature extraction network, and the audio feature extraction network is an ERes2Net-based audio feature extraction network.

[0015] Furthermore, in step (3), the feature extraction network is trained using the scenario dataset obtained in step (2), including the following steps:

[0016] Step 31: Pre-train the feature extraction network using a publicly available general dataset;

[0017] Step 32: Fine-tune the feature extraction network using the aforementioned dataset to obtain the trained feature extraction network.

[0018] Furthermore, in step 32, fine-tuning the feature extraction network using the scenario dataset specifically involves: freezing all network parameters in the feature extraction network except for the last layer, and training the last layer of the feature extraction network using the scenario dataset.

[0019] Furthermore, the multi-granularity tomography fusion model described in step (4) is expressed by the following formula:

[0020]

[0021] F=concatenate(α1·F1,α2·F2,α3·F3,α4·F4,α5·F5,α6·F6)

[0022] In the formula, F f This represents multi-granularity hierarchical fusion features. F1, F2, F3, F4, F5, and F6 represent facial features, human 3D skeleton features, text features, human interaction behavior features, background features, and audio features, respectively. α1, α2, α3, α4, α5, and α6 are learnable parameters. concatenate(*) indicates feature concatenation. W represents the weight matrix of the graph convolutional neural network. Let σ(*) denote the normalized adjacency matrix, and let σ(*) denote the nonlinear activation function.

[0023] The multi-granularity layer fusion model concatenates features with different weights, and then uses a graph convolutional network to further fuse the concatenated features to obtain fused features.

[0024] Furthermore, in step (5), the occasion recognition classifier includes a first fully connected layer and a first softmax layer, and the emotion recognition classifier includes a second fully connected layer and a second softmax layer. The inputs to the occasion recognition classifier and the emotion recognition classifier are the multi-granularity layer fusion features obtained in step four, and the outputs are the probability distributions of the occasion category and the probability distributions of the emotion category, respectively.

[0025] The occasion recognition classifier and the sentiment recognition classifier each consist of a fully connected layer and a softmax layer, respectively. The input is the fused features from the previous stage, and the output is the probability distribution of the occasion category and the sentiment category.

[0026] Furthermore, the occasion identification classifier is represented by the following formula:

[0027]

[0028] In the formula, C represents the output of the multi-granularity layer fusion features after passing through the first fully connected layer. O p represents the number of categories of occasions. O,j This indicates the probability distribution for different scenarios.

[0029] The emotion recognition classifier is represented by the following formula:

[0030]

[0031] In the formula, C represents the output of the multi-granularity layer fusion features after passing through the second fully connected layer. E p represents the number of emotion categories. E,i This represents the probability distribution corresponding to different emotions.

[0032] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: The multi-granularity hierarchical fusion method for situational cognition of this invention extracts data features from different modalities and uses graph neural networks for feature fusion, capturing complex relationships and dependencies in multi-modal data, improving the quality of feature fusion, and achieving effective fusion of multi-scale information by propagating and aggregating information in the graph structure. By fusing data from different modalities at multiple levels and in multiple granularities, it achieves accurate cognition in complex situations, enabling the system to understand "what happened" and "what emotions were expressed." Attached Figure Description

[0033] Figure 1 This is a flowchart of the multi-granularity tomographic fusion method for situational cognition according to the present invention;

[0034] Figure 2 This is a schematic diagram of the structure of a facial feature extraction network based on ResNet18;

[0035] Figure 3 This is a schematic diagram of the structure of the ST-GCN-based skeleton feature extraction network;

[0036] Figure 4 This is a schematic diagram of the structure of a text feature extraction network based on the CLIP text encoder;

[0037] Figure 5 This is a schematic diagram of the structure of a network for extracting human interaction features based on the CenterNet backbone network.

[0038] Figure 6 This is a schematic diagram of the background feature extraction network based on ViT;

[0039] Figure 7 This is a schematic diagram of the structure of an audio feature extraction network based on ERes2Net. Detailed Implementation

[0040] Example 1

[0041] like Figure 1 As shown, the multi-granularity tomographic fusion method for situational cognition of the present invention includes the following steps:

[0042] Step S1: Collect video images and audio data from different occasions. The video images cover social occasions, cultural occasions, public occasions, educational occasions, and entertainment occasions. Social occasions include family gatherings, business meetings, and weddings, cultural occasions include cultural celebrations, religious ceremonies, and ethnic activities, public occasions include public speeches and community activities, educational occasions include classroom teaching, academic lectures, and graduation ceremonies, and entertainment occasions include concerts and sports competitions. The audio data includes various audio elements present in each occasion, such as voices of people communicating and background music.

[0043] Step S2: Annotate the video images with facial features, human skeletons, contextual text descriptions, character interactions and locations, contextual background, contextual category, and emotion. Specifically, facial feature annotation uses the Retinaface face detection algorithm to detect faces in the video, obtain their position and size, and correct inaccurate detection results. Human skeleton annotation uses the VideoPose3D 3D human pose estimation algorithm to estimate the human skeletons in the video, obtain the coordinates of the characters' 3D skeletons, and manually correct them. Contextual text description annotation uses the BLIP image description model to generate different text descriptions for the acquired video images, and manually corrects inaccurate text descriptions to better reflect the actual data. Character interaction detection annotation uses the DETR object detection algorithm to detect the positions of people and objects in the acquired video images, and manually corrects the detection results. Contextual background annotation uses the SAM segmentation algorithm to segment and annotate the contextual background in the video, and manually corrects inaccurate segmentation results. Finally, all acquired video images are annotated with the contextual category and corresponding emotion.

[0044] Step S3: Please refer to Figure 2 As shown, a facial feature extraction network based on ResNet18 is constructed, with its last layer being a fully connected layer. The network's last layer is trained using the collected dataset, and the pre-trained network weights are fine-tuned to extract facial features. Please refer to [link to documentation]. Figure 3 As shown, a skeleton feature extraction network based on ST-GCN is constructed, with its last layer being a fully connected layer. The network's last layer is trained using the collected dataset, and the pre-trained network weights are fine-tuned to extract 3D human skeleton features. Please refer to [link to documentation]. Figure 4 As shown, a text feature extraction network based on the CLIP text encoder is further constructed, with its last layer being a fully connected layer. The last layer of the network is trained using the collected dataset, and the pre-trained network weights are fine-tuned to extract text features. Please refer to [link to documentation]. Figure 5 As shown, a network for extracting human interaction features based on the CenterNet backbone is constructed. Its last layer is a fully connected layer. The collected dataset is used to train the last layer of the network, and the pre-trained network weights are fine-tuned to extract human interaction features. Please refer to [link to documentation]. Figure 6 As shown, a background feature extraction network based on ViT is constructed, with its last layer being a fully connected layer. The network's last layer is trained using the collected dataset, and the pre-trained network weights are fine-tuned to extract background features. Please refer to [link to relevant documentation]. Figure 7As shown, an audio feature extraction network based on ERes2Net is constructed, with its last layer being a fully connected layer. The last layer of the network is trained using the collected dataset, and the pre-trained network weights of the model are fine-tuned to extract audio features.

[0045] Step S4: Align features of different granularities using convolutional layers to obtain features of the same size. Then, construct adaptive learnable parameters (α1, α2, α3, α4, α5, α6) to parameterize the feature importance of different granularity spectral features, and concatenate the features of each granularity spectral feature. Specifically, this can be represented as:

[0046] F=concatenate(α1·F1,α2·F2,α3·F3,α4·F4,α5·F5,α6·F6)

[0047] Where F1, F2, F3, F4, F5, and F6 represent the extracted facial features, skeletal features, text features, character interaction features, background features, and audio features, respectively, and concatenate indicates feature concatenation. Finally, feature fusion is performed on the multi-granularity hierarchical features based on a graph neural network, which can be specifically represented as:

[0048]

[0049] Where W refers to the weight matrix of the graph convolutional neural network. Let σ be the normalized adjacency matrix, and σ be the nonlinear activation function.

[0050] Step S5: Construct occasion recognition classifiers and emotion recognition classifiers based on fully connected layers and softmax. By inputting fused hierarchical features of different granularities, different occasions and emotions are identified, thus completing occasion cognition. The occasion recognition classifier can be specifically represented as follows:

[0051]

[0052] in, C represents the output of multi-granularity layer fusion features after passing through a fully connected layer. O p represents the number of categories of occasions. O,j This represents the probability distribution of the output for different contexts. A sentiment recognition classifier can be specifically represented as:

[0053]

[0054] in, C represents the output of multi-granularity layer fusion features after passing through a fully connected layer. E p represents the number of emotion categories. E,i This represents the probability distribution corresponding to different emotions.

[0055] This invention introduces contextual awareness for the first time, enabling the system to understand not only "where" it is, but also "what happened" and "what emotions were expressed." This allows the system to perform better in complex and dynamic real-world environments. It expands upon existing technologies that focus solely on understanding and analyzing physical environments and spatial layouts. It broadens application areas, such as intelligent assistants, social robots, and smart home systems, enabling these systems to provide more personalized and context-sensitive services based on specific events or activities. This invention utilizes a graph neural network to fuse data from multiple modalities at different granular levels through multi-granularity feature extraction and hierarchical fusion mechanisms. This multi-layered information integration allows the system to more accurately identify and understand context in complex environments.

Claims

1. A contextual cognition method based on multi-granularity tomographic fusion, characterized in that, Includes the following steps: (1) Collect video images and audio data from different occasions; (2) Perform character face annotation, human skeleton annotation, occasion text description annotation, character interaction detection annotation, occasion background annotation, occasion category annotation and emotion category annotation on the video images and audio data obtained in step (1) to obtain the occasion dataset; (3) Construct a feature extraction network based on deep learning. Use the occasion dataset obtained in step (2) to train the feature extraction network. The trained feature extraction network is used to extract features of different granularities and hierarchies in the occasion. Features of different granularities and hierarchies include facial features, human three-dimensional skeleton features, text features, human interaction behavior features, background features and audio features. The feature extraction network includes a facial feature extraction network, a skeleton feature extraction network, a text feature extraction network, a human interaction feature extraction network, a background feature extraction network and an audio feature extraction network. (4) A multi-granularity layer spectrum fusion model is constructed by graph convolutional neural network. The multi-granularity layer spectrum fusion model uses the features of different granularities and layers obtained in step (3) to obtain multi-granularity layer spectrum fusion features. The multi-granularity tomography fusion model is expressed by the following formula: ； ； In the formula, This indicates multi-granularity spectral fusion characteristics. , , , , and These represent facial features, human 3D skeleton features, text features, character interaction behavior features, background features, and audio features, respectively. , , , , and For learnable parameters, This indicates that feature splicing is being performed. This represents the weight matrix of a graph convolutional neural network. Represents the normalized adjacency matrix. Represents a nonlinear activation function; (5) Construct an occasion recognition classifier and an emotion recognition classifier. The occasion recognition classifier and the emotion recognition classifier complete the occasion recognition and emotion recognition based on the multi-granularity layer fusion features obtained in step (4), thus completing occasion cognition.

2. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 1, characterized in that, In step (1), the occasions include social occasions, cultural occasions, public occasions, educational occasions and entertainment occasions. The social occasions include family gatherings, business meetings and weddings. The cultural occasions include cultural celebrations, religious ceremonies and ethnic activities. The public occasions include public speeches and community activities. The educational occasions include classroom teaching, academic lectures and graduation ceremonies. The entertainment occasions include concerts and sports competitions.

3. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 1, characterized in that, In step (2), the facial character annotation uses the Retinaface face detection algorithm to detect faces in the video image, obtain the position and size of the faces, and manually correct inaccurate detection results; the human skeleton annotation uses the VideoPose3D 3D human pose estimation algorithm to estimate the human skeleton of the person in the video image, obtain the coordinates of the person's three-dimensional skeleton, and manually correct it; the occasion text description annotation uses the BLIP image description model to generate text descriptions for the acquired video image, and manually corrects inaccurate text descriptions; the human interaction detection annotation uses the DETR object detection algorithm to detect the position of people and objects and objects in the video image, and manually corrects the detection results; the occasion background annotation uses the SAM segmentation algorithm to segment and annotate the occasion background in the video image, and manually corrects inaccurate segmentation results; finally, the video image is annotated with occasion category and corresponding emotion category.

4. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 1, characterized in that, In step (2), the occasion categories include social occasions, cultural occasions, public occasions, educational occasions, and entertainment occasions; the emotion categories include happy, sad, nervous, excited, surprised, angry, embarrassed, and neutral.

5. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 1, characterized in that, In step (2), the facial feature extraction network is a facial feature extraction network based on ResNet18, the skeleton feature extraction network is a skeleton feature extraction network based on ST-GCN, the text feature extraction network is a text feature extraction network based on CLIP text encoder, the human interaction feature extraction network is a human interaction feature extraction network based on CenterNet backbone network, the background feature extraction network is a background feature extraction network based on ViT, and the audio feature extraction network is an audio feature extraction network based on ERes2Net.

6. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 1, characterized in that, In step (3), the feature extraction network is trained using the scenario dataset obtained in step (2), including the following steps: Step 31: Pre-train the feature extraction network using a publicly available general dataset; Step 32: Fine-tune the feature extraction network using the aforementioned dataset to obtain the trained feature extraction network.

7. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 6, characterized in that, In step 32, fine-tuning the feature extraction network using the scenario dataset specifically involves: freezing all network parameters of the feature extraction network except for the last layer, and training the last layer of the feature extraction network using the scenario dataset.

8. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 1, characterized in that, In step (5), the occasion recognition classifier includes a first fully connected layer and a first softmax layer, and the emotion recognition classifier includes a second fully connected layer and a second softmax layer. The inputs of the occasion recognition classifier and the emotion recognition classifier are the multi-granularity layer fusion features obtained in step four, and the outputs are the probability distributions of the occasion category and the probability distributions of the emotion category, respectively.

9. The contextual cognition method based on multi-granularity tomographic fusion as described in claim 8, characterized in that, The situation identification classifier is represented by the following formula: ； In the formula, This represents the output of the multi-granularity layer fusion features after passing through the first fully connected layer. Indicates the number of categories of occasions. This indicates the probability distribution for different scenarios. The emotion recognition classifier is represented by the following formula: ； In the formula, This represents the output of the multi-granularity layer fusion features after passing through the second fully connected layer. The number of categories representing emotions. This represents the probability distribution corresponding to different emotions.

Images