Masked position reconstruction and skeleton feature multi-task identification method for emotion behavior

Abstract

The application discloses a mask position reconstruction and skeleton feature emotional behavior multi-task identification method, and belongs to the technical field of computer vision; the method comprises the following steps: acquiring original image data of a target object; generating an occlusion mask graph by using a spatial occlusion perception module, and dividing the image into visible blocks and occlusion blocks; inputting the visible blocks into a mask feature reconstruction module, and reconstructing the occlusion blocks by using semantic prior features to generate a repaired image; inputting the reconstructed image into a skeleton collaborative multi-task module to extract multi-scale spatial features, and extracting skeleton features by using a pose estimation network to perform cross-modal fusion on the multi-scale spatial features; inputting the high-dimensional joint representation into emotion and behavior identification branches of the skeleton collaborative multi-task module respectively, and synchronously outputting the emotional state and the behavior state of the target object; the method solves the problems of target feature loss under a complex occlusion environment and feature coupling in multi-task identification by guiding image repair and feature decoupling through skeleton features.

Description

Technical Field

[0001] This application belongs to the field of computer vision and ecological monitoring technology, specifically involving a multi-task recognition method for emotion and behavior based on mask location reconstruction and skeleton features. Background Technology

[0002] Facial emotion and behavior analysis in wild animals is an important branch of computer vision applications in ecological conservation, carrying significant value in understanding the social structure of animal groups and achieving intelligent biodiversity conservation. However, due to the complex wild habitats in which Qinling golden monkeys inhabit year-round, severe natural shading (such as leaves and branches) leads to a significant loss of key features of the torso and face in images. To address this issue, traditional methods often involve acquiring data in controlled environments (such as laboratories or indoor cameras) for manual or automated identification. However, due to the uncontrollability of the real wild environment, it is not only difficult to extract the delicate facial details and limb topology of the original artwork, but also difficult to achieve large-scale monitoring of natural conditions. Therefore, employing occlusion-resistant digital visual analysis methods to restore missing features and perform joint identification is of great significance for wildlife conservation.

[0003] The challenges of wildlife monitoring, including occlusion issues, limitations of existing technologies, and decoupling failures due to a lack of pose guidance, have been addressed in recent years. Benefiting from the widespread adoption of deep learning technologies, CNNs and general-purpose Transformers have achieved some success in natural world image classification and image completion tasks. However, when faced with images of wild golden monkeys containing complex hair texture features and extremely irregular occlusion, models struggle to perceive the boundaries between current features and distractions, easily mistaking foreground distractions such as leaves for target features. Furthermore, in the joint recognition of emotions and behaviors in wild animals, learning minute facial details and torso structure layout information is crucial for models to understand fine-grained, deeper meanings. Therefore, methods such as LaMa large receptive field image inpainting, multi-gated routing networks (CGC, PLE), and spatial temporal graph convolution (ST-GCN) have emerged in related fields. While existing methods have achieved some success in single-scene scenarios, they often remain at the pixel or single semantic scale when sharing features across multiple tasks, lacking effective modeling of the animal's physical torso topology (skeleton features), and struggling to cope with structural distortions and feature loss caused by complex occlusion. When severe natural occlusion leads to a visual information vacuum, the conventional reasoning process amplifies environmental noise dramatically due to the lack of a strong prior reference in the skeletal spatial structure. This not only distorts the deep semantics of facial micro-expressions but also causes the complete collapse of the macroscopic topological structure of the limbs, making it impossible for the model to establish a physical connection between "micro-emotions" and "macro-actions." Ultimately, this leads to the complete failure of decoupling the behavioral branches and emotional branches, which are highly dependent on skeletal features.

[0004] Therefore, a multi-task recognition method is needed to improve the above-mentioned problems.

[0005] Invention / Invention Content To address the problems of existing technologies, this invention provides a multi-task method for emotion and behavior recognition based on mask location reconstruction and skeleton features, comprising the following steps: Step 1: Obtain the original image data of the target object, wherein the original image data contains at least a partially occluded area. Perform preprocessing and enhancement on the original image to obtain data. ; Step 2, transfer the data In the input spatial occlusion perception module, an occlusion mask prediction map corresponding to the occlusion area is generated. ; Step 3, predict the image based on the occlusion mask. , data The image is divided into visible and occluded blocks; the visible blocks are input into the mask feature reconstruction module, and semantic prior features extracted from the semantic prior extraction module are utilized. Feature reconstruction is performed on the occluded blocks to generate an occluded restored image. ; Step 4: Restore the unobstructed image In the input skeleton collaborative multi-task module, the skeleton collaborative multi-task module includes at least an emotion recognition branch and an action recognition branch, used to extract multi-scale spatial features and expression-related latent vectors, and to recover the image from the unoccluded image using a pose estimation network. Skeleton features are extracted and used as guiding signals to perform cross-modal fusion with the extracted multi-scale spatial features to obtain fused features, which are then concatenated with the extracted expression-related latent vectors to obtain a high-dimensional joint representation. Step 5: Input the high-dimensional joint representation into the emotion recognition branch and the behavior recognition branch respectively, and output the emotional state and behavior state of the target object simultaneously.

[0006] Furthermore, the partially occluded areas in the original image data include implicit semantic feature data and skeleton keypoint data; In step 1, the original dataset is preprocessed to obtain a multi-task dataset, specifically including: The original image is preprocessed, and adaptive cropping based on behavior bounding boxes is performed and expanded by 0.6 times. The implicit semantic features are preprocessed, and several implicit semantic vectors are precisely extracted and flattened from the pre-extracted features according to the position of the cropping box. Preprocess the skeleton key points by normalizing the relative coordinates of the skeleton key points according to the clipping offset and scaling ratio; During the training phase, random horizontal flipping, random color jitter, and random erasure enhancements are applied.

[0007] Furthermore, in step 4, the pose estimation network is a pose estimation network that has undergone domain-adaptive fine-tuning, and its fine-tuning process includes: Obtain the key point annotation data of the target object, and map the key point annotations to a Gaussian heatmap as the true value; The pre-trained pose estimation network is fine-tuned using the key point annotation data to minimize the error between the predicted heatmap and the actual heatmap. After fine-tuning convergence, the weights of the pose estimation network are solidified for subsequent feature extraction.

[0008] Furthermore, the spatial occlusion perception module in step 2 is an encoder-decoder network based on the Transformer architecture, including: Overlapping image patch embedding layer is used to divide the input image into multiple image patches and map them to serialized features; Multiple stacked Transformer encoding blocks, each including a self-attention module and a feedforward network, are used to extract context-aware serialized features; The block merging module is used to downsample and upscale the channels of the serialized features to obtain multi-scale contextual representations. The decoding module, including a multilayer perceptron and an upsampling layer, is used to decode the multi-scale context representation into an occlusion mask map that matches the resolution of the original image.

[0009] Furthermore, semantic prior features are used to reconstruct features of the occluded block, specifically including: The visible block is input into the encoder of the mask feature reconstruction module, and the structural features of the visible block are captured sequentially through the window multi-head self-attention mechanism, layer normalization and multilayer perceptron; An unoccluded reference image is input into the semantic prior extraction module to extract the global latent vector. The semantic prior features of the corresponding occluded blocks are extracted using a mask-guided marker selection operation. ; The structural features and the semantic prior features are concatenated along the sequence dimension and input into the decoder of the mask feature reconstruction module to generate an unobstructed restored image.

[0010] Furthermore, the emotion recognition branch is a spatial feature extraction module used to recover images from unoccluded images. Extracting multi-scale spatial features The multi-scale spatial features Including global context features and local details ; The behavior recognition branch is an addressing module used to retrieve latent vectors related to the expression of the target object from the latent vector space of the pre-trained visual feature extraction model. ; The skeleton collaborative multi-task module further includes: a fusion module, used to fuse the skeleton features, the multi-scale spatial features, and the expression-related latent vectors to generate a joint representation; The emotion recognition branch and behavior recognition branch are respectively connected to the output of the fusion module, and are used to decouple and classify the joint representation.

[0011] Furthermore, in step S4, the skeleton features are used as guiding signals to perform cross-modal fusion with the extracted multi-scale spatial features to obtain fused features, which are then concatenated with the extracted expression-related latent vectors to obtain a high-dimensional joint representation. Using a cross-attention mechanism, the skeleton features are used as query terms, and the multi-scale spatial features are used as key and value terms. Attention weights are calculated and attention-weighted cross-modal features are generated. The self-attention output of the cross-modal features and the skeleton features is concatenated, and then concatenated with the global context features and expression-related latent vectors in the channel dimension to obtain the fused features.

[0012] Furthermore, the spatial occlusion perception module, mask feature reconstruction module, and skeleton collaborative multi-task module are trained end-to-end using a joint loss function, which includes: Image restoration loss , pixel reconstruction loss Combating losses Perceptual feature constraints and semantic guidance loss The weighted sum is used to constrain the pixel-level and perceptual-level consistency between the restored image data and the real image data. Multi-task recognition loss Including emotion classification loss And behavioral classification loss At least one of the recognition task losses employs a class-balanced weighting strategy to reinforce the learning of the minority classes. The beneficial effects that this application can produce include: 1) The "spatial occlusion perception" mechanism provided in this application solves the problem of accurate localization in complex outdoor environments. Addressing the severe occlusion problem caused by leaves and branches in natural outdoor habitats, this invention designs a spatial occlusion perception module. Unlike traditional global feature extraction, which easily introduces environmental noise, this mechanism, through self-attention contextual interaction, can actively perceive and remove foreground interference, generating a high-precision abnormal occlusion mask, laying a precise visual localization foundation for subsequent feature repair. 2) This application constructs a "prior-driven" reconstruction paradigm, breaking through the information vacuum bottleneck under severe occlusion. To address the loss of deep facial and torso features caused by large-area occlusion, this invention innovatively proposes a feature reconstruction paradigm guided by prior knowledge. First, the semantic prior extraction module retrieves the latent semantic prior corresponding to the occluded region from the pre-trained model; then, the mask feature reconstruction module, guided by both the mask position and the semantic prior, performs high-fidelity targeted repair of visible features. This design completely eliminates the topological distortion caused by the "blind pixel filling" of traditional networks, successfully recovering discriminative fine-grained visual features; 3) This application designs a "skeleton collaboration" multi-task architecture, breaking through the barrier of cross-species topology extraction and achieving precise feature decoupling. Addressing the bottleneck of general pose networks easily generating domain shifts on wild animal targets, this invention first uses customized high-precision keypoint annotations to perform domain-adaptive fine-tuning on the pre-trained HRNet, enabling it to accurately grasp the specific trunk topology of the golden snub-nosed monkey. Based on this, a skeleton collaboration multi-task module is constructed, enabling deep cross-modal interaction between the extracted spatial multi-scale features and the fine-tuned, dedicated skeleton features. Using this dedicated skeleton topology as a powerful physical spatial reference not only effectively aligns heterogeneous features but also achieves precise decoupling of "micro-emotions" and "macro-actions" at the end of the shared representation layer, significantly improving the robustness of joint recognition. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the multi-task recognition method for emotion and behavior guided by mask location reconstruction and skeleton features in this application. Figure 2 This is the overall model architecture diagram of the multi-task recognition method of this application; Figure 3 This is a qualitative visual comparison diagram of the method proposed in this application and existing advanced models under different occlusion ratios; Figure 4 This is a visualization of the feature map evolution of the spatial occlusion perception module in this application; Figure 5 This is a graph showing the convergence curve of the loss function of the spatial occlusion perception module of the model in this application during the training period; Figure 6 This is a convergence curve of the reconstruction loss function of the mask feature reconstruction module of the model in this application during the training cycle; Figure 7 This is a convergence curve of the joint classification loss function of the skeleton-coordinated multi-task module of the model in this application during the training period. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1 This invention provides a multi-task method for emotion and behavior recognition based on mask location reconstruction and skeleton features, comprising the following steps: Step 1: Obtain the original image data of the target object, wherein the original image data contains at least a partially occluded area. Perform preprocessing and enhancement on the original image to obtain data. ; Step 2, transfer the data In the input spatial occlusion perception module (SOPM), an occlusion mask prediction map corresponding to the occlusion area is generated. ; Step 3, predict the image based on the occlusion mask. , data The data is divided into visible and occluded blocks. The visible blocks are input into the Mask Feature Reconstruction (MFRM) module, and semantic prior features extracted from the Semantic Prior Extraction (SPEM) module are utilized. Feature reconstruction is performed on the occluded blocks to generate an occluded restored image. ; Step 4: Restore the unobstructed image The input skeleton cooperative multitasking module (SSMM) includes at least an emotion recognition branch and an action recognition branch, used to extract multi-scale spatial features and expression-related latent vectors, and utilizes a pose estimation network (HRNet) to recover the image from the unoccluded state. Skeleton features are extracted and used as guiding signals to perform cross-modal fusion with the extracted multi-scale spatial features to obtain fused features, which are then concatenated with the extracted expression-related latent vectors to obtain a high-dimensional joint representation. Step 5: Input the high-dimensional joint representation into the emotion recognition branch and the behavior recognition branch respectively, and output the emotional state and behavior state of the target object simultaneously.

[0016] It should be noted that the model architecture proposed in this invention is as follows: Figure 2As shown, it comprises the following components: (a) Spatial Occlusion Perception Module (SOPM) actively extracts contextual anomaly features using a visual Transformer and self-attention mechanism, ultimately decoding and outputting a high-precision occlusion region mask. (b) Semantic Prior Extraction Module (SPEM) retrieves latent semantic priors corresponding to the occluded region from the pre-trained model, providing global knowledge guidance for feature repair. (c) Mask Feature Reconstruction Module (MFRM) performs high-fidelity directional reconstruction of fine-grained features of visible blocks under the dual guidance of mask position and prior semantics. (d) Skeleton Collaborative Multitask Module (SSMM) extracts spatial features and the dedicated skeleton features output by the fine-tuned HRNet, performs modality alignment using a cross-attention mechanism, and after decoupling in the shared representation layer, synchronously outputs the predicted distributions of the target's emotion category and behavior category via a multilayer perceptron.

[0017] Furthermore, the partially occluded areas in the original image data include implicit semantic feature data and skeleton keypoint data; In step 1, the original dataset is preprocessed to obtain a multi-task dataset, specifically including: The original image is preprocessed, and adaptive cropping based on behavior bounding boxes is performed and expanded by 0.6 times. The implicit semantic features are preprocessed, and several implicit semantic vectors are precisely extracted and flattened from the pre-extracted features according to the position of the cropping box. Preprocess the skeleton key points by normalizing the relative coordinates of the skeleton key points according to the clipping offset and scaling ratio; During the training phase, random horizontal flipping, random color jitter, and random erase enhancement are applied. It should be noted that the original image is preprocessed using an adaptive cropping strategy based on behavioral bounding boxes. The bounding box is generated from key points or manual annotations and expanded by 0.6 times to preserve contextual information. Then, it is uniformly scaled to a specific pixel resolution using bilinear interpolation. The implicit semantic features are preprocessed, and 78 implicit semantic vectors (tokens) are precisely extracted and flattened from the pre-extracted features according to the position of the cropping box. The skeleton key points are preprocessed by normalizing the relative coordinates of the skeleton key points according to the clipping offset and scaling ratio to eliminate spatial translation error. During the training phase, random horizontal flipping, random color jitter, and random erasure enhancements were applied to simulate additional occlusion noise, and standard statistics were used for channel-by-channel normalization.

[0018] Furthermore, in step 4, the pose estimation network is a pose estimation network that has undergone domain-adaptive fine-tuning, and its fine-tuning process includes: Obtain the key point annotation data of the target object, and map the key point annotations to a Gaussian heatmap as the true value; The pre-trained pose estimation network is fine-tuned using the key point annotation data to minimize the error between the predicted heatmap and the actual heatmap. After fine-tuning convergence, the weights of the pose estimation network are solidified for subsequent feature extraction. It should be noted that S101 maps the actual coordinates of the 17 labeled skeleton key points to a set of two-dimensional Gaussian heatmaps, corresponding to the... The true Gaussian heat map of the key points is recorded as follows: ; S102, input the field image with key point annotations into the pre-trained pose backbone network, and obtain its output. Predictive heatmap of key points A fine-tuning loss function is constructed using mean square error. Its expression is: (1) in, Represents the total number of key points in the skeleton and , Represents the L2 norm; S103, Minimize the fine-tuning loss function through backpropagation. The parameters of the pre-trained pose backbone network are iteratively updated. After the fine-tuning process converges, all weights of the network are solidified and used as features for the dedicated skeleton feature extraction network of the subsequent skeleton collaborative multi-task module (SSMM).

[0019] Furthermore, the spatial occlusion perception module in step 2 is an encoder-decoder network based on the Transformer architecture, including: Overlapping image patch embedding layer is used to divide the input image into multiple image patches and map them to serialized features; Multiple stacked Transformer encoding blocks, each including a self-attention module and a feedforward network, are used to extract context-aware serialized features; The block merging module is used to downsample and upscale the channels of the serialized features to obtain multi-scale contextual representations. The decoding module, including a multilayer perceptron and an upsampling layer, is used to decode the multi-scale context representation into an occlusion mask map that matches the resolution of the original image.

[0020] Specifically, in S201, serialization features are extracted through Transformer coded blocks, and the forward computation process is expressed as follows:

[0021] (2) in, and They represent the first The input and output characteristics of the layer Representation layer normalization; S202, The decoding module generates a mask for the occluded area. The calculation process is expressed as follows:

[0022] (3) in, For multi-scale context representation after block merging, The mask features are after upsampling. for Activation function.

[0023] Furthermore, in step 3, the prediction map is based on the occlusion mask. , data The process of dividing the block into visible and occluded blocks is as follows: S301, Divide the input image into a set of non-overlapping image patches. Based on occlusion mask prediction graph Set threshold Image patches are explicitly decoupled into a set of visible patches. and occlusion block set The expression is:

[0024] (4) in, Indicates the first The average mask probability of each image patch region.

[0025] Furthermore, semantic prior features are used to reconstruct features of the occluded block, specifically including: The visible block is input into the encoder of the mask feature reconstruction module, and the structural features of the visible block are captured sequentially through the window multi-head self-attention mechanism, layer normalization and multilayer perceptron; An unoccluded reference image is input into the semantic prior extraction module to extract the global latent vector. The semantic prior features of the corresponding occluded blocks are extracted using a mask-guided marker selection operation. ; The structural features and the semantic prior features are concatenated along the sequence dimension and input into the decoder of the mask feature reconstruction module to generate an unobstructed restored image. Specifically, S302 will make the visible block The encoder in the input mask feature reconstruction module (MFRM) extracts features, which are then captured through window multi-head self-attention (W-MSA), layer normalization, and multilayer perceptron. The expression is: (5) S303, input the unoccluded reference image into the semantic prior extraction module (SPEM) to extract the global latent vector. The semantic prior features of the corresponding occluded blocks are extracted using a mask-guided marker selection operation. The expression is: (6) S304, the decoder of the mask feature reconstruction module will use structural features With semantic prior features By stitching along the sequence dimensions, an occlusion-free restored image is generated. The expression is: (7) Furthermore, the emotion recognition branch is a spatial feature extraction module used to recover images from unoccluded images. Extracting multi-scale spatial features The multi-scale spatial features Including global context features and local details ; The behavior recognition branch is an addressing module used to retrieve latent vectors related to the expression of the target object from the latent vector space of the pre-trained visual feature extraction model (ViT, VisionTransformer). ; The skeleton collaborative multi-task module further includes: a fusion module, used to fuse the skeleton features, the multi-scale spatial features, and the expression-related latent vectors to generate a joint representation; The emotion recognition branch and the behavior recognition branch are respectively connected to the output of the fusion module, and are used to decouple and classify the joint representation; Furthermore, in step S4, the skeleton features are used as guiding signals to perform cross-modal fusion with the extracted multi-scale spatial features to obtain fused features, which are then concatenated with the extracted expression-related latent vectors to obtain a high-dimensional joint representation. Using a cross-attention mechanism, the skeleton features are used as query terms, and the multi-scale spatial features are used as key and value terms. Attention weights are calculated and attention-weighted cross-modal features are generated. The self-attention output of the cross-modal features and the skeleton features is concatenated, and then concatenated with the global context features and expression-related latent vectors in the channel dimension to obtain the fused features; Specifically, S401, expression-related latent vectors The expression is: (8) in, For addressing mechanism, The set of visual latent semantic features extracted by the semantic prior extraction module (SPEM); S402, Skeletal Features Multi-scale spatial features Cross-modal features generated via cross-attention mechanism The expression is: (9) in, This is the weight matrix. The feature dimension is constant; S403, Fusion Features The expression for concatenating the self-attention output of the skeleton features with cross-modal features is: (10) in, This represents the self-attention computation process. Indicates a splicing operation; S404, the network receives features of different granularities generated by multiple branches, and combines global context features. Facial expression related latent vectors and fusion features By concatenating the data along the channel dimension, we obtain the final high-dimensional joint representation. The expression is: (11) in, This represents the self-attention computation process. Indicates a splicing operation; Furthermore, the spatial occlusion perception module, mask feature reconstruction module, and skeleton collaborative multi-task module are trained end-to-end using a joint loss function, which includes: Image restoration loss Used for approximately pixel reconstruction loss Combating losses Perceptual feature constraints and semantic guidance loss The weighted sum composition of the bundle-restored image data and the pixel-level and perceptual-level consistency between the real image data and the bundle-restored image data; Multi-task recognition loss Including emotion classification loss And behavioral classification loss At least one of the recognition task losses employs a class-balanced weighting strategy to reinforce the learning of the minority classes; Specifically, image restoration loss With multi-task recognition loss ; S501, Image Restoration Loss Pixel reconstruction loss Combating losses Perceptual feature constraints and semantic guidance loss The weighted sum composition is expressed as: (12) in, , For adaptive weight parameters; S502, Multi-task recognition loss Classifying loss by emotion And behavioral classification loss with class boundary weight scaling The weighted sum is expressed as: (13) in, For the behavioral task weight parameters, the emotion classification loss uses label smoothing cross-entropy to reduce model overfitting, while the behavioral classification loss introduces Label-Distribution-AwareMargin (LDAM) loss to enhance the generalization ability of the minority class.

[0026] The target objects include wild animals, domestic animals, humans, or robots; the emotional state and behavioral state can be any two of the physiological state or the interaction state. Example 1: Emotion and Behavior Recognition in Golden Monkeys Video data of golden snub-nosed monkeys' natural habitats was collected using high-definition cameras deployed in the wild. The constructed field multi-task dataset GM-MTL contains 3155 images, rigorously divided into 2847 training images and 308 test images. Image frames containing individual golden snub-nosed monkeys were extracted. Images were labeled, including: coordinates of 17 skeletal keypoints, emotion labels (friendly, aggressive, neutral), and behavior labels (grooming, resting, fighting, walking). An adaptive cropping strategy was adopted, expanding the bounding box of the keypoints by 0.6 times to preserve contextual information, and uniformly scaling the images to 256×256 pixels. Table 1 shows the corresponding semantic label information of various ecological characteristics of golden snub-nosed monkeys, covering annotation examples of different emotion and behavior category systems.

[0027] Table 1. Examples of fine-grained text annotations for emotion and behavior in the GM-MTL multi-task dataset for golden snub-nosed monkeys. ; The Spatial Occlusion Awareness Module (SOPM) adopts a Transformer-based encoder-decoder architecture. The encoder consists of 12 Transformer blocks, each with 8 multi-head self-attention heads and a hidden layer dimension of 768. The decoder consists of 3 upsampling layers and 2 convolutional layers, ultimately outputting a mask map with the same resolution as the input image through a sigmoid function. During training, manually annotated occlusion regions are used as supervision signals, and a binary cross-entropy loss function is employed for optimization. To investigate the impact of using the SOPM on experimental results, the use of different occlusion mask ratios, and the impact of preserving the backbone network on model repair performance, this invention sets up different models for comparative evaluation from three perspectives, as shown in Table 4 below.

[0028] Here, Backbone indicates the method of retaining only the backbone network; w / oSOPM indicates the method of removing the occlusion sensing network; OR-MTF(Ours) indicates that this paper adopts the method of full SOPM.

[0029] Table 4. Quantitative results of ablation experiments in the image restoration module. ; To explore the effect of the Spatial Occlusion Awareness Module (SOPM) on the model, Table 4 compares the Backbone, w / oSOPM, and OR-MTF (Ours) models. Compared to the input methods that only retain the Backbone or remove the awareness module (w / oSOPM), the image inpainting effect using the full SOPM method is significantly improved across all occlusion ratios. At 0-10% mask ratios, OR-MTF's PSNR jumps to 36.4693, and its SSIM reaches 0.9935, demonstrating a significant advantage over w / oSOPM. This shows that SOPM has a stronger ability to specifically understand the occlusion context and express anomalous regions, achieving lower LPIPS and L1 error despite an increased number of parameters.

[0030] HRNet-W32, pre-trained on the COCO dataset, was used as the base pose network. Using 17 labeled golden snub-nosed monkey skeletal keypoints, Gaussian heatmaps with a standard deviation of 2 were generated as ground truth for the keypoint coordinates. During fine-tuning, the parameters of the first two stages were frozen, and only the last two stages were fine-tuned. The learning rate was set to 1e-4, and convergence occurred after 50 epochs. In specific implementation, such as Figure 4 As shown, to explore the evolution effect of the feature map extracted by the spatial occlusion perception module: after the input image is processed by the encoder and attention mechanism, the network gradually and accurately focuses the high-response region from the background to the edge of the occlusion, and outputs a high-precision binary mask map after decoding. This shows that SOPM has a strong ability to perceive contextual anomalies, avoiding mistaking natural occlusions (such as leaves) for the golden monkey's own feature texture.

[0031] In specific implementation, the mask feature reconstruction module adopts a mask autoencoder (MAE) architecture. The encoder receives image patches of the visible region and extracts structural features through 12 Transformer blocks. The semantic prior extraction module uses the ViT-B / 16 model pre-trained on ImageNet. For each region to be repaired, it retrieves the feature vector corresponding to the occlusion position from the ViT feature space as a semantic prior. The decoder concatenates the structural features and semantic prior and reconstructs the complete image through 8 Transformer blocks; for example... Figure 2As shown, to explore the effect of the Mask Feature Reconstruction Module (MFRM) on model image reconstruction: qualitative results show that introducing a complete MFRM significantly improves the structural fidelity of the generated image. When comparing methods such as DeepFillv2 and PConv, other methods tend to produce obvious structural distortions and blur artifacts at occlusion boundaries, failing to close key contours of the eyes or mouth. However, the unclamped images generated by the proposed method largely match the original golden snub-nosed monkey's fur texture, and the facial feature completion in severely occluded areas also conforms to objective biological laws. This indicates that the model can better understand and utilize mask prior information, thereby generating high-quality images that are more consistent with real-world natural scenes. This matching ability is attributed to the model's accurate understanding and effective fusion of the occlusion context, resulting in a more structured and holistic final image presentation.

[0032] To explore the effect of the semantic prior extraction module (SPEM) on the model: like Figure 3 The user research subjective rating statistics show that the model using SPEM to extract priors significantly improved the "restoration integrity," "biological consistency," and "semantic fidelity" of the generated images compared to the model without priors. Pre-trained latent semantic features can more accurately guide the network to understand the global semantics of occluded regions, effectively avoiding blind pixel filling.

[0033] Furthermore, to evaluate the cross-domain generalization ability of MFRM in this study, comparative experiments were conducted on the CelebA-HQ public dataset. The results show that even without hard-coding the network for face data, this module still achieved the highest PSNR (37.5214) and SSIM (0.9951) scores under 0-10% mask occlusion. This indicates that MFRM influences model behavior by precisely controlling the mask inference process, effectively utilizing latent semantic information, resulting in a better match between the generated image and the missing region. This highlights the importance of the model's accurate understanding and effective fusion of structural information from different biological domains.

[0034] The skeleton-based collaborative multitasking module includes three parallel branches: The emotion recognition branch is a spatial feature extraction module: it uses ResNet-50 as the backbone network to extract multi-scale spatial features and obtains 2048-dimensional global features through global average pooling.

[0035] The behavior recognition branch, the addressing module, uses a learnable query vector to retrieve expression-related feature vectors from the ViT feature space through an attention mechanism. The skeleton feature extraction branch uses a fine-tuned HRNet to extract heatmap features from 17 key points and maps them to a 512-dimensional skeleton feature vector through a fully connected layer. The fusion module employs a cross-attention mechanism, using skeleton features as the query (Q) and spatial and skeleton features as keys (K) and values ​​(V) to compute cross-modal attention features. The self-attention outputs of the cross-modal features and skeleton features are concatenated, and then concatenated with global and expression features along the channel dimension to obtain the final joint representation.

[0036] To investigate the impact of the internal structural components of the Skeleton Collaborative Multitask Module (SSMM) on the performance of multitask joint classification, this invention set up several model ablation experiments. As shown in Table 5, the effects of changing the network structure (removing implicit vectors, removing the occlusion perception and repair module) on model performance are explored.

[0037] Table 5 Comparison of Ablation Experiment Results of SSMM Network Structure Components ; Comparing the impact of different structural components on the final feature decoupling and classification, the complete architecture presented in this paper achieves the best OR-MTF generation effect and classification accuracy.

[0038] 1) As can be seen from the data analysis in Table 5, after removing the implicit feature vector (w / olatent-vector), the model's accuracy in emotion and behavior suffers a catastrophic decline due to the lack of high-dimensional shared priors across tasks, plummeting to 43.44% and 28.44%, respectively. This indicates that introducing pre-trained ViT latent vectors through the addressing mechanism is crucial for the model to understand and generate local fine-grained expressions.

[0039] 2) When the occlusion perception and repair modules (w / o SOPM & MFRM) are removed, the model degenerates into directly extracting multimodal features from severely occluded images, with emotion and behavior accuracy of only 64.29% and 65.26%, respectively. This proves that the asymmetric cross-attention architecture must be built on a high-fidelity de-occluded visual foundation; otherwise, environmental noise will spread along the attention mechanism, leading to decoupling failure.

[0040] In the ablation experiments of the multi-task joint loss function in this invention, the main focus is on exploring changes to the weights of the multi-task loss function. To overcome the "seesaw effect" and its impact on the model's final recognition accuracy, Table 6 below discusses methods such as removing single-task constraints, using proportional weights, and using weights with different biases.

[0041] Table 6 Comparison Results of Weight Ablation Experiments Using Multi-Task Loss Functions ; Comparison of the above loss weight ablation experiments revealed that the OR-MTF method using a targeted weighting strategy outperforms other uniform or extreme loss weights in classification performance.

[0042] 1) and removing single-task constraints ( Compared to other methods, our proposed method achieves a 55.52% improvement in emotion classification accuracy. This is because the multi-task joint learning layer heavily relies on the guidance of dual supervision signals; once the gradient backpropagation of the emotion branch is lost, the network representation layer will become completely biased towards macroscopic skeleton actions, leading to the loss of deep semantic information of subtle facial expressions in the feature space. This is directly reflected in the collapse of the expressive power of the shared layer.

[0043] 2) In chart comparisons, compared to using equal weights ( Compared to conventional settings, our proposed method improves sentiment accuracy by 36.04% and behavioral accuracy by 19.15%. This is because wild golden monkey behavioral data exhibits a severe long-tail distribution, and equal weights can cause the model to overfit the majority class during joint optimization. Our method utilizes LDAM loss to dynamically penalize the decision boundary of behavioral categories and assigns optimal multi-task joint weights, covering both local acuity and global consistency in the feature fusion process. This diverse combination of losses and weights dynamically adapts to complex interference in the wild, allowing the model to more comprehensively decouple multimodal features, thereby generating the most accurate joint classification predictions.

[0044] In practice, rigorous empirical studies were conducted to compare the performance of different models under different occlusion ratios and complex outdoor scenes. Quantitative indicators (such as PSNR, SSIM, LPIPS, classification Acc and F1 score) and qualitative analysis were used to explore the differences and semantic consistency between generated images and real images, and to compare them with other influence models. The results are shown in Tables 7 and 8.

[0045] Table 7 Comparison with state-of-the-art image inpainting control methods at different mask ratios ; ; Table 8 compares the recognition performance with state-of-the-art multi-task learning classification methods. ; In practice, rigorous empirical studies were conducted to compare the performance of different models under different occlusion ratios and complex field scenarios. Quantitative indicators and qualitative analysis were used to explore the differences and semantic consistency between generated images and real images, and the results were compared with other influence models. The results are shown in Tables 7 and 8.

[0046] LaMa, Global&Local, DeepFillv1, DeepFillv2, PConv, and AOT-GAN are common image inpainting and diffusion control methods. LaMa is a method with a large receptive field that introduces Fast Fourier Convolution (FFC). DeepFillv2 is a control method that introduces gated convolution, which can improve the model's generation effect on irregular boundaries. AOT-GAN aims to achieve efficient and high-fidelity image inpainting capabilities by aggregating context transformations, and is suitable for various masking and occlusion tasks. HPS, Cross_stitch, MMOE, CGC, PLE, and MTAN are common multi-task recognition methods. MMOE introduces a multi-gated hybrid expert mechanism to dynamically allocate feature weights; PLE is a cascaded evolution of the CGC model, which can effectively overcome the "seesaw effect" and "negative transfer" problems commonly found in multi-task learning; ST-GCN is a skeleton topology behavior recognition method that relies on learning spatial temporal graph convolution.

[0047] like Figure 2 and Figures 5-7 The comparative results show that our method outperforms other methods in both of these key metrics, demonstrating a significant advantage. Our model fully integrates a Spatial Occlusion Awareness Module (SOPM) with active anomaly detection capabilities during training, and optimizes the model and provides fine-grained mask prior guidance through a Mask Feature Reconstruction Module (MFRM), resulting in excellent performance in terms of texture style and semantic consistency of multi-task features in the generated golden monkey images. Furthermore, we use adaptive weights to calculate the total loss by combining image reconstruction losses (such as pixel reconstruction loss and adversarial loss) and recognition losses from the multi-task classification model (such as label smoothing cross-entropy and LDAM loss), and iteratively update the parameters of the multi-task network based on the total loss.

[0048] It is worth noting that: 1) This invention realizes a multi-task recognition method for emotion and behavior based on mask location reconstruction and skeleton feature guidance. It uses high-fidelity natural habitat images and multi-dimensional fine-grained semantic annotation information as training data. By micro-adapting the pre-trained pose network through domain adaptation, it solves the shortcomings of existing models in handling the small domain task of wild golden monkeys, such as lack of high-quality unoccluded priors and lack of fine-grained joint skeleton annotation. The weights of the pre-trained ViT model are frozen and a semantic prior extraction module (SPEM), a mask feature reconstruction module (MFRM), and a skeleton collaborative multi-task module (SSMM) are introduced to extract the latent prior semantics as additional guiding information to guide the directional reconstruction of missing images. This achieves precise control over the cross-modal attention interaction behavior of the network and further optimizes the classification process under the deep alignment mechanism to obtain image feature representation that integrates global context and accurate skeleton topology. This solves the problems of existing models, such as lack of fine-grained guidance, insufficient control over details, and inability to generate ideal features under large-area occlusion. 2) This invention is deeply integrated with the background of wildlife ecology conservation. Using exclusive skeletal features as the core physical reference, it effectively expresses the complex biological characteristics of wild golden monkeys, such as trunk topology, local facial details, interactive behavior, micro-expression state information, and group action logic. It achieves more refined and higher quality feature generation under complex environmental interference, effectively improves the accuracy of model generation and prediction distribution and multi-task collaboration, and enables the model to better learn and capture the characteristic proportion relationship and individual posture changes of wild animals. Thus, it achieves model generation and classification that is more in line with the characteristics of high-precision intelligent monitoring of wild animals.

[0049] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A multi-task method for emotion and behavior recognition based on mask location reconstruction and skeleton features, characterized in that, Includes the following steps: Step 1: Obtain the original image data of the target object, wherein the original image data contains at least a partially occluded area. Perform preprocessing and enhancement on the original image to obtain data. ; Step 2, transfer the data In the input spatial occlusion perception module, an occlusion mask prediction map corresponding to the occlusion area is generated. ; Step 3, predict the image based on the occlusion mask. , data The image is divided into visible and occluded blocks; the visible blocks are input into the mask feature reconstruction module, and semantic prior features extracted from the semantic prior extraction module are utilized. Feature reconstruction is performed on the occluded blocks to generate an occluded restored image. ; Step 4: Restore the unobstructed image In the input skeleton collaborative multi-task module, the skeleton collaborative multi-task module includes at least an emotion recognition branch and an action recognition branch, used to extract multi-scale spatial features and expression-related latent vectors, and to recover the image from the unoccluded image using a pose estimation network. Skeleton features are extracted and used as guiding signals to perform cross-modal fusion with the extracted multi-scale spatial features to obtain fused features, which are then concatenated with the extracted expression-related latent vectors to obtain a high-dimensional joint representation. Step 5: Input the high-dimensional joint representation into the emotion recognition branch and the behavior recognition branch respectively, and output the emotional state and behavior state of the target object simultaneously.

2. The multi-task recognition method according to claim 1, characterized in that, The partially occluded areas in the original image data include implicit semantic feature data and skeleton keypoint data; In step 1, the original dataset is preprocessed to obtain a multi-task dataset, specifically including: The original image is preprocessed, and adaptive cropping based on behavior bounding boxes is performed and expanded by 0.6 times. The implicit semantic features are preprocessed, and several implicit semantic vectors are precisely extracted and flattened from the pre-extracted features according to the position of the cropping box. Preprocess the skeleton key points by normalizing the relative coordinates of the skeleton key points according to the clipping offset and scaling ratio; During the training phase, random horizontal flipping, random color jitter, and random erasure enhancements are applied.

3. The multi-task recognition method according to claim 1, characterized in that, In step 4, the pose estimation network is a pose estimation network that has undergone domain adaptive fine-tuning, and its fine-tuning process includes: Obtain the key point annotation data of the target object, and map the key point annotations to a Gaussian heatmap as the true value; The pre-trained pose estimation network is fine-tuned using the key point annotation data to minimize the error between the predicted heatmap and the actual heatmap. After fine-tuning convergence, the weights of the pose estimation network are solidified for subsequent feature extraction.

4. The multi-task recognition method according to claim 1, characterized in that, In step 2, the spatial occlusion perception module is an encoder-decoder network based on the Transformer architecture, including: Overlapping image patch embedding layer is used to divide the input image into multiple image patches and map them to serialized features; Multiple stacked Transformer encoding blocks, each including a self-attention module and a feedforward network, are used to extract context-aware serialized features; The block merging module is used to downsample and upscale the channels of the serialized features to obtain multi-scale contextual representations. The decoding module, including a multilayer perceptron and an upsampling layer, is used to decode the multi-scale context representation into an occlusion mask map that matches the resolution of the original image.

5. The multi-task recognition method according to claim 1, characterized in that, Feature reconstruction of occluded blocks using semantic prior features includes: The visible block is input into the encoder of the mask feature reconstruction module, and the structural features of the visible block are captured sequentially through the window multi-head self-attention mechanism, layer normalization and multilayer perceptron; An unoccluded reference image is input into the semantic prior extraction module to extract the global latent vector. The semantic prior features of the corresponding occluded blocks are extracted using a mask-guided marker selection operation. ; The structural features and the semantic prior features are concatenated along the sequence dimension and input into the decoder of the mask feature reconstruction module to generate an unobstructed restored image.

6. The multi-task recognition method according to claim 1, characterized in that, The emotion recognition branch is a spatial feature extraction module used to recover images from unoccluded images. Extracting multi-scale spatial features The multi-scale spatial features Including global context features and local details ; The behavior recognition branch is an addressing module used to retrieve latent vectors related to the expression of the target object from the latent vector space of the pre-trained visual feature extraction model. ; The skeleton collaborative multi-task module further includes: a fusion module, used to fuse the skeleton features, the multi-scale spatial features, and the expression-related latent vectors to generate a joint representation; The emotion recognition branch and behavior recognition branch are respectively connected to the output of the fusion module, and are used to decouple and classify the joint representation.

7. The multi-task recognition method according to claim 6, characterized in that, In step S4, the skeleton features are used as guiding signals to perform cross-modal fusion with the extracted multi-scale spatial features to obtain fused features, which are then concatenated with the extracted expression-related latent vectors to obtain a high-dimensional joint representation. Using a cross-attention mechanism, the skeleton features are used as query terms, and the multi-scale spatial features are used as key and value terms. Attention weights are calculated and attention-weighted cross-modal features are generated. The self-attention output of the cross-modal features and the skeleton features is concatenated, and then concatenated with the global context features and expression-related latent vectors in the channel dimension to obtain the fused features.

8. The multi-task recognition method according to claim 1, characterized in that, The spatial occlusion perception module, mask feature reconstruction module, and skeleton collaborative multi-task module are trained end-to-end using a joint loss function, which includes: Image restoration loss , pixel reconstruction loss Combating losses Perceptual feature constraints and semantic guidance loss The weighted sum is used to constrain the pixel-level and perceptual-level consistency between the restored image data and the real image data. Multi-task recognition loss Including emotion classification loss And behavioral classification loss At least one of the recognition task losses employs a class-balanced weighting strategy to reinforce the learning of the minority classes.

Images