Attention mechanism based visual-text cross-modal giant panda behavior recognition method

Abstract

The application provides a visual-text cross-modal panda behavior recognition method based on an attention mechanism, relates to the technical field of the attention mechanism, and first inputs a multi-modal data set into an initial model, extracts panda behavior features, introduces a customized cross-modal attention mechanism to realize feature alignment, strengthens the interaction depth of visual and text features, and breaks through the bottleneck of insufficient semantic fusion; then, based on the features, a video frame sequence-text description pair sample is constructed, bidirectional matching learning is carried out in a unified embedding space through a cross-modal representation network, parameters are optimized with a symmetric cross-entropy loss, the model is solidified in combination with a verification and early stopping mechanism, video timing information is captured, and the problem of incomplete dynamic representation of behavior is solved; finally, the visual part of target panda behavior data is optimized, the preprocessing strategy is adjusted according to the quality parameters, the optimization result is used to assist the attention mechanism to focus on key features, the current situation that preprocessing has no unified standard and cannot be fed back is improved, and finally, the target behavior category is accurately recognized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of attention mechanism, in particular to a visual-text cross-modal giant panda behavior recognition method based on an attention mechanism. BACKGROUND

[0002] The current giant panda behavior recognition process mainly includes the following steps: first, data collection, through high-definition infrared cameras, sensors and other equipment, the images or videos of the giant panda's eating, resting and other behaviors in the captive or wild environment are captured, and the environmental information is recorded synchronously; second, data preprocessing, the original data is denoised and standardized, and the behavior categories are labeled by professional personnel to construct a structured data set; third, model construction and training, convolutional neural network, long short-term memory network and other algorithms are selected, the model parameters are iteratively optimized after the data set is divided, and the structure is adjusted to improve the classification ability; then, the model is evaluated by the test set, and the model is optimized according to the accuracy and other indicators, and the data is supplemented or the method is improved for the behaviors with high recognition error; finally, the application and dynamic update are deployed, the model is used for real-time monitoring, the behavior report and early warning are generated, and the model is updated regularly with new data to adapt to the change of behavior mode.

[0003] For example, the Chinese invention patent with the publication number CN111967358B provides a neural network gait recognition method based on an attention mechanism, which includes the following steps: dividing the training set and the test set from the reference data set; pre-training the network of the gait extraction model without embedding the attention mechanism, so that the model has good adaptability to human gait; embedding the time domain and spatial domain attention mechanism module in the network, and loading the pre-trained network model parameters; reusing the data set to train the gait recognition feature extraction model based on the attention mechanism.

[0004] For example, the Chinese invention patent with the publication number CN112257572A provides a behavior recognition method based on a self-attention mechanism, which adopts a key frame target position prediction and continuous frame action category prediction module based on a multi-angle attention mechanism, which can realize target positioning function while completing continuous frame action detection. The key frame target position prediction and continuous frame action category prediction module based on the multi-angle attention mechanism replaces the 3D convolution network, improves the parallel computing capability of the model on the GPU, and avoids the problem of weak compatibility of the model conversion or deployment due to the 3D convolution in different deep learning frameworks.

[0005] The current technology has the following technical problems:

[0006] The existing cross-modal learning still faces multiple challenges when CLIP (contrastive language-image pre-training model) is directly applied to video behavior recognition by jointly aligning image features and text features: firstly, the capture of video temporal information is insufficient, making it difficult to fully represent the dynamic process of behavior; secondly, the interaction depth and interaction relevance of visual features and text features are insufficient, failing to achieve effective fusion at the semantic level; in addition, the current behavior recognition preprocessing link lacks a unified standard, and the preprocessing results cannot effectively feedback to the attention mechanism, further affecting the focusing ability of the model on key behavior features. SUMMARY

[0007] In order to solve the technical problems of the prior art that the interaction depth and interaction relevance of visual features and text features are insufficient and the preprocessing results cannot effectively feedback to the attention mechanism, the embodiment of the present application provides a visual-text cross-modal panda behavior recognition method based on an attention mechanism. The technical solution is as follows:

[0008] A visual-text cross-modal panda behavior recognition method based on an attention mechanism is provided, which comprises the following steps: step one, inputting a multi-modal data set into an initial behavior recognition model, extracting feature data of panda behavior through a Transformer network, and introducing a cross-modal attention mechanism to align the feature data; step two, based on the feature data of panda behavior, constructing paired samples of video frame sequences-text descriptions, conducting bidirectional matching learning in a unified embedding space by a cross-modal representation network, and updating parameters by an optimizer with symmetric cross-entropy loss as the main loss function, requiring paired samples to approach each other and unpaired samples to separate from each other, and through verification and early stopping mechanism constraints, when the training end flag is monitored, the obtained behavior recognition model parameters and inference process are solidified, so that the initial behavior recognition model is marked as a behavior recognition model; step three, obtaining target panda behavior data, and performing data optimization processing on the visual data in the target panda behavior data, inputting the optimized target panda behavior data into the behavior recognition model, and analyzing and recognizing the target panda behavior category through the behavior recognition model.

[0009] The technical solution provided by the embodiment of the present application has at least the following beneficial effects:

[0010] (1) The visual-text cross-modal giant panda behavior recognition method based on the attention mechanism, first inputs the multi-modal data set into the initial model, extracts the giant panda behavior features through the Transformer network, introduces the customized cross-modal attention mechanism to realize feature alignment, strengthens the interaction depth of visual and text features, and breaks through the bottleneck of insufficient semantic fusion;Then, based on the features, the video frame sequence-text description pair sample is constructed, the cross-modal representation network is used to carry out bidirectional matching learning in the unified embedding space, the parameters are optimized by the symmetric cross-entropy loss, the model is solidified by the verification and early stopping mechanism, the video timing information is fully captured, and the problem of incomplete dynamic representation of behavior is solved;Finally, the visual part of the target giant panda behavior data is optimized, the preprocessing strategy is adjusted according to the quality parameter, the optimization result is used to assist the attention mechanism to focus on the key features, the current situation that the preprocessing has no unified standard and cannot be fed back is improved, and finally the target behavior category is accurately recognized, which provides reliable technical support for giant panda monitoring research.

[0011] (2) The invention designs a semantic anchor point driven by a behavior label, introduces a category label as an explicit anchor point, and carries out three-way interaction of category embedding, video global feature and text global feature. The cross-modal feature fusion and category discrimination effect are significantly optimized. First, the L2 norm normalization is performed on the three, to ensure that the features interact in a unified scale, and to avoid affecting the fusion accuracy due to the difference in magnitude;Then, the text feature is used to guide the video feature attention weighting, and the accurate attention weight is generated along the time dimension, effectively aggregating the key timing information of the video, solving the problem of insufficient utilization of video dynamic features in traditional fusion. Secondly, the category embedding is expanded into a homomorphic tensor, so that the category, video and text interact directly in the same space position, and the semantic correlation of the three is strengthened, breaking through the problem of missing semantic anchor point caused by relying only on video-text bidirectional interaction;Finally, the original category score is obtained through element-level multiplication and word dimension average, providing high-quality discrimination basis for subsequent loss calculation and reasoning, greatly improving the accuracy and reliability of behavior recognition.

[0012] (3) The invention realizes dynamic regulation of data cleaning through hierarchical comparison of quality representation coefficients and limit values, avoids the insufficient adaptation of unified processes to different quality data, ensures efficient processing of high-quality data and accurate optimization of low-quality data;Introducing the quality representation coefficient increment and the double judgment of data normal conditions, combined with the limit execution frequency constraint, not only maximizes the repair effect of low-quality data through parameter adjustment, but also avoids excessive optimization consumption, while still ensuring data input and triggering model initialization when repair is invalid, balancing recognition efficiency and accuracy;More quality representation coefficient deviation and visual Transformer encoder temperature coefficient linkage make the model attention weight dynamically focus on key features according to data quality, effectively compensate for the feature defects of low-quality visual data, and finally improve the robustness and accuracy of behavior recognition, adapt to the complex and changeable visual data scene in the field giant panda behavior monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0014] Figure 1 is a visual-text cross-modal giant panda behavior recognition method flowchart based on an attention mechanism provided by an embodiment of the present application.

[0015] Figure 2 is a target giant panda data optimization flowchart provided by an embodiment of the present application.

[0016] Figure 3 is a determination initialization behavior recognition model flowchart provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the present application will be described below with reference to the drawings.

[0018] In the embodiments of the present application, the words such as "example", "for example" and the like are used to represent an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be either one of the two.

[0019] In order to make the technical problems, technical solutions and advantages of the present application more clear, the following will be described in detail with reference to the drawings and specific embodiments.

[0020] An embodiment of the present application provides a visual-text cross-modal giant panda behavior recognition method based on an attention mechanism. As shown in the visual-text cross-modal giant panda behavior recognition method flowchart based on an attention mechanism, the processing flow of the method can include the following steps: Figure 1

[0021] Step one, input the multi-modal data set into the initial behavior recognition model, extract the feature data of the giant panda behavior through the Transformer network, and introduce the cross-modal attention mechanism to align the feature data.

[0022] ​The training of the behavior recognition model is based on a multi-modal dataset, and the input in the training stage includes the visual features of the pre-processed panda behavior video frame sequence (such as the frame-level feature sequence extracted by convolution and Transformer), the corresponding behavior category label and the associated text description. Generally, a pre-trained base model is used to build a training framework, for example, a video Transformer model (such as Video Swin Transformer) pre-trained on general video understanding tasks (such as action classification and temporal feature modeling) is selected as the basic backbone for visual feature learning, and a language model (such as a lightweight version of BERT) pre-trained on a large-scale text corpus is used as the basic module for text feature learning. By leveraging the general visual temporal dependency capturing ability and text semantic understanding ability learned by the base model, the training difficulty and data requirement of the model on panda specific behavior data are reduced. During the training process, the model relies on the base model to optimize the feature extraction accuracy of the visual branch for the dynamic changes of panda behavior (such as claw movements during eating and body posture transitions during climbing) by learning the mapping relationship between visual features and behavior category labels. On the other hand, by combining text descriptions and using cross-modal attention mechanisms, the visual branch and the text branch interactively learn to strengthen the understanding of behavior semantics (such as the text "panda eats bamboo" can help the model focus on the key area of "mouth contact with bamboo" in the visual). At the same time, the parameters of the base model and the subsequent task adaptation layer are iteratively optimized using a loss function (such as cross-entropy loss) to gradually transition the model from "general feature learning" to "panda specific behavior discrimination", and finally master the ability to accurately distinguish behavior categories from visual dynamic changes. The input in the model inference stage is the panda behavior video frame sequence to be recognized and the text description. After processing by the feature extraction module derived from the base model, visual feature sequences and text feature sequences are formed, and the two types of features are integrated through a cross-modal fusion layer. The output is the panda behavior category (such as eating, climbing, resting, etc.) corresponding to the video frame sequence, presented in the form of standardized labels, achieving accurate recognition and classification of panda behavior.

[0023] The multi-modal data set comprises a sequence of video frames of panda behavior, behavior category information, and a text description; the sequence of video frames has undergone standardized annotation processing; the behavior category information is a preset standardized label corresponding to at least one behavior type of the panda, such as eating, climbing, resting, etc., and is used as a template input parameter to trigger generation of the text description; and the text description is generated based on a preset template and corresponds to the behavior category information, and is in the form of a natural language sentence to map the behavior scene and core content of the panda presented in the sequence of video frames, for example, when the behavior category is eating, a text description similar to "a video of a panda eating bamboo" is generated, realizing the conversion from the behavior category to the natural language expression, and establishing the association between the text modality and the visual modality and the behavior category information.

[0024] The standardized annotation processing refers to an operation of marking and explaining the behavior of the panda in the video in accordance with unified and standardized rules and processes. Specifically, a standard definition system covering various behaviors of the panda (such as eating, climbing, resting, etc.) is first defined to ensure consistent understanding of the behavior categories by different annotators. Then, according to the system, the behavior categories presented by the panda in the collected video segments containing the behavior of the panda are marked frame by frame or segment by segment to ensure consistency and accuracy of the annotation results in terms of terminology use and behavior determination basis, thereby providing reliable and standardized basic information for subsequent multi-modal data set construction, behavior recognition model training, etc. based on these annotation data.

[0025] The feature data of the panda behavior is extracted by a Transformer network, wherein the feature data comprises video feature data and text feature data, and the specific extraction process is as follows:

[0026] S1. Obtain a video sequence to be processed, the video sequence is composed of multiple frames of images, and the video sequence is represented as , each frame of image has a preset height and width, represents the frame of image, ℝ is a set of real numbers, a tensor with real values, and 3 represents RGB three channels, T is the number of video frames, H and W are the height and width of the image, respectively.

[0027] S2. Encode the video sequence by a visual Transformer to obtain a frame-level feature sequence, specifically including:

[0028] S21. First, divide each frame of image in the video sequence into F t according to a preset patch size P × P . image blocks, and the image blocks are linearly mapped by two-dimensional convolution to obtain initial features:

[0029] ;

[0030] wherein is the number of image blocks, D is the feature dimension.

[0031] S22. Next, a learnable single [class] category label is added at the front end of the sequence of image blocks subjected to linear mapping , and video position coding is added to the sequence :

[0032] ;

[0033] Subsequently, the sequence of features after adding the position coding is input to a preset number of layers (such as L layers) of a Transformer encoder for global feature modeling, and the output dimension is unchanged. The processed features are:

[0034] ;

[0035] S23. Next, the feature corresponding to the category label is extracted as the visual feature of the corresponding single frame image:

[0036] ;

[0037] Finally, the visual features of all single frame images are arranged in sequence according to the frames to obtain a sequence of frame-level features of the video:

[0038] ;

[0039] S3. The sequence of frame-level features is modeled by a time sequence Transformer to obtain a time sequence enhanced global feature of the video, specifically including:

[0040] S31. The sequence of frame-level features is input to the time sequence Transformer, and the time sequence Transformer is composed of multiple layers of residual attention blocks for capturing inter-frame time sequence dependencies.

[0041] S32. The sequence of frame-level features is sequentially processed by multiple layers of residual attention blocks, and each layer of processing process is: the inter-frame time sequence dependencies are captured by the multi-head self-attention mechanism in the current layer of residual attention block, the features after capturing the time sequence dependencies are subjected to layer normalization processing, and then the processed features are sent to a feedforward fully connected layer for feature conversion.

[0042] Let the input be , the first The layer multi-head self-attention residual connection calculation formula is:

[0043] ;

[0044] The residual connection calculation formula of the feedforward network is:

[0045] ;

[0046] wherein, MHA(·) represents a multi-head self-attention mechanism, which can capture the time sequence dependency between frames in a frame-level feature sequence from multiple attention heads, (such as the action association between the first frame of the giant panda lifting its head and the third frame of the giant panda eating), LN is layer normalization, and MLP(·) is a feedforward fully connected layer.

[0047] After L t The layer calculation obtains a time sequence enhanced video global feature:

[0048] ;

[0049] S33. After all the layer residual attention blocks are processed, the time sequence enhanced video global feature is output.

[0050] Text feature data, the specific extraction process is: obtaining an input text description, the text description W is in the form of a word sequence, wherein, is the number of text words, w i is the th word.

[0051] First, each word is mapped to a fixed-dimensional embedding vector, e i is the word embedding vector of the i th word:

[0052] ;

[0053] The embedding operation Embed(·) is a mapping operation that maps discrete symbols (such as words, characters, etc.) to a fixed-dimensional continuous vector space.

[0054] The position encoding of the text is added again , then the text feature sequence after adding the position encoding is:

[0055] ;

[0056] The sequence z 0 after adding the position encoding is input to L txtIn the Transformer model composed of the text Transformer residual attention blocks, let Z 0= z 0, then by analogy, the text Transformer residual attention block in the i-th layer is The multi-head self-attention residual calculation formula of the i-th layer is:

[0057] ;

[0058] The feedforward residual is:

[0059] ;

[0060] where MHA(·) represents the multi-head self-attention mechanism, LN represents layer normalization, and MLP(·) represents the feedforward fully connected network.

[0061] After all the layer residual attention blocks are processed, the text word-level feature sequence is obtained:

[0062] ;

[0063] The feature of the first position of the text sequence is taken as the text global semantic vector:

[0064] ;

[0065] The cross-modal attention mechanism is specifically: the category embedding, the video global feature and the text global feature are obtained, and the three are subjected to L2 norm normalization processing; the text feature is used to guide the video feature, the similarity matrix of the text and the video is calculated, the attention weight is obtained by using the Softmax function along the time dimension, the video feature is weighted and aggregated by using the attention weight, and the text-guided video feature is generated; the category embedding is extended into a tensor with the same shape as the text-guided video feature, so that the category, the text and the video directly interact at the same space position; the features after the interaction are subjected to element-level multiplication operation, and the average value is taken along the word dimension to obtain the original category score which can be used for cross-entropy loss calculation and reasoning stage and has not been subjected to Softmax normalization.

[0066] The cross-modal attention mechanism is designed to realize efficient interaction between the visual and text modalities. The category perception triple interaction mechanism TRIPLE is proposed by taking the behavior label as the category embedding, the category embedding is used as a semantic anchor point, the video is first guided in a fine-grained manner by the text, and then the category and the aligned video-text representation are directly interacted, so that a discriminative joint feature is obtained in a simple calculation path.

[0067] Let the learnable category embedding vector be , video frame-level features are , text word-level features are , where C is the number of classes, B is the number of batches, is the number of text words, T is the number of video frames, D is the number of feature dimensions. First, L2-norm normalization is performed on the three to make them mathematically lie on the same hyper-sphere:

[0068] ;

[0069] The symbol represents forward / left assignment, represents L2-norm normalization.

[0070] Subsequently, the text features implement attention guidance on the video features at the word-level granularity, so as to calculate the similarity matrix S between the text and the video:

[0071] ;

[0072] a ∈[1, C ], b ∈[1, B ], , t ∈[1, T ], d ∈[1, D ] respectively correspond to the class, the batch, the word, the frame, and the feature index; S abit denotes the element of the matrix S corresponding to the index of the corresponding dimension abit , and the same applies hereinafter.

[0073] The Softmax function is used along the video frame dimension to obtain the attention weight matrix A :

[0074] ;

[0075] The formula involves a summation operation along the frame dimension, and in order to prevent confusion, the summation frame index is changed to ; the temperature coefficient τ1=0.01 is used to sharpen the attention weight distribution.

[0076] The video feature matrix of the text-guided attention (attention: att) is obtained by weighting and aggregating the video features with the weight: , whose components are According to the index (a, b, i, d) combination:

[0077] ;

[0078] Next, the category embedding Expansion (expend: exp) is a matrix with the same form :

[0079] ;

[0080] That is, the matrix Expansion dimension, but does not change the original element value, by its components According to the index (a, b, i, d) combination.

[0081] In the expanded space, the category, text, and video directly complete interaction at the same position. After element-level multiplication, the average along the word and feature dimension is taken to obtain the original category score logits , that is, the original category score matrix Z :

[0082] ;

[0083] Where ⊙ represents element-level multiplication (Hadamard product).

[0084] The ternary interaction mechanism takes the category embedding as the core anchor point, first lets the text look at the video in fine granularity, and then lets the category and the aligned video-text representation directly dialogue, the calculation path is simple, and the alignment effect is good.

[0085] Step two, based on the feature data of giant panda behavior, construct a pair of sample of video frame sequence-text description, by cross-modal representation network in the unified embedding space to carry out bidirectional matching learning, and by the optimizer with symmetric cross entropy loss as the main loss function to implement parameter update, requires that the paired samples approach each other, and the unpaired samples separate from each other, through the verification and early stopping mechanism constraint, when the training end flag is monitored, the obtained behavior recognition model parameters and inference process are solidified, so that the initial behavior recognition model is marked as the behavior recognition model.

[0086] The optimizer updates parameters using symmetric cross-entropy loss as the main loss function. Specifically, the main loss function is guided by the bidirectional matching relationship between video and text, and text and video. It first calculates two types of unidirectional matching losses separately, and then combines them to form the final symmetric cross-entropy loss. The video-to-text matching loss measures the similarity between video features and text features for each video in the training batch using cosine similarity. The similarity distribution is scaled by a temperature coefficient to maximize the similarity between the video and text of the same category and minimize the similarity with text of different categories. The text-to-video matching loss uses the same logic to maximize the similarity between each text and videos of the same category and minimize the similarity with videos of different categories.

[0087] In one example implementation, a bidirectional matching learning objective is designed, and the model is trained using symmetric cross-entropy loss to improve the alignment capability of cross-modal representations. Specifically, the matching relationships between video to text and text to video are considered simultaneously. By maximizing the similarity of matched pairs and minimizing the similarity of unmatched pairs, consistency constraints in the cross-modal feature space are achieved.

[0088] Let the global features of the text be The global features of the video are Both have been processed using L2 normalization. Similarity function Using cosine similarity:

[0089] ;

[0090] For video-to-text matching loss, with a batch size of B In the case of, let In order to be with the first b For a set of video tags with the same category index, the loss can be defined as:

[0091] ;

[0092] Where τ2 is a temperature coefficient hyperparameter used to scale the smoothness of the similarity distribution; K , b , As a vector index, when used as a subscript, it represents the first position of the corresponding vector. K , b , Each element.

[0093] Similarly, the text-to-video matching loss can be defined as:

[0094] ;

[0095] The final symmetric cross-entropy loss is:

[0096] ;

[0097] The loss function guides the video features and the corresponding category features to be close in the embedding space during the training process, and separates the mismatched feature pairs, thereby enhancing the performance of the model in the cross-modal retrieval and classification tasks.

[0098] Step three, obtaining target giant panda behavior data, and performing data optimization processing on visual data in the target giant panda behavior data, inputting the target giant panda behavior data after the optimization processing into the behavior recognition model, and analyzing and recognizing the target giant panda behavior category through the behavior recognition model.

[0099] In the present application, the target giant panda behavior data specifically refers to a data set to be analyzed in terms of behavior category, which includes video data recording the activity process of giant pandas and text description information supplemented later to describe the video content. When processing the data, only the video data is optimized, and the core reason is that: the video data is usually collected in complex environments such as the wild, and is easily affected by factors such as light changes, vegetation obstructions, and camera shaking, resulting in quality problems such as blurred pictures and loss of details, which in turn affects the extraction accuracy of the model for behavior characteristics; and the text description is generated by manual or automatic labeling later, and the content is stable and has no clearness-related quality fluctuations, so it does not need to be optimized. Based on this, the present application optimizes the video data first to improve the quality of visual features, and then inputs the optimized complete target data into the behavior recognition model, to ensure that the model relies on the synergistic effect of high-quality visual data and stable text description to accurately analyze and recognize the specific behavior category of giant pandas.

[0100] Figure 2 is the target giant panda data optimization flowchart provided by the embodiment of the present application, in which Figure 1After the model training of steps one to three is completed, the visual data contained in the target giant panda behavior data to be input into the trained model needs to be optimized; the quality characterization coefficient of the visual data is compared with the defined quality characterization coefficient, if the quality characterization coefficient of the visual data is greater than or equal to the defined quality characterization coefficient, the current data cleaning process is executed for data optimization, the target giant panda behavior data after optimization is input into the behavior recognition model, the target giant panda behavior category is recognized by the behavior recognition model, if the quality characterization coefficient of the visual data is less than the defined quality characterization coefficient, the current data cleaning process is adjusted, the quality characterization coefficient deviation is compared with the quality characterization coefficient deviation reference value, if the quality characterization coefficient deviation is less than the quality characterization coefficient deviation reference value, the slight deblurring kernel radius in the data cleaning process is adjusted based on the quality characterization coefficient deviation increase, if the quality characterization coefficient deviation is greater than or equal to the quality characterization coefficient deviation reference value, the slight deblurring kernel radius in the data cleaning process and the global clarity reconstruction weight are adjusted based on the quality characterization coefficient deviation increase; after the adjustment is completed, the data optimization is performed, after the data optimization is completed, the quality characterization coefficient of the visual data is updated.

[0101] Figure 3 The flow chart of the initialization behavior recognition model provided by the embodiment of the application is determined, after the data optimization is completed, the quality characterization coefficient of the visual data is updated, it is determined whether the visual data in the target giant panda behavior data is quality prewarned, if yes, the quality prewarning is performed, if no, it is judged whether there is a data normal condition, if yes, the target giant panda behavior data is input into the behavior recognition model, if no, the parameter adjustment strategy is continuously executed, after the data normal condition is monitored within the defined execution times, the target giant panda behavior data is input into the behavior recognition model, if the data normal condition is not monitored within the defined execution times, the behavior recognition model is initialized, and the target giant panda behavior data is input into the behavior recognition model.

[0102] The visual data in the target giant panda behavior data is optimized, specifically, the perception hash entropy value of the visual data, the local contrast entropy of the visual data and the interframe optical flow gradient module variance of the visual data are obtained.

[0103] Perceptual hash entropy value refers to the information entropy calculated based on the perceptual hash feature of visual data (such as images, video frames), which is used to quantify the richness and disorder of the texture details of visual data. The higher the entropy value, the more complete and high-quality the details of the visual data are. On the contrary, it means that the details are lost or blurred to a higher degree. First, the visual data is converted into a grayscale image and scaled to a fixed size (such as 8x8 pixels) to reduce the interference of redundant information. The scaled image is then subjected to discrete cosine transform (extracting low-frequency components that reflect the overall structure and key details of the image). A threshold is set to binarize the low-frequency components, generating a perceptual hash sequence composed of 0 and 1. Finally, the distribution probability of 0 and 1 in the hash sequence is calculated, and the perceptual hash entropy value is obtained based on the information entropy formula.

[0104] Local contrast entropy is an information entropy obtained by dividing the visual data into local regions, calculating the contrast of each region, and then obtaining the probability distribution. It is used to represent the complexity and recognition of the local brightness difference of visual data. The higher the entropy value, the more rich the local light and dark levels and the stronger the detail differentiation. The visual data is divided into several non-overlapping local regions according to a predetermined size (such as 16x16 pixels). The contrast of each region is calculated (commonly the difference between the maximum and minimum pixel values in the region, or the standard deviation of the region's grayscale). The distribution of all region contrast values is calculated to obtain the probability density function of the contrast. The entropy value of this distribution is calculated by substituting the information entropy formula, which is the local contrast entropy.

[0105] The inter-frame optical flow gradient module length variance is used for quantifying the intensity of inter-frame motion (such as camera shaking and target rapid movement), and the greater the variance, the more unstable the inter-frame motion, which is easy to cause picture blur or ghosting, and vice versa, the motion is more stable. Firstly, the dense optical flow algorithm (such as Farneback algorithm) is used to generate the optical flow field for the adjacent two frames of the video: by analyzing the position change of each pixel between the two frames, the horizontal motion component (horizontal movement distance) and the vertical motion component (vertical movement distance) of the pixel are determined, so as to quantify the motion state of all pixels. Then, the spatial gradient of each pixel motion vector is calculated: for the horizontal motion component of a single pixel, the difference between the horizontal motion component of the pixel and the horizontal motion component of the adjacent pixels is analyzed to obtain the horizontal direction gradient; for the vertical motion component of the pixel, the difference between the vertical motion component of the pixel and the vertical motion component of the adjacent pixels is analyzed to obtain the vertical direction gradient, so as to reflect the change rate of the motion state of each pixel in the surrounding area. Subsequently, the horizontal direction gradient and the vertical direction gradient of the same pixel are comprehensively calculated. Specifically, the square operation is performed on the gradient values of the two directions respectively, then the two square values are added, and finally the square root of the added result is taken. Through this process, the gradient information of the two directions is fused into a single value, which is the optical flow gradient module length of the pixel, which can directly reflect the intensity of the motion change of a single pixel. Finally, the optical flow gradient module length values of all pixels in the two frames are collected, and the overall dispersion of these values is calculated, that is, the variance is processed, and the result obtained is the inter-frame optical flow gradient module length variance.

[0106] The perception hash entropy value is compared with the minimum allowed perception hash entropy value, the local contrast entropy is compared with the minimum allowed local contrast entropy, and the inter-frame optical flow gradient module length variance is compared with the maximum value of the inter-frame optical flow gradient module length variance. The results of each comparison are weighted and summarized to obtain a quality representation coefficient of the visual data, wherein the quality representation coefficient of the visual data is used to quantitatively represent the clarity quality of the visual data.

[0107] In the weighted summary, each comparison result is multiplied by the corresponding weight coefficient and then accumulated. The weight coefficient refers to the proportion of the parameter in the quality representation coefficient, and the value range is between 0 and 1. Multiple sets of comparison experiments verify the rationality of the weight, adjust different weight combinations to calculate the quality representation coefficient, observe the recognition accuracy of the model for the behaviors of the giant panda such as eating, climbing and resting, and select the weight distribution scheme that makes the overall recognition accuracy of the model optimal and the robustness strongest. Finally, the specific proportion of each parameter in the quality representation coefficient is determined to ensure that the weight coefficient can objectively reflect the actual influence value of the parameter on the data quality.

[0108] The determination of the allowed minimum value of the perceptual hash entropy value, the allowed minimum value of the local contrast entropy, and the running maximum value of the frame-to-frame optical flow gradient modulus variance requires relevant technical personnel to determine them through data statistics and experimental verification in combination with the visual data characteristics and model requirements of the panda behavior recognition scene. First, a large number of wild panda video samples (covering different light, occlusion, jitter, and other scenes) are collected, and the perceptual hash entropy value, local contrast entropy, and frame-to-frame optical flow gradient modulus variance of each sample are extracted to statistically determine the distribution range and data characteristics of the three types of parameters. Second, the effective samples that allow the model to accurately identify the behavior of the panda are selected based on the core standard that the parameters correspond to the visual quality that meets the feature extraction requirements of the model, and the lower limit interval of the perceptual hash entropy value and the local contrast entropy and the upper limit interval of the frame-to-frame optical flow gradient modulus variance in the effective samples are determined. For example, a too low perceptual hash entropy value will result in loss of texture details, so the lowest value of the entropy in the effective samples is taken as the allowed minimum value. A too low local contrast entropy means that the light and shade levels of the picture are blurred, so the lowest value in the effective samples is taken as the allowed minimum value. A too high frame-to-frame optical flow gradient modulus variance represents intense motion and picture distortion, so the highest value in the effective samples is taken as the allowed maximum value. Finally, the parameter rationality is verified through multiple comparison experiments, and the parameter range is fine-tuned to the optimal recognition accuracy and robustness of the model for the panda behavior, and the allowed extreme values of the three types of parameters are finally determined.

[0109] When the visual data has intense inter-frame motion due to camera jitter, rapid target movement, etc., the frame-to-frame optical flow gradient modulus variance will increase, and intense motion easily leads to picture blurring and detail misplacement, which in turn damages the texture detail integrity of the visual data, resulting in a decrease in the perceptual hash entropy value. Since the perceptual hash entropy value depends on the richness of the texture details, loss of details will directly reduce the disorder of the 0 and 1 hash sequence. At the same time, picture blurring also weakens the light and shade differences in local areas, making the luminance gradient between adjacent areas smaller and causing the local contrast entropy to decrease simultaneously. Conversely, if the frame-to-frame optical flow gradient modulus variance is small (the inter-frame motion is smooth), the picture details are preserved intact, the perceptual hash entropy value can be maintained at a high level to reflect the richness of the texture, and the local contrast entropy will also remain at a high value due to the clear light and shade levels. In the quality representation coefficient calculation, the perceptual hash entropy value and the local contrast entropy are both positive influence factors, and the higher their values, the better the data details and discrimination. The frame-to-frame optical flow gradient modulus variance is a negative influence factor, and the higher its value, the more serious the motion interference. The three types of parameters are linked through the chain of motion stability, detail integrity, and light and shade discrimination to determine the final value of the quality representation coefficient, which in turn quantifies the overall quality of the visual data.

[0110] The quality characterization coefficient of the visual data is compared with a defined quality characterization coefficient. The defined quality characterization coefficient represents the minimum value allowed for the quality characterization coefficient and is determined by multi-stage data analysis and experimental verification around the adaptability of the giant panda visual data quality and the performance of the behavior recognition model. First, visual data samples of the giant panda in different collection scenarios (such as changes in illumination, vegetation obstruction, camera shaking, etc.) are collected, the perceptual hash entropy value, local contrast entropy, and frame-to-frame optical flow gradient modulus variance of each sample are extracted, and the quality characterization coefficient of each sample is calculated according to the preset weight (determined according to the influence of each parameter on the model feature extraction) to form an initial coefficient data set. Second, samples with different quality characterization coefficients are input into the behavior recognition model, the behavior recognition accuracy of the model for each sample is counted, and samples with recognition accuracy meeting the preset threshold (such as more than 90%) are selected to determine the minimum value of the quality characterization coefficient of the effective samples, which is taken as the defined quality characterization coefficient.

[0111] If the quality characterization coefficient of the visual data is greater than or equal to the defined quality characterization coefficient, it means that the texture detail integrity and the contrast level distinction of the visual data meet the standard, and the inter-frame motion stability is within a reasonable range, meeting the basic needs of feature extraction of the behavior recognition model. The model can effectively recognize typical behaviors of the giant panda such as eating, climbing, resting, and moving based on such data, and then the current data cleaning process is executed, mainly for optimizing data redundancy and slight interference, specifically including: removing invalid frames without the giant panda target in the video, removing local noise pixels caused by temporary obstruction of the lens, slightly calibrating the picture brightness and contrast to balance the consistency of visual features, while preserving the temporal correspondence between the text description and the video frames, ensuring that the optimized data maintains the integrity of the original behavior information and reduces the influence of irrelevant interference on model reasoning, thereby performing data optimization processing, and inputting the optimized target giant panda behavior data into the behavior recognition model.

[0112] If the quality characterization coefficient of the visual data is less than the defined quality characterization coefficient, the current data cleaning process needs to be adjusted, indicating that the overall quality of the current visual data does not meet the basic needs of the behavior recognition model. The initial data cleaning process is designed based on the assumption that the quality characterization coefficient of the visual data is greater than or equal to the defined quality characterization coefficient, and can only handle slight redundancy or interference, but cannot specifically solve the above low-quality problems (such as unable to repair severe blur or unable to suppress picture dislocation caused by severe motion). If the original process is continued, not only the data quality cannot be improved, but also effective information may be further lost due to invalid processing, leading to a significant decrease in the subsequent model recognition accuracy. Therefore, the current data cleaning process needs to be adjusted.

[0113] Adjusting the current data cleaning process refers to executing a parameter adjustment strategy, wherein the parameter adjustment strategy specifically refers to: differentiating the quality characterization coefficient from the quality characterization coefficient of the visual data, i.e., difference processing, and comparing the processing result with the quality characterization coefficient, i.e., ratio processing, the processing result is marked as quality characterization coefficient deviation, which represents the deviation between the quality characterization coefficient and the quality characterization coefficient of the visual data. The greater the quality characterization coefficient deviation, the smaller the quality of the visual data is than the quality characterization coefficient.

[0114] Comparing the quality characterization coefficient deviation with the quality characterization coefficient deviation reference value; the quality characterization coefficient deviation reference value is a critical threshold for defining the quality of the visual data as slightly substandard or severely substandard, which is formulated by relevant technical personnel.

[0115] If the quality characterization coefficient deviation is less than the quality characterization coefficient deviation reference value, i.e., the quality of the visual data is slightly substandard, the core problem is mostly slight blur (such as slight lens fogging, slight detail dispersion caused by weak light), then the slight deblurring kernel radius in the data cleaning process is adjusted based on the increase of the quality characterization coefficient deviation, i.e., the quality characterization coefficient deviation is multiplied by the current slight deblurring kernel radius, and the result is added to the current slight deblurring kernel radius, thereby completing the increase adjustment of the slight deblurring kernel radius. At this time, only the slight deblurring kernel radius is adjusted, because the deblurring kernel radius directly determines the repair range of the deblurring algorithm for the blurred area of the picture. The range of the blurred area corresponding to the slight blur is small and the degree is shallow. By increasing the deblurring kernel radius, the algorithm can more accurately cover the slightly blurred area, while avoiding introducing picture distortion caused by excessive repair.

[0116] If the quality characterization coefficient deviation is greater than or equal to the quality characterization coefficient deviation reference value, that is, the visual data quality severity does not meet the standard, there are often moderate and above blur and global texture weakening composite problems (such as overall picture blur caused by thick vegetation obstruction, interframe detail dislocation caused by camera severe shaking), and increasing the deblurring kernel radius cannot repair the global texture loss. The deblurring kernel focuses on the edge recovery of the local blur area, while the global clarity reconstruction can make up for the large-scale detail loss by adjusting the overall contrast of the picture and strengthening the texture level. Based on the quality characterization coefficient deviation, the slight deblurring kernel radius in the data cleaning process is adjusted, and based on the quality characterization coefficient deviation, the global clarity reconstruction weight in the data cleaning process is adjusted, that is, the quality characterization coefficient deviation is multiplied by the global clarity reconstruction weight, and the result is added to the current global clarity reconstruction weight, thereby completing the increase adjustment of the current global clarity reconstruction weight. Through double parameter adjustment, both the edge details of the local blur area and the texture level of the global picture are restored, which can effectively improve the quality of the severely substandard data and meet the basic requirements of model feature extraction (such as clearly showing the paw movement of the panda when eating and the limb posture when climbing). Ensure the robustness of model recognition: avoid the problem of limited quality improvement due to insufficient single repair method, and double adjustment can make the cleaned data of severely substandard data more consistent with the visual features of the standard data, reduce the recognition error of the model due to input quality fluctuations, and ensure the recognition accuracy of various behaviors of the panda.

[0117] After the adjustment is completed, the data optimization processing is performed, and after the data optimization processing is completed, the quality characterization coefficient of the visual data is updated, so as to determine whether to perform quality warning on the visual data in the target panda behavior data.

[0118] Determine whether to perform quality warning on the visual data in the target panda behavior data, and the specific determination process is: difference processing is performed on the updated quality characterization coefficient of the visual data and the quality characterization coefficient of the visual data before updating, the processing result is divided by the quality characterization coefficient of the visual data before updating, the processing result is marked as quality characterization coefficient increment, and compared with the defined quality characterization coefficient increment.

[0119] The defined quality representation coefficient increment refers to the minimum improvement range of the quality representation coefficient of the visual data after the data cleaning process. To obtain the defined quality representation coefficient increment, a plurality of groups of visual data samples of the giant panda with light and heavy quality defects are collected, and the corresponding data cleaning adjustment strategy is used to process the samples. The quality representation coefficients of each group of samples before and after cleaning are recorded, and the increment is calculated. Then, the samples with different increments after cleaning are input into the behavior recognition model, and the samples with a recognition accuracy reaching the preset qualified standard are selected. The minimum value of the quality representation coefficient increment in these effective samples is extracted as the initial reference value. Subsequently, repeated experiments are performed in different field collection scenarios such as low light and high occlusion. The initial reference value is weighted and calibrated based on the data proportion of each scenario. Finally, the stability of the calibrated value is verified by continuously inputting a plurality of low-quality samples, and the defined quality representation coefficient increment that can accurately determine whether the cleaning adjustment is effective is finally determined.

[0120] If the quality representation coefficient increment is less than or equal to the defined quality representation coefficient increment, it indicates that the cleaning adjustment is ineffective, and the data still has unresolved quality defects (such as blurred images not effectively eliminated and missing texture details). If the data is directly input into the model, it will lead to insufficient extraction accuracy of the model for the behavior characteristics of the giant panda, and further cause problems such as decreased recognition accuracy, misjudgment (such as misjudging rest as feeding), or missed judgment, affecting the reliability of the subsequent analysis results of the giant panda behavior. Therefore, a quality warning is determined for the visual data in the target giant panda behavior data, i.e., a quality defect analysis report is generated. Combined with the previous parameter records (such as the frame-to-frame optical flow gradient modulus variance and local contrast entropy), the specific reasons for the insufficient increment are located, and a defect report is uploaded to the technical personnel end.

[0121] If the quality representation coefficient increment is greater than the defined quality representation coefficient increment and there is a data normal condition, the target giant panda behavior data is input into the behavior recognition model. This means that the data meets both the effective cleaning adjustment and the quality standard, and has the basic qualifications to input into the behavior recognition model.

[0122] If the quality representation coefficient increment is greater than the defined quality representation coefficient increment and there is no data normal condition, the parameter adjustment strategy is continuously executed. After the data normal condition is monitored within the defined execution times, the target giant panda behavior data is input into the behavior recognition model. If the data normal condition is not monitored within the defined execution times, the behavior recognition model is initialized, and the target giant panda behavior data is input into the behavior recognition model.

[0123] The execution frequency is defined as the maximum number of times the parameter adjustment strategy is allowed to execute. The process of setting it needs to balance the data cleaning effect and processing efficiency, with the core focusing on two principles: avoiding excessive adjustment and ensuring quality standards. For example, first, by experimentally analyzing the quality improvement of different quality defect data after 1-5 adjustments, it is determined that most data can meet the quality standards after 3-4 adjustments, and further increasing the number of times will only increase the calculation cost without significant effect. Second, combined with the requirements of the behavior recognition model for data processing efficiency, the risk of process delay caused by excessive number of times is excluded. Finally, the maximum number of times required by most data to meet the standards and the upper limit of the number of times allowed by the time efficiency are taken as the minimum value to determine the execution frequency, ensuring that both invalid adjustment and resource waste are avoided, and the quality repair needs of most data are covered.

[0124] The data normal condition refers to the quality representation coefficient of the updated visual data being greater than or equal to the defined quality representation coefficient.

[0125] Within the defined execution frequency, if the data normal condition is not monitored after multiple parameter adjustments (i.e., the quality representation coefficient of the updated visual data does not meet the defined standard), considering that the data has been significantly improved in quality despite not fully meeting the standards, the target giant panda behavior data can be input into the behavior recognition model. At the same time, the behavior recognition model is initialized by resetting the internal parameters to enhance the model's adaptability and robustness to slight quality defects. This approach not only avoids process delays caused by excessive pursuit of data perfection, but also provides dual protection for data optimization and model adaptation through adaptive adjustments on the model side, minimizing the impact of low-quality data on giant panda behavior recognition accuracy and ensuring the efficiency and reliability of the recognition process.

[0126] The initialization process of the behavior recognition model is as follows: obtain the quality representation coefficient of the final visual data, update the quality representation coefficient deviation; based on the updated quality representation coefficient deviation, increase the temperature coefficient of the multi-head self-attention mechanism of the visual Transformer encoder in the adjustment model, thereby analyzing and identifying the target giant panda behavior category through the behavior recognition model. Multiply the quality representation coefficient deviation by the current multi-head self-attention mechanism temperature coefficient, and accumulate the result with the current multi-head self-attention mechanism temperature coefficient to complete the increase adjustment of the multi-head self-attention mechanism temperature coefficient.

[0127] The multi-head self-attention mechanism of the visual Transformer encoder is responsible for capturing the key behavior features (such as limb movements and posture outlines) of the giant panda in the picture, and the temperature coefficient directly affects the accuracy of the distribution of attention weights. When the data has slight quality defects (such as blurred details and weakened local features), the attention mechanism under the original temperature coefficient may pay excessive attention to irrelevant background information (such as vegetation and light and shadow), or fail to accurately locate the behavior feature area of the giant panda. Based on the updated deviation degree increasing temperature coefficient, the concentration degree of attention weights can be reduced, so that the model can more flexibly extract the behavior feature details (such as the claw movements and body posture trends of the giant panda in a blurred state) in the picture although they are not clear, and reduce the feature misjudgment caused by the lack of data details. The benefit of this adjustment is that it does not need to repeatedly clean the data without meaning, but through the dynamic adaptation of model parameters and data quality, it maximizes the retention of effective behavior feature information under the premise that the data is not fully qualified but has been optimized, which not only improves the robustness of the model to low-quality data, but also guarantees the recognition accuracy of the giant panda behavior category, while avoiding the efficiency loss caused by continuous data processing, and achieving the efficient recognition goal of active adaptation of the model under the condition of limited data optimization.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0129] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment according to actual needs.

[0130] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A visual-text cross-modal giant panda behavior recognition method based on attention mechanism, characterized in that, The method includes: Step 1: Input the multimodal dataset into the initial behavior recognition model, extract the feature data of giant panda behavior through the Transformer network, and introduce a cross-modal attention mechanism to align the feature data; Step 2: Based on the feature data of giant panda behavior, construct paired samples of video frame sequence and text description. The cross-modal representation network conducts bidirectional matching learning in a unified embedding space, and the optimizer updates the parameters using symmetric cross-entropy loss as the main loss function. Paired samples are required to move closer to each other, and unpaired samples are required to separate from each other. Through validation and early stopping mechanisms, when the end of training is detected, the obtained behavior recognition model parameters and inference process are solidified, thereby marking the initial behavior recognition model as the behavior recognition model. Step 3: Obtain the target giant panda's behavior data, optimize the visual data in the target giant panda's behavior data, input the optimized target giant panda's behavior data into the behavior recognition model, and analyze and identify the target giant panda's behavior category through the behavior recognition model. The aforementioned data optimization processing of visual data in the target giant panda behavioral data specifically refers to: Acquire the perceptual hash entropy value of visual data, the local contrast entropy of visual data, and the magnitude variance of inter-frame optical flow gradient of visual data; The perceptual hash entropy value is compared with the minimum perceptual hash entropy value, the local contrast entropy is compared with the minimum local contrast entropy value, and the maximum value of the inter-frame optical flow gradient magnitude variance is compared with the inter-frame optical flow gradient magnitude variance. The comparison results are weighted and summarized to obtain the quality characterization coefficient of the visual data. The quality characterization coefficient of the visual data is used to quantitatively characterize the clarity quality of the visual data. Compare the quality characterization coefficients of the visual data with the defined quality characterization coefficients; If the quality characterization coefficient of the visual data is greater than or equal to the defined quality characterization coefficient, the current data cleaning process is executed to optimize the data, and the optimized target panda behavior data is input into the behavior recognition model. If the quality characterization coefficient of the visual data is less than the defined quality characterization coefficient, the current data cleaning process will be adjusted.

2. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 1, characterized in that, The multimodal dataset includes video frame sequences of giant panda behavior, behavior category information, and text descriptions; The video frame sequence has been standardized and annotated. The behavior category information is a preset standardized label, which corresponds to at least one behavior type of the giant panda and is used as a template input parameter to trigger the generation of the text description; The text description is generated based on a preset template. Its content corresponds to the behavior category information and maps the giant panda behavior scenes and core content presented in the video frame sequence in the form of short natural language sentences.

3. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 1, characterized in that, The method involves extracting feature data of giant panda behavior using a Transformer network. This feature data includes video feature data and text feature data. The specific extraction process for the video feature data is as follows: S1. Obtain the video sequence to be processed. The video sequence consists of multiple frames of images, each of which has a preset height and width. S2. Encode the video sequence using a visual Transformer to obtain a frame-level feature sequence, specifically including: S21. Divide each frame of the video sequence into multiple image blocks according to a preset patch size, and perform linear mapping on the image blocks through two-dimensional convolution; S22. Add learnable category labels to the front of the linearly mapped image patch sequence and add positional encoding to the sequence; S23. Input the sequence after adding position encoding into a Transformer encoder with a preset number of layers for global feature modeling, extract the features corresponding to the category label as the visual features of the corresponding single frame image, arrange the visual features of all single frame images in frame order, and obtain the frame-level feature sequence of the video. S3. Model the frame-level feature sequence using a temporal Transformer to obtain temporally enhanced global video features, specifically including: S31. Input the frame-level feature sequence into a temporal Transformer, which is composed of multiple layers of residual attention blocks; S32. The frame-level feature sequence is processed by multiple residual attention blocks in sequence. The processing of each layer is as follows: the inter-frame temporal dependency is captured by the multi-head self-attention mechanism in the current layer residual attention block, the features after capturing the temporal dependency are processed by layer normalization, and then the processed features are sent to the feedforward fully connected layer for feature transformation. S33. After all the residual attention blocks in all layers have been processed, output the temporally enhanced global video features.

4. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 3, characterized in that, The specific extraction process for the text feature data is as follows: Obtain the input text description, which is in the form of a word sequence; The word sequence is preprocessed by mapping each word to a fixed-dimensional embedding vector and adding a learnable positional encoding to the embedding vector sequence. The embedded vector sequence with added position encoding is input into the Transformer model; The sequence is processed by the Transformer model. The processing of each residual attention block is as follows: the input features are processed by a multi-head self-attention mechanism, and after layer normalization, they are sent to the feedforward fully connected network for feature transformation. After all the residual attention blocks in all layers have been processed, a text feature sequence is obtained. The feature at the first position of this sequence is extracted as the global semantic vector of the text.

5. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 1, characterized in that, The cross-modal attention mechanism specifically refers to: Obtain category embeddings, video global features, and text global features, and perform L2 norm normalization on each of them; Attention guidance is applied to video features using text features. The similarity matrix between text and video is calculated. Attention weights are obtained by using the Softmax function along the time dimension. The video features are then weighted and aggregated using the attention weights to generate text-guided video features. The category embedding is extended to a tensor that is isomorphic to the text-guided video features, enabling direct interaction between the category, text, and video in the same spatial location; Element-wise multiplication is performed on the interactive features, and the average value is taken along the word dimension to obtain the original category score that can be used for cross-entropy loss calculation and inference stage, and has not been normalized by Softmax.

6. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 5, characterized in that, The optimizer performs parameter updates using symmetric cross-entropy loss as the main loss function, where the main loss function specifically refers to: With the bidirectional matching relationship between video and text and text and video as the optimization guide, we first calculate the two types of one-way matching loss separately, and then combine the two to form the final symmetric cross-entropy loss. The video-to-text matching loss is applied to each video in the training batch. Cosine similarity is used to measure the similarity between video features and text features. The similarity distribution is scaled by a temperature coefficient to maximize the similarity between the video and text of the same category and minimize the similarity with text of different categories. The same logic is used for the text-to-video matching loss, which maximizes the similarity of each text with videos of the same category and minimizes the similarity with videos of different categories.

7. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 1, characterized in that, Adjusting the current data cleaning process refers to executing a parameter adjustment strategy, whereby the parameter adjustment strategy specifically refers to: The quality characterization coefficients of the definition and the quality characterization coefficients of the visual data are differentiated, and the processing results are compared with the defined quality characterization coefficients. The processing results are marked as the deviation of the quality characterization coefficients. Compare the deviation of the quality characterization coefficient with the reference value of the deviation of the quality characterization coefficient; If the deviation of the quality characterization coefficient is less than the reference value, then the radius of the slight deblurring kernel in the data cleaning process is adjusted by increasing the deviation of the quality characterization coefficient. If the deviation of the quality characterization coefficient is greater than or equal to the reference value of the deviation of the quality characterization coefficient, the radius of the slight deblurring kernel in the data cleaning process is adjusted based on the increase of the deviation of the quality characterization coefficient, and the weight of global clarity reconstruction in the data cleaning process is also adjusted based on the increase of the deviation of the quality characterization coefficient. After adjustment, data optimization is performed. Once data optimization is complete, the quality characterization coefficient of the visual data is updated to determine whether to issue a quality warning for the visual data in the target giant panda's behavioral data.

8. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 7, characterized in that, The specific process for determining whether to issue a quality warning for the visual data in the target giant panda's behavioral data is as follows: The quality characterization coefficients of the updated visual data are compared with those of the visual data before the update. The result is divided by the quality characterization coefficient of the visual data before the update. The result is marked as the quality characterization coefficient increment and compared with the defined quality characterization coefficient increment. If the increment of the quality characterization coefficient is less than or equal to the increment of the defined quality characterization coefficient, then a quality warning is issued for the visual data in the target giant panda behavior data. If the increment of the quality characterization coefficient is greater than the increment of the defined quality characterization coefficient, and there are normal data conditions, then the target giant panda behavior data will be input into the behavior recognition model. If the increment of the quality characterization coefficient is greater than the increment of the defined quality characterization coefficient, and there is no normal data condition, the parameter adjustment strategy will continue to be executed. After the normal data condition is detected within the defined number of executions, the target giant panda behavior data will be input into the behavior recognition model. If the normal data condition is not detected within the defined number of executions, the behavior recognition model will be initialized, and the target giant panda behavior data will be input into the behavior recognition model. The normal condition for the data refers to the updated visual data having a quality characterization coefficient that is greater than or equal to the defined quality characterization coefficient.

9. The visual-text cross-modal giant panda behavior recognition method based on attention mechanism according to claim 8, characterized in that, The initialization process of the behavior recognition model is as follows: Obtain the quality characterization coefficients of the final visual data and update the deviation of the quality characterization coefficients; Based on the updated quality characterization coefficient deviation, the temperature coefficient of the multi-head self-attention mechanism of the visual Transformer encoder in the adjustment model is increased, thereby identifying the target giant panda behavior category through behavior recognition model analysis.

Images