Bimodal behavior recognition method based on basic model and competitive selection

Abstract

The invention relates to a bimodal behavior recognition method based on a basic model and competitive selection, and the method comprises the steps: dynamically generating the mask information of an infrared picture and a visible light picture through employing a competition mechanism for an infrared picture sequence and a visible light picture sequence of a user in the same scene, recording the masked position of the image as a mask mark, and taking the unmasked position as a visible mark; respectively and correspondingly inputting the infrared picture sequence, the visible light picture sequence and corresponding mask information into an infrared encoder and a visible light encoder according to the masked and unmasked positions of the image, and extracting corresponding modal features; and feeding the modal features obtained after coding to a hierarchical mask fusion module, performing calculation by combining the masked and unmasked positions of the image to obtain fusion features, and feeding the fusion features to a classifier to calculate the behavior category of the user in the corresponding scene, so that the efficiency and accuracy of a behavior recognition task can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and computer vision, and in particular relates to a dual-modal behavior recognition method based on a basic model and competitive selection. Background Art

[0002] With the continuous innovation of artificial intelligence technology, behavior recognition technology plays an increasingly important role in intelligent monitoring, autonomous driving, human-computer interaction and other fields. Traditional behavior recognition methods mainly rely on complete video frames for feature extraction and recognition. However, this method will lead to a large amount of redundant calculations when faced with spatiotemporal redundancy in video sequences, thus affecting the efficiency and performance of the system. In addition, single-modality (such as RGB images) behavior recognition methods often show limitations in diverse scenes and complex lighting conditions, limiting their widespread application.

[0003] In order to overcome these challenges, multimodal behavior recognition technology has emerged. Among them, dual-modal behavior recognition technology can significantly improve recognition accuracy and robustness by integrating information from infrared and visible light modalities. The infrared modality can capture the temperature distribution and contour information of the human body in low-light or no-light environments, while the visible light modality provides rich color, texture, and detail information in sufficient light. However, dual-modal behavior recognition also faces challenges such as differences between modalities and data imbalance. Summary of the invention

[0004] In order to solve the problems existing in the background technology and improve the accuracy and robustness of behavior recognition, the present invention provides a dual-modal behavior recognition method based on a basic model and competitive selection, including: training a dual-modal behavior recognition model, and recognizing the user's behavior through the trained dual-modal behavior recognition model, wherein the dual-modal behavior recognition model includes: an infrared encoder, a visible light encoder, a hierarchical mask fusion module and a classifier, and the behavior recognition process is as follows:

[0005] S1: For the infrared image sequence and visible light image sequence of the user in the same scene, a competition mechanism is used to dynamically generate mask information of the infrared image and visible light image, and the masked positions of the image are recorded as mask marks, and the unmasked positions are recorded as visible marks;

[0006] S2: input the infrared image sequence and the visible light image sequence and their corresponding mask information into the infrared encoder and the visible light encoder respectively according to the masked and unmasked positions of the image, and extract the corresponding modal features;

[0007] S3: The modal features obtained after encoding in step S2 are fed to the hierarchical mask fusion module, and the fused features are calculated by combining the masked and unmasked positions of the image, and the fused features are fed to the classifier to calculate the user's behavior category in the corresponding scenario.

[0008] Preferably, the training of the bimodal behavior recognition model includes:

[0009] S01: inputting the modal features of the infrared branch encoded in step S2 and the corresponding mask information into the corresponding infrared light decoder to obtain a reconstructed infrared image sequence, and constructing a first loss function according to the reconstruction result;

[0010] S02: inputting the modal features of the visible light branch encoded in step S2 and the corresponding mask information into the corresponding visible light decoder to obtain a reconstructed visible light picture sequence, and constructing a second loss function according to the reconstruction result;

[0011] S03: Construct a third loss function based on the output results of the classifier and the behavior labels of the samples, and construct the total loss function of the model based on the first loss function, the second loss function and the third loss function; based on the constructed total loss function, update the parameters of the bimodal behavior recognition model through the Adam optimizer to complete the model training.

[0012] Preferably, dynamically generating mask information for the infrared image or the visible light image using a competition mechanism includes:

[0013] S11: Build an information perception module with a kernel and stride size of 16. The information perception module calculates the average value of the feature strength along the channel direction as the score of each token;

[0014] S12: Given a 3*3 motion region, each position p in this motion region corresponds to a token;

[0015] S13: For infrared images or visible light images, use the information perception module to calculate the score of each token; in the 3×3 motion unit area, select the top two tokens as visible tokens; mask the remaining tokens and record the position of the masked token as a mask mark;

[0016] S14: After the above steps, for each infrared image or visible light image, a size of The mask image blocks are enlarged to 4 times and 2 times of the original size, respectively, to obtain mask information, which includes: The Mask_block1 and size are Mask_block2.

[0017] Preferably, step S2 includes: the infrared encoder and the visible light encoder use the same network structure to encode the input infrared picture sequence and the visible light picture sequence, and the encoding process is as follows:

[0018] S21: The input infrared image sequence or visible light image sequence can be represented as T×H×W×C, which is input into the coding layer 1 for the first stage of feature extraction; in the coding layer 1, the input image sequence is divided into several non-overlapping image blocks through PatchEmbed1, and these image blocks are converted to a specific dimension through convolution operation, and the size is obtained. feature E1; then perform a dot product operation on feature E1 and mask_block1 to mask the useless part of feature E1; the masked feature E1 is enhanced by convolutional attention and linear layers in turn, and is residually connected with the original mask feature E1, and then fed to FFN to obtain feature F1, realizing feature conversion and enhancement;

[0019] S22: Input feature F1 into coding layer 2 for the second stage of feature extraction; in coding layer 2, feature F1 is further divided into PatchEmbed2 to obtain The feature E2 is obtained by performing a dot product operation between feature E2 and mask_block2, and feature E2 is masked. The masked feature E2 is then enhanced by the convolutional attention and linear layers in turn, and is residually connected with the original mask feature E2, and then fed to FFN to obtain feature F2, thus realizing feature conversion and enhancement.

[0020] S23: Input feature F2 into coding layer 3 for feature extraction in the third stage; in coding layer 3, feature F2 is further divided into PatchEmbed3 to obtain The feature is then converted into a T×N×C3 feature E3 through a fully connected layer, and the visible mark is obtained according to step S1. The feature corresponding to the visible mark in feature E3 is retained to obtain a feature T×N v ×C3, the feature T×N v ×C3 is fed into the Transformer encoder to obtain the feature Tk_s∈N v ×C4,N v Indicates visible tokens, N indicates all tokens;

[0021] S24: Scale the obtained features F1 and F2 to the same size as feature Tk_S, then extract the unmasked features in features F1 and F2 according to the visible marks in step S1, and add them to feature Tk_S along the channel dimension to obtain a size of T×N v ×C4 infrared branch modal characteristics FI Or the modal characteristic F of the visible light branch V , T represents the number dimension of infrared light image sequence or visible light image sequence.

[0022] Preferably, reconstructing the visible light image sequence or the infrared light image sequence comprises: first defining a learnable embedding parameter x_res of size T×N×C, transforming the feature F I or feature F v The features at the visible marked positions in are embedded into x_res respectively, and the remaining position information of x_res replaces the features of the masked positions in the original infrared image or visible light image, and the embedded x_res is fed to the Transformer decoder to obtain the reconstructed infrared light image sequence or visible light image sequence.

[0023] Preferably, the step of feeding the modal features obtained after encoding in step S2 to the hierarchical mask fusion module to obtain the fused features comprises: feeding the modal features F of the infrared branch I and the modal characteristics of the visible light branch F V They are fed to two independent mask encoders respectively, where the modal features interact with the features at the visible mark position in the modality. The interacted infrared modal features and visible light modal features are concatenated together, and then another mask encoder is used to realize the interaction between the features of the infrared modality and the visible light modality to obtain deep fusion features.

[0024] Preferably, based on the Transformer Encoder, the mask encoder sets the feature value at the mask mark position in the feature attn_score calculated by the multi-head self-attention module to -∞ before performing the softmax operation; before the first linear layer of the MLP, the feature value at the mask mark position in the input feature is set to 0; the output feature of the Transformer Encoder is the deep fusion feature.

[0025] Preferably, feeding the fused features to a classifier to calculate the user's behavior category in the corresponding scenario includes:

[0026]

[0027] Among them, p represents the classification result predicted by the classifier, f c represents the fusion feature, c(·) represents the classification reconstruction decoder based on the Transformer decoder, φ(·) represents the pooling function, and s(·) represents the softmax function.

[0028] Preferably, the first loss function and the second loss function adopt mean square error loss, and the third loss function adopts cross entropy loss.

[0029] Preferably, the total loss function is as follows:

[0030]

[0031] Among them, y M To reconstruct the pixel at the masked position of the image, x M is the pixel at the masked position of the original image, Ω(·) calculates the number of masked pixels, ||·||2 refers to the l2 loss, λ is the parameter to balance the reconstruction branch and the classification branch; C represents the number of classification categories, z i represents the probability of category i predicted by the model, p i represents the true probability of category i; l r_ir and l r_vis Represent the first loss function and the second loss function respectively; l c represents the third loss function; l represents the total loss function.

[0032] The present invention has at least the following beneficial effects

[0033] The method of the present invention dynamically generates mask information by introducing a competition mechanism, and uses a hierarchical mask fusion module to achieve effective fusion of modal features, thereby improving the accuracy and robustness of behavior recognition. The competition mechanism is used to dynamically generate mask information for infrared images and visible light images, aiming to simulate the selective attention mechanism of the human visual system when processing information, and selectively mask part of the information through the competition mechanism to reduce computational redundancy and enhance the generalization ability of the model. The hierarchical fusion module is designed to make full use of the complementary information between different modalities while reducing the differences and redundant information between modalities. Through hierarchical fusion, more robust and accurate fusion features can be obtained. Through the above technical solution, behavior recognition based on infrared and visible light dual modalities can be realized, significantly improving recognition accuracy and robustness. At the same time, by introducing a competition mechanism and a hierarchical mask fusion module, the method effectively solves challenges such as differences between modalities and data imbalance, and provides strong support for the practical application of dual-modal behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the process framework of the method provided by the present invention;

[0035] Figure 2 It is a schematic diagram of the method flow of the present invention;

[0036] Figure 3 Schematic diagram of the generation process of the dual-mode motion mask of the present invention;

[0037] Figure 4 Detailed schematic diagram of the implementation of the basic model encoder of the present invention;

[0038] Figure 5 Detailed schematic diagram of the implementation of the decoder of the present invention;

[0039] Figure 6 Schematic diagram of hierarchical mask fusion of the present invention. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] See also Figure 1 to Figure 6 The present invention provides a dual-modal behavior recognition method based on a basic model and competitive selection, including: training a dual-modal behavior recognition model, and recognizing the user's behavior through the trained dual-modal behavior recognition model, wherein the dual-modal behavior recognition model includes: an infrared encoder, a visible light encoder, a hierarchical mask fusion module and a classifier, and the behavior recognition process is as follows:

[0042] S1: For the infrared image sequence and visible light image sequence of the user in the same scene, a competition mechanism is used to dynamically generate mask information of the infrared image and visible light image, and the masked positions of the image are recorded as mask marks, and the unmasked positions are recorded as visible marks;

[0043] S2: input the infrared image sequence and the visible light image sequence and their corresponding mask information into the infrared encoder and the visible light encoder respectively according to the masked and unmasked positions of the image, and extract the corresponding modal features;

[0044] S3: The modal features obtained after encoding in step S2 are fed to the hierarchical mask fusion module, and the fused features are calculated by combining the masked and unmasked positions of the image, and the fused features are fed to the classifier to calculate the user's behavior category in the corresponding scenario.

[0045] Preferably, the training of the bimodal behavior recognition model includes:

[0046] S01: inputting the modal features of the infrared branch encoded in step S2 and the corresponding mask information into the corresponding infrared light decoder to obtain a reconstructed infrared image sequence, and constructing a first loss function according to the reconstruction result;

[0047] S02: inputting the modal features of the visible light branch encoded in step S2 and the corresponding mask information into the corresponding visible light decoder to obtain a reconstructed visible light picture sequence, and constructing a second loss function according to the reconstruction result;

[0048] S03: Construct a third loss function based on the output results of the classifier and the behavior labels of the samples, and construct the total loss function of the model based on the first loss function, the second loss function and the third loss function; based on the constructed total loss function, update the parameters of the bimodal behavior recognition model through the Adam optimizer to complete the model training.

[0049] Preferably, dynamically generating mask information for the infrared image or the visible light image using a competition mechanism includes:

[0050] S11: Build an information perception module with a kernel and stride size of 16. The information perception module calculates the average value of the feature strength along the channel direction as the score of each token;

[0051] S12: Given a 3*3 motion region, each position p in this motion region corresponds to a token;

[0052] S13: For infrared images or visible light images, use the information perception module to calculate the score of each token; in the 3×3 motion unit area, select the top two tokens as visible tokens; mask the remaining tokens and record the position of the masked token as a mask mark;

[0053] S14: After the above steps, for each infrared image or visible light image, a size of The mask image blocks are enlarged to 4 times and 2 times of the original size, respectively, to obtain mask information, which includes: The Mask_block1 and size are Mask_block2.

[0054] See also Figure 1 and Figure 2, in step 1, first pass through an information perception module with a kernel and stride size of 16, and calculate the average value of the feature intensity along the channel direction as the score. Given a 3*3 motion unit, then select the top 2 tokens in the corresponding positions of infrared and visible light as visible tokens, and the rest are masked. The above process can be described as:

[0055] For infrared mode:

[0056]

[0057] For visible light mode:

[0058]

[0059] Among them, Ω represents a given 3*3 motion unit area, and p represents the position of a token in this area.

[0060] Combine the results of infrared and visible light selections and define the visible Token set as:

[0061]

[0062] Then process the masks selected by the two modalities:

[0063]

[0064] At the same time, the visible token number after the mask and the token number after the mask are recorded. The design of the motion unit mask is to stimulate the information leakage caused by time correlation to reduce the difficulty of the reconstruction task, which means that more details can be focused on in the reconstruction stage to improve the recognition performance. Figure 3 The generation process based on motion unit masks is described.

[0065] See also Figure 4 Preferably, step S2 includes: the infrared encoder and the visible light encoder use the same network structure to encode the input infrared picture sequence and the visible light picture sequence, and the encoding process is as follows:

[0066] S21: The input infrared image sequence or visible light image sequence can be represented as T×H×W×C, which is input into the coding layer 1 for the first stage of feature extraction; in the coding layer 1, the input image sequence is divided into several non-overlapping image blocks through PatchEmbed1, and these image blocks are converted to a specific dimension through convolution operation, and the size is obtained. feature E1; then perform a dot product operation on feature E1 and mask_block1 to mask the useless part of feature E1; the masked feature E1 is enhanced by convolutional attention and linear layers in turn, and is residually connected with the original mask feature E1, and then fed to FFN to obtain feature F1, realizing feature conversion and enhancement;

[0067] S22: Input feature F1 into coding layer 2 for the second stage of feature extraction; in coding layer 2, feature F1 is further divided into PatchEmbed2 to obtain The feature E2 is obtained by performing a dot product operation between feature E2 and mask_block2, and feature E2 is masked. The masked feature E2 is then enhanced by the convolutional attention and linear layers in turn, and is residually connected with the original mask feature E2, and then fed to FFN to obtain feature F2, thus realizing feature conversion and enhancement.

[0068] S23: Input feature F2 into coding layer 3 for feature extraction in the third stage; in coding layer 3, feature F2 is further divided into PatchEmbed3 to obtain The feature is then converted into a T×N×C3 feature E3 through a fully connected layer, and the visible mark is obtained according to step S1. The feature corresponding to the visible mark in feature E3 is retained to obtain a feature T×N v ×C3, the feature T×N v ×C3 is fed into the Transformer encoder to obtain the feature Tk_s∈N v ×C4,N v Indicates visible tokens, N indicates all tokens;

[0069] S24: Scale the obtained features F1 and F2 to the same size as feature Tk_S, then extract the unmasked features in features F1 and F2 according to the visible marks in step S1, and add them to feature Tk_S along the channel dimension to obtain a size of T×N v ×C4 infrared branch modal characteristics F I Or the modal characteristic F of the visible light branch V , T represents the number dimension of infrared light image sequence or visible light image sequence.

[0070] See also Figure 5 Preferably, reconstructing a visible light image sequence or an infrared light image sequence comprises: first defining a learnable embedding parameter x_res of size T×N×C, transforming the feature F I or feature F vThe features at the visible marked positions in are embedded into x_res respectively, and the remaining position information of x_res replaces the features of the masked positions in the original infrared image or visible light image, and the embedded x_res is fed to the Transformer decoder to obtain the reconstructed infrared light image sequence or visible light image sequence.

[0071] See also Figure 6 Preferably, the step of feeding the modal features obtained after encoding in step S2 to the hierarchical mask fusion module to obtain the fused features comprises: feeding the modal features F of the infrared branch I and the modal characteristics of the visible light branch F V They are fed to two independent mask encoders respectively, where the modal features interact with the features at the visible mark position in the modality. The interacted infrared modal features and visible light modal features are concatenated together, and then another mask encoder is used to realize the interaction between the features of the infrared modality and the visible light modality to obtain deep fusion features.

[0072] Preferably, based on the Transformer Encoder, the mask encoder sets the feature value at the mask mark position in the feature attn_score calculated by the multi-head self-attention module to -∞ before performing the softmax operation; before the first linear layer of the MLP, the feature value at the mask mark position in the input feature is set to 0; the output feature of the Transformer Encoder is the deep fusion feature.

[0073] Preferably, feeding the fused features to a classifier to calculate the user's behavior category in the corresponding scenario includes:

[0074]

[0075] Among them, p represents the classification result predicted by the classifier, f c represents the fusion feature, c(·) represents the classification reconstruction decoder based on the Transformer decoder, φ(·) represents the pooling function, and s(·) represents the softmax function.

[0076] Preferably, the first loss function and the second loss function adopt mean square error loss, and the third loss function adopts cross entropy loss.

[0077] Preferably, the total loss function is as follows:

[0078]

[0079] Among them, y M To reconstruct the pixel at the masked position of the image, x Mis the pixel at the masked position of the original image, Ω(·) calculates the number of masked pixels, ||·||2 refers to the l2 loss, which is a parameter to balance the reconstruction branch and the classification branch; C represents the number of classification categories, z i represents the probability of category i predicted by the model, p i represents the true probability of category i; l r_ir and l r_vis Represent the first loss function and the second loss function respectively; l c represents the third loss function; Represents the total loss function. The classifier adopts a softmax classifier.

[0080] In summary, the method of the present invention dynamically generates mask information by introducing a competition mechanism, and uses a hierarchical mask fusion module to achieve effective fusion of modal features, thereby improving the accuracy and robustness of behavior recognition. The competition mechanism is used to dynamically generate mask information for infrared images and visible light images, aiming to simulate the selective attention mechanism of the human visual system when processing information, and selectively mask part of the information through the competition mechanism to reduce computational redundancy and enhance the generalization ability of the model. The design of the hierarchical fusion module aims to make full use of the complementary information between different modalities while reducing the differences and redundant information between modalities. Through hierarchical fusion, more robust and accurate fusion features can be obtained. Through the above technical solution, behavior recognition based on infrared and visible light dual modalities can be realized, significantly improving recognition accuracy and robustness. At the same time, the method effectively solves challenges such as differences between modalities and data imbalance by introducing a competition mechanism and a hierarchical mask fusion module, providing strong support for the practical application of dual-modal behavior recognition.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A bimodal behavior recognition method based on a basic model and competitive selection, characterized in that: include: The bimodal behavior recognition model is trained to recognize the user's behavior. The bimodal behavior recognition model includes: an infrared encoder, a visible light encoder, a hierarchical mask fusion module and a classifier. The behavior recognition process is as follows: S1: For the infrared image sequence and visible light image sequence of the user in the same scene, a competition mechanism is used to dynamically generate mask information of the infrared image and visible light image, and the masked positions of the image are recorded as mask marks, and the unmasked positions are recorded as visible marks; S2: input the infrared image sequence and the visible light image sequence and their corresponding mask information into the infrared encoder and the visible light encoder respectively according to the masked and unmasked positions of the image, and extract the corresponding modal features; S3: The modal features obtained after encoding in step S2 are fed to the hierarchical mask fusion module, and the fused features are calculated by combining the masked and unmasked positions of the image, and the fused features are fed to the classifier to calculate the user's behavior category in the corresponding scenario.

2. The dual-modal behavior recognition method based on basic model and competitive selection according to claim 1, characterized in that: The training of the bimodal behavior recognition model includes: S01: inputting the modal features of the infrared branch encoded in step S2 and the corresponding mask information into the corresponding infrared light decoder to obtain a reconstructed infrared image sequence, and constructing a first loss function according to the reconstruction result; S02: inputting the modal features of the visible light branch encoded in step S2 and the corresponding mask information into the corresponding visible light decoder to obtain a reconstructed visible light picture sequence, and constructing a second loss function according to the reconstruction result; S03: Construct a third loss function based on the output results of the classifier and the behavior labels of the samples, and construct the total loss function of the model based on the first loss function, the second loss function and the third loss function; based on the constructed total loss function, update the parameters of the bimodal behavior recognition model through the Adam optimizer to complete the model training.

3. The dual-modal behavior recognition method based on basic model and competitive selection according to claim 2, characterized in that: The competitive mechanism is used to dynamically generate mask information for infrared images or visible light images, including: S11: Build an information perception module with a kernel and stride size of 16. The information perception module calculates the average value of the feature strength along the channel direction as the score of each token; S12: Given a 3*3 motion region, each position p in this motion region corresponds to a token; S13: For infrared images or visible light images, use the information perception module to calculate the score of each token; in the 3×3 motion unit area, select the top two tokens as visible tokens; mask the remaining tokens and record the position of the masked token as a mask mark; S14: After the above steps, for each infrared image or visible light image, a size of The mask image blocks are enlarged to 4 times and 2 times of the original size, respectively, to obtain mask information, which includes: The size of Mask_block1 is Mask_block2.

4. The method for bimodal behavior recognition based on basic model and competitive selection according to claim 3, characterized in that: The step S2 includes: the infrared encoder and the visible light encoder use the same network structure to encode the input infrared picture sequence and the visible light picture sequence, and the encoding process is as follows: S21: The input infrared image sequence or visible light image sequence can be represented as T×H×W×C, which is input into the coding layer 1 for the first stage of feature extraction; in the coding layer 1, the input image sequence is divided into several non-overlapping image blocks through PatchEmbed1, and these image blocks are converted to a specific dimension through convolution operation, and the size is obtained. feature E1; then perform a dot product operation on feature E1 and mask_block1 to mask the useless part of feature E1; the masked feature E1 is enhanced by convolutional attention and linear layers in turn, and is residually connected with the original mask feature E1, and then fed to FFN to obtain feature F1, realizing feature conversion and enhancement; S22: Input feature F1 into coding layer 2 for the second stage of feature extraction; in coding layer 2, feature F1 is further divided into PatchEmbed2 to obtain The feature E2 is obtained by performing a dot product operation between feature E2 and mask_block2, and feature E2 is masked. The masked feature E2 is then enhanced by the convolutional attention and linear layers in turn, and is residually connected with the original mask feature E2, and then fed to FFN to obtain feature F2, thus realizing feature conversion and enhancement. S23: Input feature F2 into coding layer 3 for feature extraction in the third stage; in coding layer 3, feature F2 is further divided into PatchEmbed3 to obtain The feature is then converted into a T×N×C3 feature E3 through a fully connected layer, and the visible mark is obtained according to step S1. The feature corresponding to the visible mark in feature E3 is retained to obtain a feature T×N v ×C3, the feature T×N v ×C3 is fed into the Transformer encoder to obtain the feature Tk_s∈N v ×C4,N v Indicates visible tokens, and N indicates all tokens; S24: Scale the obtained features F1 and F2 to the same size as feature Tk_S, then extract the unmasked features in features F1 and F2 according to the visible marks in step S1, and add them to feature Tk_S along the channel dimension to obtain a size of T×N v ×C4 infrared branch modal characteristics F I Or the modal characteristic F of the visible light branch V , T represents the number dimension of infrared light image sequence or visible light image sequence.

5. The dual-modal behavior recognition method based on basic model and competitive selection according to claim 4, characterized in that: Reconstructing a visible light image sequence or an infrared light image sequence includes: first defining a learnable embedding parameter x_res of size T×N×C, and transforming the feature F I or feature F v The features at the visible marked positions in are embedded into x_res respectively, and the remaining position information of x_res replaces the features of the masked positions in the original infrared image or visible light image, and the embedded x_res is fed to the Transformer decoder to obtain the reconstructed infrared light image sequence or visible light image sequence.

6. The dual-modal behavior recognition method based on basic model and competitive selection according to claim 5, characterized in that: The step of feeding the modal features obtained after encoding in step S2 to the hierarchical mask fusion module to obtain fused features includes: feeding the modal features F I and the modal characteristics of the visible light branch F V They are fed to two independent mask encoders respectively, where the modal features interact with the features at the visible mark position in the modality. The interacted infrared modal features and visible light modal features are concatenated together, and then another mask encoder is used to realize the interaction between the features of the infrared modality and the visible light modality to obtain deep fusion features.

7. The dual-modal behavior recognition method based on basic model and competitive selection according to claim 6, characterized in that: Based on the Transformer Encoder, the mask encoder sets the feature value at the mask mark position in the feature attn_score calculated by the multi-head self-attention module to -∞ before the softmax operation; before the first linear layer of the MLP, the feature value at the mask mark position in the input feature is set to 0; the output feature of the Transformer Encoder is the deep fusion feature.

8. The dual-modal behavior recognition method based on basic model and competitive selection according to claim 7, characterized in that: The step of feeding the fusion features to the classifier to calculate the user's behavior category in the corresponding scenario includes: Among them, p represents the classification result predicted by the classifier, f c represents the fusion feature, c(·) represents the classification reconstruction decoder based on the Transformer decoder, φ(·) represents the pooling function, and s(·) represents the softmax function.

9. The dual-modal behavior recognition method based on basic model and competitive selection according to claim 8, characterized in that: The first loss function and the second loss function use mean square error loss, and the third loss function uses cross entropy loss.

10. A dual-modal behavior recognition method based on basic model and competitive selection according to claim 9, characterized in that: The total loss function is as follows: Among them, y M To reconstruct the pixel at the masked position of the image, x M is the pixel at the masked position of the original image, Ω(·) calculates the number of masked pixels, ||·||2 refers to the l2 loss, λ is the parameter to balance the reconstruction branch and the classification branch; C represents the number of classification categories, z i represents the probability of category i predicted by the model, p i represents the true probability of category i; l r_ir and l r_vis Represent the first loss function and the second loss function respectively; l c represents the third loss function; l represents the total loss function.

Images