Cross-scene action recognition methods

Abstract

This invention provides a cross-scene action recognition method, belonging to the field of computer vision video understanding technology. The method first trains a dark-to-bright diffusion model, combining prior knowledge obtained from a large-scale diffusion model pre-trained on normal lighting data to convert input dark video frames into bright video frames. During the restoration sampling process, a specific spatiotemporal attention mechanism is integrated into the trained conditional diffusion model to alleviate the discontinuities between video frames caused by low-light enhancement methods based on image training. Subsequently, a specific self-distillation branch is designed and configured into the action recognition backbone network to extract weighted spatiotemporal features between layers of the backbone network, thereby improving the generalization ability of the action recognition network. Compared to mainstream methods in the industry, this invention achieves state-of-the-art results on existing dark video recognition datasets, while also showing a significant performance improvement over baseline results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision video understanding task technology, and specifically relates to a cross-scene action recognition method. Background Technology

[0002] Action recognition is a highly influential task in computer vision video understanding. Current research fully leverages advanced recognition network skeletons and data, such as convolutional neural network-based architectures. [1][2][3] Transformer-based models [4][5] and large-scale action recognition datasets [6][7] Although these methods have achieved remarkable results, they are usually implemented in scenarios with optimal lighting conditions, which limits their applicability in real-world environments. The challenges of this task are: (1) the low visibility of dark videos makes it difficult for the network to extract effective motion information; and (2) the lack of a backbone network for motion recognition in this scenario. At the same time, due to the lack of a large-scale training dataset of low-light videos, the performance of existing models will be greatly reduced when identifying video content with insufficient light.

[0003] Previous research has focused on recognition tasks in specific low-light scenes. [8][9]

[10]

[11] It has its limitations. First, previous works relied too heavily on basic low-light image enhancement strategies, resulting in insufficient improvement in the quality of low-light videos. Second, previous works often did not improve the action recognition backbone network for this task.

[0004] [1]Feichtenhofer,C.,Fan,H.,Malik,J.,&He,K..Slowfast networks for video recognition.In:CVPR(2019).

[0005] [2]Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, MA closer look at spatiotemporal convolutions for action recognition. In: CVPR (2018).

[0006] [3]Feichtenhofer, C.X3d:Expanding architectures for efficient videorecognition.In:CVPR(2020).

[0007] [4]Xing,Z.,Dai,Q.,Hu,H.,Chen,J.,Wu,Z.,&Jiang,Y.G.Svformer:Semi-supervised video transformer for action recognition.In:CVPR(2023).

[0008] [5]Liu,Z.,Ning,J.,Cao,Y.,Wei,Y.,Zhang,Z.,Lin,S.,&Hu,H.Video swintransformer.In:CVPR(2022).

[0009] [6]Goyal,R.,Ebrahimi Kahou,S.,Michalski,V.,Materzynska,J.,Westphal,S.,Kim,H.,...&Memisevic,R.The"something something"video database for learningand evaluating visual common sense.In:ICCV(2017).

[0010] [7]Kuehne,H.,Jhuang,H.,Garrote,E.,Poggio,T.,&Serre,T..HMDB:a largevideo database for human motion recognition.In:ICCV(2011).

[0011] [8]Hira,S.,Das,R.,Modi,A.,&Pakhomov,D.Delta Sampling R-BERT forlimited data and low-light action recognition.In:CVPR(2021).

[0012] [9]Chen,R.,Chen,J.,Liang,Z.,Gao,H.,&Lin,S..Darklight networks foraction recognition in the dark.In:CVPR(2021).

[0013]

[10] Tu, Z., Liu, Y., Zhang, Y., Mu, Q., & Yuan, J. DTCM: Joint Optimization of Dark Enhancement and Action Recognition in Videos. In: IEEE Transactions onImage Processing. (2023).

[0014]

[11] Xu, Y., Yang, J., Cao, H., Mao, K., Yin, J., & See, S. Arid: A new dataset for recognizing action in the dark. In: Deep Learning for Human Activity Recognition: Second International Workshop (2021). Summary of the Invention

[0015] This invention is made to solve the above-mentioned problems, and aims to provide a cross-scene action recognition method.

[0016] This invention provides a cross-scene action recognition method with the following features to improve the network's ability to recognize dark videos, comprising the following steps: S10, using a dark image as a condition and the corresponding real lighting image as the target, training the learnable part of a ControlNet model pre-trained on a large-scale normal lighting dataset; S20, sampling the ControlNet model trained in step S10 to obtain the dark image and its corresponding preliminary restored lighting image; S30, using the dark image, the corresponding real lighting image, and the preliminary restored lighting image concatenated along the channel dimension as input, and training a dark-to-bright conditional diffusion model using a multi-head attention mechanism; S40, using the multi-head attention mechanism... The process is replaced with a spatiotemporal attention mechanism, and a dark-to-light conditional diffusion model is used to sample video frames from dark videos to generate continuous bright video frames. In S50, based on a dual-channel action recognition backbone network structure composed of dark and bright input channels, N self-distillation branches are constructed between its residual modules. Each self-distillation branch includes a spatiotemporal fusion module and an alignment module. In S60, the video frames of dark videos and their corresponding bright video frames are input into the dark and bright input channels, respectively. After feature extraction and loss calculation by the spatiotemporal fusion and alignment modules, they are merged along the channel dimension. After merging, the output features are obtained through the self-attention module. The output features are output from the fully connected layer to the corresponding logit of the shallow residual module.

[0017] The cross-scene action recognition method provided by this invention may also have the following feature: wherein, in step S30, the training loss function of the dark-to-light conditional diffusion model is: This represents a weighted average of the loss across the entire data distribution, where t represents the time step for progressive denoising in the diffusion model, θ represents the parameters optimized during model training, and ∈ t This represents the Gaussian noise added to the image by the diffusion model at time step t, ∈ θ This represents the noise predicted by the diffusion model at time step t. This represents the cumulative noise attenuation factor, where x0 represents the original image data, which is a real illuminated image without added noise. d Represents a dark image, X b Represents a true illuminated image, P(X) d ,X b () indicates that the lighting image has been initially restored.

[0018] The cross-scene action recognition method provided by this invention may also have the following features: In step S40, the spatiotemporal attention mechanism is based on a multi-head attention mechanism and is used to recover video frames of dark videos input to the dark-to-light conditional diffusion model. For the currently recovered frame j, the query is the feature of the currently recovered frame j. The value and key are reconstructed using the features of the first frame and the previous frame j-1 of the input dark video video frames. The calculation method of the spatiotemporal attention mechanism is as follows: z represents the feature representation after the spatiotemporal attention mechanism, and Q = Q j Let K represent the query matrix of the feature input for the j-th frame, T represent the matrix transpose, and K represent the matrix input. Τ This represents the transpose of the reconstructed key matrix. The key matrix in the spatiotemporal attention mechanism is concatenated from the key matrix of the first frame and the key matrix of the previous frame along the channel dimension. K0 represents the key matrix of the first frame, K j-1 This represents the key matrix of the previous frame. The scaling factor representing the attention mechanism. The value matrix under the spatiotemporal attention mechanism is formed by concatenating the value matrix of the first frame and the value matrix of the previous frame along the channel dimension. V0 represents the value matrix of the first frame, V j-1 This represents the value matrix of the previous frame. This indicates a merge operation.

[0019] The cross-scene action recognition method provided by the present invention may also have the following feature: in step S40, the preliminary restored illumination image is used as an additional control condition for the dark-to-bright conditional diffusion model, thereby enhancing the realism and effectiveness of illumination restoration.

[0020] The cross-scene action recognition method provided by the present invention may also have the following features: in steps S50 to S60, the spatiotemporal fusion module includes a separable 3D convolutional layer and a series of upsampling layers. The separable 3D convolutional layer is used to capture spatiotemporal information from the shallower residual module in the dual-channel action recognition backbone network structure. After the upsampling operation, the dot product with the input features is used to weight the extracted features according to their correlation.

[0021] The cross-scene action recognition method provided by the present invention may also have the following feature: in steps S50 to S60, the alignment module consists of a series of separable 3D convolutional layers to ensure that the feature size of the spatiotemporal fusion module matches the reference feature size, thereby calculating the loss between them.

[0022] The cross-scene action recognition method provided by this invention may also have the following feature: In step S60, the dual-channel action recognition backbone network structure, as a neural network model, has the following loss function: Represents classification loss. This indicates the loss relative to the label. This represents the loss for the feature, where α and λ are weighting coefficients.

[0023] The cross-scene action recognition method provided by this invention may also have the following feature: wherein, the classification loss Loss related to labels Loss for features and Together they constitute the self-distillation loss, h represents the cross-entropy loss. i This represents the output logit of the i-th self-distillation branch, where y represents the corresponding label and h represents the output logit. N+1 This indicates that the final output of the neural network model is logit. This represents the Kullback-Leibler divergence loss. F represents L2 loss. i F represents the predicted feature output of the i-th self-distillation branch. N+1 This represents the predicted features of the final output of the neural network model.

[0024] The role and effect of invention

[0025] This invention utilizes a diffusion model to restore dark video frames into continuous bright video frames. The restoration result closely approximates the video under real natural lighting, solving the low visibility problem of dark videos and enabling subsequent networks to extract effective motion information.

[0026] The present invention employs a specific spatiotemporal attention mechanism in the sampling stage of the dark-to-bright diffusion model, which effectively utilizes the temporal information of a continuous segment of sampled video frames, thereby ensuring the spatiotemporal consistency of the recovered bright video frames. Furthermore, the spatiotemporal attention mechanism can be seamlessly integrated into a diffusion model trained on image data, achieving excellent recovery of continuous bright video frames without requiring additional model training.

[0027] The dual-channel action recognition backbone network based on self-distillation branch designed in this invention can effectively extract and learn action features in video sequences. The distillation process makes the network more robust and generalizable, enhances the interaction between dark and lit video information, and improves the network's recognition accuracy. Attached Figure Description

[0028] Figure 1 This is a flowchart of a cross-scene action recognition method according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the spatiotemporal attention mechanism according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the sampling process of the dark-to-light conditional diffusion model according to an embodiment of the present invention;

[0031] Figure 4 This is a comparison image of the initially restored image generated by ControlNet after fine-tuning in the test example of this invention and the final output result of the dark-to-bright conditional diffusion model;

[0032] Figure 5 The image shows the results of the dark-to-light conditional diffusion model in the test examples of this invention on the ARID dark action recognition dataset and the Dark48 dark scene recognition dataset. Detailed Implementation

[0033] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of a cross-scene action recognition method of the present invention.

[0034] <Example>

[0035] Figure 1 This is a flowchart of a cross-scene action recognition method according to an embodiment of the present invention.

[0036] like Figure 1 As shown, this embodiment provides a cross-scene action recognition method to improve the network's ability to recognize actions in dark videos, including the following steps:

[0037] S10, using the low-light image to be recovered (dark image X)d As a condition, the corresponding real-world lighting image X b As a target, the learnable parts of the ControlNet model pre-trained on a large-scale normal lighting dataset are trained.

[0038] S20, sample the ControlNet model trained in step S10 to obtain the dark image and its corresponding preliminary restored illumination image P(X). d ,X b ).

[0039] S30, X dark image d The corresponding real lighting image X b And preliminary recovery of the illumination image P(X) d ,X b After concatenation along the channel dimension, the data is used as input, and a multi-head attention mechanism is employed to train the dark-to-light conditional diffusion model.

[0040] Since existing labeled dark motion recognition video datasets lack corresponding illuminated videos, it is impossible to directly construct a dark-to-light conditional diffusion model based on such datasets. To overcome this problem, this embodiment utilizes the ExLPose image dataset for low-light human pose estimation tasks.

[13] This data serves as training data for a dark-to-light conditional diffusion model.

[0041] The training loss function for the dark-to-light conditional diffusion model is:

[0042]

[0043] in, This represents a weighted average of the loss across the entire data distribution, where t represents the time step for progressive denoising in the diffusion model, θ represents the parameters optimized during model training, and ∈ t This represents the Gaussian noise added to the image by the diffusion model at time step t, ∈ θ This represents the noise predicted by the diffusion model at time step t. This represents the cumulative noise attenuation factor, where x0 represents the original image data, which is a real illuminated image without added noise. d Represents a dark image, X b Represents a true illuminated image, P(X) d ,X b () indicates that the lighting image has been initially restored.

[0044] S40, replace the multi-head attention mechanism with a spatiotemporal attention mechanism, and use the dark-to-light conditional diffusion model to sample video frames of the dark video, including the following sub-steps S41 to S42:

[0045] S41: The multi-head attention mechanism is replaced with a spatiotemporal attention mechanism, which is seamlessly integrated into the pre-trained dark-to-light conditional diffusion model. This mechanism requires no additional training and addresses the discontinuity between adjacent video frames caused by direct sampling in image-based diffusion models.

[0046] Figure 2 This is a schematic diagram of the spatiotemporal attention mechanism according to an embodiment of the present invention. Figure 2 As shown, the spatiotemporal attention mechanism, based on a multi-head attention mechanism, is used to recover video frames of dark videos input to a dark-to-light conditional diffusion model. The specific operation is as follows:

[0047] The input consists of a continuous video frame of length k. In the spatiotemporal attention mechanism, for the currently recovered frame j, the query is the feature of the current frame j. The first frame of the input dark video and the features of the previous frame j-1 are concatenated to reconstruct the value and key. Then, the spatiotemporal features of the current frame j are calculated through the multi-head attention computing mechanism.

[0048] The calculation method for the spatiotemporal attention mechanism is as follows:

[0049]

[0050] Where z represents the feature representation after the spatiotemporal attention mechanism, and Q = Q j Let K represent the query matrix of the feature input for the j-th frame, T represent the matrix transpose, and K represent the matrix input. Τ This represents the transpose of the reconstructed key matrix. The key matrix in the spatiotemporal attention mechanism is concatenated from the key matrix of the first frame and the key matrix of the previous frame along the channel dimension. K0 represents the key matrix of the first frame, K j-1 This represents the key matrix of the previous frame. The scaling factor representing the attention mechanism. The value matrix under the spatiotemporal attention mechanism is formed by concatenating the value matrix of the first frame and the value matrix of the previous frame along the channel dimension. V0 represents the value matrix of the first frame, V j-1 This represents the value matrix of the previous frame. This indicates a merge operation.

[0051] S42, using the preliminary restored illumination image P(X) d ,X b Noise and other data are concatenated dimensionally as additional control conditions input to the dark-to-bright conditional diffusion model. Large-scale bright training data is added as prior knowledge to the model to enhance the realism and effectiveness of illumination restoration. Following a spatiotemporal attention mechanism during sampling, video frames from dark videos are sampled from the dark-to-bright conditional diffusion model to obtain the restored continuous bright video frames X.e ,like Figure 3 As shown.

[0052] In this step, the spatiotemporal attention mechanism samples a continuous video frame and provides information from the initial frame and the previous frame for the current frame, in order to maintain the continuity of the sampled bright video frames and avoid the randomness and discontinuity of each frame in the video caused by operating on a single frame.

[0053] S50 is based on a dual-channel action recognition backbone network structure consisting of a dark input channel and a bright input channel. N self-distillation branches are constructed between its residual modules. Each self-distillation branch includes a spatiotemporal fusion module and an alignment module.

[0054] The spatiotemporal fusion module consists of separable 3D convolutional layers and a series of upsampling layers. The separable 3D convolutional layers capture spatiotemporal information from the shallower residual modules in the backbone network. After upsampling, the extracted features are weighted according to their correlation by taking a dot product with the input features.

[0055] The alignment module consists of a series of separable 3D convolutional layers to ensure that the feature size passed through the spatiotemporal fusion module matches the reference feature size, thereby calculating the loss between them.

[0056] In this step, the dual-channel action recognition backbone network structure is used as a neural network model, and its loss function... Including classification loss And self-distillation loss.

[0057] Classification loss It consists of a cross-entropy loss function, including the loss between the final output of the action recognition network and its corresponding label, as well as the loss between the output of each distillation branch and its corresponding label:

[0058]

[0059] in, h represents the cross-entropy loss. i This represents the output logit of the i-th self-distillation branch, where y represents the corresponding label and h represents the output logit. N+1 This indicates that the final output of the neural network model is logit.

[0060] Self-distillation losses include losses specific to the label. and feature-specific loss

[0061] Loss related to labels The Kullback-Leibler divergence loss is used to calculate the logit between the output of each distillation branch and the final output logit of the action recognition network:

[0062]

[0063] in, This represents the Kullback-Leibler divergence loss.

[0064] Loss for features L2 loss is used to calculate the difference between the predicted features output by each distillation branch and the predicted features output by the final action recognition network:

[0065]

[0066] in, F represents L2 loss. i F represents the predicted feature output of the i-th self-distillation branch. N+1 This represents the predicted features of the final output of the neural network model.

[0067] Finally, the loss function is obtained.

[0068]

[0069] Where α is the weighting coefficient of the Kullback-Leibler divergence loss, and λ is... The weighting coefficients.

[0070] S60 inputs the video frames of the dark video and the corresponding bright video frames into the dark input channel and the bright input channel, respectively. After the spatiotemporal fusion module and the alignment module extract features and calculate the loss, they are merged along the channel dimension. After merging, the output features are obtained through the self-attention module. The output features are output from the fully connected layer to the corresponding logit of the shallow residual module.

[0071] The Top-1 accuracy of the final output action recognition is then used as the quantitative evaluation index of the cross-scene action recognition method in this embodiment.

[0072] <Test Example>

[0073] This test case uses the cross-scene action recognition method from the embodiments for corresponding testing.

[0074] In this test case, the dark-to-light conditional diffusion model accepts a fixed-size 256×256 image as input, which consists of the original dark image X. d ControlNet initially recovered the illumination image P(X) d,X b ) and the corresponding real lighting image X b It is pieced together in the channel dimension.

[0075] When fine-tuning the ControlNet training, pairs of dark and lit images are used as input to the ControlNet, where the dark image X... d As a condition, the bright image X b As the target for generation, it is used to train the adaptive part of ControlNet to obtain a preliminary recovered illumination image P(X). d ,X b During fine-tuning, to avoid adding irrelevant information to the diffusion process, "Adetailed high-quality professional image" was used as the default prompt. The number of diffusion steps in the dark-to-light conditional diffusion model training process was set to 1000 steps, and 47,000 iterations were performed.

[0076] In the dark-to-light conditional diffusion model, the attention mechanism in UNet is replaced with a specific spatiotemporal attention mechanism in the embodiment during the sampling process. DDIM sampling is used and the diffusion step count is set to 100 steps. A 16-frame dark video segment is converted into a continuous bright video segment, which serves as the input to the subsequent dual-channel action recognition backbone network structure.

[0077] Figure 4 This is a comparison image of the initially restored image generated by ControlNet after fine-tuning in the test example of this invention and the final output result of the dark-to-bright conditional diffusion model.

[0078] like Figure 4 As shown, the dark-to-bright conditional diffusion model is significantly better than the initial recovery model for restoring details and colors in video frames.

[0079] Figure 5 The image shows the results of the dark-to-light conditional diffusion model in the test examples of this invention on the ARID dark action recognition dataset and the Dark48 dark scene recognition dataset.

[0080] like Figure 5 As shown, for input dark video frames, the dark-to-light conditional diffusion model can effectively recover the colors of the video frames with the help of a large amount of prior knowledge about illumination. At the same time, the spatiotemporal attention mechanism used in the sampling process can effectively maintain the continuity between the recovered video frames.

[0081] Based on a self-distillation branch and a dual-channel action recognition backbone network structure, the system accepts video clips of a fixed size of 112×112×3×64 as input. Both dark and bright videos are scaled and normalized to the above size, and video data augmentation is performed using multi-scale cropping, random horizontal flipping, and cutout operations. The dual-channel action recognition backbone network structure adopts an R(2+1)D-34 structure, inputting the dark video and the corresponding restored bright video into a weight-shared network. Three self-distillation branches are constructed between the residual modules, each consisting of a spatiotemporal fusion module and an alignment module. The spatiotemporal fusion module consists of 3D separable convolutional layers, 3D batch normalization layers, and ReLU activation function layers. It upsamples the extracted features and performs a dot product operation with the input features to calculate their correlation, and then performs weighted summation. The alignment module uses a series of 3D separable convolutional layers to ensure that the feature size passing through the spatiotemporal fusion module matches the reference feature size, thereby calculating the loss between them. The darkness and light information from the two channels are merged along the channel dimension after passing through the spatiotemporal fusion and alignment modules. The merged features are then passed through a self-attention module to obtain the output features. The output features are then output from the fully connected layer as the corresponding logit of the shallow residual module.

[0082] For motion recognition in dark scenes, recognition accuracy is used as the evaluation metric.

[0083] The specific experimental results of the cross-scene action recognition method in this test case on the ARID dataset and the Dark48 dataset are shown in Table 1 and Table 2 below, respectively.

[0084] Table 1 (Specific experimental results on the ARID dataset)

[0085]

[0086] Table 2 (Specific experimental results on the Dark48 dataset)

[0087] method <![CDATA[DTCM

[10] ]]> <![CDATA[DarkLight-ResNext101 [9] ]]> <![CDATA[DarkLight-R(2+1)D [9] ]]> The method in this embodiment Top-1 accuracy 46.68％ 42.27％ 39.08％ 47.14％ Top-5 accuracy 75.92％ 70.47％ 71.24％ 75.65％

[0088] As shown in Tables 1 and 2, the proposed cross-scene action recognition method in the embodiments has achieved state-of-the-art results on existing dark video recognition datasets, and has a significant performance improvement over the baseline results.

[0089] The role and effect of the embodiments

[0090] This embodiment utilizes a conditional diffusion model to enhance video frames with insufficient lighting, and combines it with the ControlNet model.

[12] By integrating prior knowledge and normal lighting information, and using a diffusion model, dark video frames are restored into continuous bright video frames. The restoration result is close to the video under real natural lighting, which solves the low visibility problem of dark videos and enables subsequent networks to extract effective motion information.

[0091] This embodiment employs a specific spatiotemporal attention mechanism in the sampling stage of the dark-to-bright diffusion model. This mechanism effectively utilizes the temporal information of a continuous segment of video frames being sampled, mitigating the discontinuities between adjacent video frames caused by direct sampling in image-based diffusion models. This ensures the spatiotemporal consistency of the recovered bright video frames. Furthermore, the spatiotemporal attention mechanism can be seamlessly integrated into diffusion models trained on image data, achieving excellent recovery of continuous bright video frames without requiring additional model training.

[0092] The dual-channel action recognition backbone network based on self-distillation branch designed in this embodiment can effectively extract and learn action features in video sequences. The distillation process makes the network more robust and generalizable, enhances the interaction between dark and lit video information, and improves the network's recognition accuracy.

[0093] This embodiment integrates the self-distillation branch with the dual-channel action recognition backbone network to extract weighted spatiotemporal features from different blocks of the backbone network. This encourages the self-distillation branch to focus on information of higher importance in dark videos, thereby promoting information interaction between different blocks in the action recognition backbone architecture and effectively improving the network's ability to recognize dark videos.

[0094] The test cases leverage expertise gained from the ExLPose dataset to enhance the performance of the dark-to-light diffusion model for action recognition tasks in dark scenes.

[0095] The test cases used the Top-1 accuracy of the model's final action recognition output as a quantitative evaluation metric for the cross-scene action recognition method. The cross-scene action recognition method of this embodiment was compared with other methods, verifying the advantages of this embodiment. Furthermore, the test cases also visualized the recovered bright video frames for visual qualitative evaluation, verifying that the model's recovery results in the test cases closely approximate videos under real natural lighting, while maintaining the continuity between adjacent video frames.

[0096]

[12] Zhang, L., Rao, A., & Agrawala, M. Adding conditional control to text-to-image diffusion models. In: ICCV (2023).

[0097]

[13] Lee, S., Rim, J., Jeong, B., Kim, G., Woo, B., Lee, H.,... & Kwak, S. Human poseestimation in extremely low-light conditions. In: CVPR (2023).

[0098] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A cross-scene action recognition method, characterized in that, To improve the network's ability to identify dark videos, the following steps are included: S10 uses dark images as conditions and corresponding real-light images as targets, and trains the learnable parts of the ControlNet model pre-trained on a large-scale normal-light dataset. S20, sample the ControlNet model trained in step S10 to obtain the dark image and its corresponding preliminary restored illumination image; S30, the dark image, the corresponding real lighting image and the preliminary restored lighting image are stitched together in the channel dimension and used as input, and a dark-to-bright conditional diffusion model is trained using a multi-head attention mechanism. S40, replace the multi-head attention mechanism with a spatiotemporal attention mechanism, and use the dark-to-light conditional diffusion model to sample the video frames of the dark video to generate continuous bright video frames; S50 is based on a dual-channel action recognition backbone network structure composed of dark input channel and bright input channel. N self-distillation branches are constructed between its residual modules. Each self-distillation branch includes a spatiotemporal fusion module and an alignment module. S60, the video frames of the dark video and the corresponding bright video frames are respectively input into the dark input channel and the bright input channel. After the spatiotemporal fusion module and the alignment module extract features and calculate loss, they are merged along the channel dimension. After merging, the output features are obtained through the self-attention module. The output features are output from the fully connected layer to the corresponding logit of the shallow residual module.

2. The cross-scene action recognition method according to claim 1, characterized in that: in, In step S30, the training loss function of the dark-to-light conditional diffusion model is: This represents a weighted average of the loss across the entire data distribution, where t represents the time step for progressive denoising in the diffusion model, θ represents the parameters optimized during model training, and ∈ t This represents the Gaussian noise added to the image by the diffusion model at time step t, ∈ θ This represents the noise predicted by the diffusion model at time step t. X represents the cumulative noise attenuation factor, x0 represents the original image data, which is the real illumination image without added noise, X d X represents the dark image. b P(X) represents the actual illuminated image. d ,X b The image represents the preliminary restored illumination image.

3. The cross-scene action recognition method according to claim 1, characterized in that: in, In step S40, the spatiotemporal attention mechanism, based on the multi-head attention mechanism, is used to recover video frames of the dark video input to the dark-to-light conditional diffusion model. For the currently recovered frame j, the query is the feature of the currently recovered frame j. The value and key are reconstructed using the features of the first frame and the previous frame j-1 of the input dark video frame. The calculation method of the spatiotemporal attention mechanism is as follows: z represents the feature representation after the spatiotemporal attention mechanism, and Q = Q j Let K represent the query matrix of the feature input for the j-th frame, T represent the matrix transpose, and K represent the matrix input. Τ This represents the transpose of the reconstructed key matrix. The key matrix in the spatiotemporal attention mechanism is concatenated from the key matrix of the first frame and the key matrix of the previous frame along the channel dimension. K0 represents the key matrix of the first frame, K j-1 This represents the key matrix of the previous frame. The scaling factor representing the attention mechanism. The value matrix under the spatiotemporal attention mechanism is formed by concatenating the value matrix of the first frame and the value matrix of the previous frame along the channel dimension. V0 represents the value matrix of the first frame, V j-1 This represents the value matrix of the previous frame. This indicates a merge operation.

4. The cross-scene action recognition method according to claim 1, characterized in that: in, In step S40, the preliminary restored illumination image is also used as an additional control condition for the dark-to-bright conditional diffusion model, thereby enhancing the realism and effectiveness of illumination restoration.

5. The cross-scene action recognition method according to claim 1, characterized in that: in, In steps S50 to S60, the spatiotemporal fusion module includes a separable 3D convolutional layer and a series of upsampling layers. The separable 3D convolutional layer is used to capture spatiotemporal information from the shallower residual modules in the dual-channel action recognition backbone network structure. After upsampling, the extracted features are weighted according to their correlation by the dot product of the input features.

6. The cross-scene action recognition method according to claim 5, characterized in that: in, In steps S50 to S60, the alignment module consists of a series of separable 3D convolutional layers to ensure that the feature size of the spatiotemporal fusion module matches the reference feature size, thereby calculating the loss between them.

7. The cross-scene action recognition method according to claim 1, characterized in that: in, In step S60, the dual-channel action recognition backbone network structure is used as a neural network model, and its loss function is: Represents classification loss, This indicates the loss relative to the label. This represents the loss for the feature, where α and λ are weighting coefficients.

8. The cross-scene action recognition method according to claim 7, characterized in that: in, Classification loss Loss related to labels Loss for features and Together they constitute the self-distillation loss. h represents the cross-entropy loss. i This represents the output logit of the i-th self-distillation branch, where y represents the corresponding label and h represents the output logit. N+1 This indicates that the final output of the neural network model is logit. This represents the Kullback-Leibler divergence loss. F represents L2 loss. i F represents the predicted feature output of the i-th self-distillation branch. N+1 This represents the predicted features of the final output of the neural network model.

Images