Action recognition video unsupervised domain adaptation method and system based on correlation information alignment

Abstract

The application belongs to the field of computer vision and cross-domain transfer learning, and specifically discloses a motion recognition video domain adaptation method and system based on relevant information alignment. The method extracts video features and high-level features of a source domain and a target domain through a shared three-dimensional convolutional encoder, generates a pixel correlation matrix and a correlation feature vector by using a correlation extraction module based on a non-local operation, sets a domain discriminator for the video features and the correlation features respectively to perform global adversarial alignment, further proposes a pixel correlation difference (PCD) to map the pixel correlation matrix to a reproducing kernel Hilbert space, and combines a source domain real label and a target domain pseudo label to realize fine-grained alignment of class-level correlation information distribution, and finally performs collaborative optimization on a motion classification loss, a dual-domain adversarial loss and a PCD loss. The application effectively improves the cross-domain motion recognition accuracy and can be applied to intelligent monitoring, human-computer interaction and other scenes.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, video understanding and cross-domain transfer learning technology, and particularly relates to an unsupervised domain adaptation method and system for action recognition videos based on relevant information alignment. Background Technology

[0002] Action recognition has wide applications in intelligent surveillance, human-computer interaction, and robot perception. Existing action recognition methods typically assume that training and testing data follow the same distribution and rely on a large amount of labeled video data for supervised training. However, in practical applications, changes in the video capture environment, actor poses, background scenes, and lighting conditions can cause significant domain shifts, leading to a significant performance degradation when the model is deployed across different scenes. To mitigate this problem, unsupervised domain adaptation methods have been proposed to improve recognition performance in the unlabeled target domain using labeled source domain videos.

[0003] Existing research on video domain adaptation mostly draws on the ideas of image domain adaptation, either by directly replacing it with a three-dimensional convolutional feature extractor to process video data, or by focusing on enhancing alignment effects in the temporal direction, such as applying attention mechanisms to different time segments or introducing auxiliary tasks such as segment order prediction, in order to improve the robustness of temporal representation.

[0004] However, the aforementioned existing technologies still have significant shortcomings. First, in addition to spatial and temporal features, videos also contain relevant features formed by long-range spatiotemporal dependencies, and existing methods do not specifically align these features. The paper points out that relevant features are highly correlated with action categories and have proven effective in supervised action recognition; however, the relevant information for the same action often differs significantly in different domains due to differences in actor appearance, posture, and scene. Without alignment, the transferability of video features will be affected. Second, existing cross-domain video benchmark datasets are generally small in scale and have limited domain differences, making it difficult to reflect the real-world domain shift problem in complex scenes such as low-light conditions.

[0005] Therefore, there is an urgent need for a new video domain adaptation method that can not only align the overall features and temporal information of the video, but also effectively align long-range spatiotemporal pixel-related information, and be supplemented with a more challenging cross-domain video benchmark to improve the transfer capability of action recognition models in complex cross-domain scenarios. Summary of the Invention

[0006] The technical problem to be solved by this invention is to design an unsupervised domain adaptation method and system for action recognition videos based on relevant information alignment. In the unsupervised video domain adaptation scenario, this method effectively aligns long-range spatiotemporal pixel-related information in cross-domain videos for action recognition tasks, while taking into account the overall distribution alignment of video features and the distribution alignment of category-related information, thereby improving the performance of cross-domain action recognition. Furthermore, this invention provides a benchmark scenario for video domain adaptation with a larger domain offset to verify the effectiveness of the method in complex environments, thus solving the existing technical problems.

[0007] To address the aforementioned technical problems, this invention provides an unsupervised domain adaptation method for action recognition videos based on relevant information alignment, specifically including the following steps:

[0008] Step S1: Obtain the training dataset and preprocess it to obtain source domain motion video and target domain motion video with the same number of sampled frames and screen resolution.

[0009] Step S2: Use a shared 3D convolutional encoder to extract features from the preprocessed source domain motion video and target domain motion video respectively, and output the low-level global video features and high-level video features.

[0010] Step S3: Process the high-level video features through the correlation extraction module to generate a pixel correlation matrix and a correlation feature vector.

[0011] Step S4: Construct a multi-branch joint loss calculation structure, and simultaneously calculate the action classification loss, dual-domain adversarial loss and pixel-related difference loss based on the low-level global video features and related feature vectors of the source domain and target domain.

[0012] Step S5: Jointly optimize the total loss function, which is obtained by weighted combination of action classification loss, video domain adversarial loss, correlation domain adversarial loss, and pixel correlation difference loss; iterate training until the model converges to obtain a cross-domain action recognition model that can be directly used in the unlabeled target domain.

[0013] In a further step of this invention, the specific process of generating the pixel correlation matrix in step S3 includes: using a 1×1×1 three-dimensional convolutional layer to perform linear mapping on the high-level video features respectively, performing a softmax operation on the mapping results, and multiplying them to obtain the pixel correlation matrix; interactively fusing the pixel correlation matrix with the mapped high-level video features, and outputting the correlation feature vector after spatiotemporal pooling.

[0014] In a further step of this invention, the method for calculating the action classification loss in step S4 is as follows: the source domain bottom-level global video features and the source domain related feature vectors are concatenated and input into the action classifier, and the action classification loss is calculated based on the real action labels of the source domain video.

[0015] In a further step of this invention, the method for calculating the dual-domain adversarial loss in step S4 is as follows: a video domain discriminator and a correlation domain discriminator are set up respectively, and a gradient reversal layer is added before the bottom-level global video features and correlation feature vectors are input to the corresponding discriminators; the video domain adversarial loss and the correlation domain adversarial loss are calculated based on the discriminator output respectively, and the shared three-dimensional convolutional encoder and the correlation extraction module learn domain-invariant features through minimax adversarial training.

[0016] In a further step of this invention, the pixel correlation difference loss (PCD) in step S4 is calculated as follows: the pixel correlation matrices of the source domain and the target domain are mapped to the regenerating kernel Hilbert space as relevant information distribution samples, and the distribution difference of the pixel correlation matrices of the source domain and the target domain under the same source action category is minimized based on the action category mean embedding metric; wherein the source domain video uses real action labels to calculate the category weights, and the target domain video uses action prediction pseudo labels output by the action classifier to calculate the category weights.

[0017] Furthermore, in this invention, the pixel correlation difference loss is calculated as follows: a Gaussian kernel function is used to expand the class mean embedding in the Hilbert space of the regenerated kernel, and the kernel distances between source domain samples, between target domain samples, and between source domain and target domain samples under the same action are calculated respectively. The weighted summation is then used to obtain the class-level pixel correlation distribution difference.

[0018] Furthermore, in this invention, the total loss function is expressed as follows: in, Action classification loss, For video domain countermeasures losses, To mitigate losses in related domains, For pixel-related difference loss; , and These are the balancing weights for video domain loss, correlation domain loss, and pixel correlation difference loss, respectively.

[0019] The present invention also provides an unsupervised domain adaptation system for action recognition videos based on relevant information alignment, for performing the aforementioned unsupervised domain adaptation method for action recognition videos based on relevant information alignment, comprising the following modules:

[0020] The video preprocessing module is used to receive labeled source domain action videos and unlabeled target domain action videos, and uniformly complete video frame sampling, image size scaling and cropping preprocessing.

[0021] The shared feature encoding module has a built-in shared 3D convolutional encoder, which is used to receive preprocessed video and extract low-level global video features and high-level video features.

[0022] The relevant feature extraction module is connected to the shared feature encoding module. It calculates high-level video features based on nonlocal spatiotemporal dependency operations and outputs pixel correlation matrix and relevant feature vector.

[0023] The joint loss calculation and model training module is connected to the shared feature encoding module and the relevant feature extraction module, respectively. The joint loss calculation and model training module includes:

[0024] The multi-branch loss calculation submodule is used to simultaneously calculate action classification loss, dual-domain adversarial loss, and pixel-related difference loss;

[0025] The model training submodule is used to construct the total loss function by weighted fusion of four types of losses, iteratively update the network parameters in reverse, and output the trained cross-domain action recognition model.

[0026] Furthermore, in this invention, the multi-branch loss calculation submodule includes:

[0027] The classification unit is used to concatenate the source domain's bottom-level global video features with relevant feature vectors and calculate the action classification loss.

[0028] The dual adversarial unit has a built-in video domain discriminator, a correlation domain discriminator and a gradient inversion layer. It performs domain discrimination on the underlying global video features and correlation feature vectors respectively, and outputs video domain adversarial loss and correlation domain adversarial loss.

[0029] The PCD loss calculation unit is used to map the pixel correlation matrix to the regeneration kernel Hilbert space and calculate the category-level pixel correlation difference loss based on the source domain true label and the target domain pseudo label.

[0030] Furthermore, the system in this invention also includes a dataset construction module, which is used to fuse HMDB51 regular lighting action videos and ARID low-light action videos to construct an HMDB-ARID large-domain offset cross-domain evaluation dataset, and call the trained cross-domain action recognition model to complete the low-light scene action recognition performance test.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The unsupervised domain adaptation method for action recognition video based on relevant information alignment of the present invention extracts the overall video features and relevant features of both the source domain video and the target domain video simultaneously on the basis of a shared 3D convolutional video encoder. Adversarial domain discriminators are set for the video features and relevant features respectively, so that the feature extractor learns transferable features under the supervision of action classification. At the same time, the global distribution difference between the source domain and the target domain is reduced through adversarial training, which overcomes the defect of existing methods that only align spatial and temporal features and ignore relevant features, and significantly improves the transferability of cross-domain video features.

[0033] This invention constructs pixel correlation matrices and related feature vectors from high-level video features through a correlation extraction module. The pixel correlation matrix represents the long-range dependencies between pixel features at different spatiotemporal locations. Based on this pixel correlation matrix, instead of directly minimizing the differences between related Gram matrices, this invention proposes Pixel Correlation Difference (PCD) (used to measure and align the class-level distribution differences in pixel-related information between the source and target domains). This maps the related information distributions to a regenerating kernel Hilbert space and performs fine-grained alignment of the related information distributions between the source and target domains at the class level. This improves the effectiveness of correlation alignment and reduces the computational overhead and discriminative degradation caused by direct Gram matrix matching. Compared to existing schemes that directly minimize Gram matrix differences, this invention achieves more effective related information distribution alignment while maintaining network discriminativeness, and also has higher computational efficiency.

[0034] This invention combines action classification loss, video domain adversarial loss, correlation domain adversarial loss, and pixel correlation difference loss for joint optimization, thereby achieving coordinated alignment of the overall video representation and related information representation, and further improving the accuracy of cross-domain action recognition. Attached Figure Description

[0035] The specific embodiments of the present invention will be further explained below with reference to the accompanying drawings.

[0036] Figure 1 This is a flowchart of the action recognition video unsupervised domain adaptation method based on relevant information alignment according to the present invention.

[0037] Figure 2 This is a system block diagram of the action recognition video unsupervised domain adaptation system based on relevant information alignment according to the present invention. Detailed Implementation

[0038] Example 1

[0039] Combination Figure 1 The action recognition video unsupervised domain adaptation method based on relevant information alignment in this embodiment specifically includes the following steps:

[0040] Step S1: Obtain the training dataset and perform preprocessing to obtain source domain motion videos and target domain motion videos with the same number of sampled frames and screen resolution. Specifically, in this embodiment, the training dataset includes a set of labeled source domain motion videos and a set of unlabeled target domain motion videos; uniform preprocessing is performed on the source domain motion videos and target domain motion videos to unify the number of sampled frames and screen resolution.

[0041] Step S2: Use a shared 3D convolutional encoder to extract features from the preprocessed source domain motion video and target domain motion video respectively, and output the low-level global video features and high-level video features.

[0042] Step S3: The high-level video features are processed by the correlation extraction module to generate a pixel correlation matrix and a correlation feature vector. Specifically, in this embodiment, the correlation extraction module is constructed based on nonlocal spatiotemporal dependency operations to characterize the long-range correlation between pixels in different spatiotemporal regions within the video.

[0043] In this embodiment, preferably, the specific process of generating the pixel correlation matrix in step S3 includes: using a 1×1×1 three-dimensional convolutional layer to perform linear mapping on the high-level video features respectively, performing softmax operation on the mapping results, and multiplying them to obtain the pixel correlation matrix; interactively fusing the pixel correlation matrix with the mapped high-level video features, and outputting the correlation feature vector after spatiotemporal pooling.

[0044] In this embodiment, the same number of consecutive frames are sampled for both the source domain video and the target domain video, and the resolution is uniformly adjusted before being input into a shared three-dimensional convolutional encoder to extract video features. Then, from the higher-level video features output from the deeper layers of the encoder, a pixel correlation matrix and a correlation feature vector are constructed through a correlation extraction module. The pixel correlation matrix is ​​used to represent the correlation between the spatiotemporal pixel positions.

[0045] Step S4: Construct a multi-branch joint loss calculation structure, and simultaneously calculate the action classification loss, dual-domain adversarial loss and pixel-related difference loss based on the low-level global video features and related feature vectors of the source domain and target domain.

[0046] In this embodiment, preferably, in step S4, the method for calculating the action classification loss is as follows: the source domain bottom-level global video features and the source domain related feature vectors are concatenated and input into the action classifier, and the action classification loss is calculated based on the real action labels of the source domain video.

[0047] In this embodiment, preferably, in step S4, the method for calculating the dual-domain adversarial loss is as follows: a video domain discriminator and a correlation domain discriminator are set up respectively, and a gradient reversal layer is added before the bottom-level global video features and correlation feature vectors are input to the corresponding discriminators; the video domain adversarial loss and the correlation domain adversarial loss are calculated based on the discriminator output respectively, and the shared three-dimensional convolutional encoder and the correlation extraction module learn domain-invariant features through minimax adversarial training.

[0048] In this embodiment, the gradient inversion layer works as follows: during forward propagation, the input features remain unchanged, and during backward propagation, the gradient is inverted, so that the feature extraction network and the two-class discriminator form an adversarial game training.

[0049] In this embodiment, the source domain video features and the source domain related feature vectors are concatenated and input into the action classifier to calculate the classification loss. At the same time, a video domain discriminator and a related domain discriminator are constructed to perform domain discrimination on the video features and related features. The encoder and the related extraction module learn more domain-invariant representations through a gradient inversion layer.

[0050] In this embodiment, preferably, in step S4, the pixel correlation difference loss (PCD) is calculated as follows: the pixel correlation matrices of the source domain and the target domain are mapped to the regenerating kernel Hilbert space as relevant information distribution samples, and the distribution difference of the pixel correlation matrices of the source domain and the target domain under the same source action category is minimized based on the action category mean embedding metric; wherein the source domain video uses real action labels to calculate the category weights, and the target domain video uses action prediction pseudo labels output by the action classifier to calculate the category weights.

[0051] In this embodiment, the pixel correlation difference loss is preferably calculated as follows: a Gaussian kernel function is used to expand the class mean embedding in the Hilbert space of the regenerated kernel, and the kernel distances between source domain samples, between target domain samples, and between source domain and target domain samples under the same action are calculated respectively. The weighted summation is then used to obtain the class-level pixel correlation distribution difference.

[0052] In this embodiment, the pixel correlation matrix is ​​specifically treated as a sample of relevant information distribution, and a measure of the intra-class distribution difference between the source and target domains is established based on the category. The source domain category weights are directly given by the true labels, while the target domain category weights are determined by the pseudo-label probabilities output by the action classifier. Then, the difference in pixel correlation distribution between the source and target domains under each category is calculated using kernel function expansion to form the PCD loss.

[0053] In this embodiment, the target domain video category weights are generated by inputting the target domain splicing features into the action classifier and outputting the predicted probability of each action category as the corresponding category weight.

[0054] Step S5: Jointly optimize the total loss function, which is obtained by weighted combination of action classification loss, video domain adversarial loss, correlation domain adversarial loss, and pixel correlation difference loss; iterate training until the model converges to obtain a cross-domain action recognition model that can be directly used in the unlabeled target domain.

[0055] In this embodiment, preferably, the total loss function expression is: in, Action classification loss, For video domain countermeasures losses, To mitigate losses in related domains, For pixel-related difference loss; , and These are the balancing weights for video domain loss, correlation domain loss, and pixel correlation difference loss, respectively.

[0056] The action recognition video unsupervised domain adaptation method based on relevant information alignment in this embodiment is applicable to action recognition domain adaptation scenarios across normal lighting and dark environment conditions, and can be combined with HMDB-ARID data construction methods for model training and evaluation in low-light large-domain offset scenarios.

[0057] To verify the effectiveness of this invention in scenarios with significant domain offset, this embodiment constructs a cross-domain video dataset—the HMDB-ARID dataset: Eleven action categories from ARID are selected, namely drink, jump, pick, pour, push, run, sit, stand, turn, walk, and wave, and corresponding action categories with the same names are selected from HMDB51. The cross-domain datasets are constructed according to the official training / validation partitions of each dataset. HMDB-ARID contains 770 training videos and 330 validation videos from HMDB51, and 2288 training videos and 823 validation videos from ARID. This dataset exhibits greater statistical differences and a larger domain offset compared to existing video domain adaptation datasets.

[0058] To verify the effectiveness of the present invention, experiments were conducted on two video domain adaptation datasets, UCF-HMDBfull and HMDB-ARID.

[0059] The UCF-HMDBfull dataset includes two setup directions: UCF→HMDB and HMDB→UCF. Using I3D as the video encoder, this invention achieves a Top-1 accuracy of 85.4% in the UCF→HMDB setup and 93.8% in the HMDB→UCF setup. Compared to the lower bound performance of training only with source domain data (Source Only, 80.3% and 88.8% respectively), this invention represents improvements of 5.1 and 5.0 percentage points, respectively. Compared to the existing method SAVA (82.2% and 91.2% respectively), this invention represents improvements of 3.2 and 2.6 percentage points, respectively.

[0060] When using MFNet as the video encoder, this invention achieves a Top-1 accuracy of 85.8% under the UCF → HMDB setting and a Top-1 accuracy of 93.2% under the HMDB → UCF setting.

[0061] On the HMDB-ARID dataset, using MFNet as the video encoder, this invention achieves a Top-1 accuracy of 58.0% in the HMDB→ARID setting and 46.4% in the ARID→HMDB setting. Compared to the lower bound performance of training only with source domain data (Source Only, 48.3% and 37.9%, respectively), this invention improves performance by 9.7 percentage points and 8.5 percentage points, respectively. Compared to the existing method TA3N (22.4% and 19.8%, respectively), this invention significantly outperforms it. Compared to other typical domain adaptation methods such as DANN (50.7% / 40.6%), MK-MMD (50.2% / 40.1%), MCD (47.6% / 36.8%), and CORAL (51.3% / 41.7%), this invention achieves the best results in all these areas.

[0062] Example 2

[0063] Combination Figure 2 As shown, the action recognition video unsupervised domain adaptation system based on relevant information alignment in this embodiment is used to execute the action recognition video unsupervised domain adaptation method based on relevant information alignment in Embodiment 1, and specifically includes the following modules:

[0064] The video preprocessing module is used to receive labeled source domain action videos and unlabeled target domain action videos, and uniformly complete video frame sampling, image size scaling and cropping preprocessing.

[0065] The shared feature encoding module has a built-in shared 3D convolutional encoder, which is used to receive preprocessed video and extract low-level global video features and high-level video features.

[0066] The relevant feature extraction module, connected to the shared feature encoding module, calculates high-level video features based on nonlocal spatiotemporal dependency operations, and outputs a pixel correlation matrix and relevant feature vectors. Specifically, in this embodiment, the relevant feature extraction module includes two 1×1×1 three-dimensional convolutional mapping layers, a softmax operation layer, and a spatiotemporal pooling layer, which sequentially complete the linear mapping of high-level features, the calculation of the pixel correlation matrix, and the generation of relevant feature vectors.

[0067] The joint loss calculation and model training module is connected to the shared feature encoding module and the relevant feature extraction module, respectively. The joint loss calculation and model training module includes:

[0068] The multi-branch loss calculation submodule is used to simultaneously calculate action classification loss, dual-domain adversarial loss, and pixel-related difference loss;

[0069] The model training submodule is used to construct the total loss function by weighted fusion of four types of losses, iteratively update the network parameters in reverse, and output the trained cross-domain action recognition model.

[0070] In this embodiment, preferably, the multi-branch loss calculation submodule includes:

[0071] The classification unit is used to concatenate the source domain's bottom-level global video features with relevant feature vectors and calculate the action classification loss.

[0072] The dual adversarial unit has a built-in video domain discriminator, a correlation domain discriminator and a gradient inversion layer. It performs domain discrimination on the underlying global video features and correlation feature vectors respectively, and outputs video domain adversarial loss and correlation domain adversarial loss.

[0073] The PCD loss calculation unit is used to map the pixel correlation matrix to the regeneration kernel Hilbert space and calculate the category-level pixel correlation difference loss based on the source domain true label and the target domain pseudo label.

[0074] In this embodiment, preferably, the system further includes a dataset construction module, which is used to fuse HMDB51 regular lighting action videos and ARID low-light action videos to construct an HMDB-ARID large-domain offset cross-domain evaluation dataset, and call the trained cross-domain action recognition model to complete the low-light scene action recognition performance test.

[0075] Many specific details have been set forth in the foregoing description to provide a thorough understanding of the present invention. However, the above description is merely a preferred embodiment of the present invention, and the present invention can be implemented in many other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed above. Furthermore, any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention, or modify them into equivalent embodiments, using the methods and techniques disclosed above, without departing from the scope of the present invention. Any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. An unsupervised domain adaptation method for action recognition videos based on relevant information alignment, characterized in that: Includes the following steps: Step S1: Obtain the training dataset and preprocess it to obtain source domain motion video and target domain motion video with the same number of sampled frames and screen resolution; Step S2: Use a shared 3D convolutional encoder to extract features from the preprocessed source domain motion video and target domain motion video respectively, and output the low-level global video features and high-level video features; Step S3: Process the high-level video features through the correlation extraction module to generate a pixel correlation matrix and a correlation feature vector; Step S4: Construct a multi-branch joint loss calculation structure, and simultaneously calculate action classification loss, dual-domain adversarial loss and pixel-related difference loss based on the low-level global video features and related feature vectors of the source domain and target domain; Step S5: Jointly optimize the total loss function, which is obtained by weighted combination of action classification loss, video domain adversarial loss, correlation domain adversarial loss, and pixel correlation difference loss; iterate training until the model converges to obtain a cross-domain action recognition model that can be directly used in the unlabeled target domain.

2. The action recognition video unsupervised domain adaptation method based on relevant information alignment according to claim 1, characterized in that: In step S3, the specific process of generating the pixel correlation matrix by the correlation extraction module includes: using a 1×1×1 three-dimensional convolutional layer to perform linear mapping on the high-level video features respectively, performing softmax operation on the mapping results, and multiplying them to obtain the pixel correlation matrix; interactively fusing the pixel correlation matrix with the mapped high-level video features, and outputting the correlation feature vector after spatiotemporal pooling.

3. The action recognition video unsupervised domain adaptation method based on relevant information alignment according to claim 1, characterized in that: In step S4, the action classification loss is calculated as follows: the source domain bottom-level global video features and the source domain related feature vectors are concatenated and input into the action classifier, and the action classification loss is calculated based on the real action labels of the source domain video.

4. The action recognition video unsupervised domain adaptation method based on relevant information alignment according to claim 1, characterized in that: In step S4, the dual-domain adversarial loss is calculated as follows: a video domain discriminator and a correlation domain discriminator are set up respectively, and a gradient reversal layer is added before the bottom-level global video features and correlation feature vectors are input to the corresponding discriminators; the video domain adversarial loss and the correlation domain adversarial loss are calculated based on the discriminator output respectively, and the shared three-dimensional convolutional encoder and the correlation extraction module learn domain-invariant features through minimax adversarial training.

5. The action recognition video unsupervised domain adaptation method based on relevant information alignment according to claim 1, characterized in that: In step S4, the pixel correlation difference loss (PCD) is calculated as follows: the pixel correlation matrices of the source and target domains are mapped to the regenerating kernel Hilbert space as relevant information distribution samples. The distribution difference of the pixel correlation matrices of the source and target domains under the same source action category is minimized based on the action category mean embedding metric. The source domain video uses real action labels to calculate the category weights, and the target domain video uses action prediction pseudo-labels output by the action classifier to calculate the category weights.

6. The action recognition video unsupervised domain adaptation method based on relevant information alignment according to claim 5, characterized in that: The pixel correlation difference loss is calculated as follows: a Gaussian kernel function is used to expand the class mean embedding in the Hilbert space of the regenerated kernel, and the kernel distances between source domain samples, between target domain samples, and between source and target domain samples under the same action are calculated respectively. The weighted sum is then used to obtain the class-level pixel correlation distribution difference.

7. The action recognition video unsupervised domain adaptation method based on relevant information alignment according to claim 1, characterized in that: The expression for the total loss function is as follows: in, Action classification loss, To mitigate losses in the video domain, To mitigate losses in related domains, For pixel-related difference loss; , and These are the balancing weights for video domain loss, correlation domain loss, and pixel correlation difference loss, respectively.

8. An unsupervised domain adaptation system for action recognition video based on relevant information alignment, characterized in that: The method for performing the action recognition video unsupervised domain adaptation method based on relevant information alignment as described in any one of claims 1-7 includes the following modules: The video preprocessing module is used to receive labeled source domain action videos and unlabeled target domain action videos, and uniformly complete video frame sampling, image size scaling and cropping preprocessing; The shared feature encoding module has a built-in shared 3D convolutional encoder, which is used to receive preprocessed video and extract low-level global video features and high-level video features. The relevant feature extraction module is connected to the shared feature encoding module. It calculates high-level video features based on nonlocal spatiotemporal dependency operations and outputs pixel correlation matrix and relevant feature vector. The joint loss calculation and model training module is connected to the shared feature encoding module and the relevant feature extraction module, respectively. The joint loss calculation and model training module includes: The multi-branch loss calculation submodule is used to simultaneously calculate action classification loss, dual-domain adversarial loss, and pixel-related difference loss; The model training submodule is used to construct the total loss function by weighted fusion of four types of losses, iteratively update the network parameters in reverse, and output the trained cross-domain action recognition model.

9. The action recognition video unsupervised domain adaptation system based on relevant information alignment according to claim 8, characterized in that: The multi-branch loss calculation submodule includes: The classification unit is used to concatenate the source domain's bottom-level global video features with relevant feature vectors and calculate the action classification loss. The dual adversarial unit has a built-in video domain discriminator, a correlation domain discriminator and a gradient inversion layer. It performs domain discrimination on the underlying global video features and correlation feature vectors respectively, and outputs video domain adversarial loss and correlation domain adversarial loss. The PCD loss calculation unit is used to map the pixel correlation matrix to the regeneration kernel Hilbert space and calculate the category-level pixel correlation difference loss based on the source domain true label and the target domain pseudo label.

10. The action recognition video unsupervised domain adaptation system based on relevant information alignment according to claim 8, characterized in that: The system also includes a dataset construction module, which is used to fuse HMDB51 regular lighting motion videos and ARID low-light motion videos to construct an HMDB-ARID large-domain offset cross-domain evaluation dataset, and call the trained cross-domain motion recognition model to complete the motion recognition performance test in low-light scenes.

Images