Passive video domain adaptation method and system based on attention temporal consistency network

Abstract

The application belongs to the technical field of computer vision, video understanding and cross-domain transfer learning, and specifically discloses a passive video domain adaptation action recognition method and system based on an attention time consistent network. The application only uses a pre-trained source model and unlabelled target videos to complete adaptation, freezes the source classifier throughout the whole process and does not access the original videos of the source domain; by constructing double time sequence constraint unified local time sequence feature semantics of feature consistency and source prediction consistency, realizing time sequence feature attention weighted aggregation according to classification prediction confidence to generate local weights, and combining information maximization loss and clustering pseudo label to complete self-supervised optimization, the application effectively aligns target time sequence representation and source domain distribution without target domain labeling, and the accuracy is higher than that of existing passive domain adaptation schemes on multiple cross-domain action recognition data sets, and is suitable for private sensitive action recognition scenes such as dark light monitoring and human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, video understanding and cross-domain transfer learning technology, specifically relating to human action recognition technology under passive video domain adaptation, and particularly to a passive video domain adaptation action recognition method and system based on attention temporal consistency network. Background Technology

[0002] Video action recognition has wide applications in intelligent surveillance, behavior analysis, and human-computer interaction. With the development of deep learning and large-scale video datasets, video action recognition models have made significant progress under supervised conditions. However, video annotation is costly, and models are often affected by domain shift factors such as scene differences and acquisition conditions when deployed to new environments, leading to a significant performance degradation.

[0003] To address the performance degradation issue across domains, existing unsupervised domain adaptation methods for videos propose utilizing labeled source domain videos and unlabeled target domain videos. During the adaptation phase, both domains are accessed simultaneously to reduce domain differences through adversarial learning, statistical distribution alignment, or temporal attention mechanisms, thereby improving the target domain action recognition performance. However, these existing techniques have significant shortcomings. On one hand, these methods generally rely on the continued accessibility of the source domain video during the adaptation phase, but video data often contains sensitive information such as human behavior and scene environment, making long-term open access unsuitable in many practical applications. On the other hand, most existing passive domain adaptation methods are designed for images, processing only spatial features, and are difficult to directly transfer to video tasks. Compared to images, videos have an additional temporal dimension. Alignment based solely on the overall video level or static feature level cannot guarantee semantic consistency between different local temporal features, ultimately leading to insufficient discriminative power of the overall temporal features.

[0004] Existing unsupervised video domain adaptation methods typically require continuous access to source domain video data during the adaptation process to achieve model transfer by reducing the statistical distribution difference between the source and target domains. However, in practical applications, source domain videos often contain sensitive information such as actor behavior and scene content. Continuous access to source domain data poses significant data privacy risks, thus existing methods are insufficiently applicable in privacy-sensitive scenarios. Therefore, passive video domain adaptation scenarios, where source data is inaccessible and only a trained source model and unlabeled target domain videos are provided, are more in line with practical needs. However, most existing passive domain adaptation methods are designed for image tasks, primarily dealing with static spatial features, and are difficult to directly apply to video data. Video data has significant temporal and multimodal characteristics; simply enhancing the discriminative power of individual video features is insufficient to guarantee the effectiveness of the overall video representation. If there is a lack of consistency among local temporal features, the final aggregated overall temporal features may still be semantically ambiguous and lack discriminative power.

[0005] Therefore, there is an urgent need for a new passive video domain adaptation method that can model temporal consistency at the level of local temporal features without accessing source domain video data, and highlight more reliable local temporal information through attention mechanisms, so as to achieve effective transfer of target domain video features to source domain distribution. Summary of the Invention

[0006] The technical problem to be solved by this invention is to design a passive video domain adaptive action recognition method and system based on attention temporal consistency network. Without accessing the source domain video data, it makes full use of the temporal information in the target video to construct the overall temporal features of the target domain that match the distribution of the source domain and have good discriminativeness, thereby improving the action recognition performance under passive video domain adaptation and solving the existing technical problems.

[0007] To address the aforementioned technical problems, this invention provides a passive video domain adaptive action recognition method based on attention temporal consistency networks, specifically comprising the following steps: Step S1: Source model pre-training, wherein the source model includes at least a spatial feature extractor, a temporal feature extractor, and a classifier; the temporal feature extractor extracts multiple local temporal features by modeling the relationships between different numbers of temporally sampled frame segments in the video, and further aggregates them into overall temporal features; Step S2: Passive Adaptation in the Target Domain; When the passive adaptation step in the target domain is executed, the source domain labeled video data is no longer accessed. Only the pre-trained source model and the unlabeled target domain video are loaded, and the classifier parameters of the source model are fixed throughout the adaptation process.

[0008] Furthermore, in this invention, the passive adaptive targeting of the target domain specifically includes the following steps: Step S21: Initialization: Initialize the target domain spatial feature extractor and the target domain temporal feature extractor using the parameters of the source model's spatial feature extractor and temporal feature extractor; Step S22: Extract local temporal features: Sample temporal segments of the target video and obtain multiple sets of local temporal features through the target domain temporal feature extractor; Step S23: Construct temporal consistency constraints: Simultaneously apply feature consistency constraints and source prediction consistency constraints to the multiple sets of local temporal features; the feature consistency constraints are used to make the feature representations of different local temporal features tend to be consistent; the source prediction consistency constraints are used to make the predicted distributions of different local temporal features output by a fixed classifier tend to be consistent. Step S24: Attention weighted aggregation: Generate corresponding local weights based on the prediction confidence obtained by inputting the local temporal features of each group into a fixed classifier, and use the local weights to weight and aggregate the local temporal features of each group to obtain the weighted overall temporal features; Step S25: Self-supervised optimization: Construct a self-supervised loss function based on the weighted overall temporal features, and backpropagate to update the parameters of the target domain spatial feature extractor and the target domain temporal feature extractor to complete passive video domain adaptation; After adaptation, the optimized target network is combined with a fixed classifier to output the action recognition results of the target video.

[0009] Furthermore, in this invention, the source model pre-training includes: training the source model using labeled source domain videos, and using labeled smooth cross-entropy loss as the training loss; the temporal feature extractor is a temporal relation network (TRN), and the spatial feature extractor is a 2D residual network (ResNet).

[0010] In a further step of this invention, in step S23, the feature consistency constraint is implemented as follows: after normalizing any two sets of local time features, the cross-correlation matrix is ​​calculated, and the cross-correlation matrix is ​​constrained to approximate the identity matrix to construct the feature consistency loss; when constructing the feature consistency loss, balanced weights are set for the diagonal and off-diagonal terms of the cross-correlation matrix for joint optimization.

[0011] Furthermore, in step S23 of this invention, the source prediction consistency constraint includes a local prediction consistency sub-constraint and an overall prediction consistency sub-constraint. The local prediction consistency sub-constraint is to input each local temporal feature into a fixed classifier to obtain a local prediction distribution, and minimize the KL divergence between each local prediction distribution and the average local prediction distribution. The overall prediction consistency sub-constraint is to aggregate the mean of local time features to obtain ordinary overall time features and input them into a fixed classifier to obtain the overall prediction distribution, minimizing the difference between the overall prediction distribution and the average local prediction distribution. By setting balance coefficients for the local prediction consistency sub-constraint and the global prediction consistency sub-constraint respectively, the source prediction consistency loss is obtained.

[0012] In a further step of this invention, in step S24, the prediction confidence is calculated from the information entropy of the prediction distribution corresponding to the local time features, and the local weight calculation formula is: local weight = 1 + prediction confidence.

[0013] In a further step of this invention, in step S25, the self-supervised loss function includes temporal consistency loss, information maximization loss, and pseudo-label cross-entropy loss; the temporal consistency loss is obtained by weighted fusion of feature consistency loss and source prediction consistency loss; the information maximization loss is used to improve the certainty of single-sample prediction and the balance of global category distribution; the pseudo-label cross-entropy loss is constructed based on pseudo-labels generated by weighted overall temporal feature clustering.

[0014] Furthermore, in this invention, pseudo-labels are generated through two rounds of iterative K-means clustering: first, the cluster centers are initialized based on the globally weighted overall time features to obtain initial pseudo-labels, and then the cluster centers are updated using the initial pseudo-labels, and the final pseudo-labels are obtained by matching the cosine distance.

[0015] This invention also provides a passive video domain adaptive action recognition system based on attention temporal consistency networks, used to execute the aforementioned passive video domain adaptive action recognition method based on attention temporal consistency networks, specifically including the following modules: The source model pre-training module is used to train a source model that includes a spatial feature extractor, a temporal feature extractor, and a classifier. The target domain feature extraction module loads the source model weights to initialize the target domain extractor, fixes the classifier parameters, and extracts multiple sets of local temporal features from unlabeled target videos. The adaptive optimization module incorporates a constraint calculation unit, an attention weighting unit, and a self-supervised training unit. The constraint calculation unit calculates the temporal consistency loss corresponding to feature consistency constraints and source prediction consistency constraints. The attention weighting unit generates local weights based on the classification prediction confidence of local temporal features and aggregates them to obtain weighted overall temporal features. The self-supervised training unit constructs a self-supervised loss based on the weighted overall temporal features, updates the target domain space and temporal feature extractor parameters, and outputs an adapted action recognition model.

[0016] Furthermore, the present invention also includes an inference output module, which loads the adapted action recognition model, inputs the video to be recognized, and outputs the video action category recognition result.

[0017] This invention presents a passive video domain adaptive action recognition method based on attention-temporally consistent networks. This method uses a source model as its initialization basis and adapts to the target domain solely based on unlabeled videos. The invention proposes a cross-temporal assumption: local temporal features extracted from different segments within the same video should not only be discriminative individually but also consistent with each other and exhibit similar feature distribution patterns, thus ensuring the effectiveness of the aggregated overall temporal features. This invention designs two types of consistency constraints around local temporal features: feature consistency and source prediction consistency. Feature consistency ensures consistency and low redundancy by constraining the cross-correlation matrix between different local temporal features to approximate the identity matrix. Source prediction consistency ensures that the prediction results of local temporal features on a fixed source classifier are close to each other and further consistent with the overall temporal feature prediction, making the target features better approximate the source domain distribution. This invention designs a local weight module that assigns different weights to different local temporal features based on their prediction confidence on the source classifier, thereby focusing on local temporal features that are closer to the source domain distribution during the construction of the overall temporal features. Building upon the above, this invention further combines information maximization loss and a self-supervised pseudo-label generation mechanism to improve the global diversity and class discriminative power of the overall temporal features of the target domain. Under the condition that source domain data is inaccessible, this invention achieves effective adaptation to target video features, and compared with existing technologies, it has the following beneficial effects: (1) It solves the problem of source domain data privacy leakage. The model adaptation process does not require access to or transmission of the original source domain video. It only uses the pre-trained weights exported offline, which completely avoids the leakage of sensitive information about people and scenes in the video and meets the requirements for use in privacy compliance scenarios.

[0018] (2) Adapting to the inherent characteristics of video temporal sequence and solving the adaptation defects of passive domain in images. The first dual temporal consistency constraint (feature consistency + source prediction consistency) is created to force the semantic unity of local temporal features of different temporal segments of the same video, and solves the problem of insufficient global feature discriminability after aggregation from the root.

[0019] (3) Attention mechanism to select effective temporal segments. Based on the prediction confidence of the source classifier, local temporal features are assigned differentiated weights, which automatically weakens noise and invalid segments, and focuses on retaining effective action segments that are close to the distribution of the source domain, thereby improving the cross-domain representation alignment effect. (4) Multi-layer self-supervised joint optimization, no manual labeling of the target domain is required. It integrates three types of self-supervised loss: temporal consistency, information maximization, and cluster pseudo-label, to improve the clustering effect of feature categories without any target labels.

[0020] (5) Excellent generalization performance. In cross-domain tasks of the three major public cross-domain action recognition datasets, UCF-HMDBfull, Daily-DA and Sports-DA, it outperforms existing passive domain adaptation methods for images in all cases, and the accuracy of some tasks exceeds that of traditional video domain adaptation schemes that require source domain data; the robustness is significantly improved on large domain offset datasets such as low-light ARID. Attached Figure Description

[0021] The specific embodiments of the present invention will be further explained below with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart of the passive video domain adaptive action recognition method based on attention temporal consistency network of the present invention.

[0023] Figure 2 This is a flowchart of the passive adaptive step in the passive video domain adaptation action recognition method based on attention temporal consistency network of the present invention.

[0024] Figure 3 This is a system block diagram of the passive video domain adaptive action recognition system based on attention temporal consistency network of the present invention. Detailed Implementation Example 1

[0025] Combination Figure 1 As shown, the passive video domain adaptive action recognition method based on attention temporal consistency network in this embodiment specifically includes the following steps: Step S1: Source model pre-training, wherein the source model includes at least a spatial feature extractor, a temporal feature extractor, and a classifier; the temporal feature extractor extracts multiple local temporal features by modeling the relationships between different numbers of temporally sampled frame segments in the video, and further aggregates them into overall temporal features; In this embodiment, preferably, the source model pre-training includes: training the source model using labeled source domain videos, and using labeled smooth cross-entropy loss as the training loss; the temporal feature extractor is a temporal relation network (TRN), and the spatial feature extractor is a 2D residual network (ResNet).

[0026] In this embodiment, a source model is trained using labeled video data from the source domain. The source model includes a spatial feature extractor, a temporal feature extractor, and a classifier. The spatial feature extractor extracts the spatial representation of each frame; the temporal feature extractor models the relationships between sampled segments from different time sequences, generates multiple local temporal features, and averages these features to obtain the overall temporal features; the classifier outputs an action category prediction based on the overall temporal features. Labeled smooth cross-entropy loss is used during source model training to improve the transferability and class distribution separation of the source model.

[0027] Step S2: Passive Adaptation in the Target Domain; When the passive adaptation step in the target domain is executed, the source domain labeled video data is no longer accessed. Only the pre-trained source model and the unlabeled target domain video are loaded, and the classifier parameters of the source model are fixed throughout the adaptation process.

[0028] In this embodiment, preferably, combined with Figure 2 As shown, the target domain passive adaptive method specifically includes the following steps: Step S21: Initialization: Initialize the target domain spatial feature extractor and the target domain temporal feature extractor using the parameters of the source model's spatial feature extractor and temporal feature extractor.

[0029] Step S22: Extract local temporal features: Sample temporal segments of the target video and obtain multiple sets of local temporal features through the target domain temporal feature extractor.

[0030] In this embodiment, the target network is initialized using the spatial and temporal feature extractors from the source model in the target domain. For the input target video, multiple local temporal features are extracted at different time spans, each consisting of several video frames sampled in chronological order. The target network retains the same structure as the source model, while the source classifier remains fixed during the adaptation process.

[0031] Step S23: Construct temporal consistency constraints: Simultaneously apply feature consistency constraints and source prediction consistency constraints to the multiple sets of local temporal features; the feature consistency constraints are used to make the feature representations of different local temporal features tend to be consistent; the source prediction consistency constraints are used to make the predicted distributions of different local temporal features output by a fixed classifier tend to be consistent.

[0032] In this embodiment, different local temporal features extracted from the same target video are normalized, and the cross-correlation matrix between any two sets of local temporal features is calculated. By optimizing the cross-correlation matrix so that the diagonal elements are close to 1 and the off-diagonal elements are close to 0, the different local temporal features are made consistent in representation and redundancy is reduced, thus constructing a feature consistency loss. Each local temporal feature is directly input into a fixed source classifier to obtain the corresponding local source prediction; then the average result of the local source prediction is calculated, and the KL divergence between each local source prediction and the average result is minimized. At the same time, the overall temporal features are input into a fixed source classifier to obtain the overall prediction, and the difference between the overall prediction and the average local prediction is minimized, thus constructing a source prediction consistency loss.

[0033] In this embodiment, preferably, in step S23, the feature consistency constraint is implemented in the following way: after normalizing any two sets of local time features, the cross-correlation matrix is ​​calculated, and the cross-correlation matrix is ​​constrained to approximate the identity matrix to construct the feature consistency loss; when constructing the feature consistency loss, balanced weights are set for the diagonal and off-diagonal terms of the cross-correlation matrix for joint optimization.

[0034] In this embodiment, preferably, in step S23, the source prediction consistency constraint includes a local prediction consistency sub-constraint and an overall prediction consistency sub-constraint; The local prediction consistency sub-constraint is to input each local temporal feature into a fixed classifier to obtain a local prediction distribution, and minimize the KL divergence between each local prediction distribution and the average local prediction distribution. The overall prediction consistency sub-constraint is to aggregate the mean of local time features to obtain ordinary overall time features and input them into a fixed classifier to obtain the overall prediction distribution, minimizing the difference between the overall prediction distribution and the average local prediction distribution. By setting balance coefficients for the local prediction consistency sub-constraint and the global prediction consistency sub-constraint respectively, the source prediction consistency loss is obtained.

[0035] Step S24: Attention-weighted aggregation: Generate corresponding local weights based on the prediction confidence obtained by inputting the local time features of each group into a fixed classifier, and use the local weights to weight and aggregate the local time features of each group to obtain the weighted overall time features.

[0036] In this embodiment, the confidence level of each local temporal feature is calculated based on its prediction entropy on a fixed source classifier, and local weights are generated by adding residual terms. These local weights are then applied to each local temporal feature and its corresponding local prediction to obtain a weighted overall temporal feature and a weighted local prediction. This step ensures that local temporal features that more closely approximate the source domain distribution have a higher weight in the overall representation construction.

[0037] In this embodiment, preferably, in step S24, the prediction confidence is calculated from the information entropy of the prediction distribution corresponding to the local time features, and the local weight calculation formula is: local weight = 1 + prediction confidence.

[0038] Step S25: Self-supervised optimization: Construct a self-supervised loss function based on the weighted overall temporal features, and backpropagate to update the parameters of the target domain spatial feature extractor and the target domain temporal feature extractor to complete passive video domain adaptation.

[0039] After adaptation, the optimized target network is combined with a fixed classifier to output the action recognition results of the target video.

[0040] Preferably, in step S25, the self-supervised loss function includes temporal consistency loss, information maximization loss, and pseudo-label cross-entropy loss. Specifically, in this embodiment, independent weight coefficients are configured for temporal consistency loss, information maximization loss, and pseudo-label cross-entropy loss, and the total loss is obtained by weighted fusion and used for network backpropagation optimization. The temporal consistency loss is obtained by weighted fusion of feature consistency loss and source prediction consistency loss. The information maximization loss is used to improve the determinism of single-sample prediction and the balance of global category distribution. The pseudo-label cross-entropy loss is constructed based on pseudo-labels generated by weighted overall temporal feature clustering.

[0041] In this embodiment, pseudo-labels are preferably generated through two rounds of iterative K-means clustering: first, the cluster centers are initialized based on the globally weighted overall time features to obtain initial pseudo-labels, and then the cluster centers are updated using the initial pseudo-labels, and the final pseudo-labels are obtained by matching the cosine distance.

[0042] In this embodiment, an information maximization loss is applied to the weighted overall temporal features to make single-sample predictions in the target domain more certain, while ensuring a more balanced overall category distribution. Furthermore, clustering is performed on the weighted overall temporal features to obtain pseudo-labels for the target domain, and the target network is trained using a pseudo-label cross-entropy loss under self-supervised conditions. The final total loss is a weighted sum of the temporal consistency loss, the information maximization loss, and the pseudo-label cross-entropy loss.

[0043] The method of this embodiment was validated on three cross-domain action recognition benchmarks: UCF-HMDBfull, Daily-DA, and Sports-DA. Specifically: UCF-HMDBfull: This benchmark includes two cross-domain tasks: U101→H51 (UCF101 as the source domain and HMDB51 as the target domain) and H51→U101 (HMDB51 as the source domain and UCF101 as the target domain).

[0044] Daily-DA: This benchmark includes 10 cross-domain tasks across 4 domains (ARID, HMDB51, Moments-in-Time, Kinetics), covering 8 action categories.

[0045] Sports-DA: This benchmark includes six cross-domain tasks across three large-scale datasets (UCF101, Sports-1M, Kinetics), covering 23 action categories.

[0046] The method of this embodiment (ATCoN) is compared with the following methods: (1) Source-only model (TRN): No adaptation is performed; the source model is used directly for testing on the target domain. (2) Existing SFDA methods: including SHOT, SHOT++, CPGA, BAIT, etc.; (3) Existing VUDA methods include DANN, MK-MMD, TA³N, etc. The latter can access source domain data during the adaptation process.

[0047] The experimental results (accuracy %) on the UCF-HMDBfull benchmark are as follows:

[0048] Experimental results show that, on the UCF-HMDBfull benchmark, the method in this embodiment achieves the best performance in both cross-domain tasks. In terms of average accuracy, ATCoN reaches 82.51%, which is significantly better than all the comparison methods.

[0049] The experimental results (accuracy %) on the Daily-DA benchmark are as follows:

[0050] Experimental results show that the proposed method outperforms the optimal SFDA method on all 10 cross-domain tasks of the Daily-DA benchmark. The advantage of ATCoN is particularly evident on difficult tasks involving ARID, fully validating the robustness of this invention in handling large-domain offsets.

[0051] The experimental results (accuracy %) on the Sports-DA benchmark are as follows:

[0052] Experimental results show that, across all five cross-domain tasks on the Sports-DA benchmark, the method in this embodiment achieves the best performance on four tasks and the second-best performance on one task. In large-scale cross-domain scenarios, ATCoN also demonstrates superior performance, proving the versatility and effectiveness of the method under different data scales. Example 2

[0053] Combination Figure 3 As shown, the passive video domain adaptive action recognition system based on attention temporal consistency network in this embodiment is used to execute the passive video domain adaptive action recognition method based on attention temporal consistency network in Embodiment 1, and specifically includes the following modules: The source model pre-training module is used to train a source model that includes a spatial feature extractor, a temporal feature extractor, and a classifier. The target domain feature extraction module loads the source model weights to initialize the target domain extractor, fixes the classifier parameters, and extracts multiple sets of local temporal features from unlabeled target videos. The adaptive optimization module incorporates a constraint calculation unit, an attention weighting unit, and a self-supervised training unit. The constraint calculation unit calculates the temporal consistency loss corresponding to feature consistency constraints and source prediction consistency constraints. The attention weighting unit generates local weights based on the classification prediction confidence of local temporal features and aggregates them to obtain weighted overall temporal features. The self-supervised training unit constructs a self-supervised loss based on the weighted overall temporal features, updates the target domain space and temporal feature extractor parameters, and outputs an adapted action recognition model.

[0054] In this embodiment, preferably, it also includes an inference output module, which loads the adapted action recognition model, inputs the video to be recognized, and outputs the video action category recognition result.

[0055] The passive video domain adaptive action recognition system based on attention temporal consistency network in this embodiment can be applied to intelligent monitoring human behavior recognition, low-light scene action recognition, and human-computer interaction action classification scenarios.

[0056] Many specific details have been set forth in the foregoing description to provide a thorough understanding of the present invention. However, the above description is merely a preferred embodiment of the present invention, and the present invention can be implemented in many other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed above. Furthermore, any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention, or modify them into equivalent embodiments, using the methods and techniques disclosed above, without departing from the scope of the present invention. Any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A passive video domain adaptive action recognition method based on attention temporal consistency networks, characterized in that: Includes the following steps: Step S1: Source model pre-training, wherein the source model includes at least a spatial feature extractor, a temporal feature extractor, and a classifier; the temporal feature extractor extracts multiple local temporal features by modeling the relationships between different numbers of temporally sampled frame segments in the video, and further aggregates them into overall temporal features; Step S2: Passive Adaptation in the Target Domain; When the passive adaptation step in the target domain is executed, the source domain labeled video data is no longer accessed. Only the pre-trained source model and the unlabeled target domain video are loaded, and the classifier parameters of the source model are fixed throughout the adaptation process.

2. The passive video domain adaptive action recognition method based on attention temporal consistency network according to claim 1, characterized in that: The passive adaptive targeting domain specifically includes the following steps: Step S21: Initialization: Initialize the target domain spatial feature extractor and the target domain temporal feature extractor using the parameters of the source model's spatial feature extractor and temporal feature extractor; Step S22: Extract local temporal features: Sample temporal segments of the target video and obtain multiple sets of local temporal features through the target domain temporal feature extractor; Step S23: Construct temporal consistency constraints: Simultaneously apply feature consistency constraints and source prediction consistency constraints to the multiple sets of local temporal features; the feature consistency constraints are used to make the feature representations of different local temporal features tend to be consistent; the source prediction consistency constraints are used to make the predicted distributions of different local temporal features output by a fixed classifier tend to be consistent. Step S24: Attention weighted aggregation: Generate corresponding local weights based on the prediction confidence obtained by inputting the local time features of each group into a fixed classifier, and use the local weights to weight and aggregate the local time features of each group to obtain the weighted overall time features; Step S25: Self-supervised optimization: Construct a self-supervised loss function based on the weighted overall temporal features, and backpropagate to update the parameters of the target domain spatial feature extractor and the target domain temporal feature extractor to complete passive video domain adaptation; After adaptation, the optimized target network is combined with a fixed classifier to output the action recognition results of the target video.

3. The passive video domain adaptive action recognition method based on attention temporal consistency network according to claim 1, characterized in that: The source model pre-training includes: training the source model using labeled source domain videos, and using labeled smooth cross-entropy loss as the training loss; the temporal feature extractor is a temporal relation network (TRN), and the spatial feature extractor is a 2D residual network (ResNet).

4. The passive video domain adaptive action recognition method based on attention temporal consistency network according to claim 2, characterized in that: In step S23, the feature consistency constraint is implemented as follows: after normalizing any two sets of local time features, the cross-correlation matrix is ​​calculated, and the cross-correlation matrix is ​​constrained to approximate the identity matrix to construct the feature consistency loss; when constructing the feature consistency loss, balanced weights are set for the diagonal and off-diagonal terms of the cross-correlation matrix for joint optimization.

5. The passive video domain adaptive action recognition method based on attention temporal consistency network according to claim 2, characterized in that: In step S23, the source prediction consistency constraint includes a local prediction consistency sub-constraint and an overall prediction consistency sub-constraint. The local prediction consistency sub-constraint is to input each local temporal feature into a fixed classifier to obtain a local prediction distribution, and minimize the KL divergence between each local prediction distribution and the average local prediction distribution. The overall prediction consistency sub-constraint is to aggregate the mean of local time features to obtain ordinary overall time features and input them into a fixed classifier to obtain the overall prediction distribution, minimizing the difference between the overall prediction distribution and the average local prediction distribution. By setting balance coefficients for the local prediction consistency sub-constraint and the global prediction consistency sub-constraint respectively, the source prediction consistency loss is obtained.

6. The passive video domain adaptive action recognition method based on attention temporal consistency network according to claim 2, characterized in that: In step S24, the prediction confidence is calculated from the information entropy of the prediction distribution corresponding to the local time features, and the local weight is calculated using the formula: local weight = 1 + prediction confidence.

7. The passive video domain adaptive action recognition method based on attention temporal consistency network according to claim 2, characterized in that: In step S25, the self-supervised loss function includes temporal consistency loss, information maximization loss, and pseudo-label cross-entropy loss; the temporal consistency loss is obtained by weighted fusion of feature consistency loss and source prediction consistency loss; the information maximization loss is used to improve the certainty of single-sample prediction and the balance of global category distribution; the pseudo-label cross-entropy loss is constructed based on pseudo-labels generated by weighted overall temporal feature clustering.

8. The passive video domain adaptive action recognition method based on attention temporal consistency network according to claim 7, characterized in that: Pseudo-labels are generated through two rounds of iterative K-means clustering: first, the cluster centers are initialized based on the globally weighted overall time features to obtain the initial pseudo-labels; then, the cluster centers are updated using the initial pseudo-labels, and the final pseudo-labels are obtained by matching the cosine distance.

9. A passive video domain adaptive action recognition system based on attention temporal consistency networks, characterized in that: A passive video domain adaptive action recognition method based on an attention temporally consistent network as described in any one of claims 1-8, comprising the following modules: The source model pre-training module is used to train a source model that includes a spatial feature extractor, a temporal feature extractor, and a classifier. The target domain feature extraction module loads the source model weights to initialize the target domain extractor, fixes the classifier parameters, and extracts multiple sets of local temporal features from unlabeled target videos. The adaptive optimization module incorporates a constraint calculation unit, an attention weighting unit, and a self-supervised training unit. The constraint calculation unit calculates the temporal consistency loss corresponding to feature consistency constraints and source prediction consistency constraints. The attention weighting unit generates local weights based on the classification prediction confidence of local temporal features and aggregates them to obtain weighted overall temporal features. The self-supervised training unit constructs a self-supervised loss based on the weighted overall temporal features, updates the target domain space and temporal feature extractor parameters, and outputs an adapted action recognition model.

10. The passive video domain adaptive action recognition system based on attention temporal consistency network according to claim 9, characterized in that: It also includes an inference output module, which loads the adapted action recognition model, inputs the video to be recognized, and outputs the video action category recognition result.

Images