Method for training action recognition model based on unsupervised domain adaptation and application thereof

Abstract

The application provides a training method and application of an action recognition model based on unsupervised domain adaptation, and the training method comprises the following steps: obtaining a first training sample set and a second training sample set; training a to-be-trained action recognition model according to the first training sample set to obtain a preliminarily trained action recognition model; inputting the second training sample set into the preliminarily trained action recognition model and a pre-trained CLIP model respectively to obtain first pseudo labels and second pseudo labels of each second training sample; screening each second training sample according to the matching condition of the first pseudo label and the second pseudo label of the second training sample; and jointly training the preliminarily trained action recognition model to obtain a trained action recognition model. Through the training method and application of the action recognition model based on unsupervised domain adaptation, the generalization ability of the model can be improved, and the damage to semantic information caused by additional processing can be avoided.

Description

Technical Field

[0001] This invention relates to the field of domain-adaptive action recognition, and more particularly to a training method and application of an unsupervised domain-adaptive action recognition model. Background Technology

[0002] Video action recognition is widely used in fields such as industrial safety, intelligent transportation, and public security. For example, action recognition models can be used to analyze driver behavior to prevent safety accidents. In actual deployment, due to differences in lighting, viewpoint, and background between the training data (source domain) and the application scenario (target domain), the model accuracy will decrease significantly, which is the domain offset problem.

[0003] To mitigate domain shift, unsupervised domain adaptation has become the mainstream approach. Existing techniques mainly fall into two categories: one is feature distribution alignment, but this suffers from training instability; the other is pseudo-label self-training, where the model generates pseudo-labels for the target domain data and then iteratively fine-tunes them. This approach is more common in practical applications due to its simplicity. However, these approaches share a problem: some methods introduce additional processing during self-training to improve the model's generalization ability. But if the supervision signal is not adjusted accordingly, and the processed data is still required to maintain consistency with the original data output, the model may learn features that deviate from the true semantics, affecting cross-domain recognition accuracy and training stability.

[0004] Therefore, how to improve the generalization ability of a model while avoiding the destruction of semantic information by additional processing is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This invention provides a training method and application for an action recognition model based on unsupervised domain adaptive methods, which can improve the generalization ability of the model while avoiding the destruction of semantic information by additional processing.

[0006] This invention provides a training method for an action recognition model based on unsupervised domain adaptation, comprising: Obtain a first training sample set and a second training sample set; wherein, the first training sample set includes multiple first training samples and their action categories, and the second training sample set includes multiple second training samples; The action recognition model to be trained is trained based on the first training sample set to obtain a preliminary trained action recognition model. The second training sample set is input into the pre-trained action recognition model and the pre-trained CLIP model respectively, and the first pseudo-label and the second pseudo-label of each second training sample are obtained accordingly. For each second training sample, it is filtered based on the matching of the first pseudo-label and the second pseudo-label, and the first pseudo-label or the second pseudo-label is taken as the action category of the second training sample; wherein, the second training samples and their action categories retained after filtering are taken as the third training sample set; The first training sample set and the third training sample set are respectively input into the action recognition model that has been initially trained to obtain the corresponding first training loss and second training loss. Randomly mask each of the second training samples in the third training sample set to obtain the corresponding masked training samples; Each of the mask training samples and the corresponding second training samples are input into the pre-trained action recognition model to obtain the corresponding third training loss; The parameters of the initially trained action recognition model are updated based on the first training loss, the second training loss, and the third training loss to obtain the trained action recognition model.

[0007] In one embodiment of the present invention, training the action recognition model to be trained based on the first training sample set to obtain a pre-trained action recognition model includes: Each first training sample in the first training sample set is input into the action recognition model to be trained to obtain the predicted category of each first training sample. For each of the first training samples, calculate the cross-entropy loss based on the action category and the predicted category of the first training sample; Calculate the average of the cross-entropy loss of all the first training samples, and use it as the first training loss; The parameters of the action recognition model to be trained are updated based on the first training loss to obtain the initially trained action recognition model.

[0008] In one embodiment of the present invention, the step of inputting the second training sample set into the pre-trained action recognition model and the pre-trained CLIP model respectively to obtain the first pseudo-label and the second pseudo-label of each second training sample includes: For each second training sample in the second training sample set: The second training sample is input into the pre-trained action recognition model to obtain the first confidence level of the second training sample belonging to each action category, and the action category with the highest first confidence level is used as the first pseudo label of the second training sample. The second training sample is input into the pre-trained CLIP model to obtain the second confidence level of the second training sample belonging to each action category, and the action category with the highest second confidence level is used as the second pseudo-label of the second training sample.

[0009] In one embodiment of the present invention, the step of inputting the second training sample into a pre-trained CLIP model to obtain the second confidence level of the second training sample belonging to each action category, and using the action category with the highest second confidence level as the second pseudo-label of the second training sample, includes: The second training sample is input into the pre-trained CLIP model to obtain multiple frame-level visual features; For each frame-level visual feature, the similarity between the frame-level visual feature and each action category is calculated to obtain the predicted probability that the frame-level visual feature belongs to each action category. For each action category, the predicted probabilities of all frame-level visual features are averaged to obtain the second confidence score of the second training sample belonging to each action category. The action category with the highest confidence level is used as the second pseudo-label for the second training sample.

[0010] In one embodiment of the present invention, the step of filtering each second training sample based on the matching of the first pseudo-label and the second pseudo-label, and using the first pseudo-label or the second pseudo-label as the action category of the second training sample, includes: For each of the second training samples: If the first pseudo-label of the second training sample is consistent with the second pseudo-label, then the first pseudo-label or the second pseudo-label is taken as the action category of the second training sample; If the first pseudo-label and the second pseudo-label of the second training sample are inconsistent, then compare the first confidence level of the first pseudo-label and the second confidence level of the second pseudo-label: If at least one of the first confidence level and the second confidence level is greater than or equal to a preset threshold, then the one with the higher confidence level between the first pseudo-label and the second pseudo-label is taken as the action category of the second training sample. Otherwise, discard the second training sample.

[0011] In one embodiment of the present invention, the step of inputting each of the mask training samples and the corresponding second training samples into the pre-trained action recognition model to obtain the corresponding third training loss includes: Each of the masked training samples and the corresponding second training samples are input into the action recognition model that is initially trained, so as to obtain the masked view features of each of the masked training samples and the complete view features of the corresponding second training samples. The third training loss is calculated based on each of the masked view features and the corresponding complete view features.

[0012] In one embodiment of the present invention, the action recognition model is trained and updated based on a loss function; the calculation formula of the loss function is as follows:

[0013]

[0014]

[0015]

[0016] in, For loss function, , , These are the first training loss, the second training loss, and the third training loss, respectively. The total number of the first training samples. Let cross-entropy be the loss function. For the first The predicted category of the first training sample. For the first The action category of the first training sample The total number of the second training samples in the third training sample set. For the first The confidence level of the action category of the second training sample. For the first The predicted category of the second training sample. For the first The action categories of the second training sample For the first Mask view features of a masked training sample To stop the gradient operation, For the first The complete view features of the second training sample.

[0017] The present invention also provides a training device for an action recognition model based on unsupervised domain adaptation, comprising: The data acquisition module is used to acquire a first training sample set and a second training sample set; wherein, the first training sample set includes multiple first training samples and their action categories, and the second training sample set includes multiple second training samples. The first training module is used to train the action recognition model to be trained based on the first training sample set to obtain a preliminary trained action recognition model. The category prediction module is used to input the second training sample set into the pre-trained action recognition model and the pre-trained CLIP model respectively, and obtain the first pseudo-label and the second pseudo-label of each second training sample. The matching and filtering module is used to filter each second training sample based on the matching of the first pseudo-label and the second pseudo-label of the second training sample, and to take the first pseudo-label or the second pseudo-label as the action category of the second training sample; wherein, the second training samples and their action categories retained after filtering are used as the third training sample set; The first input module is used to input the first training sample set and the third training sample set into the action recognition model under preliminary training, respectively, to obtain the corresponding first training loss and second training loss. The random masking module is used to randomly mask each of the second training samples in the third training sample set to obtain the corresponding masked training samples. The second input module is used to input each of the mask training samples and the corresponding second training samples into the pre-trained action recognition model to obtain the corresponding third training loss. The second training module is used to update the parameters of the initially trained action recognition model based on the first training loss, the second training loss, and the third training loss, so as to obtain the trained action recognition model.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method for the action recognition model based on unsupervised domain adaptation.

[0019] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the training method for the action recognition model based on unsupervised domain adaptation.

[0020] The beneficial effects of this invention are as follows: By fusing the initial training model and the CLIP model to generate dual-path pseudo-labels and filtering them according to consistency and high confidence rules, the reliability of pseudo-labels is improved and the interference of noisy labels on training is reduced; by applying differentiated supervision constraints to the complete video and the masked video during the joint training phase, the generalization ability of the model is improved by utilizing mask enhancement while avoiding the destruction of the semantic structure of the video by the mask, thereby improving the model's adaptability to domain shifts while maintaining basic recognition accuracy. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] In the attached diagram: Figure 1 A flowchart illustrating a training method for an unsupervised domain adaptive action recognition model according to an embodiment of the present invention; Figure 2 A schematic diagram of a training device for an unsupervised domain adaptive action recognition model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0023] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0025] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0026] With the development of deep learning technology, action recognition models have demonstrated their application potential in numerous practical scenarios. In the field of industrial safety production, analyzing the action sequences of operators in surveillance videos can identify whether there are any violations or dangerous behaviors. In intelligent transportation scenarios, continuous monitoring of drivers' facial expressions and body movements helps determine whether they are driving while fatigued or distracted. In the field of public security, real-time analysis of abnormal body movements in crowds can be used to assist in the early warning of potential safety incidents. The common requirement of these scenarios is that they all rely on action recognition models to achieve accurate and timely semantic understanding of human actions in videos.

[0027] Existing action recognition models are typically trained on specific labeled datasets, which often have relatively fixed distributions in terms of lighting conditions, shooting angle, background environment, and clothing. When these models are deployed directly to real-world application environments, there is usually a significant difference between the data distribution in the actual scene and the distribution of the training data; this difference is called domain shift. Domain shift can lead to varying degrees of decrease in model recognition accuracy, thus hindering the large-scale deployment of action recognition technology in open environments. Therefore, how to effectively adapt models to the data distribution in real-world scenes has become a research topic of practical significance in the field of video action recognition.

[0028] To address the aforementioned domain offset problem, unsupervised domain adaptation methods have attracted attention due to their independence from labeled data. Existing techniques employ two approaches: one is to forcibly narrow the feature distributions of the source domain (labeled dataset) and the target domain (unlabeled dataset) through adversarial training; however, these methods are often unstable during training and highly sensitive to hyperparameters. Another approach uses a pseudo-label self-training strategy, where the action recognition model generates pseudo-labels from the training samples in the target domain and iteratively fine-tunes them. This approach is relatively simple to implement and is more widely used in practice. However, some improved schemes, in order to enhance the model's generalization ability, introduce additional processing techniques such as masking and augmentation, yet still require the processed sample output to remain consistent with the supervision signal of the original sample. This constraint may cause the model to learn feature representations that deviate from the true action semantics, thus affecting cross-domain recognition accuracy and training stability.

[0029] To address this, the present invention provides a training method and application for an unsupervised adaptive action recognition model. In the pseudo-label generation stage, this training method integrates the complementary prediction capabilities of the pre-trained action recognition model and the CLIP model, and designs selection rules based on consistency and high confidence to reduce the interference of noisy labels on training. In the self-training stage, by constructing a full-view semantic baseline branch and a masked view semantic transfer branch, differentiated supervisory constraints are applied to the complete video and the masked video respectively. This enhances the model's generalization ability using masking enhancement while minimizing damage to the video's semantic structure.

[0030] Please see Figure 1 The training method for an action recognition model based on unsupervised domain adaptation disclosed in this embodiment includes the following steps: S10, obtaining a first training sample set and a second training sample set; wherein, the first training sample set includes multiple first training samples and their action categories, and the second training sample set includes multiple second training samples.

[0031] Specifically, the first step is to obtain a first training sample set. This first training sample set consists of multiple first training samples, each labeled with a corresponding action category. The first training samples are derived from video data of complete, labeled ordinary vehicle driving behavior. This type of video data is typically collected from publicly available driving behavior databases or recorded in real-world road environments using in-vehicle cameras and then manually labeled. Each first training sample is a short video clip recording a segment of driver actions. The action categories are predefined driving behavior categories, such as normal driving, fatigued driving, smoking, making a phone call, drinking water, and looking around. Each first training sample is labeled with one of these action categories. First training sample set , can be represented as ,in, For the first The first training sample, For the first The action category of the first training sample The total number of video frames in the first training sample. , For example, the resolution of the video frame. , , The total number of the first training samples. This represents the total number of action categories.

[0032] Simultaneously, a second training sample set is acquired. This second training sample set consists of multiple training samples derived from corresponding application scenarios, such as onboard or surveillance videos of forklift driving. As industrial handling vehicles, forklifts differ from ordinary vehicles in their driving environment and operating habits. For example, the cab layout is different, the operating scenarios are mostly warehouses or factories, and the driving speed is slower but the operational movements are more significant. The second training samples are also short video clips, recording the continuous actions of the forklift driver during the execution of tasks. However, unlike other vehicles, the second training samples do not contain any action category annotations. This is because in actual industrial scenarios, annotating a large number of forklift driving videos frame by frame requires significant manpower and time, and different annotators may have different judgment standards for certain driving behaviors. Second Training Sample Set , can be represented as ,in, For the first The second training sample, The total number of video frames in the second training sample. , For example, the resolution of the video frame. , , This represents the total number of the second training samples.

[0033] The first and second training sample sets share the same action categories. For example, fatigued driving in ordinary vehicle driving behavior belongs to the same action category as fatigued driving in forklift scenarios. This setting ensures semantic consistency between the source and target domains, allowing the classification knowledge learned by the model on ordinary vehicle driving data to be transferred to forklift driving scenarios.

[0034] Both the first and second training samples are stored as video files or video frame sequences on local storage devices or remote databases. Acquisition can be achieved by reading data files from a specified path. The first training sample set is used in the subsequent initial training phase to enable the action recognition model to learn basic semantic features of driving behavior; the second training sample set is used for pseudo-label generation and filtering, as well as in the subsequent joint training phase, to gradually adapt the action recognition model to the data distribution characteristics of forklift driving scenarios.

[0035] Please see Figure 1 The training method disclosed in this embodiment also includes the following steps: S20, training the action recognition model to be trained based on the first training sample set to obtain a preliminary trained action recognition model.

[0036] In a preferred embodiment, S20 includes the following steps: S21, inputting each first training sample in the first training sample set into the action recognition model to be trained to obtain the predicted category of each first training sample.

[0037] In the initial training phase, each of the first training samples in the first training sample set is input into the action recognition model to be trained. The action recognition model consists of two parts: a feature extractor and a classifier. The feature extractor can adopt the ViT-B / 16 visual Transformer architecture.

[0038] For a given first training sample, the feature extractor first performs spatiotemporal feature encoding on each video frame of the first training sample, extracts a discriminative visual representation, and transforms it into a fixed-dimensional visual feature. Among them, the first Visual features of the first training sample , represented as: , ,in, These are the internal parameters of the feature extractor. For feature dimension, For feature extraction operations, For the first The first training sample.

[0039] The classifier then receives the visual feature, maps it to an action category space, and outputs a probability distribution vector. Each element in the probability distribution vector corresponds to the predicted probability of an action category. The predicted probability is expressed as: ,in, For mapping operations, These are the internal parameters of the classifier. .

[0040] Based on this, the action category with the highest probability value is selected as the predicted category for the first training sample. Through the above forward propagation process, each first training sample obtains a corresponding predicted category. Wherein, the... The predicted category of the first training sample is represented as follows: .

[0041] In a preferred embodiment, S20 further includes the following step: S22, for each first training sample, calculate the cross-entropy loss based on the action category and the predicted category of the first training sample.

[0042] After obtaining the predicted class for each first training sample, the predicted class for each sample is compared with the ground truth action class. Cross-entropy loss is a common metric for measuring the difference between two probability distributions. The ground truth action class is represented as a one-hot encoded probability distribution, while the predicted class is calculated as the probability distribution output by the model. Cross-entropy loss quantifies the degree of deviation between the predicted class and the action class: the closer the predicted class is to the action class, the smaller the loss value; conversely, the farther the predicted class deviates from the action class, the larger the loss value. By calculating the cross-entropy loss for each first training sample, the model's classification performance for each source domain sample under the current parameters can be quantitatively evaluated.

[0043] In a preferred embodiment, S20 further includes the following step: S23, calculating the average value of the cross-entropy loss of all first training samples as the first training loss.

[0044] After calculating the loss for all first training samples, the arithmetic mean of all cross-entropy loss values ​​is taken, and this result is used as the first training loss. The averaging method, rather than summation, is used to eliminate the influence of the number of first training samples on the magnitude of the loss, ensuring that the loss value does not change with the total number of samples, thus maintaining comparability on training sets of different sizes. The first training loss reflects the overall classification accuracy of the action recognition model on the first training sample set under the current parameter state and is a comprehensive indicator of the model's fit to the source domain. The formula for calculating the first training loss is as follows:

[0045] in, This is the first training loss. The total number of the first training samples. Let cross-entropy be the loss function. For the first The predicted category of the first training sample. For the first The action category of the first training sample.

[0046] In a preferred embodiment, S20 further includes the following step: S24, updating the parameters of the action recognition model to be trained according to the first training loss to obtain the initially trained action recognition model.

[0047] After obtaining the first training loss, all learnable parameters of the action recognition model to be trained are updated with the goal of minimizing this first training loss. Specifically, the gradient of the first training loss with respect to each parameter in the feature extractor and classifier is calculated using the backpropagation algorithm. These gradients are then propagated backward layer by layer along the network structure to determine the update direction and magnitude of each parameter.

[0048] Based on this, a preset optimizer, such as an adaptive moment estimation optimizer, is used in conjunction with a set learning rate to iteratively update the parameters of the feature extractor and classifier one or more times. In each iteration, the parameters are adjusted according to the first training loss under the current parameters, so that the first training loss gradually decreases.

[0049] When the first training loss decreases to the preset convergence condition, or the number of iterations reaches the preset upper limit, the initial training stops, and the parameters at the current moment are used as the parameters of the initially trained action recognition model. The initially trained action recognition model has learned the mapping relationship between the visual features and categories of driving behavior in the first training sample set, providing a basic classification ability for the subsequent generation of pseudo-labels for the second training sample.

[0050] Please see Figure 1 The training method disclosed in this embodiment further includes the following steps: S30, inputting the second training sample set into the pre-trained action recognition model and the pre-trained CLIP model respectively, and obtaining the first pseudo-label and the second pseudo-label of each second training sample.

[0051] In a preferred embodiment, S30 includes the following steps: S31, for each second training sample in the second training sample set: input the second training sample into the pre-trained action recognition model to obtain the first confidence level of the second training sample belonging to each action category, and take the action category with the highest first confidence level as the first pseudo-label of the second training sample.

[0052] For each second training sample in the second training sample set, it is input into the pre-trained action recognition model. After being trained on the first training sample set, this action recognition model has preliminarily grasped the mapping relationship between the visual features of various driving behaviors and action categories.

[0053] When the second training sample is passed through the forward propagation of the action recognition model, the classifier in the action recognition model outputs a probability distribution. The dimension of the probability distribution is equal to the total number of action categories. The value in each dimension represents the confidence level of the action recognition model in considering the second training sample to belong to the corresponding action category, and the sum of these confidence levels is 1. The formula for calculating the probability distribution is as follows:

[0054] in, For mapping operations, For feature extraction operations, These are the internal parameters of the feature extractor. These are the internal parameters of the classifier. , For the first The second training sample.

[0055] The confidence scores corresponding to all action categories are used as the confidence score distribution for the second training sample. The highest confidence score is then selected, and its corresponding action category is taken as the first pseudo-label for this second training sample. The first pseudo-label is essentially a category inference given by the initially trained action recognition model for unlabeled target domain samples, and its reliability is reflected by the corresponding first confidence score. The formulas for calculating the first pseudo-label and the first confidence score are as follows:

[0056]

[0057] in, For the first The first pseudo-label of the second training sample For the first The first confidence level corresponding to the first pseudo-label of the second training sample. For the first The second training sample of the first The first confidence level for each action category.

[0058] In a preferred embodiment, S30 further includes the following steps: S32, inputting the second training sample into the pre-trained CLIP model to obtain the second confidence level of the second training sample belonging to each action category, and taking the action category with the highest second confidence level as the second pseudo-label of the second training sample.

[0059] In a preferred embodiment, S32 includes the following steps: S321, inputting the second training sample into the pre-trained CLIP model to obtain multiple frame-level visual features.

[0060] For the same second training sample, it is first input into the pre-trained CLIP model. The CLIP model includes an image encoder and a text encoder. The image encoder is used to extract the visual features of the input image, and the text encoder is used to extract the semantic features of the input text. The image encoder and text encoder map inputs from different modalities to the same feature space, allowing visual features and text features to be directly compared for similarity in this space.

[0061] For the second training sample, since it is essentially a short video clip containing multiple video frames, each video frame needs to be processed separately. Each video frame in the second training sample is fed into the image encoder in the CLIP model. After encoding, each video frame generates a corresponding frame-level visual feature. This frame-level visual feature is a fixed-dimensional vector used to represent the visual semantic information contained in the video frame. For a second training sample with T frames, T frame-level visual features can be obtained.

[0062] Simultaneously, the text encoder in the CLIP model needs to be used to obtain the text features corresponding to each action category. For each driving behavior category, a unified text description template is constructed. For example, considering that the second training sample comes from a forklift driving scenario, the text template format can be designed as "a video of a forklift driver {class}", where {class} is the name of the specific action category. For example, for the fatigue driving category, the corresponding text is "a video of a forklift driver being fatigued"; for the smoking category, the corresponding text is "a video of a forklift driver smoking"; for the phone call category, the corresponding text is "a video of a forklift driver talking on the phone". Following the above method, corresponding text is constructed for each action category, resulting in a total of C texts. These C texts are input into the text encoder in the CLIP model. The text encoder performs semantic encoding on each text and outputs the corresponding text features. Each text feature is a vector of fixed dimensions, and this dimension is consistent with the feature dimension of the frame-level visual features. The frame-level visual features and text features are in the same feature space, providing a prerequisite for subsequent calculation of the similarity between the two.

[0063] In a preferred embodiment, S32 further includes the following steps: S322, for each frame-level visual feature, calculate the similarity between the frame-level visual feature and each action category to obtain the predicted probability that the frame-level visual feature belongs to each action category.

[0064] For each frame-level visual feature, the cosine similarity between that frame-level visual feature and the text features of each action category is calculated. Cosine similarity reflects the semantic closeness between the visual content of the video frame and the corresponding action category. By scaling and normalizing all cosine similarities using a temperature coefficient, the similarity of the frame-level visual feature to each action category is converted into a predicted probability, with a value ranging from 0 to 1, and the sum of the predicted probabilities for all action categories is 1. The formula for calculating the predicted probability is as follows:

[0065] in, For the first The frame-level visual feature belongs to the first The predicted probability of each action category. It is a natural exponential function. For cosine similarity, As frame-level visual features, For the first Text features of each action category The total number of action categories. For the first Text features of each action category This is the temperature coefficient.

[0066] In a preferred embodiment, S32 further includes the following step: S323, for each action category, the predicted probabilities of all frame-level visual features are averaged to obtain the second confidence level of the second training sample belonging to each action category.

[0067] After calculating and normalizing the similarity between all frame-level visual features and each action category, for each action category, the predicted probabilities of all frame-level visual features in the second training sample for that category are arithmetically averaged. This approach aims to fuse information from all video frames in the second training sample. Because a single video frame may contain blur, occlusion, or incomplete actions, it's difficult to accurately determine a complete driving behavior based on just one frame. Averaging the prediction results from multiple video frames suppresses the influence of noise from individual frames, resulting in more robust predictions. The probability distribution vector obtained after averaging serves as the second confidence level for the second training sample belonging to each action category. The formula for calculating the second confidence level is as follows:

[0068] in, For the first The second training sample of the first The second confidence level for each action category For the first The second training sample, This represents the total number of video frames in the second training sample.

[0069] In a preferred embodiment, S32 further includes the following step: S324, using the action category with the highest second confidence as the second pseudo-label of the second training sample.

[0070] After obtaining the second confidence distribution of the second training sample, the second confidence value with the largest value is selected, and its corresponding action category is used as the second pseudo-label for the second training sample. Similar to the first pseudo-label, the second pseudo-label is the category inference given by the pre-trained CLIP model based on its visual-semantic alignment ability learned on large-scale image and text data. The CLIP model has zero-shot recognition capability, generating prediction results without training on the second training sample. Its second pseudo-label provides reference information from a different source than the first pseudo-label for subsequent filtering steps. The mutual verification between the two helps identify and eliminate unreliable noise labels. The formulas for calculating the second pseudo-label and the second confidence value are as follows:

[0071]

[0072] in, For the first The second pseudo-label of the second training sample. For the first The second confidence level corresponding to the second pseudo-label of the second training sample. For the first The second training sample of the first The second confidence level for each action category.

[0073] Please see Figure 1 The training method disclosed in this embodiment further includes the following steps: S40, for each second training sample, filtering is performed based on the matching of the first pseudo-label and the second pseudo-label of the second training sample, and the first pseudo-label or the second pseudo-label is taken as the action category of the second training sample; wherein, the second training samples and their action categories retained after filtering are taken as the third training sample set. , represented as ,in, For the reserved first The second training sample, For the reserved first The action categories of the second training sample This represents the total number of the second training samples in the third training sample set.

[0074] In a preferred embodiment, S40 includes the following steps: For each second training sample: If the first pseudo-label of the second training sample is consistent with the second pseudo-label, then the first pseudo-label or the second pseudo-label is taken as the action category of the second training sample; If the first pseudo-label and the second pseudo-label of the second training sample are inconsistent, then compare the first confidence level of the first pseudo-label and the second confidence level of the second pseudo-label: If at least one of the first confidence level and the second confidence level is greater than or equal to a preset threshold, then the one with the higher confidence level between the first pseudo-label and the second pseudo-label is taken as the action category of the second training sample. Otherwise, discard the second training sample.

[0075] Since each second training sample receives two pseudo-labels from different sources—a first pseudo-label and its corresponding first confidence score from the initially trained action recognition model, and a second pseudo-label and its corresponding second confidence score from the pre-trained CLIP model—their predictions for the same second training sample may or may not be consistent due to differences in their training objectives and knowledge sources. Therefore, it is necessary to filter the pseudo-labels of each second training sample, determining whether to retain the sample and what action category to label it based on its matching performance. This process constructs a high-quality third training sample set for subsequent joint training phases.

[0076] For each second training sample in the second training sample set, the first step is to determine whether the first pseudo-label and the second pseudo-label of the second training sample are consistent. If their action categories are the same, it indicates that the initially trained action recognition model and the pre-trained CLIP model have reached a consensus on the action category of the second training sample. This mutual validation from two different models demonstrates the high reliability of the prediction to some extent. Therefore, either the first or second pseudo-label is directly used as the action category of the second training sample, and the sample is retained for subsequent training. Since the two pseudo-labels are the same at this point, the result is consistent regardless of which one is selected as the action category.

[0077] If the first pseudo-label and the second pseudo-label of the second training sample are inconsistent, it indicates that the two models disagree on the action category of the second training sample. In this case, neither prediction result can be simply accepted; further judgment using confidence levels is required. Specifically, the first confidence level corresponding to the first pseudo-label is compared with the second confidence level corresponding to the second pseudo-label, and a preset threshold is used for filtering. This preset threshold is a pre-set hyperparameter used to measure the acceptable confidence level of the prediction result. For example, the preset threshold can be set to 0.9.

[0078] If the first and second pseudo-labels are inconsistent, it is determined whether at least one of the first and second confidence scores is greater than or equal to a preset threshold. If this condition is met, the pseudo-label with the higher confidence score is taken as the action category of the second training sample. This approach is based on the fact that when two models make inconsistent predictions for the same sample, the prediction with higher confidence is often closer to the true category. Especially when the confidence score has reached or exceeded the preset threshold, it indicates that at least one model has a high degree of certainty in its prediction, and therefore there is reason to accept the prediction with higher confidence.

[0079] If the first pseudo-label and the second pseudo-label are inconsistent, and both the first and second confidence scores are less than a preset threshold, it indicates that the two models not only differ in their predictions of the second training sample, but also that their respective confidence scores are insufficient. In other words, neither prediction is sufficiently reliable as a label. In this case, forcibly labeling the second training sample with an action category and using it for subsequent training may introduce significant label noise, thus interfering with the model's optimization direction. Therefore, this second training sample should be discarded and not included in subsequent joint training.

[0080] After the above screening process, all retained second training samples are labeled with a specific action category. These second training samples and their corresponding action categories together constitute the third training sample set. Compared with the original second training sample set, the size of the third training sample set may be smaller, but the pseudo-labels of each second training sample have been verified through consistency checks and high-confidence screening, and have high reliability, providing relatively accurate supervision signals for subsequent joint training.

[0081] Please see Figure 1 The training method disclosed in this embodiment also includes the following steps: S50, inputting the first training sample set and the third training sample set into the action recognition model of preliminary training respectively to obtain the corresponding first training loss and second training loss.

[0082] In a preferred embodiment, S50 includes the following steps: S51, inputting the first training sample set into the pre-trained action recognition model to obtain the predicted category of each first training sample, and calculating the first training loss based on the action category and predicted category of each first training sample. The formula for calculating the first training loss is as follows:

[0083] in, This is the first training loss. The total number of the first training samples. Let cross-entropy be the loss function. For the first The predicted category of the first training sample. For the first The action category of the first training sample.

[0084] In the joint training phase, each first training sample from the first training sample set is first input into the pre-trained action recognition model. After forward propagation, the predicted category of each first training sample is obtained. Then, for each first training sample, its predicted category is compared with the actual labeled action category, and the cross-entropy loss is calculated. After calculating the loss for all first training samples, the arithmetic mean of all cross-entropy loss values ​​is taken to obtain the first training loss. This first training loss is calculated in the same way as the first training loss calculated in the preceding process. On the one hand, the first training loss enables the model to continuously retain its classification ability for driving behavior in the source domain while adapting to the data distribution of the target domain, preventing the model from forgetting the semantic information of the source domain it has already mastered while learning new knowledge. On the other hand, under the pseudo-label self-training framework, the pseudo-labels generated by the model for the target domain samples inevitably contain a small number of errors. If only pseudo-labels are relied upon for iterative optimization, these errors may gradually accumulate during training, leading to a decline in model performance or even deviation from the correct optimization direction. The first training loss introduces the true source domain labels as constraints, which is equivalent to providing a stable reference benchmark for the model. When the supervision signal provided by the pseudo-labels is biased, this loss can pull the model back to the correct optimization trajectory, avoid the error from being amplified round by round, and thus ensure the stability and convergence direction of joint training.

[0085] In a preferred embodiment, S50 further includes the following step: S52, inputting the third training sample set into the pre-trained action recognition model to obtain the predicted category of each second training sample, and calculating the second training loss based on the action category and predicted category of each second training sample. The formula for calculating the second training loss is as follows:

[0086] in, For the second training loss, The total number of the second training samples in the third training sample set. For the first The confidence level of the action category of the second training sample. For the first The predicted category of the second training sample. For the first The action categories of the second training sample.

[0087] Each second training sample from the third training sample set is input into the initially trained action recognition model. Forward propagation yields the predicted category for each second training sample. Each second training sample in the third training sample set has already been filtered and labeled with an action category, which is used as the true label in subsequent training. For each second training sample in the third training sample set, its predicted category is compared with the labeled action category, and the cross-entropy loss is calculated. During the calculation, the confidence level of the pseudo-label corresponding to the second training sample is used as a weight and multiplied by the cross-entropy loss of that second training sample. This ensures that samples with higher confidence levels contribute a larger proportion to the total loss, while samples with lower confidence levels contribute less, thus suppressing the interference of noise that may remain in the pseudo-labels on model optimization. After calculating the weighted loss for all retained samples, the arithmetic mean of all results is taken to obtain the second training loss. This loss fits the pseudo-labels with complete video samples, establishing a stable semantic baseline for the model and ensuring that the model has basic classification accuracy in the target domain.

[0088] Please see Figure 1 The training method disclosed in this embodiment further includes the following steps: S60, randomly masking each of the second training samples in the third training sample set to obtain the corresponding masked training samples.

[0089] Specifically, the second training sample is divided into several image patches in the spatiotemporal dimension. A random masking operation randomly selects a portion of these image patches at a certain masking rate. These selected image patches are directly discarded from the input sequence, rather than retaining their pixel values ​​and setting them to zero. The unselected image patches are retained and participate in subsequent feature encoding. This discarding masking method references the masking strategy in masked autoencoders. The difference between this and the pixel-zeroing method is that the discarding operation makes it impossible for the feature extractor to obtain information from the masked region. It must rely on the spatiotemporal context of the remaining visible image patches to infer the complete action semantics, thus more effectively prompting the model to learn to reconstruct the complete action representation from local and incomplete observation information. After the masking operation, the original second training sample becomes the corresponding masked training sample. The masked training sample is completely identical to the original second training sample in terms of video size and format, with only missing information in some image patches. The purpose of the random masking operation is to simulate the situation where some spatiotemporal information is missing in the video, prompting the model to learn to infer the complete action semantics from local and incomplete observation information. For example, the masking rate can be set to 0.5, which randomly masks half of the image patch. This ratio strikes a balance between compromising information integrity and preserving identifiable features. The formula for calculating the random mask is as follows:

[0090] in, As training samples for the mask, As the second training sample, For Bernoulli random vectors, Each element has a probability of 0.5 of taking the value 1.

[0091] Please see Figure 1 The training method disclosed in this embodiment also includes the following steps: S70, inputting each mask training sample and the corresponding second training sample into the pre-trained action recognition model to obtain the corresponding third training loss.

[0092] In a preferred embodiment, S70 includes the following steps: inputting each masked training sample and the corresponding second training sample into the pre-trained action recognition model to obtain the masked view features of each masked training sample and the complete view features of the corresponding second training sample; and calculating a third training loss based on each masked view feature and the corresponding complete view feature. The formula for calculating the third training loss is as follows:

[0093] in, For the third training loss, For the first The action categories of the second training sample For the first Mask view features of a masked training sample To stop the gradient operation, For the first The complete view features of the second training sample.

[0094] Each second training sample and its corresponding masked training sample from the third training sample set are input into the feature extractor of the pre-trained action recognition model. For the second training sample, since it has not undergone random masking and contains complete video spatiotemporal information, the feature extractor encodes it and outputs complete view features. For the masked training sample, since some image patches are obscured, the feature extractor can only encode based on the remaining visible information, outputting masked view features. The complete view features and the masked view features reside in the same feature space and have the same feature dimension.

[0095] After obtaining each masked view feature and its corresponding full view feature, the feature alignment loss between them is calculated. For example, the cosine similarity between the masked view feature and the full view feature is calculated. The cosine similarity reflects the proximity of the masked view feature and its corresponding full view feature in the semantic space. The feature alignment loss is calculated by subtracting this cosine similarity from one. When the orientations of the masked view feature and its corresponding full view feature tend to be consistent, the cosine similarity approaches 1, and the alignment loss approaches 0; conversely, when the orientations of the two features differ significantly, the alignment loss increases.

[0096] During the computation, a gradient-stopping operation is performed on the full-view features. This means blocking the backpropagation of gradients from the masked view branch to the full-view branch, ensuring that the parameters of the full-view features do not change as the masked view features are optimized. The purpose of this is to maintain the stability of the full-view features, allowing the masked view features to proactively converge towards the full-view features, rather than the two adapting to each other.

[0097] After calculating all alignment losses, the pseudo-label confidence of each second training sample is multiplied by its alignment loss as a weight. The arithmetic mean of all weighted results is then calculated to obtain the third training loss. This third training loss, through feature-level constraints, transfers the semantic knowledge of the complete view to the masked view, enabling the model to extract information relevant to the complete view even when information is missing. Figure 1 By identifying specific action characteristics, the generalization ability of the model can be improved.

[0098] Please see Figure 1The training method disclosed in this embodiment further includes the following steps: S80, updating the parameters of the initially trained action recognition model according to the first training loss, the second training loss, and the third training loss to obtain the trained action recognition model. The action recognition model is trained and updated based on a loss function; the formula for calculating the loss function is as follows:

[0099] Specifically, the first training loss, the second training loss, and the third training loss are summed to obtain the total loss of the joint training. The total loss incorporates three optimization objectives: first, the training loss maintains the model's ability to recognize driving behavior in the source domain; second, the training loss ensures the model's classification accuracy on complete videos in the target domain; and third, the training loss drives the model to extract and complete videos under masked conditions. Figure 1 Characteristic representation of the same.

[0100] With the goal of minimizing the total loss, the gradient of the total loss with respect to each learnable parameter in the action recognition model is calculated using the backpropagation algorithm, and the parameters are updated using an optimizer. This joint training process requires multiple iterations, in which the three training losses and the total loss are calculated for the current parameters, and then the parameters are updated.

[0101] As iterations proceed, the total loss gradually decreases, and the model's recognition performance in the target domain gradually improves. When the total loss decreases to a preset convergence condition, or the number of iterations reaches a preset upper limit, joint training stops, and the parameters of the action recognition model at the current moment are used as the final parameters to obtain the trained action recognition model. This model retains the classification knowledge of driving behavior in the source domain, adapts to the data distribution characteristics of the target domain, and possesses robust representation capabilities for situations where some information is missing.

[0102] As can be seen, the above scheme reduces the interference of pseudo-label noise on training by fusing the initial training model and the CLIP model to generate dual-path pseudo-labels, and by combining consistency and high-confidence screening rules. During the joint training phase, cross-entropy loss with confidence weights is used to establish a stable semantic baseline on the complete video. Simultaneously, a feature alignment mechanism between the masked view and the complete view is introduced, using cosine similarity constraints to achieve semantic knowledge transfer from the complete view to the masked view. Combined with stopping gradient operations, the stability of the semantic baseline is ensured, enabling the model to extract information similar to the complete view even when faced with missing information. Figure 1 The model captures precise action features. By iteratively optimizing the combined source domain supervised loss and target domain dual-branch self-training loss, a balance is achieved between source domain knowledge preservation and target domain adaptation. The trained model can be directly used for target domain video action recognition.

[0103] Please seeFigure 2 The present invention also provides a training device for an action recognition model based on unsupervised domain adaptive model. The training device can be applied to the training method described above. The training device includes a data acquisition module 100, a first training module 200, a category prediction module 300, a matching and filtering module 400, a first input module 500, a random mask module 600, a second input module 700, and a second training module 800.

[0104] The data acquisition module 100 is used to acquire a first training sample set and a second training sample set; wherein the first training sample set includes multiple first training samples and their action categories, and the second training sample set includes multiple second training samples.

[0105] The first training module 200 is used to train the action recognition model to be trained based on the first training sample set, so as to obtain the preliminary trained action recognition model.

[0106] The category prediction module 300 is used to input the second training sample set into the initially trained action recognition model and the pre-trained CLIP model, respectively, to obtain the first pseudo-label and the second pseudo-label of each second training sample.

[0107] The matching and filtering module 400 is used to filter each second training sample based on the matching of the first pseudo-label and the second pseudo-label, and to take the first pseudo-label or the second pseudo-label as the action category of the second training sample; wherein, the second training samples and their action categories retained after filtering are used as the third training sample set.

[0108] The first input module 500 is used to input the first training sample set and the third training sample set into the pre-trained action recognition model, respectively, to obtain the corresponding first training loss and second training loss.

[0109] The random masking module 600 is used to randomly mask each of the second training samples in the third training sample set to obtain the corresponding masked training samples.

[0110] The second input module 700 is used to input each mask training sample and the corresponding second training sample into the initially trained action recognition model to obtain the corresponding third training loss.

[0111] The second training module 800 is used to jointly train the initially trained action recognition model based on the first training sample set and the third training sample set, so as to obtain the trained action recognition model.

[0112] Specific limitations regarding the training device can be found in the limitations of the training method described above, and will not be repeated here. Each module in the aforementioned training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the memory in the electronic device in hardware form, or stored in the memory of the electronic device in software form, so that the memory can call and execute the operations corresponding to each module.

[0113] Please see Figure 3 The electronic device 900 may include a memory 910, a processor 920, and a bus, and may also include a computer program stored in the memory 910 and executable on the processor 920, such as a program for training an unsupervised domain adaptive action recognition model.

[0114] The memory 910 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 910 can be an internal storage unit of the electronic device 900, such as the portable hard drive of the electronic device 900. In other embodiments, the memory 910 can also be an external storage unit of the electronic device 900, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 900. Furthermore, the memory 910 can include both internal and external storage units of the electronic device 900. The memory 910 can be used not only to store application software and various types of data installed on the electronic device 900, such as code for training methods of unsupervised domain adaptive action recognition models, but also to temporarily store data that has been output or will be output.

[0115] In some embodiments, the processor 920 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 920 is the control unit of the electronic device 900, connecting various components of the electronic device 900 via various interfaces and lines. It executes programs or modules stored in the memory 910 (e.g., programs for training methods based on unsupervised domain adaptive action recognition models) and calls data stored in the memory 910 to perform various functions and process data of the electronic device 900.

[0116] The processor 920 executes the operating system of the electronic device 900 and various installed applications. The processor 920 executes the applications to implement the steps in the training method of the above-described unsupervised domain adaptive action recognition model.

[0117] A computer program can be divided into one or more modules. One or more modules are stored in memory 910 and executed by processor 920 to complete this application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in electronic device 900. For example, the computer program can be divided into a data acquisition module 100, a first training module 200, a category prediction module 300, a matching and filtering module 400, a first input module 500, a random mask module 600, a second input module 700, a second training module 800, etc.

[0118] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A training method for an action recognition model based on unsupervised domain adaptive model, characterized in that, include: Obtain a first training sample set and a second training sample set; wherein, the first training sample set includes multiple first training samples and their action categories, and the second training sample set includes multiple second training samples; The action recognition model to be trained is trained based on the first training sample set to obtain a preliminary trained action recognition model. The second training sample set is input into the pre-trained action recognition model and the pre-trained CLIP model respectively, and the first pseudo-label and the second pseudo-label of each second training sample are obtained accordingly. For each second training sample, it is filtered based on the matching of the first pseudo-label and the second pseudo-label, and the first pseudo-label or the second pseudo-label is taken as the action category of the second training sample; wherein, the second training samples and their action categories retained after filtering are taken as the third training sample set; The first training sample set and the third training sample set are respectively input into the action recognition model that has been initially trained to obtain the corresponding first training loss and second training loss. Randomly mask each of the second training samples in the third training sample set to obtain the corresponding masked training samples; Each of the mask training samples and the corresponding second training samples are input into the pre-trained action recognition model to obtain the corresponding third training loss; The parameters of the initially trained action recognition model are updated based on the first training loss, the second training loss, and the third training loss to obtain the trained action recognition model.

2. The training method for an action recognition model based on unsupervised domain adaptation according to claim 1, characterized in that, The step of training the action recognition model to be trained based on the first training sample set to obtain a preliminary trained action recognition model includes: Each first training sample in the first training sample set is input into the action recognition model to be trained to obtain the predicted category of each first training sample. For each of the first training samples, calculate the cross-entropy loss based on the action category and the predicted category of the first training sample; Calculate the average of the cross-entropy loss of all the first training samples, and use it as the first training loss; The parameters of the action recognition model to be trained are updated based on the first training loss to obtain the initially trained action recognition model.

3. The training method for the action recognition model based on unsupervised domain adaptation according to claim 1, characterized in that, The step of inputting the second training sample set into the pre-trained action recognition model and the pre-trained CLIP model respectively, to obtain the first pseudo-label and the second pseudo-label for each of the second training samples, includes: For each second training sample in the second training sample set: The second training sample is input into the pre-trained action recognition model to obtain the first confidence level of the second training sample belonging to each action category, and the action category with the highest first confidence level is used as the first pseudo label of the second training sample. The second training sample is input into the pre-trained CLIP model to obtain the second confidence level of the second training sample belonging to each action category, and the action category with the highest second confidence level is used as the second pseudo-label of the second training sample.

4. The training method for the action recognition model based on unsupervised domain adaptation according to claim 3, characterized in that, The step of inputting the second training sample into the pre-trained CLIP model to obtain the second confidence score of the second training sample belonging to each action category, and using the action category with the highest second confidence score as the second pseudo-label of the second training sample, includes: The second training sample is input into the pre-trained CLIP model to obtain multiple frame-level visual features; For each frame-level visual feature, the similarity between the frame-level visual feature and each action category is calculated to obtain the predicted probability that the frame-level visual feature belongs to each action category. For each action category, the predicted probabilities of all frame-level visual features are averaged to obtain the second confidence score of the second training sample belonging to each action category. The action category with the highest confidence level is used as the second pseudo-label for the second training sample.

5. The training method for an action recognition model based on unsupervised domain adaptation according to claim 1, characterized in that, For each second training sample, the process of filtering based on the matching between the first pseudo-label and the second pseudo-label, and using either the first pseudo-label or the second pseudo-label as the action category of the second training sample, includes: For each of the second training samples: If the first pseudo-label of the second training sample is consistent with the second pseudo-label, then the first pseudo-label or the second pseudo-label is taken as the action category of the second training sample; If the first pseudo-label and the second pseudo-label of the second training sample are inconsistent, then compare the first confidence level of the first pseudo-label and the second confidence level of the second pseudo-label: If at least one of the first confidence level and the second confidence level is greater than or equal to a preset threshold, then the one with the higher confidence level between the first pseudo-label and the second pseudo-label is taken as the action category of the second training sample. Otherwise, discard the second training sample.

6. The training method for an action recognition model based on unsupervised domain adaptation according to claim 1, characterized in that, The step of inputting each of the masked training samples and the corresponding second training samples into the pre-trained action recognition model to obtain the corresponding third training loss includes: Each of the masked training samples and the corresponding second training samples are input into the action recognition model that is initially trained, so as to obtain the masked view features of each of the masked training samples and the complete view features of the corresponding second training samples. The third training loss is calculated based on each of the masked view features and the corresponding complete view features.

7. The training method for an action recognition model based on unsupervised domain adaptation according to claim 6, characterized in that, The action recognition model is trained and updated based on a loss function; the formula for calculating the loss function is as follows: in, For loss function, , , These are the first training loss, the second training loss, and the third training loss, respectively. The total number of the first training samples. Let cross-entropy be the loss function. For the first The predicted category of the first training sample. For the first The action category of the first training sample The total number of the second training samples in the third training sample set. For the first The confidence level of the action category of the second training sample. For the first The predicted category of the second training sample. For the first The action categories of the second training sample For the first Mask view features of a masked training sample To stop the gradient operation, For the first The complete view features of the second training sample.

8. A training device for an unsupervised domain adaptive action recognition model, characterized in that, include: The data acquisition module is used to acquire a first training sample set and a second training sample set; wherein, the first training sample set includes multiple first training samples and their action categories, and the second training sample set includes multiple second training samples. The first training module is used to train the action recognition model to be trained based on the first training sample set, so as to obtain a preliminary trained action recognition model. The category prediction module is used to input the second training sample set into the pre-trained action recognition model and the pre-trained CLIP model respectively, and obtain the first pseudo-label and the second pseudo-label of each second training sample. The matching and filtering module is used to filter each second training sample based on the matching of the first pseudo-label and the second pseudo-label of the second training sample, and to take the first pseudo-label or the second pseudo-label as the action category of the second training sample; wherein, the second training samples and their action categories retained after filtering are used as the third training sample set; The first input module is used to input the first training sample set and the third training sample set into the action recognition model under preliminary training, respectively, to obtain the corresponding first training loss and second training loss. The random masking module is used to randomly mask each of the second training samples in the third training sample set to obtain the corresponding masked training samples. The second input module is used to input each of the mask training samples and the corresponding second training samples into the pre-trained action recognition model to obtain the corresponding third training loss. The second training module is used to update the parameters of the initially trained action recognition model based on the first training loss, the second training loss, and the third training loss, so as to obtain the trained action recognition model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The steps of the training method for an action recognition model based on unsupervised domain adaptation as described in any one of claims 1-7 are implemented when the processor executes a computer program.

10. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the steps of the training method for an action recognition model based on unsupervised domain adaptation as described in any one of claims 1-7.

Images