Multi-model collaborative learning behavior recognition method applied to cross-domain scene

Abstract

The invention discloses a multi-model collaborative learning behavior recognition method applied to a cross-domain scene. The method comprises the steps of model pre-training, pseudo label set outputting, model updating and model deployment. According to the method disclosed by the invention, the phenomenon of unstable pseudo label prediction in the existing method is relieved through technologies such as fusion of historical prediction information, cross-model integration and the like. Compared with an existing method, the method has the advantages that the target domain pseudo-label quality is greatly improved, and a better cross-domain recognition effect is achieved. Meanwhile, by introducing a double-model structure, the problem of insufficient sample feature characterization capability is relieved, and a better cross-domain behavior recognition effect is achieved.

Description

Technical Field

[0001] The present invention relates to computer vision and video understanding methods, and in particular to a multi-model collaborative learning behavior recognition method applied to cross-domain scenarios. Background Art

[0002] In the past decade, with the rapid growth of video data, related industries such as video understanding have flourished. Among them, video behavior recognition technology takes "people" as the core of video data, and is committed to using algorithms to automatically identify and understand human behavior. It is widely used in smart security, human-computer interaction, and industrial production. So far, most behavior recognition algorithms have achieved excellent recognition results based on deep learning technology, and can basically meet the actual application requirements in specific scenarios.

[0003] However, deep learning-based behavior recognition technology often uses supervised learning based on large amounts of video data to improve accuracy. These algorithms perform well when the training videos and actual application videos are in the same data distribution, but when faced with cross-domain videos with differences in lighting, background, etc., the recognition accuracy drops significantly. In the traditional supervised learning model, engineers need to continuously add training videos with manual annotations to improve cross-domain recognition accuracy. However, considering the time-consuming and labor-intensive video annotation work, behavior recognition methods based on supervised learning are difficult to achieve rapid iteration and update of algorithms.

[0004] Based on this background, unsupervised domain adaptive behavior recognition algorithms have gradually entered the field of vision of researchers. This type of method first trains the behavior recognition model through an existing labeled public video dataset (called the source domain), and then uses the unlabeled cross-domain video data collected later (called the target domain) to assist network training, and finally achieves accurate recognition of the target domain video data. In the entire process, the target domain video data with domain differences are not manually labeled, which greatly improves the efficiency of algorithm development and iteration in practical applications, and is more in line with actual application needs.

[0005] Due to the lack of label information of target domain samples, most existing cross-domain behavior recognition methods use pseudo-label technology to assist in distribution alignment. In this technology, the algorithm continuously uses the prediction results of high-confidence target domain samples as manual annotations of unlabeled data to add to network training, and finally uses the target domain pseudo-label information to achieve the recognition of target domain video behavior. Although this method has improved the cross-domain recognition effect of the model to a certain extent, there are still the following problems:

[0006] (1) Most existing pseudo-label evaluation methods directly select the target domain pseudo-label set based on the confidence information output by the source domain model. However, due to differences in data distribution, the confidence of target domain samples is generally low, and there is also the phenomenon of unstable prediction results. The above situation reduces the quality of target domain pseudo-labels and greatly affects the accuracy of cross-domain behavior recognition algorithms.

[0007] (2) Existing cross-domain behavior recognition methods only use a single structural model to extract features from cross-domain video samples. The cross-domain video feature representation capability is weak, which directly affects the performance of the cross-domain behavior recognition algorithm. Although some methods enhance features by fusing optical flow features, the time-consuming and labor-intensive optical flow extraction operation is not suitable for practical application scenarios.

[0008] Therefore, this application mainly proposes a video behavior recognition method and its deployment solution suitable for cross-domain scenarios based on unsupervised domain adaptation related technologies. Summary of the invention

[0009] Purpose of the invention: The purpose of the present invention is to provide a multi-model collaborative learning behavior recognition method for cross-domain scenarios with better cross-domain behavior recognition effect.

[0010] Technical solution: The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios provided by the present invention includes the following steps:

[0011] S1, model pre-training;

[0012] S2, output pseudo label set;

[0013] S3, model update, if it is finished, go to the next step, if it is not finished, return and loop the above steps;

[0014] S4. Model deployment.

[0015] Furthermore, step S1 includes: all source domain labeled videos are sent to the behavior recognition model for supervised learning, and two behavior recognition models based on 3D CNN and 2D CNN are used for feature extraction. All source domain videos are sampled using the interval sampling method. N video frames are sent to two networks respectively for supervised training using the standard cross entropy loss. At the end, two pre-trained source domain models are obtained.

[0016] Furthermore, step S2 includes: increasing the accuracy of generating target domain pseudo labels, on the one hand, alleviating the instability of target domain sample prediction results by fusing historical prediction information; on the other hand, further improving the quality of pseudo labels by reorganizing pseudo labels across models and making full use of complementary information of different structural models.

[0017] Furthermore, model-level pseudo-label generation: During the model training process in the previous stage, the target domain video will also be synchronously input into the two models for prediction along with the source domain video samples. All historical prediction information of the target domain video will be effectively recorded. When using 2D CNN, assuming that the target domain video V t j The historical forecast of is represented as Among them I j Represents the 2D CNN model for the target domain video V t j The historical prediction results, each element in it represents the predicted category score, n j represents the total number of predictions, so the historical prediction information of all target domain videos can be further expressed as Where N t Select the optimal pseudo-label set for the number of target domain videos.

[0018] Historical predictions The first prediction information Take as an example, first obtain the category information of the prediction in the following way:

[0019]

[0020] In the formula, express The predicted category, δ d (·) represents the value of the d-th dimension element of the corresponding feature vector. Using the same method, we can get the video V t j The set of all historical prediction categories is denoted as Finally, for this video, the weight coefficients of each category can be obtained by the following formula:

[0021]

[0022] in, Representative video V t j The category prediction weight, η(B, b) returns the number of times element b appears in set B, D represents the number of categories, n j is the total number of predictions, D is the number of categories. In this process, in order to facilitate horizontal comparison between samples, the obtained category prediction weights are normalized. Finally, W j The index corresponding to the largest element in is considered to be the target domain video V t j The predicted category, when W j When the maximum element in is not unique, the average prediction score of the sample is calculated by formula 3, and the index corresponding to the maximum value is determined as the prediction category.

[0023]

[0024] Finally, video V t j The pseudo labels and their corresponding prediction weights can be expressed as follows:

[0025]

[0026]

[0027] In the formula, Indicates the final prediction label that integrates historical prediction information; sw j Indicates the predicted category In addition, to deal with the special case where the category weights of different videos are equal, this application also uses the average probability score Calculation model for V t j The average entropy value θ j , average information entropy θ j It reflects the uncertainty of the model about the sample and is also an important reference indicator for pseudo-label screening:

[0028]

[0029] According to the above information, all target domain samples are traversed, and the target domain video set can be re-expressed as where j = 0, 1, ..., N t When generating the pseudo-label set of the current model, K target domain pseudo-labels are generated for each category in units of categories, and the samples with the largest prediction weight are selected to be assigned pseudo-labels. When the prediction weights are the same, samples with smaller average entropy values ​​are selected, and finally a model-level pseudo-label video set m containing K×D target domain videos is generated. 1 , D is the number of categories,

[0030] For the 3D CNN model, the same method is used to derive the model-level pseudo label set m 2 , subsequent m 1 With m 2 Through category-level screening, a pseudo-label set across models will be generated to further improve the quality of pseudo-labels.

[0031] Furthermore, the cross-model pseudo label set: from m 1 With m 2 The target domain categories that the two models are best at are selectively extracted from the two models, and then a cross-model pseudo label set is formed to generate high-quality pseudo labels.

[0032] Assume that m 1 With m2 The videos with pseudo labels τ are denoted as m 1[τ] = {V t 1j},m 2[τ] = {V t 2j}, where j = 1, 2, ..., K, and the average entropy value sets corresponding to the videos in the two sets are expressed as and This application calculates the average value of all elements in the above set by the following formula, denoted as and

[0033]

[0034]

[0035] Finally, the target domain video set with pseudo label τ in the generated cross-model pseudo label set will be sorted from m according to the minimum entropy principle. 1 and m 2 Adaptive extraction in the middle, that is:

[0036]

[0037] Among them, m [τ] Indicates the generated cross-model pseudo-label set, the pseudo-label is the τ video set. By traversing all target domain categories in this way, the complete cross-model pseudo-label set m = {m [0] , m [1] , ..., m [D]}, m will be added to the model update in the subsequent stage. In addition, to guide the subsequent model update, the two models will also determine the target domain category index that the two models are better at by comparing the entropy values ​​in this process, as follows:

[0038]

[0039]

[0040] Finally, step S2 obtains the pseudo label set m and the category index class that the two models are good at 2D and class 3D , because step S2 will be executed multiple times in the overall process, the target domain pseudo-label videos in the pseudo-label set m will continue to accumulate, and the class 2D and class 3D It is constantly being updated.

[0041] Furthermore, the step S3 includes: all source domain videos, target domain videos, and the current stage pseudo-label video set m are simultaneously added to the network for training, including two stages: model self-training and cross-model mutual distillation.

[0042] Furthermore, first, all source domain videos and target domain videos with pseudo labels will be trained using cross entropy loss. Assume that the 2D CNN model and the 3D CNN model are represented as G 2D and G 3D , with model G 2D For example, the specific loss function is expressed as follows:

[0043]

[0044] In the formula represents the source domain label after smoothing. Its calculation process is shown in Formula 14, where a is the smoothing coefficient; v s and v m denote the source domain and the target domain video with pseudo labels respectively; σ(·) is the Softmax activation function; δ d (·) represents the d-dimensional element of the corresponding feature; 1(·) is the indicator function. The final self-training losses of the 2D CNN and 3D CNN models can be expressed as:

[0045]

[0046] In addition, to further enhance the cross-domain recognition capability of the algorithm, the two models with different structures will also perform dynamic model mutual distillation. During the training process, the two models continuously perform model prediction on the target domain training video. Taking the 2D CNN model as an example, the video V can be obtained by the following formula: t j The prediction results and corresponding confidence levels are:

[0047]

[0048] In the formula, and Represent the prediction category and model confidence respectively, Represents 2D CNN on video V t j Based on this method, the predicted categories and corresponding confidences of the two models for all target domain videos can be obtained respectively. Finally, this application will divide all target domain videos into two sets based on the above information, as shown in the following formula:

[0049]

[0050]

[0051] Among them, class 2D With class 3D Calculated from Equation 10 and Equation 11; th is the artificially set confidence threshold. 2D In the video samples in the model mutual distillation process, the domain knowledge will be transferred from the 2D CNN model to the 3D CNN model; for D 3D For the video samples in , the direction of knowledge distillation is opposite. In the specific implementation, the output values ​​of the two models need to be smoothed first:

[0052]

[0053] Where, temp is the temperature coefficient; and Respectively represent the output values ​​of the two models for category d; and is the output value of category d after smoothing of the two models. Based on this, the following KL divergence loss function can be constructed:

[0054]

[0055] Finally, the optimization loss of the two models in the entire S3 can be expressed as Equation 25. Stage S3 will last for P epochs. When each S3 stage ends, the algorithm will independently determine whether the total number of current training rounds has reached the preset total number of rounds. Only when the preset total number of rounds is reached will the algorithm end the training and enter the S4 stage. Otherwise, the algorithm will return to step S2 for loop execution.

[0056]

[0057] Furthermore, the step S4 includes: when the step S3 is finished, two behavior recognition models G are obtained respectively. 2D With G 3D In order to meet the needs of real-time processing in practical applications, this application will make full use of various inference acceleration technologies to reduce the model inference speed. The two trained model files will be quantized using TensorRT for PTQ. In this process, some labeled video data selected from the source domain dataset will be used as a calibration set to ensure the recognition accuracy of the quantized model, and finally output two serialized int8 model files.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. This application alleviates the instability of pseudo-label prediction in existing methods by integrating historical prediction information and cross-model integration. Compared with existing methods, the quality of pseudo-labels in the target domain is greatly improved, thereby achieving better cross-domain recognition results.

[0060] 2. This application alleviates the problem of insufficient sample feature representation capabilities by introducing a dual-model structure, achieving better cross-domain behavior recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The overall process of the model;

[0062] Figure 2 It is the model pre-training process;

[0063] Figure 3 It is the pseudo label prediction process;

[0064] Figure 4 Distillation for models;

[0065] Figure 5 Lightweight the model;

[0066] Figure 6 is the model reasoning process. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0068] Combination Figure 1 ,The overall process of this application can be divided into four steps.

[0069] S1: In the model pre-training stage, all source domain labeled videos will be sent to the behavior recognition model for supervised learning. In order to enhance the representation of cross-domain sample features, this application uses two behavior recognition models based on 3D CNN and 2D CNN for feature extraction. Figure 2 , all source domain videos are sampled with N video frames using the interval sampling method, and are fed into two networks for supervised training using the standard cross entropy loss. When S1 ends, two pre-trained source domain models are obtained.

[0070] S2: To increase the accuracy of generating pseudo labels for the target domain, this method is different from the existing cross-domain behavior recognition methods. On the one hand, this application alleviates the instability of the prediction results of target domain samples by integrating historical prediction information; on the other hand, it further improves the quality of pseudo labels by reorganizing pseudo labels across models and making full use of the complementary information of different structural models. Figure 3 , the following will explain the general steps of S2 from two aspects: model-level pseudo-label generation and cross-model pseudo-label generation:

[0071] (1) Model-level pseudo-label generation

[0072] The purpose of model-level pseudo-label generation is to generate a batch of target domain samples that the model is most confident of in 2D CNN and 3D CNN respectively to assign pseudo-labels. In the model training process of the previous stage, the target domain video will also be synchronously input into the two models for prediction along with the source domain video samples, and the historical prediction information of all target domain videos will be effectively recorded. Taking 2D CNN as an example, in one embodiment, assuming that the target domain video V t j The historical forecast of is represented as Among them I j Represents the 2D CNN model for the target domain video V t j The historical prediction results, each element in it represents the predicted category score, n j represents the total number of predictions. Therefore, the historical prediction information of all target domain videos can be further expressed as Where N t is the number of target domain videos. This application will make full use of the above information to calculate the weight information of each target domain sample, and then select the optimal pseudo label set.

[0073] Historical predictions The first prediction information Take as an example, first obtain the category information of the prediction in the following way:

[0074]

[0075] In the formula, express The predicted category, δ d (·) represents the value of the d-th dimension element of the corresponding feature vector. Using the same method, we can get the video V t j The set of all historical prediction categories is denoted as Finally, for this video, the weight coefficients of each category can be obtained by the following formula:

[0076]

[0077] in, Representative video V t j The category prediction weight, η(B, b) returns the number of times element b appears in set B, D represents the number of categories, n j is the total number of predictions, and D is the number of categories. In this process, in order to facilitate horizontal comparison between samples, the obtained category prediction weights were normalized. Finally, W j The index corresponding to the largest element in is considered to be the target domain video V t jThe predicted category. When W j When the maximum element in is not unique, the average prediction score of the sample is calculated by Formula 3, and the index corresponding to the maximum value is determined as the prediction category.

[0078]

[0079] Finally, video V t j The pseudo labels and their corresponding prediction weights can be expressed as follows:

[0080]

[0081]

[0082] In the formula, Indicates the final prediction label that integrates historical prediction information; sw j Indicates the predicted category In addition, in order to deal with the special case where the category weights of different videos are equal, this application also uses the average probability score Calculation model for V t j The average entropy value θ j , average information entropy θ j It reflects the uncertainty of the model about the sample and is also an important reference indicator for pseudo-label screening:

[0083]

[0084] According to the above information, all target domain samples are traversed, and the target domain video set can be re-expressed as where j = 0, 1, ..., N t When generating the pseudo-label set of the current model, K target domain pseudo-labels will be generated for each category in units of categories. In this process, this application will give priority to selecting samples with the largest prediction weights and assigning pseudo-labels. When the prediction weights are the same, samples with smaller average entropy values ​​will be selected. Finally, a model-level pseudo-label video set m containing K×D target domain videos is generated. 1 (D is the number of categories).

[0085] Finally, for another 3D CNN model, the same method is used to obtain the model-level pseudo label set m 2 , subsequent m 1 With m 2 Through category-level screening, a cross-model pseudo-label set will be finally generated to further improve the quality of pseudo-labels.

[0086] (2) Cross-model pseudo-label set

[0087] Since the network structures of 3D CNN and 2D CNN are completely different, the cross-domain recognition accuracy of the two is also quite different. 1 With m 2 Selectively extracting the target domain categories that the two models are best at, and then forming a cross-model pseudo label set, will be more conducive to generating high-quality pseudo labels.

[0088] In one embodiment, assuming that m 1 With m 2 The videos with pseudo labels τ are denoted as m 1[τ] = {V t 1j},m 2[τ] = {V t 2j}, where j = 1, 2, ..., K. The average entropy value sets corresponding to the videos in the two sets are expressed as and This application calculates the average value of all elements in the above set by the following formula, denoted as and

[0089]

[0090] Finally, the target domain video set with pseudo label τ in the generated cross-model pseudo label set will be sorted from m according to the minimum entropy principle. 1 and m 2 Adaptive extraction in the middle, that is:

[0091]

[0092] Among them, m [τ] In the generated cross-model pseudo label set, the pseudo label is the τ video set. By traversing all target domain categories in this way, a complete cross-model pseudo label set m = {m [0] , m [1] , ..., m [D]}. m will be added to the model update in the subsequent stage. In addition, to guide the subsequent model update, the two models will also determine the target domain category index that the two models are better at by comparing the entropy values ​​in this process, as follows:

[0093]

[0094] Finally, step S2 obtains the pseudo label set m and the category index class that the two models are good at 2D and class 3D Since step S2 will be executed multiple times in the overall process, the target domain pseudo-label videos in the pseudo-label set m will continue to accumulate, and the class 2Dand class 3D It is constantly being updated.

[0095] S3: In this step, all source domain videos, target domain videos, and the current stage pseudo-label video set m will be added to the network for training at the same time. Through the training in the S3 stage, the cross-domain recognition capabilities of the two models can be improved at the same time. The S3 step mainly includes two stages: model self-training and cross-model mutual distillation, which will be described in detail below.

[0096] First, the two models effectively improve their cross-domain recognition capabilities through model self-training. In this process, all source domain videos and target domain videos with pseudo labels will be trained using cross entropy loss. Assume that the 2D CNN model and the 3D CNN model are represented as G 2D and G 3D , with model G 2D For example, the specific loss function is expressed as follows:

[0097]

[0098] In the formula represents the source domain label after smoothing. Its calculation process is shown in Formula 14, where a is the smoothing coefficient; v s and v m denote the source domain and the target domain video with pseudo labels respectively; σ(·) is the Softmax activation function; δ d (·) represents the d-dimensional element of the corresponding feature; 1(·) is the indicator function. The final self-training loss of the 2D CNN and 3D CNN models can be expressed as:

[0099]

[0100]

[0101] In addition, to further enhance the cross-domain recognition capability of the algorithm, the two models with different structures will also perform dynamic model mutual distillation. During the training process, the two models continuously perform model prediction on the target domain training video. Taking the 2D CNN model as an example, the video V can be obtained by the following formula: t j The prediction results and corresponding confidence levels are:

[0102]

[0103] In the formula, and Represent the prediction category and model confidence respectively, Represents 2D CNN on video V t jBased on this method, the predicted categories and corresponding confidences of the two models for all target domain videos can be obtained respectively. Finally, this application will divide all target domain videos into two sets based on the above information, as shown in the following formula:

[0104]

[0105] Among them, class 2D With class 3D Calculated from equations 10 and 11; th is the confidence threshold set manually. 2D In the video samples in the model mutual distillation process, the domain knowledge will be transferred from the 2D CNN model to the 3D CNN model; for D 3D For the video samples in , the knowledge distillation direction is opposite. In the specific implementation, the output values ​​of the two models need to be smoothed first:

[0106]

[0107] Where, temp is the temperature coefficient; and Respectively represent the output values ​​of the two models for category d; and is the output value of category d after smoothing of the two models. Based on this, the following KL divergence loss function can be constructed:

[0108]

[0109] Finally, combined Figure 4 , the optimization loss of the two models in the entire S3 can be expressed as Equation 25. Stage S3 will last for P epochs. When each S3 stage ends, the algorithm will autonomously determine whether the total number of current training rounds has reached the preset total number of rounds. Only when the preset total number of rounds is reached will the algorithm end the training and enter the S4 stage, otherwise the algorithm will return to step S2 for loop execution.

[0110]

[0111] S4: At the end of step S3, two behavior recognition models G will be obtained respectively. 2D With G 3D In order to meet the needs of real-time processing in practical applications, this application will make full use of various inference acceleration technologies to reduce the model inference speed. Figure 5 The two trained model files will be quantized using TensorRT for PTQ. During this process, some labeled video data selected from the source domain dataset will be used as a calibration set to ensure the recognition accuracy of the quantized model. This process ultimately outputs two serialized int8 model files.

[0112] Combination Figure 6 , and finally, this application realizes the real-time reasoning of the behavior recognition model by maintaining a double-ended queue and multi-threading. First, thread 1 will read the input video frames of the video stream in real time. When the accumulated number of video frames read reaches N frames, the N frames of video will be packaged into the double-ended queue; then, thread 2 will continue to read the stored video frame set from the queue and perform model reasoning operations. Ultimately, only when the confidence of the reasoning exceeds a certain threshold, the video stream will be finally judged as a certain behavior category and the results will be displayed.

[0113] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A multi-model collaborative learning behavior recognition method applied to cross-domain scenarios, characterized in that: The steps include: S1, model pre-training; S2, output pseudo label set; S3, model update, if it is finished, go to the next step, if it is not finished, return and loop the above steps; S4. Model deployment.

2. The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios according to claim 1 is characterized in that: The step S1 includes: all source domain labeled videos are sent to the behavior recognition model for supervised learning, and two behavior recognition models based on 3D CNN and 2D CNN are used for feature extraction. All source domain videos are sampled using an interval sampling method to sample N video frames, and are sent to two networks respectively for supervised training using a standard cross entropy loss. At the end, two pre-trained source domain models are obtained.

3. The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios according to claim 2 is characterized in that: The step S2 includes: increasing the accuracy of generating target domain pseudo labels, on the one hand, alleviating the instability of target domain sample prediction results by fusing historical prediction information; on the other hand, further improving the quality of pseudo labels by reorganizing pseudo labels across models and making full use of complementary information of different structural models.

4. The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios according to claim 3 is characterized in that: Model-level pseudo-label generation: In the model training process of the previous stage, the target domain video will also be synchronously input into the two models for prediction along with the source domain video samples. All historical prediction information of the target domain video will be effectively recorded. When using 2DCNN, assuming that the target domain video V t j The historical forecast of is represented as Among them I j Represents the 2D CNN model for the target domain video V t j The historical prediction results, each element in it represents the predicted category score, n j represents the total number of predictions, so the historical prediction information of all target domain videos can be further expressed as Where N t Select the optimal pseudo-label set for the number of target domain videos. Historical predictions The first prediction information Take as an example, first obtain the category information of the prediction in the following way: In the formula, express The predicted category, δ d (·) represents the value of the d-th dimension element of the corresponding feature vector. Using the same method, we can get the video V t j The set of all historical prediction categories is denoted as Finally, for this video, the weight coefficients of each category can be obtained by the following formula: in, Representative video V t j The category prediction weight, η(B, b) returns the number of times element b appears in set B, D represents the number of categories, n j is the total number of predictions, D is the number of categories. In this process, in order to facilitate horizontal comparison between samples, the obtained category prediction weights are normalized. Finally, W j The index corresponding to the largest element in is considered to be the target domain video V t j The predicted category, when W j When the maximum element in is not unique, the average prediction score of the sample is calculated by formula 3, and the index corresponding to the maximum value is determined as the prediction category. Finally, video V t j The pseudo labels and their corresponding prediction weights can be expressed as follows: In the formula, Indicates the final prediction label that integrates historical prediction information; sw j Indicates the predicted category In addition, to deal with the special case where the category weights of different videos are equal, this application also uses the average probability score Calculation model for V t j The average entropy value θ j , average information entropy θ j It reflects the uncertainty of the model about the sample and is also an important reference indicator for pseudo-label screening: According to the above information, all target domain samples are traversed, and the target domain video set can be re-expressed as where j = 0, 1, ..., N t When generating the pseudo-label set of the current model, K target domain pseudo-labels are generated for each category in units of categories, and the samples with the largest prediction weight are selected to be assigned pseudo-labels. When the prediction weights are the same, samples with smaller average entropy values ​​are selected, and finally a model-level pseudo-label video set m1 containing K×D target domain videos is generated, where D is the number of categories. For the 3D CNN model, the same method is used to obtain the model-level pseudo label set m2. Subsequently, m1 and m2 will be filtered at the category level to finally generate a cross-model pseudo label set to further improve the quality of the pseudo labels.

5. The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios according to claim 4 is characterized in that: Cross-model pseudo label set: Selectively extract the target domain categories that the two models are best at from m1 and m2, and then form a cross-model pseudo label set to generate high-quality pseudo labels. Assume that the videos with pseudo labels τ in m1 and m2 are represented as m 1[τ] = {V t 1j },m 2[τ] = {V t 2j }, where j = 1, 2, ..., K, and the average entropy value sets corresponding to the videos in the two sets are expressed as and This application calculates the average value of all elements in the above set by the following formula, denoted as and Finally, the target domain video set with pseudo label τ in the generated cross-model pseudo label set will be adaptively extracted from m1 and m2 according to the minimum entropy principle, that is: Among them, m [τ] Indicates the generated cross-model pseudo-label set, the pseudo-label is the τ video set. By traversing all target domain categories in this way, the complete cross-model pseudo-label set m = {m [0] , m [1] , ..., m [D] }, m will be added to the model update in the subsequent stage. In addition, to guide the subsequent model update, the two models will also determine the target domain category index that the two models are better at by comparing the entropy values ​​in this process, as follows: Finally, step S2 obtains the pseudo label set m and the category index class that the two models are good at 2D and class 3D , because step S2 will be executed multiple times in the overall process, the target domain pseudo-label videos in the pseudo-label set m will continue to accumulate, and the class 2D and class 3D It is constantly being updated.

6. The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios according to claim 5 is characterized in that: The step S3 includes: all source domain videos, target domain videos, and the current stage pseudo-label video set m are simultaneously added to the network for training, including two stages: model self-training and cross-model mutual distillation.

7. The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios according to claim 6 is characterized in that: First, all source domain videos and target domain videos with pseudo labels will be trained using cross entropy loss. Assume that the 2D CNN model and 3D CNN model are represented as and , based on the model For example, the specific loss function is expressed as follows: Formula 12 Formula 13 Formula 14 In the formula represents the source domain label after smoothing, and its calculation process is shown in Formula 14, where a is the smoothing coefficient; and Represent the source domain and the target domain video with pseudo labels respectively; is the Softmax activation function; Indicates the d-dimensional elements of the corresponding features to be extracted; is the indicator function, and the final self-training losses of the 2D CNN and 3D CNN models can be expressed as: Formula 15 Formula 16 In addition, to further enhance the cross-domain recognition capability of the algorithm, the two models with different structures will also perform dynamic model mutual distillation. During the training process, the two models continuously perform model prediction on the target domain training video. Taking the 2D CNN model as an example, the video can be obtained by the following formula: The prediction results and corresponding confidence levels are: Formula 17 Formula 18 In the formula, and Represent the prediction category and model confidence respectively, Represents 2D CNN on video Based on this method, the predicted categories and corresponding confidences of the two models for all target domain videos can be obtained respectively. Finally, this application will divide all target domain videos into two sets based on the above information, as shown in the following formula: Formula 19 Formula 20 in, and Calculated from equations 10 and 11; is a manually set confidence threshold. In the video samples, during the process of model mutual distillation, the domain knowledge will be transferred from the 2D CNN model to the 3D CNN model; For the video samples in , the direction of knowledge distillation is opposite. In the specific implementation, the output values ​​of the two models need to be smoothed first: Formula 21 Formula 22 Where, temp is the temperature coefficient; and Respectively represent the output values ​​of the two models for category d; and is the output value of category d after smoothing of the two models. Based on this, the following KL divergence loss function can be constructed: Formula 23 Formula 24 Finally, the optimization loss of the two models in the entire S3 can be expressed as Equation 25. Stage S3 will last for P epochs. When each S3 stage ends, the algorithm will independently determine whether the total number of current training rounds has reached the preset total number of rounds. Only when the preset total number of rounds is reached will the algorithm end the training and enter the S4 stage. Otherwise, the algorithm will return to step S2 for loop execution. Formula 25.

8. The multi-model collaborative learning behavior recognition method applied to cross-domain scenarios according to claim 7 is characterized in that: The step S4 includes: when the step S3 is finished, two behavior recognition models G are obtained respectively. 2D With G 3D In order to meet the needs of real-time processing in practical applications, this application will make full use of various inference acceleration technologies to reduce the model inference speed. The two trained model files will be quantized using TensorRT for PTQ. In this process, some labeled video data selected from the source domain dataset will be used as a calibration set to ensure the recognition accuracy of the quantized model, and finally output two serialized int8 model files.

Images