Action recognition method, system and equipment based on multi-model aggregation and storage medium

Abstract

The invention relates to the technical field of computer vision, and particularly discloses a motion recognition method, system and device based on multi-model aggregation and a storage medium, and the method comprises the steps: obtaining a plurality of target path weights and aggregation motion recognition results corresponding to each sample video of a target domain; based on the action identification tag corresponding to each sample video, aggregating the action identification result, the quantized value of the instance-level transferability estimation index and a plurality of target path weights, calculating a loss value and performing iterative optimization; and inputting a to-be-detected video into the trained path generation network to obtain a plurality of target path weights corresponding to the to-be-detected video, and obtaining an aggregation action recognition result of the to-be-detected video according to the plurality of target path weights corresponding to the to-be-detected video and the correspondingly trained source domain model. According to the method, the adaptability of the source domain model to the target domain scene is improved, so that the action recognition accuracy of the model in the target domain scene is improved.

Description

Background Art

[0002] Video action recognition is an important task in video understanding, and its goal is to identify the action categories contained in the input video. However, building a large-scale dataset for real-world scenarios has extremely high manpower and economic costs. Unsupervised video domain adaptation is proposed as a solution. Labeled source domain data and another unlabeled target domain data are fed into the model, and by aligning the data of the source domain and the target domain, knowledge is transferred from the labeled source domain to the unlabeled target domain to overcome domain shift. Although traditional unsupervised video domain adaptation methods can effectively alleviate domain shifts between video domains, their source data needs to be accessed during the adaptation process, which will bring huge privacy leakage risks and huge transmission costs.

[0003] Existing benchmarks and tests for passive-domain video domain migration are all based on a single source model. These traditional benchmark configurations are designed to perform fair ablation comparisons. However, they ignore the fact that in real scenarios, there are many source models with different structures that can be utilized, and these different models provide rich source domain knowledge. However, in real scenarios, there are often multiple available source models, and different source models can provide richer source domain knowledge, thereby significantly improving the adaptation results. It is currently unclear how to extend single-model passive-domain video domain migration to multiple pre-trained models. Simply aggregating all source domain pre-trained models directly is undoubtedly inefficient, because not all source domain models have good transferability in the target domain.

[0004] The existing multi-model aggregation algorithms for source-domain migration in the image field assume that the structures of these source domain models are unified when considering model aggregation, for example, the structures of the source domain models are all ResNet models. This limits the knowledge that the model can learn in the source domain. It does not take into account the existence of models with different architectures for source domain tasks, such as ResNet50 and ResNet101 based on CNN architecture, and Vision-Transformer and Swin-Transformer models based on Transformer architecture.

[0005] In addition, compared with images, videos have more temporal information and an additional time dimension, which makes it impossible to directly transfer the original image methods to the video field. Targeted design is needed for the temporal information contained in the video. Moreover, at the instance level, the difference between images and videos is greater. When performing model aggregation, the multi-model aggregation algorithm for passive domain migration in the image field only considers different weights for different models from the domain level. This ignores the large differences between different videos from the instance perspective, and thus the problem of different preferences for each video instance model. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a method, system, device and storage medium for action recognition based on multi-model aggregation.

[0007] In a first aspect, the present invention provides an action recognition method based on multi-model aggregation, and the technical solution of the method is as follows:

[0008] S1. Input any sample video of the target domain into the initial path generation network, obtain N path weights corresponding to any sample video, and determine all path weights except the minimum path weight as the target path weight; wherein N ≥ 3 and N is a positive integer, and each path weight corresponds to a screened source domain model for action recognition;

[0009] S2, inputting the any sample video into the filtered source domain model corresponding to each target path weight, respectively, to obtain a plurality of first action recognition results corresponding to the any sample video, and based on each first action recognition result corresponding to the any sample video and the corresponding target path weight, obtaining an aggregated action recognition result corresponding to the any sample video;

[0010] S3, repeatedly executing S1-S2 to obtain multiple target path weights and aggregated action recognition results corresponding to each sample video in the target domain;

[0011] S4, based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the weights of multiple target paths, calculate and optimize the parameters of the screened source domain model according to the current loss value, and obtain an optimized source domain model, and optimize the parameters of the initial path generation network according to the quantized value of the instance-level transferability estimation index corresponding to each sample video, and obtain an optimized path generation network;

[0012] S5, taking the optimized path generation network as the initial path generation network, and the optimized source domain model as the screened source domain model, and returning to execute S3, until the iterative optimization condition is met, the initial path generation network is determined as the trained path generation network, and the screened source domain model is determined as the trained source domain model;

[0013] S6. Input the video to be tested into the trained path generation network to obtain the target path weight corresponding to the video to be tested, and input the video to be tested into the trained source domain models corresponding to each target path weight respectively to obtain multiple first action recognition results corresponding to the video to be tested, and based on each first action recognition result corresponding to the video to be tested and the corresponding target path weight, obtain the aggregated action recognition result corresponding to the video to be tested.

[0014] The beneficial effects of the action recognition method based on multi-model aggregation of the present invention are as follows:

[0015] The method of the present invention improves the adaptability of the source domain model to the target domain scene, thereby improving the action recognition accuracy of the model in the target domain scene.

[0016] Based on the above solution, the action recognition method based on multi-model aggregation of the present invention can also be improved as follows.

[0017] In an optional manner, the method further includes:

[0018] Determine the quantized value of transferability between each source domain model and the target domain, and sort them in descending order according to the quantized values ​​to obtain a target queue;

[0019] The source domain models corresponding to the first N quantized values ​​of the target queue are determined as the filtered source domain models.

[0020] In an optional manner, the step of determining a quantitative value of transferability between any source domain model and the target domain includes:

[0021] Based on a first target formula, a quantitative value of transferability between any source domain model and the target domain is determined; wherein the first target formula is: IC am =E(IC i )×SUTE×TC÷E(IC i ); IC i =H(h(V i )), IC am represents the quantitative value of the transferability between any source domain model and the target domain, V i represents the i-th sample video of the target domain, h(V i ) indicates V i The output result obtained by inputting into any source domain model is H(h(V i )) indicates the calculation of h(V i ), SUTE represents the quantized value of the passive domain transferable estimation index, TC represents the quantized value of temporal consistency, and IC i represents the quantitative value of the individual certainty of any instance of the source domain model, E(IC i ) indicates calculating IC with different values i The mathematical expectation of Indicates V i Video after time perturbation, D T represents the sample set of the target domain, Indicates the calculation of different values ​​of Η(h(V i ), Indicates the calculation of different values ​​of h(V i ), represents the predicted semantics of the sample video, represents the pseudo label of the sample video, Representation calculation KL divergence, h c (V i ) indicates V i The predicted probability of class c.

[0022] In an optional manner, the step of obtaining a quantized value of an instance-level transferability estimation index corresponding to any sample video includes:

[0023] Based on the second target formula, the quantitative value of the instance-level transferability estimation index corresponding to any sample video is calculated; wherein the second target formula is: Γ i represents the quantized value of the instance-level transferability estimation index corresponding to the i-th sample video, Express Perform softmax normalization processing.

[0024] In an optional manner, the step of calculating the current loss value based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the weights of multiple target paths includes:

[0025] Substitute the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and multiple target path weights into the target loss function to calculate the current loss value; wherein the target loss function is: L = θ1L cor +L SHTC , L represents the current loss value, θ1 is the hyperparameter, and L SHTC represents the original loss obtained based on the action recognition label corresponding to each sample video and the aggregated action recognition results, L cor represents the weight loss, n represents the number of sample videos, G1(V i ) represents the weights of multiple target paths of the i-th sample video.

[0026] In an optional manner, the iterative optimization condition is: a maximum number of iterations or convergence of a loss function.

[0027] In a second aspect, the present invention provides an action recognition system based on multi-model aggregation, and the technical solution of the system is as follows:

[0028] A first processing module, a second processing module, a third processing module, a fourth processing module, an iterative training module and a detection module;

[0029] The first processing module is used to: input any sample video of the target domain into the initial path generation network, obtain N path weights corresponding to any sample video, and determine all path weights except the minimum path weight as the target path weight; wherein N≥3 and N is a positive integer, and each path weight corresponds to a screened source domain model for action recognition;

[0030] The second processing module is used to: input any of the sample videos into the filtered source domain model corresponding to each target path weight, obtain multiple first action recognition results corresponding to the any of the sample videos, and obtain an aggregated action recognition result corresponding to the any of the sample videos based on each first action recognition result corresponding to the any of the sample videos and the corresponding target path weight;

[0031] The third processing module is used to: repeatedly call the first processing module to the second processing module to obtain multiple target path weights and aggregated action recognition results corresponding to each sample video in the target domain;

[0032] The fourth processing module is used to: calculate and optimize the parameters of the screened source domain model based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the multiple target path weights according to the current loss value, and obtain the optimized source domain model, and optimize the parameters of the initial path generation network according to the quantized value of the instance-level transferability estimation index corresponding to each sample video to obtain the optimized path generation network;

[0033] The iterative training module is used to: use the optimized path generation network as the initial path generation network, the optimized source domain model as the screened source domain model, and return to call the third processing module until the iterative optimization condition is met, and the initial path generation network is determined as the trained path generation network, and the screened source domain model is determined as the trained source domain model;

[0034] The detection module is used to: input the video to be tested into the trained path generation network to obtain the target path weight corresponding to the video to be tested, and input the video to be tested into the trained source domain model corresponding to each target path weight respectively to obtain multiple first action recognition results corresponding to the video to be tested, and based on each first action recognition result corresponding to the video to be tested and the corresponding target path weight, obtain the aggregated action recognition result corresponding to the video to be tested.

[0035] The beneficial effects of the action recognition system based on multi-model aggregation of the present invention are as follows:

[0036] The system of the present invention improves the adaptability of the source domain model to the target domain scene, thereby improving the action recognition accuracy of the model in the target domain scene.

[0037] Based on the above solution, the action recognition system based on multi-model aggregation of the present invention can also be improved as follows.

[0038] In an optional manner, the method further includes: a preprocessing module; the preprocessing module is used to:

[0039] Determine the quantized value of transferability between each source domain model and the target domain, and sort them in descending order according to the quantized values ​​to obtain a target queue;

[0040] The source domain models corresponding to the first N quantized values ​​of the target queue are determined as the filtered source domain models.

[0041] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0042] The invention comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the action recognition method based on multi-model aggregation of the present invention are implemented.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0044] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the action recognition method based on multi-model aggregation of the present invention.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0047] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for action recognition based on multi-model aggregation of the present invention;

[0048] Figure 2Schematic diagram of the training process of the source domain model library;

[0049] Figure 3 It is a schematic diagram of the overall principle;

[0050] Figure 4 A schematic diagram of the structure of an embodiment of an action recognition system based on multi-model aggregation of the present invention;

[0051] Figure 5 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0053] Figure 1 A flow chart of an embodiment of a method for motion recognition based on multi-model aggregation provided by the present invention is shown. The method for motion recognition based on multi-model aggregation can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the method for motion recognition based on multi-model aggregation by calling computer-readable instructions stored in a memory through a processor. Figure 1 As shown, the following steps are included:

[0054] S1. Input any sample video of the target domain into the initial path generation network, obtain N path weights corresponding to any sample video, and determine all path weights except the minimum path weight as the target path weight.

[0055] Wherein, N≥3 and N is a positive integer, and each path weight corresponds to a screened source domain model for action recognition. In this embodiment, the value of N defaults to 3. The initial path generation network is an untrained path generation network, and the path generation network is a standard C3d network, which includes 8 convolutional layers, 5 maximum pooling layers, 2 fully connected layers, and 1 softmax output layer. The softmax output layer outputs 3 path weights, and the smallest path weight is recorded as 0. The remaining path weights are retained and determined as the target path weights. The path generation network uses the following formula to calculate the path weight: G1(V i)=f top2 (G0(V i ),k); k is the number of path weights, G0(V i ) is the N path weights corresponding to the i-th sample video, G1(V i ) are the N-1 target path weights corresponding to the i-th sample video.

[0056] S2. Input any of the sample videos into the filtered source domain model corresponding to each target path weight, obtain multiple first action recognition results corresponding to any of the sample videos, and obtain the aggregated action recognition result corresponding to any of the sample videos based on each first action recognition result corresponding to any of the sample videos and the corresponding target path weight.

[0057] The first action recognition result is the action recognition result output by the source domain model. Each target path weight corresponds to a filtered source domain model, and each filtered source domain model outputs a first action recognition result. The following formula is used to calculate the aggregated action recognition result corresponding to any sample video: output i represents the aggregated action recognition result corresponding to the i-th sample video, H j (V i ) indicates that V i The first action recognition result obtained by inputting the jth filtered source domain model, m represents the number of filtered source domain models, and A represents the Perform cumulative summation.

[0058] S3. Repeat S1-S2 to obtain multiple target path weights and aggregated action recognition results corresponding to each sample video in the target domain.

[0059] For each sample video in the target domain, S1-S2 are repeatedly performed until a plurality of target path weights and aggregated action recognition results corresponding to each sample video in the target domain are obtained.

[0060] S4. Based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the weights of multiple target paths, calculate and optimize the parameters of the screened source domain model based on the current loss value to obtain an optimized source domain model, and optimize the parameters of the initial path generation network based on the quantized value of the instance-level transferability estimation index corresponding to each sample video to obtain an optimized path generation network.

[0061] In S4, the step of calculating the current loss value based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the weights of multiple target paths includes:

[0062] Substitute the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and multiple target path weights into the target loss function to calculate the current loss value. Wherein, the target loss function is: L = θ1L cor +L SHTC , L represents the current loss value, θ1 is the hyperparameter, and L SHTC represents the original loss obtained based on the action recognition label corresponding to each sample video and the aggregated action recognition results, L cor represents the weight loss, n represents the number of sample videos, G1(V i ) represents the weights of multiple target paths of the i-th sample video. It should be noted that the value of θ1 is 0.01.

[0063] Based on the second objective formula, the quantitative value of the instance-level transferability estimation index corresponding to any sample video is calculated. The second objective formula is: Γ i represents the quantized value of the instance-level transferability estimation index corresponding to the i-th sample video, Express Perform softmax normalization processing.

[0064] It should be noted that based on the current loss value, the parameters of the screened source domain model are optimized to obtain the optimized source domain model, and according to the quantized value of the instance-level transferability estimation index corresponding to each sample video, the parameters of the initial path generation network are optimized to obtain the specific process of optimizing the path generation network, which is the existing technology and will not be elaborated here.

[0065] S5. Use the optimized path generation network as the initial path generation network and the optimized source domain model as the screened source domain model, and return to execute S3 until an iterative optimization condition is met, and determine the initial path generation network as the trained path generation network and the screened source domain model as the trained source domain model.

[0066] Among them, the iterative optimization conditions are: the maximum number of iterations or the convergence of the loss function.

[0067] S6. Input the video to be tested into the trained path generation network to obtain the target path weight corresponding to the video to be tested, and input the video to be tested into the trained source domain models corresponding to each target path weight respectively to obtain multiple first action recognition results corresponding to the video to be tested, and based on each first action recognition result corresponding to the video to be tested and the corresponding target path weight, obtain the aggregated action recognition result corresponding to the video to be tested.

[0068] The video to be tested is the video data for action recognition in this embodiment. Except for calculating the loss value, the detection method of the video to be tested is the same as that of the sample video, which will not be described in detail here.

[0069] In an optional manner, the method further includes:

[0070] Determine the quantized value of the transferability between each source domain model and the target domain, and sort them in descending order according to the quantized values ​​to obtain a target queue.

[0071] Among them, Figure 2 As shown in the figure, based on the mmaction2 framework, a pre-trained model provided for the video action recognition task is selected from a source domain model library with various structures, and trained on the source domain dataset to finally obtain multiple source domain models. Each source domain model corresponds to a quantitative value of transferability.

[0072] Specifically, the step of determining a quantitative value of transferability between any source domain model and the target domain includes: determining a quantitative value of transferability between any source domain model and the target domain based on a first target formula.

[0073] Among them, the first target formula is: IC am =E(IC i )×SUTE×TC÷E(IC i );

[0074]

[0075] IC i =H(h(V i )), IC am represents the quantitative value of the transferability between any source domain model and the target domain, V i represents the i-th sample video of the target domain, h(V i ) indicates V i The output result obtained by inputting into any source domain model is H(h(V i )) indicates the calculation of h(V i), SUTE represents the quantized value of the passive domain transferable estimation index, TC represents the quantized value of temporal consistency, and IC i represents the quantitative value of the individual certainty of any instance of the source domain model, E(IC i ) indicates calculating IC with different values i The mathematical expectation of Indicates V i Perform time perturbation (on V i The video after random time enhancement of the sampled frames in D T represents the sample set of the target domain, Indicates the calculation of different values ​​of Η(h(V i ), Indicates the calculation of different values ​​of h(V i ), represents the predicted semantics of the sample video, represents the pseudo label of the sample video, Representation calculation KL divergence, h c (V i ) indicates V i The predicted probability of class c.

[0076] The source domain models corresponding to the first N quantized values ​​of the target queue are determined as the filtered source domain models.

[0077] The default value of N is 3.

[0078] In an optional manner, the step of obtaining a quantized value of an instance-level transferability estimation index corresponding to any sample video includes:

[0079] Based on the second objective formula, the quantitative value of the instance-level transferability estimation index corresponding to any sample video is calculated.

[0080] Among them, the second objective formula is: Γ i represents the quantized value of the instance-level transferability estimation index corresponding to the i-th sample video, Express Perform softmax normalization processing.

[0081] It should be noted that Figure 3The schematic diagram of the principle of this embodiment is shown. This embodiment conducts experiments on Daily-DA, a large public dataset for video action recognition, to verify the effectiveness of this solution. DailyDA is another large-scale cross-domain action recognition benchmark. It includes 4 datasets, namely ARID (A), HMDB51 (H), Moments-in-Time (M) and Kinetics (K). Videos from 8 shared classes are used for cross-domain evaluation. Because the models in the source domain model library are pre-trained in Kinetics, in order to ensure the accuracy of the performance, the Kinetics dataset is selected to be removed. After the removal, there are three datasets left, resulting in a total of 6 tasks. For the accuracy indicator, the Mean-1 accuracy is selected, which represents the average accuracy of the Top-1 of each category. Compared with the current domain adaptation advanced algorithms SHOT, STHC and multi-model domain adaptation advanced algorithms CAiDA, KD3A, DECISION, the comparison results are shown in Table 1 below. It can be seen from Table 1 that the method of this embodiment greatly improves the action recognition accuracy of the source domain model in the target domain.

[0082] Table 1:

[0083] Method M->A H->A A->M H->M A->H M->H AVG SHOT 58.95 39.74 44.50 48.00 60.00 69.17 53.39 STHC 57.72 49.67 45.00 48.25 59.17 69.58 54.90 Decision 52.47 49.73 45.75 47.50 57.92 68.75 53.69 kd3A) 57.12 50.09 45.25 48.25 59.17 69.17 54.84 Caida 57.57 48.13 45.25 47.75 59.17 68.75 54.44 This program 63.37 48.25 51.00 51.25 68.75 79.58 60.37

[0084] The technical solution of this embodiment improves the adaptability of the source domain model to the target domain scene, thereby improving the action recognition accuracy of the model in the target domain scene.

[0085] Figure 4 FIG. 2 is a schematic diagram showing a structure of an embodiment of a multi-model aggregation-based action recognition system 200 provided by the present invention. Figure 4 As shown, the system 200 includes: a first processing module 210, a second processing module 220, a third processing module 230, a fourth processing module 240, an iterative training module 250 and a detection module 260;

[0086] The first processing module 210 is used to: input any sample video of the target domain into the initial path generation network, obtain N path weights corresponding to any sample video, and determine all path weights except the minimum path weight as the target path weight; wherein N≥3 and N is a positive integer, and each path weight corresponds to a screened source domain model for action recognition;

[0087] The second processing module 220 is used to: input the any sample video into the filtered source domain model corresponding to each target path weight, obtain multiple first action recognition results corresponding to the any sample video, and obtain the aggregated action recognition result corresponding to the any sample video based on each first action recognition result corresponding to the any sample video and the corresponding target path weight;

[0088] The third processing module 230 is used to: repeatedly call the first processing module 120 to the second processing module 220 to obtain multiple target path weights and aggregated action recognition results corresponding to each sample video in the target domain;

[0089] The fourth processing module 240 is used to: calculate and optimize the parameters of the screened source domain model based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the weights of multiple target paths according to the current loss value, and obtain the optimized source domain model, and optimize the parameters of the initial path generation network according to the quantized value of the instance-level transferability estimation index corresponding to each sample video to obtain the optimized path generation network;

[0090] The iterative training module 250 is used to: use the optimized path generation network as the initial path generation network, the optimized source domain model as the screened source domain model, and return to call the third processing module 230 until the iterative optimization condition is met, and the initial path generation network is determined as the trained path generation network, and the screened source domain model is determined as the trained source domain model;

[0091] The detection module 260 is used to: input the video to be tested into the trained path generation network to obtain the target path weight corresponding to the video to be tested, and input the video to be tested into the trained source domain models corresponding to each target path weight respectively to obtain multiple first action recognition results corresponding to the video to be tested, and based on each first action recognition result corresponding to the video to be tested and the corresponding target path weight, obtain the aggregated action recognition result corresponding to the video to be tested.

[0092] In an optional manner, the method further includes: a preprocessing module; the preprocessing module is used to:

[0093] Determine the quantized value of transferability between each source domain model and the target domain, and sort them in descending order according to the quantized values ​​to obtain a target queue;

[0094] The source domain models corresponding to the first N quantized values ​​of the target queue are determined as the filtered source domain models.

[0095] In an optional manner, the preprocessing module is specifically used for:

[0096] Based on a first target formula, a quantitative value of transferability between any source domain model and the target domain is determined; wherein the first target formula is: IC am =E(IC i )×SUTE×TC÷E(ICi ); IC i =H(h(V i )), IC am represents the quantitative value of the transferability between any source domain model and the target domain, V i represents the i-th sample video of the target domain, h(V i ) indicates V i The output result obtained by inputting into any source domain model is H(h(V i )) indicates the calculation of h(V i ), SUTE represents the quantized value of the passive domain transferable estimation index, TC represents the quantized value of temporal consistency, and IC i represents the quantitative value of the individual certainty of any instance of the source domain model, E(IC i ) indicates calculating IC with different values i The mathematical expectation of Indicates V i Video after time perturbation, D T represents the sample set of the target domain, Indicates the calculation of different values ​​of Η(h(V i ), Indicates the calculation of different values ​​of h(V i ), represents the predicted semantics of the sample video, represents the pseudo label of the sample video, Representation calculation KL divergence, h c (V i ) indicates V i The predicted probability of class c.

[0097] In an optional manner, the method further includes: an acquisition module; the acquisition module is used to:

[0098] The step of obtaining a quantitative value of an instance-level transferability estimation index corresponding to any sample video includes:

[0099] Based on the second target formula, the quantitative value of the instance-level transferability estimation index corresponding to any sample video is calculated; wherein the second target formula is: Γ i represents the quantized value of the instance-level transferability estimation index corresponding to the i-th sample video, Express Perform softmax normalization processing.

[0100] In an optional manner, the fourth processing module 240 is specifically used to:

[0101] Substitute the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and multiple target path weights into the target loss function to calculate the current loss value; wherein the target loss function is: L = θ1L cor +L SHTC , L represents the current loss value, θ1 is the hyperparameter, and L SHTC represents the original loss obtained based on the action recognition label corresponding to each sample video and the aggregated action recognition results, L cor represents the weight loss, n represents the number of sample videos, G1(V i ) represents the weights of multiple target paths of the i-th sample video.

[0102] In an optional manner, the iterative optimization condition is: a maximum number of iterations or convergence of a loss function.

[0103] The technical solution of this embodiment improves the adaptability of the source domain model to the target domain scene, thereby improving the action recognition accuracy of the model in the target domain scene.

[0104] For the above-mentioned parameters and steps for each module to implement corresponding functions in the action recognition system 200 based on multi-model aggregation in this embodiment, reference can be made to the parameters and steps in the embodiment of the action recognition method based on multi-model aggregation above, and no further details will be given here.

[0105] like Figure 5 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above-mentioned action recognition methods based on multi-model aggregation, specifically:

[0106] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any one of the motion recognition methods based on multi-model aggregation provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be repeated here.

[0107] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned action recognition methods based on multi-model aggregation.

[0108] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0109] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned action recognition methods based on multi-model aggregation.

[0110] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0111] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0112] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0113] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An action recognition method based on multi-model aggregation, characterized in that: include: S1. Input any sample video of the target domain into the initial path generation network, obtain N path weights corresponding to any sample video, and determine all path weights except the minimum path weight as the target path weight; wherein N ≥ 3 and N is a positive integer, and each path weight corresponds to a screened source domain model for action recognition; S2, inputting the any sample video into the filtered source domain model corresponding to each target path weight, respectively, to obtain a plurality of first action recognition results corresponding to the any sample video, and based on each first action recognition result corresponding to the any sample video and the corresponding target path weight, obtaining an aggregated action recognition result corresponding to the any sample video; S3, repeatedly executing S1-S2 to obtain multiple target path weights and aggregated action recognition results corresponding to each sample video in the target domain; S4, based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the weights of multiple target paths, calculate and optimize the parameters of the screened source domain model according to the current loss value, and obtain an optimized source domain model, and optimize the parameters of the initial path generation network according to the quantized value of the instance-level transferability estimation index corresponding to each sample video, and obtain an optimized path generation network; S5, taking the optimized path generation network as the initial path generation network, and the optimized source domain model as the screened source domain model, and returning to execute S3, until the iterative optimization condition is met, the initial path generation network is determined as the trained path generation network, and the screened source domain model is determined as the trained source domain model; S6. Input the video to be tested into the trained path generation network to obtain the target path weight corresponding to the video to be tested, and input the video to be tested into the trained source domain models corresponding to each target path weight respectively to obtain multiple first action recognition results corresponding to the video to be tested, and based on each first action recognition result corresponding to the video to be tested and the corresponding target path weight, obtain the aggregated action recognition result corresponding to the video to be tested.

2. The action recognition method based on multi-model aggregation according to claim 1 is characterized in that: Also includes: Determine the quantized value of transferability between each source domain model and the target domain, and sort them in descending order according to the quantized values ​​to obtain a target queue; The source domain models corresponding to the first N quantized values ​​of the target queue are determined as the filtered source domain models.

3. The action recognition method based on multi-model aggregation according to claim 2 is characterized in that: The step of determining a quantitative value of transferability between any source domain model and the target domain comprises: Based on a first target formula, a quantitative value of transferability between any source domain model and the target domain is determined; wherein the first target formula is: IC am =E(IC i )×SUTE×TC÷E(IC i ); IC i =H(h(V i )), IC am represents the quantitative value of the transferability between any source domain model and the target domain, V i represents the i-th sample video of the target domain, h(V i ) indicates V i The output result obtained by inputting into any source domain model is H(h(V i )) indicates the calculation of h(V i ), SUTE represents the quantized value of the passive domain transferable estimation index, TC represents the quantized value of temporal consistency, and IC i represents the quantitative value of the individual certainty of any instance of the source domain model, E(IC i ) indicates calculating IC with different values i The mathematical expectation of Indicates V i Video after time perturbation, D T represents the sample set of the target domain, Indicates the calculation of different values ​​of Η(h(V i ), Indicates the calculation of different values ​​of h(V i ), represents the predicted semantics of the sample video, represents the pseudo label of the sample video, Representation calculation KL divergence, h c (V i ) indicates V i The predicted probability of class c.

4. The action recognition method based on multi-model aggregation according to claim 3 is characterized in that: The step of obtaining a quantized value of an instance-level transferability estimation index corresponding to any sample video includes: Based on the second target formula, the quantitative value of the instance-level transferability estimation index corresponding to any sample video is calculated; wherein the second target formula is: Γ i represents the quantized value of the instance-level transferability estimation index corresponding to the i-th sample video, Express Perform softmax normalization processing.

5. The action recognition method based on multi-model aggregation according to claim 4 is characterized in that: The step of calculating the current loss value based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the weights of multiple target paths includes: Substitute the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and multiple target path weights into the target loss function to calculate the current loss value; wherein the target loss function is: L = θ1L cor +L SHTC , L represents the current loss value, θ1 is the hyperparameter, and L SHTC represents the original loss obtained based on the action recognition label corresponding to each sample video and the aggregated action recognition results, L cor represents the weight loss, n represents the number of sample videos, G1(V i ) represents the weights of multiple target paths of the i-th sample video.

6. The action recognition method based on multi-model aggregation according to any one of claims 1 to 5, characterized in that: The iterative optimization condition is: the maximum number of iterations or the convergence of the loss function.

7. An action recognition system based on multi-model aggregation, characterized in that: include: A first processing module, a second processing module, a third processing module, a fourth processing module, an iterative training module and a detection module; The first processing module is used to: input any sample video of the target domain into the initial path generation network, obtain N path weights corresponding to any sample video, and determine all path weights except the minimum path weight as the target path weight; wherein N≥3 and N is a positive integer, and each path weight corresponds to a screened source domain model for action recognition; The second processing module is used to: input any of the sample videos into the filtered source domain model corresponding to each target path weight, obtain multiple first action recognition results corresponding to the any of the sample videos, and obtain an aggregated action recognition result corresponding to the any of the sample videos based on each first action recognition result corresponding to the any of the sample videos and the corresponding target path weight; The third processing module is used to: repeatedly call the first processing module to the second processing module to obtain multiple target path weights and aggregated action recognition results corresponding to each sample video in the target domain; The fourth processing module is used to: calculate and optimize the parameters of the screened source domain model based on the action recognition label corresponding to each sample video, the aggregated action recognition result, the quantized value of the instance-level transferability estimation index, and the multiple target path weights according to the current loss value, and obtain the optimized source domain model, and optimize the parameters of the initial path generation network according to the quantized value of the instance-level transferability estimation index corresponding to each sample video to obtain the optimized path generation network; The iterative training module is used to: use the optimized path generation network as the initial path generation network, the optimized source domain model as the screened source domain model, and return to call the third processing module until the iterative optimization condition is met, and the initial path generation network is determined as the trained path generation network, and the screened source domain model is determined as the trained source domain model; The detection module is used to: input the video to be tested into the trained path generation network to obtain the target path weight corresponding to the video to be tested, and input the video to be tested into the trained source domain model corresponding to each target path weight respectively to obtain multiple first action recognition results corresponding to the video to be tested, and based on each first action recognition result corresponding to the video to be tested and the corresponding target path weight, obtain the aggregated action recognition result corresponding to the video to be tested.

8. The action recognition system based on multi-model aggregation according to claim 7 is characterized in that: Also includes: Preprocessing module; The pre-processing module is used for: Determine the quantized value of transferability between each source domain model and the target domain, and sort them in descending order according to the quantized values ​​to obtain a target queue; The source domain models corresponding to the first N quantized values ​​of the target queue are determined as the filtered source domain models.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the action recognition method based on multi-model aggregation as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the action recognition method based on multi-model aggregation as described in any one of claims 1 to 6.

Images