Method and device for training video action recognition model, electronic device and medium

Abstract

The application provides a video action recognition model training method and device, electronic equipment and medium, and relates to the field of financial technology. The method uses a video action recognition model to perform feature extraction on a second preset number multiplied by a fourth preset number of sample videos in each target sub-sample set, generating a feature memory library for each target sub-sample set; uses the feature memory library of each target sub-sample set for contrast learning, obtaining a loss value of contrast learning for each target sub-sample set; accumulates the loss value of contrast learning for the first preset number of target sub-sample sets as a whole loss function, and then updates the parameters of the video action recognition model using the whole loss function; the above steps are iteratively executed until the iteration end condition is met, and a target video action recognition model is obtained. The embodiment solves the problem that the traditional method can cause the model to not converge, and improves the accuracy of video action recognition by the target video action recognition model.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a training method and apparatus, electronic device and medium for a video action recognition model. Background Technology

[0002] Video action recognition tasks analyze behavior and identify anomalous behavior. For example, a video action recognition task might involve recording one or more videos and analyzing the behaviors within them. Anomalous behaviors are rare, resulting in a very small sample size, meaning very little available labeled data. In such cases, few-shot techniques can be used to achieve "few-shot video action recognition," where "few-shot" refers to a small number of samples. Humans can recognize a new object with very few samples and remember different animals from pictures in an animal book, so few-shot learning can also be used to achieve video action recognition. However, although few-shot learning can achieve video action recognition with a small number of samples, it often faces the problem of low recognition accuracy.

[0003] For example, in specific business scenarios, banks need to use real-time intelligent monitoring systems to continuously monitor ATMs (Automated Teller Machines) and bank entrances 24 / 7 to ensure the security of their branches. This monitoring system combines video and artificial intelligence technologies to identify abnormal events, enabling pre-event warnings and post-event retrospective analysis, thereby reliably ensuring the security management of bank branches. Therefore, accurately achieving video motion recognition to identify abnormal events with a small sample size becomes a pressing technical problem that needs to be solved. Summary of the Invention

[0004] In view of the above problems, this application is made to provide a training method, apparatus, electronic device, and medium for a video action recognition model that overcomes or at least partially solves the above problems. The technical solution is as follows:

[0005] Firstly, a training method for a video action recognition model is provided, including:

[0006] Based on a pre-built support set for training a video action recognition model, a first preset number of target sub-sample sets are generated; wherein the support set contains a second preset number of categories, and each category contains a third preset number of sample videos; the first preset number is less than or equal to the third preset number, and each target sub-sample set in the first preset number of target sub-sample sets contains the second preset number of categories, and each category contains a fourth preset number of sample videos.

[0007] The video action recognition model is used to extract features from the sample videos of each target sub-sample set by multiplying the second preset number by the fourth preset number, thereby generating a feature memory library for each target sub-sample set.

[0008] Comparative learning is performed using the feature memory of each target subsample set to obtain the loss value of the comparative learning for each target subsample set;

[0009] Based on the loss value of the comparative learning of each target sub-sample set, the cumulative loss value of the comparative learning of the first preset number of target sub-sample sets is used as the overall loss function, and then the parameters of the video action recognition model are updated using the overall loss function.

[0010] The above steps are executed iteratively until the iteration termination condition is met, resulting in the target video action recognition model.

[0011] In one possible implementation, generating a first preset number of target subsample sets based on a pre-built support set used for training the video action recognition model includes:

[0012] A temporary set is initialized based on a pre-built support set for training the video action recognition model, wherein the temporary set contains the second preset number of categories, and each category contains the third preset number of sample videos;

[0013] Randomly select one sample video from each of the second preset number of categories in the temporary set to generate a first initial subsample set, wherein the first initial subsample set contains the second preset number of categories, and each category contains one sample video;

[0014] Update the temporary set, and randomly select a sample video from each of the second preset number of categories in the updated temporary set to generate a second initial sub-sample set, wherein the second initial sub-sample set contains the second preset number of categories, and each category contains a sample video;

[0015] Similarly, the first preset number of initial sub-sample sets are generated, wherein each initial sub-sample set contains the second preset number of categories, and each category contains one sample video;

[0016] Enhancement processing is performed on the sample videos in each of the initial subsample sets in the first preset number of initial subsample sets to generate the first preset number of target subsample sets.

[0017] In one possible implementation, the sample videos in each of the initial sub-sample sets within the first preset number of initial sub-sample sets are enhanced to generate the first preset number of target sub-sample sets, including:

[0018] Using the fourth preset number of enhancement methods, the sample videos in each initial subsample set in the first preset number of initial subsample sets are enhanced to generate the first preset number of target subsample sets.

[0019] In one possible implementation, the step of performing contrastive learning using the feature memory of each target sub-sample set to obtain the loss value of the contrastive learning for each target sub-sample set includes:

[0020] Comparative learning is performed using the feature memory of each target subsample set to obtain the loss value of comparative learning for any category of sample videos in each target subsample set;

[0021] Based on the contrastive learning loss value of any category sample video in each target subsample set, calculate the contrastive learning loss value of all category sample videos in each target subsample set, and use it as the contrastive learning loss value of each target subsample set.

[0022] In one possible implementation, the step of performing contrastive learning using the feature memory of each target sub-sample set to obtain the contrastive learning loss value for any category of sample videos in each target sub-sample set includes:

[0023] Contrastive learning is performed using the feature memory of each target subsample set. Based on the positive and negative sample pairs in any category of sample videos in each target subsample set, the loss value of contrastive learning for any category of sample videos in each target subsample set is obtained. Among them, the positive sample pairs come from different enhanced versions of the same sample video, and the negative sample pairs come from different sample videos.

[0024] In one possible implementation, after obtaining the target video action recognition model, the method further includes:

[0025] The video data to be identified is acquired, and the target video action recognition model is used to identify the video data to obtain the action category of the video data to be identified.

[0026] Secondly, a training device for a video action recognition model is provided, comprising:

[0027] The first generation module is configured to generate a first preset number of target subsample sets based on a pre-constructed support set for training a video action recognition model; wherein the support set contains a second preset number of categories, and each category contains a third preset number of sample videos; the first preset number is less than or equal to the third preset number, and each target subsample set in the first preset number of target subsample sets contains the second preset number of categories, and each category contains a fourth preset number of sample videos.

[0028] The second generation module is used to extract features from the sample videos of each target sub-sample set by multiplying the second preset number by the fourth preset number using the video action recognition model, and generate a feature memory library for each target sub-sample set.

[0029] The contrastive learning module is used to perform contrastive learning using the feature memory of each target subsample set to obtain the contrastive learning loss value for each target subsample set.

[0030] The parameter update module is used to accumulate the loss values ​​of the comparative learning of the first preset number of target sub-sample sets as the overall loss function based on the loss values ​​of the comparative learning of each target sub-sample set, and then use the overall loss function to update the parameters of the video action recognition model.

[0031] The iteration module is used to iteratively execute the above steps until the iteration termination condition is met, thus obtaining the target video action recognition model.

[0032] In one possible implementation, the first generation module is further configured to:

[0033] A temporary set is initialized based on a pre-built support set for training the video action recognition model, wherein the temporary set contains the second preset number of categories, and each category contains the third preset number of sample videos;

[0034] Randomly select one sample video from each of the second preset number of categories in the temporary set to generate a first initial subsample set, wherein the first initial subsample set contains the second preset number of categories, and each category contains one sample video;

[0035] Update the temporary set, and randomly select a sample video from each of the second preset number of categories in the updated temporary set to generate a second initial sub-sample set, wherein the second initial sub-sample set contains the second preset number of categories, and each category contains a sample video;

[0036] Similarly, the first preset number of initial sub-sample sets are generated, wherein each initial sub-sample set contains the second preset number of categories, and each category contains one sample video;

[0037] Enhancement processing is performed on the sample videos in each of the initial subsample sets in the first preset number of initial subsample sets to generate the first preset number of target subsample sets.

[0038] In one possible implementation, the first generation module is further configured to:

[0039] Using the fourth preset number of enhancement methods, the sample videos in each initial subsample set in the first preset number of initial subsample sets are enhanced to generate the first preset number of target subsample sets.

[0040] In one possible implementation, the contrastive learning module is further used for:

[0041] Comparative learning is performed using the feature memory of each target subsample set to obtain the loss value of comparative learning for any category of sample videos in each target subsample set;

[0042] Based on the contrastive learning loss value of any category sample video in each target subsample set, calculate the contrastive learning loss value of all category sample videos in each target subsample set, and use it as the contrastive learning loss value of each target subsample set.

[0043] In one possible implementation, the contrastive learning module is further used for:

[0044] Contrastive learning is performed using the feature memory of each target subsample set. Based on the positive and negative sample pairs in any category of sample videos in each target subsample set, the loss value of contrastive learning for any category of sample videos in each target subsample set is obtained. Among them, the positive sample pairs come from different enhanced versions of the same sample video, and the negative sample pairs come from different sample videos.

[0045] In one possible implementation, the device further includes an identification module for:

[0046] After obtaining the target video action recognition model, the video data to be recognized is acquired, and the target video action recognition model is used to recognize the video data to obtain the action category of the video data to be recognized.

[0047] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the training method for the video action recognition model described in any of the preceding claims.

[0048] Fourthly, a storage medium is provided that stores a computer program, wherein the computer program is configured to execute the training method for the video action recognition model described in any of the preceding claims during runtime.

[0049] Using the above technical solutions, the video action recognition model training method, apparatus, electronic device, and medium provided in this application embodiment can generate a first preset number of target sub-sample sets based on a pre-constructed support set for training the video action recognition model; wherein the support set contains a second preset number of categories, and each category contains a third preset number of sample videos; the first preset number is less than or equal to the third preset number, and each target sub-sample set in the first preset number contains a second preset number of categories, and each category contains a fourth preset number of sample videos; the video action recognition model is used to extract features from the sample videos in each target sub-sample set multiplied by the fourth preset number, generating a feature memory for each target sub-sample set; the feature memory for each target sub-sample set is used for comparative learning to obtain the comparative learning loss value for each target sub-sample set; based on the comparative learning loss value of each target sub-sample set, the comparative learning loss value of the first preset number of target sub-sample sets is accumulated as the overall loss function, and then the parameters of the video action recognition model are updated using the overall loss function; the above steps are iteratively executed until the iteration termination condition is met, and the target video action recognition model is obtained. As can be seen, the embodiments of this application can train and iterate the video action recognition model based on target subsamples. Unlike traditional contrastive learning, this embodiment uses a delayed update model method to improve contrastive learning. That is, the loss value of the contrastive learning of the first preset number of target subsample sets is accumulated as the overall loss function, and then the parameters of the video action recognition model are updated using the overall loss function. When the iteration termination condition is met, the target video action recognition model is obtained, which solves the problem that the traditional method will cause the model to not converge, and at the same time improves the accuracy of the target video action recognition model in video action recognition.

[0050] Furthermore, the real-time intelligent monitoring system at bank ATMs and bank entrances collects video footage. The collected video footage data is converted into data that can be used for subsequent processing. Then, the target video action recognition model of this embodiment is used to perform target detection and target recognition on the preprocessed data to determine whether there is abnormal behavior, such as smashing or wielding weapons. Once abnormal behavior is detected, the system will automatically trigger an alarm mechanism. Different alarm methods will be adopted according to different scenarios and needs, such as sound alarm, image alarm, and SMS alarm. The processed data will be stored in the database for subsequent analysis and backtracking. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0052] Figure 1 A flowchart illustrating the training method of the video action recognition model provided in an embodiment of this application is shown;

[0053] Figure 2 This illustration shows a schematic diagram of the supported sets and query sets in video action recognition provided in an embodiment of this application;

[0054] Figure 3 This paper shows a structural diagram of the training device for the video action recognition model provided in an embodiment of this application;

[0055] Figure 4 A structural diagram of a training device for a video action recognition model provided in another embodiment of this application is shown;

[0056] Figure 5 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0057] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0058] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0059] To address the aforementioned technical problems, embodiments of this application provide a training method for a video action recognition model, such as... Figure 1 As shown, the training method for this video action recognition model may include the following steps S101 to S105:

[0060] Step S101: Generate a first preset number of target subsample sets based on a pre-built support set for training the video action recognition model; wherein the support set contains a second preset number of categories, and each category contains a third preset number of sample videos; the first preset number is less than or equal to the third preset number, and each target subsample set in the first preset number of target subsample sets contains a second preset number of categories, and each category contains a fourth preset number of sample videos.

[0061] In this step, the first, second, third, and fourth preset quantities can be set according to actual needs, and this embodiment does not impose any restrictions on them. For example, the first preset quantity is K, the second preset quantity is C, the third preset quantity is K, and the fourth preset quantity is M, where the first preset quantity is equal to the third preset quantity. Alternatively, the first preset quantity is N, the second preset quantity is C, the third preset quantity is K, and the fourth preset quantity is M, where the first preset quantity N is less than the third preset quantity K. It should be noted that these examples are merely illustrative and do not limit this embodiment.

[0062] Step S102: Use the video action recognition model to extract features from the sample videos of the second preset number multiplied by the fourth preset number in each target subsample set, and generate a feature memory library for each target subsample set.

[0063] Step S103: Perform comparative learning using the feature memory of each target subsample set to obtain the loss value of the comparative learning for each target subsample set.

[0064] Step S104: Based on the loss values ​​of the comparative learning of each target sub-sample set, accumulate the loss values ​​of the comparative learning of the first preset number of target sub-sample sets as the overall loss function, and then use the overall loss function to update the parameters of the video action recognition model.

[0065] Step S105: Repeat the above steps iteratively until the iteration termination condition is met to obtain the target video action recognition model.

[0066] This embodiment of the application can train and iterate the video action recognition model based on target subsamples. Unlike traditional contrastive learning, this embodiment uses a delayed update model method to improve contrastive learning. That is, the loss value of the contrastive learning of a first preset number of target subsample sets is accumulated as the overall loss function. Then, the parameters of the video action recognition model are updated using the overall loss function. When the iteration termination condition is met, the target video action recognition model is obtained. This solves the problem that the traditional method will cause the model to not converge, and at the same time improves the accuracy of the target video action recognition model in video action recognition.

[0067] This application embodiment provides a possible implementation method. Step S101 above generates a first preset number of target subsample sets based on a pre-built support set for training the video action recognition model. Specifically, it may include the following steps A1 to A4:

[0068] Step A1: Initialize a temporary set based on a pre-built support set for training the video action recognition model, wherein the temporary set contains a second preset number of categories, and each category contains a third preset number of sample videos.

[0069] As described above, the supported set contains a second preset number of categories, and each category contains a third preset number of sample videos. The temporary set initialized here also contains a second preset number of categories, and each category contains a third preset number of sample videos. For example, the second preset number is C, and the third preset number is K. Of course, this example is only illustrative and does not limit this embodiment.

[0070] Step A2: Randomly select one sample video from each of the second preset number of categories in the temporary set to generate a first initial subsample set, wherein the first initial subsample set contains the second preset number of categories, and each category contains one sample video.

[0071] Step A3: Update the temporary set. Continue to randomly select a sample video from each of the second preset number of categories in the updated temporary set to generate a second initial sub-sample set. The second initial sub-sample set contains the second preset number of categories, and each category contains a sample video. Similarly, generate a first preset number of initial sub-sample sets, where each initial sub-sample set contains the second preset number of categories, and each category contains a sample video.

[0072] Step A4: Enhance the sample videos in each initial subsample set in the first preset number of initial subsample sets to generate the first preset number of target subsample sets.

[0073] This embodiment can generate a first preset number of initial sub-sample sets based on the support set, and then enhance the sample videos in each initial sub-sample set in the first preset number of initial sub-sample sets through the enhancement method to generate a first preset number of target sub-sample sets for subsequent feature extraction. Thus, subsequent video action recognition can be achieved using very little labeled data (i.e., the support set).

[0074] This application embodiment provides a possible implementation method in which step A4 above enhances the sample videos in each of the initial sub-sample sets in the first preset number of initial sub-sample sets to generate the first preset number of target sub-sample sets. Specifically, it can be achieved by the following step A41:

[0075] Step A41: Using the fourth preset number of enhancement methods, enhance the sample videos in each initial subsample set in the first preset number of initial subsample sets to generate the first preset number of target subsample sets.

[0076] The enhancement methods here can be image flipping, image cropping, etc., and the number of enhancement methods can be a fourth preset number. Each target subsample set contains a second preset number of categories, and each category contains a fourth preset number of sample videos. For example, the second preset number is C, and the fourth preset number is M. Of course, this example is only illustrative and does not limit this embodiment.

[0077] This application embodiment provides a possible implementation method. Step S103 above uses the feature memory of each target sub-sample set to perform comparative learning to obtain the loss value of the comparative learning for each target sub-sample set. Specifically, it may include the following steps B1 to B2:

[0078] Step B1: Use the feature memory of each target subsample set to perform contrastive learning to obtain the contrastive learning loss value of any category of sample video in each target subsample set.

[0079] Step B2: Based on the contrastive learning loss value of any category of sample video in each target subsample set, calculate the contrastive learning loss value of all categories of sample video in each target subsample set, and use it as the contrastive learning loss value of each target subsample set.

[0080] This embodiment uses the feature memory of each target subsample set for comparative learning to obtain the comparative learning loss value of any category of sample video in each target subsample set. Then, based on the comparative learning loss value of any category of sample video in each target subsample set, the comparative learning loss value of all categories of sample video in each target subsample set is calculated as the comparative learning loss value of each target subsample set, so as to update the model parameters efficiently and accurately.

[0081] This application provides a possible implementation method in which step B1 above uses the feature memory of each target subsample set to perform contrastive learning, and obtains the loss value of contrastive learning for any category of sample video in each target subsample set. Specifically, it may include the following step B11:

[0082] Step B11: Perform contrastive learning using the feature memory of each target subsample set. Based on the positive and negative sample pairs in any category of sample videos in each target subsample set, obtain the loss value of contrastive learning for any category of sample videos in each target subsample set; where positive sample pairs come from different enhanced versions of the same sample video, and negative sample pairs come from different sample videos.

[0083] This embodiment uses the feature memory of each target subsample set for comparative learning. Based on the positive and negative sample pairs in any category of sample videos in each target subsample set, the loss value of comparative learning for any category of sample videos in each target subsample set is obtained. By using positive and negative sample pairs, the loss value of comparative learning for any category of sample videos in each target subsample set can be accurately reflected, so as to update the model parameters efficiently and accurately in the future.

[0084] This application embodiment provides a possible implementation method. After iteratively executing the above steps in step S105 until the iteration termination condition is met and the target video action recognition model is obtained, the following step C1 may also be included:

[0085] Step C1: Obtain the video data to be identified, and use the target video action recognition model to identify the video data to obtain the action category of the video data to be identified.

[0086] This embodiment can acquire video data to be identified, use a target video action recognition model to identify the video data to be identified, and obtain the action category of the video data to be identified. The recognition accuracy and efficiency are high.

[0087] The above introduces Figure 1 The embodiments shown have various implementation methods for each step. The training method of the video action recognition model of this application will be further explained below through specific embodiments.

[0088] In a specific embodiment, a few-shot action recognition method based on a delayed-update contrastive learning model is used. Here, "few-shot" refers to a small number of samples. Through few-shot learning, video action recognition can be achieved using very little labeled data, and more useful feature information is extracted through contrastive learning. Unlike traditional contrastive learning, this embodiment uses a delayed-update model method to improve contrastive learning, thereby increasing the overall accuracy of the video action recognition model. This embodiment can be applied to fields such as smart education, finance, and artificial intelligence for video action recognition, accurately and efficiently identifying video actions.

[0089] First, let's define what the "few-shot video action recognition" task is. For example... Figure 2 As shown, this involves introducing few-shot learning techniques into video action recognition tasks. Therefore, the video action recognition dataset needs to be divided into a support set (Set). support and query set query The support set contains C categories, and each category contains K sample videos, as shown below:

[0090]

[0091] in, and Let represent the i-th sample video of category c and its corresponding label. Here, the label is the specific category, c is a positive integer between 1 and C, and i is a positive integer between 1 and K. Therefore, the entire support set contains a total of C×K data points.

[0092] Query Set query The data includes multiple video data to be identified. Subsequently, the target video action recognition model can be used to identify the video data to be identified and obtain the action category of the video data to be identified.

[0093] The training process for few-sample video action recognition is shown in Algorithm 1 below, which consists of four core steps.

[0094] Step (1) generates a minibatch, which is the initial sub-sample set.

[0095] To improve the model's representation extraction capability, this embodiment uses contrastive learning to extract more feature information; however, contrastive learning requires the use of minibatch. Sets are supported. support There are C categories, and each category has K samples.

[0096] As shown in Algorithm 1, the initial support set is Set. support To initialize a temporary Set tempThen, a sample video is randomly selected from each of the C categories in the temporary set to generate a minibatch, as shown below:

[0097]

[0098] Among them, Set mini This minibatch contains C categories, with one sample for each category. The temporary set is then updated as follows:

[0099] Set temp =Set temp -Set mini

[0100] Because Set support Each category has K samples, so K minibatches can be generated, which are K initial subsample sets.

[0101] Step (2) Generate the MemoryBank of minibatch.

[0102] Contrastive learning often stores features in a feature memory to build a large number of positive and negative sample pairs. Since there are C classes in the minibatch in step (1), and only one sample for each class, it is impossible to generate positive sample pairs. Therefore, it is necessary to use an augmentation method to augment each sample into M samples. Samples from the same sample are positive sample pairs, and samples from different samples are negative sample pairs. The augmentation process is as follows:

[0103]

[0104] in, and This represents the i-th sample video of category c and its corresponding label, where the label is the specific category, c is a positive integer between 1 and C, and i is a positive integer between 1 and M. Aug(·) represents the data augmentation function, with a total of M augmentation methods, including image flipping, image cropping, etc. Then, feature extraction is performed on the C×M sample videos to generate a MemoryBank.

[0105]

[0106] Among them, Z t Let t represent the t-th feature vector. There are a total of C×M feature vectors in the memory. Encoder is a video action recognition model.

[0107] Step (3) is comparative learning based on memory.

[0108] Contrastive learning is implemented using the feature memory bank MemoryBank from step (2), where the loss value for the contrastive learning of the i-th class sample in the j-th minibatch is calculated as follows:

[0109]

[0110] Among them, z i and Indicates a positive sample pair. It is z i Enhanced samples, z i and z s This represents a negative sample, i.e. a sample from a different category. j takes a positive integer between 1 and K, i takes a positive integer between 1 and C, sim(·,·) represents a similarity function, such as the cosine similarity function, and ρ is a penalty coefficient that can be set according to needs, for example, to 100.

[0111] Step (4): Delay updating the Encoder model parameters.

[0112] When using a loss memory, traditional contrastive learning updates the Encoder network parameters in each minibatch. However, in this embodiment, the number of labeled samples is very small, which can lead to model non-convergence using traditional methods. Therefore, a delayed update of model parameters is adopted to address this issue. This involves accumulating the loss value for each minibatch and updating the model parameters only after one round is completed. The overall loss function is as follows:

[0113]

[0114] in, This represents the loss value of the contrastive learning for the j-th minibatch. This represents the loss value for contrastive learning of the i-th class sample in the j-th minibatch.

[0115] Algorithm 1: Few-shot contrastive learning method based on delayed update model

[0116] Input: Supports Set collections support

[0117] For i←1…E / / E represents the number of training rounds.

[0118] (1)Set temp =Set support / / Initialize temporary collection

[0119] For j←1…K / / Supports sets where each class contains K samples.

[0120] (1)Setmini =Select(Set temp / / Generate minibatch

[0121] (1)Set temp =Set temp -Set mini / / Update temporary collection

[0122] (2)

[0123] (2)

[0124] (3) Calculate the loss value using contrastive learning.

[0125] EndFor

[0126] (4) Update the parameters of the Encoder model

[0127] EndFor

[0128] This embodiment proposes a few-shot video action recognition method based on a delayed update contrastive learning model. The technical effects are as follows: (1) Video action recognition can be achieved with very little labeled data. Few-shot learning solves the problem of few labeled samples and extracts more useful feature information through contrastive learning; (2) Unlike traditional contrastive learning, this embodiment uses a delayed update model to improve contrastive learning, thereby improving the overall accuracy of the video action recognition model.

[0129] In a specific business scenario, as mentioned earlier, banks need to use a real-time intelligent monitoring system to continuously monitor ATMs and bank entrances 24 / 7 to ensure the security of their branches. This monitoring system combines video and artificial intelligence technologies to identify abnormal events, enabling early warning and post-event review, thereby reliably ensuring the security management of bank branches. With a small sample size, the target video action recognition model trained in this embodiment can accurately recognize video actions and identify abnormal events. Specifically, the real-time intelligent monitoring system for bank ATMs and bank entrances detects abnormal behavior in the following steps 1) to 6):

[0130] 1) Data acquisition: Surveillance cameras collect video footage and convert it into digital signals.

[0131] For example, a surveillance camera captures multiple images of a user withdrawing money from an ATM, such as image 1, image 2, image 3, etc., resulting in images of image 1, image 2, image 3, etc.

[0132] 2) Data preprocessing: The collected data is transformed into data that can be used for subsequent processing.

[0133] For example, image preprocessing of multiple screens such as screen 1, screen 2, screen 3, etc., when a user withdraws money from an ATM, such as filtering, screenshotting, or compression.

[0134] 3) Target recognition: The target video action recognition model trained in this embodiment is used to perform target detection and target recognition on the preprocessed data, such as face recognition and behavior analysis.

[0135] For example, preprocessed images of multiple scenes (such as screen 1, screen 2, screen 3, etc.) of a user withdrawing money from an ATM can be input into a trained target video action recognition model to extract image features and then identify the behavior type of the image scene, such as behavior type a, b, c, etc.

[0136] 4) Anomaly detection: Determine if there is any abnormal behavior, such as smashing, wielding weapons, etc.

[0137] In this step, the behavior type of the image obtained in step 3) is used to determine whether there is any abnormal behavior, such as smashing or wielding weapons.

[0138] 5) Alarm handling: Once abnormal behavior is detected, the system will automatically trigger the alarm mechanism and take different alarm methods according to different scenarios and needs, such as sound alarm, image alarm, SMS alarm, etc.

[0139] 6) Data storage: The processed data is stored in the database for subsequent analysis and backtracking.

[0140] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.

[0141] Based on the training methods for video action recognition models provided in the above embodiments, and based on the same inventive concept, this application also provides a training device for video action recognition models.

[0142] Figure 3 This is a structural diagram of the training device for the video action recognition model provided in this application embodiment. For example... Figure 3 As shown, the training device for the video action recognition model may specifically include a first generation module 310, a second generation module 320, a contrastive learning module 330, a parameter update module 340, and an iteration module 350.

[0143] The first generation module 310 is configured to generate a first preset number of target subsample sets based on a pre-constructed support set for training a video action recognition model; wherein the support set contains a second preset number of categories, and each category contains a third preset number of sample videos; the first preset number is less than or equal to the third preset number, and each target subsample set in the first preset number of target subsample sets contains the second preset number of categories, and each category contains a fourth preset number of sample videos.

[0144] The second generation module 320 is used to extract features from the sample videos of each target sub-sample set by multiplying the second preset number by the fourth preset number using the video action recognition model, and generate a feature memory library for each target sub-sample set.

[0145] The contrastive learning module 330 is used to perform contrastive learning using the feature memory of each target subsample set to obtain the contrastive learning loss value of each target subsample set.

[0146] The parameter update module 340 is used to accumulate the loss values ​​of the comparative learning of the first preset number of target sub-sample sets as the overall loss function based on the loss values ​​of the comparative learning of each target sub-sample set, and then use the overall loss function to update the parameters of the video action recognition model.

[0147] The iteration module 350 is used to iteratively execute the above steps until the iteration termination condition is met, thereby obtaining the target video action recognition model.

[0148] This application embodiment provides a possible implementation, wherein the first generation module 310 is further configured to:

[0149] A temporary set is initialized based on a pre-built support set for training the video action recognition model, wherein the temporary set contains the second preset number of categories, and each category contains the third preset number of sample videos;

[0150] Randomly select one sample video from each of the second preset number of categories in the temporary set to generate a first initial subsample set, wherein the first initial subsample set contains the second preset number of categories, and each category contains one sample video;

[0151] Update the temporary set, and randomly select a sample video from each of the second preset number of categories in the updated temporary set to generate a second initial sub-sample set, wherein the second initial sub-sample set contains the second preset number of categories, and each category contains a sample video;

[0152] Similarly, the first preset number of initial sub-sample sets are generated, wherein each initial sub-sample set contains the second preset number of categories, and each category contains one sample video;

[0153] Enhancement processing is performed on the sample videos in each of the initial subsample sets in the first preset number of initial subsample sets to generate the first preset number of target subsample sets.

[0154] This application embodiment provides a possible implementation, wherein the first generation module 310 is further configured to:

[0155] Using the fourth preset number of enhancement methods, the sample videos in each initial subsample set in the first preset number of initial subsample sets are enhanced to generate the first preset number of target subsample sets.

[0156] This application embodiment provides a possible implementation, wherein the comparison learning module 330 is further configured to:

[0157] Comparative learning is performed using the feature memory of each target subsample set to obtain the loss value of comparative learning for any category of sample videos in each target subsample set;

[0158] Based on the contrastive learning loss value of any category sample video in each target subsample set, calculate the contrastive learning loss value of all category sample videos in each target subsample set, and use it as the contrastive learning loss value of each target subsample set.

[0159] This application embodiment provides a possible implementation, wherein the comparison learning module 330 is further configured to:

[0160] Contrastive learning is performed using the feature memory of each target subsample set. Based on the positive and negative sample pairs in any category of sample videos in each target subsample set, the loss value of contrastive learning for any category of sample videos in each target subsample set is obtained. Among them, the positive sample pairs come from different enhanced versions of the same sample video, and the negative sample pairs come from different sample videos.

[0161] This application provides one possible implementation method, such as... Figure 4 As shown above, Figure 3 The device on display also includes an identification module 410, used for:

[0162] After obtaining the target video action recognition model, the video data to be recognized is acquired, and the target video action recognition model is used to recognize the video data to obtain the action category of the video data to be recognized.

[0163] Based on the same inventive concept, this application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the training method of the video action recognition model of any of the above embodiments.

[0164] In an exemplary embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The illustrated electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one type, and the structure of this electronic device 500 does not constitute a limitation on the embodiments of this application.

[0165] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0166] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0167] The memory 503 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0168] The memory 503 stores computer program code that executes the scheme of this application, and its execution is controlled by the processor 501. The processor 501 executes the computer program code stored in the memory 503 to implement the content shown in the foregoing method embodiments.

[0169] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0170] Based on the same inventive concept, this application also provides a storage medium storing a computer program, wherein the computer program is configured to execute the training method of the video action recognition model of any of the above embodiments at runtime.

[0171] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0172] Those skilled in the art will understand that the technical solution of this application, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0173] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.

[0174] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.

Claims

1. A training method for a video action recognition model, characterized in that, include: Based on the pre-built support set used to train the video action recognition model, a first preset number of target subsample sets are generated; The support set contains a second preset number of categories, and each category contains a third preset number of sample videos; the first preset number is less than or equal to the third preset number, and each target sub-sample set in the first preset number contains the second preset number of categories, and each category contains a fourth preset number of sample videos; The video action recognition model is used to extract features from the sample videos of each target sub-sample set by multiplying the second preset number by the fourth preset number, thereby generating a feature memory library for each target sub-sample set. Comparative learning is performed using the feature memory of each target subsample set to obtain the loss value of the comparative learning for each target subsample set; Based on the loss value of the comparative learning of each target sub-sample set, the cumulative loss value of the comparative learning of the first preset number of target sub-sample sets is used as the overall loss function, and then the parameters of the video action recognition model are updated using the overall loss function. The above steps are executed iteratively until the iteration termination condition is met, resulting in the target video action recognition model. The step of using the feature memory of each target sub-sample set to perform comparative learning to obtain the loss value of the comparative learning for each target sub-sample set includes: Comparative learning is performed using the feature memory of each target subsample set to obtain the loss value of comparative learning for any category of sample videos in each target subsample set; Based on the contrastive learning loss value of any category sample video in each target subsample set, calculate the contrastive learning loss value of all category sample videos in each target subsample set, and use it as the contrastive learning loss value of each target subsample set.

2. The method according to claim 1, characterized in that, The step of generating a first preset number of target subsample sets based on a pre-constructed support set used for training video action recognition models includes: A temporary set is initialized based on a pre-built support set for training the video action recognition model, wherein the temporary set contains the second preset number of categories, and each category contains the third preset number of sample videos; Randomly select one sample video from each of the second preset number of categories in the temporary set to generate a first initial subsample set, wherein the first initial subsample set contains the second preset number of categories, and each category contains one sample video; Update the temporary set, and randomly select a sample video from each of the second preset number of categories in the updated temporary set to generate a second initial sub-sample set, wherein the second initial sub-sample set contains the second preset number of categories, and each category contains a sample video; Similarly, the first preset number of initial sub-sample sets are generated, wherein each initial sub-sample set contains the second preset number of categories, and each category contains one sample video; Enhancement processing is performed on the sample videos in each of the initial subsample sets in the first preset number of initial subsample sets to generate the first preset number of target subsample sets.

3. The method according to claim 2, characterized in that, Enhancement processing is performed on the sample videos in each of the initial subsample sets within the first preset number of initial subsample sets to generate the first preset number of target subsample sets, including: Using the fourth preset number of enhancement methods, the sample videos in each initial subsample set in the first preset number of initial subsample sets are enhanced to generate the first preset number of target subsample sets.

4. The method according to claim 1, characterized in that, The step of performing contrastive learning using the feature memory of each target sub-sample set to obtain the contrastive learning loss value for any category of sample videos in each target sub-sample set includes: Contrastive learning is performed using the feature memory of each target subsample set. Based on the positive and negative sample pairs in any category of sample videos in each target subsample set, the loss value of contrastive learning for any category of sample videos in each target subsample set is obtained. Among them, the positive sample pairs come from different enhanced versions of the same sample video, and the negative sample pairs come from different sample videos.

5. The method according to any one of claims 1 to 3, characterized in that, After obtaining the target video action recognition model, the method further includes: The video data to be identified is acquired, and the target video action recognition model is used to identify the video data to obtain the action category of the video data to be identified.

6. A training device for a video action recognition model, characterized in that, include: The first generation module is used to generate a first preset number of target subsample sets based on a pre-built support set used to train the video action recognition model. The support set contains a second preset number of categories, and each category contains a third preset number of sample videos; the first preset number is less than or equal to the third preset number, and each target sub-sample set in the first preset number contains the second preset number of categories, and each category contains a fourth preset number of sample videos; The second generation module is used to extract features from the sample videos of each target sub-sample set by multiplying the second preset number by the fourth preset number using the video action recognition model, and generate a feature memory library for each target sub-sample set. The contrastive learning module is used to perform contrastive learning using the feature memory of each target subsample set to obtain the contrastive learning loss value for each target subsample set. The parameter update module is used to accumulate the loss values ​​of the comparative learning of the first preset number of target sub-sample sets as the overall loss function based on the loss values ​​of the comparative learning of each target sub-sample set, and then use the overall loss function to update the parameters of the video action recognition model. The iteration module is used to iteratively execute the above steps until the iteration termination condition is met, thereby obtaining the target video action recognition model. The contrastive learning module is further used for: Comparative learning is performed using the feature memory of each target subsample set to obtain the loss value of comparative learning for any category of sample videos in each target subsample set; Based on the contrastive learning loss value of any category sample video in each target subsample set, calculate the contrastive learning loss value of all category sample videos in each target subsample set, and use it as the contrastive learning loss value of each target subsample set.

7. The apparatus according to claim 6, characterized in that, The first generation module is also used for: A temporary set is initialized based on a pre-built support set for training the video action recognition model, wherein the temporary set contains the second preset number of categories, and each category contains the third preset number of sample videos; Randomly select one sample video from each of the second preset number of categories in the temporary set to generate a first initial subsample set, wherein the first initial subsample set contains the second preset number of categories, and each category contains one sample video; Update the temporary set, and randomly select a sample video from each of the second preset number of categories in the updated temporary set to generate a second initial sub-sample set, wherein the second initial sub-sample set contains the second preset number of categories, and each category contains a sample video; Similarly, the first preset number of initial sub-sample sets are generated, wherein each initial sub-sample set contains the second preset number of categories, and each category contains one sample video; Enhancement processing is performed on the sample videos in each of the initial subsample sets in the first preset number of initial subsample sets to generate the first preset number of target subsample sets.

8. An electronic device, characterized in that, The device includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform a training method for the video action recognition model according to any one of claims 1 to 5.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the training method of the video action recognition model according to any one of claims 1 to 5 at runtime.

Images