A multi-instance small sample based action localization method and system

Abstract

The application discloses a kind of action positioning method and system based on multiple-instance small sample, it is related to action positioning field, the method includes: obtaining video to be measured and standard action video;The video to be measured includes multiple instances to be measured;The divided video segment and standard action video are input into C3D model, obtain the C3D feature map corresponding to video segment and standard action sample feature;C3D feature map is input into center judging model, and the center video segment of each instance to be measured is filtered out, and it is expanded to different frames to both sides, generate multiple candidate regions;Adaptive pooling layer is used to align C3D feature map and candidate region, obtain alignment feature map;Alignment feature map and standard action sample feature are input into action prediction model, obtain action prediction result, and are optimized using non-maximum suppression method, complete action positioning, the application reduces the input cost and time period of model training, improves action positioning efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of action localization, and particularly relates to an action localization method and system based on multiple-instance small samples. BACKGROUND

[0002] Action localization technology based on videos can segment action clips from a large number of videos, and provide a key basis for subsequent evaluation of each action. Therefore, as an indispensable preliminary step for action evaluation in actual scenarios, action localization plays a crucial role.

[0003] The related mainstream action localization method mainly relies on a full-supervised mode, and uses a large-scale dataset and a complex model to improve the prediction accuracy. However, the premise of this method is that a large amount of labeled data must be used to train the model. In actual applications, it is not only time-consuming and laborious to obtain and label such a large amount of data, but also has high investment cost and long time period, which greatly limits its wide application. At the same time, there is currently a lack of models that can realize action localization under small sample conditions. SUMMARY

[0004] The purpose of the present application is to provide an action localization method and system based on multiple-instance small samples, which can complete the training of the model through a small amount of sample data, reduce the investment cost and time period of model training, and improve the action localization efficiency and accuracy.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides an action localization method based on multiple-instance small samples, comprising:

[0007] obtaining a to-be-tested video and a standard action video; the to-be-tested video comprises multiple to-be-tested instances; one instance is one action;

[0008] dividing the to-be-tested video to obtain multiple video clips;

[0009] inputting each video clip into a C3D model to obtain a C3D feature map corresponding to each video clip;

[0010] inputting the standard action video into the C3D model to obtain a standard action sample feature;

[0011] inputting the C3D feature map into a center judgment model to screen out a center video clip of each to-be-tested instance; the center video clip of each to-be-tested instance is a video clip corresponding to a center time of each to-be-tested instance; the center judgment model is obtained by training a regression model using a sample C3D feature map;

[0012] Using the center point of the central video segment of each test instance as the center, expand outwards to both sides of the central video segment with different frames to generate multiple candidate regions for each test instance;

[0013] An adaptive pooling layer is used to align the C3D feature map and multiple candidate regions to obtain an aligned feature map for each candidate region.

[0014] The alignment feature map of each candidate region and the standard action sample features are input into the action prediction model to obtain the action prediction result for each candidate region. The action prediction model is obtained by training the Siamese network with a sample training set. The sample training set includes: sample alignment feature map, standard action sample features, and corresponding sample prediction results.

[0015] The non-maximum suppression method is used to optimize the action prediction results of each candidate region, so as to obtain the action type, start time and end time of each test instance and complete the action localization.

[0016] Secondly, this application provides an action localization system based on multiple instance small samples, including:

[0017] The acquisition module is used to acquire the video to be tested and the standard action video; the video to be tested includes multiple test instances; each instance is one action;

[0018] The segmentation module is used to divide the video to be tested into multiple video segments;

[0019] The C3D module is used to input each video clip into the C3D model to obtain the C3D feature map corresponding to each video clip; the C3D module is also used to input standard action videos into the C3D model to obtain standard action sample features;

[0020] The center determination module is used to input the C3D feature map into the center determination model to filter out the center video segment of each test instance; each center video segment of the test instance is the video segment corresponding to the center moment of each test instance; the center determination model is obtained by training a regression model with sample C3D feature maps;

[0021] The candidate region generation module is used to generate multiple candidate regions for each test instance by expanding outwards from the center point of the central video segment of each test instance to both sides of the central video segment at different frame intervals.

[0022] An alignment module is used to align the C3D feature map and multiple candidate regions using an adaptive pooling layer to obtain an aligned feature map for each candidate region.

[0023] The prediction module is used to input the alignment feature map of each candidate region and the standard action sample features into the action prediction model to obtain the action prediction result for each candidate region. The action prediction model is obtained by training the Siamese network with a sample training set. The sample training set includes: sample alignment feature map, standard action sample features and corresponding sample prediction results.

[0024] The action determination module is used to optimize the action prediction results of each candidate region using the non-maximum suppression method, so as to obtain the action type, start time and end time of each test instance and complete the action localization.

[0025] According to the specific embodiments provided in this application, this application has the following technical effects:

[0026] This application provides a method and system for action localization based on multiple instances and few samples. Actions are localized using a trained Siamese network (i.e., an action prediction model). Since the Siamese network can map input sample pairs to a feature space through a shared neural network and determine whether sample pairs belong to the same category by calculating the distance between them, this metric-based learning approach allows the model to learn the relationships between samples from a small number of samples, without requiring a large amount of labeled data to learn the specific features of each category. Therefore, this application only requires a small amount of data to train the Siamese model, reducing training costs and time, and improving the efficiency of action localization. Simultaneously, multiple candidate regions generated from the central video segment preserve the integrity of the action within the video frame, ensuring the accuracy of the action prediction results obtained by the action prediction model, thereby improving the accuracy of action localization. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating an action localization method based on multiple instance small samples provided in an embodiment of this application;

[0029] Figure 2 A detailed flowchart illustrating an action localization method based on multiple instances and small samples, provided as an embodiment of this application;

[0030] Figure 3 This is a flowchart illustrating the process of motion localization using a twin network.

[0031] Figure 4 A schematic diagram of the candidate regions. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The definitions are as follows:

[0034] Motion localization refers to predicting the start time, end time, and type of a specific action within a long video (or clip).

[0035] Few-sample action localization refers to achieving the above function by training a model with only a few samples, rather than relying on a large number of models.

[0036] The advantage of small sample sizes is that they don't require a large number of samples or manual annotations, but training the model is more difficult. This application uses a Siamese network for small sample training, thus solving the problem of high training difficulty. Furthermore, this application can locate actions from multiple instances in a video segment without determining the video type or the temporal location of actions within the video. In addition, this application can generate multiple candidate regions by identifying a central video segment and using its center point as an anchor point, thus preserving both the start and end times of actions. Then, the model trained with small samples generates action prediction results, and non-maximum suppression is used to optimize the start and end times of actions, ultimately ensuring the accuracy of action localization. Therefore, this application not only reduces the investment cost and time required for model training but also improves the efficiency and accuracy of action localization.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] In one exemplary embodiment, such as Figures 1 to 3 As shown, a method for action localization based on multiple instances and small samples is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is described using a server as an example, and includes the following steps 1 to 8. Wherein:

[0039] Step 1: Obtain the video to be tested and the standard action video; the video to be tested includes multiple test instances; each instance is one action.

[0040] Step 2: Divide the video to be tested into multiple video segments.

[0041] In one exemplary embodiment, the entire video is divided into 16-frame segments to obtain multiple video segments.

[0042] Step 3: Input each video segment into the C3D model to obtain the C3D feature map corresponding to each video segment.

[0043] Step 4: Input the standard action video into the C3D model to obtain the standard action sample features.

[0044] Step 5: Input the C3D feature map into the center judgment model to filter out the center video segment of each test instance; the center video segment of each test instance is the video segment corresponding to the center time of each test instance; the center judgment model is obtained by training the regression model with the sample C3D feature map.

[0045] Specifically, step 5 includes:

[0046] Step 51: Input the C3D feature map into the center judgment model to obtain the probability value corresponding to the C3D feature map.

[0047] Step 52: Determine whether the probability value is greater than the probability threshold. The probability threshold is 0.5 to 1.

[0048] Step 53: If yes, then select the video segment corresponding to the C3D feature map as the center video segment.

[0049] Step 54: If not, delete the video segment corresponding to the C3D feature map.

[0050] The training process of the center judgment model specifically includes:

[0051] The sample C3D feature map is input into the regression model to obtain the detection probability value.

[0052] Based on the detection probability value and the sample probability value corresponding to the sample C3D feature map, a loss function is constructed. The parameters of the regression model are then iteratively optimized using this loss function until the loss function reaches its minimum value or the number of iterations reaches its maximum value. The iteration optimization then stops, resulting in the center judgment model. The loss function is the cross-entropy loss function.

[0053] Step 6: Using the center point of the central video segment of each test instance as the center, expand outwards to both sides of the central video segment with different frames to generate multiple candidate regions for each test instance.

[0054] In one exemplary embodiment, the candidate regions are expanded to both sides according to a plurality of pre-arranged templates of fixed size, such as... Figure 4 As shown, with the center of a 16-frame video clip as the center point, multiple fixed-size lengths are selected as radii, and the regions are expanded forward and backward according to the frame number to generate candidate regions.

[0055] Step 7: Use an adaptive pooling layer to align the C3D feature map and multiple candidate regions to obtain the aligned feature map of each candidate region.

[0056] Step 8: Input the alignment feature map and standard action sample features of each candidate region into the action prediction model to obtain the action prediction result for each candidate region. The action prediction model is trained on the Siamese network using a sample training set. The sample training set includes: sample alignment feature maps, standard action sample features, and corresponding sample prediction results. The number of sample alignment feature maps, standard action sample features, and corresponding sample prediction results in the sample training set is less than 5 pairs.

[0057] Specifically, such as Figure 2 As shown, the C3D feature map extracted from the C3D model and the candidate regions are used to extract features to obtain the alignment feature map of each candidate region. The alignment feature map of each candidate region and the standard action sample features are simultaneously input into a Siamese network for comparative learning and prediction, resulting in action similarity, candidate region center point offset, and candidate region length offset. The standard action sample is a small sample of a certain type of action as referred to in this application; that is, the similarity between the alignment feature map of the candidate region and this type of action sample is used to evaluate whether the candidate region belongs to this type of action. The predicted candidate region center point offset and candidate region length offset provide a specific prediction of the actual action region (start time, end time).

[0058] like Figure 3As shown, a Siamese network consists of two identical sub-networks, each containing convolutional layers and fully connected layers, with parameters shared between the two sub-networks. During contrastive learning and prediction, each sub-network processes the input data and extracts features that describe key information about the input data. Because the sub-networks share parameters, their processing of the input data is consistent, ensuring the comparability of the extracted feature representations. The two sub-networks share weights, meaning that parameter updates are synchronized during training. This parameter-sharing mechanism reduces the number of parameters in the model, lowers the risk of overfitting, and improves the training efficiency of the model. The Siamese network takes as input the alignment feature map of each candidate region and the standard action sample features. The distance (e.g., Euclidean distance) between the alignment feature map of each candidate region and the standard action sample features is calculated through two sub-networks in the Siamese network. The output of the Siamese network is finally passed through a fully connected layer and a Sigmoid activation function for similarity prediction, thereby obtaining the action prediction result and determining whether the sample pairs belong to the same category. This learning method, which calculates the distance between the alignment feature map of each candidate region and the standard action sample features, does not require a large amount of labeled data. This allows the Siamese network to be trained with only a small amount of sample data, thereby reducing training costs and training time and improving the efficiency of action localization.

[0059] Specifically, the training process of the action prediction model includes:

[0060] The sample alignment feature map and the standard action sample features are input into the Siamese network to obtain the prediction result.

[0061] Based on the prediction results, the sample alignment feature map, and the sample action prediction results corresponding to the standard action sample features, a loss function is constructed. The parameters of the Siamese network are iteratively optimized according to the loss function until the loss function reaches its minimum value or the iteration optimization rounds reach their maximum value. Then, the iterative optimization is stopped, and the action prediction model is obtained.

[0062] Step 9: Optimize the action prediction results of each candidate region using the non-maximum suppression method to obtain the action type, start time and end time of each test instance, and complete the action localization.

[0063] Specifically, the action prediction results include: action similarity, candidate region center point offset, and candidate region length offset; the action similarity is the similarity between the action corresponding to each aligned feature map and the action corresponding to the standard action sample feature. Step 9 specifically includes:

[0064] Step 91: Determine the action type of the action corresponding to the candidate region based on the action similarity and similarity threshold.

[0065] Specifically, step 91 includes: determining whether the action similarity is greater than a similarity threshold; if yes, then determining that the action type of the action corresponding to the candidate region is the action type corresponding to the standard action sample feature, and retaining it; if no, then determining that the action type of the action corresponding to the candidate region is inconsistent with the action type corresponding to the standard action sample feature, and deleting it.

[0066] Step 92: Determine the start and end times of the action corresponding to the candidate region based on the candidate region center point offset and the candidate region length offset. Specifically, the start time of the action corresponding to the candidate region is equal to the candidate region center point offset minus half of the candidate region length offset; the end time of the action corresponding to the candidate region is equal to the candidate region center point offset plus half of the candidate region length offset.

[0067] Step 93: Determine whether the overlap rate between the start and end times of the actions corresponding to the candidate region reaches the overlap threshold. If yes, proceed to step 94; otherwise, proceed to step 95.

[0068] Step 94: Determine that the test instance has multiple candidate regions corresponding to actions, and take the action corresponding to the candidate region with the highest action similarity as the action type, start time and end time of the corresponding test instance.

[0069] Step 95: Determine that the test instance has only one candidate region corresponding to the action, and use the action corresponding to the candidate region as the action type, start time and end time of the corresponding test instance to complete the action localization.

[0070] Specifically, since each candidate region predicts an action, a video has many candidate regions, and multiple actions can be located. However, the predicted actions may have overlapping regions. Considering that this application only addresses the case where actions do not overlap temporally, a non-maximum suppression (NMS) method is used to retain the region containing the highly reliable action (i.e., the action with the highest similarity) and delete other action predictions that have a high temporal overlap (e.g., 30%) with the region containing the highly reliable action.

[0071] The beneficial effects of the action localization method based on multiple instances and small samples proposed in this application are mainly reflected in:

[0072] (1) This application uses an action prediction model (trained Siamese network) to complete action localization. Since the Siamese network determines whether the sample pairs are of the same category by calculating the distance between sample pairs (such as Euclidean distance), this learning method of calculating the distance between sample pairs does not require a large amount of labeled data. This application only needs a small amount of sample data to complete the training of the Siamese network, thereby reducing the training cost and training time and improving the efficiency of action localization.

[0073] (2) This application generates multiple candidate regions with different frame lengths in the central video segment, which can preserve the integrity of each test instance in the video frame segment, thereby ensuring the accuracy of the action prediction model in obtaining the action prediction result and improving the accuracy of action localization.

[0074] Based on the same inventive concept, this application also provides a motion localization system based on multiple instances and a small sample size. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the motion localization system based on multiple instances and a small sample size provided below can be found in the limitations of the motion localization system method based on multiple instances and a small sample size described above, and will not be repeated here.

[0075] In one exemplary embodiment, a multi-instance small sample-based action localization system is provided, comprising:

[0076] The acquisition module is used to acquire the video to be tested and the standard action video; the video to be tested includes multiple test instances; each instance is an action.

[0077] The segmentation module is used to divide the video to be tested into multiple video segments.

[0078] The C3D module is used to input each video clip into the C3D model to obtain the C3D feature map corresponding to each video clip; the C3D module is also used to input standard action videos into the C3D model to obtain standard action sample features.

[0079] The center judgment module is used to input the C3D feature map into the center judgment model to filter out the center video segment of each test instance; the center video segment of each test instance is the video segment corresponding to the center time of each test instance; the center judgment model is obtained by training a regression model with sample C3D feature maps.

[0080] The candidate region generation module is used to generate multiple candidate regions for each test instance by expanding outwards from the center point of the central video segment of the central video segment to both sides of the central video segment with different frames.

[0081] The alignment module is used to align the C3D feature map and multiple candidate regions using an adaptive pooling layer to obtain an aligned feature map for each candidate region.

[0082] The prediction module is used to input the alignment feature map of each candidate region and the standard action sample features into the action prediction model to obtain the action prediction result of each candidate region. The action prediction model is obtained by training the Siamese network with a sample training set. The sample training set includes: sample alignment feature map, standard action sample features and corresponding sample prediction results.

[0083] The action determination module is used to optimize the action prediction results of each test instance using the non-maximum suppression method, so as to obtain the action type, start time and end time of each test instance and complete the action localization.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for action localization based on multiple instances and small samples, characterized in that, The action localization method based on multiple instance small samples includes: Acquire the video to be tested and the standard action video; the video to be tested includes multiple test instances; each instance represents one action. The video to be tested is divided into multiple video segments; Each video segment is input into the C3D model to obtain the C3D feature map corresponding to each video segment; The standard action video is input into the C3D model to obtain the standard action sample features; The C3D feature map is input into the center determination model to filter out the center video segment of each test instance; the center video segment of each test instance is the video segment corresponding to the center time of each test instance; the center determination model is obtained by training a regression model with the sample C3D feature map; Using the center point of the central video segment of each test instance as the center, expand outwards to both sides of the central video segment with different frames to generate multiple candidate regions for each test instance; An adaptive pooling layer is used to align the C3D feature map and multiple candidate regions to obtain an aligned feature map for each candidate region. The alignment feature map of each candidate region and the standard action sample features are input into the action prediction model to obtain the action prediction result for each candidate region. The action prediction model is obtained by training the Siamese network with a sample training set. The sample training set includes: sample alignment feature map, standard action sample features, and corresponding sample prediction results. The non-maximum suppression method is used to optimize the action prediction results of each candidate region, so as to obtain the action type, start time and end time of each test instance and complete the action localization.

2. The action localization method based on multiple instance small samples according to claim 1, characterized in that, The C3D feature map is input into the center determination model to filter out the center video segment of each test instance, specifically including: The C3D feature map is input into the center judgment model to obtain the probability value corresponding to the C3D feature map; Determine whether the probability value is greater than the probability threshold; If so, the video segment corresponding to the C3D feature map is selected as the central video segment; If not, delete the video segment corresponding to the C3D feature map.

3. The action localization method based on multiple instance small samples according to claim 2, characterized in that, The training process of the center judgment model specifically includes: The sample C3D feature map is input into the regression model to obtain the detection probability value; Based on the detection probability value and the sample probability value corresponding to the sample C3D feature map, a loss function is constructed, and the parameters of the regression model are iteratively optimized according to the loss function until the loss function reaches the minimum value or the iteration optimization round reaches the maximum value, then the iterative optimization stops, and the center judgment model is obtained.

4. The action localization method based on multiple instance small samples according to claim 1, characterized in that, The action prediction results include: action similarity, candidate region center point offset, and candidate region length offset; the action similarity is the similarity between the action corresponding to each aligned feature map and the action corresponding to the standard action sample feature. The non-maximum suppression method is used to optimize the action prediction results for each candidate region, obtaining the action type, start time, and end time for each instance to be tested, thus completing action localization. Specifically, this includes: Based on the action similarity and similarity threshold, the action type corresponding to the candidate region is determined; Based on the offset of the center point of the candidate region and the offset of the length of the candidate region, the start time and end time of the action corresponding to the candidate region are determined. Determine whether the overlap rate between the start and end times of the action corresponding to the candidate region reaches the overlap threshold; If so, it is determined that the test instance has multiple candidate regions corresponding to actions, and the action corresponding to the candidate region with the highest action similarity is taken as the action type, start time and end time of the corresponding test instance. If not, it is determined that the test instance has only one candidate region corresponding to the action, and the action corresponding to the candidate region is used as the action type, start time and end time of the corresponding test instance to complete the action localization.

5. The action localization method based on multiple instance small samples according to claim 4, characterized in that, Based on the action similarity and similarity threshold, the action type corresponding to the candidate region is determined, specifically including: Determine whether the similarity of the actions is greater than a similarity threshold; If so, then determine the action type of the action corresponding to the candidate region as the action type corresponding to the feature of the standard action sample, and retain it; If not, then determine that the action type corresponding to the candidate region is inconsistent with the action type corresponding to the feature of the standard action sample, and delete it.

6. The action localization method based on multiple instance small samples according to claim 4, characterized in that, The start time of the action corresponding to the candidate region is equal to the offset of the center point of the candidate region minus half the length offset of the candidate region. The end time of the action corresponding to the candidate region is equal to the offset of the center point of the candidate region plus half the length offset of the candidate region.

7. The action localization method based on multiple instance small samples according to claim 1, characterized in that, The training process of the action prediction model specifically includes: The sample alignment feature map and the standard action sample features are input into the Siamese network to obtain the prediction result; Based on the prediction results, the sample alignment feature map, and the sample action prediction results corresponding to the standard action sample features, a loss function is constructed. The parameters of the Siamese network are iteratively optimized according to the loss function until the loss function reaches its minimum value or the iteration optimization rounds reach their maximum value. Then, the iterative optimization is stopped, and the action prediction model is obtained.

8. A motion localization system based on multiple instances and small samples, characterized in that, The action localization system based on multiple instance small samples includes: The acquisition module is used to acquire the video to be tested and the standard action video; the video to be tested includes multiple test instances; each instance is one action; The segmentation module is used to divide the video to be tested into multiple video segments; The C3D module is used to input each video clip into the C3D model to obtain the C3D feature map corresponding to each video clip; the C3D module is also used to input standard action videos into the C3D model to obtain standard action sample features; The center determination module is used to input the C3D feature map into the center determination model to filter out the center video segment of each test instance; each center video segment of the test instance is the video segment corresponding to the center moment of each test instance; the center determination model is obtained by training a regression model with sample C3D feature maps; The candidate region generation module is used to generate multiple candidate regions for each test instance by expanding outwards from the center point of the central video segment of each test instance to both sides of the central video segment at different frame intervals. An alignment module is used to align the C3D feature map and multiple candidate regions using an adaptive pooling layer to obtain an aligned feature map for each candidate region. The prediction module is used to input the alignment feature map of each candidate region and the standard action sample features into the action prediction model to obtain the action prediction result for each candidate region. The action prediction model is obtained by training the Siamese network with a sample training set. The sample training set includes: sample alignment feature map, standard action sample features and corresponding sample prediction results. The action determination module is used to optimize the action prediction results of each candidate region using the non-maximum suppression method, so as to obtain the action type, start time and end time of each test instance and complete the action localization.

Images