Motion detection method, training method of motion detection model and related device

Abstract

The present application discloses an action detection method and a training method of an action detection model and related devices, wherein the action detection method includes: extracting features from an image sequence to be tested to obtain features to be processed; matching the template features of the template image sequence and the features to be processed to obtain matching features; and then predicting based on the matching features to determine the action detection result of the image sequence to be tested, the action detection result includes whether the image sequence to be tested has a template action corresponding to the template image sequence. The above scheme can improve the applicability of action detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a motion detection method, a motion detection model training method and related devices. Background Art

[0002] In real life, identifying or finding actions is extremely important in many scenarios. For example, in image search scenarios, action detection can help relevant personnel quickly narrow the search scope; or in video search scenarios, action detection of frame images in the video can help quickly locate the corresponding video content, and so on.

[0003] At present, action detection generally uses deep learning models to directly predict the image to be tested. The predicted action category can only be the action category of the sample data during the training of the deep learning model. If a new action category needs to be added, the deep learning model must be retrained using the sample data of the new action category, which reduces the efficiency of action detection. In view of this, how to improve the applicability of action detection has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a motion detection method and a training method of a motion detection model and related devices, which can improve the applicability of motion detection.

[0005] In order to solve the above-mentioned technical problems, the first aspect of the present application provides an action detection method, including: extracting features from an image sequence to be tested to obtain features to be processed, and matching the template features of the template image sequence and the features to be processed to obtain matching features, and then using the matching features to perform prediction to determine the action detection result of the image sequence to be tested, the action detection result including whether there is a template action corresponding to the template image sequence in the image sequence to be tested.

[0006] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a training method for an action detection model, including: using a feature extraction network of the action detection model to extract features from a sample image sequence to be tested and a sample template image sequence respectively, to obtain sample features to be tested of the sample image sequence to be tested and sample template features of the sample template image sequence; and using a matching module to match the sample template features and the sample features to be tested to obtain sample matching features; and then using a prediction module to perform prediction based on the sample matching features to determine a sample action detection result of the sample image sequence to be tested; finally, based on the sample action detection result, adjusting the network parameters of the action detection model.

[0007] In order to solve the above-mentioned technical problems, the third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the action detection method in the above-mentioned first aspect or the training method of the action detection model in the above-mentioned second aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the action detection method in the above first aspect or the training method of the action detection model in the above second aspect.

[0009] The above scheme extracts features from the image sequence to be tested to obtain features to be processed, and matches the template features of the template image sequence with the features to be processed to obtain matching features. On this basis, the matching features are used for prediction to determine the action detection result of the image sequence to be tested, and the action detection result includes whether the image sequence to be tested has a template action corresponding to the template image sequence. On the one hand, by predicting the matching features obtained based on the features to be processed and the template features, the detection of whether the image sequence to be tested has a template action in the template image sequence is realized. Compared with directly predicting using the image features of the image sequence to be tested, the accuracy of action detection can be improved. Moreover, since the prediction network can only predict known template actions by directly using the image features of the image sequence to be tested, the scheme determines whether the image sequence to be tested has a template action in the template image sequence by matching features. Even if the prediction network does not know the template action in the template image sequence before prediction, the detection of its template action can be realized, that is, the detection of the template action unknown to the prediction network before prediction is realized, thereby improving the applicability of action detection. On the other hand, using matching features for prediction can maximize the simplicity and efficiency of action detection.

[0010] In addition, the action detection process of the present application scheme can be executed using an action detection model. Since the present application scheme determines whether the image sequence to be tested contains a template action in the template image sequence by matching features, even if the template action in the template image sequence has not appeared in the training process of the action detection model, the current action detection model can be directly used to detect the unknown template action without the need for retraining, thereby improving the applicability of action detection and improving the efficiency of action detection.

[0011] Furthermore, when the action detection process of the present application scheme is executed using an action detection model, during the training of the action detection model, two identical feature extraction networks are used to extract features from the sample test image sequence and the sample template image sequence, and the extracted features are matched to predict the sample detection results. Finally, the model is adjusted based on the sample detection results, which can make the trained action detection model more generalized. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of an embodiment of the motion detection method of the present application;

[0013] Figure 2 is a schematic diagram of an embodiment of obtaining matching features by matching template features with features to be processed;

[0014] Figure 3 is a schematic diagram of an embodiment of an action detection model;

[0015] Figure 4 It is a flowchart of an embodiment of a training method for an action detection model of the present application;

[0016] Figure 5 is a schematic diagram of an embodiment of testing an action detection model;

[0017] Figure 6 It is a schematic diagram of the framework of an embodiment of the motion detection device of the present application;

[0018] Figure 7 It is a schematic diagram of the framework of an embodiment of a training device for an action detection model of the present application;

[0019] Figure 8 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0020] Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0024] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the motion detection method of the present application. Specifically, it may include the following steps:

[0025] Step S11: extracting features from the image sequence to be tested to obtain features to be processed.

[0026] In one implementation scenario, the image sequence to be tested includes a plurality of images to be tested, and the features to be processed of the image sequence to be tested include the features of each image to be tested. The plurality of images to be tested may be a plurality of images taken by a shooting device at a plurality of moments of the scene to be tested, or may be a plurality of video frames selected from a video of the scene to be tested taken by a shooting device. Usually, the plurality of video frames may be selected from a plurality of video frames whose maximum time interval in the video is less than a preset time interval, for example, a plurality of continuous video frames are selected from the video. The image sequence to be tested may be selected according to actual conditions, and is not specifically limited here.

[0027] In an implementation scenario, the image sequence to be tested can be feature extracted by the LBP algorithm (Local Binary Patterns), the HOG feature extraction algorithm (Histogram of Oriented Gradient), or the feature extraction network. For example, the feature extraction network can be CNN (convolution neural network) or RCNN (Region based Convolutional Neural Network). The feature extraction method can be selected according to the actual situation and is not specifically limited here.

[0028] Step S12: performing matching processing on the template features of the template image sequence and the features to be processed to obtain matching features.

[0029] The method of action detection in the present application is to detect whether there is a template action in the template image sequence in the image sequence to be tested based on the matching features between the template image sequence and the image sequence to be tested. Therefore, in an implementation scenario, before performing this step, it is also necessary to obtain the template features of the template image sequence. It should be noted that the action detection objects of the template image sequence and the image sequence to be tested are the same or similar. For example, when hand action detection is performed, the template image sequence and the image sequence to be tested are both hand action detection image sequences; when head action detection is performed, the template image sequence and the image sequence to be tested are both head action detection image sequences. The application scenario of action detection can be selected according to actual conditions and is not specifically limited here. The template image sequence includes a number of template images, and the features to be processed of the template image sequence include the features of each template image. The several template images can specifically be a number of images taken by a shooting device at several moments of a scene, or a number of video frames selected from a video of a scene shot by a shooting device. In some embodiments, in order to facilitate subsequent action detection and save processing resources, the template image may be a partial image obtained by cropping the action detection object from the original image captured during the above-mentioned shooting period. For example, if the action detection image is a person, the template image is a partial image obtained by cropping the detection frame portion of the person in the captured original image. It can be understood that the number of template images in the template image sequence and the number of images to be detected in the image sequence to be detected may be the same or different.

[0030] In an implementation scenario, the features to be processed and the template features can be extracted using the same feature extraction network. Of course, the features to be processed and the template features can also be extracted separately using two feature extraction networks with the same network structure and parameters. In the above manner, the features to be processed and the template features are extracted by a feature extraction network with the same weights, which helps to improve the robustness of the network when the categories are unbalanced, thereby making the application scenarios of the action detection method more extensive. It can be understood that the template feature can be obtained by extracting features from the template image sequence in advance using the above-mentioned feature extraction network. For example, after determining to use an image sequence of a template action as the template image sequence, the above-mentioned feature extraction network can be used to extract features from the template image sequence to obtain template features, and the template features can be saved in the feature library so that after the image sequence to be tested is obtained, it can be matched with the features to be processed of the image sequence to be tested to achieve action detection of the image sequence to be tested; or, the template image sequence can be saved first, and when it is necessary to perform action detection on the image sequence to be tested, the above-mentioned feature extraction network can be used to extract features from the template image sequence and the image sequence to be tested respectively to obtain template features and features to be processed, and then the template features and features to be processed are matched to achieve action detection of the image sequence to be tested.

[0031] In an implementation scenario, the matching feature may include a fusion feature of the template feature and the feature to be processed. Specifically, for example, the template feature and the feature to be processed may be concatenated in dimensions to obtain the matching feature.

[0032] In another implementation scenario, in order to further improve the accuracy of matching features, a matching network can be used to match template features and features to be processed to obtain matching features. Specifically, the template features and features to be processed can be used to perform a first fusion process to obtain similarity features; then the template features and features to be processed can be used to perform cross-correlation processing to obtain cross-correlation features. For example, when the template feature is c*m*n (c is the number of template features, that is, the number of template images, m is the height of the template feature, and n is the width of the template feature), the feature to be processed is c*a*b (c is the number of features to be processed, that is, the number of images to be tested, a is the height of the feature to be processed, and b is the width of the feature to be processed). First, the template feature c*m*n is pooled to c*1*1. The pooled template feature c*1*1 and the feature to be processed c*a*b are cross-correlated to obtain the cross-correlation feature c*a*b. On this basis, the similarity feature and the cross-correlation feature are subjected to a second fusion process to obtain a matching feature. For example, when the similarity feature is c*a*b and the cross-correlation feature is also c*a*b, the similarity feature and the cross-correlation feature can be spliced ​​first to obtain a splicing result 2c*a*b, and then a convolution operation is performed on the splicing result to obtain a matching feature c*a*b; or, the similarity feature and the cross-correlation feature can be spliced ​​first to determine the splicing result, and then the splicing result is subjected to a dimensionality reduction process to obtain a matching feature. The second fusion processing method can be selected according to actual conditions and is not specifically limited here. In the above method, the template feature and the feature to be processed are processed to obtain the similarity feature and the cross-correlation feature respectively, and then the matching feature is obtained through the similarity feature and the cross-correlation feature, which can further improve the efficiency of the matching feature, thereby improving the accuracy of action detection.

[0033] In one implementation scenario, in order to obtain similarity features, the template features can be downsampled to obtain sampling features. In this process, downsampling the features can further reduce the amount of calculation for the template features. After obtaining the sampling features, the sampling features and the features to be processed can be subjected to a first fusion process to obtain similarity features. Specifically, the first fusion process can be to first splice the sampling features with the features to be processed, and then perform a convolution operation to obtain the similarity features. For example, if the sampling features and the features to be processed are both c*a*b, the features obtained after splicing are 2c*a*b, and then a convolution operation is performed to obtain the similarity features c*a*b; the first fusion process can also first splice the sampling features with the features to be processed, and then perform a dimensionality reduction process to obtain the similarity features. The method for obtaining the similarity features can be selected according to the actual situation, and is not specifically limited here. The above method, by downsampling the template features, helps to reduce the amount of calculation for the template features, thereby improving the simplicity and efficiency of the action detection method.

[0034] In a specific implementation scenario, in order to obtain sampling features, the template features can be subjected to a first pooling process and a second pooling process respectively, and the first pooling feature and the second pooling feature can be obtained accordingly, wherein one of the first pooling process and the second pooling process can be an average pooling process, and the other can be a maximum pooling process. The first pooling process and the second pooling process are not specifically limited here, and can be selected according to actual conditions. After obtaining the first pooling feature and the second pooling feature, the first pooling feature and the second pooling feature can be fused, that is, the first pooling feature and the second pooling feature are spliced ​​to obtain a splicing feature with a dimension twice that of the preset dimension. For example, if the first pooling feature is c*1*1 and the second pooling feature is also c*1*1, the splicing feature obtained by splicing the first pooling feature and the second pooling feature is 2c*1*1. After obtaining the splicing feature, the splicing feature can be subjected to dimensionality reduction processing to obtain a sampling feature of a preset dimension, or the splicing feature can be subjected to a convolution operation to obtain a sampling feature of a preset dimension. The method of obtaining the sampling feature can be selected according to actual conditions, and is not specifically limited here. It should be noted that the dimensions of the template features, the features to be processed, the first pooled features, the second pooled features, the sampled features, and the copied features are all preset dimensions. In the above manner, by performing pooling processing on the template features, it is possible to reduce parameters and calculation amount while retaining the main features, prevent overfitting, and thereby maximize the simplicity and efficiency of action detection.

[0035] In a specific implementation scenario, in order to obtain a similarity feature, a copy feature of the same size as the feature to be processed can be generated by copying the sampled feature. Specifically, the features in each dimension of the sampled feature can be copied respectively to obtain a copy feature of a preset dimension and a resolution the same as the resolution of the feature to be processed. For example, if the sampled feature is c*1*1 and the resolution of the feature to be processed is c*a*b, the sampled feature is copied in each dimension, that is, after a*b copies, the copy feature is c*a*b. On this basis, the copied feature and the feature to be processed are subjected to a first fusion process to obtain a similarity feature. It should be noted that the dimensions of the template feature and the feature to be processed, the first pooled feature, the second pooled feature, the sampled feature, and the copy feature are all preset dimensions. The above method saves computational costs by copying the sampled feature, which helps to improve the efficiency of action detection.

[0036] See also Figure 2 , Figure 2 It is a schematic diagram of an embodiment of matching features obtained by matching template features and features to be processed. The template features of the template image sequence are c*h*w (c is the number of template features, h is the height of the template features, and w is the width of the template features), and the features to be processed are c*H*W (c is the number of features to be processed, H is the height of the features to be processed, and W is the width of the features to be processed). The template features are first subjected to a first pooling process and a second pooling process to obtain first pooling features and second pooling features. At this time, the first pooling features and the second pooling features are both c*1*1. The first pooling features and the second pooling features are then spliced ​​to obtain a spliced ​​feature 2c*1*1. The spliced ​​features are then subjected to a dimensionality reduction process to obtain a sampling feature c*1*1. Then, the sampling features and the features to be processed are subjected to a first fusion process to obtain similarity features. The copied features can be obtained by first copying the sampling features. Specifically, the features in each dimension of the sampling features are copied respectively to obtain a copy feature c*H*W of a preset dimension and a resolution the same as that of the features to be processed; then, the copied features and the features to be processed are subjected to a first fusion process to obtain similarity features. Specifically, the copied features and the features to be processed can be spliced ​​to obtain a 2c*H*W feature, and a convolution operation is performed on the feature to obtain a similarity feature c*H*W. In addition, the template features and the features to be processed are subjected to a cross-correlation process. Specifically, the sampling feature size 1*1 and the feature size H*W to be processed are multiplied in each c dimension to obtain a cross-correlation feature c*H*W. On this basis, the similarity features and the cross-correlation features are subjected to a second fusion process. Specifically, the similarity features and the cross-correlation features can be spliced ​​to obtain a 2c*H*W feature, and then a convolution operation is performed on the feature to obtain a matching feature c*H*W.

[0037] Step S13: Predicting based on the matching features, determining the action detection result of the image sequence to be tested, where the action detection result includes whether the image sequence to be tested has a template action corresponding to the template image sequence.

[0038] In one implementation scenario, the action detection result includes the location information of the template action in the image sequence to be tested and the confidence of the location information. It should be noted that the location information of the template action can be represented by the bounding rectangle of the template action area in the image to be tested (for example, the coordinates of the upper left corner and the lower right corner of the bounding rectangle), or by the key points in the template action area in the image to be tested. The representation method of the location information of the template action can be selected according to the actual situation and is not specifically limited here.

[0039] In an implementation scenario, in order to obtain the action detection result, the matching features can be predicted through a prediction network. For example, the prediction network can include but is not limited to: CNN, RCNN. Of course, the prediction network can also be composed of two modules, namely a regression module and a classification module. The regression module is used to predict the position information of the template action in the image sequence to be tested, and the classification module is used to predict whether there is representation information of the template action corresponding to the template image sequence in the image sequence to be tested. Specifically, the representation information can be the confidence of the position information (that is, the confidence can be understood as the template action probability), so it can be determined whether there is a template action corresponding to the template image sequence in the image sequence to be tested according to the size of the confidence, and the location of the possible template action can be determined according to the above position information. The prediction network can be selected according to the actual situation and is not limited here.

[0040] See also Figure 3 , Figure 3 is a schematic diagram of an embodiment of an action detection model, such as Figure 3 As shown, feature extraction is performed on the image sequence to be tested to obtain features to be processed, and the template features of the template image sequence and the features to be processed are matched to obtain matching features; prediction is performed based on the matching features to determine the step of action detection results of the image sequence to be tested, which can be performed by the action detection model. Specifically, the feature extraction network of the action detection model can be used to extract features of the image sequence to be tested to obtain features to be processed; then the matching network of the action detection model can be used to match the template features of the template image sequence and the features to be processed to obtain matching features; on this basis, the prediction network of the action detection model can be used to predict based on the matching features to determine the action detection results of the image sequence to be tested, and the action detection results include whether the image sequence to be tested has a template action corresponding to the template image sequence.

[0041] In a specific implementation scenario, during the training process of the action detection model, the action detection model includes a feature extraction network, the feature extraction network includes a first sub-network and a second sub-network, and the network structure and parameters of the first sub-network and the second sub-network are the same, the first sub-network can be used to extract features of a sample image sequence to be tested to obtain sample features to be processed; and the second sub-network can be used to extract features of a sample template image sequence to obtain sample template features; the matching network is used to match the sample template features with the sample features to be tested to obtain sample matching features; the prediction network is used to predict based on the sample matching features to determine the sample action detection results of the sample image sequence to be tested; based on the sample action detection results, Adjust the network parameters of the action detection model; and after the action detection model is trained, at least one of the first subnetwork and the second subnetwork is used as the feature extraction network in the trained action detection model, that is, the trained action detection model includes at least one of the first subnetwork and the second subnetwork in addition to the matching network and the prediction network. For example, the trained action detection module can only retain one of the first subnetwork and the second subnetwork (such as only retaining the first subnetwork) to extract the above-mentioned features to be processed; for another example, if the trained action detection module can retain the first subnetwork and the second subnetwork, the first subnetwork and the second subnetwork are both used to extract the above-mentioned features to be processed. The above method, by extracting features from the image sequence to be tested and obtaining matching features through sample template features and sample features to be tested, helps to improve the accuracy in the matching process, thereby improving the accuracy of action detection.

[0042] In one implementation scenario, the template features are obtained by extracting features from the template image sequence using the above-mentioned feature extraction network or the template feature extraction network, and the network structure and parameters of the template feature extraction network and the feature extraction network are the same. Exemplarily, if the trained action detection module only retains the first sub-network, the first sub-network is the feature extraction network, and the template features can be obtained by extracting features from the template image sequence through the feature extraction network, or by extracting the template features through a template feature extraction network having the same network structure and parameters as the feature extraction network (first sub-network). In this case, the template feature extraction network can be the second sub-network. The template feature extraction network can be selected according to actual conditions and is not specifically limited here.

[0043] The above scheme extracts features from the image sequence to be tested to obtain features to be processed, and matches the template features of the template image sequence with the features to be processed to obtain matching features. On this basis, the matching features are used for prediction to determine the action detection result of the image sequence to be tested, and the action detection result includes whether the image sequence to be tested has a template action corresponding to the template image sequence. On the one hand, by predicting the matching features obtained based on the features to be processed and the template features, the detection of whether the image sequence to be tested has a template action in the template image sequence is realized. Compared with directly predicting using the image features of the image sequence to be tested, the accuracy of action detection can be improved. Moreover, since the prediction network can only predict known template actions by directly using the image features of the image sequence to be tested, the scheme determines whether the image sequence to be tested has a template action in the template image sequence by matching features. Even if the prediction network does not know the template action in the template image sequence before prediction, the detection of its template action can be realized, that is, the detection of the template action unknown to the prediction network before prediction is realized, thereby improving the applicability of action detection. On the other hand, using matching features for prediction can maximize the simplicity and efficiency of action detection.

[0044] In addition, the action detection process of the present application scheme can be executed using an action detection model. Since the present application scheme determines whether the image sequence to be tested contains a template action in the template image sequence by matching features, even if the template action in the template image sequence has not appeared in the training process of the action detection model, the current action detection model can be directly used to detect the unknown template action without the need for retraining, thereby improving the applicability of action detection and improving the efficiency of action detection.

[0045] Furthermore, when the action detection process of the present application scheme is executed using an action detection model, during the training of the action detection model, two identical feature extraction networks are used to extract features from the sample test image sequence and the sample template image sequence, and the extracted features are matched to predict the sample detection results. Finally, the model is adjusted based on the sample detection results, which can make the trained action detection model more generalized.

[0046] See also Figure 4 , Figure 4 This is a flow chart of an embodiment of a training method for an action detection model of the present application. Specifically, it may include the following steps:

[0047] Step S41: using the feature extraction network of the action detection model to extract features from the sample to-be-tested image sequence and the sample template image sequence respectively, to obtain sample to-be-tested features of the sample to-be-tested image sequence and sample template features of the sample template image sequence.

[0048] In one implementation scenario, the position information and sample action category of the sample action category in the sample image sequence to be tested are marked. For example, the sample action category of the sample image sequence to be tested is marked as hand action. The position information of the sample action category can be marked by a circumscribed rectangle of the action category area in the image to be tested, or by key points in the action category area. The marking method can be selected according to actual conditions and is not specifically limited here.

[0049] In an implementation scenario, during the training of the action detection model, the action detection model includes a feature extraction network, and the feature extraction network includes a first subnetwork and a second subnetwork. The network structures and parameters of the first subnetwork and the second subnetwork are the same. The feature extraction network of the action detection model can be used to extract features of the sample image sequence to be tested and the sample template image sequence, respectively, to obtain sample features to be processed of the sample image sequence to be tested and sample template features of the sample template image sequence. Specifically, the first subnetwork can be used to first extract features of the sample image sequence to be tested to obtain sample features to be tested. Exemplarily, the sample features to be tested are c*H*W (c is the number of sample features to be tested, H is the height of the sample features to be tested, and W is the width of the sample features to be tested); and then the second subnetwork is used to extract features of the sample template image sequence to obtain sample template features. Exemplarily, the sample template features are c*h*w (c is the number of sample template features, h is the height of the sample template features, and w is the width of the sample template features). Since the sizes of the sample template image and the sample image to be tested are different, the sizes of the sample template features and the sample features to be tested will be inconsistent. In the above method, feature extraction is performed by utilizing the first sub-network and the second sub-network with the same network structure and parameters. Since the present application scheme determines whether the image sequence to be tested contains the template action in the template image sequence by matching features, even if the template action in the template image sequence has not appeared in the training process of the action detection model, the current action detection model can be directly used to realize the detection of the unknown template action.

[0050] Step S42: using a matching module to match the sample template features with the sample features to be tested, to obtain sample matching features.

[0051] In one implementation scenario, the way in which the matching module matches the sample template features and the sample features to be tested can refer to the way in which the matching module matches the template features and the features to be tested in the aforementioned disclosed embodiment, which will not be repeated here.

[0052] Step S43: using the prediction module to make predictions based on the sample matching features to determine the sample action detection results of the sample image sequence to be tested.

[0053] In an implementation scenario, the prediction module can input the matching features into a convolution layer of a preset size for feature extraction, then perform nonlinear transformation through an activation layer, and finally pass them into another convolution layer of a preset size for convolution operation, thereby obtaining the sample action detection result of the sample image sequence to be tested. In this process, the prediction module can be composed of two submodules, namely a regression submodule and a classification submodule. The regression submodule is used to predict the position information of the template action in the sample image sequence to be tested, and the classification submodule is used to predict the confidence of the position information.

[0054] Step S44: Based on the sample action detection results, adjust the network parameters of the action detection model.

[0055] In one implementation scenario, the network loss of the action detection model can be obtained through the sample action detection results, and the action detection model can be supervised by the loss function. Exemplarily, the loss function can be an L1 loss function or a triple loss function. The category of the loss function can be selected according to the actual situation and is not specifically limited here. Of course, supervision can also be performed simultaneously through multiple loss functions. When multiple loss functions are used for supervision at the same time, the multiple loss functions can be superimposed according to preset weights, and then the network parameters of the action detection model can be iteratively updated through the back propagation algorithm. Exemplarily, the location information of the sample template action can be supervised by the L1 loss function. The formula of the L1 loss function is expressed as follows:

[0056]

[0057] Among them, the location information of the sample template action can be output through the bounding rectangle. is the position information of the bounding rectangle of the template action in the sample image sequence to be tested, is the position information of the bounding rectangle of the sample template action marked in the sample image sequence to be tested, , is the center point of the circumscribed rectangle, h , w is the width and height of the bounding rectangle. On the basis of supervising the position information of the sample template action, the focal loss function can also be used to supervise whether the category action appears in the bounding rectangle of the template action in the sample image sequence to be tested. The focal loss function is expressed as follows:

[0058]

[0059] in, are two hyperparameters that can be set to 2 and 4 respectively. Represents ( x , y), Indicates that the coordinates are in ( x , y ). In addition, the similarity between the sample test image sequence and the sample template image sequence can be supervised by the triplet loss function. The triplet loss function is expressed as follows:

[0060]

[0061] Among them, a is the anchor example, p is the positive example, and n is the negative example. By optimizing the distance between the anchor example and the positive example to be smaller than the distance between the anchor example and the negative example, the similarity calculation between the sample data is realized. That is to say, a is the sample template image sequence, p and n are the sample image sequences to be tested, and p and a are image sequences with the same template action. Exemplarily, p and a are both image sequences of hand template actions; n and a are image sequences with different template actions. Exemplarily, n is an image sequence of hand template actions, and a is an image sequence of head template actions. Margin is a constant greater than 0, and d is a similarity function. At this time, the similarity calculation between the two sample data is realized by d. After obtaining the calculation results of the three loss functions, they can be superimposed by preset weights. The formula is expressed as follows:

[0062]

[0063] in, All of them are preset weights. After obtaining L through superposition processing with preset weights, the network parameters of the action detection model are adjusted. It should be noted that after the action detection model training is completed, at least one of the two feature extraction networks is retained. In the above manner, after training the action detection model, the action detection model can have better generalization ability, thereby making the model more robust.

[0064] See also Figure 5 , Figure 5This is a schematic diagram of an embodiment of testing an action detection model. After the action detection model is trained, the action detection model can also be tested. If the first sub-network and the second sub-network are both retained after the action detection model training is completed, the first sub-network or the second sub-network is used to extract features of the test image sequence to obtain the test features to be tested, and the other sub-network is used to extract features of the test template image sequence to obtain the test template features; if only one of the first sub-network and the second sub-network is retained after the action detection model training is completed, exemplarily, the first sub-network is retained in the first sub-network and the second sub-network, then the second sub-network can be used to extract features of the test template image sequence to obtain the test template features; or the test template image sequence can be extracted by the retained first sub-network to obtain the test template features, thereby forming a test template feature library. The method for obtaining the test template feature library can be selected according to the actual situation, and is not specifically limited here. After obtaining the test template feature library, the first sub-network (retained sub-network) is used to extract features of the test image sequence to obtain the features of the template to be tested, and then the matching module is used to match the test template feature library and the test features to obtain the test matching features. On this basis, the prediction module is used to perform prediction based on the test matching features to determine the test action detection result of the test image sequence to be tested, and the test action detection result includes the position information of the test template action in the test image sequence to be tested and the confidence of the position information.

[0065] The above scheme uses the feature extraction network of the action detection model to extract features from the sample image sequence to be tested and the sample template image sequence to obtain the sample features to be tested of the sample image sequence to be tested and the sample template features of the sample template image sequence, and uses the matching module to match the sample template features and the sample features to be tested to obtain the sample matching features, and then uses the prediction module to predict based on the sample matching features to determine the sample action detection results of the sample image sequence to be tested, and on this basis, adjust the network parameters of the action detection model based on the sample action detection results. During the training process of the action detection model, the first sub-network and the second sub-network are used to extract features from the sample image sequence to be tested and the sample template image sequence, and the extracted features are matched to predict the sample detection results, and finally the model is adjusted based on the sample detection results, which can make the trained action detection model more generalized.

[0066] See also Figure 6 , Figure 6It is a schematic diagram of the framework of an embodiment of the motion detection device of the present application. The motion detection device 60 includes an extraction module 61, a matching module 62 and a prediction module 63. Among them, the extraction module 61 is used to extract features of the image sequence to be tested to obtain features to be processed; the matching module 62 is used to match the template features of the template image sequence and the features to be processed to obtain matching features; the prediction module 63 is used to use the matching features to predict and determine the motion detection result of the image sequence to be tested, and the motion detection result includes whether the image sequence to be tested has a template action corresponding to the template image sequence.

[0067] The above scheme, on the one hand, predicts the matching features obtained based on the features to be processed and the template features, so as to detect whether the image sequence to be tested contains the template action in the template image sequence. Compared with directly using the image features of the image sequence to be tested for prediction, the accuracy of action detection can be improved. Moreover, since the prediction is directly based on the image features of the image sequence to be tested, the prediction network can only predict known template actions. The present scheme determines whether the image sequence to be tested contains the template action in the template image sequence through matching features. Even if the prediction network does not know the template action in the template image sequence before prediction, the template action can be detected, that is, the template action that is unknown to the prediction network before prediction is detected, thereby improving the applicability of action detection. On the other hand, using matching features for prediction can maximize the simplicity and efficiency of action detection.

[0068] In addition, the action detection process of the present application scheme can be executed using an action detection model. Since the present application scheme determines whether the image sequence to be tested contains a template action in the template image sequence by matching features, even if the template action in the template image sequence has not appeared in the training process of the action detection model, the current action detection model can be directly used to detect the unknown template action without the need for retraining, thereby improving the applicability of action detection and improving the efficiency of action detection.

[0069] In some disclosed embodiments, the action detection result includes location information of the template action in the image sequence to be detected and the confidence level of the location information.

[0070] In some disclosed embodiments, the matching module 62 includes a first fusion submodule, which is used to perform a first fusion process using the template features and the features to be processed to obtain a similarity feature; the matching module 62 includes a feature execution submodule, which is used to perform a cross-correlation process using the template features and the features to be processed to obtain a cross-correlation feature; the matching module 62 also includes a second fusion submodule, which is used to perform a second fusion process on the similarity feature and the cross-correlation feature to obtain a matching feature.

[0071] Therefore, by processing the template features and the features to be processed to obtain similarity features and cross-correlation features respectively, and then obtaining matching features through the similarity features and cross-correlation features, the efficiency of matching features can be further improved, thereby improving the accuracy of action detection.

[0072] In some disclosed embodiments, the first fusion submodule includes a downsampling unit, which is used to downsample the template features; the first fusion submodule also includes a fusion unit, which is used to perform a first fusion process on the template features after the downsampling process and the features to be processed to obtain similarity features.

[0073] Therefore, downsampling the template features can help reduce the amount of calculation on the template features, thereby improving the simplicity and efficiency of the action detection method.

[0074] In some disclosed embodiments, the downsampling unit includes a first melting subunit, which is used to perform a first pooling process and a second pooling process on the template features, respectively, to obtain first pooling features and second pooling features correspondingly; the downsampling unit includes a second fusion subunit, which is used to fuse the first pooling features and the second pooling features to obtain the template features after downsampling.

[0075] Therefore, by pooling the template features, we can reduce parameters and calculation amount while retaining the main features, prevent overfitting, and thus maximize the simplicity and efficiency of action detection.

[0076] In some disclosed embodiments, the dimensions of the template features and the features to be processed, the first pooled features, the second pooled features, and the template features after downsampling processing are all preset dimensions; the first melting subunit can splice the first pooled features and the second pooled features to obtain a spliced ​​feature with a dimension twice the preset dimension; and perform dimensionality reduction processing on the spliced ​​features to obtain the template features after downsampling processing of the preset dimension.

[0077] Therefore, by duplicating the sampled features, the computational cost is saved, which helps to improve the efficiency of action detection.

[0078] In some disclosed embodiments, the steps of performing feature extraction on the image sequence to be tested to obtain features to be processed; performing matching processing on the template features of the template image sequence and the features to be processed to obtain matching features; and performing prediction based on the matching features to determine the action detection results of the image sequence to be tested include: performing feature extraction on the image sequence to be tested using a feature extraction network of the action detection model to obtain features to be processed; performing matching processing on the template features of the template image sequence and the features to be processed using a matching network of the action detection model to obtain matching features; and performing prediction based on the matching features using a prediction network of the action detection model to determine the action detection results of the image sequence to be tested.

[0079] In some disclosed embodiments, template features are obtained by extracting features from a template image sequence using a feature extraction network or a template feature extraction network, and the template feature extraction network and the feature extraction network have the same network structure and parameters.

[0080] Therefore, extracting the features to be processed and the template features through a feature extraction network with the same weights helps to improve the robustness of the network when the categories are unbalanced, thereby improving the usage scenarios of the action detection method.

[0081] In some disclosed embodiments, during the training of the motion detection model, the feature extraction network includes a first subnetwork and a second subnetwork, and the network structures and parameters of the first subnetwork and the second subnetwork are the same. The motion detection device 60 can also use the first subnetwork to perform feature extraction on the sample image sequence to obtain the sample features to be tested; and use the second subnetwork to perform feature extraction on the sample template image sequence to obtain the sample template features; and use the matching network to match the sample template features and the sample features to be tested to obtain the sample matching features; and then use the prediction network to perform prediction based on the sample matching features to determine the sample motion detection results of the sample image sequence to be tested; on this basis, based on the sample motion detection results, adjust the network parameters of the motion detection model; and after the motion detection model training is completed, at least one of the first subnetwork and the second subnetwork is used as the feature extraction network in the trained motion detection model.

[0082] Therefore, the action detection process of the present application scheme can be executed using an action detection model. Since the present application scheme determines whether there is a template action in the template image sequence in the image sequence to be tested by matching features, even if the template action in the template image sequence has not appeared in the training process of the action detection model, the current action detection model can be directly used to detect the unknown template action without the need for retraining, thereby improving the applicability of action detection and improving the efficiency of action detection.

[0083] See also Figure 7 , Figure 7It is a schematic diagram of the framework of an embodiment of the training device of the action detection model of the present application. The training device 70 of the action detection model includes an extraction module 71, a matching module 72, a prediction module 73 and an adjustment module 74. Among them, the extraction module 71 is used to use the feature extraction network of the action detection model to extract features of the sample image sequence to be tested and the sample template image sequence respectively, and obtain the sample features to be tested of the sample image sequence to be tested and the sample template features of the sample template image sequence; the matching module 72 is used to use the matching module to match the sample template features and the sample features to be tested to obtain the sample matching features; the prediction module 73 is used to use the prediction module to predict based on the sample matching features to determine the sample action detection results of the sample image sequence to be tested; the adjustment module 74 is used to adjust the network parameters of the action detection model based on the sample action detection results.

[0084] In the above scheme, during the training process of the action detection model, two identical feature extraction networks are used to extract features from the sample test image sequence and the sample template image sequence, and the extracted features are matched to predict the sample detection results. Finally, the model is adjusted based on the sample detection results, which can make the trained action detection model more generalized.

[0085] In some disclosed embodiments, the action detection model includes a feature extraction network, the feature extraction network includes a first sub-network and a second sub-network, the network structure and parameters of the first sub-network and the second sub-network are the same; the extraction module 71 includes a first extraction sub-module, the first extraction sub-module is used to use the first sub-network to perform feature extraction on the sample image sequence to be tested, and obtain the sample features to be tested; the extraction module 71 also includes a second extraction sub-module, the second extraction sub-module is used to use the second sub-network to perform feature extraction on the sample template image sequence, and obtain the sample template features; and after the action detection model is trained, at least one of the first sub-network and the second sub-network is used as the feature extraction network in the trained action detection model.

[0086] Therefore, by using the first sub-network and the second sub-network with the same network structure and parameters to extract features from the sample test image sequence and the sample template image sequence, and matching the extracted features, the sample detection results are predicted, and finally the model is adjusted based on the sample detection results, the trained action detection model can be generalized better.

[0087] See also Figure 8 , Figure 8: is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, the memory 81 stores program instructions, and the processor 82 is used to execute the program instructions to implement the steps in any of the above-mentioned action detection method embodiments or the steps in any of the action detection model training method embodiments. Specifically, the electronic device 80 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0088] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps in any of the above-mentioned action detection method embodiments or the steps in any of the action detection model training method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0089] The above scheme, on the one hand, helps to improve the accuracy of the matching process because it first extracts features from the image sequence to be tested and obtains matching features through template features and features to be processed. On the other hand, using matching features for prediction can maximize the simplicity and efficiency of action detection.

[0090] In addition, on the one hand, by predicting the matching features obtained based on the features to be processed and the template features, the detection of whether the image sequence to be tested contains the template action in the template image sequence can be realized. Compared with directly using the image features of the image sequence to be tested for prediction, the accuracy of action detection can be improved. Moreover, since the prediction is directly based on the image features of the image sequence to be tested, the prediction network can only predict known template actions, while the present scheme determines whether the image sequence to be tested contains the template action in the template image sequence through matching features. Even if the template action in the template image sequence is unknown to the prediction network before prediction, the detection of the template action can be realized, that is, the detection of the template action that is unknown to the prediction network before prediction is realized, thereby improving the applicability of action detection. On the other hand, using matching features for prediction can maximize the simplicity and efficiency of action detection.

[0091] In addition, the action detection process of the present application scheme can be executed using an action detection model. Since the present application scheme determines whether the image sequence to be tested contains a template action in the template image sequence by matching features, even if the template action in the template image sequence has not appeared in the training process of the action detection model, the current action detection model can be directly used to detect the unknown template action without the need for retraining, thereby improving the applicability of action detection and improving the efficiency of action detection.

[0092] Furthermore, when the action detection process of the present application scheme is executed using an action detection model, during the training of the action detection model, two identical feature extraction networks are used to extract features from the sample test image sequence and the sample template image sequence, and the extracted features are matched to predict the sample detection results. Finally, the model is adjusted based on the sample detection results, which can make the trained action detection model more generalized.

[0093] See also Fig. 9 , Fig. 9 The computer-readable storage medium 90 stores program instructions 91 that can be executed by a processor, and the program instructions 91 are used to implement the steps in any of the above-mentioned action detection method embodiments or any of the above-mentioned action detection model training method embodiments.

[0094] The above scheme, on the one hand, predicts the matching features obtained based on the features to be processed and the template features, so as to detect whether the image sequence to be tested contains the template action in the template image sequence. Compared with directly using the image features of the image sequence to be tested for prediction, the accuracy of action detection can be improved. Moreover, since the prediction is directly based on the image features of the image sequence to be tested, the prediction network can only predict known template actions. The present scheme determines whether the image sequence to be tested contains the template action in the template image sequence through matching features. Even if the prediction network does not know the template action in the template image sequence before prediction, the template action can be detected, that is, the template action that is unknown to the prediction network before prediction is detected, thereby improving the applicability of action detection. On the other hand, using matching features for prediction can maximize the simplicity and efficiency of action detection.

[0095] In addition, the action detection process of the present application scheme can be executed using an action detection model. Since the present application scheme determines whether the image sequence to be tested contains a template action in the template image sequence by matching features, even if the template action in the template image sequence has not appeared in the training process of the action detection model, the current action detection model can be directly used to detect the unknown template action without the need for retraining, thereby improving the applicability of action detection and improving the efficiency of action detection.

[0096] Furthermore, when the action detection process of the present application scheme is executed using an action detection model, during the training of the action detection model, two identical feature extraction networks are used to extract features from the sample test image sequence and the sample template image sequence, and the extracted features are matched to predict the sample detection results. Finally, the model is adjusted based on the sample detection results, which can make the trained action detection model more generalized.

[0097] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0098] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0103] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A motion detection method, characterized in that: include: Extract features from the image sequence to be tested to obtain features to be processed; Performing matching processing on the template features of the template image sequence and the features to be processed to obtain matching features; wherein, performing matching processing on the template features of the template image sequence and the features to be processed to obtain matching features comprises: performing a first fusion processing on the template features and the features to be processed to obtain a similarity feature; and performing a cross-correlation processing on the template features and the features to be processed to obtain a cross-correlation feature; performing a second fusion processing on the similarity feature and the cross-correlation feature to obtain the matching feature; Prediction is performed based on the matching features to determine an action detection result of the image sequence to be tested, wherein the action detection result includes whether the image sequence to be tested has a template action corresponding to the template image sequence.

2. The method according to claim 1, characterized in that The action detection result includes position information of the template action in the image sequence to be detected and the confidence of the position information.

3. The method according to claim 1, characterized in that The first fusion process is performed using the template feature and the feature to be processed to obtain a similarity feature, including: Performing downsampling processing on the template features; The first fusion processing is performed on the template feature after the downsampling processing and the feature to be processed to obtain the similarity feature.

4. The method according to claim 3, characterized in that The downsampling process of the template feature includes: Performing a first pooling process and a second pooling process on the template features respectively, and obtaining a first pooling feature and a second pooling feature correspondingly; The first pooling feature and the second pooling feature are fused to obtain the template feature after the downsampling process.

5. The method according to claim 4, characterized in that The dimensions of the template feature, the feature to be processed, the first pooling feature, the second pooling feature, and the template feature after downsampling are all preset dimensions; The fusing the first pooling feature and the second pooling feature to obtain the template feature after the downsampling process includes: Concatenating the first pooled feature and the second pooled feature to obtain a concatenated feature whose dimension is twice the preset dimension; The concatenated features are subjected to dimensionality reduction processing to obtain the template features after the downsampling processing of the preset dimension.

6. The method according to claim 1, characterized in that The feature extraction is performed on the image sequence to be tested to obtain the feature to be processed; the template feature of the template image sequence and the feature to be processed are matched to obtain the matching feature; The step of predicting based on the matching features and determining the action detection result of the image sequence to be tested comprises: The feature extraction network of the action detection model is used to extract features from the image sequence to be tested, and the features to be processed are obtained; Using the matching network of the action detection model to match the template features of the template image sequence with the features to be processed to obtain matching features; The prediction network of the action detection model is used to perform prediction based on the matching features to determine the action detection result of the image sequence to be tested.

7. The method according to claim 6, characterized in that The template features are obtained by extracting features from a template image sequence using the feature extraction network or the template feature extraction network. The template feature extraction network and the feature extraction network have the same network structure and parameters.

8. The method according to claim 6, characterized in that During the training of the action detection model, the feature extraction network includes a first sub-network and a second sub-network, the first sub-network and the second sub-network have the same network structure and parameters, and the action detection model is trained in the following manner: Using the first sub-network to extract features from a sample image sequence to obtain features of the sample image sequence to be tested; as well as The second sub-network is used to extract features from the sample template image sequence to obtain sample template features; Using the matching network to match the sample template feature with the sample feature to be tested, to obtain a sample matching feature; Using the prediction network to make predictions based on the sample matching features, and determining a sample action detection result of the sample to-be-tested image sequence; Based on the sample action detection results, adjusting the network parameters of the action detection model; After the action detection model is trained, at least one of the first sub-network and the second sub-network is used as the feature extraction network in the trained action detection model.

9. A method for training an action detection model, characterized in that: include: Using the feature extraction network of the action detection model to extract features from the sample image sequence to be tested and the sample template image sequence respectively, to obtain sample features of the sample image sequence to be tested and sample template features of the sample template image sequence; The sample template feature and the sample feature to be tested are matched by a matching module to obtain a sample matching feature; wherein the matching of the sample template feature and the sample feature to be tested to obtain a sample matching feature includes: performing a first fusion process on the sample template feature and the sample feature to be tested to obtain a similarity feature; and performing a cross-correlation process on the sample template feature and the sample feature to be tested to obtain a cross-correlation feature; performing a second fusion process on the similarity feature and the cross-correlation feature to obtain the sample matching feature; Using a prediction module to make predictions based on the sample matching features to determine a sample action detection result of the sample image sequence to be tested; Based on the sample action detection results, network parameters of the action detection model are adjusted.

10. The method according to claim 9, characterized in that The action detection model includes a feature extraction network, the feature extraction network includes a first sub-network and a second sub-network, the first sub-network and the second sub-network have the same network structure and parameters, and the feature extraction network of the action detection model is used to extract features from a sample image sequence to be tested and a sample template image sequence, respectively, to obtain sample features of the sample image sequence to be tested and sample template features of the sample template image sequence, including: Using the first sub-network to extract features from the sample image sequence to obtain features of the sample image sequence to be tested; and The second sub-network is used to extract features from the sample template image sequence to obtain sample template features; After the action detection model is trained, at least one of the first sub-network and the second sub-network is used as the feature extraction network in the trained action detection model.

11. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the action detection method described in any one of claims 1 to 8 or the training method of the action detection model described in any one of claims 9 to 10.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the action detection method described in any one of claims 1 to 8 or the training method of the action detection model described in any one of claims 9 to 10.

Images