A pre-training-based action recognition model training method, device and equipment

Abstract

The specification discloses a pre-training action recognition model training method, device and equipment. Space data for training a space feature extraction model space and time data for training a time feature extraction model are generated, the space feature extraction model and the time feature extraction model are pre-trained, collected video data is determined as a third sample, and an action type corresponding to the video data is determined as a label of the third sample, an action recognition model is trained, an encoding layer of the action recognition model is pre-trained by constructing a reservoir network group from two reservoir network models, the pre-trained action recognition model has good generalization on time feature extraction and space feature extraction, and the pre-trained action recognition model is trained again, so that the trained action recognition model can recognize an action type through multi-channel features and has higher accuracy in action type recognition.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of artificial intelligence, and in particular, to a pre-training based action recognition model training method, device and equipment. BACKGROUND

[0002] With the development of the field of artificial intelligence, the requirement for the precision of an action recognition model is increasingly high when performing action recognition. However, the existing action recognition model often needs a large number of training samples when training, and the training cost is high.

[0003] In the prior art, in order to solve the problems of small samples and high training cost, the reserve pool calculation is widely used in the training of the action recognition model due to its fast learning speed, low training cost and other characteristics.

[0004] However, when the action recognition model is trained by the reserve pool calculation, the inductive bias degree of the reserve pool calculation is not enough, which will lead to insufficient generalization of the model trained by the reserve pool calculation, and the precision of the trained model is low. Therefore, the present specification provides a pre-training based action recognition model training method, device and equipment. SUMMARY

[0005] The present specification provides a pre-training based action recognition model training method, device and equipment to partially solve the above problems existing in the prior art.

[0006] The present specification adopts the following technical solutions:

[0007] The present specification provides a pre-training based action recognition model training method, wherein the action recognition model comprises an encoding layer and a decoding layer, the encoding layer comprises a to-be-trained spatial feature extraction model and a to-be-trained temporal feature extraction model, and the to-be-trained spatial feature extraction model and the to-be-trained temporal feature extraction model are reserve pool networks, comprising:

[0008] In response to a model training instruction, spatial data for training the spatial feature extraction model and temporal data for training the temporal feature extraction model are generated, and the spatial data and the temporal data are composed of a plurality of video data segments that are continuous in content;

[0009] According to adjacent video data segments in the spatial data, a first sample and a label of the first sample are determined, the spatial feature extraction model is pre-trained, and according to adjacent video data segments in the temporal data, a second sample and a label of the second sample are determined, and the temporal feature extraction model is pre-trained;

[0010] determining the collected video data as a third sample, and determining a motion type corresponding to the video data as a label of the third sample;

[0011] inputting the third sample into an encoding layer of a motion recognition model determined by a pre-trained spatial feature extraction model and a pre-trained temporal extraction model, determining spatial features and temporal features, inputting the spatial features and the temporal features into a decoding layer of the motion recognition model, determining a predicted motion type, training the motion recognition model according to the predicted motion type and the label of the third sample, and using the trained motion recognition model to identify a motion type corresponding to input video data.

[0012] Optionally, spatial data used for training the spatial feature extraction model is generated, and specifically includes:

[0013] determining a spatial motion trajectory in a video space;

[0014] generating a plurality of motion points in the video space according to a starting point position of the spatial motion trajectory, the center of a Gaussian distribution of each motion point being the starting point position;

[0015] generating video data of continuous motion of the plurality of motion points, with the center of the Gaussian distribution moving along the spatial motion trajectory as a target;

[0016] when the center position of the Gaussian distribution changes, segmenting the video data of continuous motion to determine a plurality of continuous video data segments as spatial data used for training the spatial feature extraction model.

[0017] Optionally, a first sample and a label of the first sample are determined according to adjacent video data segments in the spatial data, and the spatial feature extraction model is pre-trained, and specifically includes:

[0018] determining adjacent video data segments in the spatial data, and taking a video data segment earlier in time in the adjacent video data segments as the first sample;

[0019] determining, according to another video data segment, a coordinate of the center of the Gaussian distribution of the motion light point in the video data segment as a label corresponding to the first sample;

[0020] inputting the first sample into the spatial feature extraction network to determine, as a first output result, a coordinate of the center of the Gaussian distribution of the plurality of motion light points in a predicted next video output by the spatial feature extraction network;

[0021] Determine a loss according to a difference between the first output result and a label corresponding to the first sample, to reduce the loss as an optimization target, and pre-train the spatial feature extraction model.

[0022] Optionally, time data for training the time feature extraction model is generated, specifically including:

[0023] Determine a second motion trajectory in a video space;

[0024] According to the second motion trajectory, generate video data of the target object moving along the second motion trajectory;

[0025] Divide the continuous video data of the target object according to time length, and determine time data for training the time feature extraction model.

[0026] Optionally, according to adjacent video data segments in the time data, determine a second sample and a label corresponding to the second sample, and pre-train the time feature extraction model, specifically including:

[0027] Determine adjacent video data segments in the time data, and take a video data segment earlier in time in the adjacent video data segments as the second sample;

[0028] According to another video data segment, determine a moving direction feature vector of the target object as a label corresponding to the second sample;

[0029] Input the second sample into the time feature extraction network, determine a predicted next video output by the time feature extraction network, and take a moving direction feature vector of the target object in the predicted next video as a second output result, to reduce a difference between the second output result and a label corresponding to the second sample as an optimization target, and pre-train the time feature extraction model.

[0030] Optionally, determine collected video data, wherein the collected video data is data collected by an event camera.

[0031] Optionally, the encoding layer further includes a preset dynamic attention network.

[0032] Input the third sample into an encoding layer of an action recognition model determined by the pre-trained spatial feature extraction model and the pre-trained time extraction model, to determine spatial features and time features, specifically including:

[0033] According to the third sample, determine an optical flow feature corresponding to the third sample, and then input the third sample into the dynamic attention network, to determine first resolution video data.

[0034] interpolating the third sample according to the first resolution video data to determine second resolution video data;

[0035] inputting the first resolution video data and the second resolution video data into the time feature extraction network to determine first resolution time features and second resolution time features as the time features, and inputting the first resolution video data, the second resolution video data, and the optical flow features into the space feature extraction network to determine first resolution space features, second resolution space features, and optical flow feature space features as the space features.

[0036] The specification provides a pre-trained action recognition model training device, the action recognition model comprising an encoding layer and a decoding layer, the encoding layer comprising a to-be-trained space feature extraction model and a to-be-trained time feature extraction model, the to-be-trained space feature extraction model and the to-be-trained time feature extraction model being a reserve pool network, comprising:

[0037] a response module configured to, in response to a model training instruction, generate space data for training the space feature extraction model and time data for training the time feature extraction model, the space data and the time data being composed of a plurality of video data segments that are continuous in content;

[0038] a pre-training module configured to determine a first sample and a label of the first sample according to adjacent video data segments in the space data, pre-train the space feature extraction model, and determine a second sample and a label of the second sample according to adjacent video data segments in the time data, pre-train the time feature extraction model;

[0039] a collection module configured to determine collected video data as a third sample, and determine an action type corresponding to the video data as a label of the third sample;

[0040] a training module configured to input the third sample into an encoding layer of an action recognition model determined by the pre-trained space feature extraction model and the pre-trained time feature extraction model, determine space features and time features, input the space features and the time features into a decoding layer of the action recognition model, determine a predicted action type, train the action recognition model according to the predicted action type and the label of the third sample, and use the trained action recognition model to identify an action type corresponding to input video data.

[0041] The specification provides a computer-readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the above-described pre-trained action recognition model training method.

[0042] The specification provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the above-mentioned pre-trained action recognition model training method when executing the program.

[0043] The above-mentioned at least one technical solution adopted by the specification can achieve the following beneficial effects:

[0044] In the pre-trained action recognition model training method provided by the specification, by generating spatial data for training the spatial feature extraction model space and time data for training the temporal feature extraction model, the spatial feature extraction model and the temporal feature extraction model are pre-trained, and then the collected video data is determined as the third sample, and the action type corresponding to the video data is determined as the label of the third sample, and the action recognition model is trained.

[0045] As can be seen from the above method, the reserve pool network group constructed by the two reserve pool network models constructs the encoding layer of the action recognition model, and the video data generated for the training of the temporal feature extraction ability and the spatial feature extraction ability is used to pre-train the temporal feature extraction ability and the spatial feature extraction ability of each reserve pool network in the encoding layer, so that the pre-trained action recognition model has better generalization in temporal feature extraction and spatial feature extraction, and then the pre-trained action recognition model is trained, so that the trained action recognition model can recognize the action type through multi-channel features, and the accuracy of action type recognition is higher. BRIEF DESCRIPTION OF DRAWINGS

[0046] The drawings described herein are used to provide further understanding of the specification, and form a part of the specification. The illustrative embodiments of the specification and their descriptions are used to explain the specification, and do not constitute an improper limitation on the specification. In the drawings:

[0047] Figure 1 A flowchart of a pre-trained action recognition model training method provided by an embodiment of the specification is shown in the figure;

[0048] Figure 2 A schematic diagram of spatial data provided by an embodiment of the specification is shown in the figure;

[0049] Figure 3 A schematic diagram of time data provided by an embodiment of the specification is shown in the figure;

[0050] Figure 4 A schematic diagram of an action recognition process provided by an embodiment of the specification is shown in the figure;

[0051] Figure 5A schematic diagram of a pre-training-based action recognition model training device provided for an embodiment of the present specification;

[0052] Figure 6 An electronic device schematic diagram corresponding to Figure 1 provided for the present specification. DETAILED DESCRIPTION

[0053] With the development of Internet technology, users' demand for shooting videos is increasingly high, and the types of videos are also increasingly diverse, and computer vision emerges as the times require. In many fields such as the medical field, the sports field, the education field, and the short video field, the accuracy requirement of the action type recognition corresponding to the video data is also increasingly high, such as intelligent video monitoring, medical diagnosis monitoring, intelligent human-computer interaction, and identity recognition, which all need accurate action type recognition.

[0054] At present, a reserve pool calculation is usually adopted to construct an action recognition model. The reserve pool calculation is a neural network calculation framework, which is usually composed of an input layer, a reserve pool, and an output layer, and the reserve pool therein is a recurrent neural network. Through the recurrent neural network, the reserve pool calculation can mine the input time sequence information, but also leads to that the reserve pool calculation is sensitive to time sequence, but has insufficient generalization to other features, and cannot form a corresponding bias. The action type recognition is difficult to obtain an accurate result only according to the input time sequence feature, and therefore, only constructing the action recognition model according to the reserve pool calculation will lead to that the model trained is low in accuracy when performing action type recognition.

[0055] Based on this, the present specification provides a pre-training-based action recognition model training method. In one or more embodiments of the present specification, the action recognition model comprises an encoding layer and a decoding layer. The encoding layer is a reserve pool network group architecture composed of at least two reserve pool networks. The reserve pool network group architecture at least includes a time feature extraction model and a spatial feature extraction model, which are used to extract time features and spatial features of input video data. The decoding layer is used to determine an action type corresponding to the input video data according to the time features and the spatial features output by the encoding layer.

[0056] To make the purpose, technical solutions, and advantages of the present specification clearer, the technical solutions of the present specification will be described clearly and completely in the following in combination with the embodiments of the present specification and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present specification, but not all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present invention.

[0057] In the following, the technical solutions provided by the embodiments of the present specification will be described in detail in combination with the drawings.

[0058] Figure 1 A flowchart of a pre-trained action recognition model training method provided in the specification includes the following steps:

[0059] S100: In response to a model training instruction, generate spatial data for training the spatial feature extraction model and temporal data for training the temporal feature extraction model, the spatial data and the temporal data being composed of a plurality of continuous video data segments in content.

[0060] In one or more embodiments of the specification, the action recognition model training method is not limited to being performed by a specific device, such as a personal computer, a mobile terminal, and a server. However, since the subsequent steps involve model training, feature extraction, and other operations that require high computing resources, such operations are generally performed by a server, so the subsequent description of the intelligent dialogue method is based on a server. The server can be a single device or composed of multiple devices, such as a distributed server, and the specification does not limit this.

[0061] In response to the model training instruction, if the action recognition model to be trained is already stored locally on the server, the model training instruction can only be a code to start model training. If the action recognition model to be trained is not stored locally on the server, the model training instruction can include the address or related information of the action recognition model to be trained. In one or more embodiments of the specification, the specific content or form of the model training instruction is not limited.

[0062] After receiving the model training instruction, the server generates spatial data for pre-training the spatial feature extraction model and temporal data for pre-training the temporal feature extraction model. It should be noted that the temporal data and spatial data described above are a plurality of continuous video data segments, and are dynamic vision sensor (dvs) video data, i.e., dvs data.

[0063] Of course, the spatial feature extraction model and the temporal feature extraction model described above are trained by a reservoir network according to specific training data, and the temporal data and spatial data described above can also be generated by other devices. After receiving the model training instruction, the server obtains the temporal data and spatial data, and the specification does not limit this in one or more embodiments.

[0064] Then, the server trains the untrained spatial feature extraction model and the untrained temporal feature extraction model according to the spatial data and the temporal data respectively, so as to realize that the two reserve pool models in the encoding layer of the action recognition model respectively adjust the bias of the temporal feature extraction and the bias of the temporal feature extraction.

[0065] Specifically, when generating the spatial data for pre-training the spatial feature extraction model, the spatial motion trajectory in the video space can be determined first, and then a plurality of motion points can be generated in the video space according to the starting point position of the spatial motion trajectory, the center of the Gaussian distribution of each motion point being the starting point position. The video data of continuous motion of the plurality of motion points can be generated by taking the movement of the center of the Gaussian distribution along the spatial motion trajectory as a target, and when the center of the Gaussian distribution changes, the continuous motion video data can be segmented to determine a plurality of continuous video data segments as the spatial data for training the spatial feature extraction model.

[0066] Of course, a plurality of continuous spatial video data segments can also be generated first, wherein the content of each spatial video data segment is the motion of a plurality of motion points, and for each spatial video data segment, the center of the Gaussian distribution of the plurality of motion points can be determined according to the motion points in each spatial video data segment, and the centers of the Gaussian distributions corresponding to each sequential spatial video data segment can move along a preset trajectory.

[0067] Specifically, as shown in Figure 2 , a schematic diagram of spatial data provided by an embodiment of the present specification is shown. Figure 2 Each rectangular box is a video data segment of a preset resolution, the dashed line in the video data segment is the motion trajectory of the center of the Gaussian distribution of the motion points, for each video data segment, the center of the Gaussian distribution of the motion points is the center of the relatively larger black dot, which is a virtual point, and the relatively smaller black dot is the motion point. The solid arrow to the right indicates the sequence of the video data segment in time. The centers of the Gaussian distributions of these motion points move linearly in any direction within a preset distance, and after the center moves a preset distance, the direction is randomly changed based on the movement, and when the center moves to the boundary of the video data, the position of the center is randomly initialized and continues to move in the original direction. It should be noted that the above-mentioned video content is only an example provided by one or more embodiments of the present specification, and other similar video data can also be used to train the spatial feature extraction network, such as changing the center of the Gaussian distribution of the motion light point to the center of the uniform distribution of the motion light point, or changing the center of the Gaussian distribution of the motion light point to the peak value of the gamma distribution of the motion light point. The present specification does not limit this.

[0068] In addition, when generating the time data for pre-training the time feature extraction model, a time motion track in the video space is first determined, and then a target object is generated, the center of the target object being located at the starting point of the time motion track. The video data of continuous motion of the target object is generated by taking the movement of the center of the target object along the time motion track as a target, the video data of continuous motion of the target object is segmented according to time length, and the time data for training the time feature extraction model is determined.

[0069] Specifically, as shown in FIG. 1, Figure 3 Figure 3 A schematic diagram of time data provided by an embodiment of the present specification is shown in FIG. 2, wherein each rectangular frame is a video space of a video data segment of a preset resolution, for each video data segment, the solid line rectangle in the video space is the starting position of the motion of a target object moving along a preset time track, and the dotted line rectangle is the end position of the target object moving along the preset time track in the video data segment. The solid arrow to the right indicates the order of the video data segment in time, and the end position of the target object of the video data segment earlier in time is the starting position of the target object of the adjacent video data segment later in time. Of course, the time feature of the video data is the feature of the change of the video data with time, and the above-mentioned video content is extracted by the direction feature instead of the time feature, which is only an example provided by one or more embodiments of the present specification, and other features can also be used to extract the time feature, such as speed, rotation angle, etc., which is not limited in the present specification.

[0070] For the convenience of subsequent description of pre-training of the time feature extraction model and the space feature extraction model, the above-mentioned two video contents are taken as examples for description.

[0071] S102: determining a first sample and a label of the first sample according to adjacent video data segments in the space data, pre-training the space feature extraction model, and determining a second sample and a label of the second sample according to adjacent video data segments in the time data, and pre-training the time feature extraction model.

[0072] After the server obtains the time data and the space data, the server starts to train the two reserve pool models of the encoding layer respectively, the pre-trained model determined by training the time data being a pre-trained time feature extraction model, and the pre-trained model determined by training the space data being a pre-trained space feature extraction model.

[0073] ​Specifically, when training the spatial feature extraction model, adjacent video data segments in the spatial data are determined, wherein when a previous video data segment is a first sample input to the spatial feature extraction model to be trained, the coordinates of the center of the Gaussian distribution of each moving light point can be determined according to the content of the next video data segment of the video data segment, as the label corresponding to the first sample. The first sample is input to the spatial feature extraction model to be trained, so that the spatial feature extraction model to be trained predicts the coordinates of the center of the Gaussian distribution of the moving light point in the next video data segment according to the input first sample, and outputs the predicted center coordinates, so as to reduce the difference between the predicted center coordinates and the label as an optimization target, and train the spatial feature extraction model to be trained. When the prediction accuracy reaches a preset value, the pre-trained spatial feature extraction model is determined.

[0074] When training the temporal feature extraction model, adjacent video data segments in the temporal data are determined, wherein when a previous video data segment is a second sample input to the temporal feature extraction model to be trained, the moving direction of the target object can be determined according to the content of the next video data segment of the video data segment, as the label corresponding to the second sample. The second sample is input to the temporal feature extraction model to be trained, so that the temporal feature extraction model to be trained outputs the moving direction of the target object in the next video data segment as a second output result. The temporal feature extraction model is trained to reduce the difference between the second output result and the label corresponding to the second sample as an optimization target, and a pre-trained temporal feature extraction model is determined.

[0075] S104: Determine the collected video data as a third sample, and determine the action type corresponding to the video data as the label of the third sample.

[0076] Since the pre-trained temporal feature extraction model and the pre-trained spatial feature extraction model have been determined through pre-training, that is, the action recognition model to be trained at this time already has certain spatial feature extraction capability and temporal feature extraction capability, that is, a bias to spatial features and a bias to temporal features are formed. To further ensure that the action recognition model accurately recognizes the action type in the video data, the server further trains the pre-trained action recognition model.

[0077] Specifically, the video data used to train the action recognition model is first acquired as a third sample, and the video data is real human motion video data. Of course, if the action recognition model to be trained, the video data can be collected according to the actual trained model. Then, the action type corresponding to the video data is acquired, which can be manually labeled or acquired through other means.

[0078] S106: inputting the third sample, determining spatial features and temporal features by an encoding layer of the action recognition model composed of the pre-trained spatial feature extraction model and the pre-trained temporal feature extraction model, inputting the spatial features and the temporal features into a decoding layer of the action recognition model, determining a predicted action type, training the action recognition model according to the predicted action type and the label of the third sample, and using the trained action recognition model to identify an action type corresponding to input video data.

[0079] After determining the third sample, the server starts training the entire action recognition model. Specifically, the server inputs the third sample into the action recognition model to be trained, extracts spatial features and temporal features of the third sample through the encoding layer of the action recognition model, that is, the pre-trained spatial feature extraction model and the pre-trained temporal feature extraction model, encodes the extracted features, and then inputs the encoded results into the decoding layer. The decoding layer determines the action type of the third sample as the output result according to the decoding of the encoding results, and trains the action recognition model to reduce the difference between the output result and the label, and determines the trained action recognition model.

[0080] Based on Figure 1 As shown in the pre-trained action recognition model training method, the spatial feature extraction model and the temporal feature extraction model are pre-trained by generating spatial data for training the spatial feature extraction model and temporal data for training the temporal feature extraction model, and then the collected video data is determined as the third sample, and the action type corresponding to the video data is determined as the label of the third sample, and the action recognition model is trained.

[0081] As can be seen from the above method, the encoding layer of the action recognition model is constructed by the reserve pool network group constructed by the two reserve pool network models, and the temporal feature extraction capability and the spatial feature extraction capability of each reserve pool network in the encoding layer are pre-trained by generating video data for training the temporal feature extraction capability and the spatial feature extraction capability, so that the pre-trained action recognition model has good generalization in temporal feature extraction and spatial feature extraction. Then, the pre-trained action recognition model is trained, so that the trained action recognition model can identify the action type through multi-channel features, and the accuracy of action type identification is higher.

[0082] It should be noted that whether it is the temporal data and spatial data generated in step S100 or the video data collected as the third sample determined in step S104, it is in dvs data format. Of course, it can also be RGB model data, but the RGB mode data should be preprocessed before model training.

[0083] Specifically, according to the collected video data, video data of a preset resolution is determined, a preset number of pictures are determined according to the video data of the preset resolution, and the pictures are sorted. According to two adjacent pictures, a difference picture is determined and normalized, and the processed picture is arranged in order to form video data as the third sample.

[0084] It should be noted that, due to the difference in the collection principle of RGB data and dvs data, the RGB data may contain a large amount of static video. Therefore, before data conversion, the RGB data should be cut to determine the video segment containing human motion.

[0085] Of course, the above method is a processing method for the obtained third sample being RGB data. It can also be used when the input video data is RGB data during action type recognition. If the input video data is converted into a data format that can be recognized by the action recognition model. Of course, during the training process, data in two data formats can also be collected, and after preprocessing, they are all used as third samples to train the action recognition model to be trained.

[0086] In addition, since the resolution of the collected third sample is generally high, direct extraction of time and space features according to the collected third sample requires more computing resources. Therefore, in the process of extracting time and space features from the third sample, the third sample is first input into a dynamic attention network to reduce the resolution of the third sample, and then the first resolution video data is determined. Figure 1 Based on the action recognition model training method shown in the figure, the encoding layer of the action recognition model to be trained can also include a dynamic attention network for reducing the resolution of the collected third sample. The third sample is input into the dynamic attention network to determine the first resolution video data. The third sample is cut according to the first resolution video data to determine the second resolution video data.

[0087] For example, if the resolution of the collected third sample is 1260*1260, the dynamic attention network can be set to reduce the resolution of the third sample, such as convolution, sampling, etc. A 46*46 resolution video data is determined as the first resolution video data, and then the subject of the action to be recognized in the third sample is cut to determine a 46*46 resolution video as the second resolution video data. When, the first resolution and the second resolution and the resolution of the third sample can be cut according to actual needs. In one or more embodiments of the present specification, the second resolution video data is not limited to how to cut the subject to be recognized. The action subject can be determined according to the optical flow map of the third sample, or the action subject can be recognized according to other models.

[0088] Then, the first resolution video data and the second resolution video data are input into the time feature extraction network to determine the first resolution time feature and the second resolution time feature, and the first resolution video and the second resolution video data are input into the space feature extraction network to determine the first resolution space feature and the second resolution space feature.

[0089] Then, the determined two time features and two space features are input into a decoding layer of the action recognition model to determine the action type corresponding to the input third sample.

[0090] Of course, when the space feature is extracted, to better extract the space feature of the third sample, the optical flow feature of the third sample can be determined according to the input third sample, and then the optical flow feature of the third sample and the second resolution video are input into the space feature extraction model to determine the space feature of the third sample, and the space feature of the third sample fused with the historical time sequence feature is obtained.

[0091] However, whether the first resolution time data or the second resolution video data is a part of video data obtained based on the third sample, and the time feature extraction model and the space feature extraction model consume less computing resources when extracting features than the computing resources required for directly extracting features based on the third sample.

[0092] Of course, since the feature extraction based on the first resolution video data is the global feature extraction of the original sample, and the feature extraction based on the second resolution video data is the detailed feature extraction of the third sample, the extraction of the time feature and the space feature based on the first resolution video data and the second resolution video data, and the decoding based on the extracted features to determine the corresponding action type do not change the accuracy, and whether the first resolution time data or the second resolution video data is a part of video data obtained based on the third sample, and the time feature extraction model and the space feature extraction model consume less computing resources when extracting features than the computing resources required for directly extracting features based on the third sample.

[0093] In addition, when spatial feature extraction is performed on the video data of the two resolutions, to further ensure the accuracy of the action recognition model, the optical flow feature of the third sample can be extracted, and then the extracted optical flow feature is input into the spatial feature extraction model to determine the spatial feature of the optical flow feature of the third sample, so that the decoding layer of the action recognition model determines the accuracy of the action type corresponding to the third sample according to the time feature and the spatial feature of the two resolution video data of the third sample and the spatial feature of the optical flow feature. Compared with the accuracy of the action type determined according to only the time feature and the spatial feature of the first resolution video data and the second resolution video data, the accuracy of the action type is higher.

[0094] The following is an identification process of a completed action recognition model provided in the present specification, as shown in Figure 4 . Figure 4 A complete architecture of an action recognition model provided in an embodiment of the present specification.

[0095] When the video data is input into the action recognition model, the preprocessing module of the action recognition model first preprocesses the video data. First, it is determined whether the input video data is dvs data. If not, the input video data needs to be converted to obtain data of the same structure as the dvs data. If it is dvs data or after data conversion, the video data is preprocessed, such as noise reduction and the like. Then, according to the preprocessed video data, the optical flow is extracted to determine the optical flow feature of the input video data.

[0096] Then, the video data is input into the encoding layer of the action recognition model, and the resolution is extracted through the dynamic attention model of the encoding layer to determine the video data of the preset first resolution. It should be noted that the dynamic attention model identifies and extracts the moving part of the preprocessed video data, and the extraction method can be sampling, convolution, etc. The present specification does not limit this. Then, according to the video data of the first resolution, the preprocessed video data is segmented to obtain the video data of the second resolution.

[0097] The video data of the first resolution and the video data of the second resolution are input into the time feature extraction model to determine the global time feature and the detailed time feature of the input video data. The optical flow feature is input into the spatial feature extraction model to determine the spatial feature of the optical flow feature of the input video data. The video data of the first resolution and the video data of the second resolution are input into the spatial feature extraction model to determine the global spatial feature and the detailed spatial feature of the input video data.

[0098] Finally, the global temporal features and the detailed temporal features, the global spatial features and the detailed spatial features, and the spatial features of the optical flow features output by the encoding layer are decoded by a decoding layer of the action recognition model, and the action type corresponding to the input video data is determined according to the decoding result. The decoding layer can be composed of a feature selector and at least one vector selection machine corresponding to a classification. The present specification does not limit this.

[0099] The above is a pre-training-based action recognition model training method provided by one or more embodiments of the present specification. Based on the same idea, the present specification also provides a corresponding pre-training-based action recognition model training device, as shown in Figure 5 .

[0100] Figure 5 A schematic diagram of a pre-training-based action recognition model training device provided by the present specification, the device is used to execute Figure 1 the pre-training-based action recognition model training method provided by the present specification.

[0101] A pre-training-based action recognition model training device, the action recognition model comprises an encoding layer and a decoding layer, the encoding layer includes a to-be-trained spatial feature extraction model and a to-be-trained temporal feature extraction model, the to-be-trained spatial feature extraction model and the to-be-trained temporal feature extraction model are reserve pool networks, comprising:

[0102] A response module 500 for generating spatial data for training the spatial feature extraction model and temporal data for training the temporal feature extraction model in response to a model training instruction, the spatial data and the temporal data are composed of a plurality of video data segments that are continuous in content;

[0103] A pre-training module 501 for determining a first sample and a label of the first sample according to adjacent video data segments in the spatial data, pre-training the spatial feature extraction model, and determining a second sample and a label of the second sample according to adjacent video data segments in the temporal data, pre-training the temporal feature extraction model;

[0104] An acquisition module 502 for determining acquired video data as a third sample, and determining an action type corresponding to the video data as a label of the third sample;

[0105] The training module 503 is configured to input the third sample, determine spatial features and temporal features by using an encoding layer of an action recognition model determined by a pre-trained spatial feature extraction model and a pre-trained temporal feature extraction model, input the spatial features and the temporal features into a decoding layer of the action recognition model, and determine a predicted action type; train the action recognition model according to the predicted action type and a label of the third sample, and use the trained action recognition model to identify an action type corresponding to input video data.

[0106] Optionally, the response module 500 is configured to determine a spatial motion trajectory in a video space, generate a plurality of motion points in the video space according to a starting point position of the spatial motion trajectory, a center of a Gaussian distribution of each motion point is the starting point position, generate video data of continuous motion of the plurality of motion points by taking the center of the Gaussian distribution as a target to move along the spatial motion trajectory, and segment the video data of the continuous motion when the center of the Gaussian distribution changes, to determine a plurality of continuous video data segments as spatial data for training the spatial feature extraction model.

[0107] Optionally, the pre-training module 501 is configured to determine adjacent video data segments in the spatial data, use a video data segment earlier in time as the adjacent video data segments as a first sample, determine coordinates of a center of a Gaussian distribution of a motion light point in a video data segment according to another video data segment as a label corresponding to the first sample, input the first sample into the spatial feature extraction network, determine coordinates of the center of the Gaussian distribution of the plurality of motion light points in a predicted next video output by the spatial feature extraction network as a first output result, determine a loss according to a difference between the first output result and the label corresponding to the first sample, and pre-train the spatial feature extraction model by using the loss as an optimization target.

[0108] Optionally, the response module 500 is configured to determine a second motion trajectory in a video space, generate video data of motion of an object along the second motion trajectory according to the second motion trajectory, segment continuous video data of the object according to time lengths, and determine temporal data for training the temporal feature extraction model.

[0109] Optionally, the pre-training module 501 is specifically configured to determine adjacent video data segments in the time data, take a video data segment earlier in time in the adjacent video data segments as the second sample, determine a moving direction feature vector of the target object according to another video data segment, take the second sample corresponding label as the second sample, input the second sample into the time feature extraction network, determine the predicted next video output by the time feature extraction network, and take the feature vector of the moving direction of the target object in the predicted next video as the second output result, so as to reduce the difference between the second output result and the second sample corresponding label as an optimization target, and pre-train the time feature extraction model.

[0110] Optionally, the training module 503 is further configured to, when the encoding layer further includes a preset dynamic attention network, determine the third sample corresponding optical flow feature according to the third sample, input the third sample into the dynamic attention network, determine first resolution video data, perform interception on the third sample according to the first resolution video data, determine second resolution video data, input the first resolution video data and the second resolution video data into the time feature extraction network, determine first resolution time feature and second resolution time feature as the time feature, and input the first resolution video data, the second resolution video data and the optical flow feature into the spatial feature extraction network, determine first resolution spatial feature, second resolution spatial feature and optical flow feature spatial feature as the spatial feature.

[0111] The specification also provides Figure 6 The schematic structural diagram of the electronic device is shown. As Figure 6 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the above Figure 1 The pre-training based action recognition model training method. Of course, in addition to the software implementation, the specification does not exclude other implementation manners, such as logic devices or software and hardware combined manner, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but also can be hardware or logic device.

[0112] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0113] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0114] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0115] For the sake of description, the above apparatuses are described in various units with functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in implementing the present specification.

[0116] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage etc.) containing computer usable program code.

[0117] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0118] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0119] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. ​ one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0120] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0121] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0122] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0123] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0124] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0126] The various embodiments described in this specification are described using a numbering of embodiments approach: these are each individually integrated contributions pertaining to different but related aspects of the description. Each of the various embodiments can stand on its own, and each can be used to advantage in a wide variety of applications. The various embodiments can be used in combination with each other in any suitable way. Not all components of each embodiment are necessarily described with respect to each illustrated embodiment. Methods of the various embodiments can be performed in any suitable order.

[0127] The above description is embodied in the form of embodiments only and is not intended to limit the present specification. The present specification can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. A pre-training-based action recognition model training method, characterized in that, The action recognition model comprises an encoding layer and a decoding layer, the encoding layer comprises a spatial feature extraction model to be trained and a temporal feature extraction model to be trained, and the spatial feature extraction model to be trained and the temporal feature extraction model to be trained are reserve pool networks, comprising: In response to a model training instruction, spatial data for training the spatial feature extraction model and temporal data for training the temporal feature extraction model are generated, and the spatial data and the temporal data are composed of a plurality of video data segments that are continuous in content; According to adjacent video data segments in the spatial data, a first sample and a label of the first sample are determined, the spatial feature extraction model is pre-trained, and according to adjacent video data segments in the temporal data, a second sample and a label of the second sample are determined, and the temporal feature extraction model is pre-trained; The collected video data is determined as a third sample, and the action type corresponding to the video data is determined as the label of the third sample; The third sample is input, the encoding layer of the action recognition model determined by the pre-trained spatial feature extraction model and the pre-trained temporal extraction model is used to determine spatial features and temporal features, the spatial features and the temporal features are input into the decoding layer of the action recognition model, a predicted action type is determined, the action recognition model is trained according to the predicted action type and the label of the third sample, and the trained action recognition model is used to identify the action type corresponding to the input video data.

2. The method of claim 1, wherein, The spatial data for training the spatial feature extraction model space is generated, specifically comprising: A first motion trajectory in a video space is determined; According to the starting point position of the first motion trajectory, a plurality of motion points are generated in the video space, and the center of the Gaussian distribution of each motion point is the starting point position; The video data of continuous motion of the plurality of motion points is generated by taking the movement of the center of the Gaussian distribution along the first motion trajectory as a target; When the position of the center of the Gaussian distribution changes, the continuous motion video data is segmented to determine a plurality of continuous video data segments as the spatial data for training the spatial feature extraction model.

3. The method of claim 2, wherein, According to adjacent video data segments in the spatial data, a first sample and a label of the first sample are determined, and the spatial feature extraction model is pre-trained, specifically comprising: The adjacent video data segments in the spatial data are determined, and the video data segment earlier in time in the adjacent video data segments is taken as the first sample; According to another video data segment, the coordinates of the center of the Gaussian distribution of the motion light points in the video data segment are determined as the label corresponding to the first sample; The first sample is input into the spatial feature extraction network, and the coordinates of the center of the Gaussian distribution of a plurality of motion light points in the predicted next video output by the spatial feature extraction network are determined as a first output result; According to the difference between the first output result and the label corresponding to the first sample, a loss is determined, and the spatial feature extraction model is pre-trained by optimizing the loss as an objective.

4. The method of claim 1, wherein, The time data for training the time feature extraction model is generated, specifically comprising: determining a second motion trajectory in a video space; generating video data of the target object moving along the second motion trajectory according to the second motion trajectory; segmenting the continuous video data of the target object by time length to determine the time data for training the time feature extraction model.

5. The method of claim 4, wherein, According to the adjacent video data segments in the time data, a second sample and a label of the second sample are determined, and the time feature extraction model is pre-trained, specifically comprising: determining adjacent video data segments in the time data, and taking a video data segment earlier in time in the adjacent video data segments as the second sample; determining a moving direction feature vector of the target object according to another video data segment as a label corresponding to the second sample; inputting the second sample into the time feature extraction network to determine a feature vector of the moving direction of the target object in a predicted next video output by the time feature extraction network as a second output result, taking the difference between the second output result and the label corresponding to the second sample as a loss, and taking the reduction of the loss as an optimization target to pre-train the time feature extraction model.

6. The method of claim 1, wherein, determining the collected video data, wherein the collected video data is data collected by an event camera.

7. The method of claim 1, wherein, The encoding layer further comprises a preset dynamic attention network; inputting the third sample into the encoding layer of the action recognition model determined by the pre-trained spatial feature extraction model and the pre-trained time extraction model to determine the spatial feature and the time feature, specifically comprising: determining the optical flow feature corresponding to the third sample according to the third sample, and then inputting the third sample into the dynamic attention network to determine the first resolution video data; cutting the third sample according to the first resolution video data to determine the second resolution video data; inputting the first resolution video data and the second resolution video data into the time feature extraction network to determine the first resolution time feature and the second resolution time feature as the time feature, and inputting the first resolution video data, the second resolution video data and the optical flow feature into the spatial feature extraction network to determine the first resolution spatial feature, the second resolution spatial feature and the optical flow feature spatial feature as the spatial feature. 8.A device for training a pre-trained action recognition model, characterized in that, The action recognition model comprises an encoding layer and a decoding layer, the encoding layer comprises a spatial feature extraction model to be trained and a time feature extraction model to be trained, and the spatial feature extraction model to be trained and the time feature extraction model to be trained are reserve pool networks, comprising: a response module for generating spatial data for training the spatial feature extraction model and time data for training the time feature extraction model in response to a model training instruction, the spatial data and the time data being composed of a plurality of video data segments that are continuous in content; The pre-training module is configured to determine a first sample and a label of the first sample according to adjacent video data segments in the spatial data, pre-train the spatial feature extraction model, and determine a second sample and a label of the second sample according to adjacent video data segments in the temporal data, and pre-train the temporal feature extraction model. The acquisition module is configured to determine acquired video data as a third sample, and determine a motion type corresponding to the video data as a label of the third sample. The training module is configured to input the third sample into an encoding layer of a motion recognition model determined by the pre-trained spatial feature extraction model and the pre-trained temporal feature extraction model, determine spatial features and temporal features, input the spatial features and the temporal features into a decoding layer of the motion recognition model, determine a predicted motion type, and train the motion recognition model according to the predicted motion type and the label of the third sample.

9. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1-7.

Images