Method, device, system, electronic device and storage medium for determining key points of action

Abstract

The embodiments of the present application provide a method, device, system, electronic device and storage medium for determining action key points, which relate to the field of video processing technology, to solve the problem of low efficiency in positioning action key points in the prior art. The method includes: fusing a convolutional feature map and an object key point heat map within a frame to generate a spatial feature fusion map, wherein the convolutional feature map is generated by extracting convolutional features from a video segment frame by frame, and the object key point heat map is generated by detecting object key points from the video segment frame by frame; stacking the spatial feature fusion map along the time dimension to generate a spatiotemporal comprehensive feature map; generating a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object; and determining the time information and position information of the action key points of the target object in the video segment based on the spatiotemporal heat map of the target object.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a method, device, system, electronic device and storage medium for determining action key points. Background Art

[0002] With the development of video processing technology, there is a need to locate the key points of the action of the moving object in the video. The key points of the action are the key nodes of the action event, including the time node when the moving object makes the action and the location of the part that makes the action. For example, in a football game video, the special effects artist wants to add a "shock wave effect" to the ankle of the player who touches the ball at the moment of shooting. In this case, it is necessary to locate the time node of the player's shooting action and the location of the ankle that makes the action in the video.

[0003] Currently, the main method is to manually browse the video frame by frame and locate the key points of the motion of the moving object. This method has the problems of low efficiency and cannot be processed on a large scale. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and storage medium for determining action key points to solve the problem of low efficiency in positioning action key points in the prior art.

[0005] The embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for determining action key points, including:

[0007] Intra-frame fusion of the convolution feature map and the object key point heat map to generate a spatial feature fusion map, wherein the convolution feature map is generated by performing convolution feature extraction on the video segment frame by frame, and the object key point heat map is generated by performing object key point detection on the video segment frame by frame;

[0008] The spatial feature fusion map is stacked along the time dimension to generate a spatiotemporal comprehensive feature map;

[0009] Based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object, generating a spatiotemporal heat map of the target object;

[0010] Based on the spatiotemporal heat map of the target object, time information and position information of key action points of the target object in the video segment are determined.

[0011] In a second aspect, an embodiment of the present application provides a device for determining action key points, including:

[0012] A fusion module, configured to fuse the convolution feature map and the object key point heat map within a frame to generate a spatial feature fusion map, wherein the convolution feature map is generated by performing convolution feature extraction on a video segment frame by frame, and the object key point heat map is generated by performing object key point detection on the video segment frame by frame;

[0013] A stacking module, used for stacking the spatial feature fusion map along the time dimension to generate a spatiotemporal comprehensive feature map;

[0014] A generating module, configured to generate a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object;

[0015] A determination module is used to determine the time information and position information of the key points of the action of the target object in the video segment based on the spatiotemporal heat map of the target object.

[0016] In a third aspect, an embodiment of the present application provides a system for determining action key points, including:

[0017] A convolution feature extraction module is used to extract convolution features from the input video segment and generate a convolution feature map;

[0018] The object key point detection module is used to detect the object key points of the input video segment and generate the object key point heat map;

[0019] A spatiotemporal comprehensive feature processing module, used to fuse the convolution feature map output by the convolution feature extraction module and the object key point heat map output by the object key point detection module within a frame to generate a spatial feature fusion map, and to stack the spatial feature fusion map in a time series dimension to generate a spatiotemporal comprehensive feature map, and to generate a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object;

[0020] A processing module is used to output the time information and position information of the key points of the action of the target object in the video segment based on the spatiotemporal heat map of the target object output by the spatiotemporal comprehensive feature processing module.

[0021] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for determining action key points as described in the first aspect are implemented.

[0022] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for determining action key points described in the first aspect above are implemented.

[0023] In an embodiment of the present application, a convolutional feature map and an object key point heat map are generated by processing a video segment frame by frame, and the convolutional feature map and the object key point heat map are fused between frames to obtain a spatial feature fusion map. Since the convolutional feature map contains scene information and the object key point heat map contains object key point information, the obtained spatial feature fusion map retains the spatial features containing scene information and object key point information. Then, a spatiotemporal comprehensive feature map is obtained by stacking the spatial feature fusion map between frames, and the spatiotemporal comprehensive feature map retains the spatiotemporal features that can reflect the changes in the object's motion. The spatiotemporal heat map of the target object subsequently generated only retains the spatiotemporal features of the target object's motion changes, so that the time information and position information of the target object's action key points can be determined based on the spatiotemporal heat map of the target object, thereby realizing intelligent positioning of the action key points, realizing large-scale processing, and improving the efficiency of positioning the action key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a method for determining action key points provided in an embodiment of the present application;

[0025] Figure 2 It is a schematic diagram of key points of the human body provided in an embodiment of the present application;

[0026] Figure 3 is an example diagram of the action key point determination process provided by an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of the overall structure and calculation flow of the neural network algorithm model provided in the embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the training process of the neural network algorithm model provided in the embodiment of the present application;

[0029] Figure 6 It is a structural diagram of a system for determining key points of an action provided in an embodiment of the present application;

[0030] Figure 7 is a structural diagram of another action key point determination system provided in an embodiment of the present application;

[0031] Figure 8 is a structural diagram of an action key point determination device provided in an embodiment of the present application;

[0032] Fig. 9 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The present application is used to determine the key points of the actions of the moving objects in the video segments. The term "key points of actions" can also be referred to as the spatio-temporal key points of the action events, which can be understood as the key nodes of the action events, i.e., the high-energy moments when the moving objects (hereinafter referred to as "objects") initiate actions and the key parts that activate the high-energy moments. The determination of the key points of actions includes the determination of the time nodes when the objects initiate actions (the determination of the corresponding time information), and the determination of the location of the part where the actions are initiated (the determination of the corresponding position information). In the present application, the object can be a person, an animal, a virtual object that can initiate actions, or a robot that can initiate actions, etc., and the present application does not limit this.

[0035] In the prior art, the key points of the motion of the moving object are located manually by manually browsing the video frame by frame. This method has the problems of low efficiency and inability to be processed on a large scale.

[0036] With the development of artificial intelligence (AI) technology, technical solutions for realizing intelligent video understanding and processing based on artificial intelligence technology have been proposed one after another. However, the inventors found that none of these technical solutions can realize the determination of the key points of the action of the moving object in the video segment. Taking the image understanding algorithm as an example, this type of algorithm usually performs posture recognition, gesture recognition, expression recognition and other processing on the video segment frame by frame. However, the inventors found that this type of method can only understand based on single-frame image information, and cannot effectively identify the before and after time sequence association information based on the continuous change of multiple frames of video images. For example, when a player is detected in a shooting posture in a certain frame, the image understanding algorithm cannot identify whether the user is deliberately posing, or faking, or reaching this critical moment based on a complete continuous shooting action. Taking the spatiotemporal action detection algorithm as an example, this type of algorithm can extract the action segments of different objects at different times and positions in the video segment separately. However, the inventors found that this type of method can only identify the time segment of the object's action and the overall spatial area of ​​the object, and cannot locate the spatiotemporal key points of the action event. For example, the spatiotemporal motion detection algorithm can identify the area where the player is located in the time segment from when the player raises his foot to when the ball is kicked out, as well as the action category of shooting, but it cannot accurately locate the moment when the ankle touches the ball and the position of the ankle.

[0037] In view of this, the present application provides a method, device, electronic device and storage medium for determining action key points, so as to realize intelligent positioning of action key points and improve the positioning efficiency of action key points. The embodiments of the present application are described in detail below.

[0038] See also Figure 1 , Figure 1 Flowchart of the method for determining key action points provided by the embodiment of the present application. Figure 1 As shown, this embodiment provides a method for determining action key points, comprising the following steps:

[0039] Step 101: Intra-frame fusion of the convolution feature map and the object key point heat map to generate a spatial feature fusion map;

[0040] Step 102: stacking the spatial feature fusion graph along the time series dimension to generate a spatiotemporal comprehensive feature graph;

[0041] Step 103: generating a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object;

[0042] Step 104: Based on the spatiotemporal heat map of the target object, determine the time information and position information of the key points of the action of the target object in the video segment.

[0043] For the convenience of explanation, the action key point determination method of this embodiment is described below by taking the action key point determination device as the execution subject.

[0044] Before step 101, the action key point determination device may obtain a video segment containing action data of one or more moving objects, and process the video segment frame by frame to obtain a convolution feature map and an object key point heat map. The convolution feature map may be generated by performing convolution feature extraction on the video segment frame by frame, and the object key point heat map may be generated by performing object key point detection on the video segment frame by frame.

[0045] Both the convolution feature map and the object key point heat map can be obtained using existing technologies. For example, the convolution neural network can be used to extract the features of the spatial dimension of the video segment frame by frame to obtain the convolution feature map. The obtained convolution feature map contains composite information such as object information and scene information, and retains the scene information of the task. The object key point detection network can be used to detect the object key points of the video segment frame by frame to obtain the object key point heat map of the spatial dimension. The obtained object key point heat map contains the object key point feature information. Taking the human object as an example, there are already human key point detection technologies in the existing technology, and there are many human key point detection algorithms. Mainstream network models such as HRNet, CPM, Hourglass, etc. usually decompose each character in the picture into left shoulder, right shoulder, left knee, right knee and other parts based on the prior knowledge of human body structure and mark the corresponding key points, such as Figure 2 shown.

[0046] The convolution feature map contains scene features, and the object key point heat map contains object key point features.

[0047] In step 101, the action key point determination device can perform intra-frame fusion of the convolution feature map and the object key point heat map frame by frame to generate a spatial feature fusion map. The intra-frame fusion here can be understood as fusing the convolution feature map and the object key point heat map corresponding to the same video frame to obtain the spatial feature fusion map corresponding to the corresponding video frame, and the intra-frame fusion frame by frame obtains the spatial feature fusion map corresponding to each video frame.

[0048] Since the convolutional feature map contains scene information and the object key point heat map contains object key point information, the obtained spatial feature fusion map retains the spatial features containing scene information and object key point information.

[0049] In step 102, the action key point determination device can stack the spatial fusion feature maps of different video frames along the temporal dimension to generate a spatiotemporal comprehensive feature map. By stacking the spatial fusion feature maps of different video frames along the temporal dimension, the spatiotemporal comprehensive feature map retains the spatiotemporal features that can reflect the changes in the object's actions.

[0050] In step 103, the action key point determination device can generate a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object.

[0051] Here, the target object refers to a specific moving object in the video segment. When there is a moving object in the video segment, the target object is the moving object. When there are multiple moving objects in the video segment, an identifier can be added to each moving object, and the target object can be determined by the identifier of the moving object.

[0052] The spatiotemporal detection frame of the target object is obtained based on the spatiotemporal comprehensive feature map. As an example, the spatiotemporal comprehensive feature map can be detected based on a region proposal network (RPN) and a detection anchor to regress the spatiotemporal detection frame of the target object.

[0053] The spatiotemporal detection box of the target object can be used Indicates that, among them, t1..t n Represents all frames contained in the video segment, x t ,y t ,w t ,h t They represent the center point coordinates and width and height information of the spatiotemporal detection box at the tth frame respectively.

[0054] By anchoring the spatiotemporal detection frame of the target object, the obtained spatiotemporal heat map of the target object only retains the spatiotemporal characteristics of the target object's action changes, so that the time information and position information of the target object's action key points can be determined based on the spatiotemporal heat map of the target object.

[0055] Here, the spatiotemporal heat map of the target object can be understood as representing the possibility of the target object's action becoming a key moment and the possibility of becoming a key point position for activating the action moment through spatiotemporal heat.

[0056] In step 104, the action key point determination device can determine the time information and position information of the action key points of the target object in the video segment based on the spatiotemporal heat map of the target object, thereby realizing intelligent positioning of the action key points.

[0057] In this embodiment, a convolution feature map and an object key point heat map are generated by processing the video segment frame by frame, and the convolution feature map and the object key point heat map are fused between frames to obtain a spatial feature fusion map. Since the convolution feature map contains scene information and the object key point heat map contains object key point information, the obtained spatial feature fusion map retains the spatial features containing scene information and object key point information. Then, a spatiotemporal comprehensive feature map is obtained by stacking the spatial feature fusion map between frames, and the spatiotemporal comprehensive feature map retains the spatiotemporal features that can reflect the changes in the object's motion. The spatiotemporal heat map of the target object subsequently generated only retains the spatiotemporal features of the target object's motion changes, so that the time information and position information of the target object's action key points can be determined based on the spatiotemporal heat map of the target object, thereby realizing intelligent positioning of the action key points, realizing large-scale processing, and improving the efficiency of positioning the action key points.

[0058] In some embodiments, before fusing the convolution feature map and the object key point heat map within a frame to generate a spatial feature fusion map, the method further includes:

[0059] Performing N stages of feature extraction on each video frame of the video segment to generate N convolution feature maps, wherein the sampling scales of the N stages are different, and N is an integer greater than 1;

[0060] Scaling the N convolutional feature maps to a target scale to obtain N target convolutional feature maps;

[0061] The N target convolutional feature maps are merged in the channel dimension to generate the convolutional feature map.

[0062] In this embodiment, the action key point determination device can perform spatial dimension feature extraction on each video frame of the video segment based on the mainstream convolution backbone network (such as ResNet-50) to generate a multi-order convolution feature map (or multi-scale convolution feature map).

[0063] That is, the action key point determination device can perform N stages of feature extraction on each video frame of the video segment, and the sampling scales of the N stages are different, thereby generating N convolution feature maps. Then, the N convolution feature maps are scaled to the target scale, thereby obtaining N target convolution feature maps. Then, the N target convolution feature maps are merged in the channel dimension, thereby generating a multi-order convolution feature map.

[0064] In the N-stage feature extraction process, as the network processes layer by layer, the feature extraction is continuously abstracted from low-level visual features (such as lines, colors, etc.) to gradually form high-level semantic expressions (such as people, objects, etc.), and finally achieve image semantic understanding. However, this process is also accompanied by a large amount of information loss, especially the process of video action understanding requires the calculation of a large amount of low-level motion information, but in the high-level feature map, the low-level information is severely missing and difficult to restore.

[0065] Since this embodiment obtains N target convolution feature maps by scaling N convolution feature maps to the target scale, and then merges the N target convolution feature maps in the channel dimension to generate a multi-order convolution feature map, this process realizes the fusion of low-order global information and high-order semantic information, effectively solving the problem of information loss, so that the obtained appearance semantic information can better retain the action scene information.

[0066] As an example, assuming that N is equal to 5, the mainstream convolutional backbone network can perform five stages of feature extraction on each video frame of the video segment, and output feature maps with 2x, 4x, 8x, 16x, and 32x downsampling scales respectively. Assuming that the original size of the video frame is W×H×C0, the corresponding five-stage output sizes are W / 2×H / 2×C1, W / 4×H / 4×C2, W / 8×H / 8×C3, W / 16×H / 16×C4, and W / 32×H / 32×C5. The five-stage feature maps are scaled to the same scale, such as W / 16×H / 16, and then merged in the channel dimension to generate a size of W / 16×H / 16×C ms The multi-order convolution feature map of C ms =C1+C2+C3+C4+C5.

[0067] The present embodiment provides an implementation scheme for multi-stage feature extraction and fusion. As mentioned above, the present embodiment realizes the fusion of low-order global information and high-order semantic information through the process of multi-stage feature extraction and fusion, effectively solving the problem of information loss, so that the obtained appearance semantic information can better retain the action scene information and improve the positioning accuracy of action key points.

[0068] In some embodiments, the intra-frame fusion of the convolution feature map and the object key point heat map to generate a spatial feature fusion map includes:

[0069] The convolution feature map and the object key point heat map scaled to the target scale are fused in the channel dimension to generate the spatial feature fusion map.

[0070] In this embodiment, the action key point determination device can scale the object key point heat map obtained above to the same scale of the multi-order convolution feature map (for example, W / 16×H / 16×C k ), and fused with the aforementioned multi-order convolution feature map in the channel dimension to generate a spatial feature fusion map. The size of the spatial feature fusion map is, for example, W / 16×H / 16×C * , where C * =C ms +C k .

[0071] This embodiment provides an implementation scheme for fusing the object key point heat map with the multi-order convolution feature map to obtain a spatial feature fusion map. Since the appearance semantic information of the multi-order convolution feature map can better retain the action scene information, the spatial feature fusion map can better retain the action scene information and improve the positioning accuracy of the action key points.

[0072] In some embodiments, stacking the spatial feature fusion graph along the time dimension to generate a spatiotemporal comprehensive feature graph includes:

[0073] Splitting the video segment into T segments in a temporal dimension, where boundaries of adjacent segments of the T segments overlap, and T is an integer greater than 1;

[0074] The spatial feature fusion graphs corresponding to the T segments are stacked along the time dimension to generate T spatiotemporal comprehensive feature subgraphs;

[0075] The T spatiotemporal comprehensive feature subgraphs are merged to obtain the spatiotemporal comprehensive feature graph.

[0076] The present embodiment provides an implementation scheme for dividing a video segment into multiple time segments and processing them separately to obtain multiple spatiotemporal comprehensive feature subgraphs, and then merging the multiple spatiotemporal comprehensive feature subgraphs to obtain a total spatiotemporal comprehensive feature graph. Compared with the processing method without segmentation, this implementation scheme requires less memory for each processing and can be better applied to scenarios with insufficient memory.

[0077] In some embodiments, generating a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object includes:

[0078] Acquire feature information located in a first spatiotemporal detection frame in the spatiotemporal comprehensive feature map to obtain structural features of the target object and convolution features of the target object, wherein the first spatiotemporal detection frame is a spatiotemporal detection frame of the target object;

[0079] The structural features of the target object and the convolutional features of the target object are fused and then subjected to multi-layer bottleneck convolution processing to generate a spatiotemporal heat map of the target object.

[0080] This embodiment provides an implementation scheme for generating a spatiotemporal heat map based on a spatiotemporal comprehensive feature map and a spatiotemporal detection frame. Through the above-mentioned feature fusion and multi-layer bottleneck convolution processing process, the generated spatiotemporal heat map of the target object can better retain the spatiotemporal characteristics of the target object's action changes, thereby accurately determining the time information and position information of the target object's action key points, and improving the positioning accuracy of the action key points.

[0081] As an example, the action key point determination device can perform threshold filtering on the spatiotemporal comprehensive feature map based on the spatiotemporal detection frame, retain the information in the frame and remove the content outside the frame. If the video segment involves multiple objects, they are processed separately to generate multiple spatiotemporal feature maps of a single object. At this time, each spatiotemporal feature map only retains the spatiotemporal features related to a single object, including the structural features and convolution features (or multi-order convolution features) of the object, and is usually accompanied by a large amount of blank background. At this time, the action key point determination device can use bottleneck convolution for dimensionality reduction, realize cross-modal fusion of convolution features and structural features, and realize gradual feature abstraction through multi-layer bottleneck convolution processing, and finally generate a spatiotemporal heat map with a size of n×W / 16×H / 16×1.

[0082] In some embodiments, determining the time information and position information of the key action points of the target object in the video segment based on the spatiotemporal heat map of the target object includes:

[0083] Based on the heat value of the spatiotemporal heat map of the target object, determine M blocks of the spatiotemporal heat map of the target object, where M is an integer greater than or equal to 1;

[0084] Determine a target heat point whose heat value meets a preset condition in the M blocks;

[0085] Based on the target heat points of the M blocks, time information and position information of action key points of the target object in the video segment are determined.

[0086] This embodiment provides an implementation scheme for determining action key points based on a spatiotemporal heat map. Specifically, the action key point determination device first divides the spatiotemporal heat map into multiple blocks based on the heat values ​​of the spatiotemporal heat map, then determines the target heat points whose heat values ​​meet the preset conditions in each block, and then determines the time information and position information of the action key points based on the target heat points of all blocks.

[0087] Since the spatiotemporal heat map represents the possibility of an action becoming a key moment and the possibility of becoming a key point position that activates the action moment through spatiotemporal heat, the greater the spatiotemporal heat, the greater the possibility of the action becoming a key moment and the greater the possibility of becoming a key point position that activates the action moment. Therefore, each determined block can be an area with a larger heat value in the spatiotemporal heat map. Correspondingly, the target heat point of each block can also be a point with a larger heat value in each block. As an example, the target heat point can also be a point with the largest heat value in each block.

[0088] In this embodiment, the blocks and target heat points in the blocks are determined by the heat values ​​of the spatiotemporal heat map, so that the time information and position information of the determined action key points are accurate and reliable, thereby improving the positioning accuracy of the action key points.

[0089] In some embodiments, determining the M blocks of the spatiotemporal heat map of the target object based on the heat value of the spatiotemporal heat map of the target object includes:

[0090] Eliminate the area whose heat value is less than a preset threshold in the spatiotemporal heat map of the target object;

[0091] The remaining areas of the spatiotemporal heat map of the target object are clustered to obtain the M blocks.

[0092] This embodiment provides an implementation scheme for obtaining multiple blocks from a spatiotemporal heat map. Specifically, the action key point determination device may first remove the areas in the spatiotemporal heat map whose heat values ​​are less than a preset threshold, and then cluster the remaining areas to obtain multiple blocks. In addition, the action key point determination device may also directly divide the areas whose heat values ​​are greater than or equal to the preset threshold from the spatiotemporal heat map according to the size of the heat values, thereby obtaining multiple blocks.

[0093] It should be noted that after obtaining the target hot point (such as the maximum hot point) of each block, the spatiotemporal position of the target hot point in the video segment, that is, the time and position in the video segment, can be inferred based on the reverse calculation of the previous processing.

[0094] The following combination Figure 3 , the action key point determination process of the embodiment of the present application is exemplarily described.

[0095] like Figure 3 As shown, the action key point determination process includes steps S1 to S7, which are as follows:

[0096] Step S1: Acquire a video segment containing one or more moving objects.

[0097] Step S2: Generate multi-order convolution feature maps frame by frame.

[0098] Specifically, based on the mainstream convolutional backbone network, multiple stages of feature extraction are performed on each frame of the video, with a different sampling scale at each stage. The convolutional feature map of each stage is then scaled to the same scale, and multiple convolutional feature maps are merged in the channel dimension to generate a multi-order convolutional feature map.

[0099] Step S3: Generate object key point heat map frame by frame.

[0100] Specifically, an object key point detection network is used to detect object key points in each frame of the video segment, and a heat map of object key points in the spatial dimension is output.

[0101] Step S4: Intra-frame fusion generates a spatial feature fusion map.

[0102] Specifically, the object key point heat map obtained in step S3 is scaled to the same scale and merged with the multi-order convolution feature map obtained in step S2 in the channel dimension to generate a spatial feature fusion map.

[0103] Step S5: Stack frames to generate a spatiotemporal comprehensive feature map.

[0104] Specifically, the spatial fusion feature maps generated from different video frames are stacked along the temporal dimension to generate a spatiotemporal comprehensive feature map.

[0105] Step S6.1: Output spatiotemporal detection boxes based on RPN and detection anchors.

[0106] Step S6.2: Generate a spatiotemporal heat map by combining the spatiotemporal detection box and the spatiotemporal comprehensive feature map.

[0107] Step S7: Output the time information and position information of the key points of the action.

[0108] The action key point determination process of the embodiment of the present application can be implemented by a neural network algorithm model. FIG4 shows the overall structure and calculation process of the neural network algorithm model. The training process of the neural network algorithm model can be found in Figure 5 .

[0109] like Figure 5 As shown in Figure 1, the training process of the neural network algorithm model includes the following steps:

[0110] Step T1: Obtain labeled data.

[0111] Specifically, multiple segments of video data containing relevant moving objects can be obtained, and the spatiotemporal detection frame and spatiotemporal key points of each moving object are annotated for each segment of video to form the annotated data. The spatiotemporal detection frame is the video period and spatial region of each frame where each object is located, while the spatiotemporal key points mark the key moments of the target object's actions and the key point position information that activates the key moments for the algorithm model to perform pre-training and subsequent recognition. The spatiotemporal key points are contained in the spatiotemporal detection frame, and the key point positions are Figure 3 A subset of the object key points detected in step S3.

[0112] Step T2: Process each video segment to generate spatiotemporal detection boxes and spatiotemporal heat maps.

[0113] For details, please refer to Figure 3From step S2 to step S6.2, each video segment in the labeled data is processed to generate a spatiotemporal detection box and a spatiotemporal heat map.

[0114] Step T3: Build a multi-task learning mechanism to implement model training.

[0115] Specifically, multi-task learning can be performed in combination with the outputs of step T1 and step T2 to implement model training. Usually, the spatiotemporal coordinates of the original key point annotations obtained in step T1 are too fine, and the spatiotemporal key points are relatively sparse, which is not conducive to model training. At this time, it can be converted into a Gaussian heat map based on the Gaussian kernel function, and then combined with mainstream loss functions such as MSE and wing loss and the gradient return mechanism to construct a supervision signal for the spatiotemporal heat map. The supervision signal of the spatiotemporal detection box can be constructed with reference to mainstream loss functions such as crossentropy and smooth L1. The two supervision signals are weighted and summed to construct a multi-task learning mechanism.

[0116] The action key point determination method provided in the embodiment of the present application can accurately locate or identify the action key points of the moving object, and can be applied to sports event editing and super AI special effects projects. For example, the time-space key points of the shooting action in a football game can be identified, and "shock wave special effects" can be added to the ankle of the player touching the ball, or the time-space key points of the dunk action in a basketball game can be identified, and "explosion special effects" can be added to the wrist of the player touching the frame, etc., to achieve cool and creative video production.

[0117] The beneficial effects of the embodiments of the present application include: realizing intelligent positioning of key action points, enabling large-scale processing, and improving the efficiency of positioning key action points.

[0118] The present application also provides a system for determining key points of an action. Figure 6 As shown, the action key point determination system M0 includes:

[0119] The convolution feature extraction module M1 is used to extract convolution features from the input video segment and generate a convolution feature map;

[0120] The object key point detection module M2 is used to detect the object key points of the input video segment and generate a heat map of the object key points;

[0121] The spatiotemporal comprehensive feature processing module M3 is used to fuse the convolution feature map output by the convolution feature extraction module and the object key point heat map output by the object key point detection module within a frame to generate a spatial feature fusion map, and to stack the spatial feature fusion map in the time dimension to generate a spatiotemporal comprehensive feature map, and to generate a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object;

[0122] The processing module M4 is used to output the time information and position information of the key points of the action of the target object in the video segment based on the spatiotemporal heat map of the target object output by the spatiotemporal comprehensive feature processing module.

[0123] In some embodiments, Figure 7 As shown, the action key point determination system M0 also includes:

[0124] Model training module M5: used to collect and save original video data and spatiotemporal key point annotations, generate ground truth spatiotemporal heat maps, and perform Figure 5 The relevant calculation logic in steps T1 to T3 shown is based on the "video + spatiotemporal heat map" pairing training data, and trains the neural network based on the loss function and gradient return mechanism of multi-task learning. The neural network consists of a convolutional feature extraction module M1, an object key point detection module M2 and a spatiotemporal comprehensive feature processing module M3.

[0125] The action key point determination system provided in the embodiment of the present application can implement each process implemented in the action key point determination method embodiment and achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0126] See also Figure 8 , Figure 8 is a structural diagram of the action key point determination device provided in the embodiment of the present application. Figure 8 As shown, the action key point determination device 800 includes:

[0127] A fusion module 801 is used to fuse the convolution feature map and the object key point heat map within a frame to generate a spatial feature fusion map, wherein the convolution feature map is generated by performing convolution feature extraction on the video segment frame by frame, and the object key point heat map is generated by performing object key point detection on the video segment frame by frame;

[0128] A stacking module 802 is used to stack the spatial feature fusion graph along the time series dimension to generate a spatiotemporal comprehensive feature graph;

[0129] A generating module 803 is used to generate a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object;

[0130] The determination module 804 is used to determine the time information and position information of the key points of the action of the target object in the video segment based on the spatiotemporal heat map of the target object.

[0131] In some embodiments, the action key point determination device 800 further includes:

[0132] An extraction module, configured to perform N-stage feature extraction on each video frame of the video segment to generate N convolution feature maps, wherein the sampling scales of the N stages are different, and N is an integer greater than 1;

[0133] A scaling module, used for scaling the N convolutional feature maps to a target scale to obtain N target convolutional feature maps;

[0134] A merging module is used to merge the N target convolutional feature maps in the channel dimension to generate the convolutional feature map.

[0135] In some embodiments, the fusion module 801 is specifically used to:

[0136] The convolution feature map and the object key point heat map scaled to the target scale are fused in the channel dimension to generate the spatial feature fusion map.

[0137] In some embodiments, the stacking module 802 includes:

[0138] a splitting unit, configured to split the video segment into T segments in a temporal dimension, wherein boundaries of adjacent segments of the T segments overlap, and T is an integer greater than 1;

[0139] A stacking unit, used for stacking the spatial feature fusion graphs corresponding to the T segments along the time dimension respectively, so as to generate T spatiotemporal comprehensive feature subgraphs;

[0140] A merging unit is used to merge the T spatiotemporal comprehensive feature sub-graphs to obtain the spatiotemporal comprehensive feature graph.

[0141] In some embodiments, the generation module 803 includes:

[0142] an acquisition unit, configured to acquire feature information in a first spatiotemporal detection frame in the spatiotemporal comprehensive feature map to obtain a structural feature of the target object and a convolution feature of the target object, wherein the first spatiotemporal detection frame is a spatiotemporal detection frame of the target object;

[0143] A processing unit is used to fuse the structural features of the target object and the convolution features of the target object and perform multi-layer bottleneck convolution processing to generate a spatiotemporal heat map of the target object.

[0144] In some embodiments, the determination module 804 includes:

[0145] A first determining unit is used to determine M blocks of the spatiotemporal heat map of the target object based on the heat value of the spatiotemporal heat map of the target object, where M is an integer greater than or equal to 1;

[0146] A second determining unit, configured to determine a target heat point in the M blocks whose heat values ​​meet a preset condition;

[0147] The third determination unit is used to determine the time information and position information of the action key points of the target object in the video segment based on the target heat points of the M blocks.

[0148] In some embodiments, the first determining unit includes:

[0149] A removal subunit, used to remove areas in the spatiotemporal heat map of the target object whose heat value is less than a preset threshold;

[0150] The clustering subunit is used to cluster the remaining areas of the spatiotemporal heat map of the target object to obtain the M blocks.

[0151] The action key point determination device provided in the embodiment of the present application can implement each process implemented in the above-mentioned action key point determination method embodiment and achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0152] like Fig. 9 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1002, a memory 1001, and a computer program stored in the memory 1001 and executable on the processor 1002. When the computer program is executed by the processor 1002, each process of the above-mentioned action key point determination method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0153] The processor 1002 is used to run the above-mentioned action key point determination system and execute the relevant calculations involved in the above-mentioned action key point determination method. Its existence includes but is not limited to a central processing unit (CPU), a field programmable gate array (FPGA), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processor (NPU), an application specific integrated circuit (ASIC), and a system on chip (SOC), and can be any one or more combinations.

[0154] The memory 1001 is used to store all long-term and short-term calculation data in the calculation process of the above method. It can exist in the form of physical media such as mobile phone memory, digital camera memory, USB flash drive, mobile hard disk, CD-ROM optical disk storage device, etc. It can be composed of a single or multiple physical media, and its logical media includes read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk, etc.

[0155] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned action key point determination method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0158] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for determining action key points, characterized in that: include: Intra-frame fusion of the convolution feature map and the object key point heat map to generate a spatial feature fusion map, wherein the convolution feature map is generated by performing convolution feature extraction on the video segment frame by frame, and the object key point heat map is generated by performing object key point detection on the video segment frame by frame; The spatial feature fusion map is stacked along the time dimension to generate a spatiotemporal comprehensive feature map; Based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object, a spatiotemporal heat map of the target object is generated, wherein the step of generating the spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object comprises: acquiring feature information located in a first spatiotemporal detection frame in the spatiotemporal comprehensive feature map to obtain structural features of the target object and convolutional features of the target object, wherein the first spatiotemporal detection frame is the spatiotemporal detection frame of the target object; fusing the structural features of the target object and the convolutional features of the target object, and performing multi-layer bottleneck convolution processing to generate the spatiotemporal heat map of the target object; Based on the spatiotemporal heat map of the target object, time information and position information of key action points of the target object in the video segment are determined.

2. The method according to claim 1, characterized in that Before fusing the convolution feature map and the object key point heat map within a frame to generate a spatial feature fusion map, the method further includes: Performing N stages of feature extraction on each video frame of the video segment to generate N convolution feature maps, wherein the sampling scales of the N stages are different, and N is an integer greater than 1; Scaling the N convolutional feature maps to a target scale to obtain N target convolutional feature maps; The N target convolutional feature maps are merged in the channel dimension to generate the convolutional feature map.

3. The method according to claim 2, characterized in that The convolution feature map and the object key point heat map are intra-frame fused to generate a spatial feature fusion map, including: The convolution feature map and the object key point heat map scaled to the target scale are fused in the channel dimension to generate the spatial feature fusion map.

4. The method according to claim 1, characterized in that: The step of stacking the spatial feature fusion graph along the time dimension to generate a spatiotemporal comprehensive feature graph includes: Splitting the video segment into T segments in a temporal dimension, where boundaries of adjacent segments of the T segments overlap, and T is an integer greater than 1; The spatial feature fusion graphs corresponding to the T segments are stacked along the time dimension to generate T spatiotemporal comprehensive feature subgraphs; The T spatiotemporal comprehensive feature subgraphs are merged to obtain the spatiotemporal comprehensive feature graph.

5. The method according to claim 1, characterized in that The determining, based on the spatiotemporal heat map of the target object, time information and position information of key action points of the target object in the video segment includes: Based on the heat value of the spatiotemporal heat map of the target object, determine M blocks of the spatiotemporal heat map of the target object, where M is an integer greater than or equal to 1; Determine a target heat point whose heat value meets a preset condition in the M blocks; Based on the target heat points of the M blocks, time information and position information of action key points of the target object in the video segment are determined.

6. The method according to claim 5, characterized in that The determining, based on the heat value of the spatiotemporal heat map of the target object, M blocks of the spatiotemporal heat map of the target object comprises: Eliminate the area whose heat value is less than a preset threshold in the spatiotemporal heat map of the target object; The remaining areas of the spatiotemporal heat map of the target object are clustered to obtain the M blocks.

7. An action key point determination device, characterized in that: include: A fusion module, configured to fuse the convolution feature map and the object key point heat map within a frame to generate a spatial feature fusion map, wherein the convolution feature map is generated by performing convolution feature extraction on a video segment frame by frame, and the object key point heat map is generated by performing object key point detection on the video segment frame by frame; A stacking module, used for stacking the spatial feature fusion map along the time dimension to generate a spatiotemporal comprehensive feature map; A generation module, used to generate a spatiotemporal heat map of the target object based on the spatiotemporal comprehensive feature map and the spatiotemporal detection frame of the target object, the generation module comprising: an acquisition unit, used to acquire feature information located in a first spatiotemporal detection frame in the spatiotemporal comprehensive feature map to obtain the structural features of the target object and the convolution features of the target object, the first spatiotemporal detection frame being the spatiotemporal detection frame of the target object; a processing unit, used to fuse the structural features of the target object and the convolution features of the target object and perform multi-layer bottleneck convolution processing to generate the spatiotemporal heat map of the target object; A determination module is used to determine the time information and position information of the key points of the action of the target object in the video segment based on the spatiotemporal heat map of the target object.

8. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for determining action key points as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the action key point determination method according to any one of claims 1 to 6 are implemented.

Images