Video action positioning method and device and electronic equipment

Abstract

The invention provides a video action positioning method and device and electronic equipment. The method comprises the following steps: dividing a to-be-identified video into N video clips; performing action recognition on the N video clips to obtain first action category probabilities of the N video clips; determining the motion category probability confidence coefficient of each image frame in the video clip, wherein the motion category probability confidence coefficient of the image frame represents that the motion category probability of the image frame is the confidence coefficient of the first motion category probability of the video clip; for each frame of image in the to-be-identified video, determining a second action category probability of the image based on M action category probability confidence coefficients of the image and a first action category probability of M video clips where the image is located, the M action category probability confidence coefficients including action category probability confidence coefficients of the image in the M video clips; and based on the second action category probability of the image in the to-be-identified video, determining a time sequence action positioning result of the to-be-identified video. And the video action positioning accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular to a video motion positioning method, apparatus, and electronic device. Background Technology

[0002] Video-based action recognition and localization is a process that automatically identifies and locates various actions by analyzing video data, and it has a wide range of applications. Action recognition refers to identifying the category of an action, while temporal action localization refers to locating the time when an action occurs in a video, that is, locating the image frame in which the action is located. Temporal action localization tasks are generally divided into two stages: searching for candidate segments and identifying the actions in the candidate segments; localization requires not only labeling the time segment in which the action occurs but also identifying the action category.

[0003] Video action recognition and temporal localization methods in related technologies typically involve segmenting the video into multiple shorter video segments, then using an action recognition model to predict the action category probability for each video segment. Finally, the action category probabilities at the video segment level are processed to obtain the action localization result for the entire video. However, this approach limits the temporal resolution of action localization to the length of the segmented segments, which can easily reduce the accuracy of action localization. Summary of the Invention

[0004] This application provides a video motion localization method, apparatus, and electronic device to solve the problem of low accuracy in existing video motion localization.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a video action localization method, the method comprising:

[0007] The video to be identified is divided into N video segments, where N is an integer greater than 1;

[0008] Action recognition is performed on the N video segments to obtain the first action category probability for each of the N video segments;

[0009] For each of the N video segments, determine the confidence level of the action category probability of each image frame in the video segment. The confidence level of the action category probability of the image frame represents the confidence level that the action category probability of the image frame is the first action category probability of the video segment.

[0010] For each frame of the video to be identified, the second action category probability of the image is determined based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer.

[0011] Based on the probability of the second action category in the image of the video to be identified, the temporal action localization result of the video to be identified is determined.

[0012] Secondly, embodiments of this application provide a video motion positioning device, the device comprising:

[0013] The segmentation module is used to divide the video to be identified into N video segments, where N is an integer greater than 1;

[0014] The recognition module is used to perform action recognition on the N video segments and obtain the first action category probability of each video segment in the N video segments;

[0015] The first determining module is used to determine, for each of the N video segments, the action category probability confidence level of each image frame in the video segment within the video segment, wherein the action category probability confidence level of the image frame represents the confidence level that the action category probability of the image frame is the first action category probability of the video segment;

[0016] The second determining module is used to determine the second action category probability of each frame of the image in the video to be identified, based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer.

[0017] The third determining module is used to determine the temporal action localization result of the video to be identified based on the second action category probability of the image in the video to be identified.

[0018] Thirdly, embodiments of this application provide an electronic device, including a transceiver and a processor.

[0019] The processor is used for:

[0020] The video to be identified is divided into N video segments, where N is an integer greater than 1;

[0021] Action recognition is performed on the N video segments to obtain the first action category probability for each of the N video segments;

[0022] For each of the N video segments, determine the confidence level of the action category probability of each image frame in the video segment. The confidence level of the action category probability of the image frame represents the confidence level that the action category probability of the image frame is the first action category probability of the video segment.

[0023] For each frame of the video to be identified, the second action category probability of the image is determined based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer.

[0024] Based on the probability of the second action category in the image of the video to be identified, the temporal action localization result of the video to be identified is determined.

[0025] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the video motion positioning method described in the first aspect.

[0026] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video motion localization method described in the first aspect.

[0027] In a sixth aspect, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in the first aspect above.

[0028] In the video action localization method of this embodiment, after determining the first action category probability of each video segment in N video segments of the video to be identified, the confidence level of the action category probability of each image frame in the video segment can be obtained. The confidence level of the action category probability of the image frame is the confidence level that the action category probability of the image frame is the first action category probability of the video segment. Then, using the confidence level of the action category probability of the image in M ​​video segments and the first action category probability of the M video segments in which the image is located, the second action category probability of the image is determined. That is, the first action category probability at the segment level is refined to the second category probability at the image frame level using the confidence level of the action category probability of the image frame in the video segment. The video action localization is achieved by using the more refined second action category probability at the image frame level, thereby obtaining the temporal action localization result of the video to be identified, which can improve the accuracy of video temporal action localization. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of a video motion localization method provided in an embodiment of this application;

[0031] Figure 2 This is one of the schematic diagrams of a video motion localization method provided in an embodiment of this application;

[0032] Figure 3 This is a second schematic diagram of a video motion localization method provided in an embodiment of this application;

[0033] Figure 4 This is the third schematic diagram of a video motion localization method provided in the embodiments of this application;

[0034] Figure 5 This is the fourth schematic diagram of a video motion localization method provided in the embodiments of this application;

[0035] Figure 6 This is a schematic diagram of the structure of a video motion positioning device provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] See Figure 1 , Figure 1 This is a flowchart of a video motion localization method provided in an embodiment of this application, which can be applied to electronic devices, such as... Figure 1 As shown, the video motion localization method provided in this embodiment includes the following steps:

[0039] Step 101: Divide the video to be identified into N video segments, where N is an integer greater than 1.

[0040] Each video segment may include at least two image frames (also referred to as at least two image frames). As an example, adjacent video segments in N video segments may overlap (i.e., adjacent video segments may have overlapping image frames) or not overlap. As an example, at least two adjacent video segments in N video segments may overlap.

[0041] Step 102: Perform action recognition on N video clips to obtain the first action category probability for each of the N video clips.

[0042] Action recognition can be performed on each of the N video segments to obtain the first action category probability of each video segment, which is also the segment-level action category probability. The first action category probability can also be understood as the first action category probability vector. The first action category probability of a video segment can include the probability value of the video segment belonging to each of the multiple action categories (for example, k+1 categories, which may include a background category and k action categories, where k is a positive integer).

[0043] Step 103: For each of the N video segments, determine the confidence level of the action category probability of each image frame in the video segment. The confidence level of the action category probability of the image frame represents the confidence level of the action category probability of the image frame as the first action category probability of the video segment.

[0044] For each of the N video segments, the confidence level of the action category probability of each image frame in the video segment can be determined. That is, the confidence level of the action category probability of the image frame is the first action category probability of the image frame in the video segment in which the image frame is located. Thus, the confidence level of the action category probability of each image frame in the N video segments in the video segment in which the image frame is located can be determined. In other words, if the image frame is an overlapping frame of two adjacent video segments, then the number of confidence levels of the action category probability of the image frame is two, that is, it includes the confidence level of the action category probability of the image frame in each video segment in which the image frame is located.

[0045] Step 104: For each frame of the video to be identified, determine the second action category probability of the image based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer.

[0046] Step 105: Determine the temporal action localization result of the video to be identified based on the probability of the second action category in the image of the video to be identified.

[0047] It can be understood that an image (or image frame) belongs to M video segments, and the image is an overlapping image frame of the M video segments. By using the confidence scores of the M action category probabilities of the image in the video to be identified, and the first action category probabilities of the M video segments to which the image belongs, the second action category probability of the image is determined. Each frame in the video to be identified can have its second action category probability determined in this way, thus obtaining the second action category probability of each frame in the video to be identified. In other words, by using the action category probability confidence scores, the action category probabilities at the segment level can be refined to the category probabilities at the image frame level. Then, using the second action category probabilities of the images in the video to be identified, the temporal action localization result of the video to be identified is determined. The second action category probability of an image can include the probability values ​​of the image belonging to each of the multiple action categories (i.e., action category probability values).

[0048] In the method of this embodiment, after determining the first action category probability of each video segment in N video segments of the video to be identified, the confidence level of the action category probability of each image frame in the video segment can be obtained. The confidence level of the action category probability of the image frame is the confidence level that the action category probability of the image frame is the first action category probability of the video segment. Then, using the confidence level of the action category probability of the image in M ​​video segments and the first action category probability of the M video segments in which the image is located, the second action category probability of the image is determined. That is, the first action category probability at the segment level is refined to the second category probability at the image frame level using the confidence level of the action category probability of the image frame in the video segment. The video action localization is achieved by using the more refined second action category probability at the image frame level, thereby obtaining the temporal action localization result of the video to be identified, which can improve the accuracy of video temporal action localization.

[0049] In some embodiments, the temporal action localization result of the video to be identified is determined based on the probability of a second action category in an image of the video to be identified, including:

[0050] For each frame of the video to be identified, the action category probability value of the candidate action category in the second action category probability of the image is determined as the image score, where the candidate action category is the category with the highest probability in the second action category probability of the image;

[0051] The candidate action category of the first target image in the video to be identified is updated to a preset category, and the score of the first target image is updated to the action category probability value of the preset category in the second action category probability of the first target image. The first target image is an image in the video to be identified whose action category probability value of the candidate action category is less than a preset probability threshold and whose candidate action category is not a preset category.

[0052] Based on the updated candidate action categories and updated scores of the images in the video to be identified, the temporal action localization result of the video to be identified is determined.

[0053] In other words, for each frame of the video to be identified, the category with the highest probability value in the second action category probability of the image can be used as the candidate action category of the image. The probability value of the candidate action category in the second action category probability of the image can be determined as the score of the image. Then, the non-preset category can be filtered based on the probability filtering mechanism. That is, the first target image that is not a preset category and whose action category probability value of the candidate action category is less than the preset probability threshold is updated in terms of category. Specifically, the candidate action category of the first target image is updated to the preset category, and the score of the first target image is updated to the action category probability value of the preset category in the second action category probability of the first target image. It should be noted that if the candidate action category of the first target image in the video to be identified is updated, then the candidate action category of the first target image is updated. For images in the video to be identified whose candidate action category probability value is greater than or equal to a preset probability threshold or whose candidate action category is a preset category, their candidate action category and score can remain unchanged. That is, the category and score are not updated for the time being. In the updated candidate action category and updated score of the images in the video to be identified, it can be understood that for images where the candidate action category and score are updated, that is, the updated candidate action category and updated score, and for images that are not updated, the updated candidate action category and updated score of these images are also the originally determined candidate action category and score.

[0054] As an example, the preset category can be the background category among multiple categories (i.e., multiple action categories), or it can be understood as a non-action category. As an example, the preset probability threshold can be preset according to needs or historical experience, without specific limitations. For example, the preset probability threshold can be 0.3, etc.

[0055] In this embodiment, the smaller the probability value of an action category, the lower the probability of belonging to that action category. The candidate action category is the category with the highest probability. If the probability value of a candidate action category is less than a preset probability threshold, it means that the probability of belonging to that candidate action category is low. Therefore, the category can be updated by using the preset probability threshold. This allows the candidate action categories of the first target image that are not in the preset category and whose probability value of the candidate action category is less than the preset probability threshold to be updated to the preset category, thereby improving the accuracy of the category of the first target image and thus improving the accuracy of subsequent video temporal action localization.

[0056] In some embodiments, the temporal action localization result of the video to be identified is determined based on the updated candidate action categories and updated scores of images in the video to be identified, including:

[0057] Images that are adjacent and have the same action category in the video to be identified are merged into candidate sub-segments, and the second target image in the video to be identified is formed into a separate candidate sub-segment, resulting in K candidate sub-segments of the video to be identified. The action categories of the second target image and its adjacent images in the video to be identified are different. The score of each candidate sub-segment is the average of the updated scores of the images in the candidate sub-segment. The action category of the candidate sub-segment is the action category of the images in the candidate sub-segment, and K is an integer greater than 1.

[0058] For each of the K candidate sub-segments, if the length of the candidate sub-segment is less than or equal to the first length threshold, the action category of the candidate sub-segment is updated to a preset category, and the score of the candidate sub-segment is updated to the mean of the action category probability values ​​of the preset category in the second action category probability of the image in the candidate sub-segment.

[0059] Based on the updated action categories and updated scores of K candidate sub-segments, the temporal action localization result of the video to be identified is determined.

[0060] It is understood that the candidate sub-segments in this embodiment can also be called candidate regions (i.e., video candidate regions). In this embodiment, adjacent image frames with the same action category can be merged, and each merge yields one candidate sub-segment. For images without adjacent images of the same category, a separate candidate sub-segment can be formed. In this way, a total of K candidate sub-segments can be obtained. Then, the action category of the candidate sub-segments can be updated based on a length-based filtering mechanism. For example, if the length of the candidate sub-segment (i.e., the number of image frames included in the candidate sub-segment) is less than or equal to a first length threshold, the action category of the candidate sub-segment is updated to a preset category, and the score of the candidate sub-segment is updated to the average of the action category probability values ​​of the preset category among the second action category probabilities of the images included in the candidate sub-segment. It should be noted that among the updated action categories and updated scores of the K candidate sub-segments, the action categories and scores of the merged candidate sub-segments are updated. The updated action categories and updated scores of these candidate sub-segments are the same as the updated action categories and scores after the above process. For candidate sub-segments formed by the image alone, the updated action categories and updated scores are also the same as the updated candidate action categories and updated scores of the image that formed the candidate sub-segment alone.

[0061] In this embodiment, adjacent images with the same action category can be merged into candidate sub-segments. Images without adjacent images of the same category are formed into separate candidate sub-segments, resulting in K candidate sub-segments. The score of each candidate sub-segment is the average of the updated scores of the images in the candidate sub-segment, and the action category of each candidate sub-segment is the action category of the images in the candidate sub-segment. Subsequently, if the length of a candidate sub-segment is less than or equal to a first length threshold, the action category of the candidate sub-segment can be updated to a preset category, and the score of the candidate sub-segment can be updated to the average of the probability values ​​of the preset action categories of the images in the candidate sub-segment. Based on the updated action categories and updated scores of the K candidate sub-segments, the temporal action localization result of the video to be identified can be determined to improve the accuracy of temporal action localization.

[0062] In some embodiments, the temporal action localization result of the video to be identified is determined based on the updated action categories and updated scores of K candidate sub-segments, including:

[0063] Merge adjacent candidate sub-segments from the K candidate sub-segments whose updated action categories are all preset categories, and keep the remaining candidate sub-segments from the K candidate sub-segments unchanged to obtain L first sub-segments. The score of the merged sub-segment in the L first sub-segments is the weighted sum of the updated scores of the merged sub-segments. The score of the candidate sub-segments in the L first sub-segments is the updated score of the candidate sub-segments. L is a positive integer less than or equal to K. The remaining candidate sub-segments are the remaining sub-segments from the K candidate sub-segments excluding the merged candidate sub-segments.

[0064] For each of the m sub-segment pairs that meet the preset requirements in the L first sub-segments, merge the sub-segment pair with one sub-segment between the sub-segment pairs to obtain a second sub-segment. The category of the second sub-segment is the action category of the sub-segment pair. The score of the second sub-segment is the average of the action category probabilities of the first action category in the second action category probabilities of the image in the second sub-segment. The first action category is the action category of the second sub-segment. The preset requirements include that the two sub-segments in the sub-segment pair have the same action category and are not of the preset category, and the action category of one sub-segment between the sub-segment pairs is of the preset category and the length of one sub-segment between the sub-segment pairs is less than or equal to the second length threshold.

[0065] Based on the action categories and scores of m second sub-segments and the action categories and scores of other first sub-segments, the temporal action localization result of the video to be identified is determined. The other first sub-segments are the unmerged sub-segments among the L first sub-segments.

[0066] It is understood that each candidate sub-segment among the remaining candidate sub-segments can form a first sub-segment independently. Each time adjacent candidate sub-segments with updated action categories of the same preset category are merged, a first sub-segment can be obtained. In this way, L first sub-segments can be obtained, that is, L first sub-segments include merged sub-segments and the remaining candidate sub-segments. The score of the merged sub-segment among the L first sub-segments is the weighted sum of the updated scores of the merged sub-segments, and the score of each candidate sub-segment among the remaining candidate sub-segments among the L first sub-segments is the updated score of that candidate sub-segment.

[0067] Then, for any two first sub-segments belonging to the same category but not to a preset category, if the sub-segments between them all belong to the preset category and the distance does not exceed a second preset length threshold, these two first sub-segments and the sub-segments of the preset category between them are merged into a second sub-segment. The score of the second sub-segment is then set to the average of the probability values ​​of all image frames included in the second sub-segment for that preset category in the second action category probability. Afterwards, based on the action categories and scores of the obtained m second sub-segments and the action categories and scores of the other first sub-segments, the temporal action localization result of the video to be identified is determined to improve localization accuracy. It should be noted that the other first sub-segments are the unmerged sub-segments among the L first sub-segments, or, in other words, the sub-segments other than those in the m sub-segment pairs that meet the preset requirements.

[0068] In some embodiments, determining the probability confidence of each image frame in a video segment for the action category within the video segment includes:

[0069] The confidence level of the action category probability of an image frame in a video clip is determined based on the distance between the center positions of the image frame and the video clip. The confidence level of the action category probability of an image frame in a video clip is inversely correlated with the distance between the center positions of the image frame and the video clip. The center position of the video clip is determined based on the number of image frames in the video clip.

[0070] That is, the greater the distance, the lower the confidence level, and the smaller the distance, the higher the confidence level. In this embodiment, multiple video segment-level action category probabilities can be fused to obtain more granular image frame-level action category probabilities. Different positions within a video segment have different confidence levels; the closer to the center of the video segment, the higher the confidence level, and the closer to the edge of the video segment, the lower the confidence level, thereby improving the accuracy of image frame-level action category probabilities.

[0071] In some embodiments, determining the second action category probability of an image based on the confidence scores of M action category probabilities of the image and the first action category probabilities of the M video segments to which the image belongs includes:

[0072] The ratio of the first value to the second value is taken as the second action category probability of the image. The second value is the sum of the confidence scores of the M action category probabilities of the image. The first value is the result of weighted summation of the first action category probabilities of the M video segments to which the image belongs, based on the confidence scores of the M action category probabilities of the image.

[0073] If an image is located in the overlapping portion of adjacent video segments, it may belong to more than one video segment, thus having more than one confidence level. For example, if an image belongs to M video segments, it has M action category probability confidence levels. These M action category probability confidence levels can be used as weights for the first action category probability of the M video segments to which the image belongs. A weighted sum of the first action category probabilities of the M video segments is obtained to get a first value. The sum of the M action category probability confidence levels is used as a second value. The ratio of the first value to the second value is used as the second action category probability of the image, thereby improving the accuracy of the second action category probability.

[0074] The process of the above method will be specifically described below with some specific embodiments.

[0075] In related technologies, the general process of video action recognition and temporal localization methods is as follows: First, the video is divided into multiple shorter video segments, and then the action recognition model is used to predict the action category probability of each video segment. Finally, the segment-level action category probabilities are processed into the action localization result of the entire video. However, this segment-level action recognition and localization method has the following limitations: (1) The temporal resolution of action localization is limited by the time length of the segment, which easily reduces the precision of action localization; (2) Since the video input information has a decay effect near the boundary of the segment, it is unreasonable to distribute the same category probability equally to all image frames within the video segment. To this end, this application proposes a multi-video segment probability fusion scheme to generate a more fine-grained image frame-level category probability from the video segment-level category probability. Based on the category probability after multi-video segment probability fusion, a temporal action localization calibration strategy is further proposed. This strategy generates the recognition result of long segments in a coarse-to-fine manner by gradually expanding and filtering local segments, thereby obtaining a more accurate temporal action boundary.

[0076] The solutions in this application can be applied to various scenarios. For example, they can be applied to the scenario of process action recognition and localization, providing a video-based method and system for process action recognition and localization. The method includes: 1) an action recognition network that analyzes segmented short video clips and outputs process action category recognition results at the video clip level; 2) a multi-segment probabilistic fusion strategy that, based on a preset prior confidence level, refines the action category recognition results at the segment level to the action category recognition results at the frame level; 3) a temporal action localization calibration strategy that, based on the frame-level action category recognition results, generates long-segment temporal action localization results by progressively filtering, merging, and smoothing local clips, thereby obtaining more accurate temporal action boundaries.

[0077] The solutions in this application mainly involve the following:

[0078] An action recognition network can analyze segmented video clips and output the action category recognition results at the video clip level, i.e., the probability of the first action category;

[0079] A multi-segment probability fusion strategy, which refines the action category recognition results at the video segment level to the action category recognition results at the frame level based on a preset prior confidence level, i.e., the second action category probability;

[0080] A temporal action localization calibration strategy is proposed. Based on the action category recognition results at the frame level, the strategy generates long-segment temporal action localization results by progressively filtering, merging and smoothing local segments, thereby obtaining more accurate temporal action boundaries, i.e., the temporal action localization results of the video.

[0081] The overall process principle of the technical solution in this application proposal is as follows: Figure 2 As shown, the overall algorithm process includes: (1) Action recognition (action recognition of video segments through an action recognition network); (2) Multi-segment probability fusion (probabilistic fusion of multiple video segments); (3) Temporal action localization calibration.

[0082] For action recognition networks:

[0083] For example, the video to be identified is divided into N segments, which may overlap with each other (e.g., adjacent segments). Then, F (positive integer) frames are extracted from each segment, with a sampling interval of T. These F frames are processed by an action recognition network (e.g., a neural network) to obtain the action class probability of this video segment, i.e., the first action class probability. For example, in Figure 2In this dataset, there are k+1 categories, including one background class and k action classes (e.g., category 1, category 2, ..., category k, i.e., category 1 - category k). After each video segment undergoes action recognition through the network, the probability values ​​of the video segment belonging to each of the k+1 categories can be obtained, which is the first action class probability of the video segment. In this part, the VideoMAE basic model architecture can be used, where ViT is selected as the network backbone, and pre-trained weights on the Kinetics-710 dataset are used for model initialization. After multiple rounds of training on a self-built process action recognition dataset, the network can predict and output the action class probabilities at the video segment level. For example, for video segment S, the corresponding action class probability can be output. ViT structure such as Figure 3 The portion to the left of the dashed line is shown. As an example, in action recognition, images in a video clip can be divided into n×n (for example, n can be 16) image patches. For example, as an example... Figure 3 The image is divided into 9 image patches, which are then flattened into one-dimensional vectors. These vectors are added to the position embedding vector of the image patch within the image, and then input into the encoder (TransformerEncoder). Finally, the linear programming language (MLP) head outputs the action class probability of the video segment, i.e., the first action class probability. The specific structure of the encoder is as follows: Figure 3 The right side of the dotted line.

[0084] For multi-segment probability fusion:

[0085] The aforementioned action recognition network can predict action category probabilities at the output video segment level, and the action recognition results at the video segment level have video segment-level resolution. Since video input information experiences attenuation near the boundaries of video segments, it is unreasonable to distribute the same action category probability equally among all image frames within a video segment, especially for positions near segment boundaries. In this application embodiment, a multi-segment probability fusion strategy is proposed, such as... Figure 4 As shown. The purpose of multi-segment probability fusion is to fuse action category probabilities at the level of multiple video segments to obtain more fine-grained frame-level action category probabilities. The confidence level of action category probabilities in a video segment follows a Gaussian prior distribution. Different positions within a video segment have different confidence levels; the confidence level is higher closer to the center of the video segment and lower closer to the edge. As an example, the confidence level of an image can be calculated using the following formula:

[0086] ;

[0087] in, Indicates video clip Images inside confidence level Represents an image frame With video clips Distance from the center Indicates standard deviation, The value can be preset, and can be preset according to actual inspection or actual needs, without specific limitations.

[0088] Based on this confidence calculation formula, the action category probabilities at the segment level can be assigned to all image frames within a segment with different confidence levels. For each image frame, all video segments containing that frame need to be aggregated to obtain the action category probability for each frame, i.e., the second action category probability. As an example, the second action category probability can be calculated using the following formula:

[0089] ;

[0090] in, Representing an image The probability of the action category, i.e., the image The probability of the second action category, Indicates video clip Action category probability, i.e., video segment The probability of the first action category, Represents all images A collection of video clips, for example, if there are M video clips including images, then... It includes M video clips. For example, such as... Figure 4 If two video clips overlap, for the overlapping frame image, the confidence level of the image in the two video clips is used to fuse the first action category probabilities of the two video clips using the above formula to obtain the second action category probability of the image.

[0091] For timing-based positioning calibration:

[0092] After obtaining the second action category probability for each frame, for each frame, the action category with the highest probability value among its second action category probabilities is selected as the candidate action category for that frame. Simultaneously, the probability of that frame in that candidate action category is recorded as the score for that frame, as illustrated in the diagram. Figure 5 As shown in the first line of the document. To obtain accurate temporal action boundaries, a temporal action localization calibration strategy is proposed to process the action recognition results of each frame image, generating the temporal action localization results of the entire video in a coarse-to-fine manner. This strategy includes the following steps:

[0093] Probability-based filtering mechanism: First, based on a preset probability threshold... Filtering out low-scoring frames requires attention to detail. This step only filters foreground (non-background) categories, reclassifying these low-scoring frames as background. Simultaneously, the frame's score is updated to the background probability value within the second action category probability set for that frame. Figure 5 As shown in the second line, for example, Figure 5 Preset probability threshold ;

[0094] Candidate region generation mechanism: Then, all adjacent frames of the same category are merged to obtain coarse candidate regions (candidate sub-fragments). The score of each candidate region is the average score of all the image frames it contains, such as... Figure 5 As shown in the third line of the document;

[0095] Length-based filtering mechanism: then based on a first length threshold Filter out those with a length not exceeding The candidate regions are identified, and their scores are updated to the mean of the probability values ​​of the background class of all frames contained within them, such as... Figure 5 As shown in the fourth line, for example, Figure 5 middle, .

[0096] Candidate region merging mechanism: Merge all adjacent candidate regions that are both in the background class to obtain a new candidate region, i.e., the first sub-segment. The score of the new candidate region is then set to the weighted sum of the scores of all its constituent candidate regions. The weight is the length of each candidate region divided by the sum of their lengths. Figure 5 As shown in the fifth line of the document.

[0097] Candidate region smoothing mechanism. Finally, for every two foreground candidate regions belonging to the same category, if the portions between them both belong to the background class and the distance does not exceed the second length threshold... Then, these candidate regions, along with the intermediate background candidate regions, are merged into a new foreground candidate region. The score of the new candidate region is then updated to the average probability of all image frames it contains in that foreground category. Figure 5 The sixth line in the example is shown below. Figure 5 middle, .

[0098] This strategy generates action candidate regions from frame-level action recognition results in a coarse-to-fine manner, ultimately yielding the temporal action localization results for the entire video.

[0099] The solution implemented in this application provides finer-grained frame-level action category probabilities. Based on the category probabilities obtained through multi-segment probability fusion, longer segment recognition results can be obtained, leading to more accurate temporal action boundaries. It also exhibits stronger generalization ability for actions in different scenarios and processes.

[0100] like Figure 6 As shown, Figure 6 This is a structural schematic diagram of a video motion positioning device 600 provided in an embodiment of this application, as shown below. Figure 6 As shown, the video motion positioning device 600 includes:

[0101] The segmentation module 601 is used to divide the video to be identified into N video segments, where N is an integer greater than 1;

[0102] The recognition module 602 is used to perform action recognition on N video segments and obtain the first action category probability of each video segment in the N video segments;

[0103] The first determining module 603 is used to determine the confidence level of the action category probability of each image frame in the video segment for each of the N video segments. The confidence level of the action category probability of the image frame represents the confidence level of the action category probability of the image frame as the first action category probability of the video segment.

[0104] The second determining module 604 is used to determine the second action category probability of an image for each frame of an image in the video to be identified, based on the confidence of the probability of the image of M action categories and the first action category probability of the image in the M video segments to which the image is located. The confidence of the probability of the M action categories includes the confidence of the probability of the image of the action categories in the M video segments, where M is a positive integer.

[0105] The third determining module 605 is used to determine the temporal action localization result of the video to be identified based on the probability of the second action category of the image in the video to be identified.

[0106] In some embodiments, the third determining module 605 includes:

[0107] The score determination module is used to determine the action category probability value of the candidate action category in the second action category probability of the image as the score of the image for each frame of the video to be identified, wherein the candidate action category is the category with the highest probability in the second action category probability of the image.

[0108] The first update module is used to update the candidate action category of the first target image in the video to be identified to a preset category, and to update the score of the first target image to the action category probability value of the preset category in the second action category probability of the first target image. The first target image is an image in the video to be identified whose action category probability value of the candidate action category is less than a preset probability threshold and whose candidate action category is not a preset category.

[0109] The result determination module is used to determine the temporal action localization result of the video to be identified based on the updated candidate action categories and updated scores of the images in the video to be identified.

[0110] In some embodiments, the result determination module includes:

[0111] The candidate sub-segment acquisition module is used to merge adjacent images with the same action category in the video to be identified into candidate sub-segments and to form a separate candidate sub-segment from the second target image in the video to be identified, thereby obtaining K candidate sub-segments of the video to be identified. In the video to be identified, the action categories of the second target image and its adjacent images are different. The score of each candidate sub-segment is the average of the updated scores of the images in the candidate sub-segment. The action category of the candidate sub-segment is the action category of the images in the candidate sub-segment, and K is an integer greater than 1.

[0112] The second update module is used to update the action category of each candidate sub-segment in the K candidate sub-segments to a preset category when the length of the candidate sub-segment is less than or equal to a first length threshold, and to update the score of the candidate sub-segment to the average of the action category probability values ​​of the preset category in the second action category probability of the image in the candidate sub-segment.

[0113] The motion localization result determination module is used to determine the temporal motion localization result of the video to be identified based on the updated motion categories and updated scores of K candidate sub-segments.

[0114] In some embodiments, the motion localization result determination module includes:

[0115] The first processing submodule is used to merge adjacent candidate sub-segments among K candidate sub-segments whose updated action categories are all preset categories, and to keep the remaining candidate sub-segments among K candidate sub-segments unchanged, so as to obtain L first sub-segments. The score of the merged sub-segment among the L first sub-segments is the weighted sum of the updated scores of the merged sub-segments. The score of the candidate sub-segments among the L first sub-segments is the updated score of the candidate sub-segments. L is a positive integer less than or equal to K. The remaining candidate sub-segments are the remaining sub-segments among the K candidate sub-segments excluding the merged candidate sub-segments.

[0116] The second processing submodule is used to merge one sub-segment between each of the m sub-segment pairs that meet the preset requirements in the L first sub-segments to obtain a second sub-segment. The category of the second sub-segment is the action category of the sub-segment pair. The score of the second sub-segment is the average of the action category probability values ​​of the first action category in the second action category probability of the image in the second sub-segment. The first action category is the action category of the second sub-segment. The preset requirements include that the two sub-segments in the sub-segment pair have the same action category and are not of the preset category, and the action category of one sub-segment between the sub-segment pair is of the preset category and the length of one sub-segment between the sub-segment pair is less than or equal to the second length threshold.

[0117] The localization result determination submodule is used to determine the temporal action localization result of the video to be identified based on the action categories and scores of m second sub-segments and the action categories and scores of other first sub-segments. The other first sub-segments are the unmerged sub-segments among the L first sub-segments.

[0118] In some embodiments, the first determining module is specifically used for:

[0119] The confidence level of the action category probability of an image frame in a video clip is determined based on the distance between the center positions of the image frame and the video clip. The confidence level of the action category probability of an image frame in a video clip is inversely correlated with the distance between the center positions of the image frame and the video clip. The center position of the video clip is determined based on the number of image frames in the video clip.

[0120] In some embodiments, the second determining module is specifically used for:

[0121] The ratio of the first value to the second value is taken as the second action category probability of the image. The second value is the sum of the confidence scores of the M action category probabilities of the image. The first value is the result of weighted summation of the first action category probabilities of the M video segments to which the image belongs, based on the confidence scores of the M action category probabilities of the image.

[0122] The video motion positioning device 600 provided in this embodiment can realize the various processes of the above-described video motion positioning method, with one-to-one correspondence of technical features and the same technical effect. To avoid repetition, it will not be described again here.

[0123] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described video motion positioning method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0124] For details, see Figure 7This application also provides an electronic device, which is a relay device, including a bus 701, a transceiver 702, an antenna 703, a bus interface 704, a processor 705, and a memory 706.

[0125] The processor 705 is used for:

[0126] The video to be identified is divided into N video segments, where N is an integer greater than 1;

[0127] Perform action recognition on N video clips to obtain the first action category probability for each of the N video clips;

[0128] For each of the N video segments, determine the confidence level of the action category probability of each image frame in the video segment. The confidence level of the action category probability of the image frame represents the confidence level of the action category probability of the image frame as the first action category probability of the video segment.

[0129] For each frame of the video to be identified, the second action category probability of the image is determined based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer.

[0130] Based on the probability of the second action category in the image of the video to be identified, the temporal action localization result of the video to be identified is determined.

[0131] In some embodiments, the processor 705 is specifically used for:

[0132] For each frame of the video to be identified, the action category probability value of the candidate action category in the second action category probability of the image is determined as the image score, where the candidate action category is the category with the highest probability in the second action category probability of the image;

[0133] The candidate action category of the first target image in the video to be identified is updated to a preset category, and the score of the first target image is updated to the action category probability value of the preset category in the second action category probability of the first target image. The first target image is an image in the video to be identified whose action category probability value of the candidate action category is less than a preset probability threshold and whose candidate action category is not a preset category.

[0134] Based on the updated candidate action categories and updated scores of the images in the video to be identified, the temporal action localization result of the video to be identified is determined.

[0135] In some embodiments, the processor 705 is specifically used for:

[0136] Images that are adjacent and have the same action category in the video to be identified are merged into candidate sub-segments, and the second target image in the video to be identified is formed into a separate candidate sub-segment, resulting in K candidate sub-segments of the video to be identified. The action categories of the second target image and its adjacent images in the video to be identified are different. The score of each candidate sub-segment is the average of the updated scores of the images in the candidate sub-segment. The action category of the candidate sub-segment is the action category of the images in the candidate sub-segment, and K is an integer greater than 1.

[0137] For each of the K candidate sub-segments, if the length of the candidate sub-segment is less than or equal to the first length threshold, the action category of the candidate sub-segment is updated to a preset category, and the score of the candidate sub-segment is updated to the mean of the action category probability values ​​of the preset category in the second action category probability of the image in the candidate sub-segment.

[0138] Based on the updated action categories and updated scores of K candidate sub-segments, the temporal action localization result of the video to be identified is determined.

[0139] In some embodiments, the processor 705 is specifically used for:

[0140] Merge adjacent candidate sub-segments from the K candidate sub-segments whose updated action categories are all preset categories, and keep the remaining candidate sub-segments from the K candidate sub-segments unchanged to obtain L first sub-segments. The score of the merged sub-segment in the L first sub-segments is the weighted sum of the updated scores of the merged sub-segments. The score of the candidate sub-segments in the L first sub-segments is the updated score of the candidate sub-segments. L is a positive integer less than or equal to K. The remaining candidate sub-segments are the remaining sub-segments from the K candidate sub-segments excluding the merged candidate sub-segments.

[0141] For each of the m sub-segment pairs that meet the preset requirements in the L first sub-segments, merge the sub-segment pair with one sub-segment between the sub-segment pairs to obtain a second sub-segment. The category of the second sub-segment is the action category of the sub-segment pair. The score of the second sub-segment is the average of the action category probabilities of the first action category in the second action category probabilities of the image in the second sub-segment. The first action category is the action category of the second sub-segment. The preset requirements include that the two sub-segments in the sub-segment pair have the same action category and are not of the preset category, and the action category of one sub-segment between the sub-segment pairs is of the preset category and the length of one sub-segment between the sub-segment pairs is less than or equal to the second length threshold.

[0142] Based on the action categories and scores of m second sub-segments and the action categories and scores of other first sub-segments, the temporal action localization result of the video to be identified is determined. The other first sub-segments are the unmerged sub-segments among the L first sub-segments.

[0143] In some embodiments, the processor 705 is specifically used for:

[0144] The confidence level of the action category probability of an image frame in a video clip is determined based on the distance between the center positions of the image frame and the video clip. The confidence level of the action category probability of an image frame in a video clip is inversely correlated with the distance between the center positions of the image frame and the video clip. The center position of the video clip is determined based on the number of image frames in the video clip.

[0145] In some embodiments, the processor 705 is specifically used for:

[0146] The ratio of the first value to the second value is taken as the second action category probability of the image. The second value is the sum of the confidence scores of the M action category probabilities of the image. The first value is the result of weighted summation of the first action category probabilities of the M video segments to which the image belongs, based on the confidence scores of the M action category probabilities of the image.

[0147] exist Figure 7 In this document, a bus architecture (represented by bus 701) is used. Bus 701 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 705 and memory represented by memory 706. Bus 701 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 704 provides an interface between bus 701 and transceiver 702. Transceiver 702 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 705 is transmitted over a wireless medium via antenna 703, which further receives data and transmits data to processor 705.

[0148] Processor 705 manages bus 701 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 706 can be used to store data used by processor 705 during operation.

[0149] Optionally, the processor 705 can be a CPU, ASIC, FPGA, or CPLD.

[0150] The processor 705 of the electronic device provided in this embodiment can implement each process of each embodiment of the above-described video motion positioning method. The technical features correspond one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0151] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described video motion positioning method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0152] This application provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the method described in the embodiment. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0155] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video motion localization method, characterized in that, The method includes: The video to be identified is divided into N video segments, where N is an integer greater than 1; Action recognition is performed on the N video segments to obtain the first action category probability for each of the N video segments; For each of the N video segments, determine the confidence level of the action category probability of each image frame in the video segment. The confidence level of the action category probability of the image frame represents the confidence level that the action category probability of the image frame is the first action category probability of the video segment. For each frame of the video to be identified, the second action category probability of the image is determined based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer. Based on the probability of the second action category in the image of the video to be identified, the temporal action localization result of the video to be identified is determined.

2. The method according to claim 1, characterized in that, The step of determining the temporal action localization result of the video to be identified based on the second action category probability of the images in the video to be identified includes: For each frame of the video to be identified, the action category probability value of the candidate action category in the second action category probability of the image is determined as the score of the image, wherein the candidate action category is the category with the highest probability in the second action category probability of the image; The candidate action category of the first target image in the video to be identified is updated to a preset category, and the score of the first target image is updated to the action category probability value of the preset category in the second action category probability of the first target image. The first target image is an image in the video to be identified whose action category probability value of the candidate action category is less than a preset probability threshold and whose candidate action category is not the preset category. Based on the updated candidate action categories and updated scores of the images in the video to be identified, the temporal action localization result of the video to be identified is determined.

3. The method according to claim 2, characterized in that, The step of determining the temporal action localization result of the video to be identified based on the updated candidate action categories and updated scores of the images in the video to be identified includes: Images that are adjacent and have the same action category in the video to be identified are merged into candidate sub-segments, and the second target image in the video to be identified is formed into a separate candidate sub-segment, resulting in K candidate sub-segments of the video to be identified. The action category of the second target image in the video to be identified is different from that of its adjacent images. The score of each candidate sub-segment is the average of the updated scores of the images in the candidate sub-segment. The action category of the candidate sub-segment is the action category of the images in the candidate sub-segment, and K is an integer greater than 1. For each of the K candidate sub-segments, if the length of the candidate sub-segment is less than or equal to a first length threshold, the action category of the candidate sub-segment is updated to the preset category, and the score of the candidate sub-segment is updated to the average of the action category probability values ​​of the preset category in the second action category probability of the image in the candidate sub-segment. Based on the updated action categories and updated scores of the K candidate sub-segments, the temporal action localization result of the video to be identified is determined.

4. The method according to claim 3, characterized in that, The determination of the temporal action localization result of the video to be identified based on the updated action category and updated score of the K candidate sub-segments includes: The candidate sub-segments that are adjacent and whose updated action categories are all the preset categories among the K candidate sub-segments are merged, and the remaining candidate sub-segments among the K candidate sub-segments are kept unchanged to obtain L first sub-segments. The score of the merged sub-segment among the L first sub-segments is the weighted sum of the updated scores of the merged sub-segments. The score of the candidate sub-segments among the L first sub-segments is the updated score of the candidate sub-segments. L is a positive integer less than or equal to K. The remaining candidate sub-segments are the remaining sub-segments among the K candidate sub-segments excluding the merged candidate sub-segments. For each of the m sub-segment pairs that meet the preset requirements among the L first sub-segments, the sub-segment pair is merged with one sub-segment between the sub-segment pairs to obtain a second sub-segment. The category of the second sub-segment is the action category of the sub-segment pair. The score of the second sub-segment is the average of the action category probability values ​​of the first action category in the second action category probability of the image in the second sub-segment. The first action category is the action category of the second sub-segment. The preset requirements include that the action categories of the two sub-segments in the sub-segment pair are the same and not the preset category, and the action category of one sub-segment between the sub-segment pairs is the preset category and the length of one sub-segment between the sub-segment pairs is less than or equal to a second length threshold. Based on the action categories and scores of m second sub-segments and the action categories and scores of other first sub-segments, the temporal action localization result of the video to be identified is determined, wherein the other first sub-segments are the unmerged sub-segments among the L first sub-segments.

5. The method according to any one of claims 1-4, characterized in that, Determining the probability confidence of the action category of each image frame in the video segment includes: The confidence level of the action category of the image frame in the video segment is determined based on the distance between the center position of the image frame and the center position of the video segment. The confidence level of the action category of the image frame in the video segment is inversely correlated with the distance between the image frame and the center position of the video segment. The center position of the video segment is determined based on the number of image frames in the video segment.

6. The method according to any one of claims 1-4, characterized in that, The determination of the second action category probability of the image based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments to which the image belongs includes: The ratio of the first value to the second value is taken as the second action category probability of the image. The second value is the sum of the confidence scores of the M action category probabilities of the image. The first value is the result of weighted summation of the first action category probabilities of the M video segments to which the image is located, based on the confidence scores of the M action category probabilities of the image.

7. A video motion positioning device, characterized in that, The device includes: The segmentation module is used to divide the video to be identified into N video segments, where N is an integer greater than 1; The recognition module is used to perform action recognition on the N video segments and obtain the first action category probability of each video segment in the N video segments; The first determining module is used to determine, for each of the N video segments, the action category probability confidence level of each image frame in the video segment within the video segment, wherein the action category probability confidence level of the image frame represents the confidence level that the action category probability of the image frame is the first action category probability of the video segment; The second determining module is used to determine the second action category probability of each frame of the image in the video to be identified, based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer. The third determining module is used to determine the temporal action localization result of the video to be identified based on the second action category probability of the image in the video to be identified.

8. An electronic device, characterized in that, Including transceivers and processors, The processor is used for: The video to be identified is divided into N video segments, where N is an integer greater than 1; Action recognition is performed on the N video segments to obtain the first action category probability for each of the N video segments; For each of the N video segments, determine the confidence level of the action category probability of each image frame in the video segment. The confidence level of the action category probability of the image frame represents the confidence level that the action category probability of the image frame is the first action category probability of the video segment. For each frame of the video to be identified, the second action category probability of the image is determined based on the confidence scores of the M action category probabilities of the image and the first action category probabilities of the M video segments in which the image is located. The confidence scores of the M action category probabilities include the confidence scores of the action category probabilities of the image in the M video segments, where M is a positive integer. Based on the probability of the second action category in the image of the video to be identified, the temporal action localization result of the video to be identified is determined.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

11. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-6.

Images