Action positioning method and device, electronic equipment and computer readable storage medium

Abstract

The application provides an action positioning method and device, electronic equipment and a computer readable storage medium, wherein the method comprises: obtaining at least one target video segment containing a target object in a video; obtaining a first action recognition result of at least one image frame in the target video segment and a second action recognition result of the target video segment; and obtaining an action positioning result of the video based on the first action recognition result and the second action recognition result. The step of obtaining the action recognition result can be realized by an artificial intelligence module. In the process of action positioning, the application does not need to obtain an optical flow image, has small memory capacity, small calculation amount, high accuracy, and a wider application range.

Description

Technical Field

[0001] This application relates to the fields of computer and artificial intelligence technology, and more specifically, to a motion localization method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Temporal action localization (TAL) detects the location and type of each action within a continuous, unsegmented video. In recent years, deep learning-based methods have achieved remarkable success in computer vision. Deep neural networks have also outperformed traditional methods in action localization, significantly advancing the field. However, existing action localization methods employ large and computationally intensive models, demanding significant memory and computing power, thus limiting their applicability. Therefore, it is necessary to improve existing action localization methods. Summary of the Invention

[0003] The purpose of this application is to at least solve one of the aforementioned technical defects. The technical solution provided by the embodiments of this application is as follows:

[0004] In a first aspect, embodiments of this application provide an action localization method, including:

[0005] Obtain at least one target video segment that contains the target object;

[0006] Acquire the first action recognition result of at least one image frame in the target video segment and the second action recognition result of the target video segment;

[0007] Based on the first action recognition result and the second action recognition result, the action localization result of the video is obtained.

[0008] In one optional embodiment of this application, obtaining at least one target video segment containing the target object includes:

[0009] Perform target object detection on at least one image frame in the video, and obtain at least one target video segment based on the target object detection results.

[0010] In one optional embodiment of this application, target object detection is performed on at least one image frame in the video, and at least one target video segment is obtained based on the target object detection result, including:

[0011] At least one first image frame is obtained from the video, and target object detection is performed on the obtained first image frame to obtain the corresponding target object detection result;

[0012] If the target object detection result indicates that the current first image frame obtained at the current time contains the target object, then the current first image frame, the previous first image frame relative to the current time, and the image frames between the current first image frame and the previous first image frame are cached. If the total number of cached image frames at the current time is not less than the first preset value, then the cached image frames at the current time are output as the corresponding target video segment.

[0013] If the target object detection result indicates that the current first image frame acquired at the current moment does not contain the target object, and if the total number of cached image frames at the current moment is not less than the second preset value, then the cached image frame at the current moment will be output as the corresponding target video segment.

[0014] In one optional embodiment of this application, the method further includes:

[0015] After outputting the corresponding target video segment, if the target object detection result indicates that the current first image frame obtained at the current moment contains the target object, then the cached image frames of the third preset value before the current first image frame are retained, and other cached image frames are cleared.

[0016] After outputting the corresponding target video segment, if the target object detection result indicates that the current first image frame obtained at the current moment does not contain the target object, then the cached image frame at the current moment is cleared.

[0017] In one optional embodiment of this application, obtaining a first action recognition result of at least one image frame in the target video segment and a second action recognition result of the target video segment includes:

[0018] At least one second image frame is obtained from the target video segment, and action recognition is performed on the obtained at least one second image frame to obtain the corresponding first action recognition result;

[0019] Based on the first action recognition result, obtain the second action recognition result.

[0020] In one optional embodiment of this application, action recognition is performed on the acquired second image frame to obtain the corresponding first action recognition result, including:

[0021] Extract spatial information corresponding to at least one second image frame, and obtain the corresponding input features based on the spatial information;

[0022] Using at least one temporal residual module, time information corresponding to at least one second image frame is extracted based on input features, and corresponding output features are obtained based on input features and time information;

[0023] Based on the output features, obtain the first action recognition result corresponding to at least one second image frame.

[0024] In one optional embodiment of this application, the temporal residual module includes a first branch and a second branch. Through at least one temporal residual module, it extracts temporal information corresponding to at least one second image frame based on input features, and obtains corresponding output features based on the input features and the temporal information, including:

[0025] Through the first branch, the weight information corresponding to the second image frame is obtained based on the input features, and then the first feature containing the first time information is obtained based on the weight information and the input features.

[0026] Through the second branch, feature information of adjacent second image frames corresponding to the second image frame is obtained based on the input features, and corresponding second features containing second time information are obtained based on the feature information of adjacent second image frames.

[0027] Based on the first feature and the second feature, the corresponding output features are obtained.

[0028] In one optional embodiment of this application, obtaining a second action recognition result based on a first action recognition result includes:

[0029] The first action recognition results of at least one second image frame are fused to obtain the second action recognition result.

[0030] In one optional embodiment of this application, the first action recognition result includes first probability values ​​of actions for each preset action type contained in the corresponding second image frame, and the second action recognition result includes second probability values ​​of actions for each preset action type contained in the corresponding target video segment. The first action recognition results of at least one second image frame are fused to obtain the second action recognition result, including:

[0031] The average value of the first probability value corresponding to each preset action type in at least one second image frame is obtained respectively;

[0032] Normalize the average value of the first probability value corresponding to each preset action type to obtain the probability value corresponding to each preset action type.

[0033] The probability values ​​of each preset action type are used as the second probability values ​​of each preset action type for the actions contained in the target video segment, thus obtaining the second action recognition result.

[0034] In one optional embodiment of this application, the first probability value is the non-normalized log probability Logits value.

[0035] In one optional embodiment of this application, obtaining the motion localization result of the video based on the first motion recognition result and the second motion recognition result includes:

[0036] Based on the first action recognition result and the second action recognition result, obtain the action localization result of the corresponding target video segment;

[0037] Based on the motion localization results of the target video segment, obtain the motion localization results of the video.

[0038] In one optional embodiment of this application, based on the first action recognition result and the second action recognition result, the action localization result of the corresponding target video segment is obtained, including:

[0039] Based on the first action recognition result and the second action recognition result, at least one candidate nomination is determined;

[0040] Based on each candidate nomination, the corresponding action range in the target video segment is identified to determine the target candidate nomination;

[0041] The action type and action range corresponding to the target candidate nomination are used as the action localization results of the target video segment.

[0042] In one optional embodiment of this application, at least one candidate nomination is determined based on the first action recognition result and the second action recognition result, including:

[0043] Based on the first action recognition result and the second action recognition result, determine the nomination threshold corresponding to at least one preset action type;

[0044] For each of the at least one preset action types, at least one candidate nomination is determined based on the corresponding nomination threshold and the first action recognition result.

[0045] In one optional embodiment of this application, for each of the at least one preset action types, at least one candidate nomination is determined based on the corresponding nomination threshold and the first action recognition result, including:

[0046] Each candidate second image frame is determined, and the first probability value of the action contained in the candidate second image frame being the preset action type is greater than the corresponding nomination threshold;

[0047] Based on each candidate second image frame, at least one candidate nomination is determined.

[0048] In one optional embodiment of this application, based on each candidate nomination, the corresponding action interval in the target video segment is identified to determine the target candidate nomination, including:

[0049] Obtain the video segment features corresponding to the target video segment;

[0050] Based on each candidate nomination, the video segment features corresponding to the action interval are determined, and the corresponding candidate features are obtained.

[0051] Each candidate feature is identified to determine the target candidate nomination.

[0052] In one optional embodiment of this application, based on each candidate nomination, video segment features corresponding to the action interval are determined to obtain corresponding candidate features, including:

[0053] Determine the video segment features corresponding to the action range of each candidate nomination;

[0054] The video segment features corresponding to the action interval of each candidate nomination are sampled into features of a predetermined length to obtain the corresponding candidate features.

[0055] In one optional embodiment of this application, based on each candidate nomination, the corresponding action interval in the target video segment is identified to determine the target candidate nomination, including:

[0056] Based on each candidate nomination, the corresponding action range in the target video segment is identified, and the action recognition score corresponding to each candidate nomination is obtained.

[0057] For each candidate nomination, the continuous action recognition score of the candidate nomination is determined based on the action recognition score and action range corresponding to the candidate nomination;

[0058] The candidate with the highest continuous action recognition score is selected as the target candidate.

[0059] In one optional embodiment of this application, based on the first action recognition result and the second action recognition result, the action localization result of the corresponding target video segment is obtained, including:

[0060] Based on the second action recognition result, the target action type in the target video segment is determined;

[0061] Based on the first action recognition result, obtain the target key image frame corresponding to the target action type;

[0062] Based on the target key image frames, obtain the target action range corresponding to the target action type;

[0063] The target action type and the corresponding target action range are used as the action localization result of the target video segment.

[0064] In one optional embodiment of this application, determining the target action type based on the second action recognition result includes:

[0065] The preset action type corresponding to the maximum second probability value in the second action recognition result is taken as the target action type.

[0066] In one optional embodiment of this application, based on the first action recognition result, obtaining the target key image frame corresponding to the target action type includes:

[0067] Based on the first action recognition result, obtain a first probability value corresponding to the target action type in at least one second image frame;

[0068] The second image frame with the highest first probability value corresponding to the target action type in at least one second image frame is taken as the target key image frame.

[0069] In one optional embodiment of this application, obtaining the target action range corresponding to the target action type based on the target key image frame includes:

[0070] Using at least one preset action range width as the window width, perform a sliding window operation on the target video segment to obtain at least one sliding window containing the target key image frame;

[0071] Based on the first probability value of the target action type corresponding to the second image frame in the sliding window, the score value of the sliding window is obtained, and the sliding window with the largest score value is determined as the target sliding window;

[0072] Based on the first probability value corresponding to each preset action type in the second image frame of the target sliding window, obtain the probability value corresponding to each preset action type in the target sliding window;

[0073] If the preset action type corresponding to the maximum probability value in the target sliding window is the target action type, then the action range corresponding to the target sliding window is determined as the target action range.

[0074] In one optional embodiment of this application, obtaining the motion localization result of the video based on the motion localization result of the target video segment includes:

[0075] If two adjacent target action intervals in the video correspond to the same target action type, and the difference between the end time of the earlier target action interval and the start time of the later target action interval is no greater than the third preset duration, then the two target action intervals are merged to obtain the motion localization result of the video.

[0076] In one optional embodiment of this application, merging the two target action ranges includes:

[0077] The start time of the earlier target action interval will be used as the start time of the merged target action interval, and the end time of the later target action interval will be used as the end time of the merged target action interval.

[0078] Secondly, embodiments of this application provide a positioning device, including:

[0079] The target video segment acquisition module is used to acquire at least one target video segment containing the target object.

[0080] The action recognition result acquisition module is used to acquire the first action recognition result of at least one image frame in the target video segment and the second action recognition result of the target video segment.

[0081] The motion localization result acquisition module is used to acquire motion localization results of the video based on the first motion recognition result and the second motion recognition result.

[0082] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor;

[0083] The memory contains computer programs;

[0084] A processor for executing computer programs to implement the methods provided in the first aspect embodiment or any alternative embodiment of the first aspect.

[0085] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the first aspect embodiment or any optional embodiment of the first aspect.

[0086] The beneficial effects of the technical solution provided in this application are:

[0087] After acquiring the target video segment containing the target object in the video to be located, the first action recognition result of multiple image frames in each target video segment and the second action recognition result of the target video segment are acquired. Then, based on each first action recognition result and second action recognition result, the action localization result of the video to be located is obtained. This scheme does not require the acquisition of optical flow images during the action localization process, requires less memory capacity, has less computation, and has high accuracy, making it more widely applicable. Attached Figure Description

[0088] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0089] Figure 1 A flowchart illustrating an action localization method provided in an embodiment of this application;

[0090] Figure 2 This is a flowchart illustrating an example of an action localization method in an embodiment of this application.

[0091] Figure 3 A schematic diagram illustrating the motion localization process of the input video in an embodiment of this application;

[0092] Figure 4 This is a flowchart illustrating a pedestrian detection process in one example of an embodiment of this application.

[0093] Figure 5 This is a flowchart illustrating the action recognition process in one example of an embodiment of this application.

[0094] Figure 6 This is a schematic diagram of the structure of a temporal residual network in one example of an embodiment of this application;

[0095] Figure 7 This is a schematic diagram of the structure of a timing residual block in one example of an embodiment of this application;

[0096] Figure 8a This is a schematic diagram illustrating the global branching effect in one example of an embodiment of this application;

[0097] Figure 8b This is a schematic diagram of an attention subnetwork in one example of an embodiment of this application;

[0098] Figure 8c This is a schematic diagram of a basketball playing action as an example of an embodiment of this application;

[0099] Figure 8d This is a schematic diagram illustrating the effect of a local branch in an example of an embodiment of this application;

[0100] Figure 8e This is a schematic diagram of a neighborhood sub-network in one example of an embodiment of this application;

[0101] Figure 8f This is a schematic diagram of the structure of a timing residual module in one example of an embodiment of this application;

[0102] Figure 8g This is a schematic diagram illustrating action recognition using a temporal residual network in one example of an embodiment of this application.

[0103] Figure 9 This is a flowchart illustrating the process of obtaining the second action recognition result of a target video segment in one example of an embodiment of this application;

[0104] Figure 10 This is a flowchart illustrating the process of obtaining motion localization results from a video in one example of an embodiment of this application.

[0105] Figure 11a This is a schematic diagram of an action recognition result according to an embodiment of this application;

[0106] Figure 11b This is a schematic diagram illustrating the use of a nomination threshold to obtain candidate nominations according to an embodiment of this application;

[0107] Figure 11c This is a flowchart illustrating an action positioning process according to an embodiment of this application.

[0108] Figure 12aThis is a schematic diagram illustrating the process of obtaining the target action type and target action range of a target video segment in one example of an embodiment of this application;

[0109] Figure 12b This is a schematic diagram of the time pyramid sliding window in one example of an embodiment of this application;

[0110] Figure 13a This is a schematic diagram illustrating the processing of original video image frames in one example of an embodiment of this application;

[0111] Figure 13b This is a schematic diagram illustrating the processing of original video image frames in one example of an embodiment of this application;

[0112] Figure 14 This is a schematic diagram illustrating the input video action localization process in one example of an embodiment of this application;

[0113] Figure 15 A structural block diagram of a positioning device provided in an embodiment of this application;

[0114] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0115] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0116] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0117] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0118] Temporal motion localization refers to obtaining the motion category of each motion and its corresponding start and end times from a given unsegmented, continuous long video containing multiple motions.

[0119] Traditional action localization methods typically obtain the category of each action through action recognition, and then obtain the start and end times of each action through temporal action proposal. Examples include action recognition methods based on TSM (Temporal Shift Module) and temporal action proposal methods based on DBG (Dense Boundary Generator).

[0120] TSM is a two-stream network, typically used for processing RGB and optical flow images. First, it extracts the optical flow sequence from the RGB image using a specific algorithm. Then, the image and optical flow are processed separately as two sub-branches of the network, and finally, the results are fused. TSM allows for the acquisition of temporal information even when using only RGB images for convolutional computation. This module allows for sacrificing the optical flow component in some scenarios to improve computational speed and achieve better real-time performance. In image convolutional neural networks, the output feature shape of each layer is typically (n, h, w, c), where n is the number of images in a batch, h is the feature height, w is the feature width, and c is the number of feature channels. In video action recognition problems, this shape is typically (n*t, h, w, c), where t is the number of images selected in a video. Taking n=1 as an example, the shape is simplified to (t, h*w, c). A certain number of channels are selected, and the data for those channels is shifted forward in the time dimension; an equal number of channels are selected, and the data for those channels is shifted backward in the time dimension. Then, the feature dimensions are restored, and the original convolutional operations are performed. Because the features have been shifted in the time dimension, some time-dimensional information is obtained in the original convolutional network.

[0121] The DBG method first extracts spatial and temporal features from RGB and optical flow images using an action recognition network. Then, it obtains action score features and dual-stream features using a DSB (Dual Stream BaseNet) network. The action score features are used to obtain an action confidence table via an ACR (Action-aware Completeness Regression) network, while the dual-stream features are obtained using a TBC (Temporal Boundary Classification) network to obtain start and end time confidence tables. These three tables are combined to obtain the intervals of candidate actions. Finally, a Soft NMS (Soft Non-maximum Suppression) method is used to filter out duplicate intervals to obtain the final action interval. This method achieves an AR@50 score of 40.89 on THUMOS14, which is still relatively low and difficult to apply to real-world products. Here, AR@50 means: AR stands for Average Recall, and 50 represents the average number of nominations per video. The overall meaning of AR@50 being 40.89 is that when a video is given 50 candidate intervals, it can find 40.89% of the correct intervals among all the correct intervals.

[0122] In summary, the above-mentioned motion localization methods have the following problems:

[0123] (1) In action recognition tasks, the TSM method based on 2D convolution obtains temporal information through time shifting, but the information obtained is limited because the temporal information obtained by TSM can only represent the changes between different frames, but cannot express the meaning of these changes. Temporal semantic information refers to a deeper level of temporal dimension information that can express the specific semantics contained in the changes between different image frames, such as expressing the motion state of "dancing". TSM ignores the impact of temporal semantic information on action recognition and localization. However, temporal semantic information is very important for videos, and the lack of temporal semantic information may lead to incorrect action localization results. For example, if the action category of a video is dancing, the action category of the first frame image is dancing, but the action category of the third frame image does not have obvious features for the action category of dancing. From the third frame image, the action is more similar to walking. If temporal semantic information can be extracted at this time, this temporal semantic information can focus on the motion state of the entire video, and thus the action recognition and localization can be accurately performed based on the temporal semantic information.

[0124] (2) In the temporal action nomination task, the DBG method has a low average recall rate and is difficult to apply to actual products.

[0125] (3) Current action recognition methods and temporal action nomination methods all require the use of optical flow data, which means that optical flow needs to be extracted when processing videos, resulting in low running speed and difficulty in ensuring real-time performance.

[0126] To address the aforementioned issues, this application provides an action localization method, which will be described in detail below.

[0127] Figure 1 This is a flowchart illustrating an action localization method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method may include:

[0128] Step S201: Obtain at least one target video segment containing the target object.

[0129] In this context, the target object refers to the initiator of the action in the action localization task, which can be a person, an animal, or other moving object. This application primarily uses the localization of human actions to illustrate the method provided, but it is not limited thereto. It is understood that a video clip may contain one or more target objects; for example, it could be pedestrian A and / or pedestrian B.

[0130] In this context, target video segments are those containing the target object. Motion localization of a video essentially involves localizing the motion of these target video segments, and then obtaining the localization result for the entire video based on the motion localization results of these target video segments. Therefore, it is necessary to first filter out these target video segments containing the target object from the video that needs to be localized.

[0131] Step S202: Obtain the first action recognition result of at least one image frame in the target video segment and the second action recognition result of the target video segment.

[0132] The target video segment contains multiple image frames. During the action recognition process, each image frame can have a corresponding action recognition result, i.e., the first action recognition result. Simultaneously, during the action recognition process, each target video segment as a whole can have a corresponding action recognition result, i.e., the second action recognition result.

[0133] Specifically, the action recognition results of some image frames in the target video segment are obtained respectively, that is, multiple first action recognition results are obtained, and based on these multiple first action recognition results, the second recognition result of the target video segment is obtained.

[0134] Step S203: Based on the first action recognition result and the second action recognition result, obtain the action localization result of the video.

[0135] Specifically, for each target video segment in the video, its motion localization result can be obtained based on multiple first motion recognition results corresponding to multiple image frames and its own second motion recognition result. Then, based on the motion localization results of multiple target video segments in the video, the motion localization result of the entire video is obtained.

[0136] The solution provided in this application, after obtaining the target video segment containing the target object in the video to be located, obtains the first action recognition result of multiple image frames in each target video segment and the second action recognition result of the target video segment. Then, based on each first action recognition result and the second action recognition result, the action localization result of the video to be located is obtained. This solution does not require the acquisition of optical flow images during the action localization process, requires less memory capacity, has less computation, and has high accuracy, thus having a wider range of applications.

[0137] like Figure 2 As shown, the specific implementation process of the action localization method provided in this application mainly includes the following tasks: target object detection, action recognition, and action localization. Specifically, as follows: Figure 3 As shown, the task of action recognition may include the steps of video representation + recognizer, and the task of action localization may include the steps of temporal action proposal + recognizer and fusion steps. The action localization method provided in this application specifically includes:

[0138] 1. Target object detection:

[0139] (1) The input is a video to be located. Figure 3 (Taking a sparsely sampled video frame as an example), the video contains at least one RGB image frame.

[0140] (2) Use an object detection module (such as a pedestrian detection module) to detect target objects in the video. If the target object is a pedestrian, select the time interval containing the pedestrian as the candidate interval for action recognition. This step is the target object detection, which is used as the basis for determining the target video segment. This step can avoid performing action localization on all video content, and only perform action localization on the video segment containing the target object, thus reducing the computational load of action localization.

[0141] (3) Divide the interval containing the target object (such as a pedestrian) into video segments every t seconds. If the interval is less than t seconds, divide it into a video segment according to the actual time length. In other words, select the video segment containing the target object (such as a pedestrian) as the target video segment.

[0142] Among them, two adjacent video segments can have overlapping frames, that is, the first few frames of the current video segment can be the same as the last few frames of the previous adjacent video segment, and the last few frames of the current video segment can be the same as the first few frames of the next adjacent video segment, so as to ensure the recognition and positioning effect of the image frames at the boundary. The specific number of overlapping frames can be set according to the actual situation, and is not limited here.

[0143] (4) For each target video segment, extract k image frames. If there are fewer than k images, copy the last image frame to make up the difference. It should be noted that these copied image frames need to be discarded in the subsequent motion localization stage.

[0144] The above steps can also be understood as: sampling video segments in the interval with a target object (such as a pedestrian) according to a fixed number of frames to obtain the target video segments for motion localization, where the fixed number of frames can be k (for example, k = 25).

[0145] Through the above target object detection steps, several target video segments can be obtained. Each target video segment contains consecutive video frames (also called image frames) with a target object (such as a pedestrian).

[0146] 2. Video representation + Recognizer

[0147] K image frames of a target video segment are input into an action recognition network containing a temporal residual module for action recognition. This process yields a first action recognition result for each of the k image frames and a second action recognition result for the target video segment. For example, this could involve obtaining k scores for the k frames and a score for the video segment, where each score represents the probability of an action. For instance, when performing action recognition on target video segments Δt1, Δt2, and Δt3, the action recognition results for Δt1 and Δt2 might show the highest probability of running, while the second action recognition result for Δt3 might show the highest probability of jumping.

[0148] The above steps can also be understood as follows: dividing each target video segment into fixed-length segments and performing action recognition to obtain the corresponding first action recognition result and second action recognition result. This step can obtain sufficient temporal semantic information through the Temporal Residual Module (TRM).

[0149] 3. Sequential Action Nomination + Recognizer

[0150] Based on the first action recognition result of each of the k image frames and the second action recognition result of the target video segment, the presence of action in the target video segment, the action range and action type are calculated using an adaptive multi-scale localization method or a temporal pyramid approach, thus obtaining the action localization result of the target video segment. For the input video, this step yields video segments containing a start time point, an end time point, and an action category (the duration of each video segment is less than a threshold), for example, obtaining the start time point t. 1s The end time point is t 1e The action category is running, and the starting time is t. 2s The end time point is t 2e The action category is running, and the starting time point is t. 3s The end time point is t 3e And three video segments with the action category of jumping. This step has low computation and good performance.

[0151] 4. Integration

[0152] The results from each target video segment are fused to obtain the action range and action category of the video to be localized, i.e., to obtain the action localization result of the video. This yields the final video segment containing the start time point, end time point, and action category. For example, fusing the three video segments in the example above yields the action localization result of the video at time t. 1s To t 2e The action range is for the action category of running, in t 3s To t 3e The action category for the action range is jump.

[0153] The following section will further explain the target object detection, action recognition, and action localization steps mentioned above.

[0154] In one optional embodiment of this application, obtaining at least one target video segment containing the target object includes:

[0155] Perform target object detection on at least one image frame in the video, and obtain at least one target video segment based on the target object detection results.

[0156] Specifically, target object detection is performed on multiple image frames in the video to be located. Based on the detection results, it is determined which video segments contain the target object and these video segments are identified as target video segments.

[0157] In one optional embodiment of this application, target object detection is performed on at least one image frame in the video, and at least one target video segment is obtained based on the target object detection result, including:

[0158] At least one first image frame is obtained from the video, and target object detection is performed on the obtained first image frame to obtain the corresponding target object detection result;

[0159] If the target object detection result indicates that the current first image frame obtained at the current time contains the target object, then the current first image frame, the previous first image frame relative to the current time, and the image frames between the current first image frame and the previous first image frame are cached. If the total number of cached image frames at the current time is not less than the first preset value, then the cached image frames at the current time are output as the corresponding target video segment.

[0160] If the target object detection result indicates that the current first image frame acquired at the current moment does not contain the target object, and if the total number of cached image frames at the current moment is not less than the second preset value, then the cached image frame at the current moment will be output as the corresponding target video segment.

[0161] It is understandable that when performing target object detection on a video to be located, target object detection can be performed on all image frames in the video, and the target video segment can be determined based on the target object detection results. However, to reduce the number of target object detections and improve computational efficiency, instead of performing full detection on all image frames in the video, a subset of image frames (the first image frame) can be extracted for target object detection. For example, the first image frame can be obtained from the video at a first preset time interval; in other words, image frames (the first image frames) for target object detection are sampled from the video at equal time intervals (i.e., the first preset time interval). The first preset time interval can be set according to actual needs, for example, it can be set to 1 second. Alternatively, multiple first image frames can be randomly obtained from the video; in other words, first image frames for target object detection are sampled from the video at non-equal time intervals. The interval duration during the sampling process can be set according to actual needs. After sampling, target object detection is performed on the obtained first image frames, and the time of each sampling can be called the sampling time. Since the action of the target object can last for a certain period of time, multiple consecutive image frames will contain the target object. Therefore, based on the target detection result of an image frame, it is possible to predict whether its adjacent image frames contain the target object.

[0162] Specifically, for the current image frame acquired at the current moment (or the current sampling moment), target object detection is performed. If the target object detection result indicates that it contains a target object, then the current first image frame corresponding to the current moment, the previous first image frame corresponding to the previous sampling moment, and the image frames between the two sampling moments are all considered to contain the target object, and these image frames are cached. Then, it is further determined whether the total number of cached image frames at the current moment is greater than or equal to a first preset value (i.e., whether the duration of the video segment composed of the cached image frames at the current moment is greater than or equal to a certain preset duration). If it is greater than or equal to the first preset value, the cached image frames at the current moment are output as the corresponding target video segment; if it is less than the first preset value, the target object detection and cached image frame count determination for the next sampling moment are continued. At the same time, in this case, after outputting the corresponding target video segment, all cached image frames except those within the third preset value before the current first image frame are cleared. The cached image frames within the third preset value are retained to ensure that there is overlap between two adjacent target video segments, reducing the impact of subsequent action positioning errors on the fusion of target action intervals.

[0163] If target object detection is performed on the current image frame acquired at the current moment, and the target object detection result indicates that it does not contain a target object, then it is assumed that the current first image frame corresponding to the current moment, the previous first image frame corresponding to the previous sampling moment, and the image frames between the two sampling moments do not contain a target object. Then, it is further determined whether the total number of cached image frames at the current moment is greater than or equal to a second preset value. If it is greater than or equal to the second preset value, the cached image frame at the current moment is output as the corresponding target video segment; if it is less than the second preset value, the target object detection and cached image frame count determination for the next sampling moment are continued. It is understandable that the second preset value can be set to 0. That is, in this case, if a cached image frame exists at the current moment, the cached image frame at the current moment is output as the target video segment; if no cached image frame exists at the current moment, the target object detection and cached image frame count determination for the next sampling moment are performed. Simultaneously, in this case, the cached image frame at the current moment is cleared after outputting the corresponding target video segment.

[0164] The following example, using pedestrians as the target, illustrates the above process. Figure 4 As shown, the following steps may be included:

[0165] (1) Decode the video to be located, sample and obtain the first image frame every t seconds in time sequence (i.e. the first preset duration, one optional value is t=1), and input a pre-processed first image frame into the pedestrian detection module.

[0166] (2) Perform pedestrian detection on the video, that is, detect whether there are pedestrians in the first image frame. One available pedestrian detection method is SSD (Single Shot MultiBox Detector).

[0167] (3) Determine whether the first image frame at the current time contains pedestrians. If it does, then set t i-1 seconds (previous image acquisition time) to t i All RGB images between the current time (or the current image acquisition time) are added to the buffer.

[0168] (4) Further determine whether the length of the cached image frame at the current moment has reached t. n seconds (an optional value is t) n =8), if the target video segment is reached, the current cached image frame is output as the target video segment, and all cached image frames except for the last 1 second are cleared; if the target video segment is not reached, the system waits for the next first image frame for pedestrian detection. The above determination of whether the length of the current cached image frame reaches t... n The seconds can be used to determine the cache length.

[0169] (5) If the first image frame at the current moment does not contain a pedestrian, then determine whether there is a cached image frame at the current moment. If there is a cached image frame, then output the cached image frame at the current moment as the target video segment and clear the cached image frame; if there is no cached image frame, then wait for the next first image frame to perform pedestrian detection. The above determination of whether there is a cached image frame at the current moment can be called cache judgment.

[0170] It should be noted that the processing of the cached image frames in steps (4) and (5) above after outputting the target video segment can be collectively referred to as clearing the cache.

[0171] In one optional embodiment of this application, obtaining a first action recognition result of at least one image frame in the target video segment and a second action recognition result of the target video segment includes:

[0172] At least one second image frame is obtained from the target video segment, and action recognition is performed on the obtained at least one second image frame to obtain the corresponding first action recognition result;

[0173] Based on the first action recognition result, obtain the second action recognition result.

[0174] Specifically, for each target video segment, similar to the target object detection process, computational efficiency is improved by extracting a portion of image frames (i.e., second image frames). For example, multiple second image frames can be obtained by sampling the target video segment at equal time intervals, i.e., obtaining a second image frame from the target video segment at a second preset time interval; alternatively, multiple second image frames can be obtained by sampling the target video segment at non-equal time intervals, where the second preset time interval and the interval between non-equal time interval sampling can be set according to actual needs. After obtaining the first action recognition result for each second image frame, the second action recognition result for the target video segment can be obtained based on the multiple first action recognition results.

[0175] For example, such as Figure 5 As shown, the specific process of the above action recognition can include the following steps:

[0176] (1) For each target video segment, it contains n image frames, where n is at least 1.

[0177] (2) Sample k second image frames at equal time intervals (i.e., the second preset duration), where the value of k is determined by the input size of the action recognition network (i.e., the sequential residual network). An optional value for k is 25. If k > n, copy the last image frame in the target video segment until the number of sampled frames reaches k, and record the number of copied images x.

[0178] (3) Input k second image frames into the action recognition network to obtain the first action recognition result for each second image frame. Specifically, the input consists of k preprocessed second image frames with dimensions (k, h, w, c), where k is the number of second image frames, h is the height of the second image frame, w is the width of the second image frame, and c is the number of channels in the RGB image, for example, c = 3. After passing through the temporal residual network, the first action recognition result for each second image frame can be obtained. The dimensions of the first action recognition result are (k, m+1), where k is the number of second image frames, and m+1 is the number of action types, i.e., including m preset action types and 1 other action type (the other action type can be an unrecognizable action type or a stationary action (i.e., no action)).

[0179] (4) The first action recognition results of each second image frame are fused and processed to obtain the second action recognition result of the corresponding target video segment, wherein the second action recognition result is a one-dimensional vector with a length of m+1.

[0180] In one optional embodiment of this application, action recognition is performed on the acquired second image frame to obtain the corresponding first action recognition result, including:

[0181] Extract spatial information corresponding to at least one second image frame, and obtain the corresponding input features based on the spatial information;

[0182] Using at least one temporal residual module, time information corresponding to at least one second image frame is extracted based on input features, and corresponding output features are obtained based on input features and time information;

[0183] Based on the output features, obtain the first action recognition result corresponding to at least one second image frame.

[0184] Specifically, multiple acquired second image frames can be input into a pre-trained temporal residual network, which can extract spatial information corresponding to the multiple second image frames. This temporal residual network includes a temporal residual module. Input features containing the aforementioned spatial information are input into the temporal residual module, which can extract first and second temporal information of the second image frames, and further obtain first features containing the first temporal information and second features containing the second temporal information. Finally, based on the first and second features, corresponding output features are obtained. Then, the temporal residual network outputs the first action recognition result corresponding to each second image frame based on these output features.

[0185] For example, a temporal residual network structure such as Figure 6 As shown, it includes a backbone network and a temporal residual block (TRblock). Specifically, the backbone network can be MobileNetv2, and the corresponding temporal residual network structure details are shown in Table 1, where t is the dilation coefficient in the temporal residual block, c is the number of output feature channels, n is the number of temporal residual blocks, and s is the stride value of the first temporal residual block.

[0186] Table 1

[0187] <![CDATA[224 2 ×3]]> 2D Convolution - 32 1 2 <![CDATA[112 2 ×32]]> Timing residual block 1 16 1 1 <![CDATA[112 2 ×16]]> Timing residual block 6 24 2 2 <![CDATA[56 2 ×24]]> Timing residual block 6 32 3 2 <![CDATA[28 2 ×32]]> Timing residual block 6 64 4 2 <![CDATA[14 2 ×64]]> Timing residual block 6 96 3 1 <![CDATA[14 2 ×96]]> Timing residual block 6 160 3 2 <![CDATA[7 2 ×160]]> Timing residual block 6 320 1 1 <![CDATA[7 2 ×320]]> 2D convolution 1×1 - 1280 1 1 <![CDATA[7 2 ×1280]]> Average pooling 7×7 - - 1 - 1×1×1280 2D convolution 1×1 - m+1 -

[0188] The structure of the time-series residual block is as follows: Figure 7As shown, if the shape of the input features of a temporal residual block is the same as the shape of the output features (i.e., its stride = 1), a temporal residual module (TRM) is added to the temporal residual block, and it can be called a first-type temporal residual block. If the shape of the input features of a temporal residual block is different from the shape of the output features (i.e., its stride = 2), then a temporal residual block does not include a TRM and can be called a second-type temporal residual block. Specifically, the second-type temporal residual block can be used to extract spatial information from multiple second image frames and output features containing spatial information. The features containing spatial information output by the second-type temporal residual block are used as the input features of the first-type temporal residual block. The TRM in the first-type temporal residual block further extracts temporal information from multiple second image frames based on the spatial information in the input features, and then outputs output features containing both spatial and temporal information. It should be noted that, as can be seen from Table 1, the first type of temporal residual block can use the output features containing spatial information of the second type of residual block as input features, or it can use the output features containing spatial information of the ordinary convolutional layer as input features.

[0189] Specifically, refer to again Figure 7 The initial input features of the first type of temporal residual block are first processed by the temporal residual module to extract corresponding output features containing first and second temporal information. These output features are then sequentially processed by a 1×1 convolutional kernel, a 3×3 depthwise convolutional kernel, and a 1×1 convolutional kernel to extract corresponding spatial information. Thus, the output features at this point contain both the temporal information extracted by the temporal residual module and the spatial information extracted by subsequent convolutions. This output feature, containing both temporal and spatial information, is then fused with the initial input features to obtain the final output feature of the first type of temporal residual module. The initial input features of the second type of temporal residual block are sequentially processed by a 1×1 convolutional kernel, a 3×3 depthwise convolutional kernel (with a stride of 2), and a 1×1 convolutional kernel to extract corresponding spatial information. The output is a final feature containing only the corresponding spatial information.

[0190] In one optional embodiment of this application, the temporal residual module includes two parallel branches. Through at least one temporal residual module, it extracts temporal information corresponding to at least one second image frame based on input features, and obtains corresponding output features based on the input features and temporal information, including:

[0191] Through the first branch, attention information corresponding to the second image frame is obtained based on the input features, and then the first feature containing the first time information is obtained based on the attention information and the input features.

[0192] Through the second branch, feature information of adjacent second image frames corresponding to the second image frame is obtained based on the input features, and second features containing second time information are obtained based on the feature information of adjacent second image frames.

[0193] Based on the first feature and the second feature, the corresponding output features are obtained.

[0194] Specifically, the timing residual module contains two branches. The first branch can also be called the global branch, and the second branch can also be called the local branch. These two branches can be set in parallel or in sequence, with the aim of obtaining more complex temporal semantic information.

[0195] The global branch can employ an attention sub-network. Through pooling operations, it treats the features of each second image frame as a whole, considering the importance of each second image frame's features to the overall picture. This attention sub-network simultaneously collects key semantic and background features, allowing the model to determine which frames are more effective for action recognition, while other frames with less significant features are weakened. Its intuitive effect is as follows: Figure 8a As shown, this network obtains the importance of each frame, which, when applied to the original image, means that the attention level of each frame changes. For example... Figure 8a In the image, the second frame from the left is of higher importance and therefore receives more attention, while the fourth frame is of the lowest importance and receives less attention, and so on for the other frames.

[0196] Specifically, an example of an attention subnetwork is as follows: Figure 8b As shown, feature maps containing low-level spatial and temporal information are taken as input, and first max pooling and average pooling are performed. Max pooling can extract key semantic information for action recognition. For example, Figure 8c In the diagram, frames 1 through 4 represent dribbling, shooting, shooting, and returning to the ground, respectively. The content within each frame represents key semantic features that aid in recognition. Frame a1 is a key semantic feature of frame 1. In frame 2, frame a2 represents the person shooting, so the basketball hoop in frame b2 has better semantic features. Frame a3 in frame 3 is similar to frame b3. Frame a4 in frame 4 also only represents the person shooting, with weaker semantic features. Average pooling can preserve global information; for example, not every frame is equally important for action recognition. Background information in some frames helps with action recognition; for instance, the basketball hoop, as background information, helps identify the action of playing basketball. Figure 8cThe basketball hoop in the background is crucial information for identifying basketball. The results of max pooling and average pooling are concatenated and then input into a global temporal feature extractor. This extractor extracts temporal information using one-dimensional convolution in a richer feature space and activates it with an activation function. This allows for the acquisition of semantic information for each frame from a global perspective with less computation. Subsequently, an attention decoder calculates the importance of each frame based on the temporal information and assigns it a value from 0 to 1, for example... Figure 8c The values ​​for each frame are 0.6, 0.95, 0.88, and 0.4. The importance of each frame is fused into the feature map by multiplying the importance value of each frame by the corresponding feature map, thus outputting a feature map containing the attention information for each frame.

[0197] Local branches can employ neighborhood subnetworks, obtaining temporal information through convolution in the temporal dimension. This means acquiring the influence of adjacent frames on the current frame from a specific temporal perspective, specifically considering the relationship between the features of each second image frame and the features of its adjacent second image frames. Information from adjacent frames helps identify the operation in the current frame. For example, if the action category in a video is dancing, the third frame might not have obvious features for this action category; simply seeing the action as walking doesn't definitively determine if it's dancing. However, with the continuous information from four adjacent frames, it becomes clear that it's dancing. Therefore, increasing the temporal receptive field and acquiring more temporal information is crucial. Its intuitive effect is as follows: Figure 8d As shown, after the input features pass through the neighborhood sub-network, the third frame image obtains information from the second and fourth frames. Passing through the neighborhood sub-network again, since the second frame has received information from the first frame, and the fourth frame has received information from the fifth frame, the third frame indirectly obtains information from the first and fifth frames while simultaneously receiving information from the second and fourth frames. For example, if a person is running between the first and fifth frames, the information contained is a, b, c, d, and e (a, b, c, d, and e refer to the same person). After passing through the neighborhood sub-network once, the third frame image obtains information b from the second frame and information d from the fourth frame. At this point, the third frame image can contain information b, c, and d, while the first frame contains information a, b, and c, and the fourth frame contains information c, d, and e. Passing through the neighborhood sub-network again, the third frame again obtains information from the second and fourth frames, and at this point, the third frame image can contain information a, b, c, d, and e, which helps to identify a person's actions from a temporal perspective.

[0198] Specifically, an example of a neighborhood subnetwork is as follows: Figure 8eAs shown, feature maps containing low-level spatial and temporal information are taken as input and encoded into a high-dimensional space by a feature encoder to extract richer temporal semantic information. The original feature maps contain information such as color and texture, while using higher dimensions allows for the acquisition of temporal semantic information such as orientation, velocity, and rotation. Next, a local temporal feature extractor performs convolution in the temporal dimension, along with batch normalization (BN) and activation function operations. The local temporal feature extractor aims to extract temporal information from adjacent frames. Each operation can extract more features from the previous and next frames, increasing the temporal receptive field of motion and addressing the problem of insufficient temporal semantic information. This is because for some actions, temporal information is more important than spatial information; that is, the neighborhood sub-network focuses on acquiring motion state. For example, for... Figure 8d In frame 3, the man is moving. Using a local temporal feature extractor, frame 3 will extract information from frames 2 and 4 to determine his motion state. In one embodiment, the temporal receptive field can be represented as Δt = 2*n + 1, where n represents the number of neighboring subnetworks. Finally, a feature decoder decodes and integrates the features, outputting a feature map containing information from adjacent frames.

[0199] For example, the structure of a timing residual module is as follows: Figure 8f As shown, the size of the input feature is (n*t, h, w, c), where n is the number of batches (an optional value of 128 during training, and n=1 during application), t is the feature corresponding to the t second image frames in a target video segment, h is the feature height, w is the feature width, and c is the number of feature channels. The processing flow of the global branch and the local branch in the figure are explained below.

[0200] The processing flow in a global branch can specifically include the following steps:

[0201] (1) Transform the dimensions of the input features into (n, t, h, w, c), and then perform average pooling φ1 and max pooling φ2 on the input features in the spatial and channel dimensions. The operations φ1 and φ2 are as follows:

[0202]

[0203]

[0204] The feature dimensions of the outputs φ1 and φ2 are (n, t, 1).

[0205] (2) The two pooling results are concatenated together in dimension c, and the output feature size is (n, t, 2).

[0206] (3) Perform a one-dimensional convolution in the t dimension with a kernel length of k (one optional parameter is 5) and an expansion coefficient of β (one optional parameter is 4). Increase the number of channels by β times and output the feature size as (n, t, 2β). At the same time, use the ReLU activation function for activation.

[0207] (4) Use one-dimensional convolution to reduce the number of channels to 1, the kernel length is 1 (k=1), and the output feature shape is (n, t).

[0208] (5) Use the Sigmoid function to normalize the result to the range of 0 to 1. The output feature size is (n*t, 1, 1, 1). This output feature can be understood as the weight of the time information corresponding to each second image frame in each batch (i.e. t second image frames) in the action recognition process. This weight can also be called the attention information corresponding to each second image frame.

[0209] (6) Multiply the result of (5) with the original input features to obtain the output features of the global branch, i.e. the corresponding first feature, the size of the first feature is (n*t, h, w, c).

[0210] The overall calculation formula is as follows:

[0211] L(X)=Sigmoid(Conv1D(δ(Conv1D(Concat(X 1t ,X 2t ),K,2β)),1,1))

[0212] Where L(X) is the first feature of the global branch output, and Concat(X) 1t ,X 2t ) represents combining two features X 1t and X 2t The connection is made by Conv1D(A,B,C), which means that the input feature A is convolved by a one-dimensional convolution kernel of size B, and the number of output channels is C. δ represents the ReLU activation function and Sigmoid represents the Sigmoid function.

[0213] The process of handling local branches can specifically include the following steps:

[0214] (1) Transform the size of the input feature to (n, t, h*w, c)

[0215] (2) Expand the number of channels to γ ​​times using a 1×1 convolutional kernel (an optional parameter is 6), and perform batch normalization (BN) and ReLU activation operations. The corresponding formula expression for this step is as follows:

[0216] X′=δ(PW(X,(1,1),γC))=δ(BN(Conv2D(X,(1,1),γC))

[0217] Wherein, PW(A,B,C) represents the convolution of input feature A with a two-dimensional convolution kernel of size B and then batch normalization, with C output channels, Conv2D(A,B,C) represents the convolution of input feature A with a two-dimensional convolution kernel of size B and C output channels, and X′ is the output feature of this process.

[0218] (3) Perform depthwise convolution using a 3×1 convolution kernel in the time dimension to enable information exchange between adjacent second image frames, thus allowing each second image frame to acquire the feature information of its neighboring second image frames. Simultaneously, perform BN and ReLU activation operations. The corresponding formula expression for this step is as follows:

[0219] X′=δ(DW(X,(3,1),γC))=δ(BN(DepthConv2D(X,(3,1),γC))

[0220] Where DW(A,B,C) means that the input feature A is processed by depthwise convolution using a two-dimensional convolution kernel of size B and then batch normalized, and the number of output channels is C. DepthConv2D(A,B,C) means that the input feature A is processed by depthwise convolution using a two-dimensional convolution kernel of size B, and the number of output channels is C. X′ is the output feature of this step.

[0221] (4) Use a 1×1 convolution kernel to restore the number of channels to c, and perform BN operation. The formula expression for this step is as follows:

[0222] X′=PW(X,(1,1),C)=BN(Conv2D(X,(1,1),C)

[0223] (5) Transform the result into (n*t, h, w, c) with the same shape as the input feature, and output the second feature. The overall calculation formula is as follows:

[0224] G(X)=PW(δ(DW(δ(PW(X,(1,1),γG)),(3,1),γC)),(1,1),C)

[0225] Here, G(X) is the second feature of the local branch output.

[0226] Finally, the results of the two branches are added together according to their respective positions to obtain the TRM output, as shown in the following formula:

[0227] Z = L(X) + G(X)

[0228] In the embodiments of this application, Figure 8g This paper demonstrates the complete process of action recognition using temporal residual networks. Compared to temporal motion networks, temporal residual networks can address the problem of insufficient temporal semantic information. Specifically, for example... Figure 8g As shown in the left part, n frames (the second image frames) are sampled from the video as input, and after passing through convolution, multiple temporal residual blocks, and integration operations in the temporal residual network, the first action recognition result and the second action recognition result are output. Each temporal residual block is as follows: Figure 8g As shown in the middle section, semantic information can be obtained from frames and feature maps. Specifically, for each temporal residual block, temporal information can be obtained through the temporal residual module in the block, and spatial information can be obtained through the inverse residual block in the block, thereby obtaining high-level spatial and temporal feature maps (also known as high-level semantic feature maps). Further, the temporal residual module in each temporal residual block is as follows: Figure 8g As shown on the right, it includes an attention network and a neighborhood sub-network, which can respectively obtain the importance information of the frame and the information of adjacent frames. The results of the two networks are added together to obtain the output of the temporal residual module, which can transform the low-level spatial and temporal feature maps into high-level temporal feature maps.

[0229] In this embodiment, multiple temporal residual modules can be used to extract time information corresponding to at least one second image frame based on input features, and to obtain corresponding output features based on input features and time information.

[0230] Generally, using multiple temporal residual modules expands the temporal and spatial receptive fields, allowing for the extraction of higher-level temporal features.

[0231] Expanding the temporal receptive field means being able to see a wider range in the time dimension. A larger temporal receptive field focuses more on the continuity of an action. Taking a video as an example, assuming the input is a video frame at time point A, when using a block with a temporal residual module for the first forward pass, a slightly wider range of content can be seen in the time dimension, that is, video frames covering a larger time range (the first range) including time point A can be seen. When using a block with a temporal residual module for the second forward pass, a wider range of content can be seen in the time dimension than when the block was used for the first time, that is, video frames covering a larger time range including the first range can be seen.

[0232] Expanding the spatial receptive field, or the ability to see more content at a spatial scale, helps the model understand what's in an image at a semantic level. Take a frame as an example: a frame contains a person shooting a basketball. Using a block with a temporal residual module, you might only see a small portion of the person's upper body, making it difficult to understand what that part actually is. Using two blocks with temporal residual modules provides a larger area, potentially showing a small portion of the upper and lower body—perhaps resembling parts of a person—but without certainty. Using four blocks with temporal residual modules provides an even larger area, potentially showing the entire person, making it easier to determine that the frame contains a person.

[0233] More blocks with temporal residual modules can extract higher-level features. Each block can output features of a higher level than the input features. The following example uses two blocks, but more blocks will actually be used. The image contains a basketball hoop. Block 1's input is the features of the upper left corner of the backboard, which has a relatively low feature level, containing only color information. Block 1's output and Block 2's input are the features of half the backboard, including information such as the backboard's pattern. Here, Block 1 can understand not only color features but also line features, resulting in a higher information level output. Block 2's output is the features of the entire backboard and rim in the image, understanding not only color and line features but also positional information. At this point, the model, through these two blocks, understands that the image contains a basketball hoop, achieving a higher information level. Semantic information has been extracted, but the location of the hoop and its relationship to surrounding objects are still unknown. If higher-level information is needed, more blocks are required.

[0234] In this embodiment, convolutional kernels are used to extract temporal information. By controlling the size of the convolutional kernels, information in the corresponding time dimension is extracted to expand the temporal receptive field. For example, a convolutional kernel with a size of 3*1*c is used at a specific location. in (Number of input channels) * c out (Number of output channels) means the temporal receptive field of a block is 3. Spatial information is extracted using convolutional kernels, and the size of the kernels is controlled to extract information corresponding to the space. For example, a kernel size of 3*3*c can be used at a specific location. in *c out The spatial receptive field is then 3*3 (one representing length and one representing width). High-level features are extracted using convolution and residual operations, increasing the number of channels in the convolution kernel to enhance feature richness. For example, a convolution kernel with shape w*h*c can be used at the very beginning. in *1 is used to extract color features. Multiple blocks are stacked to extract higher-level features.

[0235] In this embodiment of the application, sufficient temporal semantic information can be obtained through at least one temporal residual module to improve the accuracy of subsequent action recognition and action localization.

[0236] In one optional embodiment of this application, obtaining a second action recognition result based on a first action recognition result includes:

[0237] The first action recognition results of at least one second image frame are fused to obtain the second action recognition result.

[0238] In one optional embodiment of this application, the first action recognition result includes first probability values ​​of actions of different preset action types contained in the corresponding second image frame, and the second action recognition result includes second probability values ​​of actions of various preset action types contained in the corresponding target video segment. The first action recognition results of at least one second image frame are fused to obtain the second action recognition result, including:

[0239] The average value of the first probability value corresponding to each preset action type in at least one second image frame is obtained respectively;

[0240] Normalize the average value of the first probability value corresponding to each preset action type to obtain the probability value corresponding to each preset action type.

[0241] The probability values ​​of each preset action type are used as the second probability values ​​of each preset action type for the actions contained in the target video segment, thus obtaining the second action recognition result.

[0242] The first probability value mentioned above can be the non-normalized log probability Logits value.

[0243] Specifically, the size of the first action recognition result is (n*t, m+1), where n is the number of target video segments in a batch (an optional value of 128 during training, n=1 during application), t is the number of second image frames in a target video segment, and m+1 is the number of recognizable action types m plus other action types, i.e., the number of preset action types. Then, as... Figure 9 As shown, fusing the first action recognition results of at least one second image frame to obtain the second action recognition result may specifically include the following steps:

[0244] (1) The size of each first action recognition result is (n*t, m+1). Its size is transformed to (n, t, m+1). Then, average pooling is performed on each first action recognition result in the t dimension to obtain the pooling result. The formula expression for this step is as follows:

[0245]

[0246] Where, xi,m+1 It is the first probability value (which can be the Logits value) corresponding to each preset action type in the i-th second image frame.

[0247] (2) For the pooling result obtained in (1), whose size is (n, m+1), the result is constrained to between 0 and 1 by the softmax function to obtain the second action recognition result of the target video segment. The formula expression for this step is as follows:

[0248]

[0249] It should be noted that the temporal residual network can be pre-trained based on training samples to optimize network parameters. The trained model can then be used for action recognition. During training, the training samples are video clips containing only one action, labeled with the action type number (e.g., eating: 1, running: 2, and so on). Other types are categorized as 0; samples not recognizable by the temporal residual network are labeled as "other samples." The loss function used during training is:

[0250]

[0251] Where, for a target video segment sample, L is the loss value corresponding to that target video segment sample, j represents the action type index, m is the maximum action type index of that target video segment, and g j This represents the true label of the target video segment (if the target video segment corresponds to the j-th action type, then g). j =1, and other g j =0, for example, assuming the action type index j = 1, 2, then if the action type index corresponding to the target video segment sample is 1, then the target video segment sample g1 = 1, g2 = 0), z j This represents the predicted value of the temporal residual network, i.e., the probability that the action type corresponding to the target video segment sample is the j-th action type.

[0252] In one optional embodiment of this application, obtaining the motion localization result of the video based on the first motion recognition result and the second motion recognition result includes:

[0253] Based on the first action recognition result and the second action recognition result, obtain the action localization result of the corresponding target video segment;

[0254] Based on the motion localization results of the target video segment, obtain the motion localization results of the video.

[0255] Specifically, after obtaining the first action recognition results and the second action recognition results of multiple second image frames in each target video segment using the aforementioned method, the corresponding action localization results are obtained based on these recognition results. Then, the action localization results of each target video segment are fused to obtain the video's action localization result. For example... Figure 10 As shown, obtaining motion localization results from a video can include the following steps:

[0256] (1) Input the first action recognition result and the second action recognition result corresponding to each target video segment.

[0257] (2) Obtain the action type and target action range in the corresponding target video segment based on the first action recognition result and the second action recognition result.

[0258] (3) Integrate the motion localization results of all target video segments according to motion type and target motion range, and merge adjacent and similar motions into the same motion range.

[0259] (4) Output the category of each action and the start and end times of the action.

[0260] In one feasible implementation, based on the first action recognition result and the second action recognition result, the action localization result of the corresponding target video segment is obtained, including:

[0261] Based on the first action recognition result and the second action recognition result, at least one candidate nomination is determined. The candidate nomination is used to indicate the action range of any preset action type in the target video segment.

[0262] Based on each candidate nomination, the corresponding action range in the target video segment is identified to determine the target candidate nomination;

[0263] The action type and action range corresponding to the target candidate nomination are used as the action localization results of the target video segment.

[0264] Specifically, the target video segment recognition result includes the first action recognition result of each second image frame in the video segment and the second action recognition result of the target video segment. An adaptive nomination generator generates a smaller number of more accurate candidate nominations at different time scales (i.e., a series of sub-time intervals) based on the target video segment recognition result. Any preset action type may correspond to one or more candidate nominations. Within the time range of each candidate nomination, features corresponding to the target video segment are extracted and recognized, ultimately yielding the recognition result. This method can improve the accuracy and continuity of action localization and reduce computation.

[0265] In this embodiment of the application, the step of determining at least one candidate nomination based on the first action recognition result and the second action recognition result can be implemented in the following way:

[0266] Based on the first action recognition result and the second action recognition result, determine the nomination threshold corresponding to at least one preset action type;

[0267] For each of the at least one preset action types, at least one candidate nomination is determined based on the corresponding nomination threshold and the first action recognition result.

[0268] Specifically, based on the first action recognition result and the second action recognition result, each nomination threshold is adaptively generated. As an example, Figure 11a An example is shown of the first action recognition result (score per frame) and the second action recognition result (score per sub-video segment, where the sub-video segment can be understood as the aforementioned target video segment) for a target video segment. The following example... Figure 11a Taking the preset action type "jump" as an example, this explains the method for generating the corresponding nomination threshold.

[0269] like Figure 11a As shown, the average score (sub-video segment score) for the jump is 0.76, and the highest score among all frames of the jump is 0.95. In this embodiment, the formula for calculating the nomination threshold is:

[0270] s1 = min(0.5, average score)

[0271] s2 = max score

[0272] In the formula, the Average score is the average score of the jump, which is 0.76, and the Max score is the highest score in each frame, which is 0.95.

[0273] The adaptive nomination threshold is calculated using s1 and s2. The number of adaptive nomination thresholds can be determined according to requirements, and this embodiment does not limit the number. The specific calculation formula is as follows:

[0274]

[0275] Where, τ i Let represent the i-th threshold, and n represent the number of nomination thresholds. Taking n=3 as an example, the calculated nomination threshold for a jump is:

[0276]

[0277] This gives us three thresholds for the preset action type "jump".

[0278] Furthermore, after obtaining a nomination threshold for a preset action type, candidate nominations for that preset action type are obtained by combining the nomination threshold with the results of each frame of that preset action type.

[0279] As an example, such as Figure 11b As shown, Figure 11b The curve in the image represents the score curve for each frame of a preset action type, with the horizontal axis representing time. The three solid horizontal lines represent three nomination thresholds, allowing for the acquisition of candidate nominations at different scales. Optionally, each candidate second image frame is determined, where the first probability value of the action contained in the candidate second image frame for the preset action type is greater than the corresponding nomination threshold. Based on the time values ​​of each candidate second image frame, at least one candidate nomination is determined; that is, a range greater than any nomination threshold constitutes a candidate nomination. Optionally, if the time length of a candidate nomination is less than a certain value, it can be directly discarded to reduce computational load. For example, in... Figure 11b In this process, the two candidate nominations with a time length of 1 determined based on τ1 are directly discarded, resulting in only one candidate nomination based on τ1. Two candidate nominations are obtained based on τ2, and one based on τ3. The candidate nomination corresponding to τ1 emphasizes accuracy, while the candidate nomination corresponding to τ3 emphasizes continuity. Therefore, this embodiment uses different time granularities to generate candidate nominations.

[0280] In this way, candidate nominations for each preset action type can be obtained. In practical applications, it is also possible not to determine candidate nominations for each preset action type. For example, preset action types with sub-video segment scores below a certain value or scores below a certain value for each frame can be directly ignored. Those skilled in the art can set this according to the actual situation, and this application embodiment does not limit it.

[0281] By adaptively generating thresholds and producing candidate nominations at different time scales, the number of nominations can be reduced, and nominations at different time granularities can be filtered with high confidence to reduce computation.

[0282] Furthermore, after obtaining a series of candidate nominations, based on each candidate nomination, the corresponding action interval in the target video segment is identified to determine the target candidate nominations, including:

[0283] Obtain the video segment features corresponding to the target video segment;

[0284] Based on each candidate nomination, the video segment features corresponding to the action interval are determined, and the corresponding candidate features are obtained.

[0285] Each candidate feature is identified to determine the target candidate nomination.

[0286] The video segment features corresponding to the target video segment can be features extracted from the target video segment using any feature extraction method, or they can be the output features of the last temporal residual block among the multiple temporal residual blocks mentioned above.

[0287] Specifically, based on each candidate nomination, the video segment features corresponding to the action interval are determined as candidate features. That is, based on the time range represented by each candidate nomination, the features corresponding to the time range in the target video segment are determined as candidate features.

[0288] In one feasible implementation, the process can also use a sampler to sample a fixed length of features from the video segment features corresponding to the target video segment within the time range of the candidate nomination. Specifically, based on each candidate nomination, the video segment features of the corresponding action interval are determined to obtain the corresponding candidate features, including: determining the video segment features of the action interval corresponding to each candidate nomination; and sampling the video segment features of the action interval corresponding to each candidate nomination as features of a predetermined length to obtain the corresponding candidate features.

[0289] As an example, suppose the sub-video segment (target video segment) has 25 frames, and each frame has its corresponding recognition result and its corresponding feature map. If a candidate nomination contains 12 frames, and the fixed input of the recognizer used for recognition is 8 frames, then the 12 frame feature maps need to be converted into 8 frame feature maps through a certain method (such as linear interpolation) before the 8 frame feature maps can be input into the recognizer for recognition.

[0290] In this embodiment of the application, based on each candidate nomination, the video segment features corresponding to the action range are determined to obtain the corresponding candidate features, which may include:

[0291] Based on each candidate nomination, the corresponding action range in the target video segment is identified, and the action recognition score corresponding to each candidate nomination is obtained.

[0292] For each candidate nomination, the continuous action recognition score of the candidate nomination is determined based on the action recognition score and action range corresponding to the candidate nomination;

[0293] The candidate with the highest continuous action recognition score is selected as the target candidate.

[0294] The identification process can employ a preset identifier. Those skilled in the art can select or train a suitable identifier based on the actual situation. This application does not limit the specific identifier used in this embodiment.

[0295] In this embodiment of the application, it is considered that, generally speaking, shorter candidate nominations will have higher recognition scores than longer candidate nominations because longer nominations always contain some transitional states. For example Figure 11bIn the graph, the candidate nominations corresponding to τ1 are included in the candidate nominations corresponding to τ3. The score curves show that the frame scores of the candidate nominations corresponding to τ1 are high, but the intervals are short, while some scores of the candidate nominations corresponding to τ3 are low, but the intervals are long. To maintain continuity while maintaining accuracy, a time scale can be added to the recognition criteria. Specifically, after recognizing all candidate nominations and obtaining the corresponding action recognition scores, the recognition results can be transformed using the following formula:

[0296] s′=s×logΔt

[0297] Where s represents the action recognition result (action recognition score) of a candidate nomination, Δt represents the time length (action interval) of the candidate nomination, and s' represents the final recognition result (continuous action recognition score), which takes into account the continuity of the action. The candidate nomination with the highest score in a certain category among all final recognition results (target candidate nomination) is selected, and the action type identified by it is the action type of that sub-video segment (target video segment). The identified start and end time points (action interval) are the start and end time points of the sub-video segment.

[0298] For example, the results of a continuous motion localization are shown in Table 2. Candidate A has the highest raw score, but is too short, so its final score is the lowest. Candidate B is the longest, but its raw score is too low, so its final score is not high. Candidate C has a good raw score and an appropriate length, so it can be used as a target candidate.

[0299] Table 2

[0300] Candidate Nomination A 0.92 4 0.553 Candidate Nomination B 0.69 8 0.623 Candidate Nomination C 0.82 6 0.638

[0301] In this embodiment of the application, by means of Figure 11c A specific example is shown. Figure 11c The left side shows the overall scheme, which involves performing action recognition on sub-video segments to obtain sub-video segment recognition results (i.e., first action recognition results and second action recognition results). Then, through action nomination and a recognizer, the sub-video segment localization result (i.e., the action localization result of the target video segment) is obtained. The action nomination (adaptive proposal generator) and recognizer constitute the action localization part of the video. The method used in this example is adaptive multi-scale localization. The adaptive proposal generator generates candidate nominations with different time scales, and the recognizer obtains the final action distinction and action category in terms of accuracy and continuity. Figure 11c The process shown in the middle section refers to the action recognition result, specifically the aforementioned first and second action recognition results. The sub-video segment features utilize intermediate results from the action recognition process, which can be as follows: Figure 11cThe characteristics of the output of the last temporal residual block among the multiple temporal residual blocks shown on the right side can be found in the section on... Figure 8g The description will not be repeated here. An adaptive nomination generator generates candidate nominations (a series of sub-time intervals) based on the sub-video segment recognition results. Then, a sampler samples a fixed length of features from the sub-video segment features within the time range of the candidate nominations, inputs these features into the recognizer, and finally outputs the sub-video segment localization result, i.e., the action localization result of the target video segment.

[0302] Furthermore, based on the first action recognition result and the second action recognition result, the action localization result of the corresponding target video segment is obtained, including:

[0303] Based on the second action recognition result, the target action type in the target video segment is determined;

[0304] Based on the first action recognition result, obtain the target key image frame corresponding to the target action type;

[0305] Based on the target key image frames, obtain the target action range corresponding to the target action type;

[0306] The target action type and the corresponding target action range are used as the action localization result of the target video segment.

[0307] Specifically, the recognition result of the target video segment includes the first action recognition result X (size can be (k, m+1), where k is the frame number and m+1 is the action type number) of each second image frame in the video segment and the second action recognition result Y (size can be a one-dimensional vector with length m+1) of the target video segment. By obtaining the target action interval corresponding to the target video segment, the start and end times of the action in the target video segment can be determined, as well as the target action type in the target video segment. Figure 12a As shown, the specific process for determining the target action range and target action type may include the following steps:

[0308] (1) First, the target action type number of the target video segment is obtained from the second action recognition result of the target video segment according to the following formula:

[0309] a = argmax(Y)

[0310] The preset action type corresponding to the highest probability value in the second action recognition result is taken as the target action type. The obtained 'a' is the index of the target action type. If 'a' = 0, it means that the video segment is identified as another action type. It is possible that in the target video segment, a certain action appears for a very short time, causing other action types to dominate the overall target video segment. Therefore, it is necessary to check if there is a short action interval. In this case, the index of the action type with the second highest probability value in the target video segment is obtained as 'a', and the action type with the second highest probability value is taken as the target action type.

[0311] (2) Based on the first action recognition result of each second image frame in the target video segment, the average score S of the a-th action type in the k frames is calculated using the formula. a :

[0312]

[0313] Where, ∑ i x i,a To sum the first probability values ​​corresponding to the a-th action type in all second image frames in the sliding window, k is the first dimension in the first action recognition result, which is the number of sampled second image frames.

[0314] Simultaneously, the target key image frame corresponding to the a-th action type is calculated using the following formula:

[0315] t i =argmax(X a )

[0316] Among them, t i For the target key image frame, X a The first probability value (which can be a Logits value) corresponding to the a-th action type in each second image frame is used. Specifically, based on the first action recognition result, the Logits value corresponding to the target action type (i.e., the a-th action type) in at least one second image frame is obtained; the second image frame with the largest Logits value corresponding to the target action type in at least one second image frame is taken as the target key image frame.

[0317] (3) Based on the target key image frame, obtain the target action interval corresponding to the target action type. For example, the time pyramid method can be used to calculate the target action interval. Specifically, with the width of at least one preset action interval as the window width, perform a sliding window operation on the target video segment to obtain at least one sliding window containing the target key image frame; based on the Logits value of the target action type corresponding to the second image frame in the sliding window, obtain the score value of the sliding window, and determine the sliding window with the largest score value as the target sliding window; based on the Logits value corresponding to each preset action type in the second image frame in the target sliding window, obtain the probability value corresponding to each preset action type in the target sliding window; if the preset action type corresponding to the maximum probability value in the target sliding window is the target action type, then the precise position (action interval) corresponding to the target sliding window is determined as the target action interval, otherwise the output segment has no action.

[0318] The score for the sliding window can be calculated using the following formula:

[0319]

[0320] Where Score is the score of the sliding window, Δt is the action range corresponding to the sliding window, and x i,a Let ∑ be the first probability value (which can be a Logits value) corresponding to the a-th action type in the i-th second image frame. Δt x i,a To sum the first probability values ​​corresponding to the a-th action type in all second image frames within the sliding window, weight represents the weight, which can be set according to the actual situation, for example, it can be 0.5, S a The probability value corresponding to each preset action type in the target sliding window, based on the aforementioned average score, can be calculated using the following formula:

[0321]

[0322] Where, x i,j p represents the numerical values ​​of the i-th frame and the j-th action in each frame's recognition result; j Let be the probability value of the j-th action within the target action interval (start time, end time). This represents the summation of the first probability values ​​corresponding to the j-th action type in all second image frames within the target action interval, where m+1 represents m action types and one other category.

[0323] For example, such as Figure 12b As shown, a sliding window operation is performed on the target video segment with window widths of Δt1, Δt2, and Δt3, respectively. Since the widths of the three windows increase sequentially, this sliding window operation can be understood as a pyramid sliding window. Here, Δt represents the smallest unit of time, such as 0.1 seconds. iThis implies different time scales. t l This represents the search space at different time scales. a This refers to the frame score of action type 'a'. First, find the keyframe with the highest frame score, and then use sliding windows with different time scales to locate the positions on either side of the keyframe. To allow for comparison of results from different time scales, a threshold can be used for selection; the threshold is S mentioned above. a Then, the score for each sliding window can be calculated, which is the Score calculation mentioned above. Select the target sliding window with the highest score, and use the formula for calculating the probability value of each preset action type in the target sliding window to determine whether the action type in this target sliding window is action type 'a'. Calculate the probability value for each action corresponding to the target sliding window. If the probability value of action type 'a' is the highest, retain the action localization result; otherwise, discard it.

[0324] Compared to traditional methods, this approach offers the advantage of requiring less computation, with the final score calculated from frame scores, eliminating the need for an additional inference module. It considers both accuracy and continuity, using a threshold to ensure comparability of recommendations across different time scales.

[0325] In one optional embodiment of this application, obtaining motion localization results for the video based on motion localization results of the target video segment includes:

[0326] If two adjacent target action intervals in the video correspond to the same target action type, and the difference between the end time of the earlier target action interval and the start time of the later target action interval is no greater than the third preset duration, then the two target action intervals are merged to obtain the motion localization result of the video.

[0327] Specifically, merging the two target action ranges includes:

[0328] The start time of the earlier target action interval will be used as the start time of the merged target action interval, and the end time of the later target action interval will be used as the end time of the merged target action interval.

[0329] Understandably, if the target action types corresponding to two adjacent target action intervals in the video are not the same, or if the difference between the end time of the earlier target action interval and the start time of the later target action interval is greater than the third preset duration, then the motion localization results of the two target video segments will be output separately as the motion localization results of the video. The third preset duration can be set according to actual needs, for example, it can be set to 1 second.

[0330] The method of this application embodiment will be further illustrated by an example below, such as... Figure 13a and 13bAs shown, this method can run on mobile terminals (such as mobile phones). The mobile terminal captures video through an image acquisition device; this captured video can be called the raw video. Image and video preprocessing can be performed on the raw video first. In the processing of offline and online videos on mobile devices, various formats of frame images from videos acquired through different methods are uniformly converted to RGB format. After acquiring the RGB image, as shown... Figure 13a and 13b As shown, the shorter side of the image (width w, height h) is reduced to a fixed size s1, and the longer side of the image is reduced to s1' according to the formula.

[0331] The formula is:

[0332]

[0333] After resizing the image, it is cropped by extracting a 2×2 square portion from the image, centered at the origin, as the input to the next-level network. The sizes of s1 and s2 can be set according to actual needs.

[0334] like Figure 14 As shown, after processing the original video as described above, the input video is obtained. Then, pedestrian detection is performed on the input video, resulting in three target video segments containing pedestrians, namely △t1, △t2, and △t3 (△t1, △t2, and △t3 are all less than or equal to the preset maximum duration △t). max Action recognition is performed on three target video segments respectively. The action recognition result for target video segment △t1 is preset action type 2, the action recognition result for target video segment △t2 is preset action type 2, and the action recognition result for target video segment △t3 is preset action type 4. For each target video segment, action localization is performed based on action recognition, that is, the target action range is obtained. Specifically, the target action range for target video segment △t1 is [t 1s , t 1e The target action range for the target video segment △t2 is [t]. 2s , t 2e The target action range for the target video segment △t3 is [t]. 3s , t 3e After determining the target action range for each target video segment, the results of the target video segments that can be integrated are then integrated. Specifically, since target video segment △t1 and target video segment △t2 have the same target action type, and the end time t of the earlier target action range is... 1e The start time t of the target action interval that follows 2sSubsequently, since the target action ranges of the two target video segments overlap, they can be merged into a single motion localization result, where the target action type is preset action type 2 and the target action range is [t]. 1s , t 2e Therefore, the final motion localization result corresponding to this input video segment is "{Preset motion type 2, target motion range [t] ]. 1s , t 2e ]}, {Preset action type 4, target action range [t 3s , t 3e ]}”.

[0335] The video motion localization method provided in this application embodiment can be applied to the following application scenarios:

[0336] 1. Offline Processing: For videos in the album, users can select a video for motion localization processing. After processing, users will obtain multiple video segments, each containing one or more actions. Users can choose to save, analyze, edit, or share these segments on social media platforms as needed.

[0337] 2. Online Processing: Users can record video using their camera if this function is enabled. During recording, the video content is analyzed, and when recording is complete, real-time motion localization results are provided. The motion localization results are consistent with offline processing, and users can process them as needed.

[0338] Figure 15 A structural block diagram of a positioning device provided in an embodiment of this application is shown below. Figure 15 As shown, the device 1300 may include: a target video segment acquisition module 1301, an action recognition result acquisition module 1302, and an action localization result acquisition module 1303, wherein:

[0339] The target video segment acquisition module 1301 is used to acquire at least one target video segment containing the target object in the video;

[0340] The action recognition result acquisition module 1302 is used to acquire the first action recognition result of at least one image frame in the target video segment and the second action recognition result of the target video segment;

[0341] The motion localization result acquisition module 1303 is used to acquire motion localization results of the video based on the first motion recognition result and the second motion recognition result.

[0342] The solution provided in this application, after obtaining the target video segment containing the target object in the video to be located, obtains the first action recognition result of multiple image frames in each target video segment and the second action recognition result of the target video segment. Then, based on each first action recognition result and the second action recognition result, the action positioning result of the video to be located is obtained. This solution does not require the acquisition of optical flow images during the action positioning process, requires less memory capacity, has less computation, and has high accuracy, thus having a wider range of applications.

[0343] In one optional embodiment of this application, the target video segment acquisition module is specifically used for:

[0344] Perform target object detection on at least one image frame in the video, and obtain at least one target video segment based on the target object detection results.

[0345] In an optional embodiment of this application, the target video segment acquisition module is further configured to:

[0346] At least one first image frame is obtained from the video, and target object detection is performed on the obtained first image frame to obtain the corresponding target object detection result;

[0347] If the target object detection result indicates that the current first image frame obtained at the current moment contains the target object, then the current first image frame, the previous first image frame, and the image frames between the current first image frame and the previous first image frame are cached. If the total number of cached image frames at the current moment is not less than the first preset value, then the cached image frames at the current moment are output as the corresponding target video segment.

[0348] If the target object detection result indicates that the current first image frame acquired at the current moment does not contain the target object, and if the total number of cached image frames at the current moment is not less than the second preset value, then the cached image frame at the current moment will be output as the corresponding target video segment.

[0349] In an optional embodiment of this application, the device may further include a cache clearing module for:

[0350] After outputting the corresponding target video segment, if the target object detection result indicates that the current first image frame obtained at the current moment contains the target object, then the cached image frames of the third preset value before the current first image frame are retained, and other cached image frames are cleared.

[0351] After outputting the corresponding target video segment, if the target object detection result indicates that the current first image frame acquired at the current moment does not contain the target object, then the cached image frame at the current moment is cleared. In an optional embodiment of this application, the action recognition result acquisition module is specifically used for:

[0352] At least one second image frame is obtained from the target video segment, and action recognition is performed on the obtained second image frame to obtain the corresponding first action recognition result;

[0353] Based on the first action recognition result, obtain the second action recognition result.

[0354] In an optional embodiment of this application, the action recognition result acquisition module is further used for:

[0355] Extract spatial information corresponding to at least one second image frame, and obtain the corresponding input features based on the spatial information;

[0356] Using at least one temporal residual module, time information corresponding to at least one second image frame is extracted based on input features, and corresponding output features are obtained based on input features and time information;

[0357] Based on the output features, obtain the first action recognition result corresponding to at least one second image frame.

[0358] In one optional embodiment of this application, the temporal residual module includes a first branch and a second branch, and the action recognition result acquisition module is further used for:

[0359] Through the first branch, the weight information corresponding to each second image frame is obtained based on the input features, and then the first feature containing the first time information is obtained based on the weight information and the input features.

[0360] Through the second branch, feature information of adjacent second image frames corresponding to each second image frame is obtained based on the input features, and corresponding second features containing second time information are obtained based on the feature information of adjacent second image frames.

[0361] Based on the first feature and the second feature, the corresponding output features are obtained.

[0362] In an optional embodiment of this application, the action recognition result acquisition module is further used for:

[0363] The first action recognition results of at least one second image frame are fused to obtain the second action recognition result.

[0364] In one optional embodiment of this application, the first action recognition result includes first probability values ​​for actions of different preset action types contained in the corresponding second image frame, and the second action recognition result includes second probability values ​​for actions of different preset action types contained in the corresponding target video segment. The action recognition result acquisition module is further configured to:

[0365] The average value of the first probability value corresponding to each preset action type in at least one second image frame is obtained respectively;

[0366] The average value of the first probability value corresponding to each preset action type is normalized to obtain the probability value corresponding to different preset action types.

[0367] The probability values ​​of each preset action type are used as the second probability values ​​of each preset action type for the actions contained in the target video segment, thus obtaining the second action recognition result.

[0368] In one optional embodiment of this application, the first probability value is the non-normalized log probability Logits value.

[0369] In one optional embodiment of this application, the motion localization result acquisition module includes a first motion localization result acquisition submodule and a second motion localization result acquisition submodule, wherein:

[0370] The first action localization result acquisition submodule is used to obtain the action localization result of the corresponding target video segment based on the first action recognition result and the second action recognition result.

[0371] The second motion localization result acquisition submodule is used to acquire the motion localization result of the video based on the motion localization result of the target video segment.

[0372] In one optional embodiment of this application, the first action positioning result acquisition submodule is specifically used for:

[0373] Based on the first action recognition result and the second action recognition result, at least one candidate nomination is determined;

[0374] Based on each candidate nomination, the corresponding action range in the target video segment is identified to determine the target candidate nomination;

[0375] The action type and action range corresponding to the target candidate nomination are used as the action localization results of the target video segment.

[0376] In an optional embodiment of this application, the first action positioning result acquisition submodule is further used for:

[0377] Based on the first action recognition result and the second action recognition result, determine the nomination threshold corresponding to at least one preset action type;

[0378] For each of the at least one preset action types, at least one candidate nomination is determined based on the corresponding nomination threshold and the first action recognition result.

[0379] In an optional embodiment of this application, for each of the at least one preset action type, the first action localization result acquisition submodule is further configured to:

[0380] Each candidate second image frame is determined, and the first probability value of the action contained in the candidate second image frame being the preset action type is greater than the corresponding nomination threshold;

[0381] Based on each candidate second image frame, at least one candidate nomination is determined.

[0382] In an optional embodiment of this application, the first action positioning result acquisition submodule is further used for:

[0383] Obtain the video segment features corresponding to the target video segment;

[0384] Based on each candidate nomination, the video segment features corresponding to the action interval are determined, and the corresponding candidate features are obtained.

[0385] Each candidate feature is identified to determine the target candidate nomination.

[0386] In an optional embodiment of this application, the first action positioning result acquisition submodule is further used for:

[0387] Determine the video segment features corresponding to the action range of each candidate nomination;

[0388] The video segment features corresponding to the action interval of each candidate nomination are sampled into features of a predetermined length to obtain the corresponding candidate features.

[0389] In an optional embodiment of this application, the first action positioning result acquisition submodule is further used for:

[0390] Based on each candidate nomination, the corresponding action range in the target video segment is identified, and the action recognition score corresponding to each candidate nomination is obtained.

[0391] For each candidate nomination, the continuous action recognition score of the candidate nomination is determined based on the action recognition score and action range corresponding to the candidate nomination;

[0392] The candidate with the highest continuous action recognition score is selected as the target candidate.

[0393] In one optional embodiment of this application, the first action positioning result acquisition submodule is specifically used for:

[0394] Based on the second action recognition result, the target action type in the target video segment is determined;

[0395] Based on the first action recognition result, obtain the target key image frame corresponding to the target action type;

[0396] Based on the target key image frames, obtain the target action range corresponding to the target action type;

[0397] The target action type and the corresponding target action range are used as the action localization result of the target video segment.

[0398] In an optional embodiment of this application, the first action positioning result acquisition submodule is further used for:

[0399] The preset action type corresponding to the highest probability value in the second action recognition result is taken as the target action type.

[0400] In an optional embodiment of this application, the first action positioning result acquisition submodule is further used for:

[0401] Based on the first action recognition result, obtain the first probability value corresponding to the target action type in at least one second image frame;

[0402] The second image frame with the highest first probability value corresponding to the target action type in at least one second image frame is taken as the target key image frame.

[0403] In an optional embodiment of this application, the first action positioning result acquisition submodule is further used for:

[0404] Using at least one preset action range width as the window width, perform a sliding window operation on the target video segment to obtain at least one sliding window containing the target key image frame;

[0405] Based on the first probability value of the target action type corresponding to the second image frame in the sliding window, the score value of the sliding window is obtained, and the sliding window with the largest score value is determined as the target sliding window;

[0406] Based on the first probability value corresponding to each preset action type in the second image frame of the target sliding window, obtain the probability value corresponding to each preset action type in the target sliding window;

[0407] If the preset action type corresponding to the maximum probability value in the target sliding window is the target action type, then the action range corresponding to the target sliding window is determined as the target action range.

[0408] In one optional embodiment of this application, the second action positioning result acquisition submodule is specifically used for:

[0409] If two adjacent target action intervals in the video correspond to the same target action type, and the difference between the end time of the earlier target action interval and the start time of the later target action interval is no greater than the third preset duration, then the two target action intervals are merged to obtain the motion localization result of the video.

[0410] In an optional embodiment of this application, the second action positioning result acquisition submodule is further used for:

[0411] The start time of the earlier target action interval will be used as the start time of the merged target action interval, and the end time of the later target action interval will be used as the end time of the merged target action interval.

[0412] The following is for reference. Figure 16 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., performing...). Figure 1 The diagram illustrates the structure of the terminal device or server 1400 of the method shown. The electronic devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 16 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0413] The electronic device includes a memory and a processor. The memory stores a program for executing the methods described in the various method embodiments above. The processor is configured to execute the program stored in the memory. The processor may be referred to as processing device 1401 as described below. The memory may include at least one of read-only memory (ROM) 1402, random access memory (RAM) 1403, and storage device 1408 as described below, as specifically shown below:

[0414] like Figure 16 As shown, electronic device 1400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1402 or a program loaded from storage device 1408 into random access memory (RAM) 1403. RAM 1403 also stores various programs and data required for the operation of electronic device 1400. Processing device 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.

[0415] Typically, the following devices can be connected to I / O interface 1405: input devices 1406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1409. Communication device 1409 allows electronic device 1400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 16Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0416] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1409, or installed from storage device 1408, or installed from ROM 1402. When the computer program is executed by processing device 1401, it performs the functions defined in the methods of embodiments of this application.

[0417] It should be noted that the computer-readable storage medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0418] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0419] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0420] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0421] Obtain at least one target video segment containing the target object; obtain a first action recognition result and a second action recognition result of at least one image frame in the target video segment; and obtain the action localization result of the video based on the first action recognition result and the second action recognition result.

[0422] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0423] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0424] The modules or units described in the embodiments of this application can be implemented in software or hardware. The names of modules or units do not necessarily limit the specific unit; for example, a target tracking status acquisition module can also be described as a "module for acquiring target tracking status".

[0425] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0426] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0427] The apparatus provided in this application embodiment can implement at least one of multiple modules through an AI model. AI-related functions can be executed through non-volatile memory, volatile memory, and a processor.

[0428] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as central processing unit (CPU), application processor (AP), etc., or pure graphics processing unit, such as graphics processing unit (GPU), vision processing unit (VPU), and / or AI-specific processors, such as neural processing unit (NPU).

[0429] The one or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0430] Here, "providing through learning" refers to obtaining predefined operating rules or an AI model with desired characteristics by applying a learning algorithm to multiple learning datasets. This learning can be performed within the device itself, in which the AI ​​is executed according to the embodiment, and / or can be implemented via a separate server / system.

[0431] This AI model can contain multiple neural network layers. Each layer has multiple weight values, and the computation of a layer is performed using the results of the previous layer and the multiple weights of the current layer. Examples of neural networks include, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), and Deep Q-Networks.

[0432] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0433] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific methods implemented by the computer-readable medium described above when executed by an electronic device can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

Claims

1. A motion localization method, characterized in that, include: Identify at least one target video segment that contains the target object; Obtain at least one image frame from the at least one target video segment; The first action recognition result of the at least one image frame is obtained through an action recognition network; Based on the first action recognition result, obtain the second action recognition result of the target video segment; Based on the first action recognition result and the second action recognition result, the action localization result of the video is obtained; The process of obtaining the first action recognition result includes: The input feature map corresponding to the at least one image frame is processed by a temporal residual network, wherein the temporal residual network includes: An attention subnetwork is configured to perform max pooling and average pooling on the input feature map to obtain a first output feature map that includes attention information for each of the at least one image frame. The neighborhood subnetwork is configured to extract local temporal information from adjacent frames to obtain a second output feature map including the local temporal information.

2. The method according to claim 1, characterized in that, Determining the at least one target video segment includes: The video is subjected to target object detection, and the at least one target video segment is obtained based on the target object detection results.

3. The method according to claim 1, characterized in that, Determining the at least one target video segment includes: At least one first image frame is obtained from the video, and target object detection is performed on the at least one first image frame to obtain the target object detection result; Based on the target object detection result indicating that the current first image frame obtained at the current time contains the target object, the current first image frame, the previous first image frame relative to the current first image frame, and the intermediate image frames between the current first image frame and the previous first image frame are cached. Based on the indication that the total number of cached image frames at the current time is not less than a first preset value, the cached image frame at the current time is output as the at least one target video segment. Based on the target object detection result indicating that the current first image frame obtained at the current moment does not contain the target object, and if the total number of cached image frames at the current moment is not less than a second preset value, the cached image frame at the current moment is output as the at least one target video segment.

4. The method according to claim 3, characterized in that, The method further includes: After outputting the at least one target video segment, based on the target object detection result indicating that the current first image frame obtained at the current time contains the target object, the cached image frames before the current first image frame with a third preset value are retained, and other cached image frames are cleared. After outputting the at least one target video segment, based on the target object detection result indicating that the current first image frame acquired at the current time does not contain the target object, the cached image frame at the current time is cleared.

5. The method according to claim 1, characterized in that, The acquisition of the first action recognition result of the at least one image frame includes: Action recognition is performed on the at least one second image frame to obtain the first action recognition result.

6. The method according to claim 5, characterized in that, The step of performing action recognition on the at least one second image frame to obtain the corresponding first action recognition result includes: Extract spatial information corresponding to at least one second image frame, and obtain input features based on the spatial information; Using a temporal residual network, temporal information corresponding to at least one second image frame is extracted based on the input features, and output features are obtained based on the input features and the temporal information. Based on the output features, the first action recognition result corresponding to the at least one second image frame is obtained.

7. The method according to claim 6, characterized in that, The temporal residual network includes a first branch and a second branch. The step of extracting temporal information corresponding to at least one second image frame based on the input features using the temporal residual network, and obtaining corresponding output features based on the input features and the temporal information, includes: Through the first branch, weight information corresponding to the second image frame is obtained based on the input features, and then a first feature containing first time information is obtained based on the weight information and the input features; Through the second branch, feature information of the second image frame adjacent to the second image frame is obtained based on the input features, and a second feature containing second time information is obtained based on the feature information of the adjacent second image frame; The output feature is obtained based on the first feature and the second feature.

8. The method according to claim 5, characterized in that, The step of obtaining the second action recognition result based on the first action recognition result includes: The first action recognition result of at least one second image frame is fused to obtain the second action recognition result.

9. The method according to claim 8, characterized in that, The first action recognition result includes first probability values ​​for each preset action type of the actions contained in the corresponding second image frame, and the second action recognition result includes second probability values ​​for each preset action type of the actions contained in the corresponding target video segment. The step of fusing the first action recognition results of at least one second image frame to obtain the second action recognition result includes: The average value of the first probability value corresponding to each preset action type in at least one second image frame is obtained respectively; The average value of the first probability value corresponding to each preset action type is normalized to obtain the second probability value corresponding to each preset action type. The second probability value of each preset action type is used as the second action recognition result.

10. The method according to claim 9, characterized in that, The first probability value is the non-normalized logarithmic probability Logits.

11. The method according to claim 1, characterized in that, The step of obtaining the action localization result of the video based on the first action recognition result and the second action recognition result includes: Based on the first action recognition result and the second action recognition result, the action localization result of the at least one target video segment is obtained; Based on the motion localization results of the target video segment, obtain the motion localization results of the video.

12. The method according to claim 11, characterized in that, Based on the first action recognition result and the second action recognition result, the action localization result of the at least one target video segment is obtained, including: Based on the first action recognition result and the second action recognition result, at least one candidate nomination is determined; Based on each candidate nomination, the action range in the target video segment is identified to determine the target candidate nomination; The action type and action range corresponding to the target candidate nomination are used as the action localization result of the at least one target video segment.

13. The method according to claim 12, characterized in that, The step of determining at least one candidate nomination based on the first action recognition result and the second action recognition result includes: Based on the first action recognition result and the second action recognition result, a nomination threshold corresponding to at least one preset action type is determined; For each of the at least one preset action types, at least one candidate nomination is determined based on the corresponding nomination threshold and the first action recognition result.

14. The method according to claim 13, characterized in that, For each of the at least one preset action types, based on the corresponding nomination threshold and the first action recognition result, at least one candidate nomination is determined, including: At least one candidate second image frame is determined, wherein the first probability value of the action contained in the candidate second image frame is greater than the corresponding nomination threshold. Based on the at least one candidate second image frame, at least one candidate nomination is determined.

15. The method according to claim 12, characterized in that, The step of identifying action regions in the target video segment based on each candidate nomination and determining target candidate nominations includes: Obtain the first video segment features corresponding to the target video segment; Based on each candidate nomination, the second video segment feature corresponding to the action interval is determined from the features of the first video segment; Based on the features of the second video segment, target candidate nominations are determined.

16. The method according to claim 15, characterized in that, The step of determining the second video segment features corresponding to the action interval based on each candidate nomination includes: The third video segment features corresponding to the action range of each candidate nomination are determined from the features of the first video segment; The third video segment features corresponding to the action interval of each candidate nomination are sampled as features of a predetermined length to obtain the corresponding second video segment features.

17. The method according to claim 12, characterized in that, The step of identifying the corresponding action interval in the target video segment based on each candidate nomination and determining the target candidate nomination includes: Based on each candidate nomination, the corresponding action interval in the target video segment is identified to obtain the action recognition score corresponding to each candidate nomination; For each candidate nomination, the continuous action recognition score of the candidate nomination is determined based on the action recognition score and action range corresponding to the candidate nomination; The candidate with the highest continuous action recognition score is selected as the target candidate.

18. The method according to claim 11, characterized in that, Based on the first action recognition result and the second action recognition result, the action localization result of the at least one target video segment is obtained, including: Based on the second action recognition result, the target action type in the target video segment is determined; Based on the first action recognition result, obtain the target key image frame corresponding to the target action type; Based on the target key image frame, obtain the target action range corresponding to the target action type; The target action type and the corresponding target action range are used as the action localization result of the at least one target video segment.

19. The method according to claim 18, characterized in that, The step of determining the target action type in the target video segment based on the second action recognition result includes: The preset action type corresponding to the maximum second probability value in the second action recognition result is taken as the target action type.

20. The method according to claim 18, characterized in that, The step of obtaining the target key image frame corresponding to the target action type based on the first action recognition result includes: Based on the first action recognition result, obtain a first probability value corresponding to the target action type in at least one second image frame; The second image frame with the highest first probability value corresponding to the target action type in the at least one second image frame is taken as the target key image frame.

21. The method according to claim 18, characterized in that, The step of obtaining the target action range corresponding to the target action type based on the target key image frame includes: Using at least one preset action range width as the window width, a sliding window operation is performed on the target video segment to obtain at least one sliding window containing the target key image frame; Based on the first probability value of the target action type corresponding to the second image frame in the sliding window, the score value of the sliding window is obtained, and the sliding window with the largest score value is determined as the target sliding window; Based on the first probability value corresponding to each preset action type in the second image frame of the target sliding window, obtain the probability value corresponding to each preset action type in the target sliding window; If the preset action type corresponding to the maximum probability value in the target sliding window is the target action type, then the action range corresponding to the target sliding window is determined as the target action range.

22. The method according to claim 11, characterized in that, The process of obtaining the motion localization result of the video based on the motion localization result of the target video segment includes: If two adjacent target action intervals in the video correspond to the same target action type, and the difference between the end time of the earlier target action interval and the start time of the later target action interval is no greater than a third preset duration, then the two target action intervals are merged to obtain the action positioning result of the video.

23. The method according to claim 22, characterized in that, The merging of the two target action ranges includes: The start time of the earlier target action interval is used as the start time of the merged target action interval, and the end time of the later target action interval is used as the end time of the merged target action interval.

24. The method according to claim 1, characterized in that, The first action recognition result corresponds to a first action recognition score, which indicates the probability of at least one preset action type occurring in the at least one image frame; The second action recognition result corresponds to a second action recognition score, which indicates the probability of the at least one preset action type occurring in the at least one target video segment; The motion localization result indicates whether a preset motion type appears in the at least one target video segment.

25. An electronic device, characterized in that, Including memory and processor; The memory stores computer programs; The processor is configured to execute the computer program to implement the method of any one of claims 1 to 24.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 24.

Images