A video frame labeling method and device, computer equipment and storage medium

Abstract

The disclosure provides a video frame labeling method and device, computer equipment and a storage medium, wherein the method comprises: extracting each video frame from a to-be-labeled video according to a preset time interval; scrolling a video frame extraction strip in a first area of a video labeling page according to the order of each video frame in the to-be-labeled video, wherein each video frame is sequentially displayed in the video frame extraction strip; wherein the number of video frames simultaneously displayed in the first area is not more than a first preset number; and in response to a labeling operation on a target video frame in the video frame extraction strip, obtaining a labeled video segment determined according to the target video frame.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a video frame annotation method, apparatus, computer device, and storage medium. Background Technology

[0002] Temporal Action Detection (TAD), an important and complex task in video understanding, aims to identify each action instance performed by a target object in a video to be labeled, along with the corresponding video time segment. For example, it identifies the video time segments in which a target object performs actions such as singing or dancing. Existing labeling methods suffer from slow labeling speed and low efficiency.

[0003] Therefore, improving the speed and efficiency of video frame annotation has become an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides at least one video frame annotation method, apparatus, computer device, and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide a video frame annotation method, including:

[0006] Each video frame is extracted from the video to be labeled according to a preset time interval;

[0007] According to the order of the video frames in the video to be annotated, a video frame extraction band is scrolled and displayed in the first area of ​​the video annotation page. The video frame extraction band contains the video frames displayed in sequence. The number of video frames displayed in the first area at the same time is no greater than a first preset number.

[0008] In response to the annotation operation on the target video frame in the video frame extraction band, an annotated video segment determined based on the target video frame is obtained.

[0009] In one possible implementation, before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the method further includes:

[0010] The video to be annotated is played in the second area of ​​the video annotation page; wherein, during the playback of the video to be annotated, the video frame that scrolls to the center line position of the first area matches the currently playing video frame of the video to be annotated.

[0011] The step of obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band includes:

[0012] In response to the annotation operation on the target video frame at the midline position, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region and the video frame strip scrolling in the first region.

[0013] In one possible implementation, the annotation operation includes a marker bit creation operation; the step of obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for a target video frame in the video frame extraction band includes:

[0014] During the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the first target video frame in the video frame extraction band, the first target time corresponding to the center line position of the first region is determined as the pairing annotation position, and it is determined whether there is a generated pairing annotation position.

[0015] If not, during the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the second target video frame in the video frame extraction band, the second target time corresponding to the center line position is determined as the new pairing annotation position.

[0016] The video segment located between the first target time and the second target time is labeled as a labeled video segment, and the unpaired label position corresponding to the first target time and the new unpaired label position are taken as the matched label position.

[0017] In one possible implementation, after obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the method further includes:

[0018] In response to a selection operation for any labeled video segment, the first video frame corresponding to the labeled video segment is aligned with the center line of the first region, and the playback progress of the video to be labeled in the second region is aligned with the start time of the labeled video segment.

[0019] In one possible implementation, after labeling the video segment located between the first target time and the second target time as a labeled video segment bounded by the first target video frame and the second target video frame, the method further includes:

[0020] Based on the first target time and the second target time, annotation segment indication information is generated and displayed in the third area of ​​the video annotation page. Additionally, on the video timeline corresponding to the video to be annotated in the second area, time segment annotation lines matching the first target time and the second target time are generated and displayed.

[0021] In one possible implementation, the process of scrolling and displaying the video frame extraction band in a first area of ​​the video annotation page according to the chronological order of the respective video frames in the video to be annotated further includes:

[0022] In response to a drag operation on any video frame in the video frame extraction band, a target time corresponding to the centerline position is determined after the drag operation ends.

[0023] At the midline position, matching video frames whose playback time matches the target time are displayed, and the playback progress of the video to be labeled in the second region is aligned with the target time.

[0024] In one possible implementation, the process of scrolling and displaying the video frame extraction band in a first area of ​​the video annotation page according to the chronological order of the respective video frames in the video to be annotated further includes:

[0025] In response to a selection operation on any video frame currently displayed in the first region, the selected video frame is displayed at the center line position of the first region, and the playback progress of the video to be labeled in the second region is aligned with the time corresponding to the video frame.

[0026] In one possible implementation, the step of extracting video frames from the video to be labeled at preset time intervals includes:

[0027] According to a preset time interval, each video frame is extracted from the video to be labeled, and the video frames are divided into at least one video frame group according to the order of each video frame in the video to be labeled; wherein, the number of video frames in a video frame group is a second preset number.

[0028] After dividing the video frames into at least one video frame group, the method further includes:

[0029] In response to a video frame group switching operation, the first video frame group corresponding to the currently displayed target video frame is determined at the center line position of the first region;

[0030] According to the switching direction indicated by the video frame group switching operation, the first video frame of the second video frame group adjacent to the first video frame group is displayed at the center line position.

[0031] In one possible implementation, the process of scrolling and displaying the video frame extraction band in a first area of ​​the video annotation page according to the chronological order of the respective video frames in the video to be annotated further includes:

[0032] In response to a display progress adjustment operation, determine the first video frame currently being displayed at the center line position of the first region;

[0033] Based on the progress adjustment direction corresponding to the display progress adjustment operation, and according to the preset adjustment interval, determine the second video frame corresponding to the first video frame, and display the second video frame at the center line position;

[0034] Align the playback progress of the video to be labeled in the second region with the moment when the second video frame is matched.

[0035] In one possible implementation, the labeled video segment is a video segment in the video to be labeled where the target object performs the target action;

[0036] Before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the method further includes:

[0037] Using an action boundary detection algorithm, keyframes that match the action boundary of the target action are extracted from the video to be labeled. The keyframes are then displayed in a scrolling manner in the fourth area of ​​the video labeling page according to their order in the video. The keyframes contain the keyframes displayed in sequence. The keyframes corresponding to the action boundary include the video frame when the target object starts performing the target action and the video frame when the target object stops performing the target action.

[0038] The step of obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band includes:

[0039] In response to a trigger operation for any keyframe, the keyframe moment corresponding to the keyframe is determined, and a video frame matching the keyframe moment is displayed at the center line position of the first region, and the playback progress of the video to be annotated in the second region is aligned with the keyframe moment.

[0040] In response to the annotation operation on the target video frame in the video frame extraction band, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region, the video frame extraction band displayed in the first region, and the keyframe band displayed in the fourth region.

[0041] Secondly, embodiments of this disclosure also provide a video frame annotation device, comprising:

[0042] The extraction module is used to extract video frames from the video to be labeled according to a preset time interval.

[0043] The display module is used to scroll and display a video frame extraction strip in a first area of ​​the video annotation page according to the order of the video frames in the video to be annotated. The video frame extraction strip contains the video frames displayed in sequence. The number of video frames displayed in the first area at the same time is no greater than a first preset number.

[0044] The annotation module is used to respond to the annotation operation of the target video frame in the video frame extraction band and obtain the annotated video segment determined according to the target video frame.

[0045] In one possible implementation, the display module, before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is further configured to:

[0046] The video to be annotated is played in the second area of ​​the video annotation page; wherein, during the playback of the video to be annotated, the video frame that scrolls to the center line position of the first area matches the currently playing video frame of the video to be annotated.

[0047] The annotation module, when obtaining an annotated video segment determined based on the target video frame in response to an annotation operation on a target video frame in the video frame extraction band, is configured to:

[0048] In response to the annotation operation on the target video frame at the midline position, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region and the video frame strip scrolling in the first region.

[0049] In one possible implementation, the annotation operation includes a marker bit creation operation; the annotation module, when obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for a target video frame in the video frame extraction band, is configured to:

[0050] During the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the first target video frame in the video frame extraction band, the first target time corresponding to the center line position of the first region is determined as the pairing annotation position, and it is determined whether there is a generated pairing annotation position.

[0051] If not, during the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the second target video frame in the video frame extraction band, the second target time corresponding to the center line position is determined as the new pairing annotation position.

[0052] The video segment located between the first target time and the second target time is labeled as a labeled video segment, and the unpaired label position corresponding to the first target time and the new unpaired label position are taken as the matched label position.

[0053] In one possible implementation, the display module, after obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is configured to:

[0054] In response to a selection operation for any labeled video segment, the first video frame corresponding to the labeled video segment is aligned with the center line of the first region, and the playback progress of the video to be labeled in the second region is aligned with the start time of the labeled video segment.

[0055] In one possible implementation, after labeling the video segment located between the first target time and the second target time as a labeled video segment bounded by the first target video frame and the second target video frame, the labeling module is further configured to:

[0056] Based on the first target time and the second target time, annotation segment indication information is generated and displayed in the third area of ​​the video annotation page. Additionally, on the video timeline corresponding to the video to be annotated in the second area, time segment annotation lines matching the first target time and the second target time are generated and displayed.

[0057] In one possible implementation, the display module, while scrolling and displaying the video frame extraction strip in the first area of ​​the video annotation page according to the sequential order of each video frame in the video to be annotated, is further configured to:

[0058] In response to a drag operation on any video frame in the video frame extraction band, a target time corresponding to the centerline position is determined after the drag operation ends.

[0059] At the midline position, matching video frames whose playback time matches the target time are displayed, and the playback progress of the video to be labeled in the second region is aligned with the target time.

[0060] In one possible implementation, the display module, while scrolling and displaying the video frame extraction strip in the first area of ​​the video annotation page according to the sequential order of each video frame in the video to be annotated, is further configured to:

[0061] In response to a selection operation on any video frame currently displayed in the first region, the selected video frame is displayed at the center line position of the first region, and the playback progress of the video to be labeled in the second region is aligned with the time corresponding to the video frame.

[0062] In one possible implementation, the extraction module, when extracting video frames from the video to be labeled according to a preset time interval, is used to:

[0063] According to a preset time interval, each video frame is extracted from the video to be labeled, and the video frames are divided into at least one video frame group according to the order of each video frame in the video to be labeled; wherein, the number of video frames in a video frame group is a second preset number.

[0064] The display module, after dividing the video frames into at least one video frame group, is further configured to:

[0065] In response to a video frame group switching operation, the first video frame group corresponding to the currently displayed target video frame is determined at the center line position of the first region;

[0066] According to the switching direction indicated by the video frame group switching operation, the first video frame of the second video frame group adjacent to the first video frame group is displayed at the center line position.

[0067] In one possible implementation, the display module, while scrolling and displaying the video frame extraction strip in the first area of ​​the video annotation page according to the sequential order of each video frame in the video to be annotated, is further configured to:

[0068] In response to a display progress adjustment operation, determine the first video frame currently being displayed at the center line position of the first region;

[0069] Based on the progress adjustment direction corresponding to the display progress adjustment operation, and according to the preset adjustment interval, determine the second video frame corresponding to the first video frame, and display the second video frame at the center line position;

[0070] Align the playback progress of the video to be labeled in the second region with the moment when the second video frame is matched.

[0071] In one possible implementation, the labeled video segment is a video segment in the video to be labeled where the target object performs the target action;

[0072] The extraction module, before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video extraction band, is further configured to:

[0073] Using an action boundary detection algorithm, key frames that match the action boundary of the target action are extracted from the video to be labeled; the key frames corresponding to the action boundary include the video frame when the target object starts to perform the target action and the video frame when the target object stops performing the target action.

[0074] The display module, before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is further configured to:

[0075] According to the order of the keyframes in the video to be annotated, a keyframe band is scrolled and displayed in the fourth area of ​​the video annotation page, and the keyframe band contains the keyframes displayed in sequence.

[0076] The annotation module, when obtaining an annotated video segment determined based on the target video frame in response to an annotation operation on a target video frame in the video frame extraction band, is configured to:

[0077] In response to a trigger operation for any keyframe, the keyframe moment corresponding to the keyframe is determined, and a video frame matching the keyframe moment is displayed at the center line position of the first region, and the playback progress of the video to be annotated in the second region is aligned with the keyframe moment.

[0078] In response to the annotation operation on the target video frame in the video frame extraction band, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region, the video frame extraction band displayed in the first region, and the keyframe band displayed in the fourth region.

[0079] Thirdly, an optional implementation of this disclosure also provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0080] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.

[0081] For a description of the effects of the aforementioned video frame annotation device, computer equipment, and computer-readable storage medium, please refer to the description of the aforementioned video frame annotation method; it will not be repeated here.

[0082] The video frame annotation method, apparatus, computer device, and storage medium provided in this disclosure reduce the number of video frames to be processed and improve the speed and efficiency of video frame annotation because the video frames in the displayed video frame extraction band are extracted from the video to be annotated. This means the number of video frames in the video frame extraction band is necessarily less than the number of video frames in the video to be annotated. Furthermore, since the first area can simultaneously display a first preset number of video frames, multiple frames can be displayed simultaneously, facilitating the simultaneous browsing of at least some of the displayed video frames. This allows for quick and intuitive location of the target video to be annotated from the first preset number of displayed video frames, further improving the efficiency of video frame annotation.

[0083] Furthermore, the video frame annotation method, apparatus, computer equipment, and storage medium provided in this disclosure display a video frame extraction strip in a first area and the video to be annotated in a second area, ensuring that the video frame at the center position of the first area matches the currently playing video frame of the video to be annotated. This facilitates quickly locating the vicinity of the video segment to be annotated by using the video frame extraction strip, and then determining the accurate annotation time and completing the annotation by matching the playing video to be annotated. By using the first and second areas in combination, the accuracy of video frame annotation can be improved.

[0084] Furthermore, the video frame annotation method, apparatus, computer equipment, and storage medium provided in this disclosure extract each key frame that matches the action boundary of the target action and display the key frame band in a scrolling manner in the fourth region. This facilitates the rapid determination of matching video frames that match the key frames in the extracted video frame band based on the key frame band. Then, annotation is performed based on the matching video frames and the video to be annotated. This not only ensures the accuracy of video frame annotation but also improves the speed and efficiency of video frame annotation.

[0085] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0086] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0087] Figure 1 A flowchart of a video frame annotation method provided by an embodiment of this disclosure is shown;

[0088] Figure 2 A schematic diagram illustrating a video frame-skipping strip provided in an embodiment of this disclosure is shown;

[0089] Figure 3 A schematic diagram illustrating a video frame extraction process provided in an embodiment of this disclosure is shown.

[0090] Figure 4 A schematic diagram of a video annotation page provided in an embodiment of this disclosure is shown;

[0091] Figure 5 This diagram illustrates an operation schematic of video frame annotation provided in an embodiment of the present disclosure;

[0092] Figure 6 This illustration shows a specific implementation diagram of a video frame annotation method provided in this disclosure.

[0093] Figure 7 A schematic diagram of a video frame annotation apparatus provided in an embodiment of this disclosure is shown;

[0094] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0096] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0097] In this article, "multiple or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0098] Research has shown that while neural network models can be used to annotate the video segments corresponding to each action instance performed by a target object within a video to be annotated, the complexity of the TAD task and the diversity of application scenarios result in low model prediction accuracy. In other words, the accuracy of the video frame annotations output by the model for the video to be annotated is low. To improve the accuracy of the identified action instances and video segments, the common method is for annotators to watch the video frame by frame and annotate based on the viewing results. However, when the video to be annotated is long or numerous, this method of watching and annotating frame by frame is inefficient. Therefore, improving the speed and efficiency of video frame annotation has become an urgent problem to be solved.

[0099] Based on the above research, this disclosure provides a video frame annotation method, apparatus, computer device, and storage medium. Since the video frames in the displayed video frame extraction band are extracted from the video to be annotated, the number of video frames in the video frame extraction band is necessarily less than the number of video frames in the video to be annotated. Using video frames from the video frame extraction band for video annotation reduces the number of video frames that need to be processed, improving the speed and efficiency of video frame annotation. Furthermore, since the first area can simultaneously display a first preset number of video frames, i.e., multiple frames can be displayed simultaneously, it facilitates simultaneous browsing of at least some of the displayed video frames. This allows for intuitive and rapid location of the target video to be annotated from the first preset number of displayed video frames, further improving the efficiency of video frame annotation.

[0100] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0101] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0102] To facilitate understanding of this embodiment, a video frame annotation method disclosed in this disclosure will first be described in detail. The execution subject of the video frame annotation method provided in this disclosure is generally a terminal device or other processing device with certain computing power. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, personal digital assistant device (PDA), handheld device, computer device, etc. In some possible implementations, the video frame annotation method can be implemented by the processor calling computer-readable instructions stored in the memory.

[0103] The video frame annotation method provided in this disclosure is illustrated below, taking a computer device as the execution subject as an example.

[0104] like Figure 1 The flowchart shown is a video frame annotation method provided in an embodiment of this disclosure, which may include the following steps:

[0105] S101: Extract each video frame from the video to be labeled according to the preset time interval.

[0106] Here, a video frame refers to a single image frame in the video to be labeled. The video to be labeled can be a video taken using a camera device in any scene. The video to be labeled may include a target object, and each target object may correspond to one or more action instances to be labeled. One action instance may correspond to multiple consecutive video frames in the video to be labeled. The target object can be any object with the ability to perform actions, such as a person, animal, or other object.

[0107] An action instance can correspond to any kind of action, and this disclosure does not impose any specific limitations. For example, an action instance can correspond to actions such as singing, dancing, playing a musical instrument, drinking water, eating, typing, exercising, playing games, cycling, etc.

[0108] For example, in a live streaming scenario, the video to be labeled can be a live stream video of a musician playing an instrument, and the action instance to be labeled can be the action of the musician playing an instrument. For instance, if the musician plays an instrument both between 1 minute and 3 minutes and between 10 minutes and 12 minutes of the video to be labeled, then 1 minute to 3 minutes corresponds to one action instance to be labeled, and 10 minutes to 12 minutes corresponds to another action instance to be labeled.

[0109] The preset time interval can be set according to the actual annotation needs, and this embodiment does not impose a specific limitation. For example, the preset time interval can be 3 seconds (S), 5 seconds, 10 seconds, 1 minute, three minutes, etc.

[0110] In practice, for the video to be labeled, video frames can be extracted sequentially from the video at preset time intervals. For example, video frames can be extracted at preset time intervals of 5 seconds, that is, one video frame is extracted from the video to be labeled every 5 seconds, thus obtaining the extracted video frames. The extracted video frames can include the video frame corresponding to the 1st second, the 6th second, the 11th second, the 16th second, and so on. Optionally, in addition to setting a preset time interval, a preset extraction interval can also be preset. The preset extraction interval is used to represent the frame difference between two consecutive video frames extracted from the video to be labeled. Taking a preset extraction interval of 5 frames as an example, one video frame can be extracted from the video to be labeled every 5 frames, thus obtaining the extracted video frames. The extracted video frames can include the 1st, 6th, 11th, 16th, and so on.

[0111] S102: According to the order of each video frame in the video to be annotated, the video frame extraction band is scrolled and displayed in the first area of ​​the video annotation page. The video frame extraction band contains each video frame displayed in sequence; wherein, the number of video frames displayed in the first area at the same time is no greater than the first preset number.

[0112] Here, the first preset number is the maximum number of video frames simultaneously displayed in the first area. When the video frame extraction band is located within the first area, and the number of video frames simultaneously displayed in the video frame extraction band is the first preset number, the area corresponding to the video frame extraction band overlaps with the first area. The video annotation page is the page used to annotate the video to be annotated. The video frame extraction band includes each extracted video frame, and each video frame is displayed sequentially within the frame extraction band. For example... Figure 2 The diagram illustrates a video frame extraction strip provided in this embodiment. The first preset quantity is 12. The video frame extraction strip is located in the first area of ​​the video annotation page. The strip currently contains 12 sequentially displayed video frames: frame 1 at 1 second, frame 2 at 6 seconds, frame 3 at 11 seconds, frame 4 at 16 seconds, frame 5 at 21 seconds, and so on, up to frame 12 at 56 seconds. Each video frame can be scrolled to the left within the strip. After the frame corresponding to 1 second is displayed, the frame corresponding to 61 seconds will be displayed after the frame corresponding to 56 seconds. When the last extracted video frame is displayed, the display can restart from the frame corresponding to 1 second.

[0113] The frame extraction band may include a timeline located in the middle, through which each video frame can sequentially pass. Optionally, for... Figure 2 In this case, when starting to scroll through the video frame extraction strip in the first area, the video frame corresponding to the 1st second can be placed on the timeline first, and then the extracted video frames can be displayed sequentially. For example... Figure 3 The diagram shown illustrates a video frame extraction process according to an embodiment of this disclosure. Figure 3 The 'a' in the diagram indicates that when the video frame skipping starts scrolling, video frame 1 corresponding to the 1st second is placed on the timeline, and the area to the left of the timeline is a blank area. Figure 3 In the text, 'b' indicates that after scrolling for a period of time, the video frame 1 corresponding to the 1st second is located on the left side of the video frame skipping band. The video frame displayed at the timeline is the video frame 5 corresponding to the 21st second. The rightmost part of the video frame skipping band is a display diagram of the video frame 11 corresponding to the 51st second. Among them, only a part of the video frame 11 is shown. Figure 3 The 'c' in the diagram represents a scenario where, after a period of scrolling, video frame 1 at the 1st second is located at the far left of the video frame skipping band, and video frame 12 at the 56th second is located at the far right of the video frame skipping band. The timeline overlaps with the starting position of video frame 7. In the display diagrams corresponding to 'a' and 'b', the number of video frames simultaneously displayed in the first area is less than the first preset number.

[0114] For example, a video frame extraction strip can be constructed based on the order of the extracted video frames in the video to be labeled, and then the video frame extraction strip can be scrolled and displayed in the first area of ​​the video labeling page.

[0115] S103: In response to the annotation operation for the target video frame in the video frame extraction band, an annotated video segment determined based on the target video frame is obtained.

[0116] Here, the target video frame can be any frame in the video frame extraction band. The annotation operation can be a user-initiated operation to specify the annotation of the target video frame. The annotated video segment is a video segment in the video to be annotated where the target object performs the target action corresponding to the action instance. There can be at least one annotated video segment, and the number of annotated video segments can vary in different videos to be annotated.

[0117] For example, in response to the annotation operation for the video frame corresponding to the 11th second displayed in the video frame extraction band, and the annotation operation for the video frame corresponding to the 21st second, the video segment corresponding to the 11th second to the 21st second in the video to be annotated can be taken as an annotated video segment.

[0118] Since each video frame is extracted using a preset time interval, for the video to be labeled, there may be multiple video frames that are not extracted within the preset time interval. Therefore, a single video frame in the video extraction band can represent all the video frames that were not extracted within the preset time interval corresponding to that video frame, as well as the video frame itself. For example, for the video frame corresponding to the 1st second, it can represent all the video frames corresponding to the 1st to 5th seconds. Furthermore, to ensure that the total duration of each video frame in the video extraction band is consistent with the total duration of the video to be labeled, the time for each video frame to pass through the timeline corresponding to the video extraction band can be made equal to the preset interval. For example, for... Figure 3 In the 'a', the time elapsed from the leftmost boundary line of the video frame corresponding to the 1st second aligning with the time line to the rightmost boundary line of the video frame corresponding to the 1st second aligning with the time line can be a preset time interval.

[0119] In practice, when any position of any target video frame aligns with the timeline, in response to the annotation operation, the video frame in the video to be annotated that matches the time corresponding to that position is used as the starting video frame for annotated video segment. For example, in response to the annotation operation, if the middle position of the video frame corresponding to the 1st second is located on the timeline, then the video frame in the video to be annotated that matches the time corresponding to the middle position (i.e., the 3rd second) can be used as the starting video frame for annotated video segment. The video frame corresponding to the 3rd second in the video to be annotated is the starting video frame determined based on the target video frame and used as an annotated video segment. During the process of scrolling through the extracted video frames, the various annotated video segments in the video to be annotated can be determined.

[0120] In this way, since the video frames in the displayed video frame extraction band are extracted from the video to be labeled, the number of video frames in the video frame extraction band is necessarily less than the number of video frames in the video to be labeled. Using the video frames in the video frame extraction band for video annotation can reduce the number of video frames that need to be processed, thereby improving the speed and efficiency of video frame annotation. Furthermore, since the first area can display a first preset number of video frames simultaneously, that is, multiple frames can be displayed at the same time, which facilitates the simultaneous browsing of at least some of the displayed video frames. This allows for intuitive and quick location of the target video to be labeled from the first preset number of displayed video frames, further improving the efficiency of video frame annotation.

[0121] In one embodiment, to further improve annotation accuracy, before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the video to be annotated can also be played in the second area of ​​the video annotation page; wherein, during the playback of the video to be annotated, the video frame scrolled to the center line position of the first area is matched with the video frame currently being played in the video to be annotated.

[0122] For example, such as Figure 4 The diagram shown is a schematic of a video annotation page provided in an embodiment of this disclosure. The first area displays a video frame extraction strip, and the second area plays the video to be annotated. The center line of the first area corresponds to the timeline in the video frame extraction strip.

[0123] In practical implementation, the scrolling speed of each video frame in the video frame extraction band can be set so that the time it takes for a video frame in the first region to pass the center line matches a preset time interval. This ensures that the video frame scrolling to the center line of the first region is consistent with the currently playing video frame in the video to be labeled. Specifically, the time corresponding to the video frame at the center line of the first region can be made consistent with the time corresponding to the currently playing video frame in the video to be labeled. In other words, when one video frame in the video frame extraction band passes the timeline, multiple video frames matching the preset time interval are played in the video to be labeled. For example, for... Figure 4 The video frame extraction band in the video shows that the video frame corresponding to the 1st second passes through the timeline, and the video to be labeled plays the video frames located from the 1st second to the 5th second.

[0124] For example, the video frame extraction band corresponds to the timeline (i.e., the video timeline) from left to right, and the length of the timeline is the same as the duration of the video to be annotated. Each position of each video frame in the video frame extraction band corresponds to a moment when it crosses the timeline. Figure 3 As shown in b, when the P position of video frame 5 corresponding to the 21st second crosses the timeline, the corresponding time is 0:0:23, 0 hours 0 minutes 23 seconds.

[0125] Furthermore, for 103, in response to the annotation operation of the target video frame at the center line position, the annotated video segment determined according to the target video frame can be obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second area and the video frame strip scrolling in the first area.

[0126] Here, the target video frame is the video frame located at the center line.

[0127] For example, in Figure 4When the target video frame in the video frame extraction band reaches the center line (i.e., the timeline) and is passing through the center line, the user can initiate an annotation operation when the target object begins to perform the target action, based on the video frames in the video to be annotated played in the second area. The position where the center line overlaps with the target video frame is designated as the starting point of the annotated video segment, and the time corresponding to this overlap is taken as the start time of the annotated video segment. Afterwards, the video frame extraction band can continue to scroll and display subsequent video frames. If a new target video frame appears, the user can initiate an annotation operation when the target object finishes performing the target action, based on the video frames in the video to be annotated played in the second area. The position where the center line overlaps with the new target video frame is designated as the end point of the annotated video segment, and the time corresponding to this overlap is taken as the end time of the annotated video segment. In this way, an annotated video segment can be obtained. Then, while continuing to scroll and display the various video frames, the annotation of the video to be annotated can be completed, identifying the various annotated video segments included in the video to be annotated.

[0128] In this way, by displaying a video frame extraction strip in the first area and the video to be labeled in the second area, and ensuring that the video frame at the center line of the first area matches the currently playing video frame of the video to be labeled, it is easy to quickly locate the vicinity of the video segment to be labeled by using the video frame extraction strip, and then determine the accurate labeling time and complete the labeling by matching the playing video to be labeled. By using the first and second areas in combination, the accuracy of video frame labeling can be improved.

[0129] In one embodiment, to further improve the video annotation speed, while scrolling through the video frame extraction strip in the first area of ​​the video annotation page according to the order of each video frame in the video to be annotated, in response to the selection operation of any video frame currently displayed in the first area, the selected video frame is displayed at the center line position of the first area, and the playback progress of the video to be annotated in the second area is aligned with the time corresponding to the video frame.

[0130] Here, the selection operation for a video frame can be the user clicking on that video frame.

[0131] For example, when the currently displayed video frame at the center line of the first area is the video frame corresponding to 26 seconds, the user can determine, by examining the various video frames displayed in the video frame extraction band, that the target object can only perform the target action at the video frame corresponding to 26 seconds. In response to the user's click on the video frame corresponding to 26 seconds, the video frame corresponding to 26 seconds can be displayed at the center line of the first area, and the playback progress of the video to be annotated in the second area can be aligned with the time corresponding to that video frame (i.e., the video to be annotated in the second area jumps to 26 seconds for playback). Afterwards, the user can determine the time to initiate the annotation operation based on the video to be annotated playing in the second area. In this way, by responding to the user's selection operation, frame-skipping playback can be achieved, improving annotation speed, while avoiding the situation where video segments to be annotated are missed due to frame-skipping playback directly in the video to be annotated, thus improving annotation accuracy.

[0132] Furthermore, if a user discovers, based on the video frames displayed in the video frame extraction strip, that the target action begins at frame 26 and ends at frame 41, then in response to the user clicking on frame 41, the video frame corresponding to frame 41 can be displayed at the center line of the first area, and the playback progress of the video to be annotated in the second area can be aligned with the time corresponding to that frame (i.e., the video to be annotated in the second area jumps to frame 41 for playback). The user can then determine the time to initiate the annotation operation based on the video to be annotated playing in the second area. In this way, based on two consecutive annotation operations, the video segment corresponding to an action instance can be annotated, resulting in an accurately annotated video segment.

[0133] In another implementation, since the video frame extraction band displays a timeline from left to right, users can skip frames by clicking on video frames or by clicking on the timeline position corresponding to any video frame. For example, in response to a user clicking on the timeline position corresponding to any video frame, the timeline position at the center line can be jumped to the timeline position corresponding to the clicked video frame. At this time, the video frame displayed at the center line position will also become the clicked video frame. Furthermore, the moment matching the timeline position of the clicked video frame can be determined, and the playback progress of the video to be annotated in the second area can be aligned with that moment. Thus, users can skip frames and annotate by clicking on the timeline, improving the flexibility of video annotation.

[0134] In another embodiment, to further improve the video annotation speed, the first video frame currently being displayed can be determined at the center line position of the first area in the first area in response to the display progress adjustment operation, while the video frame extraction band is scrolling and displayed in the first area according to the order of each video frame in the video to be annotated.

[0135] For example, the progress adjustment operation can be the user clicking the progress adjustment keys on the keyboard (such as the A and D keys). The progress adjustment operation can include controlling playback forward and controlling playback backward. For example, the A key plays backward, and the D key plays forward.

[0136] In practice, in response to the display progress adjustment operation, the first video frame currently being displayed can be located at the center line position of the first area.

[0137] Then, the progress adjustment direction corresponding to the operation can be adjusted according to the display progress, and the second video frame corresponding to the first video frame can be determined according to the preset adjustment interval, and the second video frame can be displayed at the center line position.

[0138] Here, the progress adjustment direction can include forward playback and backward playback, and the progress adjustment direction can be determined based on the button pressed by the user. The preset adjustment interval can be a preset adjustment duration and / or a preset adjustment frame count.

[0139] Taking a preset adjustment duration of 10 seconds and a forward playback direction as an example, after determining the first video frame, the corresponding jump time 10 seconds later can be determined based on the time corresponding to the first video frame. The video frame corresponding to this jump time is then used as the second video frame and displayed at the center line. Simultaneously, the playback progress of the video to be labeled in the second area can be aligned with the jump time.

[0140] Alternatively, in response to a progress adjustment operation, the overlap position between the center line of the first area and the first video frame can be determined. Based on the time corresponding to this overlap position and a preset adjustment duration, the jump time can be determined. Further, a second video frame can be determined based on the jump time. Then, a target position within the second video frame that matches the jump time can be determined. Afterward, the second video frame can be displayed at the center line, with the target position overlapping the center line. Simultaneously, the playback progress of the video to be annotated in the second area can be aligned with the jump time.

[0141] Taking a preset adjustment interval of 5 frames and a forward playback direction as an example, the second video frame, which is 5 frames away from the first video frame and is displayed before the first video frame, can be moved to the center line position for display. Simultaneously, the playback progress of the video to be annotated in the second area can be aligned with the jump time corresponding to the second video frame.

[0142] In another embodiment, to further improve the video annotation speed, while scrolling through the video frame extraction band in the first area of ​​the video annotation page according to the order of each video frame in the video to be annotated, the target time corresponding to the center line position can be determined in response to a drag operation on any video frame in the video frame extraction band after the drag operation ends.

[0143] For example, a user can drag any currently displayed video frame in the video frame extraction band. The computer device can respond to this dragging operation and determine the target time corresponding to the center line position after the dragging operation ends. For instance, the time corresponding to the center line position at the start of the dragging operation might be time 1, and the target time corresponding to the center line position after the dragging operation ends could be time 2. Depending on the dragging direction, time 1 can be earlier or later than time 2.

[0144] Furthermore, at the midline position, matching video frames whose playback time matches the target time are displayed, and the playback progress of the video to be labeled in the second area is aligned with the target time.

[0145] In practice, the process can begin by identifying the video frames that match the target time. Then, the matching position within each video frame that corresponds to the target time can be determined. Finally, the matching video frame can be displayed at the center line, with the matching position overlapping the center line. Simultaneously, the playback progress of the video to be labeled in the second area can be aligned with the target time.

[0146] Alternatively, the matching video frames can be displayed directly at the center line, where the starting position of the matching video frames overlaps with the center line. The starting position of a video frame can be understood as the position where the video frame first passes through the center line.

[0147] In one embodiment, to manage the individual video frames included in the video frame extraction band, the video frames can be grouped during the extraction process. Specifically, for S101, each video frame can be extracted from the video to be labeled at a preset time interval, and the video frames can be divided into at least one video frame group according to the order of each video frame in the video to be labeled; wherein the number of video frames in a video frame group is a second preset number.

[0148] Here, the second preset quantity is the maximum number of video frames in a video frame group. The order of the video frames in the video to be labeled is the corresponding playback order of the video frames in the video to be labeled.

[0149] Taking a second preset quantity of 12 as an example, after extracting video frames from the video to be labeled at preset time intervals, the extracted video frames are then grouped into video frame groups according to their order within the video to be labeled. Each video frame group contains 12 video frames. The video frame groups have a sequential order, determined by the order of the video frames within that group within the video to be labeled. For example, the playback order of the video frames in the first video frame group is higher than the playback order of the video frames in the second video frame group.

[0150] After dividing the video frame groups, the process of scrolling and displaying the video frame extraction band in the first area of ​​the video annotation page can also respond to the video frame group switching operation to determine the first video frame group corresponding to the currently displayed target video frame at the center line position of the first area.

[0151] Here, video frame group operations can be performed by the user pressing the video frame group toggle keys on the keyboard (such as the W and S keys). Video frame group operations correspond to different switching directions. Specifically, the switching direction can include switching to the previous group and switching to the next group.

[0152] For example, in response to a user clicking the S key, the first video frame group in which the currently displayed target video frame is located can be determined at the center line of the first region.

[0153] Then, according to the switching direction indicated by the video frame group switching operation, the first video frame of the second video frame group adjacent to the first video frame group can be displayed at the center line position.

[0154] Taking the user clicking the S key as an example of switching video frame groups, the determined switching direction can be to switch to the next group. Then, the playback order of the second video frame group, which is located after the first video frame group and adjacent to the first video frame group, can be determined. Afterward, the first frame of the second video frame group can be displayed at the center line position.

[0155] In this way, by switching video frame groups and grouping the extracted video frames, the maximum span of frame skipping playback can reach a second preset number of frames, improving the flexibility and speed of the frame skipping playback method. Adjusting the playback in this way before annotating the video can further increase the speed of video annotation.

[0156] In one embodiment, the annotation operation includes a marker creation operation. The marker creation operation can be an action performed by a user clicking a marker creation button (such as the Q key). For S103, the following steps can be followed:

[0157] S103-1: During the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the first target video frame in the video frame extraction band, determine the first target time corresponding to the center line position of the first region as the pairing annotation position, and determine whether there is a generated pairing annotation position.

[0158] Here, the first target video frame is the video frame displayed at the center line position. The pairing and marking position is the position where the center line of the first region overlaps with the first target video frame after the user initiates the mark creation operation (e.g., Figure 3 The P position (indicated by b) is also the video position corresponding to the first target time in the video to be labeled. The generated unpaired label positions are the label positions generated before the unpaired label positions corresponding to the first target time were determined. The first target time can be the position at the center line of the first region that overlaps with the first target video frame, corresponding to the time on the timeline.

[0159] In practice, labeling a video segment requires two label creation operations: one for the start time and one for the end time. The corresponding label positions from these two operations will be used as a pair of matching label positions. Therefore, after generating the matching label positions for the first target time, it's necessary to first determine if any already generated matching label positions exist. If so, the current label creation operation is for labeling the end time, which can complete the match. The video segment located between the time corresponding to the already generated matching label position and the first target time will be labeled as a single labeled video segment, and the matching label position from the current operation and the already generated matching label position will be considered as matched label positions. If not, the current label creation operation is for labeling the start time.

[0160] It should be noted that the marker creation operation will not affect the display process of video frames in the video frame extraction band; it is only used to create marker positions.

[0161] S103-2: If not, then during the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the second target video frame in the video frame extraction band, determine the second target time corresponding to the center line position as the new pairing annotation position.

[0162] Here, the second target video frame can be a video frame that, during the process of extracting frames from the animated video display, is located at the center line position at a time later than or equal to the time when the first target video frame is located at the center line position. Optionally, the first target video frame and the second target video frame can be the same, but when the two marker bit creation operations are initiated, the overlap position between the center line position and the first target video frame is different, corresponding to different times.

[0163] In practice, if no matching annotation position has been generated, the current marker creation operation is used to mark the start time. Subsequently, during the scrolling display of the video frame extraction strip, in response to the marker creation operation for the second target video frame, the second target time corresponding to the centerline position is determined as the new matching annotation position. The second target time can be the time corresponding to the position on the timeline where the centerline position of the first region overlaps with the second target video frame.

[0164] S103-3: Label the video segment located between the first target time and the second target time as labeled video segment, and take the unpaired label position corresponding to the first target time and the new unpaired label position as the matched label position.

[0165] In practice, the selected area can be the frame-extracted region between the first and second target times within the video frame extraction band, and video segments within the video to be labeled that fall between the first and second target times can be labeled as labeled video segments. Furthermore, after obtaining new labeling positions to be matched, it can be determined whether any already generated labeling positions exist. Here, since the labeling position corresponding to the first target time has already been generated but not yet paired, it can be considered a generated labeling position. Then, the labeling position corresponding to the first target time and the new labeling position can be paired to obtain two matched labeling positions. By using the labeling position corresponding to the first target time and the new labeling position as matched labeling positions, it is possible to prevent newly created labeling positions from being paired with these two positions, thus avoiding errors in video labeling.

[0166] In one embodiment, after marking the video segment located between the first target time and the second target time as a marked video segment with the first target video frame and the second target video frame as the boundary, it is also possible to generate marked segment indication information based on the first target time and the second target time, and display the marked segment indication information in the third area of ​​the video annotation page, and generate and display time segment annotation lines that match the first target time and the second target time on the video timeline corresponding to the video to be annotated in the second area.

[0167] In practice, for each annotated video segment, annotation information can be generated based on the first and second target times corresponding to that segment. This annotation information indicates the time period corresponding to each annotated video segment. The generated annotation information can be displayed in the third area of ​​the video annotation page. Figure 4 The image shows the annotation information 1 corresponding to a labeled video segment 1. Additionally, for each generated annotation segment indication, a delete button can be associated (e.g., ...). Figure 4 The delete button shown can be used to delete the labeled segment information and the labeled video segment corresponding to the labeled segment information.

[0168] In another implementation, the annotation segment can be selected in response to a user's click on any annotation segment indicator in the third area. Then, in response to a user-initiated deletion operation (e.g., clicking the delete button (such as the E key)), the annotation segment indicator and the corresponding labeled video segment can be deleted.

[0169] Additionally, time segment markers are used to indicate the time corresponding to the video segment to be annotated on the video timeline. Specifically, time segment markers can be displayed on the video timeline using methods such as highlighting and bolding. For example... Figure 4 As shown, on the video timeline corresponding to the video to be annotated in the second region, the time segment annotation line 1 corresponding to video annotation segment 1 is displayed.

[0170] In addition, the video timeline in the second area can also display the marked lines for each predicted time segment (such as...). Figure 4 The six predicted time segment annotation lines shown in the image are time marker lines corresponding to each predicted video segment after video annotation is performed using a pre-trained neural network model. Because the prediction accuracy of the neural network model decreases in video annotation tasks, the accuracy of the predicted video segments is poor, resulting in differences between the generated predicted time segment annotation lines and those obtained using the video frame annotation method provided in this embodiment. By displaying the time segment annotation lines and the predicted time segment annotation lines on the video timeline, a comparison can be made between the model annotation results and the target annotation results obtained using the video frame annotation method provided in this embodiment, thus determining the quality of the model annotation. Furthermore, the model parameters of the neural network model can be adjusted using the target annotation results, thereby improving the model's prediction accuracy.

[0171] In one embodiment, after obtaining the labeled video segment determined based on the target video frame in response to the labeling operation for the target video frame in the video frame extraction band, the system can also align the first video frame corresponding to the labeled video segment with the center line position of the first region in response to the selection operation for any labeled video segment, and align the playback progress of the video to be labeled in the second region with the start time of the labeled video segment.

[0172] Here, the selection operation for any labeled video segment can be the user double-clicking the labeled segment indicator information in the third area, the user double-clicking the time segment label line on the video timeline, or the user double-clicking the selected area corresponding to the labeled video segment in the video frame extraction band.

[0173] For example, after annotating each video segment from the video to be annotated, in response to a double-click operation on annotated video segment 1, the first video frame corresponding to annotated video segment 1 can be aligned with the center line of the first region, and the playback progress of the video to be annotated in the second region can be aligned with the start time of the annotated video segment. The time corresponding to the position where the first video frame overlaps with the center line of the first region is the start time of annotated video segment 1.

[0174] For example, in response to a double-click operation on the selected area 1 corresponding to the labeled video segment 1, the starting position of the selected area 1 can be aligned with the center line of the first area, and the playback progress of the video to be labeled in the second area can be aligned with the starting time of the labeled video segment 1.

[0175] Furthermore, after aligning the first video frame corresponding to labeled video segment 1 with the center line of the first region, the system can also respond to user deletion operations on labeled video segment 1 by deleting labeled video segment 1, deselecting the selection area 1 corresponding to labeled video segment 1, and deleting the labeled segment indicator information 1 corresponding to labeled video segment 1. The deletion operation can be an operation where the user clicks the E key, or an operation where the user clicks the delete button associated with labeled segment indicator information 1, etc.

[0176] like Figure 5 The diagram shown illustrates an operation for video frame annotation according to an embodiment of this disclosure, wherein the user can annotate video frames... Figure 5 The video frame extraction function performs the following operations: ① drag video frames, ② select any video frame ( Figure 5 (See the example diagram for clicking video frame 7), ③ Click the selected area, ④ Initiate marker creation operation (Q key), ⑤ Switch to the previous group of operations (W key), ⑥ Switch to the next group of operations (S key), ⑦ Play forward operation (D key), ⑧ Play backward operation (A key), ⑨ Delete the selected area (E key).

[0177] like Figure 6 The diagram illustrates a specific implementation of video frame annotation according to an embodiment of this disclosure. After obtaining the video to be annotated, a video frame extraction band can be displayed in the first area of ​​the video annotation page. Optionally, a neural network model can be used to pre-annotate the video to be annotated, determining each predicted video segment and displaying the prediction area corresponding to each predicted video segment. Then, the video frame annotation method provided in this embodiment can be used for annotation. The annotation process can involve time selection, which depends on the operations ①, ②, ⑤, ⑥, ⑦, and ⑧ described above. Afterwards, by executing operation ④, the selection can begin and end, resulting in selected areas and annotated video segments. Simultaneously, annotation segment indication information and its associated delete button can be generated and displayed.

[0178] Furthermore, the system can first respond to double-clicking the selected area or clicking the labeled segment indicator information, and then respond to the delete operation to delete the selected area, labeled segment indicator information, labeled video segments, etc. The delete operation can include clicking the E key or clicking the delete button.

[0179] In one embodiment, the labeled video segment can be a video segment in the video to be labeled where the target object performs the target action. To further improve the video labeling speed, before obtaining the labeled video segment determined based on the labeled video frame in response to the labeling operation of the target video frame in the video frame extraction band, an action boundary detection algorithm can be used to extract each key frame that matches the action boundary of the target action from the video to be labeled, and display the key frame band in the fourth area of ​​the video labeling page in the order of each key frame in the video to be labeled. The key frame band contains each key frame displayed in sequence; the key frames corresponding to the action boundary include the video frame when the target object starts to perform the target action and the video frame when the target object stops performing the target action.

[0180] Here, the keyframes that match the action boundaries include the video frame when the target object begins to perform the target action, and the video frame when the target object ends to perform the target action.

[0181] Boundary detection algorithms include, but are not limited to, frame difference methods, optical flow methods, and action boundary detection algorithms, which are recognition techniques that can mark important scenes or scene transitions. The keyframe band contains sequentially displayed keyframes, and this keyframe band can be scrolled through in the fourth area of ​​the video annotation page.

[0182] A keyframe can correspond to a video frame in a video frame extraction band. For example, if the keyframe corresponds to time 1, the corresponding video frame can be the video frame at time 1, or it can be any video frame at the time corresponding to the midline position that includes time 1. For example, if the video frame at time 1 passes the midline position at times 1S, 2S, 3S, 4S, and 5S, then if the keyframe corresponds to time 3S, the corresponding video frame is the same as the video frame at time 1S.

[0183] Furthermore, after displaying the keyframe strip, in response to a trigger operation for any keyframe, the keyframe moment corresponding to that keyframe can be determined, and a video frame matching the keyframe moment can be displayed at the center line position of the first region. Additionally, the playback progress of the video to be annotated in the second region can be aligned with the keyframe moment. Simultaneously, triggered keyframes can be displayed at the timelines within the keyframe strip, where the starting position of the keyframe overlaps with the timeline within the keyframe strip.

[0184] Here, the keyframe moment is the moment corresponding to the position where the keyframe overlaps with the timeline in the keyframe band, or it can be directly the moment corresponding to the keyframe.

[0185] For example, in response to a trigger operation on keyframe 1, the keyframe moment 1 corresponding to keyframe 1 is determined. Then, video frame 1 matching keyframe moment 1 is determined, and position 1 within video frame 1 matching keyframe moment 1 is determined. Afterward, video frame 1 can be displayed at the center line position, with position 1 in video frame 1 overlapping the center line position. Simultaneously, the playback progress of the video to be annotated in the second region can be aligned with keyframe moment 1. In this way, synchronization is achieved among the video frame extraction strip, the keyframe strip, and the video to be annotated.

[0186] Alternatively, if the keyframe time is the same as the keyframe time, after determining the video frame 1 that matches keyframe time 1, video frame 1 can be directly displayed at the center line position of the first region, where the center line position overlaps with the start position of video frame 1. Simultaneously, the playback progress of the video to be annotated in the second region can be aligned with keyframe time 1, that is, aligned with the time corresponding to the start position of video frame 1.

[0187] Then, in response to the annotation operation on the target video frame in the video frame extraction band, the annotated video segment determined based on the target video frame can be obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second area, the video frame extraction band displayed in the first area, and the keyframe band displayed in the fourth area.

[0188] Here, users can determine the timing and initiate annotation operations based on the video to be annotated played in the second area, the video frame extraction strip displayed in the first area, and the keyframe strip displayed in the fourth area, thereby achieving accurate annotation of the video to be annotated.

[0189] By responding to user clicks on keyframes in the keyframe strip, the system synchronizes the keyframe strip, the video frame extraction strip, and the video frames in the video to be annotated. This allows users to quickly locate the video frames that may need annotation based on the keyframes. By annotating the video frames played in the video to be annotated and the video frames displayed in the video frame extraction strip, precise annotation of the action instances performed on the target object can be achieved, resulting in accurate annotated video segments.

[0190] In the above embodiments, by extracting each keyframe that matches the action boundary of the target action and displaying the keyframe band in the fourth area, it is convenient to quickly determine the matching video frames that match the keyframes in the extracted video frame band based on the keyframe band. Then, the matching video frames and the video to be labeled are labeled, which can not only ensure the accuracy of video frame labeling, but also improve the speed and efficiency of video frame labeling.

[0191] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0192] Based on the same inventive concept, this disclosure also provides a video frame annotation device corresponding to the video frame annotation method. Since the principle of the device in this disclosure for solving the problem is similar to the video frame annotation method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0193] like Figure 7 The diagram shown is a schematic representation of a video frame annotation device provided in an embodiment of this disclosure, comprising:

[0194] The extraction module 701 is used to extract each video frame from the video to be labeled according to a preset time interval.

[0195] The display module 702 is used to scroll and display a video frame extraction band in a first area of ​​the video annotation page according to the order of the video frames in the video to be annotated. The video frame extraction band contains the video frames displayed in sequence. The number of video frames displayed in the first area at the same time is not greater than a first preset number.

[0196] The annotation module 703 is used to respond to the annotation operation of the target video frame in the video frame extraction band to obtain the annotated video segment determined according to the target video frame.

[0197] In one possible implementation, the display module 702, before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is further configured to:

[0198] The video to be annotated is played in the second area of ​​the video annotation page; wherein, during the playback of the video to be annotated, the video frame that scrolls to the center line position of the first area matches the currently playing video frame of the video to be annotated.

[0199] The annotation module 703, when obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is configured to:

[0200] In response to the annotation operation on the target video frame at the midline position, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region and the video frame strip scrolling in the first region.

[0201] In one possible implementation, the annotation operation includes a marker bit creation operation; the annotation module 703, when obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is configured to:

[0202] During the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the first target video frame in the video frame extraction band, the first target time corresponding to the center line position of the first region is determined as the pairing annotation position, and it is determined whether there is a generated pairing annotation position.

[0203] If not, during the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the second target video frame in the video frame extraction band, the second target time corresponding to the center line position is determined as the new pairing annotation position.

[0204] The video segment located between the first target time and the second target time is labeled as a labeled video segment, and the unpaired label position corresponding to the first target time and the new unpaired label position are taken as the matched label position.

[0205] In one possible implementation, the display module 702, after obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is configured to:

[0206] In response to a selection operation for any labeled video segment, the first video frame corresponding to the labeled video segment is aligned with the center line of the first region, and the playback progress of the video to be labeled in the second region is aligned with the start time of the labeled video segment.

[0207] In one possible implementation, after labeling the video segment located between the first target time and the second target time as a labeled video segment bounded by the first target video frame and the second target video frame, the labeling module 703 is further configured to:

[0208] Based on the first target time and the second target time, annotation segment indication information is generated and displayed in the third area of ​​the video annotation page. Additionally, on the video timeline corresponding to the video to be annotated in the second area, time segment annotation lines matching the first target time and the second target time are generated and displayed.

[0209] In one possible implementation, the display module 702, while scrolling and displaying the video frame extraction strip in the first area of ​​the video annotation page according to the sequential order of each video frame in the video to be annotated, is further configured to:

[0210] In response to a drag operation on any video frame in the video frame extraction band, a target time corresponding to the centerline position is determined after the drag operation ends.

[0211] At the midline position, matching video frames whose playback time matches the target time are displayed, and the playback progress of the video to be labeled in the second region is aligned with the target time.

[0212] In one possible implementation, the display module 702, while scrolling and displaying the video frame extraction strip in the first area of ​​the video annotation page according to the sequential order of each video frame in the video to be annotated, is further configured to:

[0213] In response to a selection operation on any video frame currently displayed in the first region, the selected video frame is displayed at the center line position of the first region, and the playback progress of the video to be labeled in the second region is aligned with the time corresponding to the video frame.

[0214] In one possible implementation, the extraction module 701, when extracting video frames from the video to be labeled according to a preset time interval, is used to:

[0215] According to a preset time interval, each video frame is extracted from the video to be labeled, and the video frames are divided into at least one video frame group according to the order of each video frame in the video to be labeled; wherein, the number of video frames in a video frame group is a second preset number.

[0216] The display module 702, after dividing the video frame into at least one video frame group, is further configured to:

[0217] In response to a video frame group switching operation, the first video frame group corresponding to the currently displayed target video frame is determined at the center line position of the first region;

[0218] According to the switching direction indicated by the video frame group switching operation, the first video frame of the second video frame group adjacent to the first video frame group is displayed at the center line position.

[0219] In one possible implementation, the display module 702, while scrolling and displaying the video frame extraction strip in the first area of ​​the video annotation page according to the sequential order of each video frame in the video to be annotated, is further configured to:

[0220] In response to a display progress adjustment operation, determine the first video frame currently being displayed at the center line position of the first region;

[0221] Based on the progress adjustment direction corresponding to the display progress adjustment operation, and according to the preset adjustment interval, determine the second video frame corresponding to the first video frame, and display the second video frame at the center line position;

[0222] Align the playback progress of the video to be labeled in the second region with the moment when the second video frame is matched.

[0223] In one possible implementation, the labeled video segment is a video segment in the video to be labeled where the target object performs the target action;

[0224] Before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the extraction module 701 is further configured to:

[0225] Using an action boundary detection algorithm, key frames that match the action boundary of the target action are extracted from the video to be labeled; the key frames corresponding to the action boundary include the video frame when the target object starts to perform the target action and the video frame when the target object stops performing the target action.

[0226] The display module 702, before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is further configured to:

[0227] According to the order of the keyframes in the video to be annotated, a keyframe band is scrolled and displayed in the fourth area of ​​the video annotation page, and the keyframe band contains the keyframes displayed in sequence.

[0228] The annotation module 703, when obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, is configured to:

[0229] In response to a trigger operation for any keyframe, the keyframe moment corresponding to the keyframe is determined, and a video frame matching the keyframe moment is displayed at the center line position of the first region, and the playback progress of the video to be annotated in the second region is aligned with the keyframe moment.

[0230] In response to the annotation operation on the target video frame in the video frame extraction band, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region, the video frame extraction band displayed in the first region, and the keyframe band displayed in the fourth region.

[0231] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0232] Based on the same technical concept, embodiments of this application also provide a computer device. (Refer to...) Figure 8 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application, comprising:

[0233] The system includes a processor 81, a memory 82, and a bus 83. The memory 82 stores machine-readable instructions executable by the processor 81. The processor 81 executes these machine-readable instructions, and when executed, performs the following steps: S101: Extracting video frames from the video to be annotated at preset time intervals; S102: Displaying a video frame extraction strip in a first area of ​​the video annotation page, according to the order of the video frames in the video to be annotated, where the video frame extraction strip contains sequentially displayed video frames; wherein the number of video frames simultaneously displayed in the first area is no greater than a first preset number; and S103: Responding to an annotation operation on a target video frame in the video frame extraction strip, obtaining an annotated video segment determined based on the target video frame.

[0234] The aforementioned memory 82 includes a main memory 821 and an external memory 822. The main memory 821, also known as internal memory, is used to temporarily store the computational data in the processor 81, as well as the data exchanged with external memory such as a hard disk. The processor 81 exchanges data with the external memory 822 through the main memory 821. When the computer device is running, the processor 81 and the memory 82 communicate through the bus 83, so that the processor 81 executes the execution instructions mentioned in the above method embodiments.

[0235] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the video frame annotation method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0236] The computer program product of the video frame annotation method provided in this disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the video frame annotation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0237] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0238] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0241] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0242] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0243] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A video frame annotation method, characterized in that, include: Each video frame is extracted from the video to be labeled according to a preset time interval; According to the order of the video frames in the video to be annotated, a video frame extraction band is scrolled and displayed in the first area of ​​the video annotation page. The video frame extraction band contains the video frames displayed in sequence. The number of video frames displayed in the first area at the same time is no greater than a first preset number. In response to the annotation operation on the target video frame in the video frame extraction band, an annotated video segment determined based on the target video frame is obtained, wherein the annotated video segment is a video segment in the video to be annotated where the target object performs the target action corresponding to the action instance. The annotation operation includes a marker bit creation operation; the step of obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band includes: During the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the first target video frame in the video frame extraction band, the first target time corresponding to the center line position of the first region is determined as the pairing annotation position, and it is determined whether there is a generated pairing annotation position. If not, during the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the second target video frame in the video frame extraction band, the second target time corresponding to the center line position is determined as the new pairing annotation position. The video segment located between the first target time and the second target time is labeled as a labeled video segment, and the unpaired label position corresponding to the first target time and the new unpaired label position are taken as the matched label position.

2. The method according to claim 1, characterized in that, Before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the method further includes: The video to be annotated is played in the second area of ​​the video annotation page; wherein, during the playback of the video to be annotated, the video frame that scrolls to the center line position of the first area matches the currently playing video frame of the video to be annotated. The step of obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band includes: In response to the annotation operation on the target video frame at the midline position, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region and the video frame strip scrolling in the first region.

3. The method according to claim 1, characterized in that, After obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the method further includes: In response to a selection operation for any labeled video segment, the first video frame corresponding to the labeled video segment is aligned with the center line of the first region, and the playback progress of the video to be labeled in the second region is aligned with the start time of the labeled video segment.

4. The method according to claim 1, characterized in that, After labeling the video segments located between the first target time and the second target time as labeled video segments bounded by the first target video frame and the second target video frame, the method further includes: Based on the first target time and the second target time, annotation segment indication information is generated and displayed in the third area of ​​the video annotation page. Additionally, on the video timeline corresponding to the video to be annotated in the second area, time segment annotation lines matching the first target time and the second target time are generated and displayed.

5. The method according to claim 2, characterized in that, The process of scrolling and displaying the video frame extraction band in the first area of ​​the video annotation page according to the chronological order of each video frame in the video to be annotated also includes: In response to a drag operation on any video frame in the video frame extraction band, a target time corresponding to the centerline position is determined after the drag operation ends. At the midline position, matching video frames whose playback time matches the target time are displayed, and the playback progress of the video to be labeled in the second region is aligned with the target time.

6. The method according to claim 1, characterized in that, The process of scrolling and displaying the video frame extraction band in the first area of ​​the video annotation page according to the chronological order of each video frame in the video to be annotated also includes: In response to a selection operation on any video frame currently displayed in the first region, the selected video frame is displayed at the center line position of the first region, and the playback progress of the video to be labeled in the second region is aligned with the time corresponding to the video frame.

7. The method according to claim 1, characterized in that, The step of extracting video frames from the video to be labeled according to a preset time interval includes: According to a preset time interval, each video frame is extracted from the video to be labeled, and the video frames are divided into at least one video frame group according to the order of each video frame in the video to be labeled; wherein, the number of video frames in a video frame group is a second preset number. After dividing the video frames into at least one video frame group, the method further includes: In response to a video frame group switching operation, the first video frame group corresponding to the currently displayed target video frame is determined at the center line position of the first region; According to the switching direction indicated by the video frame group switching operation, the first video frame of the second video frame group adjacent to the first video frame group is displayed at the center line position.

8. The method according to claim 1, characterized in that, The labeled video segment is a video segment in the video to be labeled where the target object performs the target action; Before obtaining the annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band, the method further includes: Using an action boundary detection algorithm, keyframes that match the action boundary of the target action are extracted from the video to be labeled. The keyframes are then displayed in a scrolling manner in the fourth area of ​​the video labeling page according to their order in the video. The keyframes contain the keyframes displayed in sequence. The keyframes corresponding to the action boundary include the video frame when the target object starts performing the target action and the video frame when the target object stops performing the target action. The step of obtaining an annotated video segment determined based on the target video frame in response to the annotation operation for the target video frame in the video frame extraction band includes: In response to a trigger operation for any keyframe, the keyframe moment corresponding to the keyframe is determined, and a video frame matching the keyframe moment is displayed at the center line position of the first region, and the playback progress of the video to be annotated in the second region is aligned with the keyframe moment. In response to the annotation operation on the target video frame in the video frame extraction band, an annotated video segment determined based on the target video frame is obtained; wherein, the annotation operation is initiated by the user based on the video to be annotated played in the second region, the video frame extraction band displayed in the first region, and the keyframe band displayed in the fourth region.

9. A video frame annotation device, characterized in that, include: The extraction module is used to extract video frames from the video to be labeled according to a preset time interval. The display module is used to scroll and display a video frame extraction band in a first area of ​​the video annotation page according to the order of the video frames in the video to be annotated. The video frame extraction band contains the video frames displayed in sequence. The number of video frames displayed in the first area is not greater than a first preset number. The annotation module is configured to respond to an annotation operation on a target video frame in the video frame extraction band, and obtain an annotated video segment determined based on the target video frame, wherein the annotated video segment is a video segment in the video to be annotated where the target object performs the target action corresponding to the action instance. The annotation operation includes a marker bit creation operation. When the annotation module obtains an annotated video segment determined based on the target video frame in response to an annotation operation on a target video frame in the video frame extraction band, the annotation module is configured to: During the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the first target video frame in the video frame extraction band, the first target time corresponding to the center line position of the first region is determined as the pairing annotation position, and it is determined whether there is a generated pairing annotation position. If not, during the scrolling display of the video frame extraction band, in response to the marker bit creation operation for the second target video frame in the video frame extraction band, the second target time corresponding to the center line position is determined as the new pairing annotation position. The video segment located between the first target time and the second target time is labeled as a labeled video segment, and the unpaired label position corresponding to the first target time and the new unpaired label position are taken as the matched label position.

10. A computer device, characterized in that, include: The processor and the memory, wherein the memory stores machine-readable instructions executable by the processor, the processor is configured to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the processor performs the steps of the video frame annotation method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer device, performs the steps of the video frame annotation method as described in any one of claims 1 to 8.

Images