Video labeling method and device, equipment and storage medium

Abstract

The invention provides a video labeling method, device and equipment and a storage medium, and the video labeling method comprises the steps: automatically determining a key frame containing a target object from a to-be-labeled video through a key frame matching mode, and segmenting the to-be-labeled video by taking the key frame as a video segmentation point, according to the method, target tracking processing is carried out on two groups of image sequences in front of and behind the key frame in the video to be labeled, so that the video labeling method does not need to independently judge whether the first frame in the video to be labeled contains the target object or not on the basis of not changing the algorithm working principle of the target tracking algorithm; automation of video labeling is realized, and efficiency and accuracy of video labeling are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video annotation method, apparatus, device and storage medium. Background Art

[0002] In scenarios such as excavator operation and loader shoveling materials, it is often necessary to record the working process of the operating vehicles in the scene through video. However, in addition to the target objects related to the video shooting task, such as the operating vehicles and shoveling materials, the captured video often also contains background elements such as trees and warehouses that are not related to the video shooting task. Therefore, in order to improve the data quality of the captured video data, users need to mark the above-mentioned target objects contained in the video data so that they can more intuitively and quickly lock the marked target objects from the marked video.

[0003] At present, target tracking algorithms can be used to track the target objects specified in the video and mark the positions of the target objects in each image frame in the video. However, since target tracking algorithms can usually only track the image sequences in which the target object appears in the first frame and the shooting time is continuous, target tracking algorithms cannot track the video data in which the target object does not appear in the first frame, which is not convenient for realizing the automated process of video annotation, resulting in reduced efficiency and accuracy of video annotation. Summary of the invention

[0004] In view of this, the present application provides a video annotation method, apparatus, device and storage medium, which automatically determine the key frames containing the target object from the video to be annotated by means of key frame matching, and use the key frames as video segmentation points to segment the video to be annotated, and perform target tracking processing on two groups of image sequences located before and after the key frames in the video to be annotated, respectively. Therefore, without changing the algorithm working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame in the video to be annotated contains the target object, thereby realizing the automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0005] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0006] In a first aspect, an embodiment of the present application provides a video annotation method, the video annotation method comprising:

[0007] According to the standard key frame of the target object, an image frame matching the standard key frame is determined from the video to be labeled as the key frame of the target object in the video to be labeled; wherein the standard key frame includes the position mark information of the target object;

[0008] The standard key frame, the key frame and a first image sequence are combined to obtain a first sub-video, and the standard key frame, the key frame and a second image sequence are combined to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be labeled; and the second image sequence represents an image sequence composed of image frames after the key frame in the video to be labeled;

[0009] Taking the target object as a tracking target, performing target tracking processing on the first sub-video and the second sub-video respectively, and obtaining target tracking processing results corresponding to the first sub-video and the second sub-video respectively;

[0010] According to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are reorganized to obtain the target tracking result of the target object in the video to be labeled.

[0011] In a second aspect, an embodiment of the present application provides a video annotation device, the video annotation device comprising:

[0012] A matching module, configured to determine, from the video to be labeled, an image frame matching the standard key frame according to the standard key frame of the target object as the key frame of the target object in the video to be labeled; wherein the standard key frame includes the position mark information of the target object;

[0013] A grouping module, used for combining the standard key frame, the key frame and a first image sequence to obtain a first sub-video, and combining the standard key frame, the key frame and a second image sequence to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be labeled; and the second image sequence represents an image sequence composed of image frames after the key frame in the video to be labeled;

[0014] A target tracking module, configured to use the target object as a tracking target, perform target tracking processing on the first sub-video and the second sub-video respectively, and obtain target tracking processing results corresponding to the first sub-video and the second sub-video respectively;

[0015] The reorganization module is used to reorganize the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the order of the image frames in the video to be labeled, so as to obtain the target tracking result of the target object in the video to be labeled.

[0016] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned video annotation method when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video annotation method are executed.

[0018] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0019] The embodiments of the present application provide a video annotation method, apparatus, device and storage medium, which automatically determine the key frames containing the target object from the video to be annotated by means of key frame matching, and segment the video to be annotated using the key frames as video segmentation points, and perform target tracking processing on two groups of image sequences located before and after the key frames in the video to be annotated, respectively. Therefore, without changing the algorithm working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame in the video to be annotated contains the target object, thereby realizing the automation of video annotation and effectively improving the efficiency and accuracy of video annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a process flow of a video annotation method provided in an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of the structure of a first sub-video and a second sub-video provided in an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of a flow chart of a method for determining a standard key frame provided in an embodiment of the present application is shown;

[0024] Figure 4 A flow chart of a method for performing deduplication processing on target tracking results of multiple target objects in a video to be labeled provided by an embodiment of the present application is shown;

[0025] Figure 5A schematic diagram of the structure of a video annotation device provided in an embodiment of the present application is shown;

[0026] Figure 6 A schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0028] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0029] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0030] At present, target tracking algorithms can be used to track the target objects specified in the video and mark the positions of the target objects in each image frame in the video. However, since target tracking algorithms can usually only track the image sequences in which the target object appears in the first frame and the shooting time is continuous, target tracking algorithms cannot track the video data in which the target object does not appear in the first frame, which is not convenient for realizing the automated process of video annotation, resulting in reduced efficiency and accuracy of video annotation.

[0031] Based on this, the embodiments of the present application provide a video annotation method, apparatus, device and storage medium, which automatically determine the key frames containing the target object from the video to be annotated by means of key frame matching, and use the key frames as video segmentation points to segment the video to be annotated, and perform target tracking processing on two groups of image sequences located before and after the key frames in the video to be annotated, respectively. Therefore, without changing the algorithm working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame in the video to be annotated contains the target object, thereby realizing the automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0032] In one embodiment of the present application, a video annotation method can be run on a terminal device or a server. The terminal device can be a local terminal device. When the video annotation method is run on a server, the video annotation method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device (i.e., a terminal device).

[0033] To facilitate understanding of the embodiments of the present application, a video annotation method, apparatus, device, and storage medium provided in the embodiments of the present application are described in detail below.

[0034] Reference Figure 1 As shown, Figure 1 A schematic flow chart of a video annotation method provided in an embodiment of the present application is shown, wherein the video annotation method includes steps S101-S104; specifically:

[0035] S101, according to a standard key frame of a target object, determining an image frame matching the standard key frame from a video to be labeled as a key frame of the target object in the video to be labeled.

[0036] S102: Combine the standard key frame, the key frame and a first image sequence to obtain a first sub-video, and combine the standard key frame, the key frame and a second image sequence to obtain a second sub-video.

[0037] S103: Taking the target object as a tracking target, target tracking processing is performed on the first sub-video and the second sub-video respectively to obtain target tracking processing results corresponding to the first sub-video and the second sub-video respectively.

[0038] S104: According to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are reorganized to obtain the target tracking result of the target object in the video to be labeled.

[0039] The above-mentioned video annotation method provided in the embodiment of the present application automatically determines the key frame containing the target object from the video to be annotated by means of key frame matching, and divides the video to be annotated using the key frame as the video segmentation point, and performs target tracking processing on two groups of image sequences located before and after the key frame in the video to be annotated, respectively. Therefore, without changing the algorithm working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame in the video to be annotated contains the target object, thereby realizing the automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0040] The following is an exemplary description of each step in the above-mentioned video annotation method provided in the embodiment of the present application:

[0041] S101, according to a standard key frame of a target object, determining an image frame matching the standard key frame from a video to be labeled as a key frame of the target object in the video to be labeled.

[0042] Here, the target object represents the entity object that needs to be labeled in the video to be labeled, wherein the specific entity object type represented by the target object can be determined according to the video shooting scene of the video to be labeled. The target object can represent only one entity object or include multiple entity objects of different categories. The embodiment of the present application does not impose any limitation on the specific entity object type and the specific number of entity objects represented by the target object.

[0043] For example, taking the video shooting scene of the video to be labeled as an excavator operation scene, the target object can be the excavator body (including the arm and bucket), the excavator body, materials to be loaded (such as soil piles, material piles, etc.), trucks transporting materials, etc.

[0044] Here, the annotated key frame includes the position mark information of the target object, that is, the standard key frame can be obtained by marking the target object in the image frame containing the target object.

[0045] It should be noted that the above-mentioned standard key frames may come from the video to be labeled or may not come from the video to be labeled. For example, according to the video shooting scene of the video to be labeled, multiple videos from the video shooting scene can be obtained. It is only necessary to select image frames containing target objects from the multiple videos and mark the target objects contained in the image frames to obtain the above-mentioned standard image frames. The standard image frames obtained at this time are suitable for video labeling of target objects in all videos to be labeled in the video shooting scene, which is conducive to improving the efficiency of video labeling.

[0046] Here, considering that the above-mentioned standard key frame may not belong to the video to be annotated, when executing step S101, the image frame with the highest degree of matching with the above-mentioned standard key frame can be determined from the video to be annotated by feature matching as the key frame of the target object in the video to be annotated (i.e., the image frame matching the above-mentioned standard key frame).

[0047] Specifically, as an optional embodiment, the above step S101 can be performed by ORB (Oriented FAST and rotated BRIEF, a feature detection and feature description algorithm) feature matching. In the ORB feature matching method, for each image frame in the video to be labeled, the ORB algorithm can first detect the key points in the image frame, and then calculate a feature vector for each key point (that is, the ORB feature corresponding to the key point) to obtain multiple ORB features corresponding to the image frame. Similarly, multiple ORB features corresponding to the above standard key frame can also be obtained by the ORB algorithm. When the Euclidean distance between the ORB features is less than a preset threshold, it can be determined that two ORB features are matched, and the more the number of matched ORB features is, the higher the degree of matching between the image frame and the above standard key frame is, so that the image frame with the highest degree of matching with the above standard key frame can be determined from the video to be labeled as the key frame of the target object in the video to be labeled.

[0048] It should be noted that when executing step S101, the feature matching method that can be used is not unique, and the embodiment of the present application does not impose any limitation on the specific implementation method of the above feature matching.

[0049] S102: Combine the standard key frame, the key frame and a first image sequence to obtain a first sub-video, and combine the standard key frame, the key frame and a second image sequence to obtain a second sub-video.

[0050] Here, the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be annotated.

[0051] Specifically, Figure 2 A schematic diagram of the structure of a first sub-video and a second sub-video provided in an embodiment of the present application is shown. Figure 2 As shown, the key frame determined in the above step S101 is used as the video segmentation point to segment the video to be labeled, and multiple image frames (from Figure 2Starting from the first frame in the video to be annotated as shown, until the frame before the key frame), wherein, considering that the target tracking algorithm can usually only perform target tracking processing on the image sequence in which the target object appears in the first frame and the shooting time is continuous, therefore, when executing step S102, after placing the standard key frame and the key frame containing the target object in the front, it is also necessary to reverse the multiple image frames before the key frame in the video to be annotated to obtain the first image sequence whose shooting time is continuous with the above-mentioned key frame, so as to obtain the first sub-video (i.e., the video that can be processed by the target tracking algorithm for target tracking) composed of the above-mentioned standard key frame, the above-mentioned key frame and the above-mentioned first image sequence in the order of arrangement of the above-mentioned standard key frame, the above-mentioned key frame and the above-mentioned first image sequence.

[0052] Here, the second image sequence represents an image sequence consisting of image frames located after the key frame in the video to be labeled.

[0053] Specifically, Figure 2 As shown, after the video to be labeled is segmented using the key frame determined in the above step S101 as the video segmentation point, multiple image frames located after the key frame in the video to be labeled can also be determined (starting from the frame after the key frame to the last frame in the video to be labeled), wherein, considering that the target tracking algorithm can usually only perform target tracking processing on the image sequence in which the target object appears in the first frame and the shooting time is continuous, therefore, when executing step S102, after placing the standard key frame and the key frame containing the target object in the front, the above multiple image frames located after the key frame whose shooting time is continuous with the key frame can be directly used as the second image sequence, so as to obtain the second sub-video (i.e., the video that can be processed by the target tracking algorithm) composed of the above standard key frame, the above key frame and the above second image sequence according to the arrangement order of the above standard key frame, the above key frame and the above second image sequence.

[0054] S103: Taking the target object as a tracking target, target tracking processing is performed on the first sub-video and the second sub-video respectively to obtain target tracking processing results corresponding to the first sub-video and the second sub-video respectively.

[0055] Here, when the target object contained in the first sub-video is tracked by the target tracking algorithm, based on the position mark information of the target object contained in the standard key frame, the target tracking algorithm can use the standard key frame located at the front in the first sub-video as prompt information related to the tracking target (equivalent to clarifying that the target object marked in the standard key frame is the tracking target), and then according to the prompt information, the key frames with continuous shooting time in the first sub-video + the first image sequence are subjected to target tracking processing to obtain the target tracking processing result corresponding to the first sub-video (i.e., the first sub-video marked with the position information of the target object in each image frame in the first sub-video).

[0056] Here, when performing target tracking processing on the above-mentioned target object contained in the second sub-video through the target tracking algorithm, the target tracking algorithm can also use the above-mentioned standard key frame located at the front in the second sub-video as prompt information related to the tracking target, thereby performing target tracking processing on the above-mentioned key frames + the above-mentioned second image sequence with continuous shooting time in the second sub-video based on the prompt information, and obtaining the target tracking processing result corresponding to the second sub-video (that is, the second sub-video marked with the position information of the target object in each image frame in the second sub-video).

[0057] It should be noted that the above-mentioned target tracking algorithm can be a TrackAnything algorithm, or it can be other target tracking algorithms that can only perform target tracking processing on image sequences in which the target object appears in the first frame and the shooting time is continuous. The embodiments of the present application do not impose any limitation on the specific algorithm represented by the above-mentioned target tracking algorithm.

[0058] S104: According to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are reorganized to obtain the target tracking result of the target object in the video to be labeled.

[0059] Here, referring to the contents of the above steps S102-S103, it can be known that since the above-mentioned first image sequence in the first sub-video is an image sequence obtained by reversing the image frame located before the key frame in the video to be annotated, therefore, after obtaining the target tracking processing results corresponding to the above-mentioned first sub-video and the above-mentioned second sub-video respectively, it is also necessary to adjust the arrangement order of each image frame in the target tracking processing results corresponding to the above-mentioned first sub-video and the above-mentioned second sub-video respectively according to the sorting of the image frames in the video to be annotated, and obtain the annotated video with the same sorting as the video to be annotated and with the position mark information of the target object as the target tracking result of the target object in the video to be annotated.

[0060] In the embodiment of the present application, in combination with the above description, the above step S104 can be performed according to the method shown in the following steps a1-a4, specifically:

[0061] Step a1: Obtain the target tracking processing result of the first image sequence from the target tracking processing result corresponding to the first sub-video.

[0062] Here, reference Figure 2 As shown, since the video to be labeled does not contain the above-mentioned standard key frames, and the above-mentioned key frames appear in the above-mentioned first sub-video and the above-mentioned second sub-video at the same time, the target tracking processing results of the first image sequence (that is, the first image sequence marked with the position information of the target object in each image frame in the first image sequence) can be obtained from the target tracking processing results corresponding to the first sub-video.

[0063] Step a2: Obtain the target tracking result of the second image sequence from the target tracking result corresponding to the second sub-video.

[0064] Here, reference Figure 2 As shown, since the video to be labeled does not contain the above-mentioned standard key frames, and the above-mentioned key frames appear in the above-mentioned first sub-video and the above-mentioned second sub-video at the same time, the target tracking processing results of the second image sequence (that is, the second image sequence marked with the position information of the target object in each image frame in the second image sequence) can be obtained from the target tracking processing results corresponding to the second sub-video.

[0065] Step a3: Obtain the target tracking result of the key frame from the target tracking result corresponding to the first sub-video or the second sub-video.

[0066] Here, reference Figure 2 As shown, since the above-mentioned key frame appears in the first sub-video and the second sub-video at the same time, the target tracking processing result of the key frame (that is, the key frame marked with the position information of the target object in the key frame) can be obtained from the target tracking processing result corresponding to the first sub-video, or the target tracking processing result of the key frame can be obtained from the target tracking processing result corresponding to the second sub-video, as long as it is ensured that there is no repeated acquisition.

[0067] Step a4: According to the order of image frames in the video to be labeled, adjust the order of image frames in the target tracking processing result of the first image sequence, the target tracking processing result of the key frame, and the target tracking processing result of the second image sequence to obtain the target tracking result of the target object in the video to be labeled.

[0068] Here, reference Figure 2As shown, according to the order of the image frames in the video to be labeled, the marked image frames in the target tracking processing results of the first image sequence can be played back in reverse to obtain the target tracking processing results of the first image sequence after the backlash, and then according to the arrangement order of the target tracking processing results of the first image sequence after the backlash, the target tracking processing results of the key frames and the target tracking processing results of the second image sequence, a labeled video composed of the above three target tracking processing results (i.e., the target tracking results of the target object in the video to be labeled) is obtained.

[0069] It should be noted that, no matter there is one or more target objects, each target object contained in the video to be labeled (that is, each entity object included in the target object) can be labeled according to the labeling method shown in steps S101-S104, and the repetitions will not be repeated here.

[0070] The specific implementation process of each of the above steps in the embodiment of the present application is described in detail below:

[0071] Regarding the method for determining the standard key frame in step S101 above, in an optional implementation, Figure 3 A schematic diagram of a process for determining a standard key frame provided by an embodiment of the present application is shown. Figure 3 As shown, before executing step S101, the method includes steps S301-S302; specifically:

[0072] S301 , according to a video shooting scene of the video to be labeled, determining a target image frame containing the target object from a plurality of original videos corresponding to the video shooting scene.

[0073] Here, the original video may be a historical video shot in a video shooting scene. For example, taking the video shooting scene as an excavator operation scene, the original video may be a historical video of multiple excavation operations of the excavator shot within a certain time period.

[0074] Specifically, when there are multiple image frames containing the target object in the multiple original videos, as an optional embodiment, an image frame containing the target object and showing the target object most clearly and completely can be determined from the multiple original videos as the target image frame.

[0075] It should be noted that in the embodiment of the present application, it is only necessary to ensure that the target image frame contains the target object, and the embodiment of the present application does not impose any limitation on the specific selection criteria of the target image frame.

[0076] S302, inputting the target image frame into an image segmentation model, annotating the target object contained in the target image frame through the image segmentation model, and outputting the annotated target image frame as a standard key frame of the target object.

[0077] Here, the above-mentioned target image frame is input into the image segmentation model. The image segmentation model can use the specified target object as a segmentation hint, predict the local image area belonging to the same entity category as the above-mentioned segmentation hint from the target image frame, and generate a segmentation mask corresponding to each predicted local area as the segmentation mask corresponding to the above-mentioned target object (that is, the target object contained in the target image frame is marked); wherein, the above-mentioned segmentation mask corresponding to the target object is the position marking information of the target object in the target image frame.

[0078] It should be noted that the above-mentioned image segmentation model includes but is not limited to: SAM model (Segment Anything model), Mask-RCNN model, etc.; the embodiment of the present application does not impose any limitation on the specific model structure of the above-mentioned image segmentation model.

[0079] Based on the video annotation method shown in the above steps S101-S104, considering that there are multiple target objects that need to be annotated in the video to be annotated (that is, the target objects include multiple physical objects of different categories), since the multiple target objects may appear in the same image frame, there may be a problem of overlapping position marks of the multiple target objects in the same image frame (that is, the above segmentation masks).

[0080] Here, in order to solve the above problem, in an optional implementation, Figure 4 A flow chart of a method for deduplicating target tracking results of multiple target objects in a video to be labeled provided by an embodiment of the present application is shown, such as Figure 4 As shown, after executing step S104, the method includes steps S401-S402; specifically:

[0081] S401, when the target object includes multiple entity objects of different categories, judging whether there are abnormal image frames with overlapping position marks between the target tracking results of the multiple entity objects according to the target tracking results respectively corresponding to the multiple entity objects in the video to be labeled.

[0082] Here, when there are multiple target objects (that is, the target objects include multiple entity objects of different categories), the target tracking process is performed on multiple different entity objects appearing in the same video to be labeled. Therefore, according to the target tracking results corresponding to the multiple entity objects in the video to be labeled (that is, the segmentation masks of each entity object in each image frame in the video to be labeled), if it is determined that the segmentation masks (that is, mask marks) of multiple (two or more) entity objects in the same image frame overlap, then the image frame where the segmentation masks overlap can be determined to be the above-mentioned abnormal image frame, and the overlapping segmentation mask is the local image area where the position marks in the abnormal image frame overlap.

[0083] S401: If it is determined that the abnormal image frame exists, segmentation prediction is performed on the local image region where the position markers in the abnormal image frame overlap, and the entity objects to which different pixel units in the local image region belong are determined based on the segmentation prediction result.

[0084] Here, the method for performing segmentation prediction on the local image region with overlapping position markers in the abnormal image frame (ie, the overlapping segmentation mask) is not unique, and the embodiments of the present application do not impose any limitation on this.

[0085] Specifically, based on the fact that the above-mentioned local image area (i.e., the overlapping segmentation mask) contains multiple sub-image areas surrounded by region boundaries (i.e., in the overlapping segmentation mask, multiple local areas surrounded by line segments often appear), in a first optional implementation, for the multiple sub-image areas contained in the above-mentioned local image area, according to the region boundaries that enclose the sub-image areas, the entity object whose position marker is connected to the region boundary can be determined as the segmentation prediction result of the sub-image area.

[0086] As an exemplary explanation, taking the target objects including the excavator body and the excavated material pile as an example, if the segmentation masks of the excavator body and the material pile in the same image frame overlap, it can be determined that the image frame is the above-mentioned abnormal image frame, and the segmentation mask of the overlapping part is determined to be the local image area where the above-mentioned position marks overlap. At this time, for each local area surrounded by line segments in the segmentation mask of the overlapping part (i.e., each sub-image area), it can be determined whether the area boundary of the local area (i.e., the line segments surrounding the local area) is connected with the segmentation mask of the excavator body or the segmentation mask of the material pile. If the area boundary of the local area is connected with the segmentation mask of the excavator body, it can be determined that the local area belongs to the segmentation mask of the excavator body. If the area boundary of the local area is connected with the segmentation mask of the material pile, it can be determined that the local area belongs to the segmentation mask of the excavator body. If the area boundary of the local area is connected with the segmentation mask of the material pile, it can be determined that the local area belongs to the segmentation mask of the material pile. If there is no connection, the local area can be removed to continue to judge the next local area.

[0087] Specifically, in the second optional implementation, the above-mentioned local image area (i.e., the overlapping segmentation mask) can be input into the image segmentation model with the help of an image segmentation model, and the image segmentation model can be used to perform segmentation prediction on the entity objects contained in the local image area to obtain the segmentation prediction result of the local image area.

[0088] Here, the above-mentioned local image region (i.e., the overlapping segmentation mask) is input into the image segmentation model. Based on the known information that the multiple overlapping entity objects in the above-mentioned local image region belong to each other, the image segmentation model can use the above-mentioned known information as a segmentation hint, and predict from the local image region the image regions that belong to the same entity category as each entity object involved in the above-mentioned segmentation hint, and generate a segmentation mask corresponding to each predicted image region as the segmentation prediction result of the image region.

[0089] Specifically, as another optional embodiment, the above two optional implementation methods of segmentation prediction may be combined to comprehensively determine the segmentation prediction result of the local image area where the position markers overlap according to the method described in the following steps b1-b4:

[0090] Step b1: for a plurality of sub-image regions included in the local image region, according to the region boundaries enclosing the sub-image regions, determine the entity objects whose position markers are connected to the region boundaries as the segmentation prediction results of the sub-image regions.

[0091] Here, the specific implementation of step b1 is the same as the first optional implementation described above, and the repeated parts are not repeated here.

[0092] Step b2: input the local image area into an image segmentation model, and use the image segmentation model to perform segmentation prediction on the entity objects contained in the local image area to obtain a segmentation prediction result of the local image area.

[0093] Here, the specific implementation of step b2 is the same as the second optional implementation described above, and the repeated parts will not be repeated here.

[0094] Step b3: If the segmentation prediction result of the sub-image area matches the segmentation prediction result of the local image area, determine the entity object to which the sub-image area belongs according to the segmentation prediction result of the sub-image area.

[0095] Here, if the segmentation prediction result of the sub-image region is consistent with (ie, matches) the segmentation prediction result of the local image region, then the entity object to which the sub-image region belongs can be directly determined as the segmentation prediction result of the sub-image region.

[0096] Step b4: If the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, remove the sub-image region from the local image region.

[0097] Here, if the segmentation prediction result of the sub-image area is inconsistent with (ie, does not match) the segmentation prediction result of the local image area, the sub-image area is removed from the local image area (equivalent to removing the sub-image area to which the entity object belongs is uncertain).

[0098] Based on the above-mentioned video annotation method provided in the embodiment of the present application, the key frame containing the target object is automatically determined from the video to be annotated by means of key frame matching, and the video to be annotated is segmented using the key frame as the video segmentation point, and target tracking processing is performed on two groups of image sequences located before and after the key frame in the video to be annotated, respectively. Therefore, without changing the algorithm working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame in the video to be annotated contains the target object, thereby realizing the automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0099] Based on the same inventive concept, the present application also provides a video annotation device corresponding to the above-mentioned video annotation method. Since the principle of solving the problem by the video annotation device in the embodiment of the present application is similar to the above-mentioned video annotation method in the embodiment of the present application, the implementation of the video annotation device can refer to the implementation of the above-mentioned video annotation method, and the repeated parts will not be repeated.

[0100] Reference Figure 5 As shown, Figure 5 A schematic diagram of the structure of a video annotation device provided in an embodiment of the present application is shown, wherein the video annotation device comprises:

[0101] A matching module 501 is used to determine, from the video to be labeled, an image frame matching the standard key frame according to the standard key frame of the target object as the key frame of the target object in the video to be labeled; wherein the standard key frame includes the position mark information of the target object;

[0102] The grouping module 502 is used to combine the standard key frame, the key frame and a first image sequence to obtain a first sub-video, and to combine the standard key frame, the key frame and a second image sequence to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be labeled; and the second image sequence represents an image sequence composed of image frames after the key frame in the video to be labeled;

[0103] A target tracking module 503 is used to perform target tracking processing on the first sub-video and the second sub-video respectively with the target object as the tracking target, and obtain target tracking processing results corresponding to the first sub-video and the second sub-video respectively;

[0104] The reorganization module 504 is used to reorganize the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the order of the image frames in the video to be labeled, so as to obtain the target tracking result of the target object in the video to be labeled.

[0105] In an optional implementation, the video annotation device further includes: a standard key frame determination module, wherein the standard key frame determination module is used to:

[0106] According to the video shooting scene of the video to be labeled, determining a target image frame containing the target object from a plurality of original videos corresponding to the video shooting scene;

[0107] The target image frame is input into an image segmentation model, the target object contained in the target image frame is labeled by the image segmentation model, and the labeled target image frame is output as a standard key frame of the target object.

[0108] In an optional implementation, when the target tracking processing results corresponding to the first sub-video and the second sub-video are reorganized according to the order of the image frames in the video to be labeled, the reorganization module 504 is used to:

[0109] Obtaining a target tracking result of the first image sequence from the target tracking result corresponding to the first sub-video;

[0110] Obtaining the target tracking result of the second image sequence from the target tracking result corresponding to the second sub-video;

[0111] Obtaining the target tracking result of the key frame from the target tracking result corresponding to the first sub-video or the second sub-video;

[0112] According to the order of the image frames in the video to be labeled, the target tracking processing result of the first image sequence, the target tracking processing result of the key frame, and the order of the image frames in the target tracking processing result of the second image sequence are adjusted to obtain the target tracking result of the target object in the video to be labeled.

[0113] In an optional implementation, the video annotation device further includes: a deduplication module, wherein the deduplication module is used to:

[0114] When the target object includes multiple entity objects of different categories, judging whether there are abnormal image frames with overlapping position marks between the target tracking results of the multiple entity objects according to the target tracking results respectively corresponding to the multiple entity objects in the video to be labeled;

[0115] If it is determined that the abnormal image frame exists, segmentation prediction is performed on the local image region where the position markers in the abnormal image frame overlap, and the entity objects to which different pixel units in the local image region belong are determined based on the segmentation prediction result.

[0116] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the deduplication module is used to:

[0117] For a plurality of sub-image regions included in the local image region, based on region boundaries enclosing the sub-image regions, entity objects whose position markers are connected to the region boundaries are determined as segmentation prediction results of the sub-image regions.

[0118] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the deduplication module is used to:

[0119] The local image region is input into an image segmentation model, and segmentation prediction is performed on the entity object contained in the local image region through the image segmentation model to obtain a segmentation prediction result of the local image region.

[0120] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the deduplication module is used to:

[0121] For a plurality of sub-image regions included in the local image region, according to the region boundaries enclosing the sub-image regions, determining a physical object whose position marker is connected to the region boundary as a segmentation prediction result of the sub-image region;

[0122] Inputting the local image region into an image segmentation model, performing segmentation prediction on the entity object contained in the local image region through the image segmentation model, and obtaining a segmentation prediction result of the local image region;

[0123] If the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, determining the entity object to which the sub-image region belongs according to the segmentation prediction result of the sub-image region;

[0124] If the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, the sub-image region is removed from the local image region.

[0125] Based on the above-mentioned video annotation device provided in the embodiment of the present application, the key frame containing the target object is automatically determined from the video to be annotated by means of key frame matching, and the video to be annotated is segmented using the key frame as the video segmentation point, and target tracking processing is performed on two groups of image sequences located before and after the key frame in the video to be annotated, respectively. Therefore, on the basis of not changing the algorithm working principle of the target tracking algorithm, the video annotation method does not need to separately determine whether the first frame in the video to be annotated contains the target object, thereby realizing the automation of video annotation and effectively improving the efficiency and accuracy of video annotation.

[0126] Based on the same inventive concept, the present application also provides an electronic device corresponding to the above-mentioned video annotation method. Since the principle of solving the problem by the electronic device in the embodiment of the present application is similar to the above-mentioned video annotation method in the embodiment of the present application, the implementation of the electronic device can refer to the implementation of the above-mentioned video annotation method, and the repeated parts will not be repeated.

[0127] Figure 6 A structural diagram of an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus 603, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device runs a video annotation method in the embodiment, the processor 601 communicates with the memory 602 through the bus 603, and the processor 601 executes the machine-readable instructions. When the processor 601 executes the machine-readable instructions, the following steps are implemented, specifically:

[0128] According to the standard key frame of the target object, an image frame matching the standard key frame is determined from the video to be labeled as the key frame of the target object in the video to be labeled; wherein the standard key frame includes the position mark information of the target object;

[0129] The standard key frame, the key frame and a first image sequence are combined to obtain a first sub-video, and the standard key frame, the key frame and a second image sequence are combined to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be labeled; and the second image sequence represents an image sequence composed of image frames after the key frame in the video to be labeled;

[0130] Taking the target object as a tracking target, performing target tracking processing on the first sub-video and the second sub-video respectively, and obtaining target tracking processing results corresponding to the first sub-video and the second sub-video respectively;

[0131] According to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are reorganized to obtain the target tracking result of the target object in the video to be labeled.

[0132] In an optional implementation, the processor 601 is further configured to:

[0133] According to the video shooting scene of the video to be labeled, determining a target image frame containing the target object from a plurality of original videos corresponding to the video shooting scene;

[0134] The target image frame is input into an image segmentation model, the target object contained in the target image frame is labeled by the image segmentation model, and the labeled target image frame is output as a standard key frame of the target object.

[0135] In an optional implementation, when reorganizing the target tracking processing results respectively corresponding to the first sub-video and the second sub-video according to the order of the image frames in the video to be labeled, the processor 601 is configured to:

[0136] Obtaining a target tracking result of the first image sequence from the target tracking result corresponding to the first sub-video;

[0137] Obtaining the target tracking result of the second image sequence from the target tracking result corresponding to the second sub-video;

[0138] Obtaining the target tracking result of the key frame from the target tracking result corresponding to the first sub-video or the second sub-video;

[0139] According to the order of the image frames in the video to be labeled, the target tracking processing result of the first image sequence, the target tracking processing result of the key frame, and the order of the image frames in the target tracking processing result of the second image sequence are adjusted to obtain the target tracking result of the target object in the video to be labeled.

[0140] In an optional implementation, the processor 601 is further configured to:

[0141] When the target object includes multiple entity objects of different categories, judging whether there are abnormal image frames with overlapping position marks between the target tracking results of the multiple entity objects according to the target tracking results respectively corresponding to the multiple entity objects in the video to be labeled;

[0142] If it is determined that the abnormal image frame exists, segmentation prediction is performed on the local image region where the position markers in the abnormal image frame overlap, and the entity objects to which different pixel units in the local image region belong are determined based on the segmentation prediction result.

[0143] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the processor 601 is configured to:

[0144] For a plurality of sub-image regions included in the local image region, based on region boundaries enclosing the sub-image regions, entity objects whose position markers are connected to the region boundaries are determined as segmentation prediction results of the sub-image regions.

[0145] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the processor 601 is configured to:

[0146] The local image region is input into an image segmentation model, and segmentation prediction is performed on the entity object contained in the local image region through the image segmentation model to obtain a segmentation prediction result of the local image region.

[0147] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the processor 601 is configured to:

[0148] For a plurality of sub-image regions included in the local image region, according to the region boundaries enclosing the sub-image regions, determining a physical object whose position marker is connected to the region boundary as a segmentation prediction result of the sub-image region;

[0149] Inputting the local image region into an image segmentation model, performing segmentation prediction on the entity object contained in the local image region through the image segmentation model, and obtaining a segmentation prediction result of the local image region;

[0150] If the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, determining the entity object to which the sub-image region belongs according to the segmentation prediction result of the sub-image region;

[0151] If the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, the sub-image region is removed from the local image region.

[0152] Through the above-mentioned electronic device provided in the embodiment of the present application, the key frame containing the target object is automatically determined from the video to be labeled by means of key frame matching, and the video to be labeled is segmented using the key frame as the video segmentation point, and target tracking processing is performed on two groups of image sequences located before and after the key frame in the video to be labeled, respectively. Therefore, without changing the algorithm working principle of the target tracking algorithm, the video labeling method does not need to separately determine whether the first frame in the video to be labeled contains the target object, thereby realizing the automation of video labeling and effectively improving the efficiency and accuracy of video labeling.

[0153] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:

[0154] According to the standard key frame of the target object, an image frame matching the standard key frame is determined from the video to be labeled as the key frame of the target object in the video to be labeled; wherein the standard key frame includes the position mark information of the target object;

[0155] The standard key frame, the key frame and a first image sequence are combined to obtain a first sub-video, and the standard key frame, the key frame and a second image sequence are combined to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be labeled; and the second image sequence represents an image sequence composed of image frames after the key frame in the video to be labeled;

[0156] Taking the target object as a tracking target, performing target tracking processing on the first sub-video and the second sub-video respectively, and obtaining target tracking processing results corresponding to the first sub-video and the second sub-video respectively;

[0157] According to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are reorganized to obtain the target tracking result of the target object in the video to be labeled.

[0158] In an optional implementation, the processor is further configured to:

[0159] According to the video shooting scene of the video to be labeled, determining a target image frame containing the target object from a plurality of original videos corresponding to the video shooting scene;

[0160] The target image frame is input into an image segmentation model, the target object contained in the target image frame is labeled by the image segmentation model, and the labeled target image frame is output as a standard key frame of the target object.

[0161] In an optional implementation, when reorganizing the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the order of the image frames in the video to be labeled, the processor is used to:

[0162] Obtaining a target tracking result of the first image sequence from the target tracking result corresponding to the first sub-video;

[0163] Obtaining the target tracking result of the second image sequence from the target tracking result corresponding to the second sub-video;

[0164] Obtaining the target tracking result of the key frame from the target tracking result corresponding to the first sub-video or the second sub-video;

[0165] According to the order of the image frames in the video to be labeled, the target tracking processing result of the first image sequence, the target tracking processing result of the key frame, and the order of the image frames in the target tracking processing result of the second image sequence are adjusted to obtain the target tracking result of the target object in the video to be labeled.

[0166] In an optional implementation, the processor is further configured to:

[0167] When the target object includes multiple entity objects of different categories, judging whether there are abnormal image frames with overlapping position marks between the target tracking results of the multiple entity objects according to the target tracking results respectively corresponding to the multiple entity objects in the video to be labeled;

[0168] If it is determined that the abnormal image frame exists, segmentation prediction is performed on the local image region where the position markers in the abnormal image frame overlap, and the entity objects to which different pixel units in the local image region belong are determined based on the segmentation prediction result.

[0169] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the processor is configured to:

[0170] For a plurality of sub-image regions included in the local image region, based on region boundaries enclosing the sub-image regions, entity objects whose position markers are connected to the region boundaries are determined as segmentation prediction results of the sub-image regions.

[0171] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the processor is configured to:

[0172] The local image region is input into an image segmentation model, and segmentation prediction is performed on the entity object contained in the local image region through the image segmentation model to obtain a segmentation prediction result of the local image region.

[0173] In an optional implementation, when performing segmentation prediction on the local image region where the position markers in the abnormal image frame overlap, the processor is configured to:

[0174] For a plurality of sub-image regions included in the local image region, according to the region boundaries enclosing the sub-image regions, determining a physical object whose position marker is connected to the region boundary as a segmentation prediction result of the sub-image region;

[0175] Inputting the local image region into an image segmentation model, performing segmentation prediction on the entity object contained in the local image region through the image segmentation model, and obtaining a segmentation prediction result of the local image region;

[0176] If the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, determining the entity object to which the sub-image region belongs according to the segmentation prediction result of the sub-image region;

[0177] If the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, the sub-image region is removed from the local image region.

[0178] Through the above-mentioned computer-readable storage medium provided in the embodiment of the present application, the key frame containing the target object is automatically determined from the video to be labeled by means of key frame matching, and the video to be labeled is segmented using the key frame as the video segmentation point, and target tracking processing is performed on two groups of image sequences located before and after the key frame in the video to be labeled, respectively. Therefore, without changing the algorithm working principle of the target tracking algorithm, the video labeling method does not need to separately determine whether the first frame in the video to be labeled contains the target object, thereby realizing the automation of video labeling and effectively improving the efficiency and accuracy of video labeling.

[0179] In an embodiment of the present application, the computer-readable storage medium can also execute other machine-readable instructions when run by a processor to execute the video annotation method as described in other embodiments. For the specific steps and principles of the video annotation method, please refer to the description of the method side embodiment, which will not be repeated here.

[0180] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0185] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video annotation method, characterized in that: The video annotation method comprises: According to the standard key frame of the target object, an image frame matching the standard key frame is determined from the video to be labeled as the key frame of the target object in the video to be labeled; wherein the standard key frame includes the position mark information of the target object; The standard key frame, the key frame and a first image sequence are combined to obtain a first sub-video, and the standard key frame, the key frame and a second image sequence are combined to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be labeled; and the second image sequence represents an image sequence composed of image frames after the key frame in the video to be labeled; Taking the target object as a tracking target, performing target tracking processing on the first sub-video and the second sub-video respectively, and obtaining target tracking processing results corresponding to the first sub-video and the second sub-video respectively; According to the order of the image frames in the video to be labeled, the target tracking processing results corresponding to the first sub-video and the second sub-video are reorganized to obtain the target tracking result of the target object in the video to be labeled.

2. The video annotation method according to claim 1, characterized in that: The video annotation method further includes: According to the video shooting scene of the video to be labeled, determining a target image frame containing the target object from a plurality of original videos corresponding to the video shooting scene; The target image frame is input into an image segmentation model, the target object contained in the target image frame is labeled by the image segmentation model, and the labeled target image frame is output as a standard key frame of the target object.

3. The video annotation method according to claim 1, characterized in that: The step of reorganizing the target tracking processing results respectively corresponding to the first sub-video and the second sub-video according to the order of the image frames in the video to be labeled includes: Obtaining a target tracking result of the first image sequence from the target tracking result corresponding to the first sub-video; Obtaining the target tracking result of the second image sequence from the target tracking result corresponding to the second sub-video; Obtaining the target tracking result of the key frame from the target tracking result corresponding to the first sub-video or the second sub-video; According to the order of the image frames in the video to be labeled, the target tracking processing result of the first image sequence, the target tracking processing result of the key frame, and the order of the image frames in the target tracking processing result of the second image sequence are adjusted to obtain the target tracking result of the target object in the video to be labeled.

4. The video annotation method according to claim 1, characterized in that: The video annotation method further includes: When the target object includes multiple entity objects of different categories, judging whether there are abnormal image frames with overlapping position marks between the target tracking results of the multiple entity objects according to the target tracking results respectively corresponding to the multiple entity objects in the video to be labeled; If it is determined that the abnormal image frame exists, segmentation prediction is performed on the local image region where the position markers in the abnormal image frame overlap, and the entity objects to which different pixel units in the local image region belong are determined based on the segmentation prediction result.

5. The video annotation method according to claim 4, characterized in that: The segmentation prediction of the local image region where the position markers overlap in the abnormal image frame includes: For a plurality of sub-image regions included in the local image region, based on region boundaries enclosing the sub-image regions, entity objects whose position markers are connected to the region boundaries are determined as segmentation prediction results of the sub-image regions.

6. The video annotation method according to claim 4, characterized in that: The segmentation prediction of the local image area with overlapping position markers in the abnormal image frame also includes: The local image region is input into an image segmentation model, and segmentation prediction is performed on the entity object contained in the local image region through the image segmentation model to obtain a segmentation prediction result of the local image region.

7. The video annotation method according to claim 4, characterized in that: The segmentation prediction of the local image area with overlapping position markers in the abnormal image frame also includes: For a plurality of sub-image regions included in the local image region, according to the region boundaries enclosing the sub-image regions, determining a physical object whose position marker is connected to the region boundary as a segmentation prediction result of the sub-image region; Inputting the local image region into an image segmentation model, performing segmentation prediction on the entity object contained in the local image region through the image segmentation model, and obtaining a segmentation prediction result of the local image region; If the segmentation prediction result of the sub-image region matches the segmentation prediction result of the local image region, determining the entity object to which the sub-image region belongs according to the segmentation prediction result of the sub-image region; If the segmentation prediction result of the sub-image region does not match the segmentation prediction result of the local image region, the sub-image region is removed from the local image region.

8. A video annotation device, characterized in that: The video annotation device comprises: A matching module, configured to determine, from the video to be labeled, an image frame matching the standard key frame according to the standard key frame of the target object as the key frame of the target object in the video to be labeled; wherein the standard key frame includes the position mark information of the target object; A grouping module, used for combining the standard key frame, the key frame and a first image sequence to obtain a first sub-video, and combining the standard key frame, the key frame and a second image sequence to obtain a second sub-video; wherein the first image sequence represents an image sequence obtained by reversing the image frames before the key frame in the video to be labeled; and the second image sequence represents an image sequence composed of image frames after the key frame in the video to be labeled; A target tracking module, configured to use the target object as a tracking target, perform target tracking processing on the first sub-video and the second sub-video respectively, and obtain target tracking processing results corresponding to the first sub-video and the second sub-video respectively; The reorganization module is used to reorganize the target tracking processing results corresponding to the first sub-video and the second sub-video respectively according to the order of the image frames in the video to be labeled, so as to obtain the target tracking result of the target object in the video to be labeled.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the video annotation method as described in any one of claims 1 to 8 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video tagging method according to any one of claims 1 to 8 are executed.

Images