Target marking method, electronic device and computer-readable storage medium

Abstract

The present application discloses a target labeling method, an electronic device and a computer-readable storage medium, the method comprising: obtaining a video frame to be labeled, determining all the targets to be labeled in the video frame to be labeled; determining the target type of the video frame to be labeled based on at least one historical target in the labeled video frame; wherein the target type includes a newly added target and an existing target, the newly added target includes a target that is different from at least one historical target, and the existing target includes a target that matches any historical target in at least one historical target; based on the labeling data of at least one historical target stored and the target type of the newly added target, generating labeling data of the newly added target and labeling the newly added target; and / or, labeling the existing target using the labeling data of the historical target that matches the existing target; wherein the labeling data includes the target type and identifier of the target. The above scheme can improve the accuracy of target labeling.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a target marking method, an electronic device and a computer-readable storage medium. Background Art

[0002] Target annotation refers to the annotation of targets in video frames. The annotated targets can be used to train the target tracking algorithm to obtain the trained target tracking model. Therefore, the accuracy of target annotation directly affects the training effect of the target tracking model. In complex application scenarios, the number of targets contained in continuous video frames is large, which makes target annotation prone to errors. In view of this, how to improve the accuracy of target annotation has become an urgent problem to be solved. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a target labeling method, an electronic device and a computer-readable storage medium, which can improve the accuracy of target labeling.

[0004] To solve the above technical problems, the first aspect of the present application provides a target labeling method, which includes: obtaining a video frame to be labeled, and determining all targets to be labeled in the video frame to be labeled; based on at least one historical target in an already labeled video frame, determining the target type of the video frame to be labeled; wherein the target type includes newly added targets and existing targets, the newly added targets include targets that are different from the at least one historical target, and the existing targets include targets that match any historical target in the at least one historical target; based on the stored labeling data of the at least one historical target and the target type of the newly added target, generating labeling data for the newly added target and labeling the newly added target; and / or, labeling the existing target using the labeling data of the historical target that matches the existing target; wherein the labeling data includes the target type and identifier of the target.

[0005] To solve the above technical problems, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer storage medium on which program data is stored, and when the program data is executed by a processor, the method described in the first aspect is implemented.

[0007] In the above scheme, after obtaining the video frame to be marked, all the targets to be marked are determined in the video frame to be marked, and the targets to be marked are compared with at least one historical target in the marked video frame to determine the target type of the target to be marked. All the targets to be marked may include at least one of existing targets and newly added targets. When the video frame to be marked is a newly added target, based on the stored annotation data of the historical targets and the target type of the newly added target, the annotation data of the newly added target is generated and the newly added target is marked, so that the annotation data of the newly added target can be distinguished from the annotation data of the historical targets, reducing the probability of mislabeling of the annotation data. When the video frame to be marked is an existing target, the annotation data of the historical targets matching the existing targets are used to mark the existing targets, thereby improving the efficiency of marking the existing targets, wherein the annotation data includes the target type and identification of the target. Therefore, when marking the targets to be marked in each video frame to be marked, the annotation data can be quickly determined for the existing targets, and the annotation data that is different from the historical targets can be determined for the newly added targets, thereby improving the accuracy of target marking. Based on the above annotation method, the annotation data can also be verified to ensure the uniqueness of the annotation data corresponding to each target. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flowchart of an implementation method of the target marking method of the present application;

[0010] Figure 2 It is a flowchart of another implementation method of the target labeling method of the present application;

[0011] Figure 3 This application Figure 2 A schematic diagram of a flow chart of an implementation method corresponding to step S205;

[0012] Figure 4 It is a schematic diagram of the topological structure of an implementation method of the target marking system of the present application;

[0013] Figure 5 It is a schematic diagram of the topological structure of an embodiment of the electronic device of the present application;

[0014] Figure 6 It is a schematic diagram of the topological structure of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0017] The target labeling method provided in this application is used to label targets in video frames. The corresponding execution subject of the target labeling method in this application can be a server. When the server receives the video frame to be labeled, it can label the target in the video frame to be labeled. In addition, the target labeling method in this application can also be labeled manually, and this application does not make specific restrictions on this.

[0018] See also Figure 1 , Figure 1 : is a flow chart of an implementation method of a target labeling method of the present application, the method comprising:

[0019] S101: Obtain a video frame to be labeled, and determine all objects to be labeled in the video frame to be labeled.

[0020] Specifically, the video frames in which all targets in the video frames have been labeled are labeled video frames, the targets in the labeled video frames are historical targets, and the video frames in which all targets in the video frames have not been labeled are video frames to be labeled. After the video frames to be labeled are obtained, all targets to be labeled are determined in the video frames to be labeled.

[0021] In one application, a video frame to be labeled is obtained, identifiable targets are calibrated, and target calibration results on the video frame to be labeled are obtained, so that all targets to be labeled in the video frame to be labeled are determined based on the target calibration results.

[0022] In an application scenario, video data is obtained and played in time sequence. When the video data is paused during the playing process, a video frame to be labeled is obtained, and an object to be labeled in the video frame to be labeled is determined.

[0023] S102: Determine the target type of the video frame to be labeled based on at least one historical target in the labeled video frame.

[0024] Specifically, the target types include newly added targets and existing targets. The newly added targets include targets that are different from at least one historical target, and the existing targets include targets that match any historical target in at least one historical target.

[0025] Furthermore, all the targets to be labeled may all be existing targets, or all the targets to be labeled may all be newly added targets, or all the targets to be labeled may include existing targets and newly added targets.

[0026] In one application, the following operations are performed for all the targets to be labeled: the targets to be labeled are sequentially compared with historical targets in at least some labeled video frames to determine the categories of the targets to be labeled, wherein the categories of the targets to be labeled include newly added targets and existing targets.

[0027] In a specific application scenario, the target to be labeled is compared with historical targets in a preset number of labeled video frames before the video frame to be labeled in sequence, where the preset value can be the number of video frames corresponding to a group of GOPs, so that the target to be labeled in the video frame to be labeled is only compared with targets in some video frames with similar timing, so as to reduce the process of repeated comparison of historical targets that have left the monitoring area and improve the comparison efficiency.

[0028] S103: Based on the stored annotation data of at least one historical target and the target type of the newly added target, generate annotation data of the newly added target and annotate the newly added target, and / or annotate the existing target using the annotation data of the historical target that matches the existing target.

[0029] Specifically, the labeling data includes the target type and identifier of the target. When the target to be labeled is a newly added target, the target type of the newly added target is determined, and then based on the target type of the newly added target and the labeling data of all stored historical targets, the labeling data matching the newly added target is generated and the newly added target is labeled, wherein the labeling data matching any newly added target is different from other targets. When the target to be labeled is an existing target, the labeling data of the historical target matching the existing target is obtained to label the existing target.

[0030] In one application, the annotation data is in the form of target type plus identification, the same target uses the same annotation data, and the judgment of the same target is based on the similarity between the target to be annotated and the historical target. When the similarity between the target to be annotated and any historical target exceeds the similarity threshold, the corresponding target to be annotated and the historical target whose similarity exceeds the similarity threshold are determined to be the same target, and the corresponding target to be annotated is taken as the existing target. Therefore, the existing target is matched with the historical target whose similarity exceeds the similarity threshold.

[0031] Furthermore, when the target to be annotated is an existing target, determine the historical target in the annotated video frame that belongs to the same target as the target in the video frame to be annotated, and annotate the annotated data corresponding to the historical target that matches the existing target as the same historical target. When the target to be annotated is a newly added target, obtain the target type of the newly added target and the identifier in the annotated data of the historical target, generate an identifier for the newly added target that is different from the historical target, and use the newly generated identifier and the target type of the newly added target together to form the annotated data that matches the newly added target.

[0032] In a specific application scenario, the identifier uses an 8-bit hexadecimal number, and the initial identifier starts from 0x00000000. When the target to be labeled includes a new target, the maximum identifier used in the historical target is determined, and a new identifier greater than the current maximum identifier is generated for the new target based on the maximum identifier, and the target type of the new target is determined, and the target type and the new identifier are used as the labeling data of the new target. When the target to be labeled includes a historical target, in the stage of determining the target category of the target to be labeled, it has been determined which historical target in the labeled video frame belongs to the same target, and the labeling data corresponding to the historical target that matches the existing target is labeled for the corresponding existing target, so that the same target uses the same labeling data.

[0033] In one implementation, all the objects to be labeled in the video frame to be labeled are traversed, and after the object categories of all the objects to be labeled are determined, the objects to be labeled are labeled based on the object categories corresponding to all the objects to be labeled.

[0034] In another embodiment, the target types of the targets to be labeled in the video frames to be labeled are determined in sequence, and after each target category of a target to be labeled is obtained, the target to be labeled is labeled based on the target category of the target to be labeled, and finally all the targets to be labeled in the video frames to be labeled are traversed.

[0035] It can be understood that the annotated video frames to be annotated are used as annotated video frames, and the annotated objects to be annotated are used as historical objects.

[0036] Optionally, when multiple labeled video frames are obtained, the labeled data corresponding to the historical targets in the multiple labeled video frames can be checked, and the labeled data can be corrected based on the test results to obtain the final labeled data of the historical targets, thereby ensuring the uniqueness of the final labeled data of each target and improving the accuracy of target labeling.

[0037] In the above scheme, after obtaining the video frame to be marked, all the targets to be marked are determined in the video frame to be marked, and the targets to be marked are compared with at least one historical target in the marked video frame to determine the target type of the target to be marked. All the targets to be marked may include at least one of existing targets and newly added targets. When the video frame to be marked is a newly added target, based on the stored annotation data of the historical targets and the target type of the newly added target, the annotation data of the newly added target is generated and the newly added target is marked, so that the annotation data of the newly added target can be distinguished from the annotation data of the historical targets, reducing the probability of mislabeling of the annotation data. When the video frame to be marked is an existing target, the annotation data of the historical targets matching the existing targets are used to mark the existing targets, thereby improving the efficiency of marking the existing targets, wherein the annotation data includes the target type and identification of the target. Therefore, when marking the targets to be marked in each video frame to be marked, the annotation data can be quickly determined for the existing targets, and the annotation data that is different from the historical targets can be determined for the newly added targets, thereby improving the accuracy of target marking. Based on the above annotation method, the annotation data can also be verified to ensure the uniqueness of the annotation data corresponding to each target.

[0038] See also Figure 2 , Figure 2 : is a flow chart of another implementation method of the target labeling method of the present application, the method comprising:

[0039] S201: Obtain a video frame to be labeled, and determine all objects to be labeled in the video frame to be labeled.

[0040] Specifically, a video frame without target labeling is used as a video frame to be labeled. After the video frame to be labeled is obtained, all the targets to be labeled in the video frame are labeled.

[0041] In one application, after obtaining the video frame to be labeled, the target and background of the video frame are identified using a target calibration algorithm, thereby extracting all the targets to be labeled in the video frame.

[0042] S202: Determine whether the target to be marked is a newly added target.

[0043] Specifically, the target types of the targets to be labeled include newly added targets and existing targets. The newly added targets include targets that are different from historical targets, and the existing targets include targets that match any historical targets in the labeled video frames.

[0044] Further, after obtaining the target to be labeled, the target to be labeled is compared with the historical targets in the labeled video frames to determine whether the target to be labeled is a newly added target. If the target to be labeled is a newly added target, proceed to step S203, otherwise, proceed to step S204.

[0045] S203: Based on the stored annotation data of at least one historical target and the target type of the newly added target, generate annotation data of the newly added target and annotate the newly added target.

[0046] Specifically, the annotation data corresponding to the historical target has been pre-stored after the annotation is completed, and the annotation data includes the target type and the identifier, wherein the target type is pre-defined based on the classification rule. When the target to be annotated is a newly added target, an identifier different from the historical target is generated based on the annotation data corresponding to the historical target, and the target type and the identifier of the newly added target are used together to annotate the newly added target.

[0047] In one application method, among the annotation data of at least one historical target that has been stored, the identifier with the largest or smallest value is used as a reference identifier; based on the number of newly added targets and the value of the reference identifier, the identifier corresponding to each newly added target is obtained; based on the target type and identifier of each newly added target, the annotation data matching each newly added target is generated; each newly generated annotation data is annotated for each newly added target, and the annotation data matching the newly added target is stored.

[0048] Specifically, the numerical values ​​identified in the annotation data of at least one stored historical target can be used from large to small or from small to large. When the numerical values ​​are used from small to large, the identifier with the largest numerical value is used as a reference identifier. When the numerical values ​​are used from large to small, the identifier with the smallest numerical value is used as a reference identifier. According to the number of newly added targets, corresponding identifiers are generated for each newly added target in turn. Based on the target type and identifier of each newly added target, annotation data matching each information target is generated. The newly generated annotation data corresponding to each newly added target is used to label the newly added target. After the labeling is completed, the annotation data matching the newly added target is stored.

[0049] In one application scenario, when multiple new targets are obtained, an identification set corresponding to the new targets is generated based on the number of new targets and reference identifications. When generating labeling data for each new target, after determining the target type of each new target, an identification is extracted from the identification set and used together with the target type to generate labeling data for the new target. The extracted identification is deleted from the identification set to avoid duplication of identifications of the new targets.

[0050] In another application scenario, the stored annotation data of historical targets are stored in a target list. In the target list, the annotation data corresponding to the targets are stored in the target list in the order in which the targets appear, using the target type and an identifier, thereby facilitating the determination of the order in which the targets appear and improving the efficiency of searching for annotation data corresponding to historical targets.

[0051] Furthermore, the target list can be displayed on a visual interface so that the annotated data of historical targets can be retrieved when annotating new targets, thereby improving the efficiency of annotation. The target type is predefined based on the classification rules in the application scenario. In different application scenarios, the same target can be classified into different types so that it can be flexibly used in different application scenarios. For example, the target type corresponding to a moving vehicle can be a motor vehicle or a moving object.

[0052] In one application method, the annotation data of stored historical targets are stored in a target list, and the identifiers in the target list are sorted from small to large. The largest identifier in the annotation data corresponding to the historical target is determined from the target list as a reference identifier. When the annotation data corresponding to the newly added target in the target list is arranged from small to large according to the identifier, the latest identifier can be extracted from the target list to obtain the reference identifier, thereby improving the extraction efficiency.

[0053] Furthermore, the newly added targets are arranged in order, and the values ​​of the current reference identifiers are added in sequence according to the number of the newly added targets until the number of the newly added targets is reached, thereby obtaining the identifier corresponding to each newly added target.

[0054] In a specific application scenario, suppose that four targets have been marked in the marked video frame, and the identifiers of the four historical targets are 1, 2, 3, and 4 respectively. Then the maximum identifier maxID in the marked data corresponding to the historical targets is 4. When N new targets need to be marked in the video frame to be marked, the identifiers corresponding to the N new targets are maxID+1…maxID+N, so that the identifier corresponding to each new target is guaranteed to be an independent identifier, thereby distinguishing different targets.

[0055] Furthermore, when identifying new targets, the target type and identifier of each new target are combined to form the labeling data of the new target, thereby using the target type and identifier to form the labeling data, thereby improving the uniqueness of the labeling data corresponding to the new target, and labeling the new target with the newly generated labeling data so that the new target matches the corresponding labeling data, and then adding the labeling data matching the new target to the target list, and placing the labeling data corresponding to the identifier with a larger numerical value at the front of the target list to facilitate determining the maximum identifier in the target list.

[0056] It can be understood that when the identifiers in the annotation data are used from large to small, the identifier with the smallest value in the annotation data of the historical target is selected as the reference identifier, and the reference identifier is decremented based on the number of reference identifiers and newly added targets. This application will not go into details.

[0057] Optionally, before annotating each newly added target with newly generated annotation data and storing the annotation data that matches the newly added target, the method also includes: comparing the identifier in the newly generated annotation data with the identifier of the target that has been annotated in the video frame to be annotated, so as to determine whether each newly generated identifier is different from the identifier of the target that has been annotated in any video frame to be annotated.

[0058] Specifically, before the newly generated annotation data is used to annotate the newly added target, the identifier of the annotation data is re-detected to ensure that the identifier in the newly generated annotation data is unique in the current video frame to be annotated, thereby reducing the probability of mislabeling due to human modification of the identifier or reuse of the identifier. The newly generated annotation data is the annotation data generated based on the target type and identifier of the newly added target.

[0059] In one application scenario, when a new target in a video frame to be labeled is to be labeled, an identifier is first applied for, and then the labeling data is generated based on the applied identifier. Before labeling the new target, the identifier in the labeling data and the identifier of the labeling data corresponding to all the targets to be labeled in the current frame are extracted. If the extracted identifier is repeated with the identifier in the labeling data corresponding to other targets in the video frame to be labeled, it is prompted that the corresponding labeling data is illegal, and then the identifier is reapplied for, and the identifier corresponding to the new target is generated again based on the reference identifier.

[0060] S204: Label the existing target using the labeling data of the historical target that matches the existing target.

[0061] Specifically, when the target to be labeled is an existing target, the existing target is labeled using the labeling data of the historical target that matches the existing target, wherein the similarity between the historical target that matches the existing target and the existing target exceeds a similarity threshold.

[0062] In one application, the annotation data of historical targets are stored in a target list. When the video frame to be annotated is an existing target, the annotation data of the historical target matching the existing target is extracted from the target list to annotate the existing target, thereby improving the annotation efficiency and accuracy.

[0063] In an application scenario, after determining all the targets to be labeled in the video frame to be labeled, the targets to be labeled are judged one by one to determine whether the target to be labeled is a new target or a historical target, and each target to be labeled is identified one by one. When the target to be labeled is a new target, an identification is applied for the new target, and an identification of the new target is generated based on the reference identification. The target type and identification of the new target are combined into the annotation data to label the new target. After the labeling is completed, the annotation data of the new target is stored. When the target to be labeled is an existing target, the annotation data of the historical target that is determined to be the same target as the existing target is extracted, that is, the annotation data of the historical target whose similarity with the existing target exceeds the similarity threshold is used to label the existing target. Among them, the labeled video frame to be labeled is used as the labeled video frame, and the labeled target to be labeled is used as the historical target.

[0064] S205: In response to obtaining multiple labeled video frames, the obtained multiple labeled video frames are all used as video frames to be detected, and based on the frame sequence number and annotation data of the video frames to be detected, the annotation data corresponding to the historical targets in the video frames to be detected are corrected to determine the annotation data that finally matches the historical targets in the video frames to be detected.

[0065] Specifically, after completing the labeling work of all video frames to be labeled or accumulating a large number of labeled video frames, the labeled video frames are used as video frames to be detected in order to determine the accuracy of the labeled data of the video frames to be detected, and the labeled data and frame numbers of the video frames to be detected are used to determine whether the video frames to be detected are mislabeled and / or missed, so as to correct the labeled data corresponding to the historical targets in the video frames to be detected, determine the final matching labeled data of the historical targets in the video frames to be detected, and improve the accuracy of the final matching labeled data, so as to use the labeled video frames to train the target tracking model and improve the training effect of the tracking model.

[0066] Furthermore, targets usually exist continuously in video frames, so historical targets corresponding to the same identifier should usually appear multiple times in consecutive video frames. When the same annotation data does not appear continuously in the frame sequence numbers of the video frames to be detected, the historical targets may be missed. If the historical targets are indeed missed, the missed historical targets are corrected. When the number of times the annotation data appears is less than the number threshold, the historical targets may be mislabeled. If the historical targets are indeed mislabeled, the mislabeled historical targets are corrected to improve the accuracy of target labeling.

[0067] In one application, see Figure 3 , Figure 3 This application Figure 2 A flowchart of an implementation method corresponding to step S205 in FIG. 1 is shown in FIG. 1 , wherein step S205 specifically includes:

[0068] S301: Determine a missing label detection result of the historical target in the video frame to be detected based on the frame sequence number of the video frame to be detected and the label data corresponding to the historical target in the video frame to be detected.

[0069] Specifically, the historical targets with the same identification in the video frame to be detected are regarded as the same target, and the frame numbers of the video frames with the same historical targets marked with annotation data are determined based on the annotation data of the historical targets. The missed label detection result of the historical targets is determined by detecting the continuity of the frame numbers.

[0070] In one application scenario, the frame numbers of the video frames to be detected corresponding to the same identifier are extracted. If the frame numbers are continuous from the beginning to the end, no prompt is given. If there are two or more frame number intervals in the frame number, it is prompted that the historical target corresponding to the identifier between the frame number intervals may be missed.

[0071] In one application, based on the frame number of the video frame to be detected and the annotation data corresponding to the historical target in the video frame to be detected, the missing mark detection result of the historical target in the video frame to be detected is determined, including: sorting the video frames to be detected corresponding to the historical target with the same identification according to the frame number to obtain at least one frame number interval segment corresponding to the historical target with the same identification; in response to the number of frame number interval segments being one, determining the missing mark detection result of the historical target in the video frame to be detected as no missing mark has occurred; in response to the number of frame number interval segments exceeding one, determining the number of frames between any two adjacent frame number interval segments, when the number of frames between any two adjacent frame number interval segments is greater than or equal to the interval interval threshold, determining the missing mark detection result of the historical target in the video frame to be detected as no missing mark has occurred, and when there is at least one group of two adjacent frame number interval segments with the number of frames between the interval being less than the interval interval threshold, determining the missing mark detection result of the historical target in the video frame to be detected as missing mark has occurred.

[0072] Specifically, the annotated video frames of the historical target corresponding to the same identifier are sorted from small to large according to the frame number, and at least one frame number interval corresponding to the historical target is obtained. If there is only one frame number interval, the missing label detection result of the historical target corresponding to only one frame number interval is that no missing label occurs. When more than one frame number interval is obtained, the number of frames between every two adjacent frame number intervals is determined, wherein the number of frames between the intervals includes the number of video frames between two adjacent frame number intervals, that is, the maximum frame number in the frame number interval with a smaller frame number. The number of video frames between the minimum frame number in the frame number interval segment with a larger frame number. If the number of frames between any two adjacent frame number interval segments is less than the interval threshold, it is determined that there is a missing mark in the historical target, and the output missing mark detection result is that a missing mark has occurred. If the number of frames between any two adjacent frame number interval segments is greater than or equal to the interval interval threshold, it is determined that the historical target appears in different time periods and no missing mark has occurred, and the output missing mark detection result is that no missing mark has occurred, thereby distinguishing between missing marks and scenes where targets appear in different time periods, and improving the accuracy of missing mark judgment.

[0073] In a specific application scenario, the interval threshold is 20 video frames, and the frame numbers of all the identifiers in a video data segment are counted. Assuming that there are 200 frames of video data with three historical targets, the frame number intervals of the first historical target are [1-110] and [120-200], respectively, and the interval between the two frame number intervals is less than the interval threshold, then the historical target may have missed labels between the frame number interval [111-119], and the missed label detection result corresponding to the historical target is output as missed labels. The frame number interval of the second historical target is [1-100] and has not appeared since then, then it is determined that the historical target has not missed labels, and the missed label detection result corresponding to the historical target is output as no missed labels. The frame number intervals of the third historical target are [1-50] and [50-200], respectively, and the interval between the two frame number intervals is greater than the interval threshold, then it is determined that the historical target entered the monitoring area at different time periods, and the historical target has not missed labels, and the missed label detection result of the historical target is output as no missed labels. By detecting the relationship between the frame sequence number interval and the interval threshold, the missed label detection result of the historical target can be determined more accurately. It can be understood that the interval threshold can be customized based on the frame interval between video frames in the video data.

[0074] S302: Determine a mislabeled detection result of the historical target in the video frame to be detected based on the annotation data corresponding to the historical target in the video frame to be detected.

[0075] Specifically, historical targets with the same identification in the video frame to be detected are regarded as the same target, and the mislabeled detection result of the historical targets is determined by detecting the consistency of the target types in the labeled data corresponding to the same historical target.

[0076] In an application scenario, the number of target types in the labeled data corresponding to the same identifier is theoretically one, and the mislabeling detection result of the historical target is determined by counting the number of target types in the labeled data of the historical target corresponding to the same identifier.

[0077] In one application method, based on the annotation data corresponding to the historical targets in the video frame to be detected, the mislabeling detection result of the historical targets in the video frame to be detected is determined, including: classifying the historical targets in the video frame to be detected with the same identification, classifying the historical targets with the same target type into one category, and obtaining the number of classifications corresponding to the historical targets in the video frame to be detected with the same identification; in response to the number of classifications being one, determining that the mislabeling detection result of the historical targets in the video frame to be detected is that no mislabeling has occurred; in response to the number of classifications being more than one, determining that the mislabeling detection result of the historical targets in the video frame to be detected is that mislabeling has occurred.

[0078] Specifically, based on the target type in the labeled data corresponding to the same historical target, the historical targets are classified, and the targets with the same type are determined to be the same type of historical targets, and the number of classifications corresponding to the historical targets in the video frames to be detected with the same identification is obtained. If the number of classifications exceeds one, it is determined that the historical target is mislabeled, and the output error labeling detection result is that mislabeling has occurred. If the number of classifications is one, it is determined that the historical target is not mislabeled, and the output error labeling detection result is that no mislabeling has occurred.

[0079] In a specific application scenario, the target types in the annotation data corresponding to the historical targets with the same identification are counted. If there is more than one target type, it is determined that the historical target is mislabeled. If there is only one target type, it is determined that the historical target is not mislabeled. The mislabeling detection result corresponding to each historical target is output. The frame number and target type in the annotation data can be used to determine whether the historical target is mislabeled, so that the annotation data corresponding to the mislabeled historical target can be reviewed and corrected.

[0080] S303: Based on the missing label detection result and / or the mislabel detection result, correct the label data corresponding to the historical target that has been missed and / or mislabeled, and obtain the label data that finally matches the historical target in the video frame to be detected.

[0081] Specifically, after obtaining the missing label detection results and / or mislabeling detection results corresponding to each historical target, the historical targets with missing labels and / or mislabeling are reviewed, so as to correct the labeling data of the historical targets that are determined to have missing labels and / or mislabeling, and use the corrected labeling data as the final matching labeling data of the historical targets.

[0082] In a specific application scenario, obtaining a video frame to be labeled includes: receiving a video frame in a target video sent by a client; wherein the video frame is a video frame correspondingly displayed after the client pauses playing the target video; and using the video frame as a video frame to be labeled.

[0083] Specifically, the client plays the target video. When the client pauses the playback of the target video, the corresponding video frame displayed after the target video is paused is uploaded to the server, so that the server receives the video frame in the target video sent by the client and uses the video frame uploaded by the client as the video frame to be marked.

[0084] Optionally, the target video is played based on a time sequence, and the client uploads video frames in the target video sequentially, or the client uploads key frames in the target video sequentially.

[0085] Furthermore, based on the frame number and annotation data of the video frame to be detected, the annotation data corresponding to the historical target in the video frame to be detected is corrected, and after the annotation data of the historical target in the video frame to be detected is determined to be finally matched, it also includes: returning the annotated video frame to the client, and the annotated video frame returned to the client includes: the annotated target and its corresponding final matching annotation data.

[0086] Specifically, after the correction of the labeled video frames is completed, the labeled video frames are returned to the client. The labeled video frames returned to the client include the labeled targets and their corresponding final matching annotation data, so that any client that receives the labeled video frames can use the labeled targets and their corresponding final matching annotation data to train the target tracking model, thereby realizing automation of target labeling and correction on the server side and improving the efficiency of the client in obtaining labeled video frames.

[0087] In this embodiment, the stored annotation data is used to annotate the existing targets in the video frame to be annotated, and annotation data is generated for the new target based on the stored annotation data. The annotation data matching the new target is stored, so as to improve the annotation efficiency and reduce the probability of duplication of annotation data, so as to facilitate the missed and mislabeled detection of historical targets in multiple annotated video frames, and correct the annotation data based on the missed and / or mislabeled detection results to improve the accuracy of the annotation data.

[0088] See also Figure 4 , Figure 4It is a schematic diagram of the topological structure of an implementation method of a target labeling system of the present application, wherein the target labeling system 40 includes a client 401 and a server 402, wherein the client 401 is used to upload a video frame to the server 402, the server 402 obtains the video frame to be labeled, determines all the targets to be labeled in the video frame to be labeled, and determines the target type of the video frame to be labeled based on at least one historical target in the labeled video frame, wherein the target type includes newly added targets and existing targets, the newly added targets include targets that are different from historical targets, and the existing targets include targets that match any historical targets in the labeled video frame, based on the stored labeling data of the historical targets and the target type of the newly added targets, generates labeling data for the newly added targets and labels the newly added targets, and / or, uses the labeling data of the historical targets that match the existing targets to label the existing targets, wherein the labeling data includes the target type and identifier of the target.

[0089] In a specific application scenario, the target annotation system 40 is opened, and the client 401 plays the target video to be annotated. The target video can only be annotated when it is in a paused playback state. When the video is played to a specified frame, a pause operation is performed. The specified frame is the frame to be annotated, and the specified frame is uploaded to the server 402. The server 402 obtains the video frame to be annotated, and determines the target type of the target to be annotated in the video frame to be annotated. The target type has been preset when the annotation task is created. For example, if the items in the car in the video are to be annotated, the target type can be a pendant, a sun visor, a perfume box, a tissue box, a pillow, a hat, an advertising sticker, etc. A unique identifier will be generated each time annotated to correspond to the target to be annotated.

[0090] Furthermore, to ensure the uniqueness of each target to be labeled, when the target to be labeled is a newly added target, the server 402 will store a maximum identifier for each frame. The maximum identifier is the largest identifier among the labeled targets in each frame. When a new identifier is to be added to a frame, an identifier will be applied for first, and the maximum identifier will be assigned to the maximum identifier + 1. The latest maximum identifier will be used as the identifier of the newly added target. After generating the labeled data, the server 402 will not save it immediately, but will extract the identifier in the data and compare it with the identifiers of all the labeled data in the current frame. If the extracted identifier is repeated with the labeled data in the current frame, it is considered illegal and a duplicate identifier error message is returned to the client 401. The client 401 applies for an identifier again. If the extracted identifier is unique, it is considered legal and the labeled data of the newly added target is generated.

[0091] Furthermore, the annotation data adopts a combination of target type and identifier, and the annotation data is stored in the target list. The annotation data of the stored historical targets are stored in the target list, and the identifiers in the target list are sorted from small to large. The largest identifier in the annotation data corresponding to the historical target is determined from the target list as a reference identifier. When the annotation data corresponding to the newly added target in the target list is arranged from small to large according to the identifier, the latest identifier can be extracted from the target list to obtain the reference identifier, thereby improving the extraction efficiency.

[0092] In a specific application scenario, suppose that a certain frame A is labeled first. There are two targets in frame A, one is a motor vehicle and the other is a non-motor vehicle. The type of motor vehicle is selected and the motor vehicle is labeled, and the obtained identification is 1. The type of non-motor vehicle is selected and the non-motor vehicle is labeled, and the obtained identification is 2. The targets of non-motor vehicles and motor vehicles are added to the target list. The target list will have two types of annotation data, "motor vehicle-target identification 1" and "non-motor vehicle-target identification 2". When the video switches from frame A to frame B for annotation, assuming that frame B also has the same motor vehicle as frame A, it is annotated with "motor vehicle-target identification 1" in the target list. In this way, the types and identifications of the same motor vehicle targets in frames B and A are kept consistent, and the annotation efficiency is high.

[0093] Furthermore, if the labeling is completed, multiple labeled video frames are obtained, and the target types of the same target identification are counted to see whether they are consistent. If there is an inconsistency, it indicates that the target identification is mislabeled. It is also counted whether the same target identification is a continuous frame, that is, whether the frame number is continuous. If the frame number is not continuous, it indicates that the target may be missing.

[0094] In a specific application scenario, all the targets stored in the labeled video frames, that is, the video frames to be detected, are traversed. If the frame number of a target is discontinuous, there is a possibility of missing the label, which needs to be corrected. If the type of a target is inconsistent in different frames, it can be considered that the target is mislabeled and needs to be corrected.

[0095] The above scheme is applied to the scenario where the client 401 and the server 402 interact. The server 402 labels the target to be labeled, determines the labeling data for the new target that is different from the historical target, and improves the accuracy of the target labeling. Based on the above labeling method, the labeling data can also be verified to ensure the uniqueness of the labeling data corresponding to each target.

[0096] See also Figure 5 , Figure 5It is a schematic diagram of the topological structure of an embodiment of an electronic device of the present application. The electronic device 50 includes a memory 501 and a processor 502 coupled to each other, wherein the memory 501 stores program data (not shown), and the processor 502 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0097] See also Figure 6 , Figure 6 It is a topological structure diagram of an implementation method of a computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program data 600. When the program data 600 is executed by the processor, the method in any of the above embodiments is implemented. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0098] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0101] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target labeling method, characterized in that: The method comprises: Obtain a video frame to be labeled, and determine all objects to be labeled in the video frame to be labeled; Determine the target type of the video frame to be labeled based on at least one historical target in the labeled video frame; wherein the target type includes a newly added target and an existing target, the newly added target includes a target that is different from the at least one historical target, and the existing target includes a target that matches any historical target in the at least one historical target; Based on the stored annotation data of the at least one historical target and the target type of the newly added target, generate annotation data of the newly added target and annotate the newly added target; and / or, annotate the existing target using the annotation data of the historical target that matches the existing target; wherein the annotation data includes the target type and identifier of the target; In response to obtaining a plurality of the labeled video frames, the plurality of the labeled video frames are all used as the video frames to be detected, and based on the frame sequence numbers and the labeled data of the video frames to be detected, the labeled data corresponding to the historical targets in the video frames to be detected are corrected to determine the labeled data that the historical targets in the video frames to be detected finally match; wherein, when the historical targets in the video frames to be detected are corrected, there is a missing label detection result corresponding to the missing label detection result; The missing mark detection result is obtained based on the following steps: sorting the video frames to be detected corresponding to the historical targets with the same identification according to the frame number to obtain at least one frame number interval segment corresponding to the historical targets with the same identification; in response to the number of the frame number interval segments being one, determining the missing mark detection result of the historical targets in the video frame to be detected as no missing mark has occurred; in response to the number of the frame number interval segments being more than one, determining the number of frames between any two adjacent frame number interval segments, when the number of frames between any two adjacent frame number interval segments is greater than or equal to the interval interval threshold, determining the missing mark detection result of the historical targets in the video frame to be detected as no missing mark has occurred, and when there is at least one group of two adjacent frame number interval segments with the number of frames between the interval being less than the interval interval threshold, determining the missing mark detection result of the historical targets in the video frame to be detected as missing mark has occurred.

2. The target labeling method according to claim 1, characterized in that: The generating, based on the stored annotation data of the at least one historical target and the target type of the newly added target, annotation data of the newly added target and annotating the newly added target comprises: Among the stored annotation data of the at least one historical target, taking the identifier with the largest or smallest value as a reference identifier; Based on the number of the newly added targets and the value of the reference identifier, obtaining an identifier corresponding to each of the newly added targets; Based on the target type and identifier of each newly added target, generating the labeling data matched by each newly added target; Each of the newly added targets is labeled with the newly generated labeling data, and the labeling data matched by the newly added targets is stored.

3. The target labeling method according to claim 2, characterized in that: Before labeling each of the newly added targets with the newly generated labeling data and storing the labeling data matched by the newly added targets, the method further includes: The identifiers in the newly generated annotation data are compared with the identifiers of the objects that have been annotated in the video frame to be annotated, so as to determine whether each newly generated identifier is different from the identifier of any object that has been annotated in the video frame to be annotated.

4. The target labeling method according to claim 1, characterized in that: The method of correcting the annotation data corresponding to the historical target in the video frame to be detected based on the frame sequence number and the annotation data of the video frame to be detected, and determining the annotation data that the historical target in the video frame to be detected finally matches, includes: Determine a missing label detection result of the historical target in the video frame to be detected based on the frame sequence number of the video frame to be detected and the label data corresponding to the historical target in the video frame to be detected; and Determining a mislabeled detection result of the historical target in the video frame to be detected based on the labeling data corresponding to the historical target in the video frame to be detected; Based on the missing label detection result and / or the mislabel detection result, the label data corresponding to the historical target with missing labels and / or mislabeling is corrected to obtain the label data that finally matches the historical target in the video frame to be detected.

5. The target labeling method according to claim 4, characterized in that: The determining, based on the labeling data corresponding to the historical target in the video frame to be detected, a mislabeled detection result of the historical target in the video frame to be detected includes: Classifying the historical targets in the to-be-detected video frames with the same identifier, classifying the historical targets with the same target type into one category, and obtaining the number of categories corresponding to the historical targets in the to-be-detected video frames with the same identifier; In response to the number of classifications being one, determining that a mislabeling detection result of the historical target in the to-be-detected video frame is that no mislabeling occurs; In response to the number of classifications exceeding one, determining that a mislabeling detection result of the historical target in the to-be-detected video frame is that mislabeling occurs.

6. The target labeling method according to claim 1, characterized in that: The step of obtaining a video frame to be labeled includes: Receiving a video frame in a target video sent by a client; wherein the video frame is a video frame correspondingly displayed by the client after pausing the playing of the target video; Using the video frame as a video frame to be labeled; After correcting the annotation data corresponding to the historical target in the video frame to be detected based on the frame sequence number and the annotation data of the video frame to be detected and determining the annotation data that the historical target in the video frame to be detected finally matches, the method further includes: The labeled video frame is returned to the client, and the labeled video frame returned to the client includes: the labeled target and its corresponding final matching label data.

7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images