Object labeling information modification method, program product, equipment and storage medium

Abstract

The embodiment of the invention provides an object annotation modification method, a program product, equipment and a storage medium. The method comprises the steps of displaying continuous multi-frame images where an annotated object is located in a first display area of a user interface, and displaying a first-frame image where the annotated object is located in a second display area of the user interface; the image content of the image comprises a labeling object and carries labeling information; in response to an annotation information modification instruction of a to-be-modified frame input by a user, modifying annotation information of a first annotation object in the to-be-modified frame into target annotation information; the first annotation object is an annotation object of to-be-modified annotation information in the to-be-modified frame; the target annotation information is annotation information of the first annotation object in a target first frame, and the target first frame is a first frame image where the first annotation object is located. According to the method, the user quickly positions the target first frame where the first labeling object is located from the second display area, correct target labeling information is obtained from the target first frame, the modification efficiency is improved, and the modification accuracy is ensured.

Description

Technical Field

[0001] The present application relates to the field of annotation technology, and in particular to an object annotation modification method, program product, device and storage medium. Background Art

[0002] At present, with the widespread application of digital image and video data, it is increasingly important to quickly and accurately label objects in continuous frames. However, traditional object labeling methods often have problems such as low labeling efficiency and cumbersome operation, especially when a large number of object labeling information needs to be modified in batches. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an object annotation modification method, program product, device and storage medium to achieve the technical effect of improving the efficiency and accuracy of object annotation modification.

[0004] A first aspect of an embodiment of the present application provides a method for modifying object annotation information, the method comprising:

[0005] Displaying multiple consecutive frames of images where each annotated object is located in a first display area of ​​the user interface, and displaying a first frame of image where each annotated object is located in a second display area of ​​the user interface; wherein the image content of each frame of the image includes one or more annotated objects, and each frame of the image carries annotation information of the included annotated objects;

[0006] In response to a user inputted annotation information modification instruction for a frame to be modified, the annotation information of a first annotated object in the frame to be modified is modified to target annotation information; wherein, the first annotated object is an annotated object of the annotation information to be modified in the frame to be modified; the target annotation information is annotation information of the first annotated object in a target first frame, and the target first frame is a first frame image where the first annotated object is located.

[0007] In the above implementation process, by designing the first display area and the second display area in the user interface, the user can more clearly compare whether the annotation information of each annotated object in the continuous multiple frames is consistent by using the grouped tiled display method. And the first frame image of each annotated object is displayed in the second display area, so that when modifying the annotation information of the first annotated object, the target first frame where the first annotated object is located can be quickly located from the second display area, so that the correct target annotation information can be obtained from the target first frame, without the user switching to the previous and next frames of the frame to be modified to find the correct annotation information, and it can also avoid the situation where the annotation information is modified incorrectly due to the disorder of annotation in continuous multiple frames, thereby improving the modification efficiency and ensuring the modification accuracy.

[0008] Furthermore, the method further comprises:

[0009] The image frame after the modification of the annotation information is moved to the continuous multiple-frame image corresponding to the target annotation information for display.

[0010] In the above implementation process, the modified image frame is moved to the continuous multi-frame image corresponding to the target annotation information for display, so that the image frame grouping in the video collection is displayed correctly, and the user can verify whether the modification is correct in the first area after the updated display, ensuring the accuracy of the modification and providing a convenient modification verification method.

[0011] Furthermore, the frame to be modified includes multiple frames; before modifying the annotation information of the first annotated object in the frame to be modified to the target annotation information, the method further includes:

[0012] Determining that the first annotated objects in the plurality of frames to be modified belong to the same object category; and / or

[0013] It is determined that the multiple frames to be modified are different frames in the video set.

[0014] In the above implementation process, in the case of multiple frames to be modified, by judging whether each first annotated object in the multiple frames to be modified belongs to the same category, and judging whether the multiple frames to be modified are different frames in the video set, the user is prevented from misselecting an image frame, thereby ensuring the accuracy of the modification of the annotation information.

[0015] Furthermore, before modifying the annotation information of the first annotated object in the frame to be modified into the target annotation information, the method further includes:

[0016] If the modification instruction indicates the target first frame, determining that a second annotated object indicated by the target annotation information in the target first frame belongs to the same object category as the first annotated object;

[0017] If the modification instruction does not indicate the target first frame, the target first frame where the first labeled object is located is matched from the second display area.

[0018] In the above implementation process, when the modification instruction indicates that there is a target first frame, by judging whether the second annotated object and the first annotated object belong to the same object category, the situation of wrongly modifying the annotation information is avoided, and the accuracy of the modification is ensured. When the modification instruction does not indicate the target first frame, the target first frame where the first annotated object is located is automatically matched in the second display area to improve the degree of automation.

[0019] Furthermore, the step of modifying the annotation information of the first annotated object in the frame to be modified into target annotation information includes:

[0020] If the frame to be modified already carries the target annotation information, and the target annotation information indicates a third annotated object in the frame to be modified, obtaining a component attribute of the third annotated object, where the component attribute is used to indicate whether the third annotated object has multiple components;

[0021] Based on the component attribute, at least the annotation information of the first annotation object is modified to the target annotation information.

[0022] In the above implementation process, when it is detected that the frame to be modified already carries the target annotation information, different modification strategies are selected based on the component attributes of the third annotation object, which not only supports one-to-many modification of annotation information under multiple component attributes, but also avoids mistakenly changing the annotation information under a single component attribute to a one-to-many annotation relationship, thereby ensuring the accuracy of the modification.

[0023] Further, the modifying at least the annotation information of the first annotation object to the target annotation information based on the component attribute includes:

[0024] If the component attribute is a multi-component attribute, modifying the annotation information of the first annotated object to the target annotation information;

[0025] If the component attribute is a single-component attribute, the annotation information of the first annotated object is modified to the target annotation information, and new annotation information is created for the third annotated object in the frame to be modified.

[0026] In the above implementation process, based on the component attributes of the third labeled object, it is chosen to directly modify the annotation information of the first labeled object, or to create new annotation information for the third labeled object while modifying it, so that the annotation information modification of labeled objects with multiple component attributes and single component attributes can be supported at the same time.

[0027] Furthermore, the method further comprises:

[0028] In response to a new annotation information creation instruction input by the user for the frame to be modified, new annotation information is created for the first annotated object in the frame to be modified.

[0029] A second aspect of an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, any method described in the first aspect is implemented.

[0030] According to a third aspect of the present application, an electronic device is provided, the electronic device comprising:

[0031] processor;

[0032] a memory for storing processor-executable instructions;

[0033] Wherein, when the processor calls the executable instruction, it implements the operation of any method described in the first aspect.

[0034] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a method for modifying object annotation information provided in an embodiment of the present application;

[0037] Figure 2-Figure 3 A schematic diagram of a user interface provided in an embodiment of the present application;

[0038] Figure 4 A flowchart of another method for modifying object annotation information provided in an embodiment of the present application;

[0039] Figure 5 A hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0041] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0042] In the related art, the labeler checks frame by frame whether the labeling information of the same object in multiple consecutive frames of images is consistent by playing continuous frames. The labeling information is, for example, a labeling ID (Identity document). If the image includes multiple labeled objects, the image needs to be divided into blocks, and then continuous frames are played for each block for frame-by-frame inspection. If it is observed that a certain object is incorrectly labeled in some frames, the labeling information is modified, and then the playback inspection is continued until all frames are checked. Repeat the cycle and continue to check other block areas in the image in the same way until all labeled objects are checked. However, when the number of frames is large or there are many objects in a single frame, this inspection method not only leads to a long labeling time, affecting the labeling efficiency, but also easily causes missed detection errors, affecting the accuracy of labeling.

[0043] In addition, the situation of disordered annotation information is very complicated. When modifying the annotation information of a labeled object in the current frame, it is often necessary to switch to the previous and next frames of the current frame to determine the correct annotation information of the labeled object, and then switch back to the current frame to manually enter the correct annotation information. However, the operation of switching frames back and forth is cumbersome and time-consuming. If the same labeled object has disordered annotation information in multiple consecutive frames, it may cause modification errors. Therefore, how to improve the efficiency and accuracy of batch modification of annotation information of a large number of objects has become a technical problem that needs to be solved urgently in this field.

[0044] To this end, the present application provides a method for modifying object annotation information, including: Figure 1 Steps 110 to 130 are shown.

[0045] Step 110: Displaying multiple consecutive frames of images where each annotated object is located in a first display area of ​​the user interface, and displaying a first frame of image where each annotated object is located in a second display area of ​​the user interface.

[0046] The image content of each frame of the image includes one or more of the annotated objects, and each frame of the image carries the annotation information of the included annotated objects.

[0047] Exemplarily, a video set includes a plurality of continuous frames of images, the image content of each frame of image includes one or more annotated objects, and each frame of image carries the annotation information of the annotated objects included therein. The annotation information includes, for example, but is not limited to, object classification, and annotation ID, etc. The annotation information may be automatically generated in an upstream link or obtained by manual annotation. The plurality of frames of images in the video set may be displayed in a user interface for users, such as annotators, to view and modify the annotation information.

[0048] Optionally, before executing step 110, the method may further include: obtaining a user input annotation object selection instruction, and determining that the continuous multiple frames of images where each selected annotation object is located and the first frame of image where each annotation object is located are the image frames to be displayed. Exemplarily, the user can filter out the objects to be inspected according to the object classification, thereby obtaining the image frames to be inspected and displayed.

[0049] The user interface includes at least a first display area and a second display area. The first display area is used to display a plurality of consecutive frames of images where each labeled object is located. Each frame of the consecutive multiple frames of images carries the labeling information of the corresponding labeled object. It is worth noting that if the image content of some image frames includes multiple labeled objects, then these image frames will be repeatedly displayed multiple times in the first display area. For example, if the image content of frames 5-20 in a video collection includes a labeled object vehicle, and the image content of frames 10-30 includes a labeled object pedestrian, this means that the image content of frames 10-20 includes multiple labeled objects. The first display area is grouped by labeled objects, and the plurality of consecutive frames where the labeled objects are located are displayed in groups. Then the first display area will display frames 5-20 where the labeled object vehicle is located, and frames 10-30 where the labeled object pedestrian is located, so frames 10-30 will be repeatedly displayed in the first display area.

[0050] Figure 2 One embodiment of the user interface is shown. Figure 2 As shown, the first display area 210 can be designed as a grid display area. The grid display area includes a plurality of grids, and the continuous multiple frames of images are displayed in the form of grid tile arrangement, and each grid displays a frame of image. Figure 2 The first display area 210 displays the 1st to 6th frames where the labeled object car 1 is located, and each frame image carries the identification information of "car1", displays the 1st to 14th frames where the labeled object car 2 is located, and each frame image carries the identification information of "car2", and so on.

[0051] Optionally, the continuous multiple frames of images displayed in the first display area may be a complete image, or a partial image area including the labeled object in the image, such as Figure 2 shown.

[0052] The user interface also includes a second display area, which is used to display the first frame image of each annotated object. The first frame image can be a complete image in which the annotated object appears for the first time, or a scaled image of the annotated object in the complete image in which the annotated object appears for the first time. The first frame image carries the annotation information of the corresponding annotated object.

[0053] For example, in the above example, if the image content of the 5th to 20th frames of the video set includes the labeled object vehicle, and the image content of the 10th to 30th frames includes the labeled object pedestrian, then the first frame image of the labeled object vehicle can be the complete image of the 5th frame, or a thumbnail of the labeled object vehicle in the complete image of the 5th frame. Figure 2 As shown, the labeled object car 1 first appears in the first frame, so the second display area 220 displays a thumbnail of the labeled object car 1 in the first frame, and so on.

[0054] In addition, considering that the image content of the image frame may include multiple annotated objects, and in order to facilitate the annotator to quickly locate and match the annotated objects from the image frame, optionally, each frame image of the first display area and / or the first frame image of each frame of the second display area may also be annotated with an identification frame of the annotated object. The identification frame can be of any shape. Taking a rectangular identification frame as an example, the rectangular identification frame can be obtained by traversing the pixels of the annotated object in the image frame and taking the minimum and maximum values ​​of the pixels as the coordinates of the upper left corner and the lower right corner of the rectangular identification frame.

[0055] Optionally, if the first frame image is a thumbnail image of the marked object in the complete image where it first appears, the identification frame can be enlarged outward by a preset ratio, such as 10%, and the image obtained after the enlargement is used as the thumbnail, that is, the first frame image.

[0056] Step 130: In response to a user inputted instruction for modifying annotation information of a frame to be modified, modify the annotation information of a first annotated object in the frame to be modified to target annotation information.

[0057] The first annotated object is an annotated object whose annotation information is to be modified in the frame to be modified; the target annotation information is annotation information of the first annotated object in the target first frame, and the target first frame is the first frame image where the first annotated object is located.

[0058] Exemplarily, the user can enter a modification instruction in the user interface. For example, the user can select the frame to be modified in the first display area and enter a modification instruction. The modification instruction is used to instruct to modify the annotation information of the frame to be modified. It can be understood that since the first display area is grouped by each annotated object and displays the continuous multiple frames of images where the annotated objects are located in groups, the frame to be modified is at least a part of the image in the continuous multiple frames where a certain annotated object is located. Therefore, when selecting the frame to be modified, it can be determined which annotated object in the frame to be modified has its annotation information modified. The annotated object whose annotation information is to be modified in the frame to be modified is called the first annotated object.

[0059] After receiving the modification instruction, the first frame image where the first annotated object is located, that is, the target first frame, can be determined. For example, the target first frame of the first annotated object is determined in the second display area. Then, the annotation information of the first annotated object in the target first frame is determined as the target annotation information, and the annotation information of the first annotated object in the frame to be modified is modified to the target annotation information.

[0060] As in the above example, the image content of the 5th to 20th frames of the video set includes a labeled object vehicle, and the labeling ID of the labeled object vehicle is "100". However, in the 13th frame, the labeling ID is disordered and becomes "001". Then, when the labeling information modification instruction for the frame to be modified, the target first frame, the 5th frame, where the first labeled object vehicle is located is first determined. Then, the target labeling information "100" of the first labeled object vehicle in the target first frame, the 5th frame, is determined, and the labeling information of the first labeled object vehicle in the frame to be modified, the 13th frame, is modified to the target labeling information "100", thereby completing the correction of the labeling information.

[0061] It can be seen that the present application designs a first display area and a second display area in the user interface, and uses the first display area to display the continuous multiple frames where each annotated object is located in groups. Compared with the inspection method of playing continuous frames in the related art, the method of tiled display of continuous multiple frames allows the user to more clearly compare whether the annotation information of each annotated object in the continuous multiple frames is consistent in the first display area. And the first frame image where each annotated object is located is displayed in the second display area, so that when modifying the annotation information of the first annotated object, the target first frame where the first annotated object is located can be quickly located from the second display area, so that the correct target annotation information can be obtained from the target first frame, without the user switching to the previous and next frames of the frame to be modified to find the correct annotation information, and can better avoid the situation where the annotation information is modified incorrectly due to the confusion of annotations in multiple consecutive frames, thereby improving the modification efficiency while ensuring the modification accuracy.

[0062] The following describes steps 110 to 130 in detail.

[0063] According to some embodiments of the present application, the annotation information modification method further includes step 140.

[0064] Step 140: moving the image frame after the modification of the annotation information to the continuous multiple frames of images corresponding to the target annotation information for display.

[0065] It is understandable that before the annotation information is modified, since the annotation information about the first annotated object carried by the frame to be modified is wrong, the frame to be modified will be grouped and displayed under the wrong annotated object in the first area. In this embodiment, the image frame after the annotation information modification is completed is moved to the continuous multi-frame image corresponding to the target annotation information for display, so that the image frames in the video set are grouped correctly.

[0066] For instance, in the above example, the labeled object vehicle appears in the 5th to 20th frames of the video collection, and the labeled object vehicle's labeling ID is "100". However, the labeled object vehicle's labeling ID becomes "001" in the 13th frame. If the labeling ID "001" corresponds to the labeled object tree, then the 13th frame will be incorrectly displayed in the continuous multi-frame image corresponding to the labeled object tree. At the same time, only the 5th to 12th frames and the 14th to 20th frames will be displayed for the labeled object vehicle in the first area. After the labeling ID of the labeled object vehicle in the 13th frame is modified to "100", the modified 13th frame can be moved to the continuous multi-frame image corresponding to the labeling ID "100" for display, that is, moved to the continuous multi-frame image corresponding to the labeled object vehicle for display.

[0067] It can be seen that this embodiment moves the modified image frame to the continuous multi-frame image corresponding to the target annotation information for display, so that the image frame grouping in the video set is displayed correctly, and the user can verify whether the modification is correct in the first area after the updated display, ensuring the accuracy of the modification and providing a convenient modification verification method.

[0068] According to some embodiments of the present application, the frame to be modified may be one frame or multiple frames. Exemplarily, the user may input modification instructions for multiple frames to be modified in the multiple selection mode, and multiple selections across annotated objects are supported. Specifically, in the multiple selection mode, the user may check multiple image frames whose annotation information needs to be modified in the first display area as frames to be modified. Alternatively, the user may select all image frames where the currently annotated object is located in the first display area or the second display area as frames to be modified, and may continue to check other frames to be modified. Figure 3 It shows that the user has selected multiple frames to be modified in the first display area, and the frames to be modified are highlighted in yellow.

[0069] In the case that the frame to be modified includes multiple frames, before executing step 130 to modify the annotation information of the first annotated object in the frame to be modified to the target annotation information, step 121 and / or step 122 are first executed.

[0070] Step 121: Determine whether the first labeled objects in multiple frames to be modified belong to the same object category.

[0071] In the case of multiple frames to be modified, the first annotated object in each frame to be modified may be the same or different. Therefore, in the case of selecting multiple frames to be modified, first determine whether the first annotated objects in all frames to be modified belong to the same object category. If they belong to the same object category, continue to execute the subsequent steps; if there are at least two frames to be modified whose corresponding first annotated objects do not belong to the same object category, then the subsequent annotation information modification action will cause objects of different categories to be annotated with the same annotation information, resulting in an annotation error. Therefore, if they do not belong to the same object category, the annotation information modification is automatically stopped, and a prompt message indicating that the modification failed is output.

[0072] Step 122: Determine that the multiple frames to be modified are different frames in the video set.

[0073] As described above, for image frames whose image content includes multiple annotated objects, they will be repeatedly displayed multiple times in the first display area. Then the user may mistakenly select two frames to be modified that belong to the same frame. In order to avoid user misselection, in the case of multiple frames to be modified, first determine whether the selected multiple frames to be modified are from different frames in the video set. If so, continue to execute subsequent steps; if there are at least two frames to be modified that belong to the same frame, in order to avoid modifying the annotation information of two annotated objects in the same frame image to be consistent, automatically stop modifying the annotation information, and output a prompt message that the modification failed.

[0074] It can be seen that in the case of multiple frames to be modified, this embodiment avoids the user from mistakenly selecting an image frame by judging whether each first annotated object in the multiple frames to be modified belongs to the same category, and judging whether the multiple frames to be modified are different frames in the video set, thereby ensuring the accuracy of the modification of the annotation information.

[0075] Based on any of the above embodiments, before executing step 130 to modify the annotation information of the first annotated object in the frame to be modified into the target annotation information, step 123 or step 124 is first executed.

[0076] Step 123: If the modification instruction indicates the target first frame, determine that the second annotated object indicated by the target annotation information in the target first frame belongs to the same object category as the first annotated object.

[0077] Exemplarily, after the user has selected the frame to be modified in the first display area, he can select the target first frame in the second display area and enter a modification instruction. The modification instruction entered at this time indicates the target first frame selected by the user. Since the target first frame is manually selected by the user, in order to avoid the user's wrong selection, before modifying the annotation information, first determine whether the second annotation object indicated by the target annotation information in the target first frame belongs to the same object category as the first annotation object. If they belong to the same object category, continue to execute the subsequent steps; if they do not belong to the same object category, stop modifying the annotation information, and output a prompt message that the modification failed.

[0078] For example, in the above example, the user selects the 13th frame as the frame to be modified. If the user selects a target first frame in the second display area, and the target annotation information corresponding to the target first frame is "100", and the second annotation object indicated by the target annotation information "100" in the target first frame is a vehicle, which belongs to the same "vehicle" category as the first annotation object, then the annotation information of the first annotation object vehicle in the 13th frame to be modified can be modified to the target annotation information "100". Conversely, if the user selects a target first frame in the second display area, and the target annotation information corresponding to the target first frame is "200", and the second annotation object indicated by the target annotation information "200" in the target first frame is a pedestrian, which does not belong to the same object category as the first annotation object vehicle, then the annotation information modification can be stopped, and a prompt message indicating that the modification failed can be output.

[0079] Step 124: If the modification instruction does not indicate the target first frame, match the target first frame where the first annotated object is located from the second display area.

[0080] Exemplarily, after the user has selected the frame to be modified in the first display area, the user can directly input a modification instruction, and the system will automatically match the target first frame where the first annotated object is located in the frame to be modified from the second display area. For example, image recognition technology can be used to match a first frame image whose similarity with the first annotated object is greater than a similarity threshold from multiple first frame images as the target first frame, and in the matched target first frame, the second annotated object indicated by the target annotation information must belong to the same object category as the first annotated object.

[0081] It can be seen that in this embodiment, when the modification instruction indicates that there is a target first frame, by judging whether the second annotated object and the first annotated object belong to the same object category, the situation of incorrectly modifying the annotation information is avoided, thereby ensuring the accuracy of the modification. When the modification instruction does not indicate the target first frame, the target first frame where the first annotated object is located is automatically matched in the second display area to improve the degree of automation.

[0082] On the basis of any of the above embodiments, in step 130 , modifying the annotation information of the first annotated object in the frame to be modified into target annotation information specifically includes steps 131 and 132 .

[0083] Step 131: if the frame to be modified already carries the target annotation information, and the target annotation information indicates a third annotated object in the frame to be modified, then obtaining a component attribute of the third annotated object, where the component attribute is used to indicate whether the third annotated object has multiple components;

[0084] Step 132: Based on the component attribute, at least modify the annotation information of the first annotation object to the target annotation information.

[0085] Exemplarily, each annotated object may carry component attributes. The component attributes are used to indicate whether the annotated object has multiple components. The component attributes of the annotated object can be set according to actual needs, and the same annotated object can have different component attributes in different scenes. Component attributes include multi-component attributes and single-component attributes. Multi-component attributes mean that the annotated object can be split into multiple components, while single-component attributes mean that the annotated object cannot be split into multiple components. In the annotation process, the annotated object with multi-component attributes can be split into multiple components, and the annotation information of each component belonging to the same annotated object is at least partially the same. For example, if the annotated object vehicle has a multi-component attribute, and the components of the annotated object vehicle include a body and a grounding point, etc., then the annotation information of the annotated object body and the annotated object grounding point can both be set to car1, or the annotation information of the annotated object body can be set to car1-1, and the annotation information of the annotated object grounding point can be set to car1-2, and so on. Therefore, for annotated objects with multi-component attributes, their annotation information will correspond to multiple components, and there is a one-to-many relationship between the annotation information and the image content. For annotated objects with single-component attributes, they cannot be split into multiple components. Therefore, the annotation information of annotated objects with single-component attributes only corresponds to one annotated object, and there is a one-to-one relationship between the annotation information and the image content.

[0086] Based on this, when modifying the annotation information of the first annotated object in the frame to be modified, it is necessary to first determine whether the frame to be modified itself already carries the target annotation information. If not, the annotation information of the first annotated object to be modified can be directly modified to the target annotation information. If the frame to be modified itself already carries the target annotation information, it is necessary to determine the component attributes of the third annotated object indicated by the target annotation information in the frame to be modified, and adopt different modification strategies according to the component attributes, that is, at least modify the annotation information of the first annotated object in the frame to be modified to the target annotation information.

[0087] It can be seen that when this embodiment detects that the frame to be modified already carries the target annotation information, different modification strategies are selected based on the component attributes of the third annotated object. It not only supports one-to-many modification of annotation information under multiple component attributes, but also avoids mistakenly changing the annotation information under a single component attribute to a one-to-many annotation relationship, thereby ensuring the accuracy of the modification.

[0088] In some embodiments, step 132 of modifying at least the annotation information of the first annotation object to target annotation information based on the component attribute may specifically include step 1321 or step 1322 .

[0089] Step 1321: If the component attribute is a multi-component attribute, modify the annotation information of the first annotated object to the target annotation information.

[0090] Exemplarily, if the third annotated object has a multi-component attribute, it means that the third annotated object has multiple components, and the target annotation information supports a one-to-many annotation relationship, so the annotation information of the first annotated object in the frame to be modified can be directly modified to the target annotation information. The target annotation information indicates both the first annotated object and the third annotated object in the modified image frame.

[0091] Step 1322: If the component attribute is a single-component attribute, modify the annotation information of the first annotated object to the target annotation information, and create new annotation information for the third annotated object in the frame to be modified.

[0092] Exemplarily, if the third annotated object is a single-component attribute, it means that the third annotated object does not support being split into multiple components, and the target annotation information is a one-to-one annotation relationship, so while modifying the annotation information of the first annotated object in the frame to be modified to the target annotation information, new annotation information is created for the third annotated object in the frame to be modified. The target annotation information indicates the first annotated object in the modified image frame, and the newly created annotation information indicates the third annotated object in the modified image frame.

[0093] It can be seen that this embodiment chooses to directly modify the annotation information of the first annotated object based on the component attributes of the third annotated object, or creates new annotation information for the third annotated object while modifying it, so that it can simultaneously support the modification of annotation information of annotated objects with multiple component attributes and single component attributes.

[0094] According to some embodiments of the present application, the annotation information modification method may further include the step of: creating new annotation information for the frame to be modified in response to an annotation information creation instruction input by the user for the frame to be modified, creating new annotation information for the first annotated object in the frame to be modified.

[0095] It is understandable that, in addition to the modification method of replacing the existing annotation information with the annotation information of the first annotated object, the user can also choose to create new annotation information for the first annotated object in the frame to be modified. Similarly, the frame to be modified may include one or more frames. In the case where the frame to be modified includes multiple frames, the user can avoid misselecting the image frame by executing step 121 and / or step 122 to ensure the accuracy of the modification of the annotation information. In addition, the image frame carrying the newly created annotation information can be moved to the multiple frames of images corresponding to the newly created annotation information for display.

[0096] In addition, the present application also provides a Figure 4 The object annotation information modification method shown. Specifically, after entering the filtering grid view, the user can filter the annotated objects that need to be checked according to the object classification. In response to the annotated object selection instruction, the continuous multiple frames of images where the annotated objects that meet the filtering conditions are located can be tiled and displayed in the first display area of ​​the user interface in the order of annotation ID and frame number. The first display area is also called the grid display area. And the first frame image where the annotated object is located is displayed in the second display area of ​​the user interface. Each first frame image carries the annotation ID of the corresponding annotated object. The first frame image is a thumbnail image of the annotated object in the complete image where it first appears. The second display area is also called the thumbnail area. At the same time, each frame image in the first display area and / or each first frame image in the second display area is also marked with an identification box of the annotated object.

[0097] In the first display area, the user can quickly check whether there are other objects marked in some image frames under the same marked object. After locating the image frame with the wrong marking ID, the user can select one or more frames to be modified. Specifically, it supports checking one or more frames to be modified in the multi-selection mode. The marked object with the marking ID to be modified in the frame to be modified is called the first marked object.

[0098] After selecting the frame to be modified, the user can quickly locate the target first frame where the first annotated object is located in the second display area to determine the correct target annotation ID. Alternatively, the user can choose to create a new annotation ID for the first annotated object in the frame to be modified.

[0099] After receiving the modification instruction or creation instruction for the frame to be modified input by the user (step 401), if the frame to be modified includes multiple frames, it is determined whether the multiple frames to be modified are different frames in the video set (step 402). If there are at least two frames to be modified that belong to the same frame, the annotation ID modification fails (step 412). If the multiple frames to be modified are different frames in the video set, it is continued to determine whether the first annotated objects in all the frames to be modified belong to the same object classification (step 403). If there are at least two frames to be modified The corresponding first annotated objects do not belong to the same object classification, the annotation ID modification fails (step 412). If the first annotated objects in all the frames to be modified belong to the same object classification, and the instruction input by the user is a creation instruction, a new annotation ID is created for the first annotated objects in all the frames to be modified (step 404), that is, the first annotated objects in all the frames to be modified correspond to the same created annotation ID. At the same time, the image frame after the annotation ID is created is moved to the continuous multi-frame image corresponding to the newly created annotation ID for display (step 411).

[0100] If the first annotated objects in all frames to be modified belong to the same object classification, and the instruction input by the user is a modification instruction, the target annotation ID is determined from the target first frame where the first annotated object is located (step 405). Then, it is determined whether the second annotated object indicated by the target annotation ID in the target first frame belongs to the same object classification as the first annotated object (step 406). If they do not belong to the same object classification, the annotation ID modification fails (step 412). If they belong to the same object classification, it is further determined whether the target annotation ID is already carried in the selected frame to be modified (step 407). If not, the annotation ID of the first annotated object in the frame to be modified is directly modified to the target annotation ID (step 408), and the image frame after the annotation ID is modified is moved to the continuous multi-frame image corresponding to the target annotation ID for display (step 411). If it is carried, it is further determined whether the component attribute of the third annotated object indicated by the target annotation ID in the frame to be modified is a single component attribute or a multi-component attribute (step 409).

[0101] If the third annotated object is a multi-component attribute, the annotation ID of the first annotated object in the frame to be modified is directly modified to the target annotation ID (step 408), and the image frame with the modified annotation ID is moved to the continuous multi-frame image corresponding to the target annotation ID for display (step 411). If the third annotated object is a single-component attribute, the annotation ID of the first annotated object in the frame to be modified is modified to the target annotation ID, and a new annotation ID is created for the third annotated object in the frame to be modified (step 410). Then the image frame with the modified annotation ID is moved to the corresponding continuous multi-frame image for display (step 411).

[0102] The execution order of step 402 and step 403 can be interchanged. If the frame to be modified includes only one frame, step 402 and step 403 can be skipped, and step 404 or step 405 and subsequent steps can be directly executed based on the creation instruction or modification instruction input by the user.

[0103] It can be seen that the object annotation information modification method provided by the present application can effectively assist users to quickly locate the correct annotation ID of each annotated object by adding a second display area to the user interface to display the first frame image of each annotated object, saving the operation of cutting frames back and forth to find the correct annotation ID, especially in scenes with densely annotated objects, which further improves convenience.

[0104] It also supports multi-frame selection mode, so that the selected frames to be modified can be merged into the target annotation ID with one click, or a new annotation ID can be created with one click, achieving the effect of quickly modifying the annotation ID during the annotation or quality inspection process.

[0105] Based on the object annotation information modification method described in any of the above embodiments, the present application also provides a computer program product, which includes one or more computer programs or instructions. The computer program or instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. When the computer program is executed by a processor, the object annotation information modification method described in any of the above embodiments is implemented.

[0106] Based on the object annotation information modification method described in any of the above embodiments, the present application also provides: Figure 5 A schematic diagram of the structure of an electronic device is shown in FIG. Figure 5 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement a method for modifying object annotation information described in any of the above embodiments.

[0107] The present application also provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to execute a method for modifying object annotation information described in any of the above embodiments.

[0108] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0110] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0111] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0112] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0113] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for modifying object annotation information, characterized in that: The method comprises: Displaying multiple consecutive frames of images where each annotated object is located in a first display area of ​​the user interface, and displaying a first frame of image where each annotated object is located in a second display area of ​​the user interface; wherein the image content of each frame of the image includes one or more annotated objects, and each frame of the image carries annotation information of the included annotated objects; In response to a user inputted annotation information modification instruction for a frame to be modified, the annotation information of a first annotated object in the frame to be modified is modified to target annotation information; wherein, the first annotated object is an annotated object of the annotation information to be modified in the frame to be modified; the target annotation information is annotation information of the first annotated object in a target first frame, and the target first frame is a first frame image where the first annotated object is located.

2. The method according to claim 1, characterized in that The method further comprises: The image frame after the modification of the annotation information is moved to the continuous multiple-frame image corresponding to the target annotation information for display.

3. The method according to claim 1, characterized in that The frame to be modified includes multiple frames; before modifying the annotation information of the first annotated object in the frame to be modified to the target annotation information, the method further includes: Determining that the first annotated objects in the plurality of frames to be modified belong to the same object category; and / or It is determined that the multiple frames to be modified are different frames in the video set.

4. The method according to claim 1, characterized in that: Before modifying the annotation information of the first annotated object in the frame to be modified into target annotation information, the method further includes: If the modification instruction indicates the target first frame, determining that a second annotated object indicated by the target annotation information in the target first frame belongs to the same object category as the first annotated object; If the modification instruction does not indicate the target first frame, the target first frame where the first labeled object is located is matched from the second display area.

5. The method according to claim 1, characterized in that The step of modifying the annotation information of the first annotated object in the frame to be modified into target annotation information includes: If the frame to be modified already carries the target annotation information, and the target annotation information indicates a third annotated object in the frame to be modified, obtaining a component attribute of the third annotated object, where the component attribute is used to indicate whether the third annotated object has multiple components; Based on the component attribute, at least the annotation information of the first annotation object is modified to the target annotation information.

6. The method according to claim 5, characterized in that The modifying, based on the component attribute, at least the annotation information of the first annotation object to the target annotation information includes: If the component attribute is a multi-component attribute, modifying the annotation information of the first annotated object to the target annotation information; If the component attribute is a single-component attribute, the annotation information of the first annotated object is modified to the target annotation information, and new annotation information is created for the third annotated object in the frame to be modified.

7. The method according to claim 1, characterized in that The method further comprises: In response to a new annotation information creation instruction input by the user for the frame to be modified, new annotation information is created for the first annotated object in the frame to be modified.

8. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, when the processor calls the executable instructions, the operation of any method described in claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed by a processor, the steps of any method described in claims 1-7 are implemented.

Images