Label correction method and device for image semantic segmentation region
By creating area correction auxiliary graphics and generating coordinate binding relationships, the problem of low pre-labeling accuracy of image semantic segmentation areas is solved, and efficient label correction is achieved, saving time and cost.
Patent Information
- Application Number
- CN202411202715.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, the pre-labeling accuracy of the image semantic segmentation area is low, resulting in low efficiency of manual label correction, time-consuming and costly.
By creating a regional correction auxiliary graph, using user operation instructions to move and rotate the graph, make its edges come into contact with the boundary of the target semantic segmentation area, generate coordinate binding relationships, and update the coordinate data of the target semantic segmentation area, thereby realizing the simultaneousness of multi-boundary points Drag and mark corrections.
It greatly improves the labeling and correction efficiency of image semantic segmentation areas, saves time and cost, and avoids inefficient and time-consuming operational boundary points one by one.
Smart Images

Figure CN119091144A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of semantic annotation, and in particular to a method and device for correcting annotation of an image semantic segmentation region. Background Art
[0002] Semantic segmentation refers to the classification of each pixel in an image, so as to achieve semantic understanding of the entire image. Traditional semantic segmentation annotation tools allow users to draw points and connect them into closed areas to complete the annotation. Among them, the annotated points are called "boundary points", and the line connecting the annotated points is the boundary of the annotated area. In order to save time and cost of manual annotation and improve efficiency, a deep network model is usually used to pre-annotate the original image to obtain a pre-annotated semantic segmentation area. However, the pre-annotated semantic segmentation area is not accurate and has low precision. In related technologies, the annotation correction of the pre-annotated semantic segmentation area is usually completed manually. However, as mentioned above, manual annotation consumes a lot of time and cost. Therefore, how to improve the annotation correction efficiency of the semantic segmentation area of the image has become an urgent problem to be solved. Summary of the invention
[0003] In order to overcome the problems existing in the related art, this specification provides a method and device for labeling and correcting image semantic segmentation areas.
[0004] According to a first aspect of an embodiment of this specification, a method for correcting annotations of an image semantic segmentation region is provided, the method comprising:
[0005] Determine the coordinate data of the target semantic segmentation area;
[0006] Creating a region correction auxiliary graph based on the target semantic segmentation region, wherein the edge of the region correction auxiliary graph includes at least two edge points;
[0007] Based on the user operation instruction, controlling the area correction auxiliary graphic to move and / or rotate;
[0008] In response to a movement and / or rotation instruction of the region correction auxiliary graphic, if an edge point on an edge of the region correction auxiliary graphic contacts a boundary point on a boundary of the target semantic segmentation region and there are at least two intersection points, generating a first coordinate binding relationship based on the at least two intersection points;
[0009] Based on the first coordinate binding relationship, the coordinate data of the target semantic segmentation area is updated.
[0010] According to a second aspect of an embodiment of this specification, a device for correcting annotations in an image semantic segmentation region is provided, the device comprising:
[0011] A data confirmation unit, used to determine the coordinate data of the target semantic segmentation area;
[0012] A graphics creation unit, configured to create a region correction auxiliary graphics based on the target semantic segmentation region, wherein the edge of the region correction auxiliary graphics includes at least two edge points;
[0013] A graphic display unit, used to control the area correction auxiliary graphic to move and / or rotate based on a user operation instruction;
[0014] a processing unit, configured to, in response to a movement and / or rotation instruction of the region correction auxiliary graphic, generate a first coordinate binding relationship based on the at least two intersection points when an edge point on the edge of the region correction auxiliary graphic contacts a boundary point on the boundary of the target semantic segmentation region and there are at least two intersection points;
[0015] A coordinate updating unit is used to update the coordinate data of the target semantic segmentation area based on the first coordinate binding relationship.
[0016] According to a third aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the method described in the embodiments of the first aspect above.
[0017] According to a fourth aspect of the embodiments of this specification, there is provided an electronic device, including:
[0018] processor;
[0019] a memory for storing processor-executable instructions;
[0020] The processor implements the method described in the embodiment of the first aspect above by running the executable instructions.
[0021] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the embodiments of the first aspect above are implemented.
[0022] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:
[0023] In an embodiment of the present specification, after the coordinate data of the target semantic segmentation area is determined so that the target semantic segmentation area is displayed in the display page of the image to be processed, a region correction auxiliary graphic including at least two edge points is established based on the target semantic segmentation area, and based on the user operation instruction, the region correction auxiliary graphic is controlled to move and / or rotate. During the movement of the region correction auxiliary graphic, when there are two intersections between the edge of the region correction auxiliary graphic and the boundary of the target semantic segmentation area, a first coordinate binding relationship is generated based on the at least two intersections, and based on the first coordinate binding relationship, the coordinate data of the target semantic segmentation area is updated, so that at least two boundary points of the target semantic segmentation area are pushed at the same time by moving and / or rotating the region correction auxiliary graphic, thereby avoiding the inefficiency and time-consuming process of operating the boundary points of the target semantic segmentation area one by one to perform annotation correction, greatly improving the annotation correction efficiency of the image semantic segmentation area, and saving time and cost.
[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.
[0026] Figure 1 It is a comparative schematic diagram of semantic segmentation region annotation correction in the related technology.
[0027] Figure 2 This is a flowchart of a method for labeling and correcting an image semantic segmentation region according to an exemplary embodiment of the present specification.
[0028] Figure 3 It is a comparative schematic diagram of the movement of an area correction auxiliary graphic according to an exemplary embodiment of this specification.
[0029] Figure 4 It is a flowchart of coordinate binding according to an exemplary embodiment of this specification.
[0030] Figure 5 This is an example diagram of semantic segmentation region annotation correction shown in this specification according to an exemplary embodiment.
[0031] Figure 6 This is a block diagram of a device for labeling and correcting an image semantic segmentation region according to an exemplary embodiment of the present specification.
[0032] Figure 7It is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION
[0033] Semantic segmentation refers to the classification of each pixel in an image, thereby achieving semantic understanding of the entire image. Unlike traditional image classification tasks, semantic segmentation requires the labeling of all pixels in the image, so it plays an important basic role in tasks such as target detection. Traditional semantic segmentation annotation tools require users to draw points and connect them into closed areas to complete the annotation. The annotated points are called "boundary points", and the line connecting the annotated points is the boundary of the annotated area. In order to save manual annotation time and cost and improve efficiency, deep network models are usually used to pre-annotate the original images to obtain pre-annotated semantic segmentation areas. For example Figure 1 As shown, Figure 1 It is a comparative schematic diagram of semantic segmentation region annotation correction in the related art. It can be seen from Figure A that the original image contains three different semantic segmentation regions after pre-annotation, among which the annotation of the yellow area is not accurate, and some areas are not selected. It is necessary to modify it based on the pre-annotated semantic segmentation area through manual annotation, so that the annotation of the yellow area is corrected to the effect shown in Figure B.
[0034] In the related art, annotation correction is usually completed by manually dragging the boundary points that need to be modified to the correct position. However, due to the fine annotation of deep network models, there are likely to be many boundary points in semantic segmentation. Even if a small area is modified, several boundary points need to be dragged, so the annotation efficiency is low.
[0035] In other solutions, a new segmentation region with the same semantics as the pre-annotated semantic segmentation region that needs to be corrected can be created, and the two can be merged into a new semantic segmentation region to achieve the correction of the annotation. However, the new semantic segmentation region also needs to be manually and meticulously annotated along the target boundary line, and the efficiency cannot be improved. In addition, the merging process requires calculation and takes a long time.
[0036] Therefore, the present specification provides a method for correcting annotations in image semantic segmentation regions, which can improve the efficiency of correcting annotations in image semantic segmentation regions and save time and cost.
[0037] Next, the embodiments of this specification are described in detail.
[0038] like Figure 2 As shown, Figure 2 This is a flowchart of a method for labeling and correcting an image semantic segmentation region according to an exemplary embodiment of the present specification, which specifically includes the following steps:
[0039] S201, determining coordinate data of a target semantic segmentation area.
[0040] In an exemplary embodiment, the pre-annotated file contains coordinate data of a pre-annotated semantic segmentation area obtained by pre-annotating the image to be processed; the method also includes: obtaining a pre-annotated file corresponding to the image to be processed, the pre-annotated file containing coordinate data of multiple pre-annotated semantic segmentation areas obtained by pre-annotating the image to be processed; and according to the received area selection signal, the selected pre-annotated semantic segmentation area is used as the target semantic segmentation area.
[0041] After the deep network model pre-annotates the image to be processed, the model outputs a pre-annotated file. The pre-annotated file includes the coordinate data of multiple pre-annotated semantic segmentation areas obtained after the model pre-annotation. After the pre-annotated file corresponding to the image to be processed is imported into the annotation platform, the semantic segmentation area is displayed in the display page of the image to be processed on the front end of the platform.
[0042] After the deep network model completes the image semantic segmentation task, it usually obtains multiple segmentation regions with different semantics. Each segmentation region represents a specific category or object in the image, such as pedestrians, vehicles, buildings, sky, etc. Therefore, the output pre-annotated file contains the coordinate data of multiple pre-annotated segmentation regions with different semantics obtained after pre-annotating the image to be processed. The display page of the image to be processed on the front end of the annotation platform usually distinguishes and displays different pre-annotated semantic segmentation regions. For example Figure 1 As shown in Figure A in , it can be seen that there are three pre-annotated segmented areas with different semantics in the current pre-annotated image, namely the yellow area, the pink area and the purple area.
[0043] For the semantic segmentation area pre-annotated by the current model, the user determines whether it is necessary to correct the annotation. If not, the annotation correction process ends; if so, the user selects one of the pre-annotated semantic segmentation areas as the area to be corrected at the front end by clicking the mouse. After the annotation platform backend receives the area selection instruction, the selected pre-annotated semantic segmentation area is used as the target semantic segmentation area, and the coordinate data of the target semantic segmentation area is queried. In the subsequent annotation correction process, the area correction auxiliary graphics only perform annotation correction on the selected target semantic segmentation area, without affecting the coordinate data of other pre-annotated semantic segmentation areas.
[0044] By selecting an area as the target semantic segmentation area for processing, it is helpful to improve the accuracy, efficiency and user interaction experience of subsequent annotation correction of the semantic segmentation area, especially when processing images containing multiple objects and complex backgrounds, the effect is more obvious.
[0045] It should be noted that, in some other embodiments, the image to be processed may only have one pre-annotated semantic segmentation region after being pre-annotated, and this specification does not limit this.
[0046] S202: Create a region correction auxiliary graph based on the target semantic segmentation region, wherein the edge of the region correction auxiliary graph includes at least two edge points.
[0047] The user uses the area correction auxiliary tool on the front end of the annotation platform to create an area correction auxiliary graphic in the display page, wherein the edge of the area correction auxiliary graphic includes at least two edge points, and the back end of the annotation platform responds to the creation instruction of the area correction auxiliary graphic to create a corresponding area correction auxiliary graphic.
[0048] In an exemplary embodiment, the method further includes: receiving a modification instruction of the area correction auxiliary graphic, the modification instruction being used to instruct modification of at least one of the following attributes of the area correction auxiliary graphic: initial position, shape, size and tilt angle; and modifying the created area correction auxiliary graphic according to the modification instruction.
[0049] The modification instruction is used to indicate at least one of the following attributes of the area correction auxiliary graphic: initial position, shape, size and tilt angle. Those skilled in the art can define the attribute category according to their own needs, and this specification does not limit this. Users can customize the starting position, specific shape, size and tilt angle of the area correction auxiliary graphic through shortcuts. After receiving the creation instruction, the annotation platform backend stores the coordinate data of the area correction auxiliary graphic.
[0050] S203: Based on the user operation instruction, control the area correction auxiliary graphic to move and / or rotate.
[0051] When the region correction auxiliary graphic is created, the user controls the movement and / or rotation of the region correction auxiliary graphic on the front-end display page to push the boundary of the target semantic segmentation region. The back-end controls the movement of the region correction auxiliary graphic according to the received movement instruction and / or rotation instruction.
[0052] For example, Figure 3 As shown, Figure 3 It is a comparative schematic diagram of the movement of an area correction auxiliary graphic according to an exemplary embodiment of this specification. Figure 3 Two examples of creating and moving area correction auxiliary graphics are included in the . Figure 3 The two regional correction auxiliary graphics shown are only for comparison and reference. In actual situations, when marking corrections, only one regional correction auxiliary graphic moves on the display page.
[0053] like Figure 3 As shown in the figure, for different pre-annotation defects, corrections can be made by defining regional correction auxiliary graphics of different shapes. For example, regional correction auxiliary graphic 1 can be defined as a rectangle to push the long boundary of the target semantic segmentation area and quickly fill the gap; while regional correction auxiliary graphic 2 can be defined as a downward triangle to finely modify the defects of the boundary of the target semantic segmentation area in a small range. By defining different regional correction auxiliary graphic attributes, it is adapted to the correction of polygonal annotations obtained by semantic segmentation, flexibly responding to different semantic segmentation areas, and improving the flexibility and accuracy of annotation correction.
[0054] By using region correction auxiliary graphics to push the target semantic segmentation region boundary, multiple boundary points can be dragged simultaneously, improving the efficiency of annotation correction.
[0055] It should be noted that, during the movement of the area correction auxiliary graphic, the user can also adjust the size, shape and other attributes of the currently moving area correction auxiliary graphic through modification instructions, thereby further improving the flexibility of annotation correction.
[0056] S204, in response to the movement and / or rotation instructions of the area correction auxiliary graphic, if the edge points on the edge of the area correction auxiliary graphic are in contact with the boundary points on the boundary of the target semantic segmentation area and there are at least two intersection points, a first coordinate binding relationship is generated based on the at least two intersection points.
[0057] Since the region correction auxiliary graphic is a tool for pushing the boundary of the target semantic segmentation region, the edge of the region correction auxiliary graphic will contact the boundary of the target semantic segmentation region during the movement to form an intersection. In response to the movement and / or rotation instructions of the region correction auxiliary graphic, after the edge of the region correction auxiliary graphic contacts the boundary of the target semantic segmentation region, the back end determines the edge point corresponding to the contact position between the two on the edge and the boundary point corresponding to the boundary. If the edge point on the edge of the region correction auxiliary graphic contacts the boundary point on the boundary of the target semantic segmentation region and there are at least two intersections, the coordinates of the boundary point and the edge point corresponding to the at least two intersections are bound to a first coordinate binding relationship.
[0058] Specifically, when the edge of the area correction auxiliary graphic contacts the boundary of the target semantic segmentation area to form an intersection, the coordinates of the intersection formed by the two will be separately bound. When there are two or more intersections, it is necessary to generate multiple single-point binding relationships at the same time so that multiple boundary points can move simultaneously following the corresponding bound edge points. The embodiment of this specification defines the single-point coordinate binding of at least two intersections as the first coordinate binding relationship.
[0059] In an exemplary embodiment, the method further includes: if an edge point on the edge of the region correction auxiliary graphic contacts a boundary point on the boundary of the target semantic segmentation region and there is an intersection, generating a second coordinate binding relationship based on the intersection.
[0060] In some other embodiments, when there is only one intersection, it is only necessary to generate a binding relationship for this one intersection. In the embodiment of this specification, the single-point coordinate binding of the single intersection is defined as the second coordinate binding relationship.
[0061] After the second coordinate binding relationship is generated, if the number of intersections of edge points on the edge of the area correction auxiliary graphic and boundary points on the boundary of the target semantic segmentation area increases to at least two intersections, a first coordinate binding relationship is generated based on the at least two intersections.
[0062] In some other embodiments, if there is only one intersection point when the edge of the region correction auxiliary graphic initially contacts the boundary of the target semantic segmentation region, a second coordinate binding relationship is generated, and after the region correction auxiliary graphic is moved or rotated, the number of intersection points increases to two or more, then the original second coordinate binding relationship can be changed to a first coordinate binding relationship.
[0063] As described above, the first coordinate binding relationship records the single-point coordinate binding relationship between each edge point on the at least two intersections and the corresponding boundary points, and the second coordinate binding relationship records the single-point coordinate binding relationship between the edge point on the only intersection and the corresponding boundary point. When the annotation platform backend receives a move instruction, it starts to determine whether the edge of the area correction auxiliary graphic is in contact with the boundary of the target semantic segmentation area. If not, it continues to determine. If yes, single-point coordinate binding is performed.
[0064] like Figure 4 As shown, Figure 4 This is a flowchart of a single point coordinate binding according to an exemplary embodiment of this specification. The specific steps are as follows:
[0065] S2401: The edge of the region correction auxiliary graphic contacts the boundary of the target semantic segmentation region.
[0066] S2402, determine whether the coordinates of the contact position match the coordinate data of the target semantic segmentation area or the area correction auxiliary tool. If the coordinates of the contact position only match the coordinate data of the target semantic segmentation area, proceed to step S2403; if the coordinates of the contact position only match the coordinate data of the area correction auxiliary graphic, proceed to step S2404; if the coordinates of the contact position match the coordinate data of both the target semantic segmentation area and the area correction auxiliary tool, proceed to step S2405.
[0067] In an exemplary embodiment, the coordinate data predefined for the target semantic segmentation region includes the coordinates of each boundary endpoint of the target semantic segmentation region; the coordinate data predefined for the region correction auxiliary graphic includes the coordinates of the vertices of the region correction auxiliary graphic.
[0068] The coordinates of the contact position are matched with the coordinate data predefined for the target semantic segmentation area and the coordinate data predefined for the area correction auxiliary graphic respectively, and it is determined on which end point of the target semantic segmentation area and the area correction auxiliary graphic the contact point is located based on the matching result.
[0069] S2403, adding the coordinates of the contact position to the coordinate data of the area correction auxiliary graphic.
[0070] If the coordinates of the contact position match the coordinates of any boundary endpoint of the target semantic segmentation area, then the any boundary endpoint is determined as the boundary point corresponding to the contact position on the corresponding boundary. At this time, the coordinates of the contact position are located on the boundary endpoint of the target semantic segmentation area, and the boundary endpoint is in contact with the edge of the area correction auxiliary graphic.
[0071] That is, when contact occurs between the first point on the edge of the area correction auxiliary graphic and the boundary point on the boundary of the target semantic segmentation area, if the coordinates of the first point have been recorded in the coordinate data of the area correction auxiliary graphic, the first point is used as the edge point to generate the single-point coordinate binding relationship; otherwise, the coordinates of the first point are added to the coordinate data of the area correction auxiliary graphic, and the first point is used as the edge point to generate the single-point coordinate binding relationship.
[0072] A coordinate point corresponding to the first point is added to the coordinate data predefined for the area correction auxiliary graphic, and is determined as an edge point corresponding to the corresponding edge.
[0073] S2404, adding the coordinates of the contact position to the coordinate data of the target semantic segmentation area.
[0074] If the coordinates of the contact position match the coordinates of any vertex of the area correction auxiliary graphic, then the any vertex is determined as the edge point corresponding to the contact position on the corresponding edge. At this time, the coordinates of the contact position are located on the vertex of the area correction auxiliary graphic, and the vertex is in contact with the boundary of the target semantic segmentation area.
[0075] That is, when contact occurs between an edge point on the edge of the area correction auxiliary graphic and a second point on the boundary of the target semantic segmentation area, if the coordinates of the second point have been recorded in the coordinate data of the target semantic segmentation area, the second point is used as the boundary point to generate the single-point coordinate binding relationship; otherwise, the coordinates of the second point are added to the coordinate data of the target semantic segmentation area, and the second point is used as the boundary point to generate the single-point coordinate binding relationship.
[0076] A coordinate point corresponding to the second point is added to the coordinate data predefined for the target semantic segmentation area, and is determined as a corresponding boundary point on the corresponding boundary.
[0077] S2405, binding the coordinates of the boundary point and the edge point corresponding to the same contact position.
[0078] The edge point corresponding to the contact position between the two on the edge and the boundary point corresponding to the boundary are determined, and the coordinates of the boundary point and the edge point corresponding to the same contact position are bound so that the bound boundary point and edge point can move simultaneously.
[0079] After the contact is confirmed and the coordinates are bound, the region correction auxiliary graphic is moved one pixel based on the movement instruction, and the contact point bound to the target semantic segmentation area moves accordingly. It is re-determined whether the edge of the moved region correction auxiliary graphic is in contact with the boundary of the target semantic segmentation area, and the next cycle is entered until a stop movement instruction is received.
[0080] By binding multiple groups of boundary points of the target semantic segmentation area and edge points of the region correction auxiliary graphics during the region correction auxiliary graphics binding and promotion process, simultaneous operation of multiple boundary points is achieved, thereby improving the efficiency of target semantic segmentation area annotation and correction.
[0081] S205: Update the coordinate data of the target semantic segmentation area based on the first coordinate binding relationship.
[0082] If a first coordinate binding relationship is generated between the edge point and the boundary point, the multiple bound boundary points on the target semantic segmentation area will move simultaneously with the edge points bound to the boundary points in the area correction auxiliary graphic, causing the coordinates of the boundary points to change. The back end updates the coordinate data of the target semantic segmentation area, and updates the display position of the moved boundary points in the front-end display page based on the corresponding coordinate change information.
[0083] In another embodiment, if a second coordinate binding relationship is generated between the edge point and the boundary point, the single bound boundary point on the target semantic segmentation area will move simultaneously with the edge point bound to the boundary point in the area correction auxiliary graphic, so that the coordinates of the boundary point change. The back end updates the coordinate data of the target semantic segmentation area, and updates the display position of the moved boundary point in the front-end display page based on the corresponding coordinate change information.
[0084] In an exemplary embodiment, after receiving an operation termination instruction of the area correction auxiliary graphic, the positional relationship between the coordinates in the coordinate data of the target semantic segmentation area is determined; if at least three coordinates are located on a line segment, the coordinates that are not at the endpoints on the line segment are filtered out.
[0085] When the backend receives the operation termination instruction of the region correction auxiliary graphic, the coordinates of each boundary point of the target semantic segmentation region are determined based on the coordinate data of the target semantic segmentation region. Specifically, the coordinates of each boundary point of the target semantic segmentation region can be grouped as at least three boundary points to determine whether they are located on a line segment. If so, the coordinates on the line segment that are not at the two end points of the line segment are filtered out.
[0086] By filtering out non-endpoint boundary points on the boundary of the target semantic segmentation region, the coordinate data of the target semantic segmentation region finally obtained can be simplified.
[0087] In an exemplary embodiment, the image to be processed to which the target semantic segmentation area belongs contains multiple pre-annotated semantic segmentation areas, and the method also includes: when there is a common boundary between the target semantic segmentation area and another pre-annotated semantic segmentation area in the image to be processed, if the coordinates of any boundary point on the common boundary change, the coordinate data of the other pre-annotated semantic segmentation area is also updated based on the corresponding coordinate change information.
[0088] Specifically, after the model completes pre-annotation, the output semantic segmentation region may contain two different semantic segmentation regions with common edges, for example Figure 1 There is a common edge between the yellow area and the purple area. In the case where there is a common boundary between the target semantic segmentation area and another pre-labeled semantic segmentation area, if the coordinates of any boundary point on the common boundary change, the coordinate data of the boundary point with the coordinate change in the other pre-labeled semantic segmentation area of the common edge are also updated based on the corresponding coordinate change information.
[0089] The above scheme makes it possible to eliminate the need to repeatedly adjust the coordinate data of the same boundary points that have changed in another pre-annotated semantic segmentation region on the same edge after the annotation correction of the target semantic segmentation region is completed, thereby further improving the efficiency of annotation correction and saving time.
[0090] Next, the solution of this application is described according to specific embodiments:
[0091] like Figure 5 As shown, Figure 5 This is an example diagram of semantic segmentation region annotation correction shown in this specification according to an exemplary embodiment. Among them, the light blue area is the target semantic segmentation area to be corrected, and the red area is the created region correction auxiliary graphic. Assuming that the current user creates a region correction auxiliary graphic, the initial position of the region correction auxiliary graphic is as shown in Figure A, and the upper boundary of the target semantic segmentation area needs to be adjusted. Therefore, the subsequent movement instruction received is to move directly upward, and the annotation correction process is obtained as follows:
[0092] exist Figure 5 In Figure A, when an upward movement instruction is received, the backend calculates the coordinates of the contact position between the current region correction auxiliary graph and the target semantic segmentation region. At this time, the coordinates of the contact position are matched with the coordinate data of the region correction auxiliary graph and the target semantic segmentation region respectively:
[0093] Contact position coordinates: (4,2), (3,2), (2,2)
[0094] Area auxiliary correction graphic coordinates: (1,2), (1,1), (5,1), (5,2)
[0095] Target semantic segmentation region coordinates: (0,4), (0,0), (6,0), (6,4), (4,2), (3,3), (3,2), (2,4), (2,2)
[0096] It can be seen that the coordinates of the current contact position only match the coordinate data of the target semantic segmentation area. Therefore, the coordinates of the contact position are added to the coordinate data of the regional auxiliary correction graphic, and the edge points and boundary points in the regional auxiliary correction graphic and the target semantic segmentation area corresponding to the coordinates of the contact position are bound to the first coordinates. Based on the movement instruction, the regional auxiliary correction graphic is moved up one pixel to obtain the display page shown in Figure B.
[0097] exist Figure 5 In Figure B, it can be seen that due to the binding of the corresponding boundary points, the boundary points of the three contact positions in the target semantic segmentation area follow the region correction auxiliary graph and move up one pixel. The coordinates of the contact positions between the current region correction auxiliary graph and the target semantic segmentation area are recalculated, and the coordinates of the contact positions are matched with the coordinate data of the region correction auxiliary graph and the target semantic segmentation area, respectively, to obtain:
[0098] Contact position coordinates: (4,3), (3,3), (2,3)
[0099] Area auxiliary correction graphic coordinates: (1,3), (1,2), (5,2), (5,3)
[0100] Target semantic segmentation area coordinates: (0,4), (0,0), (6,0), (6,4), (4,3), (3,3), (2,4), (2,3)
[0101] It can be seen that the coordinates of the current contact position only match the coordinate data of the target semantic segmentation area, so the coordinates of the contact position are added to the coordinate data of the regional auxiliary correction figure. In the process of moving the regional auxiliary correction figure upward, since the original boundary point (3,2) before the movement coincides with another boundary point (3,3) after moving up one pixel, the duplicate boundary point is deleted.
[0102] The first coordinates of the edge points and boundary points corresponding to the coordinates of the contact position in the region auxiliary correction graphic and the target semantic segmentation area are bound, and the region auxiliary correction graphic is moved up by one pixel based on the movement instruction to obtain the display page shown in Figure C.
[0103] exist Figure 5 In Figure C, it can be seen that due to the binding of the corresponding boundary points, the boundary points of the three contact positions in the target semantic segmentation area follow the region correction auxiliary figure and move up by one pixel. The upper boundary becomes a straight boundary. The coordinates of the contact position between the current region correction auxiliary figure and the target semantic segmentation area are recalculated, and the coordinates of the contact position are matched with the coordinate data of the region correction auxiliary figure and the target semantic segmentation area, respectively, to obtain:
[0104] Contact position coordinates: (1,4), (2,4), (3,4), (4,4), (5,4)
[0105] Area auxiliary correction graphic coordinates: (1,4), (1,3), (5,3), (5,4)
[0106] Target semantic segmentation area coordinates: (0,4), (0,0), (6,0), (6,4), (4,4), (3,4), (2,4)
[0107] It can be seen that among the coordinates of the current contact position, (2,4), (3,4), and (4,4) only match the coordinate data of the target semantic segmentation area, so the coordinates of the contact position are added to the coordinate data of the regional auxiliary correction figure. (1,4) and (5,4) only match the coordinate data of the regional auxiliary correction figure, so the coordinates of the contact position are added to the coordinate data of the target semantic segmentation area. In the process of moving the regional auxiliary correction figure upward, since the original boundary point (2,3) before the movement coincides with another boundary point (2,4) after moving up one pixel, the duplicate boundary point is deleted.
[0108] If the user ends the moving area correction auxiliary graphics at this time, the final coordinate data of the current target semantic segmentation area is: (0,4), (0,0), (6,0), (6,4), (4,4), (3,4), (2,4), (1,4), (5,4). After filtering out the coordinates that are not at the endpoints of the line segment, the final coordinate data of the final target semantic segmentation area is:
[0109] (0,4),(0,0),(6,0),(6,4).
[0110] In an embodiment of the present specification, after the coordinate data of the target semantic segmentation area is determined so that the target semantic segmentation area is displayed in the display page of the image to be processed, a region correction auxiliary graphic including at least two edge points is established based on the target semantic segmentation area, and based on the user operation instruction, the region correction auxiliary graphic is controlled to move and / or rotate. During the movement of the region correction auxiliary graphic, when there are two intersections between the edge of the region correction auxiliary graphic and the boundary of the target semantic segmentation area, a first coordinate binding relationship is generated based on the at least two intersections, and based on the first coordinate binding relationship, the coordinate data of the target semantic segmentation area is updated, so that at least two boundary points of the target semantic segmentation area are pushed at the same time by moving and / or rotating the region correction auxiliary graphic, thereby avoiding the inefficiency and time-consuming process of operating the boundary points of the target semantic segmentation area one by one to perform annotation correction, greatly improving the annotation correction efficiency of the image semantic segmentation area, and saving time and cost.
[0111] Corresponding to the aforementioned method embodiments, this specification also provides device embodiments.
[0112] like Figure 6 As shown, Figure 6 This is a block diagram of a device for labeling and correcting an image semantic segmentation region according to an exemplary embodiment of the present specification, the device comprising:
[0113] A data confirmation unit 601 is used to determine the coordinate data of the target semantic segmentation area;
[0114] A graphics creation unit 602 is used to create a region correction auxiliary graphic based on the target semantic segmentation region, wherein the edge of the region correction auxiliary graphic includes at least two edge points;
[0115] A graphic display unit 603, used to control the area correction auxiliary graphic to move and / or rotate based on a user operation instruction;
[0116] The processing unit 604 is configured to generate a first coordinate binding relationship based on the at least two intersection points in response to the movement and / or rotation instruction of the region correction auxiliary graphic, when an edge point on the edge of the region correction auxiliary graphic contacts a boundary point on the boundary of the target semantic segmentation region and there are at least two intersection points;
[0117] The coordinate updating unit 605 is used to update the coordinate data of the target semantic segmentation area based on the first coordinate binding relationship.
[0118] In an exemplary embodiment, the apparatus further includes: an intersection confirmation unit 606, which generates a second coordinate binding relationship based on the intersection if an edge point on the edge of the region correction auxiliary graphic contacts a boundary point on the boundary of the target semantic segmentation region and there is an intersection, and updates the coordinate data of the target semantic segmentation region based on the second coordinate binding relationship;
[0119] After the second coordinate binding relationship is generated, if the number of intersections of edge points on the edge of the area correction auxiliary graphic and boundary points on the boundary of the target semantic segmentation area increases to at least two intersections, a first coordinate binding relationship is generated based on the at least two intersections.
[0120] In an exemplary embodiment, the apparatus further includes: a region selection unit 607, configured to obtain a pre-annotated file corresponding to the image to be processed, wherein the pre-annotated file contains coordinate data of a plurality of pre-annotated semantic segmentation regions obtained by pre-annotating the image to be processed;
[0121] According to the received region selection signal, the selected pre-labeled semantic segmentation region is used as the target semantic segmentation region.
[0122] In an exemplary embodiment, the image to be processed to which the target semantic segmentation area belongs contains multiple pre-annotated semantic segmentation areas; the device also includes: a common edge adjustment unit 608, which is specifically used to update the coordinate data of the other pre-annotated semantic segmentation area based on the corresponding coordinate change information when there is a common boundary between the target semantic segmentation area and another pre-annotated semantic segmentation area in the image to be processed, if any boundary point on the common boundary changes its coordinates.
[0123] In an exemplary embodiment, the first coordinate binding relationship records the single-point coordinate binding relationship between each edge point on the at least two intersections and the corresponding boundary points, and the second coordinate binding relationship records the single-point coordinate binding relationship between the edge point on the only intersection and the corresponding boundary point. The device further includes: a coordinate binding unit 609, which is specifically used to, when a contact occurs between a first point on the edge and a boundary point on the boundary, if the coordinates of the first point have been recorded in the coordinate data of the area correction auxiliary graphic, use the first point as the edge point to generate the single-point coordinate binding relationship; otherwise, add the coordinates of the first point to the coordinate data of the area correction auxiliary graphic, and use the first point as the edge point to generate the single-point coordinate binding relationship;
[0124] And / or, in the event of contact between an edge point on the edge and a second point on the boundary, if the coordinates of the second point have been recorded in the coordinate data of the target semantic segmentation area, the second point is used as the boundary point to generate the single-point coordinate binding relationship; otherwise, the coordinates of the second point are added to the coordinate data of the target semantic segmentation area, and the second point is used as the boundary point to generate the single-point coordinate binding relationship.
[0125] In an exemplary embodiment, the device also includes: a coordinate screening unit 610, which is specifically used to determine the positional relationship between each coordinate in the coordinate data of the target semantic segmentation area after receiving the operation termination instruction of the area correction auxiliary graphic; if at least three coordinates are located on a line segment, the coordinates on the line segment that are not at the endpoints are screened out.
[0126] In an exemplary embodiment, the device also includes: a graphics creation unit 611, which is specifically used to receive a modification instruction of the area correction auxiliary graphic, and the modification instruction is used to indicate the modification of at least one of the following attributes of the area correction auxiliary graphic: initial position, shape, size and tilt angle; according to the modification instruction, modify the created area correction auxiliary graphic.
[0127] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.
[0128] Based on the same concept as the aforementioned method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.
[0129] Figure 7 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of this specification. Figure 7 At the hardware level, the electronic device includes a processor 701, an internal bus 702, a network interface 703, a memory 704, and a non-volatile memory 705, and may also include hardware required for other services. The processor 701 reads the corresponding computer program from the non-volatile memory 705 into the memory 704 and then runs it. Of course, in addition to the software implementation, this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0130] The devices or modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0131] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, a method for labeling and correcting an image semantic segmentation region as shown in any of the aforementioned embodiments is implemented.
[0132] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0133] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0134] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
Claims
1. A method for correcting the annotation of image semantic segmentation regions, characterized in that: include: Determine the coordinate data of the target semantic segmentation area; Creating a region correction auxiliary graph based on the target semantic segmentation region, wherein the edge of the region correction auxiliary graph includes at least two edge points; Based on the user operation instruction, controlling the area correction auxiliary graphic to move and / or rotate; In response to a movement and / or rotation instruction of the region correction auxiliary graphic, if an edge point on an edge of the region correction auxiliary graphic contacts a boundary point on a boundary of the target semantic segmentation region and there are at least two intersection points, generating a first coordinate binding relationship based on the at least two intersection points; Based on the first coordinate binding relationship, the coordinate data of the target semantic segmentation area is updated.
2. The method according to claim 1, characterized in that: The method further comprises: If an edge point on the edge of the area correction auxiliary graphic contacts a boundary point on the boundary of the target semantic segmentation area and there is an intersection, a second coordinate binding relationship is generated based on the intersection, and based on the second coordinate binding relationship, the coordinate data of the target semantic segmentation area is updated; After the second coordinate binding relationship is generated, if the number of intersections of edge points on the edge of the area correction auxiliary graphic and boundary points on the boundary of the target semantic segmentation area increases to at least two intersections, a first coordinate binding relationship is generated based on the at least two intersections.
3. The method according to claim 1, characterized in that: The method further comprises: Acquire a pre-annotated file corresponding to the image to be processed, wherein the pre-annotated file contains coordinate data of multiple pre-annotated semantic segmentation areas obtained by pre-annotating the image to be processed; According to the received region selection signal, the selected pre-labeled semantic segmentation region is used as the target semantic segmentation region.
4. The method according to claim 1, characterized in that The image to be processed to which the target semantic segmentation area belongs contains a plurality of pre-labeled semantic segmentation areas; the method further comprises: In the case where there is a common boundary between the target semantic segmentation region and another pre-annotated semantic segmentation region in the image to be processed, if the coordinates of any boundary point on the common boundary change, the coordinate data of the other pre-annotated semantic segmentation region is also updated based on the corresponding coordinate change information.
5. The method according to claim 2, characterized in that: The first coordinate binding relationship records the single-point coordinate binding relationship between each edge point on the at least two intersection points and each corresponding boundary point, and the second coordinate binding relationship records the single-point coordinate binding relationship between the edge point on the only intersection point and the corresponding boundary point, and the method further includes: In the case where a first point on the edge contacts a boundary point on the boundary, if the coordinates of the first point have been recorded in the coordinate data of the target semantic segmentation area, the first point is used as the edge point to generate the single-point coordinate binding relationship; otherwise, the coordinates of the first point are added to the coordinate data of the target semantic segmentation area, and the first point is used as the edge point to generate the single-point coordinate binding relationship; and / or, In the event of contact between an edge point on the edge and a second point on the boundary, if the coordinates of the second point have been recorded in the coordinate data of the area correction auxiliary graphic, the second point is used as the boundary point to generate the single-point coordinate binding relationship; otherwise, the coordinates of the second point are added to the coordinate data of the area correction auxiliary graphic, and the second point is used as the boundary point to generate the single-point coordinate binding relationship.
6. The method according to claim 1, characterized in that The method further comprises: After receiving the operation termination instruction of the region correction auxiliary graphic, determining the positional relationship between each coordinate in the coordinate data of the target semantic segmentation region; If at least three coordinates are on a line segment, the coordinates that are not at the endpoints of the line segment are filtered out.
7. The method according to claim 1, characterized in that The method further comprises: receiving a modification instruction of the region correction auxiliary graphic, wherein the modification instruction is used to instruct to modify at least one of the following attributes of the region correction auxiliary graphic: initial position, shape, size and tilt angle; According to the modification instruction, the created area correction auxiliary graphic is modified.
8. An image semantic segmentation and annotation device, characterized in that: include: A data confirmation unit, used to determine the coordinate data of the target semantic segmentation area; A graphics creation unit, configured to create a region correction auxiliary graphics based on the target semantic segmentation region, wherein the edge of the region correction auxiliary graphics includes at least two edge points; A graphic display unit, used to control the area correction auxiliary graphic to move and / or rotate based on a user operation instruction; a processing unit, configured to, in response to a movement and / or rotation instruction of the region correction auxiliary graphic, generate a first coordinate binding relationship based on the at least two intersection points when an edge point on the edge of the region correction auxiliary graphic contacts a boundary point on the boundary of the target semantic segmentation region and there are at least two intersection points; A coordinate updating unit is used to update the coordinate data of the target semantic segmentation area based on the first coordinate binding relationship.
9. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.
11. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
