Image annotation method and device, electronic equipment and storage medium
By generating the target clustering center and superpixel, and using image processing tools for labeling, the problem of low labeling efficiency and accuracy of image semantic segmentation in the prior art is solved, and a fast and efficient labeling process is achieved.
Patent Information
- Application Number
- CN202411696314.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-25
AI Technical Summary
In the prior art, the image semantic segmentation labeling method requires manual point-by-point labeling, which has low efficiency and accuracy, making it difficult to quickly and effectively process massive labeling data.
By acquiring the initial image and preprocessing, multiple target clustering centers are generated, clustering to form superpixels, and marking the superpixels based on the image processing tool to generate a first labeling point, and finally filtering out the target area image.
It improves the efficiency and quality of image semantic segmentation labeling, reduces manual labeling costs, and realizes the rapid and efficient processing of massive labeling data.
Smart Images
Figure CN119693639A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence deep learning, and specifically, to an image annotation method, device, electronic device, and storage medium. Background Art
[0002] In the field of artificial intelligence deep learning, computer vision has made remarkable achievements, including semantic segmentation technology. Image semantic segmentation is to classify each pixel in the image. It is currently widely used in medical images (for example, for image segmentation of tumors to assist medical treatment), unmanned driving (can be used for obstacle avoidance and behavior prediction), remote sensing images (for example, in road / lake / house / vehicle scenes), etc.
[0003] In computer vision deep learning tasks, artificial intelligence algorithm engineers need a large amount of image target annotation data. Obtaining a large amount of annotated data sources is an important, tedious, and labor-intensive step. Traditional image semantic segmentation annotation methods mark coordinate points of the contour of the semantic segmentation target area of the annotated image, connect these points into a closed area, and store and export the coordinate information of all points in the target area. This annotation method requires manual annotation of the target area on its boundary line one by one, and the annotation efficiency and annotation accuracy are relatively low. Therefore, how to quickly and effectively perform image semantic segmentation annotation on the annotated image data and effectively improve the annotation efficiency and annotation quality has become a problem that needs to be solved in this field. Summary of the invention
[0004] The embodiments of the present application provide an image annotation method, device, electronic device, and storage medium, aiming to solve the problem of how to quickly and effectively perform image semantic segmentation annotation on image data to be annotated.
[0005] A first aspect of an embodiment of the present application provides an image annotation method, the method comprising:
[0006] Obtain an initial image, and preprocess the initial image to obtain an image to be labeled;
[0007] Based on an image processing tool, generating a plurality of target cluster centers on the image to be labeled;
[0008] Clustering the pixel points on the image to be labeled based on the multiple target cluster centers to generate multiple superpixels on the image to be labeled;
[0009] Based on the foreground annotation information, the image to be annotated containing the plurality of superpixels is marked based on the image processing tool to obtain a plurality of first annotation points;
[0010] Pixel points in the image to be annotated are screened based on the first annotation points to obtain a target area image.
[0011] In an optional implementation, the generating a plurality of target cluster centers on the image to be labeled based on an image processing tool includes:
[0012] Get the preset number of superpixels;
[0013] Based on the image processing tool, initially generating multiple cluster centers on the image to be labeled;
[0014] Clustering the pixel points on the image to be labeled based on the multiple cluster centers, and generating a plurality of initial superpixels having the preset number of superpixels on the image to be labeled;
[0015] Reselecting a pixel point with the smallest gradient for a plurality of pixel points in each of the initial superpixels to obtain an updated cluster center in each of the initial superpixels;
[0016] Re-clustering is performed based on the updated cluster center, and the above steps are repeated until the position of the updated cluster center in the target repetition round relative to the updated cluster center in the previous repetition round remains unchanged, and the multiple updated cluster centers of the target repetition round are used as the multiple target cluster centers.
[0017] In an optional implementation, the foreground annotation information is used as a reference, and based on the image processing tool, the image to be annotated containing the plurality of superpixels is marked to obtain a plurality of first annotation points, including:
[0018] Acquire the foreground annotation information, where the foreground annotation information is marking information drawn in the foreground area of the image to be annotated based on the image processing tool;
[0019] Taking the foreground annotation information as a reference, and based on the image processing tool, superpixels in the image to be annotated are screened;
[0020] Traversing all superpixels in the image to be annotated, and taking the superpixel containing the pixel point corresponding to the foreground annotation information as the target superpixel;
[0021] The plurality of target superpixels are processed based on the image processing tool to obtain the plurality of first annotation points, and the area enclosed by the plurality of first annotation points at least includes the plurality of target superpixels.
[0022] In an optional implementation, before filtering the pixel points in the image to be annotated based on the first annotation points to obtain the target area image, the method further includes:
[0023] In response to a user's mark point moving operation instruction, first position information is determined, and a position of a target mark point is moved based on the first position information, the target mark point being any one of the plurality of first mark points, and / or,
[0024] In response to a user's command to add a new annotation point, second position information is determined, and a new first annotation point is generated on the image to be annotated based on the second position information, and / or,
[0025] In response to a user's annotation point deletion command, third position information is determined, and the original first annotation point is deleted on the image to be annotated based on the third position information.
[0026] In an optional implementation, preprocessing the initial image to obtain the image to be labeled includes:
[0027] Obtaining size information of a visible area of the image processing tool;
[0028] Scaling the initial image based on the scale information so that the size of the initial image after the scaling is consistent with the size of the visible area;
[0029] Performing a first color space transformation on the initial image after the scaling process to obtain an image in an XYZ color space;
[0030] Perform a second color space transformation on the image in the XYZ color space to obtain an image in the CEILab color space as the image to be labeled.
[0031] In an optional implementation, before labeling the image to be labeled containing the plurality of superpixels based on the image processing tool based on the foreground labeling information to obtain a plurality of first labeling points, the method further includes:
[0032] Obtaining size information of each superpixel;
[0033] Based on the distance between the target cluster centers of adjacent superpixels, determining the neighboring superpixels of each superpixel, wherein the neighboring superpixels are superpixels corresponding to the target cluster center that is closest to the target cluster center of the superpixel;
[0034] Based on the preset superpixel size, the superpixels are classified based on the size information of each superpixel to obtain a first superpixel and a second superpixel, wherein the first superpixel is a superpixel whose size information is greater than the preset superpixel size, and the second superpixel is a superpixel whose size information is less than or equal to the preset superpixel size;
[0035] The second superpixel is merged with adjacent superpixels of the second superpixel.
[0036] In an optional implementation, after the pixel points in the image to be annotated are screened based on the first annotation points to obtain the target area image, the method further includes:
[0037] Constructing multiple sets of training data, wherein the training data at least includes the initial image, the target area image, and position information and semantic information of the multiple first annotation points;
[0038] The multiple sets of training data are input into an image semantic segmentation model, and the image semantic segmentation model is iteratively trained based on the multiple training data to obtain a target model, where the target model is used to pre-annotate the input image to generate a target area image containing annotated points.
[0039] A second aspect of an embodiment of the present application provides an image annotation device, the device comprising:
[0040] The image to be annotated module is used to obtain an initial image and preprocess the initial image to obtain an image to be annotated;
[0041] A cluster center module, used for generating a plurality of target cluster centers on the image to be labeled based on an image processing tool;
[0042] A superpixel module, used for clustering the pixel points on the image to be labeled based on the multiple target cluster centers, and generating multiple superpixels on the image to be labeled;
[0043] A labeling module, used for labeling the image to be labeled containing the plurality of superpixels based on the image processing tool based on the foreground labeling information to obtain a plurality of first labeling points;
[0044] The target area image module is used to filter the pixel points in the image to be annotated based on the first annotation points to obtain a target area image.
[0045] A third aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image annotation method described in any one of the first aspect of the embodiments of the present application.
[0046] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the image annotation method described in any one of the first aspects of the embodiments of the present application are implemented.
[0047] Beneficial effects:
[0048] This application clusters the image to be annotated based on image processing tools to construct multiple superpixels, and generates marking points based on the image processing tools to obtain the target area image. The efficiency and speed of generating marking points by the image processing tools are improved through superpixels, and image semantic segmentation and annotation of the image data to be annotated can be achieved quickly and effectively, effectively improving the annotation efficiency and annotation quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0050] Figure 1 is a flowchart of the steps of an image annotation method proposed in one embodiment of the present application;
[0051] Figure 2a-2d A set of example graphs of performing multiple clustering optimization of superpixels on an image to be annotated in an image annotation method proposed in an embodiment of the present application;
[0052] Figure 3a-3f It is another set of example diagrams of performing multiple clustering optimization of superpixels on an image to be annotated in an image annotation method proposed in an embodiment of the present application;
[0053] Figures 4a-4d This is an example diagram of generating annotation points based on an image to be annotated containing superpixels in an image annotation method proposed in an embodiment of the present application;
[0054] Figure 5a-5b is an example diagram of marking points in an image marking method proposed in an embodiment of the present application;
[0055] Figure 6 is another example diagram of annotated points in an image annotation method according to an embodiment of the present application;
[0056] Figure 7 It is a schematic diagram of the training process of an image semantic segmentation model in an image annotation method proposed in one embodiment of the present application;
[0057] Figure 8 is a schematic diagram of scaling an initial image in an image annotation method proposed in an embodiment of the present application;
[0058] Fig. 9 is a structural schematic diagram of an image annotation device proposed in one embodiment of the present application;
[0059] Fig.10 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION
[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0061] In the drawings, the size of the constituent elements, the thickness of the layer or the area may be exaggerated for the sake of clarity. Therefore, any implementation of the present disclosure is not necessarily limited to the size shown in the drawings, and the shapes and sizes of the components in the drawings do not reflect the true proportions. In addition, the drawings schematically show ideal examples, and any implementation of the present disclosure is not limited to the shapes or values shown in the drawings.
[0062] In the field of artificial intelligence deep learning, computer vision has made remarkable achievements, including semantic segmentation technology. Image semantic segmentation is to classify each pixel in the image. It is currently widely used in medical images (for example, for image segmentation of tumors to assist medical treatment), unmanned driving (can be used for obstacle avoidance and behavior prediction), remote sensing images (for example, in road / lake / house / vehicle scenes), etc. This field is currently divided into different types of semantic segmentation, such as supervised semantic segmentation, unsupervised semantic segmentation, and video semantic segmentation.
[0063] In the computer vision deep learning task, artificial intelligence algorithm engineers need a large amount of image target annotation data. Obtaining a large amount of annotated data sources is an important, tedious, and labor-intensive step. For the image semantic segmentation annotation task, the industry currently provides a variety of solutions. Traditional image semantic segmentation annotation methods include: installing software with a cutout function to perform cutout and layering processing on the semantic segmentation target area of the image to be annotated, and finally storing and exporting multiple pictures containing the target area. This annotation method not only requires the installation of various software, plug-ins, and system environment dependencies, but also has a high threshold for use. In addition, the large amount of data generated increases the transmission and storage pressure of the server; or by marking the coordinate points of the contour of the semantic segmentation target area of the image to be annotated, these points are connected into a closed area, and the coordinate information of all the points in the target area is stored and exported. This annotation method requires manual annotation of the target area on its boundary line one by one, and the annotation efficiency and annotation accuracy are relatively low. Therefore, how to quickly and effectively perform image semantic segmentation annotation on the annotated image data and effectively improve the annotation efficiency and annotation quality has become a problem that needs to be solved in this field.
[0064] In view of this, an embodiment of the present application provides an image annotation method. Figure 1 FIG. 1 shows a flowchart of a method for annotating an image according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0065] S101, obtaining an initial image, and preprocessing the initial image to obtain an image to be labeled.
[0066] When the step S101 is specifically implemented, the preprocessing includes resizing the initial image and converting the color space. First, the size information of the visible area of the image processing tool is obtained. The image processing tool is a tool that includes graphics drawing and processing functions. Based on the image processing tool, the input image can be drawn, segmented, etc. to generate annotation points on the input image to achieve annotation of the initial image. Optionally, the image processing tool is a Canvas tool. The size information is used to characterize the size of the visible area of the input image processed by the image processing tool. Figure 8 FIG. 1 is a schematic diagram showing a scaling process of an initial image in an image annotation method proposed in an embodiment of the present application. Figure 8 As shown, in order to ensure that the image processing tool does not cause the image area to be lost or blank due to the mismatch between the size of the initial image and the size of the visible area when processing the input initial image, after obtaining the initial image, it is necessary to scale the initial image based on the scale information so that the size of the scaled initial image is consistent with the size of the visible area. If the aspect ratio of the initial image is larger than that of the visible area, the initial image is scaled proportionally to a width equal to the width of the visible area, otherwise the height is scaled proportionally to a height equal to the height of the visible area.
[0067] In an optional embodiment, the image processing tool is a Canvas tool, and the width of the image finally displayed in the visible area of the Canvas container is drawWidth pixels and the height is drawHeight pixels. The original initial image is obtained, and the height of the initial image is imgHeight pixels and the width is imgWidth pixels. The height of the visible area of the Canvas container is obtained as clientHeight pixels and the width is clientWidth pixels. If the ratio of the height to the width of the initial image, imgRatio, is greater than or equal to the ratio of the height to the width of the container Canvas, clientRatio, the image in the input visible area (i.e., the initial image after the scaling process) drawHeight is assigned to clientHeight, and drawWidth is assigned to imgWidth*(clientHeight / imgHeight); if the ratio of the height to the width of the initial image, imgRatio, is less than the ratio of the height to the width of the container Canvas, drawWidth is assigned to clientWidth, and drawHeight is assigned to imgHeight*(clientWidth / imgWidth). Therefore, it can be concluded that the scaling ratio of the initial image is equal to imgHeight / drawHeight or imgWidth / drawWidth, and this ratio value is used to finally correct the submitted annotation data.
[0068] In an embodiment of the present application, after acquiring the initial image, it is necessary to transform the color space of the initial image, convert the colored initial image into a 5-dimensional feature vector under the CEILab color space and pixel coordinates x and y, so as to cluster the pixels in the image. Specifically, the color data of the initial image is an RGBA (Red, Green, Blue, Alpha) color space, and the data type is a Uint8ClampedArray in the JavaScript language. First, the initial image after the scaling process is transformed into a first color space to obtain an image in an XYZ color space; the image in the XYZ color space is transformed into a second color space to obtain an image in a CEILab color space as the image to be annotated.
[0069] S102: Generate multiple target cluster centers on the image to be labeled based on an image processing tool.
[0070] When step S102 is specifically implemented, Figure 2a-2d is a set of example graphs of performing multiple clustering optimization of superpixels on an image to be annotated in an image annotation method proposed in an embodiment of the present application, such as Figure 2a-2dAs shown, firstly, a preset number of superpixels is obtained, where a superpixel is an area consisting of a group of connected pixels with similar appearance (such as Figure 2a Each block image area in the middle image in the image is generally similar in texture, color, brightness, etc. In the embodiment of the present application, the number of cluster centers initialized when the initial image is clustered is determined by presetting the number of superpixels. The more the number of preset superpixels, the higher the accuracy of the target area image obtained by the image annotation method provided by the present application.
[0071] It is easy to understand that the preset number of superpixels can be directly represented by the number of superpixels, or by the size of the superpixels. When the preset number of superpixels is represented by the size of the superpixels, the number of superpixels can be calculated by the total pixels of the image to be labeled and the size of each superpixel.
[0072] Subsequently, a distance metric is constructed for the 5-dimensional feature vector corresponding to the image to be annotated, and based on the image processing tool, a plurality of cluster centers are initialized and generated on the image to be annotated. Optionally, the initialized cluster centers can be cluster centers randomly generated on the image to be annotated based on the preset number of superpixels. Exemplarily, according to the parameter "preset number of superpixels" K and the total number of pixels N of the image to be annotated, an S*S area centered on a certain pixel can be evenly distributed in the image, S=Math.sqrt(N / K).
[0073] Then, the pixels on the image to be annotated are clustered based on the multiple cluster centers, and multiple initial superpixels of the preset number of superpixels are generated on the image to be annotated; the pixel with the smallest gradient is reselected for the multiple pixel points in each of the initial superpixels to obtain the updated cluster center in each of the initial superpixels, and the cluster center is updated by gradient to avoid the updated cluster center falling on the contour boundary of the superpixel and affecting the subsequent clustering effect. Specifically, according to the distance formula between the pixel and the cluster center, the CEILab color space data of the image to be annotated is input, the color gradient of the pixel is calculated, that is, the color gradient of the current pixel and its horizontal and vertical adjacent pixels is calculated to obtain the color gradient set edgeMap; through the CEILAB color space and the color gradient set edgeMap, the pixel with the smallest gradient is reselected as the center in the 2S*2S area of the seed point to obtain the updated cluster center newCenters. In an embodiment of the present application, the initialized pixel center corresponds to the S*S area. In order to accelerate the convergence of the algorithm, when updating the cluster center, the search range of the simple linear iterative clustering algorithm is 2S*2S, that is, by re-clustering the two superpixels corresponding to the initial cluster center.
[0074] Finally, re-clustering is performed based on the updated cluster center, and the above steps are repeated for multiple repetitions until the position of the updated cluster center in the target repetition round relative to the updated cluster center in the previous repetition round remains unchanged, or the error converges, and the multiple updated cluster centers of the target repetition round are used as the multiple target cluster centers. In general, the convergence error of the algorithm decreases significantly in several iterations, and experiments have shown that running the algorithm 4 to 10 times is sufficient.
[0075] Figure 3a-3f is another set of example diagrams of performing multiple clustering optimization of superpixels on an image to be annotated in an image annotation method proposed in an embodiment of the present application, such as Figure 3a-3f As shown (after one round of iterations, the effect is as follows Figure 3a After two rounds of iterations, the results are as follows Figure 3b After three rounds of iterations, the results are as follows Figure 3c After four rounds of iterations, the results are as follows Figure 3d After six rounds of iterations, the results are as follows Figure 3c After ten rounds of iterations, the results are as follows Figure 3f ), in the process of multiple re-clustering, the image area corresponding to each superpixel will gradually change. The more times of iterative updates, the stronger the consistency of the image features corresponding to each pixel in each superpixel, and the higher the efficiency and accuracy of subsequent annotation based on the superpixels of the multiple iterations.
[0076] S103: Clustering the pixel points on the image to be labeled based on the multiple target cluster centers, and generating multiple superpixels on the image to be labeled.
[0077] When the step S103 is specifically implemented, since in the process of forming superpixels through multiple clustering, only the number of superpixels is limited, and the area of each superpixel is not limited, the superpixels obtained after the iterative optimization may have superpixels that are too small in size. These superpixels with smaller sizes contain less image information and are difficult to play a greater positive role in the subsequent annotation and model training process. Therefore, in the embodiment of the present application, it is necessary to merge the smaller superpixels into larger superpixels. Specifically, the size information of each superpixel is obtained; based on the distance between the target cluster centers of adjacent superpixels, the adjacent superpixels of each superpixel are determined, and the adjacent superpixels are the superpixels corresponding to the target cluster centers that are closest to the target cluster centers of the superpixels; based on the preset superpixel size, the superpixels are classified based on the size information of each superpixel to obtain the first superpixel and the second superpixel, wherein the first superpixel is a superpixel whose size information is greater than the preset superpixel size, and the second superpixel is a superpixel whose size information is less than or equal to the preset superpixel size; the second superpixel is merged with the adjacent superpixel of the second superpixel.
[0078] In an embodiment of the present application, after the second superpixel and the adjacent superpixel are merged, the target cluster center of the adjacent superpixel is used as the cluster center of the merged superpixel. After the merging is completed, the pixels on the image to be annotated are clustered based on the multiple target cluster centers, and multiple superpixels are generated on the image to be annotated. The embodiment of the present application forms multiple superpixels through multiple clustering and pixel merging, which greatly improves the consistency of the image semantics represented by each superpixel. Annotation based on the multiple superpixels can improve the accuracy of the generated annotation points and avoid introducing pixels of the background area into the acquired target area image, which affects the annotation accuracy. At the same time, using a smaller-sized superpixel as the annotation unit has a smaller granularity than the entire image, which can greatly improve the annotation accuracy and efficiency.
[0079] S104 , using the foreground annotation information as a reference, based on the image processing tool, marking the image to be annotated that includes the multiple superpixels to obtain multiple first annotation points.
[0080] When step S104 is specifically implemented, Figures 4a-4d An example diagram of generating annotation points based on an image to be annotated containing superpixels in an image annotation method proposed in an embodiment of the present application is shown. Figure 2a-2d , Figures 4a-4d As shown in the left figure in , the user draws a graphic representing the foreground area of the image on the image to be annotated through the image processing tool, and determines the foreground image corresponding to the target area to be annotated through the graphic. Exemplarily, the image processing tool is a Canvas tool. When "Use Brush Tool" is clicked, the Canvas free drawing mode is turned on, that is, setIsDrawingMode(true); click "Foreground Brush" to set the brush color to green, that is, freeDrawingBrush.color=rgb(0,128,0); click "Background Brush" to set the brush color to red, that is, freeDrawingBrush.color=rgb(255,0,0). Generally, using the foreground brush can meet the needs of most segmentation image target areas.
[0081] Subsequently, obtain the foreground annotation information, where the foreground annotation information is marker information drawn in the foreground area of the image to be annotated based on the image processing tool; based on the foreground annotation information, filter the superpixels in the image to be annotated using the image processing tool; traverse all the superpixels in the image to be annotated, and take the superpixels containing the pixel points corresponding to the foreground annotation information as target superpixels, and the target superpixels form a foreground image; process multiple target superpixels using the image processing tool to obtain the multiple first annotation points, and the area enclosed by the multiple first annotation points at least includes the multiple target superpixels. Exemplarily, the image processing tool is a Canvas drawing tool. After the user obtains the foreground annotation information by drawing with a foreground brush, traverse the superpixel set indexMap. If a superpixel is marked by the foreground brush, the f mark (foreground) of the pixel set in this area is incremented by 1, that is, indexMap[i].mask.f++. If a superpixel is marked by a background edge brush, the b mark (background) of the pixel set in this area is incremented by 1, that is, indexMap[i].mask.b++. The finally retained area after segmentation is the pixel set where mask.f>0 and mask.b==0, and the foreground image as shown in the right figure in Figure 2a is obtained.
[0082] In some alternative embodiments, after determining the target superpixels (the foreground image), obtain the number of custom annotation points and the number of boundary pixels of the foreground image; based on the number of custom annotation points corresponding to the corresponding boundary and the number of boundary pixels of the foreground image, determine the number of the first annotation points. Specifically, according to the calculation of the Alpha channel of the foreground image, obtain the coordinate array ImageArray of the boundary pixels of the foreground image, and the data structure is like [[x1,y1],[x2,y2]..]; according to the number of custom annotation points PointCount configured customarily, if PointCount> the length of the Array, the number of the first annotation points generated on the image processing tool corresponds to the length of ImageArray; if PointCount< the length of ImageArray, by default, uniformly select the first annotation points at intervals in the middle of the boundary pixels Array of the foreground image, so that the number of the generated first annotation points is PointCount.
[0083] Exemplarily, scan the foreground image through a depth-first traversal algorithm to obtain the boundary point pixel set S of the foreground image. Let the ratio of the length of S to the length of the number of custom annotation points Points defined by the user be taken as an integer dis, traverse the set S with a step of dis, and draw the first annotation points at intervals. As Figure 2cThe right picture shows the foreground image with 100 first annotation points set. Figure 2d The right picture shows the foreground image with 50 first annotation points set; Figure 4b The right picture shows the foreground image with 100 first annotation points set. Figure 4d Set the foreground image of the 50 first annotation points.
[0084] In an embodiment of the present application, the image processing tool is used to annotate the image to be annotated containing multiple superpixels to obtain the target superpixel, without the need for the annotator to manually annotate each point along the contour of the target segmented area from scratch, thereby greatly improving the efficiency and quality of image annotation.
[0085] In some optional implementations, after the first annotation point is generated, since the first annotation point is completely generated based on the image processing tool, in order to further improve the accuracy of the training data, the first annotation point can also be fine-tuned based on the user's operation instruction. Specifically, in response to the user's annotation point movement operation instruction, the first position information is determined, and the position of the target annotation point is moved based on the first position information, and the target annotation point is any one of the multiple first annotation points, and / or, in response to the user's annotation point addition command, the second position information is determined, and a new first annotation point is generated on the image to be annotated based on the second position information, and / or, in response to the user's annotation point deletion command, the third position information is determined, and the original first annotation point is deleted on the image to be annotated based on the third position information. In the embodiment of the present application, by adjusting the position, moving, adding and deleting the first annotation point after it is generated, the accuracy of the first annotation point can be improved while minimizing the user's workload. The user does not need to manually generate all the annotation points, but only needs to make personalized adjustments to a small part of the automatically generated annotation points, thereby reducing the user's work cost.
[0086] In some optional implementations, before generating the first annotation point, the color and size of the first annotation point, the text information corresponding to the annotation point, etc. may be pre-set. Specifically, it includes but is not limited to the following contents:
[0087] Define the Circle class as an object for drawing solid points. The input parameters are the horizontal coordinate x and the vertical coordinate y. The radius can be configured; the fill color of the marked point can be configured. The default value is rgb(250,250,0). The Circle object is displayed as a yellow dot on the Canvas.
[0088] Define the Text class as a text label object for drawing solid points. The input parameters are the corresponding Circle object and the label text content name. The horizontal and vertical coordinates of the Text object are the horizontal and vertical coordinates of the Circle. The font format fontFamily can be configured, and the default value is "consolas, PingFangSC-Ultralight, Microsoft-Yahei, sans-serif"; the font color fill can be configured, and the default value is rgb(0,0,0); the background color backgroundColor of the font can be configured, and the default value is rgba(255,255,255,0.8); the font weight fontWeight can be configured, and the default value is 800; whether the label is visible can be configured, that is, the visible Boolean value field of the Text object can be configured, and the default value is true. This field can be used by the annotator to display and hide the text label with the shortcut key v operation; the display effect of the Text object on the Canvas is the index text 0, 1, 2, 3...39 in the yellow dot.
[0089] Define the Group class as a pair of Circle and Text group objects that correspond to each other. The input parameters are Circle object, Text object, region index parentId, and index of the annotation point in a region. The biggest difference between the Group class and Circle and Text is that when modifying the annotation point (such as moving the annotation point), the subject of interaction is Group instead of Circle and Text, which simplifies the code logic for executing related function calculations. A Group object is displayed on the Canvas as a yellow point (such as P7) and the text label (7) corresponding to the point (P7).
[0090] Define the BText class as a text label object for a marked area (a collection of several Group objects). The input parameters are the horizontal coordinate x and the vertical coordinate y. The default horizontal coordinates and vertical coordinates are Kx and Ky+16, which are the horizontal coordinates of the Group K at the bottom (with the largest vertical coordinate value) of all Group objects in the corresponding marked area. The annotator can move the position of the BText object, and the maximum moving range does not exceed the horizontal coordinate +30 and the vertical coordinate +30 of the Group closest to the area.
[0091] There can be several marked areas in Canvas. Figure 5a-5b An example diagram of annotated points in an image annotation method proposed in an embodiment of the present application is shown. Figure 5a-5bAs shown in the figure, the area connected by several annotation points of "anime face" is a annotation area, and its classification label is "anime face". Each annotation area corresponds to an areaId. If a new annotation area is created, the areaId value is Date.now(), which is the timestamp when the annotation area is created. The annotation area can be moved as a whole, the label can be modified, deleted, displayed, and hidden.
[0092] When creating a new annotation point, select the areaId to be annotated. The global variable activePointIndex represents the annotation point index of the current area (such as Figure 5a-5b areaId and activePointIndex together index the currently operated area and the annotation point. New annotation points can be inserted in sequence. For example, if there are already points 0, 1, 2, 3, the index of the newly inserted point is 4. If the annotation of another target area is performed after the annotation of the target area is completed, the areaId and activePointIndex are reset.
[0093] When deleting a marked point, the point already has the areaId, index pointIndex, and the Group object of the adjacent nodes connected to each other recorded, so after deleting the point, the adjacent nodes must be connected to each other.
[0094] S105 : Filter pixel points in the image to be annotated based on the first annotation points to obtain a target area image.
[0095] When step S105 is specifically implemented, the pixels in the image to be annotated are screened based on the first annotation points to obtain a target area image, which includes the multiple first annotation points and the target superpixels (foreground image) in the area surrounded by the multiple first annotation points. The target area image is an annotated image obtained by automatic annotation based on an image processing tool. In order to realize that the image semantic segmentation model can be automatically pre-annotated, the target area image is used to train the image semantic segmentation model.
[0096] Specifically, after the pixel points in the image to be annotated are screened based on the first annotation points to obtain the target area image, the method further includes: constructing multiple sets of training data, the training data at least including the initial image, the target area image, the position information and semantic information of the multiple first annotation points, exemplarily, the data structure of the first annotation point information and the semantic information is in JSON format, and the specific data example is as follows:
[0097]
[0098]
[0099] Subsequently, the multiple sets of training data are sent to the server and input into the image semantic segmentation model. The image semantic segmentation model is iteratively trained based on the multiple training data to obtain a target model, which is used to pre-annotate the input image to generate a target area image containing the annotation points. Figure 6 Another example diagram of annotated points in an image annotation method according to an embodiment of the present application is shown. Figure 7 FIG. 1 shows a schematic diagram of a training process of an image semantic segmentation model in an image annotation method proposed in an embodiment of the present application. Figure 7 As shown, AI model pre-annotation data refers to the process of generating the first annotation point by annotation based on the image processing tool, the polygon annotation method for annotating the target area contour refers to the process of determining the target area image based on the first annotation point, and the simple linear iterative distance algorithm for segmenting the image refers to the process of generating multiple superpixels based on the image to be annotated. As shown in Figure 5, Figure 5 is an example of annotated data (target area image) obtained after image pre-annotation, the annotated data includes a foreground image, annotation points and semantic information (animated face), and the training data is transmitted in the format of a JSON data structure to the system server where the image semantic segmentation model is located for subsequent training and testing processes.
[0100] Optionally, after loading a new batch of training data, the labelFeature field in JSON is loaded by default, and the points array in labelFeature is traversed. The index 0 of each sub-array in points is the horizontal coordinate and the index 1 is the vertical coordinate. Circle objects, Text objects, and Group objects are created, and all Group objects are connected. Then, a BText object is created according to the tagName field in labelFeature, and the following is obtained: Figure 6 The labeled data shown.
[0101] In some optional implementations, an operation undo stack opStack may also be defined, and various operations performed by the annotator in the image processing tool (such as Canvas), such as adding a new annotation point (new), moving an annotation point (move), deleting an annotation point (del), etc., may be recorded in opStack through a push operation, and the function of undoing this operation may be realized through a pop operation. For example, when this operation is move, the position of the annotation point before and after the move on the Canvas may be recorded in opStack, and after the pop operation, this move operation is canceled, and the annotation point returns to the position before the move.
[0102] In some optional implementations, the current annotation information is recorded in LocalStorage cache to prevent unexpected errors in image processing tools or interruptions in device operation, which may lead to loss of annotation information and increase the annotation cost. The data structure of the annotation data recorded in LocalStorage is consistent with the JSON data structure in the training data and the data structure of the pre-annotated data, thereby realizing friendly on-site recovery of interrupted annotation tasks.
[0103] In some optional embodiments, in order to improve the annotation efficiency, the embodiments of the present application support shortcut key operations, such as creating points (left mouse click), modifying points (dragging the annotation points with the mouse), deleting points (DELETE key), undoing (right mouse click), shrinking and enlarging images (mouse wheel), showing and hiding labels (W key), showing and hiding annotation traces (E key), dragging pictures (Alt+mouse drag in Window system, Option key+mouse drag in MacOS system), adjusting the annotation point radius (<,>), adjusting the line thickness of the annotation area connection line ([,])... and so on.
[0104] An embodiment of the present application provides an image annotation method, which clusters an image to be annotated based on an image processing tool to construct multiple superpixels, and generates marker points based on the image processing tool to obtain a target area image. The efficiency and speed of the image processing tool in generating marker points are improved through superpixels, so as to achieve fast and effective image semantic segmentation and annotation of the image data to be annotated, and effectively improve the annotation efficiency and annotation quality.
[0105] Based on the same inventive concept, the present application embodiment discloses an image annotation device. Fig. 9 FIG. 4 shows a schematic diagram of the structure of an image annotation device proposed in an embodiment of the present application. Fig. 9 As shown, the device comprises:
[0106] The image to be annotated module is used to obtain an initial image and preprocess the initial image to obtain an image to be annotated;
[0107] A cluster center module, used for generating a plurality of target cluster centers on the image to be labeled based on an image processing tool;
[0108] A superpixel module, used for clustering the pixel points on the image to be labeled based on the multiple target cluster centers, and generating multiple superpixels on the image to be labeled;
[0109] A labeling module, used for labeling the image to be labeled containing the plurality of superpixels based on the image processing tool based on the foreground labeling information to obtain a plurality of first labeling points;
[0110] The target area image module is used to filter the pixel points in the image to be annotated based on the first annotation points to obtain a target area image.
[0111] In an optional implementation, the cluster center module includes:
[0112] A superpixel number submodule is used to obtain a preset superpixel number;
[0113] An initialization cluster center submodule is used to initialize and generate multiple cluster centers on the image to be labeled based on the image processing tool;
[0114] An initial superpixel submodule, used for clustering the pixel points on the image to be labeled based on the multiple cluster centers, and generating a plurality of initial superpixels of the preset number of superpixels on the image to be labeled;
[0115] A clustering update submodule, used for reselecting a pixel point with the smallest gradient for a plurality of pixel points in each of the initial superpixels to obtain an updated cluster center in each of the initial superpixels;
[0116] A repetition submodule is used to perform re-clustering based on the updated cluster center, repeat the above steps until the position of the updated cluster center in the target repetition round relative to the updated cluster center in the previous repetition round remains unchanged, and use the multiple updated cluster centers of the target repetition round as the multiple target cluster centers.
[0117] In an optional implementation, the marking module includes:
[0118] A foreground annotation information submodule, used to obtain the foreground annotation information, where the foreground annotation information is marking information drawn on the foreground area of the image to be annotated based on the image processing tool;
[0119] A first screening submodule, used for screening superpixels in the image to be labeled based on the image processing tool based on the foreground labeling information;
[0120] A target superpixel submodule is used to traverse all superpixels in the image to be annotated, and take the superpixel containing the pixel point corresponding to the foreground annotation information as the target superpixel;
[0121] The annotation submodule is used to process the multiple target superpixels based on the image processing tool to obtain the multiple first annotation points, and the area enclosed by the multiple first annotation points at least includes the multiple target superpixels.
[0122] In an optional embodiment, the device further comprises:
[0123] a marking point moving module, configured to determine first position information in response to a marking point moving operation instruction from a user, and to move the position of a target marking point based on the first position information, wherein the target marking point is any one of the plurality of first marking points, and / or
[0124] a marking point adding module, configured to determine second position information in response to a marking point adding command from a user, and generate a newly added first marking point on the image to be marked based on the second position information, and / or,
[0125] The annotation point deletion module is used to determine the third position information in response to the user's annotation point deletion command, and delete the original first annotation point on the image to be annotated based on the third position information.
[0126] In an optional implementation, the to-be-annotated image module includes:
[0127] A visible area information acquisition submodule, used to acquire the size information of the visible area of the image processing tool;
[0128] A scaling submodule, configured to scale the initial image based on the scale information so that the size of the initial image after scaling is consistent with the size of the visible area;
[0129] A first color space transformation submodule, used for performing a first color space transformation on the initial image after the scaling process to obtain an image in an XYZ color space;
[0130] The second color space transformation submodule is used to perform a second color space transformation on the image in the XYZ color space to obtain an image in the CEILab color space as the image to be labeled.
[0131] In an optional embodiment, the device further comprises:
[0132] A superpixel size module, used to obtain size information of each superpixel;
[0133] A neighboring superpixel module, used to determine the neighboring superpixels of each superpixel based on the distances between the target cluster centers of adjacent superpixels, wherein the neighboring superpixels are superpixels corresponding to the target cluster center that is closest to the target cluster center of the superpixel;
[0134] A superpixel classification module, configured to classify the superpixels based on the size information of each superpixel, taking a preset superpixel size as a reference, to obtain a first superpixel and a second superpixel, wherein the first superpixel is a superpixel whose size information is greater than the preset superpixel size, and the second superpixel is a superpixel whose size information is less than or equal to the preset superpixel size;
[0135] A superpixel merging module is used to merge the second superpixel with the adjacent superpixel of the second superpixel.
[0136] In an optional embodiment, the device further comprises:
[0137] A training data module, used to construct multiple sets of training data, wherein the training data at least includes the initial image, the target area image, and the position information and semantic information of the multiple first annotation points;
[0138] The model optimization module is used to input the multiple sets of training data into the image semantic segmentation model, iteratively train the image semantic segmentation model based on the multiple training data, and obtain a target model, wherein the target model is used to pre-annotate the input image to generate a target area image containing annotated points.
[0139] Based on the same inventive concept, the present application embodiment discloses an electronic device, Fig.10 A schematic diagram of an electronic device proposed in an embodiment of the present application is shown, such as Fig.10 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory 110 and the processor 120 are connected to each other through a bus communication, the memory 110 stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set can be loaded and run on the processor 120 to implement the steps in the image annotation method disclosed in the embodiment of the present application.
[0140] Based on the same inventive concept, an embodiment of the present application discloses a computer-readable storage medium, on which is stored at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the steps in the image annotation method disclosed in the embodiment of the present application.
[0141] Based on the same inventive concept, an embodiment of the present application discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the image annotation method disclosed in the embodiment of the present application.
[0142] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other.
[0143] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0146] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0147] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0148] The above is a detailed introduction to an image annotation method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An image annotation method, characterized in that: The method comprises: Obtain an initial image, and preprocess the initial image to obtain an image to be labeled; Based on an image processing tool, generating a plurality of target cluster centers on the image to be labeled; Clustering the pixel points on the image to be labeled based on the multiple target cluster centers to generate multiple superpixels on the image to be labeled; Based on the foreground annotation information, the image to be annotated containing the plurality of superpixels is marked based on the image processing tool to obtain a plurality of first annotation points; Pixel points in the image to be annotated are screened based on the first annotation points to obtain a target area image.
2. The image annotation method according to claim 1, characterized in that: The method of generating a plurality of target cluster centers on the image to be labeled based on the image processing tool includes: Get the preset number of superpixels; Based on the image processing tool, initially generating multiple cluster centers on the image to be labeled; Clustering the pixel points on the image to be labeled based on the multiple cluster centers, and generating a plurality of initial superpixels having the preset number of superpixels on the image to be labeled; Reselecting a pixel point with the smallest gradient for a plurality of pixel points in each of the initial superpixels to obtain an updated cluster center in each of the initial superpixels; Re-clustering is performed based on the updated cluster center, and the above steps are repeated until the position of the updated cluster center in the target repetition round relative to the updated cluster center in the previous repetition round remains unchanged, and the multiple updated cluster centers of the target repetition round are used as the multiple target cluster centers.
3. The image annotation method according to claim 1, characterized in that: The step of marking the image to be marked containing the plurality of superpixels based on the image processing tool based on the foreground marking information to obtain a plurality of first marking points includes: Acquire the foreground annotation information, where the foreground annotation information is marking information drawn in the foreground area of the image to be annotated based on the image processing tool; Taking the foreground annotation information as a reference, and based on the image processing tool, superpixels in the image to be annotated are screened; Traversing all superpixels in the image to be annotated, and taking the superpixel containing the pixel point corresponding to the foreground annotation information as the target superpixel; The plurality of target superpixels are processed based on the image processing tool to obtain the plurality of first annotation points, and the area enclosed by the plurality of first annotation points at least includes the plurality of target superpixels.
4. The image annotation method according to claim 1, characterized in that: Before filtering the pixel points in the image to be annotated based on the first annotation points to obtain the target area image, the method further includes: In response to a user's mark point moving operation instruction, first position information is determined, and a position of a target mark point is moved based on the first position information, the target mark point being any one of the plurality of first mark points, and / or, In response to a user's command to add a new annotation point, second position information is determined, and a new first annotation point is generated on the image to be annotated based on the second position information, and / or, In response to a user's annotation point deletion command, third position information is determined, and the original first annotation point is deleted on the image to be annotated based on the third position information.
5. The image annotation method according to claim 1, characterized in that: The preprocessing of the initial image to obtain the image to be labeled includes: Obtaining size information of a visible area of the image processing tool; Scaling the initial image based on the scale information so that the size of the initial image after the scaling is consistent with the size of the visible area; Performing a first color space transformation on the initial image after the scaling process to obtain an image in an XYZ color space; Perform a second color space transformation on the image in the XYZ color space to obtain an image in the CEILab color space as the image to be labeled.
6. The image annotation method according to claim 1, characterized in that: Before the image to be annotated containing the plurality of superpixels is labeled based on the image processing tool based on the foreground annotation information to obtain a plurality of first annotation points, the method further includes: Obtaining size information of each superpixel; Based on the distance between the target cluster centers of adjacent superpixels, determining the neighboring superpixels of each superpixel, wherein the neighboring superpixels are superpixels corresponding to the target cluster center that is closest to the target cluster center of the superpixel; Based on the preset superpixel size, the superpixels are classified based on the size information of each superpixel to obtain a first superpixel and a second superpixel, wherein the first superpixel is a superpixel whose size information is greater than the preset superpixel size, and the second superpixel is a superpixel whose size information is less than or equal to the preset superpixel size; The second superpixel is merged with adjacent superpixels of the second superpixel.
7. The image annotation method according to any one of claims 1 to 6, characterized in that: After the pixel points in the image to be annotated are screened based on the first annotation points to obtain the target area image, the method further includes: Constructing multiple sets of training data, wherein the training data at least includes the initial image, the target area image, and position information and semantic information of the multiple first annotation points; The multiple sets of training data are input into an image semantic segmentation model, and the image semantic segmentation model is iteratively trained based on the multiple training data to obtain a target model, where the target model is used to pre-annotate the input image to generate a target area image containing annotated points.
8. An image annotation device, characterized in that: The device comprises: The image to be annotated module is used to obtain an initial image and preprocess the initial image to obtain an image to be annotated; A cluster center module, used for generating a plurality of target cluster centers on the image to be labeled based on an image processing tool; A superpixel module, used for clustering the pixel points on the image to be labeled based on the multiple target cluster centers, and generating multiple superpixels on the image to be labeled; A labeling module, used for labeling the image to be labeled containing the plurality of superpixels based on the image processing tool based on the foreground labeling information to obtain a plurality of first labeling points; The target area image module is used to filter the pixel points in the image to be annotated based on the first annotation points to obtain a target area image.
9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the image annotation method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the image annotation method described in any one of claims 1 to 7 are implemented.