Image segmentation-based model training method, device, system, and electronic device

Abstract

The application discloses a model training method, device and system based on image segmentation and electronic equipment, comprising: receiving a user inputted annotation figure used for image segmentation of a target annotation image; determining shape information, key point information and attribute information of the annotation figure, and representing the information in a text form to constitute annotation information of the image segmentation; and transmitting the annotation information to a database for storage and used for model training of the target annotation image. The application can effectively reduce the transmission amount of image segmentation annotation information and save transmission resources.

Description

Technical Field

[0001] This application relates to image processing technology, and in particular to a model training method, apparatus, system, storage medium, and electronic device based on image segmentation. Background Technology

[0002] Image segmentation involves dividing pixels in an image into different parts according to certain rules and assigning them different labels. In industrial defect detection, it's used for foreground extraction and defect detection, predicting the location of various defects in an image. Furthermore, using the original image before segmentation (hereinafter referred to as the target labeled image) and the various labels assigned after segmentation, various types of models can be trained.

[0003] Image segmentation annotations, used to label the segmented image, are typically described using masks. This is because image segmentation requires high precision in annotations, often down to the pixel level. Therefore, when common rectangular or polygonal annotation methods are insufficient, pen annotations are necessary. Pen annotations represent an irregular area drawn by the user, which cannot be represented by specific geometric shapes. Therefore, to ensure that pen annotations are included, image annotation information is generally represented using masks.

[0004] The mask image has a white background, and different grayscale values ​​are used for different target categories. When training various models using images, the target pixels in the training image can be calculated by performing an AND operation between the mask image and the image to be trained.

[0005] For model training using target labeled images and image segmentation labels, the user's annotations on the target labeled images can be transmitted to the model trainer for model training. Since the image annotation information uses a mask image, the mask image needs to be transmitted when transmitting the annotations. However, the mask image has the same resolution as the target labeled image, and the image is relatively large, which is not conducive to network transmission. Summary of the Invention

[0006] This application provides a model training method, apparatus, system, storage medium, and electronic device based on image segmentation, which can reduce the amount of data transmitted for annotation and improve transmission efficiency.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] A model training method based on image segmentation includes:

[0009] Receives user-input labeled graphics used for image segmentation of the target labeled image;

[0010] The shape information, key point information, and attribute information of the labeled graphic are determined and represented in text form to form the annotation information for image segmentation;

[0011] The annotation information is transmitted to a database for storage, and the annotation information is used to train a model on the target annotated image.

[0012] Preferably, when the user inputs the labeled graphic, the user indicates that the input should be performed using a regular graphic method, and the shape information of the labeled graphic is determined to be the shape information of the regular graphic indicated by the user.

[0013] When the user inputs the labeled graphic, the user is instructed to input it using a custom graphic method, and the shape information of the labeled graphic is determined to be a custom shape.

[0014] Preferably, when the shape information of the labeled graphic is a rectangle, the key point information includes the number of key points and the position information of the four corner points;

[0015] And / or, when the shape information of the labeled graphic is a polygon, the key point information includes the number of key points and the position information of all corner points;

[0016] And / or, when the shape information of the labeled graphic is circular, the key point information includes the center position information and the radius / diameter length information;

[0017] And / or, when the shape information of the labeled graphic is the custom shape, the key point information includes: initial key point position information, all first distances, and all second distances; wherein, the method for determining the initial key point, the first distance, and the second distance includes:

[0018] For the original image before image segmentation, each pixel in the original image is traversed sequentially according to the image array index;

[0019] The first pixel located within the labeled graphic is taken as the initial key point;

[0020] The first pixel outside the labeled graphic after each first-class keypoint is designated as a second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint is calculated as the first distance. The first-class keypoints include the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint.

[0021] Calculate the index difference between each first-class keypoint and its previous adjacent second keypoint as the second distance; in the original image, the last second distance is the difference between w*h and the index of the last second-class keypoint; where w and h are the width and height of the original image, respectively.

[0022] Preferably, the method further includes:

[0023] When it is necessary to display the labeled graphic, obtain the labeled information;

[0024] Based on the shape information, key point information, and attribute information of the labeled graphic included in the acquired annotation information, the labeled graphic is drawn and displayed.

[0025] Preferably, the annotation information is constructed according to a set file format;

[0026] The step of transmitting the annotation information to the database for storage includes: parsing the annotation information into structured data and storing it in the database;

[0027] The step of obtaining the annotation information includes: obtaining the annotation information stored in the form of structured data from the database, and reorganizing it according to the file format.

[0028] A model training method based on image segmentation includes:

[0029] Retrieve annotation information for image segmentation of the target labeled image from the database;

[0030] The annotation information and the target annotated image are input into the model training algorithm to train the model using the target annotated image;

[0031] The annotation information includes shape information, key point information, and attribute information of the annotation graphic represented by text, and the annotation graphic is used for image segmentation.

[0032] Preferably, the database stores structured data obtained after parsing the annotation information;

[0033] The step of obtaining the annotation information for image segmentation of the target annotated image from the database includes:

[0034] The structured data is retrieved from the database and reorganized into the annotation information.

[0035] Preferably, inputting the labeled information into the model training algorithm includes:

[0036] The shape information, key point information, and attribute information of the labeled graphics included in the labeled information are input into the model training algorithm; or,

[0037] Based on the shape information, key point information, and attribute information of the labeled graphic included in the annotation information, a mask image corresponding to the labeled graphic is generated and input into the model training algorithm.

[0038] An image segmentation annotation client includes: a user interface unit, an annotation information generation unit, and a database interface unit;

[0039] The user interface unit is used to receive annotation graphics input by the user for image segmentation of the target annotation image;

[0040] The annotation information generation unit is used to determine the shape information, key point information and attribute information of the annotation graphic, and represent them in text form to form the annotation information of the image segmentation.

[0041] The database interface unit is used to transmit the annotation information to the database for storage, and the annotation information is used to train the model on the target annotated image.

[0042] Preferably, the annotation graphic received by the user interface unit is input by the user through a regular graphic, and the annotation information generation unit determines the shape information of the annotation graphic to be the shape information of the regular graphic indicated by the user.

[0043] The annotation graphic received by the user interface unit is input by the user through a custom method, and the annotation information generation unit determines that the shape information of the annotation graphic is a custom shape;

[0044] And / or,

[0045] When the user interface unit determines that the shape information of the regular graphic is a polygon, the annotation information generation unit determines that the key point information includes the number of key points and the position information of all corner points;

[0046] And / or, when the user interface unit determines that the shape information of the regular graphic is a polygon, the annotation information generation unit determines that the key point information includes the number of key points and the position information of all corner points;

[0047] And / or, when the user interface unit determines that the shape information of the regular graphic is a circle, the annotation information generation unit determines that the key point information includes the center position information and the radius / diameter length information;

[0048] And / or, when the user interface unit determines that the shape information of the labeled graphic is the custom shape, the labeling information generation unit determines the key point information including: initial key point position information, all first distances, and all second distances; wherein, in the labeling information generation unit, the method for determining the initial key point, the first distance, and the second distance includes:

[0049] For the original image before image segmentation, each pixel in the original image is traversed sequentially according to the image array index;

[0050] The first pixel located within the labeled graphic is taken as the initial key point;

[0051] The first pixel outside the labeled graphic after each first-class keypoint is designated as a second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint is calculated as the first distance. The first-class keypoints include the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint.

[0052] Calculate the index difference between each first-class keypoint and its previous adjacent second keypoint as the second distance; in the original image, the last second distance is the difference between w*h and the index of the last second-class keypoint.

[0053] And / or,

[0054] The annotation client further includes a display unit, which is used to obtain the annotation information when the annotation graphic needs to be displayed, and is also used to draw and display the annotation graphic based on the shape information, key point information and attribute information of the annotation graphic included in the obtained annotation information;

[0055] And / or,

[0056] The annotation information generation unit is further configured to perform the process according to a set file format when composing the annotation information;

[0057] In the database interface unit, transmitting the annotation information to the database storage includes: parsing the annotation information into structured data and storing it in the database;

[0058] In the echo unit, obtaining the annotation information includes: obtaining the annotation information stored in the form of structured data from the database, and reorganizing it according to the file format.

[0059] A model trainer includes: an acquisition unit, an input unit, and a training unit;

[0060] The acquisition unit is used to acquire annotation information for image segmentation of the target labeled image from the database;

[0061] The input unit is used to input the annotation information and the target annotation image into the model training algorithm;

[0062] The training unit is used to train a model on the target labeled image;

[0063] The annotation information includes shape information, key point information, and attribute information of the annotation graphic represented by text, and the annotation graphic is used for image segmentation.

[0064] Preferably, the database stores structured data obtained after parsing the annotation information;

[0065] In the acquisition unit, the step of acquiring the annotation information for image segmentation of the target labeled image from the database includes: acquiring the structured data from the database and reorganizing it into the annotation information;

[0066] And / or,

[0067] In the input unit, the step of inputting the annotation information into the model training algorithm includes:

[0068] The shape information, key point information, and attribute information of the labeled graphic included in the annotation information are input into the model training algorithm; or, based on the shape information, key point information, and attribute information of the labeled graphic included in the annotation information, a mask image corresponding to the labeled graphic is generated and input into the model training algorithm.

[0069] A model training system based on image segmentation includes one or more annotation clients as described above, a model trainer as described above, and a database;

[0070] The database is used to store annotation information sent by the annotation client.

[0071] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, can implement the image segmentation-based model training method described above.

[0072] An electronic device, comprising at least a computer-readable storage medium and a processor;

[0073] The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the image segmentation-based model training method described in any of the preceding claims.

[0074] As can be seen from the above technical solution, the system receives user-inputted annotation graphics for image segmentation of the target annotation graphics; determines the shape information, key point information, and attribute information of the annotation graphics, and represents them in text form to form the annotation information for image segmentation; transmits the annotation information to a database for storage, and uses it for model training on the target annotation image. Since the annotation information is represented in text form, compared to directly transmitting a mask image, this application can effectively reduce the amount of data transmitted and improve transmission efficiency. Furthermore, when it is necessary to display the annotation information, the annotation information can be obtained, and based on the shape information, key point information, and attribute information of the annotation graphics included in the obtained annotation information, the annotation graphics can be drawn and displayed. Thus, the displayed annotation information still includes the drawing information when the user inputs the annotation graphics, thereby enabling the editing of the displayed annotation graphics. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the basic structure of the image segmentation annotation system in this application;

[0076] Figure 2 This is a flowchart illustrating the annotation method for image segmentation in this application;

[0077] Figure 3a A schematic diagram for drawing labeled graphics according to user instructions. Figure 1 ;

[0078] Figure 3b A schematic diagram for drawing labeled graphics according to user instructions. Figure 2 ;

[0079] Figure 3c Schematic diagram 3 showing how to draw annotated graphics according to user instructions;

[0080] Figure 4 A basic structural diagram of the client-side annotation;

[0081] Figure 5 A schematic diagram of the basic structure of a model trainer;

[0082] Figure 6 This is a schematic diagram of the basic structure of the annotation server;

[0083] Figure 7 This is a schematic diagram of the basic structure of the electronic device in this application. Detailed Implementation

[0084] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0085] The basic idea of ​​this application is to use text format to represent the annotation information of image segmentation in the model training based on image segmentation, which can effectively reduce the amount of data transmitted when transmitting annotation information and improve transmission efficiency.

[0086] This application provides an image segmentation annotation system that can be used to implement the annotation method in this application. Users can draw image segmentation annotation graphics on the annotation client and display the corresponding annotation graphics on other annotation clients. Figure 1 This is a schematic diagram of the architecture of the image segmentation-based model training system in this application. At its most basic level, the system includes: one or more annotation clients, a database, and a model trainer; more preferably, it may also include an annotation server for processing based on annotation information.

[0087] Figure 2 This diagram illustrates the basic workflow of the image segmentation-based model training method in this application. The workflow is described from the perspective of the interactions between the components of the annotation system. This annotation method can... Figure 1 Implementation in the system shown, such as Figure 2 As shown, the method includes:

[0088] Step 201: The annotation client receives the annotation graphics input by the user for image segmentation of the target annotation image.

[0089] Users can input labeled graphics in existing ways, such as by inputting them using existing rule-based graphics or by inputting them using custom graphics.

[0090] For existing regular images, specifically, users can click the button to select an existing regular graphic, indicating that the annotation graphic should be input in the manner of that regular graphic, and the client will draw the corresponding annotation graphic based on the user's input.

[0091] For example, when a user clicks the "Rectangle" button, they are instructed to input a rectangle as the label; the label shape is then a rectangle. The user presses and drags the mouse to begin drawing, and releases it to end the drawing. Starting from the initial point, the coordinates of the four corners of the rectangle are recorded clockwise. "point":"x1 y1 x2 y2 x3 y3 x4 y4", as shown. Figure 3a As shown, draw a rectangle.

[0092] For example, when a user clicks the "Polygon" button, indicating that a polygon is to be entered as the annotation, the annotation shape is a polygon. Drawing begins after the mouse button is pressed and released; one contour point of the polygon is recorded. The remaining contour points are drawn sequentially, ending the drawing process once all contour points of the polygon have been drawn. The coordinates of each contour point of the polygon are recorded from the starting point in the drawing order. "point":"x1 y1 x2 y2 x3 y3 x4 y4 x5 y5 x6 y6 x7y7 x8 y8", such as... Figure 3b As shown, draw the polygon.

[0093] For inputting custom graphics, specifically, users can click on an existing custom graphics drawing button (such as a pen button) to indicate that the annotation graphics are input in the form of custom graphics, and the client will draw the corresponding annotation graphics according to the user's input.

[0094] For example, a user clicks the "pen" button, indicating that they should input a custom annotation graphic; the mouse is pressed to start drawing, drawing each pixel the mouse hovers over, and recording all pixel information after drawing is complete. "pixelData":"368107,26,990,34,990,34,989,42,980,44,977,50,974,52,970,54,970,54,966,60,964,62,960,64,960,67,956,69,954,70,954,70,954,70,951,74,950,74" 949,75,948,77,947,77,947,77,946,78,946,78,946,78,946,78,946,78,946,78,946,78,944,80,943,81,942,82,942,82,940,84,940,84,940,84,939,85,939,85,939,85,939,85,939 9,85,939,39,4,42,939,85,939,81,943,81,943,81,943,81,943,81,943,21,1,55,947,77,947,77,947,77,947,75,949,72,952,72,952,70,954,68,956,68,956,12,1,55,956 ,13,2,51,958,13,4,49,958,13,5,44,963,14,20,27,963,14,15,32,963,19,3,37,966,58,966,56,968,54,971,51,975,47,977,47,977,40,987,33,991,31,998,20,342553", as shown Figure 3c As shown, draw a custom shape.

[0095] Step 202: The annotation client determines the shape information, key point information, and attribute information of the annotation graphic.

[0096] In this application, the general attribute information of the labeled graphics refers to some attributes that express image information, such as ID information and color information. The shape information and key point information of the labeled graphics are used to jointly represent the shape and specific location of the labeled graphics. By using the shape information, key point information, and attribute information of the labeled graphics to represent the labeled graphics, and expressing the above information in text form, the labeled graphics used for image segmentation in this application can be expressed in text form. The labeled graphics expressed in this text form serve as the annotation information for image segmentation in this application, instead of using a mask image, thereby greatly reducing the amount of annotation information transmitted.

[0097] Specifically, the attribute information can be determined using existing methods. The shape information and key point information of the labeled graphic can be determined as follows:

[0098] If the user indicates that the annotation graphic is input using a regular graphic method in step 201, then the shape information of the annotation graphic is determined to be the shape information of the regular graphic indicated by the user. For example, if the user clicks the "rectangle" button in step 201 to indicate that a rectangular annotation graphic is input as the annotation, then the shape information of the annotation graphic is determined to be the shape information of the rectangular annotation graphic indicated by the user, that is, the shape information of the annotation graphic is a rectangle. Of course, if the regular graphic indicated by the user is a polygon or a circle, then the shape information of the annotation graphic is determined to be a polygon or a circle, respectively.

[0099] If the user in step 201 indicates that the annotation graphic is input using a custom graphic method, then the shape information of the annotation graphic is determined to be a custom shape. For example, if the user clicks the "pen" button in step 201 and indicates that a custom annotation graphic is input as the annotation, then the shape information of the annotation graphic is determined to be a custom shape, and the specific shape is expressed through subsequent key point information.

[0100] Key point information is used to express crucial information such as the specific location of annotated graphics. The content of key point information may differ depending on the shape of the annotated graphic. Below are examples of key point information for several typical shapes:

[0101] 1. When the shape information of the labeled graphic is a rectangle, the key point information can include the number of key points and the position information of the four corner points; where the number of key points is 4, the position information of the corner points can be the coordinate values ​​of the corner points;

[0102] 2. When the shape information of the labeled graphic is a polygon, the key point information can include the number of key points and the position information of all corner points; where the number of key points is the number of sides of the polygon, and the position information of the corner points can be the coordinate values ​​of the corner points;

[0103] 3. When the shape information of the labeled graphic is a circle, the key point information may include the center position information and the length information of the radius or diameter;

[0104] 4. When the shape information of the labeled graphic is a custom shape, the key point information may include: initial key point position information, all first distances, and all second distances; wherein, the method for determining the initial key points, first distances, and second distances may include:

[0105] For the original image before image segmentation, traverse each pixel in the original image sequentially according to the image array index;

[0106] Use the first pixel located within the labeled graphic as the initial key point;

[0107] The first pixel outside the labeled graphic after each first-class keypoint is taken as the second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint is calculated as the first distance. The first-class keypoints include the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint.

[0108] Calculate the index difference between each first-class keypoint and its previous adjacent second keypoint as the second distance; in the original image, the last second distance is the difference between w*h and the index of the last second-class keypoint; w and h are the width and height of the original image, respectively.

[0109] To clearly explain the method for determining keypoint information under the custom shape described above, an exemplary calculation method for keypoint information is given below. In this calculation method, all pixels of the image are traversed in the order of the image array indices [0~w*h-1] of the target labeled image, and relevant processing is performed. Specifically, the processing includes the following steps:

[0110] Step 1: Traverse the pixels until the first pixel in the labeled graphic is found, and record the index1 of the first pixel in the labeled graphic, which is the position information of the initial pixel.

[0111] Step 2: Continue traversing until pixel index2 exceeds the range of the labeled graphic, and get len1 = index2 – index1, which is the first distance. index1 is the first pixel of the first type, and index2 is the first pixel of the second type.

[0112] Step 3: Continue traversing until the next pixel point index3 enters the labeled graphic, and get dist1 = index3 – index2, which is the first second distance, and index3 is the first type of pixel;

[0113] Step 4: Continue traversing until pixel index4 exceeds the range of the labeled graphic, and get len2 = index4 – index3, which is the second first distance, where index4 is a second type of pixel;

[0114] Step 5: Continue traversing until the next pixel (index5) enters the labeled graphic, and get dist2 = index5 – index4, which is the second second distance. Index5 is a first-class pixel.

[0115] Step 6: Repeat steps 2-5 until the maximum length of the image array is exceeded. Record the last second distance as dist(n) = w*h – index(2n). This calculation has a prerequisite: when drawing the annotation graphics, if the drawing area exceeds the original image, the boundary of the original image is used as the boundary of the annotation graphics. That is, the final generated annotation graphics must be within the range of the original image. This rule conforms to current restrictions on annotation graphics drawing.

[0116] Step 7: Generate key point information, including "point": "index1 len1 dist1 len2 dist2……len(n)dist(n)".

[0117] Step 203: The annotation client uses text to represent the shape information, key point information, and attribute information of the annotated graphics in step 202, forming the annotation information for image segmentation.

[0118] In step 202, the shape information, key point information, and attribute information of the labeled graphic have been determined to represent the graphic. Since the shape information, key point information, and attribute information can all be conveniently represented in text form, the entire labeled graphic can also be represented in text form. Various existing methods can be used to represent the shape information, key point information, and attribute information in text form, and this application does not limit this approach.

[0119] After representing the shape, key point, and attribute information of the labeled graphics in text form, this text-represented information can be organized into a specific text format file, such as a JSON file, to represent the annotation information for image segmentation.

[0120] The following uses a JSON file as an example to illustrate the representation of annotation information. A typical JSON file may include fields such as ID, description, imageWidth, and imageHeight. As an example, the description field of the JSON file can be used to carry annotation information. Specifically, a set of custom subfields can be set in the description for each type of annotation graphic to carry the shape information, key point information, and attribute information of the annotation graphic.

[0121] For example, for pen-type annotation graphics, the `type` subfield can carry the shape information "pencil", the `pixelData` subfield can carry key point information, and the `id`, `labelId`, `labelColor`, `labelName`, `grayLevel`, `globalCompositeOperation`, `singleChar`, `imageWidth`, and `imageHeight` subfields can carry various attribute information of the pen-type annotation graphics; for rectangle-type annotation graphics, the `type` subfield can carry the shape information "rect", the `pointCount` subfield can carry the number of key points in the key point information, the `points` subfield can carry the coordinates of the four corner points in the key point information, and the `id`, `labelId`, `labelColor`, `labelName`, `grayLevel`, `singleChar`, `imageWidth`, and `imageHeight` subfields can carry various attribute information of the rectangle-type annotation graphics; for polygon-type annotation graphics, the `type` subfield can carry the shape information "polygon", and the `pointCount` subfield can carry the key point information... The number of key points (i.e., the number of corner points) is set. The `points` subfield carries the coordinates of each corner point in the key point information. The `id`, `labelId`, `labelColor`, `labelName`, `grayLevel`, `singleChar`, `imageWidth`, and `imageHeight` subfields carry various attribute information of polygon-type annotation graphics. For circular, annular, or sector-type annotation graphics, the `type` subfield carries the shape information "circle". The `x` and `y` subfields carry the horizontal and vertical coordinates of the center pixel in the key point information, respectively. The `outerRadius` subfield carries the radius length of the circle or the outer radius length of the annular ring in the key point information. The `innerRadius` subfield carries the inner radius length of the annular ring in the key point information. The `startAngle` and `endAngle` subfields carry the start and end angles of the sector in the key point information, respectively. The `id`, `labelId`, `labelColor`, `labelName`, `grayLevel`, `singleChar`, `imageWidth`, and `imageHeight` subfields carry various attribute information of circular, annular, or sector-type annotation graphics.

[0122] After setting up the subfields as described in the example above, for each labeled graphic in the target labeled image, select the set of custom subfields mentioned above according to the category to which the labeled graphic belongs, and assign values ​​to the set of custom subfields based on the labeling information of the labeled graphic. In this way, the goal of representing the labeling information using a JSON file is achieved.

[0123] Step 204: The annotation client transmits the annotation information to the database for storage.

[0124] Through the aforementioned step 203, the image segmentation annotation information is organized in text form and uploaded to the database in the system. Since the annotation information is represented in text form, the amount of annotation information transmitted is greatly reduced.

[0125] Step 205: Obtain the annotation information for image segmentation of the target labeled image from the database.

[0126] The annotation information includes shape information, key point information, and attribute information of the annotation graphic represented by text. This annotation graphic is the annotation graphic that the user inputs in step 201 for image segmentation.

[0127] When assembling the annotation information in step 203, the specified file format is followed. Therefore, storing the annotation information in the database can be done by directly saving a file in the corresponding format, such as the aforementioned JSON file. When retrieving the annotation information in this step, the corresponding format file can be directly retrieved from the database, and the shape information, key point information, and attribute information of the annotated graphics can be extracted.

[0128] Step 206: Input the annotation information and the target annotated image into the model training algorithm to train the model on the target annotated image.

[0129] To train the model, the target labeled image and the annotation information obtained from image segmentation of the target labeled image are input into the model training algorithm. This allows for the training of various types of models as needed. Specifically, when inputting the annotation information into the model training algorithm, the annotation information can be directly input for model training. Alternatively, a mask image corresponding to the labeled image can first be generated based on the shape, key point, and attribute information of the labeled image included in the annotation information, and then the mask image can be input into the model training algorithm for model training.

[0130] This concludes the basic image segmentation-based model training method flow of this application. Steps 201-204 describe the specific processing methods on the annotation client side, while steps 205-206 describe the specific processing methods on the model trainer side. Through the processing and interaction on both sides, this application effectively reduces the amount of annotation information transmitted and improves transmission efficiency during image segmentation-based model training.

[0131] Based on the basic system structure and methodology described above, as mentioned earlier, the model training system may also include a labeling server to facilitate the processing of annotation information and the display of labeled graphics.

[0132] In step 204, the annotation client transmits the annotation information to the database for storage. Optionally, this can be implemented by first transmitting the annotation information to an annotation server, which then parses the annotation information into structured data and saves it to the database. To better facilitate data storage, querying, and retrieval, the annotation server can parse the annotation information according to the organization method in step 203 to obtain the shape information, key point information, and attribute information of the annotated graphic, and then organize this information into structured data and save it to the database. This ensures effective querying and retrieval of data in the database.

[0133] In addition, to further realize the display of labeled graphics for image segmentation, preferably, Figure 2 The process shown can further include the following steps:

[0134] Step 207: When it is necessary to display the annotated graphics, the annotation client obtains the annotation information.

[0135] The annotation client retrieves the image segmentation annotation information corresponding to the annotation graphic that needs to be displayed.

[0136] The annotation information can be obtained directly from the database, or, if the system includes an annotation server, it can be obtained from the database through the annotation server. The obtained annotation information is still represented in text form, thus effectively reducing the amount of annotation information transmitted and improving transmission efficiency.

[0137] Assuming the database stores annotation information in the aforementioned structured data manner, the specific processing when retrieving annotation information from the annotation server may include:

[0138] Step 207-1: When it is necessary to display the annotated graphics, the annotation client sends a request to the annotation server to obtain annotation information;

[0139] Step 207-2: After receiving the request, the annotation server retrieves the corresponding structured data from the database.

[0140] Step 207-3: The annotation server reassembles the structured data into annotation information and sends it to the annotation client.

[0141] After the annotation server retrieves the corresponding structured data from the database, it reassembles the structured data into annotation information in the manner described in step 206 and sends it to the annotation client. In this way, the annotation information transmitted between the annotation client and the annotation server is still represented in text form, which can effectively reduce the amount of annotation information transmitted.

[0142] Step 208: The annotation client draws and displays the annotation graphics based on the shape information, key point information, and attribute information of the annotation graphics included in the acquired annotation information.

[0143] The annotation client extracts the shape information, key point information, and attribute information of the annotation graphic from the annotation information. Based on this information, it parses the drawing information when the user inputs the annotation graphic, such as the four corner points of the rectangle, the corner points of the polygon, and the coordinates of all pixels of the pen. It then calls the client's local drawing API to draw the annotation graphic according to the example given in 201 and displays it.

[0144] Meanwhile, since the drawing information of the user-input annotation graphic can be obtained by parsing the shape information and key point information in the annotation information, the annotation graphic can be edited based on this information. For example, for a rectangular annotation graphic, the four sides can be dragged to adjust the width and height of the rectangle or the position of the rectangle can be adjusted by dragging the mouse; for a polygon, a corner point can be selected and dragged to adjust the position of the corner point, or the overall position of the polygon can be adjusted by dragging the mouse; for a pen, a pen graphic can be selected to continue drawing or to erase a part with the eraser. After completion, the pixels can be merged or deleted to generate a new pen graphic. After selecting multiple pen graphics, the pens can be merged or the merged pen graphics can be split.

[0145] In addition, when the system includes an annotation server, assuming that the database stores annotation information in the manner of the aforementioned structured data, the process of obtaining annotation information from the database in step 205 may specifically include: the annotation server obtaining structured data from the database based on the model training requirements and reorganizing it into annotation information.

[0146] More specifically, when training a network model, the annotation server can retrieve structured data related to the annotation information of all images to be trained from the database and reorganize the corresponding structured data into annotation information.

[0147] At this point, Figure 2The illustrated method flow is now complete. This method is described in this application from the perspective of the interaction between multiple components of an image segmentation-based model training system. Below, we summarize the processing methods of each component from its own perspective:

[0148] The client-side processing of annotations specifically includes:

[0149] The system receives user-input annotation graphics for image segmentation of the target labeled image; determines the shape, key point, and attribute information of the annotation graphics and represents them in text form to form the annotation information for image segmentation; and transmits the annotation information to the database for storage and use in model training on the target labeled image.

[0150] Furthermore, when it is necessary to display the labeled graphics, the labeling client can optionally obtain the labeling information; and then draw and display the labeled graphics based on the shape information, key point information and attribute information of the labeled graphics included in the obtained labeling information.

[0151] The processing of the model trainer specifically includes:

[0152] Retrieve annotation information for image segmentation of the target labeled image from the database;

[0153] The annotation information and the target annotated image are input into the model training algorithm to train the model using the target annotated image;

[0154] The annotation information includes shape information, key point information, and attribute information of the annotation graphics represented by text. The annotation graphics are used for image segmentation.

[0155] If the system includes a labeling server, then the processing by the labeling server can specifically include:

[0156] Receive image segmentation annotation information; wherein the annotation information includes shape information, key point information and attribute information of the annotation graphic represented by text, and the annotation graphic is used for image segmentation; after parsing the annotation information into structured data, save it into the database.

[0157] Furthermore, to enable the display of labeled graphics, when the labeling server receives a request to obtain labeling information, it retrieves structured data from the database; then, it reassembles the structured data into labeling information and sends it.

[0158] In addition, to meet the training requirements of neural networks, the annotation server can also obtain structured data from the database and reorganize it into annotation information. This annotation information can be directly used for model training of target annotated images. Alternatively, it can generate a mask map corresponding to the annotated graphic based on the shape information, key point information, and attribute information of the annotated graphic included in the annotation information, for use in model training.

[0159] The image segmentation-based model training method described in this application enables the generation of text-based annotation information by representing the labeled graphics used for image segmentation in text form, thereby effectively reducing the amount of data transmitted and stored. Furthermore, in the prior art, the method of transmitting annotation graphic information via mask images results in the loss of shape information such as rectangles and polygons. When the annotation information is displayed, the graphic cannot be edited. However, this application utilizes the shape information, key point information, and attribute information included in the annotation information to recover the drawing information when the user inputs the annotation graphic, thus enabling secondary editing of the annotation graphic. This includes adjusting the position, corner points, continuing to draw and erase the pen image, merging pens, and splitting pen images, solving the problems of not being able to distinguish annotation instances and being unable to edit the displayed annotation graphics. Furthermore, it can also effectively realize neural network training based on mask images.

[0160] As mentioned above, Figure 1 The system shown can be used for implementation Figure 2 The image segmentation-based model training method is shown. This application also provides... Figure 1 The annotation client and model trainer in the system shown. Figure 4 This is a basic structural diagram of the client-side interface. (Example:) Figure 4 As shown, the annotation client includes: a user interface unit, an annotation information generation unit, and a database interface unit.

[0161] The user interface unit is used to receive annotation graphics input by the user for image segmentation of the target annotation image;

[0162] The annotation information generation unit is used to determine the shape information, key point information and general attribute information of the annotation graphic, and represent them in text form to form the annotation information for image segmentation.

[0163] The database interface unit is used to transmit annotation information to the database for storage. The annotation information is used to train the model on the target annotated image.

[0164] Optionally, the annotation graphic received by the user interface unit is input by the user through a regular graphic, and the annotation information generation unit determines the shape information of the annotation graphic to be the shape information of the regular graphic indicated by the user.

[0165] The annotation graphic received by the user interface unit is input by the user through a custom method, and the annotation information generation unit determines the shape information of the annotation graphic to be a custom shape.

[0166] Optionally, when the user interface unit determines that the shape information of the regular graphic is a polygon, the annotation information generation unit determines the key point information, including the number of key points and the position information of all corner points;

[0167] And / or, when the user interface unit determines that the shape information of the regular graphic is a polygon, the annotation information generation unit determines the key point information, including the number of key points and the position information of all corner points;

[0168] And / or, when the user interface unit determines that the shape information of the regular graphic is a circle, the annotation information generation unit determines the key point information, including the center position information and the radius / diameter length information;

[0169] And / or, when the user interface unit determines that the shape information of the labeled graphic is a custom shape, the labeling information generation unit determines the key point information, including: initial key point position information, all first distances, and all second distances; wherein, in the labeling information generation unit, the methods for determining the initial key points, first distances, and second distances include:

[0170] For the original image before image segmentation, each pixel in the original image is traversed sequentially according to the image array index;

[0171] The first pixel located within the labeled graphic is taken as the initial key point;

[0172] The first pixel outside the labeled graphic after each first-class keypoint is taken as the second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint is calculated as the first distance. The first-class keypoint includes the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint.

[0173] Calculate the index difference between each first-class keypoint and its previous adjacent second keypoint as the second distance; in the original image, the last second distance is the difference between w*h and the index of the last second-class keypoint.

[0174] Optionally, the annotation client further includes a display unit, which is used to obtain annotation information when it is necessary to display the annotation graphic, and to draw and display the annotation graphic based on the shape information, key point information and general attribute information of the annotation graphic included in the obtained annotation information.

[0175] Optionally, the annotation information generation unit is further configured to perform annotation information generation according to a set file format.

[0176] In the database interface unit, the specific processing of transmitting annotation information to the database storage may include: parsing the annotation information into structured data and storing it in the database;

[0177] In the echo unit, the specific processing of obtaining annotation information may include: retrieving annotation information stored in the form of structured data from the database and reorganizing it according to the set file format.

[0178] Figure 5 A schematic diagram of the basic structure of the model trainer provided in this application. Figure 5 As shown, the model trainer includes: an acquisition unit, an input unit, and a training unit.

[0179] The acquisition unit is used to obtain the annotation information for image segmentation of the target labeled image from the database;

[0180] The input unit is used to input the annotation information and the target annotated image into the model training algorithm.

[0181] The training unit is used to train the model on the target labeled image;

[0182] The annotation information includes shape information, key point information, and attribute information of the annotation graphics represented by text. The annotation graphics are used for image segmentation.

[0183] Optionally, the database stores the structured data obtained after parsing the annotation information;

[0184] In the acquisition unit, the process of obtaining the annotation information for image segmentation of the target labeled image from the database may specifically include: obtaining structured data from the database and reorganizing it into annotation information.

[0185] Optionally, in the input unit, the processing of inputting labeled information into the model training algorithm may specifically include:

[0186] The shape, key point, and attribute information of the labeled graphic included in the annotation information can be input into the model training algorithm; or, based on the shape, key point, and attribute information of the labeled graphic included in the annotation information, a mask image corresponding to the labeled graphic can be generated and input into the model training algorithm.

[0187] Figure 6 This is a schematic diagram of the basic structure of the annotation server provided in this application. Figure 6 As shown, the annotation server includes: a client interface unit, a parsing unit, and a database interface unit.

[0188] The client interface unit is used to receive annotation information for image segmentation of the target labeled image; the annotation information includes shape information, key point information and general attribute information of the labeled graphic represented by text, and the labeled graphic is used for image segmentation; the parsing unit is used to parse the annotation information into structured data; and the database interface unit is used to save the structured data into the database.

[0189] Optionally, the client interface unit is further configured to receive a request for obtaining annotation information and notify the database interface unit;

[0190] The database interface unit is further used to retrieve structured data from the database after receiving a notification from the client interface unit;

[0191] The parsing unit is used to reorganize structured data into labeled information and send it through the client interface unit.

[0192] Optionally, the database interface unit can be used to obtain structured data from the database based on the training requirements of the neural network, and notify the parsing unit that the structured data is used for the training of the neural network.

[0193] The parsing unit can be further used to reorganize the structured data used for neural network training into labeled information, and generate a mask image corresponding to the labeled image based on the shape information, key point information and general attribute information of the labeled image included in the labeled information and send it out.

[0194] This application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can perform the steps in the image segmentation-based model training method described above. In practical applications, the computer-readable medium may be included in the devices / apparatus / systems of the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium stores instructions, which, when executed by a processor, can perform the steps in the image segmentation-based model training method described above.

[0195] According to the embodiments disclosed in this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, but not intended to limit the scope of protection of this application. In the embodiments disclosed in this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0196] Figure 7 An electronic device is also provided for this application. For example... Figure 7 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0197] The electronic device may include a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 702 is executed, an image segmentation-based model training method can be implemented.

[0198] Specifically, in practical applications, this electronic device may also include components such as a power supply 703 and an input / output unit 704. Those skilled in the art will understand that... Figure 7 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0199] The processor 701 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 702, and calling data stored in the memory 702, it performs various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0200] The memory 702 can be used to store software programs and modules, i.e., the aforementioned computer-readable storage medium. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on server usage, etc. Furthermore, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0201] The electronic device also includes a power supply 703 that supplies power to the various components. This power supply can be logically connected to the processor 701 via a power management system, enabling functions such as charging, discharging, and power consumption management. The power supply 703 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components.

[0202] The electronic device may also include an input / output unit 704, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit output 704 can also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A model training method based on image segmentation, characterized in that, include: Receives user-input labeled graphics used for image segmentation of the target labeled image; The shape information, key point information, and attribute information of the labeled graphic are determined and represented in text form to form the annotation information for image segmentation; The annotation information is transmitted to a database for storage, and the annotation information is used to train a model on the target annotated image. When the shape information of the labeled graphic is a custom shape, the key point information includes: initial key point position information, all first distances, and all second distances; wherein, the method for determining the initial key point, the first distance, and the second distance includes: For the original image before image segmentation, each pixel in the original image is traversed sequentially according to the image array index; The first pixel located within the labeled graphic is taken as the initial key point; The first pixel outside the labeled graphic after each first-class keypoint is designated as a second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint in the image array is calculated as the first distance. The first-class keypoints include the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint. Both the first-class keypoints and the second-class keypoints are pixels in the image array. Calculate the index difference between each first-class keypoint and its previous adjacent second keypoint in the image array as the second distance; in the original image, the last second distance is w. The difference between h and the index of the last second-class keypoint in the image array; where w and h are the width and height of the original image, respectively.

2. The method according to claim 1, characterized in that, When the user inputs the marked graphic, the user indicates that the input should be made in a regular graphic manner, and the shape information of the marked graphic is determined to be the shape information of the regular graphic indicated by the user. When the user inputs the labeled graphic, the user is instructed to input it using a custom graphic method, and the shape information of the labeled graphic is determined to be a custom shape.

3. The method according to claim 2, characterized in that, When the shape information of the labeled graphic is a rectangle, the key point information includes the number of key points and the position information of the four corner points; And / or, when the shape information of the labeled graphic is a polygon, the key point information includes the number of key points and the position information of all corner points; And / or, when the shape information of the labeled graphic is circular, the key point information includes the center position information and the radius / diameter length information.

4. The method according to claim 1, characterized in that, The method further includes: When it is necessary to display the labeled graphic, obtain the labeled information; Based on the shape information, key point information, and attribute information of the labeled graphic included in the acquired annotation information, the labeled graphic is drawn and displayed.

5. The method according to claim 4, characterized in that, The annotation information is constructed according to the set file format; The step of transmitting the annotation information to the database for storage includes: parsing the annotation information into structured data and storing it in the database; The step of obtaining the annotation information includes: obtaining the annotation information stored in the form of structured data from the database, and reorganizing it according to the file format.

6. A model training method based on image segmentation, characterized in that, include: Retrieve annotation information for image segmentation of the target labeled image from the database; The annotation information and the target annotated image are input into the model training algorithm to train the model using the target annotated image; The annotation information includes shape information, key point information, and attribute information of the annotation graphic represented by text, and the annotation graphic is used for image segmentation; When the shape information of the labeled graphic is a custom shape, the key point information includes: initial key point position information, all first distances, and all second distances; wherein, the method for determining the initial key point, the first distance, and the second distance includes: For the original image before image segmentation, each pixel in the original image is traversed sequentially according to the image array index; The first pixel located within the labeled graphic is taken as the initial key point; The first pixel outside the labeled graphic after each first-class keypoint is designated as a second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint in the image array is calculated as the first distance. The first-class keypoints include the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint. Both the first-class keypoints and the second-class keypoints are pixels in the image array. Calculate the index difference between each first-class keypoint and its previous adjacent second keypoint in the image array as the second distance; in the original image, the last second distance is w. The difference between h and the index of the last second-class keypoint in the image array; where w and h are the width and height of the original image, respectively.

7. The method according to claim 6, characterized in that, The database stores structured data obtained after parsing the annotation information; The step of obtaining the annotation information for image segmentation of the target annotated image from the database includes: The structured data is retrieved from the database and reorganized into the annotation information.

8. The method according to claim 6 or 7, characterized in that, Inputting the labeled information into the model training algorithm includes: The shape information, key point information, and attribute information of the labeled graphics included in the labeled information are input into the model training algorithm; or, Based on the shape information, key point information, and attribute information of the labeled graphic included in the annotation information, a mask image corresponding to the labeled graphic is generated and input into the model training algorithm.

9. An image segmentation annotation client, characterized in that, include: User interface unit, annotation information generation unit, and database interface unit; The user interface unit is used to receive annotation graphics input by the user for image segmentation of the target annotation image; The annotation information generation unit is used to determine the shape information, key point information and attribute information of the annotation graphic, and represent them in text form to form the annotation information of the image segmentation. The database interface unit is used to transmit the annotation information to the database for storage, and the annotation information is used to train the model on the target annotated image; When the user interface unit determines that the shape information of the labeled graphic is a custom shape, the labeling information generation unit determines the key point information, including: initial key point position information, all first distances, and all second distances; wherein, in the labeling information generation unit, the method for determining the initial key point, the first distance, and the second distance includes: For the original image before image segmentation, each pixel in the original image is traversed sequentially according to the image array index; The first pixel located within the labeled graphic is taken as the initial key point; The first pixel outside the labeled graphic after each first-class keypoint is designated as a second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint in the image array is calculated as the first distance. The first-class keypoints include the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint. Both the first-class keypoints and the second-class keypoints are pixels in the image array. Calculate the index difference between each first-class keypoint and its previous adjacent second keypoint in the image array as the second distance; in the original image, the last second distance is w. The difference between h and the index of the last second-class keypoint in the image array; where w and h are the width and height of the original image, respectively.

10. The annotation client according to claim 9, characterized in that, The annotation graphic received by the user interface unit is input by the user through a regular graphic, and the annotation information generation unit determines the shape information of the annotation graphic to be the shape information of the regular graphic indicated by the user. The annotation graphic received by the user interface unit is input by the user through a custom method, and the annotation information generation unit determines that the shape information of the annotation graphic is a custom shape; And / or, When the user interface unit determines that the shape information of the regular graphic is a polygon, the annotation information generation unit determines that the key point information includes the number of key points and the position information of all corner points; And / or, when the user interface unit determines that the shape information of the regular graphic is a polygon, the annotation information generation unit determines that the key point information includes the number of key points and the position information of all corner points; And / or, when the user interface unit determines that the shape information of the regular graphic is a circle, the annotation information generation unit determines that the key point information includes the center position information and the radius / diameter length information; And / or, The annotation client further includes a display unit, which is used to obtain the annotation information when the annotation graphic needs to be displayed, and is also used to draw and display the annotation graphic based on the shape information, key point information and attribute information of the annotation graphic included in the obtained annotation information; And / or, The annotation information generation unit is further configured to perform the process according to a set file format when composing the annotation information; In the database interface unit, transmitting the annotation information to the database storage includes: parsing the annotation information into structured data and storing it in the database; In the echo unit, obtaining the annotation information includes: obtaining the annotation information stored in the form of structured data from the database, and reorganizing it according to the file format.

11. A model trainer, characterized in that, include: Acquisition unit, input unit, and training unit; The acquisition unit is used to acquire annotation information for image segmentation of the target labeled image from the database; The input unit is used to input the annotation information and the target annotation image into the model training algorithm; The training unit is used to train a model on the target labeled image; The annotation information includes shape information, key point information, and attribute information of the annotation graphic represented by text, and the annotation graphic is used for image segmentation; When the shape information of the labeled graphic is determined to be a custom shape, the labeling information generation unit determines the key point information including: initial key point position information, all first distances, and all second distances; wherein, in the labeling information generation unit, the method for determining the initial key point, the first distance, and the second distance includes: For the original image before image segmentation, each pixel in the original image is traversed sequentially according to the image array index; The first pixel located within the labeled graphic is taken as the initial key point; The first pixel outside the labeled graphic after each first-class keypoint is designated as a second-class keypoint. The index difference between each second-class keypoint and its preceding adjacent first-class keypoint is calculated as the first distance. The first-class keypoints include the initial keypoint and the first pixel inside the labeled graphic after each second-class keypoint. Calculate the index difference between each first-class keypoint and its preceding adjacent second keypoint as the second distance; in the original image, the last second distance is w. The difference between h and the index of the last second-class keypoint.

12. The model trainer according to claim 11, characterized in that, The database stores structured data obtained after parsing the annotation information; In the acquisition unit, the step of acquiring the annotation information for image segmentation of the target labeled image from the database includes: acquiring the structured data from the database and reorganizing it into the annotation information; And / or, In the input unit, the step of inputting the annotation information into the model training algorithm includes: The shape information, key point information, and attribute information of the labeled graphic included in the annotation information are input into the model training algorithm; or, based on the shape information, key point information, and attribute information of the labeled graphic included in the annotation information, a mask image corresponding to the labeled graphic is generated and input into the model training algorithm.

13. A model training system based on image segmentation, characterized in that, Includes one or more annotation clients as described in claim 9 or 10, a model trainer as described in claim 11 or 12, and a database; The database is used to store annotation information sent by the annotation client.

14. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instructions are executed by the processor, they can implement the image segmentation-based model training method according to any one of claims 1 to 8.

15. An electronic device, characterized in that, The electronic device includes at least a computer-readable storage medium and a processor; The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the image segmentation-based model training method according to any one of claims 1 to 8.

Images