Interactive segmentation annotation method and apparatus, computer device, and storage medium

Abstract

The present application relates to an interactive segmentation annotation method and apparatus, a computer device, and a storage medium. The method comprises: in a box selection annotation mode, in response to a region box selection operation on an image to be segmented, determining an interactive segmentation box located on said image; determining an interactive segmentation region on the basis of the range covered by the interactive segmentation box on said image, the interactive segmentation region being a region where there is a need to perform segmentation annotation on said image; in response to interaction points annotated in the interactive segmentation region, determining interaction point type information and interaction point position information of the interaction points, the interaction point type information being used for representing whether the interaction points are a pixel points needing to be subjected to segmentation annotation; and on the basis of the interaction point type information and the interaction point position information, performing segmentation annotation on image content in the interactive segmentation region to obtain segmentation annotation information of the image to be segmented. By means of the present application, the efficiency of segmentation annotation can be improved.

Description

Interactive segmentation and annotation method, device, computer equipment and storage medium Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an interactive segmentation and annotation method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the rapid development of computer vision technology, image segmentation and annotation has become an essential step. Currently, image segmentation and annotation are mainly completed by manually clicking on each edge pixel in the image. However, in cases where the image has complex contours and many edge pixels, this segmentation and annotation method is very inefficient. Summary of the Invention

[0003] Based on this, it is necessary to provide an interactive segmentation and annotation method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above technical problems, which can improve the efficiency of segmentation and annotation.

[0004] In a first aspect, the present application provides an interactive segmentation and annotation method, comprising:

[0005] In the box selection and annotation mode, in response to the region box selection operation on the image to be segmented, an interactive segmentation box located on the image to be segmented is determined;

[0006] Determine the interactive segmentation area based on the range covered by the interactive segmentation frame on the image to be segmented; the interactive segmentation area is the area that needs to be segmented and annotated in the image to be segmented;

[0007] In response to the interaction point marked in the interaction segmentation area, the interaction point type information and interaction point location information of the interaction point are determined; the interaction point type information is used to indicate whether the interaction point is a pixel point that needs to be segmented and marked;

[0008] According to the interaction point type information and the interaction point position information, the image content of the interaction segmentation area is segmented and annotated to obtain segmentation annotation information of the image to be segmented.

[0009] In a second aspect, the present application provides an interactive segmentation and annotation device, comprising:

[0010] A frame selection and annotation module is used to determine an interactive segmentation frame located on the image to be segmented in response to a region frame selection operation on the image to be segmented in a frame selection and annotation mode;

[0011] The region determination module is used to determine the interactive segmentation region according to the range covered by the interactive segmentation frame on the image to be segmented; the interactive segmentation region is the region that needs to be segmented and annotated in the image to be segmented;

[0012] An information determination module is used to respond to interaction points marked in the interaction segmentation area and determine interaction point type information and interaction point location information of the interaction point; the interaction point type information is used to indicate whether the interaction point of the corresponding type is a pixel point that needs to be segmented and marked;

[0013] The segmentation and annotation module is used to segment and annotate the image content of the interactive segmentation area according to the interaction point type information and the interaction point position information, and obtain the segmentation and annotation information of the image to be segmented.

[0014] In a third aspect, the present application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method when executing the computer program.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0016] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps in the above method.

[0017] The interactive segmentation and annotation method, apparatus, computer device, computer-readable storage medium, and computer program product described above determine an interactive segmentation frame located on the image to be segmented in response to a region selection operation on the image to be segmented in a frame selection and annotation mode. Based on the range covered by the interactive segmentation frame on the image to be segmented, the interactive segmentation region of the image to be segmented is determined, thereby limiting the range of subsequent automatic image segmentation and annotation, thereby improving segmentation and annotation efficiency. Based on the interaction points annotated within the interactive segmentation region, interaction point type information and interaction point location information are determined. This allows for clear identification of which interaction points are pixels to be segmented and annotated, and which are pixels not to be segmented and annotated, providing prior information for subsequent automatic segmentation and annotation of the interactive segmentation region. This ensures the accuracy of the segmentation and annotation information obtained by segmenting and annotating the image content of the interactive segmentation region based on the interaction point type information and interaction point location information. In other words, in this application, users no longer need to annotate each pixel along the edge of the defect to be segmented. Automated segmentation and annotation can be achieved by simply performing the simple interactive operations of region selection and annotating interaction points, thereby improving segmentation and annotation efficiency. In addition, during automatic segmentation and annotation, by using the interaction point type information and interaction point location information as prior information for segmentation and annotation, the accuracy of segmentation and annotation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG1 is a flow chart of an interactive segmentation and annotation method provided in an embodiment of the present application;

[0019] FIG2 is a flow chart of another interactive segmentation and annotation method provided in an embodiment of the present application;

[0020] FIG3 is a structural block diagram of an interactive segmentation and annotation device provided in an embodiment of the present application;

[0021] FIG4 is a diagram showing the internal structure of a computer device provided in an embodiment of the present application;

[0022] FIG5 is a diagram showing the internal structure of another computer device provided in an embodiment of the present application;

[0023] FIG6 is a diagram showing the internal structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0025] In some embodiments, as shown in FIG1 , an interactive segmentation and annotation method is provided. This embodiment uses a terminal application as an example to illustrate the method. It is understood that the method can be implemented by the terminal alone or through interaction between the terminal and a server. In this embodiment, the method includes but is not limited to the following steps:

[0026] S202 : In a box selection and annotation mode, in response to a region box selection operation on the image to be segmented, determining an interactive segmentation box located on the image to be segmented.

[0027] The image to be segmented refers to an image that has defects and needs to have the defects segmented out.

[0028] The box selection annotation mode refers to a mode in which an interactive segmentation box is defined on the image to be segmented. The region selection operation refers to an operation in which an interactive segmentation box is defined on the image to be segmented to specify the interactive segmentation area. In some embodiments, the interactive segmentation box can be an irregular polygon or a regular polygon, such as a rectangle.

[0029] In some embodiments, the user can select an appropriate interactive segmentation frame on the image to be segmented, referring to the attribute parameters of the defect to be segmented in the image to be segmented. The defect to be segmented refers to a defect in the image to be segmented, and the attribute parameters of the defect to be segmented refer to parameters used to characterize the shape and size of the defect to be segmented.

[0030] Specifically, before segmenting and annotating the image to be segmented, the user can select an annotation mode. If the selected mode is the box selection annotation mode, the user can select a region in the image to be segmented based on the shape and size of the defect to be segmented. In response to the region selection operation, the terminal determines the interactive segmentation frame selected by the user on the image to be segmented.

[0031] S204 : Determine an interactive segmentation area according to a range covered by the interactive segmentation frame on the image to be segmented.

[0032] The interactive segmentation area is the area where the image to be segmented needs to be segmented and annotated.

[0033] Specifically, the terminal may directly determine the range currently covered by the interactive segmentation frame in the image to be segmented as the interactive segmentation area. The terminal may also determine the range covered by the interactive segmentation frame in the image to be segmented before it is moved, as well as the range covered by the interactive segmentation frame in the image to be segmented after each movement, as the interactive segmentation area.

[0034] S206 : In response to the interaction points marked in the interaction segmentation area, determine the interaction point type information and interaction point location information of the interaction point.

[0035] The interaction point refers to the pixel point annotated in the interactive segmentation area through the interactive operation. The interaction point type information is used to indicate whether the interaction point is a pixel point that needs to be segmented and annotated.

[0036] In some embodiments, the user may mark the corresponding interaction point by clicking or touching the interaction segmentation area displayed on the terminal interaction interface.

[0037] Specifically, after determining the interaction segmentation area, the user marks the interaction point on the interaction segmentation area, and the terminal obtains the interaction point type information and interaction point location information of the corresponding interaction point in response to the interaction point marked by the user on the interaction segmentation area.

[0038] S208 : Segment and annotate the image content of the interactive segmentation area according to the interaction point type information and the interaction point position information to obtain segmentation annotation information of the image to be segmented.

[0039] Segmentation and annotation refers to the process of segmenting and reasoning the image content of the interactive segmented region and annotating the segmentation reasoning results. It can be understood that segmentation reasoning is used to predict which parts of the interactive segmented region are defects.

[0040] Specifically, the terminal uses the interaction point type and location information as prior information for model reasoning. Using the image segmentation model, the terminal performs segmentation reasoning on the image content of the interaction segmentation area to obtain an image segmentation mask for the image to be segmented. The terminal then annotates the image segmentation mask to obtain segmentation annotation information for the image to be segmented.

[0041] It should be noted that an image segmentation model is a model used to predict the probability that each pixel in an interactive segmentation area is a defect. In the industrial field, image segmentation models are commonly used interactive deep learning models.

[0042] In some embodiments, the present application provides two pre-trained image segmentation models for users to choose from. These models are divided into precise mode and smooth mode based on their recognition accuracy and speed. Users can choose different annotation modes based on their computer configuration. Precise mode loads a large model, which has higher energy consumption and accuracy, while smooth mode loads a small model, which has lower energy consumption and accuracy.

[0043] As can be seen, in the embodiment of the present application, in the frame selection and annotation mode, in response to the region selection operation on the image to be segmented, an interactive segmentation frame is determined on the image to be segmented; based on the range covered by the interactive segmentation frame on the image to be segmented, the interactive segmentation region that needs to be segmented and annotated for the image to be segmented is determined to limit the range of subsequent automatic segmentation and annotation of the image, thereby improving the efficiency of segmentation and annotation. In response to the interactive points annotated in the interactive segmentation region, the interactive point type information and interactive point location information of the interactive points are determined. In this way, it is possible to clearly identify which interactive points are pixels that need to be segmented and annotated, and which interactive points are pixels that do not need to be segmented and annotated, providing prior information for the subsequent automatic segmentation and annotation of the interactive segmentation region. In this way, the accuracy of the segmentation and annotation information obtained by segmenting and annotating the image content of the interactive segmentation region based on the interactive point type information and interactive point location information can be guaranteed. In other words, in the present application, the user no longer needs to annotate each pixel along the edge of the defect to be segmented. Only the simple interactive operation of selecting the region and annotating the interactive points can achieve automatic segmentation and annotation, thereby improving the efficiency of segmentation and annotation. In addition, during automatic segmentation and annotation, by using the interaction point type information and interaction point location information as prior information for segmentation and annotation, the accuracy of segmentation and annotation can be improved.

[0044] In some embodiments, the interaction point type information includes a positive interaction point type and a negative interaction point type; the interaction point corresponding to the positive interaction point type is a pixel point that needs to be segmented and labeled; the interaction point corresponding to the negative interaction point type is a pixel point that does not need to be segmented and labeled; the interaction point location information includes positive interaction point coordinates and negative interaction point coordinates. Step 208 includes: cropping the image to be segmented based on multiple region coordinates used to represent the region range of the interaction segmentation frame to obtain an image of interest; performing segmentation reasoning on the interaction segmentation region based on the image of interest, the positive interaction point type, the positive interaction point coordinates, the negative interaction point type, and the negative interaction point coordinates to obtain an image segmentation mask for the image to be segmented; and annotating the image segmentation mask to obtain segmentation annotation information for the image to be segmented.

[0045] The image of interest refers to an image with the same size as the corresponding interactive segmentation region.

[0046] In some embodiments, the specific type of interaction point can be distinguished through different annotation operations. For example, if the user clicks the left button of the mouse, the pixel clicked by the left button is a positive interaction point; if the user clicks the right button of the mouse, the pixel clicked by the right button is a negative interaction point.

[0047] Specifically, the terminal determines multiple region coordinates representing the region extent of the interactive segmentation frame. If the interactive segmentation frame is a rectangular frame, the terminal may determine the pixel coordinates of the upper left and lower right vertices of the interactive segmentation frame to obtain the region coordinates, or determine the pixel coordinates of the lower left and upper right vertices of the interactive segmentation frame to obtain the region coordinates, or determine the pixel coordinates of any three or four vertices to obtain the region coordinates. The terminal locates the interactive segmentation frame in the image to be segmented based on the region coordinates and crops the interactive segmentation frame to obtain an image of interest. The terminal uses the positive interaction point type, positive interaction point coordinates, negative interaction point type, and negative interaction point coordinates as prior information for model inference and performs segmentation inference on the image content of the image of interest using an image segmentation model to obtain an image segmentation mask corresponding to each of the at least one interactive segmentation regions. Based on the image segmentation mask corresponding to each of the at least one interactive segmentation regions, a segmentation mask for the entire image to be segmented, i.e., the image segmentation mask of the image to be segmented, is determined. The terminal then annotates the image segmentation mask to obtain segmentation annotation information for the image to be segmented.

[0048] It can be seen that in this embodiment, the user marks the interaction points on the interactive segmentation area, providing the model with prior information on which pixels or the areas where the pixels are located are likely to be defects. Compared with the traditional technology of directly performing segmentation reasoning on the entire image, this can more effectively improve the accuracy of segmentation reasoning.

[0049] In some embodiments, segmentation reasoning of the image content of an image of interest includes three parts: pre-processing, model reasoning, and post-processing. Among them, the underlying pre-processing and post-processing are implemented in C++, and the Emscripten cross-compilation tool is used to compile the C++ code into Webassembly (Wasm), which can be run on a web page. The model reasoning part can be implemented by calling some suitable deep learning reasoning frameworks. Among them, C++ is a high-level computer programming language that is generated by the extension and upgrade of the C language. Emscripten is a compiler based on LLVM. Wasm is a binary format, and this binary format can run on a large number of operating systems and architectures.

[0050] In some embodiments, the pre-processing process includes: the terminal crops the interactive segmentation area according to the horizontal coordinate of the upper left vertex of the interactive segmentation area, the vertical coordinate of the upper left vertex, the width and length of the interactive segmentation area to obtain an image of interest. The terminal normalizes the image of interest, such as dividing each pixel value in the image of interest by 255 to obtain a normalized image. The terminal reduces the normalized image according to a preset scaling ratio to obtain a certain size, such as a reduced image of 256×256. The terminal converts the reduced image from HWC format to CHW format to obtain a format converted image. The terminal processes all the interaction point information to filter out all the interaction points within the area of ​​interest, and reduces them by the same scaling ratio as the normalized image to obtain new interaction points. It should be noted that the purpose of scaling is to improve the efficiency of model inference.

[0051] The HWC format arranges image dimensions in the order of height, width, and number of channels. For example, an RGB image with a shape of 256×256×3 is represented as [256, 256, 3] in the HWC format. The CHW format arranges image dimensions in the order of number of channels, height, and width. For example, an RGB image with a shape of 3×256×256 is represented as [3, 256, 256] in the CHW format. As you can see, the CHW format is commonly used to represent image dimensions in computer vision and deep learning.

[0052] In some embodiments, the model inference process includes: the terminal inputs the format-converted image and the new interaction point into the deep learning inference framework for segmentation inference, and outputs an image segmentation mask as large as the interaction segmentation area.

[0053] In some embodiments, the post-processing process includes: the terminal activating the image segmentation mask, i.e., performing sigmod processing, and amplifying the image segmentation mask according to a preset scaling ratio to obtain an amplified image. The amplified image is mapped to an original image mask having the same image size as the image to be segmented to obtain the image segmentation mask.

[0054] It should be noted that in this application, both pre-processing and post-processing can be encapsulated into interfaces and connected to the model inference part at the front end. In addition, in order to improve performance, the pre-processing and post-processing parts can be optimized, such as using the wasm vector instruction set to optimize them to run faster. Among them, the instruction set is a set of instructions used by the central processing unit (CPU) to calculate and control the computer system. The vector instruction set is a single instruction multiple data stream (SIMD), and one instruction can process multiple data at the same time. For example, the normalization step in the pre-processing is to divide each pixel value in the image by 255. After using the instruction set, the corresponding instructions can be used to process multiple data at once, that is, multiple pixel values ​​can be divided by 255 at one time.

[0055] It can be understood that this application realizes efficient segmentation reasoning using browser resources through the above method, ensuring accuracy and low latency, while providing services to an unlimited number of cloud users without generating additional resource consumption.

[0056] In some embodiments, the segmentation annotation information of the image to be segmented includes at least one target region and corresponding segmentation annotation information; the at least one target region is a region identified as a foreground in the image segmentation mask. The step of "annotating the image segmentation mask to obtain segmentation annotation information for the image to be segmented" includes: if the at least one target region is connected, generating a unified segmentation annotation information for the at least one target region; if the at least one target region is not connected, generating corresponding segmentation annotation information for each target region.

[0057] Specifically, after the segmentation is complete, the user can perform a labeling confirmation operation on the terminal interactive interface. The terminal will respond to the labeling confirmation operation and determine the one or more target areas identified as the foreground. If the target areas are connected, only one feature label is generated, namely the segmentation label information. If the target areas are not connected, multiple different labels with the same name are generated for each target area, and the corresponding segmentation label information is obtained.

[0058] In some embodiments, the user can press the enter key or click the "Confirm Marking" button on the terminal interaction interface to complete the marking confirmation operation.

[0059] It can be seen that in this embodiment, the terminal can perform targeted labeling processing according to the connectivity of the target area, and can be flexibly applied to different labeling occasions.

[0060] In some embodiments, the image segmentation mask includes a target region identified as a foreground. After the step of "performing segmentation reasoning on the interactive segmentation region based on the image of interest, the positive interaction point coordinates, and the negative interaction point coordinates to obtain an image segmentation mask for the image to be segmented," the interactive segmentation and annotation method of the present application further includes: in response to a region editing operation performed on the image segmentation mask, performing region editing processing on the target region in the image segmentation mask to obtain an adjusted image segmentation mask.

[0061] The region editing operation refers to the operation of filling or erasing the internal region.

[0062] Specifically, after obtaining the image segmentation mask of the image to be segmented, the terminal draws the image segmentation mask on the web page view, and the user determines whether the current image segmentation mask needs to be adjusted. If the user needs to adjust the current image segmentation mask, the target area in the image segmentation mask can be filled or erased. The terminal adjusts the target area in the image segmentation mask in response to the area filling or erasing operation to obtain the adjusted image segmentation mask.

[0063] It can be seen that in this embodiment, the user can improve the quality of the adjusted image segmentation mask by editing the automatically segmented image segmentation mask.

[0064] In some embodiments, if the user needs to supplement the image segmentation mask with a brush, the user can select a corresponding brush tool, determine the brush size and color, and use the set brush to paint the target area in the image segmentation mask to obtain an adjusted image segmentation mask.

[0065] In some embodiments, if the user needs to erase and correct the image segmentation mask, the user can select an eraser tool and determine the size of the eraser, and use the set eraser to erase the target area to obtain an adjusted image segmentation mask.

[0066] In some embodiments, the frame selection annotation mode includes a custom frame selection annotation mode and a fixed frame selection annotation mode. Step 202 includes at least one of the following:

[0067] In the custom box selection and annotation mode, in response to a custom region selection operation on the image to be segmented, a custom interactive segmentation box located on the image to be segmented is determined;

[0068] In the fixed frame selection and annotation mode, in response to a move operation performed on a pre-generated interactive segmentation frame, a moved interactive segmentation frame is determined.

[0069] It can be seen that in this embodiment, by setting different frame selection methods, different frame selection requirements can be met, thereby improving the flexibility of annotation.

[0070] In some embodiments, in the fixed frame selection and annotation mode, the terminal will automatically generate an interactive splitting frame of a fixed size, such as a 256×256 interactive splitting frame, and it will be semi-transparent by default and follow the mouse display. The user can move the fixed-size interactive splitting frame by moving the mouse.

[0071] In some embodiments, the step of “in a custom box selection and annotation mode, in response to a custom region selection operation on the image to be segmented, determining a custom interactive segmentation box located on the image to be segmented” includes at least one of the following:

[0072] In the custom frame selection and annotation mode, in response to at least one custom region frame selection operation performed on the image to be segmented, determining the interactive segmentation frame customized when the most recent custom frame selection operation was performed, so as to determine the range covered by the customized interactive segmentation frame as the interactive segmentation region to be segmented and annotated;

[0073] In the custom frame selection and annotation mode, in response to the trigger point input on the image to be segmented, the original custom interactive segmentation frame is controlled to move along with the trigger point, and the range covered by the original custom interactive segmentation frame and the interactive segmentation frame after each movement is determined as the interactive segmentation area that needs to be segmented and annotated.

[0074] The trigger point refers to a pixel point in the image to be segmented determined through interaction, such as clicking or touching.

[0075] In some embodiments, in custom selection mode, the user can uncheck "Follow Mouse." In this case, the user can directly select a custom interactive segmentation frame in the image to be segmented that covers the area where the defect to be segmented is located. If the user feels that the custom interactive segmentation frame is not in the right position, they can click again to move it or reselect the custom segmentation frame until they are satisfied.

[0076] In other embodiments, in the custom frame selection mode, the user can check the "Follow the mouse" option. This is because the image to be segmented may have defects with special shapes, such as slender defects. If the user directly selects a custom interactive segmentation frame in the image to be segmented that can cover the range of the defect to be segmented, the ratio of the custom interactive segmentation frame to the defect to be segmented will be large, thereby affecting the accuracy of the segmentation annotation. Taking the above situation into account, in this application, the user can first select an original custom interactive segmentation frame in the image to be segmented that can cover a part of the range of the defect to be segmented, and then enter a trigger point to move the original custom interactive segmentation frame. By continuously entering the trigger point, the custom interactive segmentation frame is continuously moved until the custom interactive segmentation frame before and after each move can cover the entire range of the defect to be segmented.

[0077] It can be seen that in this embodiment, the user can flexibly select any of the above custom frame selection modes according to the specific situation of the defect to determine the interactive segmentation area that needs to be segmented and labeled, thereby meeting different segmentation and labeling requirements.

[0078] In some embodiments, if the user checks the follow mouse option in the custom frame selection mode, after the mouse moves out of the custom interaction splitting frame, the front custom interaction splitting frame will follow the mouse and be displayed semi-transparently. After clicking, a custom interaction splitting frame of the same size can be generated again, and the target area in the custom interaction splitting frame can be automatically identified.

[0079] In some embodiments, if the user unchecks the "Follow Mouse" option in custom selection mode, after moving the mouse outside the custom interactive segmentation frame, the user needs to hold down the mouse to select a new interactive segmentation frame, clearing the previous interactive segmentation frame. The user then clicks and identifies the target area within the new interactive segmentation frame. Clicking outside the custom interactive segmentation frame will not produce any effect.

[0080] In some embodiments, the interactive segmentation and annotation method of the present application also includes: in the point selection and annotation mode, in response to the point selection and annotation operation on the defect to be segmented in the image to be segmented, determining multiple target interaction points located on the defect to be segmented; segmenting and annotating the image to be segmented according to the multiple target interaction points to obtain segmentation and annotation information of the image to be segmented.

[0081] Specifically, in point-and-click annotation mode, users can use their mouse to click on the image to be segmented, such as clicking on individual pixels on the defect to be segmented, to obtain target interaction points. The terminal then segments and annotates the image to be segmented based on the multiple interaction points to obtain segmentation annotation information for the image to be segmented.

[0082] In some embodiments, as shown in FIG2 , the interactive segmentation and annotation method of the present application specifically includes: starting quick annotation and determining the annotation mode. If you choose to click on the annotation, click on the target area. If you choose to select a fixed frame annotation, the fixed range box size will be displayed translucently following the mouse, and the target area will be selected. If you choose to customize the frame annotation, select the annotation range. If you then check to follow the mouse, the fixed range box size will be displayed translucently following the mouse and the target area will be selected. If you check not to follow the mouse, select the annotation range again and click on the target area. After clicking on the target area, confirm whether the brush needs to be supplemented. If the brush does not need to be supplemented, confirm whether it needs to be erased and corrected. If it does not need to be erased and corrected, confirm the annotation. If the brush needs to be supplemented, select the brush tool, select the brush size and color, and paint the target area. If the erase correction is needed, select the eraser tool, select the eraser size, and erase the target area. After confirming the annotation, if the mask is continuous, one feature and one annotation are generated, and the annotation is ended after the feature is selected. If the mask is not continuous, multiple annotations are generated for one feature, and the annotation is ended after the feature is selected.

[0083] In some embodiments, the interactive segmentation and annotation method of the present application further includes, but is not limited to, the following steps:

[0084] (1) Determine the annotation mode. In the point selection annotation mode, execute step (2). In the custom frame selection annotation mode, execute step (3) or (4). In the fixed frame selection annotation mode, execute step (5).

[0085] (2) In response to a point selection and annotation operation on a defect to be segmented in the image to be segmented, a plurality of target interaction points located on the defect to be segmented are determined; and the image to be segmented is segmented and annotated according to the plurality of target interaction points to obtain segmentation annotation information of the image to be segmented.

[0086] (3) In response to at least one custom region selection operation performed on the image to be segmented, determine the interactive segmentation frame customized when the most recent custom region selection operation is performed, determine the range covered by the customized interactive segmentation frame as the interactive segmentation region that needs to be segmented and labeled, and execute step (6).

[0087] (4) In response to a trigger point input on the image to be segmented, the original customized interactive segmentation frame is controlled to move along with the trigger point, and the range covered by the original customized interactive segmentation frame and the interactive segmentation frame after each movement is determined as the interactive segmentation area that needs to be segmented and labeled, and step (6) is executed.

[0088] (5) In response to the movement operation performed on the pre-generated interactive segmentation frame, the moved interactive segmentation frame is determined, and the range covered by the moved interactive segmentation frame is determined as the interactive segmentation area that needs to be segmented and labeled, and step (6) is executed.

[0089] (6) In response to the interaction points marked in the interaction segmentation area, determine the interaction point type information and interaction point location information of the interaction point.

[0090] (7) The image to be segmented is cropped according to the multiple region coordinates used to represent the region range of the interactive segmentation frame to obtain an image of interest.

[0091] (8) Based on the image of interest, the coordinates of the positive interaction points, and the coordinates of the negative interaction points, perform segmentation reasoning on the interactive segmentation region to obtain the image segmentation mask of the image to be segmented. If the image segmentation mask needs to be adjusted, execute step (9). If the image segmentation region mask does not need to be adjusted, execute step (10).

[0092] (9) In response to the region editing operation performed on the image segmentation mask, a region editing process is performed on the target region in the image segmentation mask to obtain an adjusted image segmentation mask, and step (10) is executed.

[0093] (10) When at least one target region is connected, a segmentation annotation information is uniformly generated for the at least one target region; when at least one target region is not connected, segmentation annotation information corresponding to each target region is generated separately.

[0094] It should be noted that this application proposes a fast annotation tool that can effectively solve the current problem of labeling complex contours, which requires constant delineation of the target area, which is time-consuming and laborious. It makes segmentation annotation faster and more accurate, providing a more efficient annotation method for subsequent annotation. In particular, the combination of a brush and eraser covers a wider range of annotation scenarios and has greater applicability.

[0095] It should be understood that, although the steps in the flowcharts of the above-mentioned embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0096] Based on the same inventive concept, the present application also provides an interactive segmentation and annotation device. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more of the following embodiments of the interactive segmentation and annotation device can be found in the aforementioned limitations of the interactive segmentation and annotation method, and will not be further elaborated here.

[0097] As shown in FIG3 , an embodiment of the present application provides an interactive segmentation and labeling apparatus 300 , including:

[0098] A frame selection and annotation module 302 is configured to determine an interactive segmentation frame located on the image to be segmented in response to a frame selection operation on a region of the image to be segmented in a frame selection and annotation mode;

[0099] The region determination module 304 is used to determine the interactive segmentation region based on the range covered by the interactive segmentation frame on the image to be segmented; the interactive segmentation region is the region that needs to be segmented and annotated in the image to be segmented;

[0100] The information determination module 306 is used to determine the interaction point type information and interaction point location information of the interaction point in response to the interaction point marked in the interaction segmentation area; the interaction point type information is used to indicate whether the interaction point of the corresponding type is a pixel point that needs to be segmented and marked;

[0101] The segmentation and annotation module 308 is used to segment and annotate the image content of the interactive segmentation area according to the interactive point type information and the interactive point position information, and obtain segmentation and annotation information of the image to be segmented.

[0102] The interactive segmentation and annotation device 300, in response to a region selection operation on an image to be segmented in a selection and annotation mode, determines an interactive segmentation frame located on the image to be segmented. Based on the range covered by the interactive segmentation frame on the image to be segmented, it determines an interactive segmentation region for the image to be segmented and annotated, thereby limiting the range of subsequent automatic image segmentation and annotation, thereby improving segmentation and annotation efficiency. In response to interaction points annotated within the interactive segmentation region, it determines interaction point type information and interaction point location information for the interaction points. This ensures that which interaction points are pixels that need to be segmented and annotated and which are not, providing prior information for subsequent automatic segmentation and annotation of the interactive segmentation region. This ensures the accuracy of the segmentation and annotation information obtained by segmenting and annotating the image content of the interactive segmentation region based on the interaction point type information and interaction point location information. In other words, in this application, users no longer need to annotate each pixel along the edge of the defect to be segmented. Instead, they only need to perform the simple interactive operations of selecting a region and annotating interaction points to achieve automated segmentation and annotation, thereby improving segmentation and annotation efficiency. In addition, during automatic segmentation and annotation, by using the interaction point type information and interaction point location information as prior information for segmentation and annotation, the accuracy of segmentation and annotation can be improved.

[0103] In some embodiments, the interaction point type information includes a positive interaction point type and a negative interaction point type; the interaction point corresponding to the positive interaction point type is a pixel point that needs to be segmented and annotated; the interaction point corresponding to the negative interaction point type is a pixel point that does not need to be segmented and annotated; the interaction point location information includes positive interaction point coordinates and negative interaction point coordinates. In terms of segmenting and annotating the image content of the interactive segmentation area according to the interaction point type information and the interaction point location information to obtain the segmentation annotation information of the image to be segmented, the segmentation annotation module 308 is specifically used to: crop the image to be segmented according to multiple area coordinates used to characterize the area range of the interactive segmentation frame to obtain an image of interest; perform segmentation reasoning on the interactive segmentation area according to the image of interest, the positive interaction point type, the positive interaction point coordinates, the negative interaction point type and the negative interaction point coordinates to obtain an image segmentation mask of the image to be segmented; and annotate the image segmentation mask to obtain the segmentation annotation information of the image to be segmented.

[0104] In some embodiments, the segmentation annotation information for the image to be segmented includes at least one target region and corresponding segmentation annotation information; the at least one target region is the region identified as the foreground in the image segmentation mask. In annotating the image segmentation mask to obtain the segmentation annotation information for the image to be segmented, the segmentation annotation module 308 is specifically configured to: if at least one target region is connected, generate a unified segmentation annotation for the at least one target region; if at least one target region is not connected, generate corresponding segmentation annotation information for each target region.

[0105] In some embodiments, the image segmentation mask includes a target region identified as a foreground. The interactive segmentation and annotation apparatus 300 of the present application further includes a region editing module, which is specifically configured to: in response to a region editing operation performed on the image segmentation mask, perform region editing processing on the target region in the image segmentation mask to obtain an adjusted image segmentation mask.

[0106] In some embodiments, the frame selection and annotation mode includes a custom frame selection and annotation mode and a fixed frame selection and annotation mode. In terms of determining an interactive segmentation frame located on the image to be segmented in response to a frame selection operation on a region of the image to be segmented, the frame selection and annotation module 302 is specifically configured to: in the custom frame selection and annotation mode, determine a custom interactive segmentation frame located on the image to be segmented in response to a custom frame selection operation on the image to be segmented; and in the fixed frame selection and annotation mode, determine a moved interactive segmentation frame in response to a move operation on a pre-generated interactive segmentation frame.

[0107] In some embodiments, in response to a custom area frame selection operation performed on the image to be segmented, determining a custom interactive segmentation frame located on the image to be segmented, the frame selection and annotation module 302 is specifically used to: in a custom frame selection and annotation mode, in response to at least one custom area frame selection operation performed on the image to be segmented, determine the custom interactive segmentation frame when the most recent custom frame selection operation is performed, so as to determine the range covered by the custom interactive segmentation frame as the interactive segmentation area that needs to be segmented and annotated; in a custom frame selection and annotation mode, in response to a trigger point input on the image to be segmented, control the original custom interactive segmentation frame to move along with the trigger point, and determine the range covered by the original custom interactive segmentation frame and the interactive segmentation frame after each movement as the interactive segmentation area that needs to be segmented and annotated.

[0108] In some embodiments, the interactive segmentation and annotation device 300 of the present application also includes a point selection and annotation module, which is specifically used to: in the point selection and annotation mode, in response to the point selection and annotation operation performed on the defect to be segmented in the image to be segmented, determine multiple target interaction points located on the defect to be segmented; segment and annotate the image to be segmented according to the multiple target interaction points to obtain segmentation and annotation information of the image to be segmented.

[0109] Each module in the interactive segmentation and annotation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0110] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 4. The computer device includes a processor, memory, an input / output (I / O) interface, and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data related to interactive segmentation and annotation. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements the steps of the interactive segmentation and annotation method described above.

[0111] In some embodiments, a computer device is provided, which may be a terminal. Its internal structure may be as shown in Figure 5. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, while the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and computer program stored in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication, which may be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements the steps of the interactive segmentation and annotation method described above. The display unit of the computer device is used to produce a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0112] Those skilled in the art will understand that the structure shown in Figure 4 or Figure 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0113] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0114] In some embodiments, an internal structure diagram of a computer-readable storage medium is provided as shown in FIG6 . The computer-readable storage medium stores a computer program that implements the steps in the above-mentioned method embodiments when executed by a processor.

[0115] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0116] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0117] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0118] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An interactive segmentation and annotation method, characterized in that: include: In the box selection and annotation mode, in response to a region box selection operation on the image to be segmented, determining an interactive segmentation box located on the image to be segmented; Determine an interactive segmentation region according to the range covered by the interactive segmentation frame on the image to be segmented; the interactive segmentation region is a region where the image to be segmented needs to be segmented and annotated; In response to the interaction point marked in the interaction segmentation area, determining the interaction point type information and the interaction point location information of the interaction point; The interaction point type information is used to indicate whether the interaction point is a pixel point that needs to be segmented and labeled; The image content of the interactive segmentation area is segmented and annotated according to the interaction point type information and the interaction point position information to obtain segmentation and annotation information of the image to be segmented.

2. The method according to claim 1, characterized in that The interaction point type information includes a positive interaction point type and a negative interaction point type; the interaction point corresponding to the positive interaction point type is a pixel point that needs to be segmented and marked; The interaction points corresponding to the negative interaction point type are pixel points that do not need to be segmented and marked; The interaction point location information includes positive interaction point coordinates and negative interaction point coordinates; segmenting and annotating the image content of the interaction segmentation area according to the interaction point type information and the interaction point location information to obtain segmentation and annotation information of the image to be segmented, including: cropping the image to be segmented according to multiple area coordinates used to characterize the area range of the interaction segmentation frame to obtain an image of interest; performing segmentation reasoning on the interaction segmentation area according to the image of interest, the positive interaction point type, the positive interaction point coordinates, the negative interaction point type and the negative interaction point coordinates to obtain an image segmentation mask of the image to be segmented; annotating the image segmentation mask to obtain segmentation and annotation information of the image to be segmented.

3. The method according to claim 2, characterized in that The segmentation annotation information of the image to be segmented includes at least one target area and corresponding segmentation annotation information; the at least one target area is an area identified as a foreground in the image segmentation mask; The labeling process is performed on the image segmentation mask to obtain the segmentation labeling information of the image to be segmented, including: when the at least one target area is connected, uniformly generating a segmentation labeling information for the at least one target area; In the case that the at least one target region is not connected, segmentation annotation information corresponding to each target region is generated respectively.

4. The method according to claim 2, characterized in that: After performing segmentation reasoning on the interactive segmentation area according to the image of interest, the positive interaction point coordinates and the negative interaction point coordinates to obtain the image segmentation mask of the image to be segmented, the method further includes: in response to the region editing operation performed on the image segmentation mask, performing region editing processing on the target region in the image segmentation mask to obtain an adjusted image segmentation mask.

5. The method according to claim 1, characterized in that The box selection and annotation mode includes a custom box selection and annotation mode and a fixed box selection and annotation mode; in the box selection and annotation mode, in response to the area box selection operation on the image to be segmented, determining the interactive segmentation box located on the image to be segmented includes at least one of the following: In the custom box selection and annotation mode, in response to a custom area box selection operation performed on the image to be segmented, a custom interactive segmentation box located on the image to be segmented is determined; In the fixed frame selection and labeling mode, in response to a move operation performed on a pre-generated interactive segmentation frame, a moved interactive segmentation frame is determined.

6. The method according to claim 5, characterized in that In the custom box selection and annotation mode, in response to a custom area box selection operation performed on the image to be segmented, determining a custom interactive segmentation box located on the image to be segmented includes at least one of the following: In the custom box selection and annotation mode, in response to at least one custom area box selection operation performed on the image to be segmented, determining an interactive segmentation box that was customized when the most recent custom box selection operation was performed, so as to determine a range covered by the custom interactive segmentation box as an interactive segmentation area that needs to be segmented and annotated; In the custom box selection and annotation mode, in response to a trigger point input on the image to be segmented, the original custom interactive segmentation frame is controlled to move along with the trigger point, and the range covered by the original custom interactive segmentation frame and the interactive segmentation frame after each movement is determined as the interactive segmentation area that needs to be segmented and labeled.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In the point selection and marking mode, in response to a point selection and marking operation performed on a defect to be segmented in the image to be segmented, a plurality of target interaction points located on the defect to be segmented are determined; The image to be segmented is segmented and annotated according to the multiple target interaction points to obtain segmentation and annotation information of the image to be segmented.

8. An interactive segmentation and labeling device, characterized in that: include: A box selection and annotation module is used to determine an interactive segmentation box located on the image to be segmented in response to a region box selection operation performed on the image to be segmented in a box selection and annotation mode; A region determination module, used to determine an interactive segmentation region according to the range covered by the interactive segmentation frame on the image to be segmented; the interactive segmentation region is a region where the image to be segmented needs to be segmented and marked; An information determination module, configured to respond to the interaction points marked in the interaction segmentation area and determine the interaction point type information and interaction point location information of the interaction points; The interaction point type information is used to indicate whether the interaction point of the corresponding type is a pixel point that needs to be segmented and labeled; The segmentation and annotation module is used to segment and annotate the image content of the interactive segmentation area according to the interactive point type information and the interactive point position information to obtain segmentation and annotation information of the image to be segmented.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images