Image annotation method, device, equipment and storage medium

Abstract

One or more embodiments of the present application propose an image annotation method, apparatus, device, and storage medium. The method may include obtaining an annotation operation performed by a user on an image to be annotated. The contour lines drawn by the annotation operation form a connected area. Based on the annotation operation, the image to be annotated is segmented to obtain a predicted segmentation result of the image to be annotated. Based on the predicted segmentation result of the image to be annotated, an annotation result of the image to be annotated is obtained.

Description

Technical Field

[0001] The present application relates to computer technology, and specifically to image annotation methods, devices, equipment and storage media. Background Art

[0002] With the advancement of science and technology, deep learning technology has developed rapidly. In order to facilitate the training of deep learning models, data needs to be labeled.

[0003] Currently, in image annotation related technologies, users usually manually annotate images, which is very inefficient and has high labor costs. Summary of the invention

[0004] In view of this, the present application discloses at least one image annotation method. The method comprises: obtaining an annotation operation performed by a user on an image to be annotated; wherein the contour lines drawn by the annotation operation form a connected area; based on the annotation operation, segmenting the image to be annotated to obtain a predicted segmentation result of the image to be annotated; and obtaining an annotation result of the image to be annotated based on the predicted segmentation result of the image to be annotated.

[0005] In some of the illustrated embodiments, the contour lines drawn by the above-mentioned annotation operation form a connected area in the above-mentioned image to be annotated; or, the contour lines drawn by the above-mentioned annotation operation and the boundary of the above-mentioned image to be annotated jointly form a connected area.

[0006] In some of the illustrated embodiments, the above-mentioned image to be labeled is segmented based on the above-mentioned labeling operation to obtain a predicted segmentation result of the above-mentioned image to be labeled, including: based on the above-mentioned labeling operation, the above-mentioned image to be labeled is preliminarily segmented to obtain a preliminary segmentation result; wherein the above-mentioned preliminary segmentation result includes an initial foreground area, an initial background area and an initial edge area between the above-mentioned initial foreground area and the above-mentioned initial background area of ​​the above-mentioned image to be labeled; based on the above-mentioned preliminary segmentation result, the above-mentioned image to be labeled is predictively segmented to obtain a predicted segmentation result of the above-mentioned image to be labeled.

[0007] In some of the illustrated embodiments, the image to be annotated is preliminarily segmented based on the annotation operation, including: displaying the first area and the second area into which the image to be annotated is divided by the contour lines outlined by the annotation operation; obtaining the user's selection operation; wherein the selection operation is used to select the initial foreground area and / or the initial background area from the first area and the second area.

[0008] In some of the illustrated embodiments, the image to be annotated is preliminarily segmented based on the annotation operation, including: determining the contour line or the contour line and the vicinity of the contour line as the initial edge area between the initial foreground area and the initial background area.

[0009] In some of the illustrated embodiments, based on the preliminary segmentation result, the image to be labeled is predictively segmented to obtain the predicted segmentation result of the image to be labeled, including: predictively segmenting the preliminary segmentation result and the image to be labeled through an image segmentation model to obtain the predicted segmentation result; or, inputting the preliminary segmentation map corresponding to the preliminary segmentation result and the image to be labeled into the image segmentation model for processing to obtain the predicted segmentation result of the image to be labeled.

[0010] In some of the illustrated embodiments, the preliminary segmentation map corresponding to the preliminary segmentation result and the image to be labeled are input into the image segmentation model for processing to obtain a predicted segmentation result of the image to be labeled, including: encoding the image to be labeled and the preliminary segmentation map corresponding to the preliminary segmentation result to obtain a coded feature map; decoding the coded feature map to obtain a predicted segmentation result of the image to be labeled.

[0011] In some of the illustrated embodiments, the encoding process of the image to be labeled and the preliminary segmentation result to obtain a coded feature map includes: fusing the image to be labeled and the preliminary segmentation map corresponding to the preliminary segmentation result to obtain a fused image; encoding the fused image to obtain a coded feature map.

[0012] In some of the illustrated embodiments, the above-mentioned fused image is encoded to obtain a coded feature map, including: processing the above-mentioned fused image separately by multiple parallel dilated convolution branches included in the image segmentation model to obtain processing results corresponding to the above-mentioned multiple dilated convolution branches; wherein at least some of the multiple dilated convolution branches correspond to different dilated rates; and fusing the processing results corresponding to the multiple dilated convolution branches to obtain the above-mentioned coded feature map.

[0013] In some of the illustrated embodiments, the method further includes: before encoding the fused image, reducing the fused image by a preset multiple to obtain the reduced fused image; encoding the fused image to obtain a coding feature map includes: encoding the reduced fused image to obtain a coding feature map.

[0014] In some of the illustrated embodiments, the above-mentioned encoded feature map is decoded to obtain the predicted segmentation result of the above-mentioned image to be labeled, including: fusing the intermediate feature map generated in the above-mentioned encoding process with the above-mentioned encoded feature map to obtain a fused feature map; decoding the above-mentioned fused feature map to obtain the predicted segmentation result of the above-mentioned image to be labeled.

[0015] In some of the illustrated embodiments, the annotation result of the image to be annotated is obtained based on the predicted segmentation result of the image to be annotated, including: using the predicted segmentation result as the annotation result of the image to be annotated; or, according to the preliminary segmentation result, correcting the predicted segmentation result to obtain the annotation result of the image to be annotated.

[0016] In some of the illustrated embodiments, the predicted segmentation results are corrected based on the preliminary segmentation results to obtain the annotation results of the image to be annotated, including: when the first pixel point included in the initial foreground area does not belong to the foreground pixel point in the predicted segmentation result, the first pixel point is corrected to a foreground pixel point; and / or, when the second pixel point included in the initial background area does not belong to the background pixel point in the predicted segmentation result, the second pixel point is corrected to a background pixel point.

[0017] In some of the illustrated embodiments, the method further includes: displaying the annotation result; and adjusting the annotation result according to an adjustment operation performed by a user on the displayed annotation result to obtain a final annotation result of the image to be annotated.

[0018] In some of the illustrated embodiments, the displaying of the annotation results includes: displaying the annotation results in the image to be annotated, or, respectively displaying the image to be annotated and the annotation results in different display areas of a display interface, so that a user can view the difference between the annotation results and the target object in the image to be annotated.

[0019] In some of the illustrated embodiments, displaying the annotation results in the image to be annotated includes: displaying the annotation results as a new layer overlay at a corresponding position of the layer where the image to be annotated is located; or, rendering different areas of the image to be annotated based on the annotation results to obtain and display rendering results.

[0020] In some of the illustrated embodiments, the method further includes: before obtaining the annotation operation performed by the user on the image to be annotated, displaying multiple brush configuration options for the user to select; wherein the multiple brush configuration options correspond to brush lines of different widths.

[0021] The present application also proposes an image annotation device, which includes: an acquisition module, used to obtain the annotation operation performed by a user on the image to be annotated; wherein the contour lines outlined by the above annotation operation form a connected area; a segmentation processing module, used to segment the above image to be annotated based on the above annotation operation to obtain a predicted segmentation result of the above image to be annotated; and a annotation result obtaining module, used to obtain the annotation result of the above image to be annotated based on the above predicted segmentation result of the above image to be annotated.

[0022] In some of the illustrated embodiments, the contour lines drawn by the above-mentioned annotation operation form a connected area in the above-mentioned image to be annotated; or, the contour lines drawn by the above-mentioned annotation operation and the boundary of the above-mentioned image to be annotated jointly form a connected area.

[0023] In some of the illustrated embodiments, the segmentation processing module includes: a preliminary segmentation module, used to perform preliminary segmentation on the image to be labeled based on the labeling operation to obtain a preliminary segmentation result; wherein the preliminary segmentation result includes an initial foreground area, an initial background area, and an initial edge area between the initial foreground area and the initial background area of ​​the image to be labeled; and a predictive segmentation module, used to perform predictive segmentation on the image to be labeled based on the preliminary segmentation result to obtain a predictive segmentation result of the image to be labeled.

[0024] In some of the illustrated embodiments, the preliminary segmentation module includes: a first display module for displaying the first area and the second area into which the image to be annotated is divided by the contour lines outlined by the annotation operation; an acquisition module for acquiring the user's selection operation; wherein the selection operation is used to select the initial foreground area and / or the initial background area from the first area and the second area.

[0025] In some of the illustrated embodiments, the preliminary segmentation module includes: a determination module that determines the contour line or the contour line and the vicinity of the contour line as an initial edge region between the initial foreground region and the initial background region.

[0026] In some of the illustrated embodiments, the above-mentioned prediction segmentation module includes: a first prediction segmentation submodule, which is used to predict the segmentation of the above-mentioned preliminary segmentation result and the above-mentioned image to be labeled through an image segmentation model to obtain a predicted segmentation result; or, a second prediction segmentation submodule, which is used to input the preliminary segmentation map corresponding to the above-mentioned preliminary segmentation result and the above-mentioned image to be labeled into the image segmentation model for processing to obtain the predicted segmentation result of the above-mentioned image to be labeled.

[0027] In some of the illustrated embodiments, the second prediction segmentation submodule includes: an encoding processing module, used to encode the preliminary segmentation map corresponding to the image to be labeled and the preliminary segmentation result to obtain a coding feature map; a decoding module, used to decode the coding feature map to obtain the prediction segmentation result of the image to be labeled.

[0028] In some of the illustrated embodiments, the above-mentioned encoding processing module includes: a first fusion module, used to fuse the above-mentioned image to be labeled with the preliminary segmentation map corresponding to the above-mentioned preliminary segmentation result to obtain a fused image; an encoding processing sub-module, used to encode the above-mentioned fused image to obtain a coding feature map.

[0029] In some of the illustrated embodiments, the encoding processing submodule is used to process the fused image respectively through multiple parallel dilated convolution branches included in the image segmentation model to obtain processing results corresponding to the multiple dilated convolution branches respectively; wherein at least some of the multiple dilated convolution branches correspond to different dilated rates; and the processing results corresponding to the multiple dilated convolution branches are fused to obtain the encoding feature map.

[0030] In some of the illustrated embodiments, the apparatus further includes: an image reduction module, for reducing the fused image by a preset multiple before encoding the fused image, to obtain the reduced fused image; and the encoding processing submodule, for encoding the reduced fused image, to obtain an encoding feature map.

[0031] In some of the illustrated embodiments, the above-mentioned decoding processing module includes: a second fusion module, which is used to fuse the intermediate feature map generated in the above-mentioned encoding process with the above-mentioned encoding feature map to obtain a fused feature map; a decoding processing sub-module, which decodes the above-mentioned fused feature map to obtain the predicted segmentation result of the above-mentioned image to be labeled.

[0032] In some of the illustrated embodiments, the above-mentioned annotation result obtaining module includes: a first obtaining sub-module, used to use the above-mentioned predicted segmentation result as the annotation result of the above-mentioned image to be annotated; or, a second obtaining sub-module, used to correct the above-mentioned predicted segmentation result according to the above-mentioned preliminary segmentation result to obtain the annotation result of the above-mentioned image to be annotated.

[0033] In some of the illustrated embodiments, the second obtaining submodule includes: a first correction module, used to correct the first pixel point included in the initial foreground area to a foreground pixel point when the first pixel point included in the initial foreground area does not belong to the foreground pixel point in the predicted segmentation result; and / or, a second correction module, used to correct the second pixel point included in the initial background area to a background pixel point when the second pixel point included in the initial background area does not belong to the background pixel point in the predicted segmentation result.

[0034] In some of the illustrated embodiments, the apparatus further includes: a second display module for displaying the annotation results; and an adjustment module for adjusting the annotation results according to adjustment operations performed by a user on the displayed annotation results to obtain the final annotation results of the image to be annotated.

[0035] In some of the illustrated embodiments, the second display module further includes: a first display submodule, used to display the annotation result in the image to be annotated, or a second display submodule, used to respectively display the image to be annotated and the annotation result in different display areas of the display interface, so that the user can view the difference between the annotation result and the target object in the image to be annotated.

[0036] In some of the illustrated embodiments, the first display submodule is used to: display the annotation results as a new layer overlay at a corresponding position of the layer where the image to be annotated is located; or, based on the annotation results, render different areas of the image to be annotated to obtain and display the rendering results.

[0037] In some of the illustrated embodiments, the apparatus further includes: a brush configuration option display module, configured to display a plurality of brush configuration options for the user to select before obtaining the annotation operation performed by the user on the image to be annotated; wherein the plurality of brush configuration options correspond to brush lines of different widths.

[0038] The present application also proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image annotation method shown in any of the above embodiments.

[0039] The present application also provides an electronic device, the device comprising: a processor;

[0040] A memory for storing the processor executable instructions; wherein the processor is configured to call the executable instructions stored in the memory to implement the image annotation method shown in any of the above embodiments.

[0041] It can be seen from the above technical solution that since the above image annotation system can generate a predicted segmentation result map corresponding to the above object to be annotated based on the annotation operation performed by the user on the image to be annotated, and then obtain the predicted segmentation result of the above image to be annotated according to the adjustment made by the user on the above predicted segmentation result map, and then obtain the annotation result of the above image to be annotated based on the above predicted segmentation result of the above image to be annotated, the image annotation system can assist manual annotation, reduce user workload, improve image annotation efficiency, and reduce manual operation costs.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A method flow chart of an image annotation method shown in this application;

[0045] Figure 2 A schematic diagram of a preliminary segmentation result diagram shown in this application;

[0046] Figure 3 A schematic diagram of a preliminary segmentation result diagram shown in this application;

[0047] Figure 4 A schematic diagram of an image including a first area and a second area shown in the present application;

[0048] Figure 5 This is a schematic diagram of an image after the user completes the selection operation shown in this application;

[0049] Figure 6 A schematic diagram of a predicted segmentation result diagram shown in this application;

[0050] Figure 7 A schematic diagram of a labeling result graph corresponding to the labeling result shown in this application;

[0051] Figure 8 A method flow chart of an image segmentation method shown in this application;

[0052] Fig. 9 A schematic diagram of a flow chart of an image segmentation method shown in this application;

[0053] Fig.10 A schematic diagram of a flow chart of an image segmentation method shown in this application;

[0054] Fig.11 A schematic diagram of the structure of a spatial pyramid pooling layer shown in this application;

[0055] Fig.12 A schematic diagram of the structure of a coding unit shown in this application;

[0056] Fig.13 A schematic diagram of the structure of a decoding unit shown in this application;

[0057] Fig.14 This is a schematic diagram of the structure of an image annotation device shown in this application;

[0058] Fig.15 This is a schematic diagram of the structure of an electronic device shown in this application. DETAILED DESCRIPTION

[0059] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0060] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "above", and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0061] With the advancement of science and technology, deep learning technology has developed rapidly. In order to facilitate the training of deep learning models, data needs to be labeled.

[0062] Currently, in image annotation related technologies, users usually manually annotate images, which is very inefficient and has high labor costs.

[0063] For example, in image segmentation tasks, users usually use polygon annotation tools to annotate images. During the annotation process, users usually need to use polygon annotation tools to mark a large number of points on the edge of the object to be segmented to form a polygonal area to segment the object.

[0064] In practical applications, the edges of the object to be segmented are usually tortuous. Therefore, it takes a lot of time for users to mark a large number of points to depict the edges of the object to be segmented in order to segment a complete image of the object to be segmented.

[0065] For another example, in a panoramic segmentation task, since the image includes many objects to be labeled, it takes several hours for the user to complete the labeling of many labeled objects. It can be seen that the current efficiency of manual labeling is very low.

[0066] Based on this, the present application proposes an image annotation method. The method enables the system to obtain the predicted segmentation result of the image to be annotated based on the annotation operation performed by the user on the image to be annotated, and then obtain the annotation result of the image to be annotated based on the predicted segmentation result of the image to be annotated, thereby assisting manual annotation through the image annotation system, reducing the user's workload, improving image annotation efficiency, and reducing manual operation costs.

[0067] See also Figure 1 , Figure 1 This is a flow chart of an image annotation method shown in this application.

[0068] like Figure 1 As shown, the above method may include:

[0069] S102, obtaining a labeling operation performed by a user on the image to be labeled; wherein the contour lines drawn by the labeling operation form a connected area;

[0070] S104, based on the above-mentioned labeling operation, performing segmentation processing on the above-mentioned image to be labeled to obtain a predicted segmentation result of the above-mentioned image to be labeled;

[0071] S106: Obtain a labeling result of the image to be labeled based on the predicted segmentation result of the image to be labeled.

[0072] The image annotation system (hereinafter referred to as the "system") may be a software system equipped with an auxiliary annotation algorithm. Based on the algorithm, the system may perform predictive segmentation processing on the image to be annotated based on the annotation operation performed by the user on the image to be annotated, and obtain a predictive segmentation result of the image to be annotated.

[0073] The above system can usually be installed in a hardware device, and the auxiliary image annotation is completed through the computing power provided by the hardware environment installed in the hardware device.

[0074] In some embodiments, the system may be installed on a terminal device. For example, the system may be installed on a terminal device in the form of a client. The system may interact with a user through an interactive module (e.g., an interactive interface) provided by the terminal device to obtain the user's annotation operation on the image to be annotated, thereby completing auxiliary image annotation.

[0075] In some embodiments, the above system can be mounted on a server (including a server cluster or a cloud server). For example, the above system can be mounted on a server in the form of a platform. When executing the above S102, the terminal device corresponding to the above server can interact with the user through the interaction module (for example, an interactive interface) provided by itself, so that the user can perform annotation operations on the image to be annotated. After the above terminal device obtains the above annotation operation, the above terminal device can send the data related to the above annotation operation to the above server. The above system can complete the auxiliary image annotation based on the obtained above annotation operation.

[0076] It can be understood that if the above system is installed on a server, the work tasks of interacting with the user can be carried out in the terminal device corresponding to the above server. When performing auxiliary annotation operations, the above server can obtain user operation data (including any user operation, such as adjustment operations, annotation operations) from the terminal device corresponding to the above server, and complete the above steps S102-S106 based on the obtained data.

[0077] The following description will be given by taking the above system installed in a terminal device as an example.

[0078] The above-mentioned terminal device or server device can be an electronic device such as a laptop computer, a computer terminal, a touch screen computer, a PAD terminal or other portable devices, which is not particularly limited in this application. The above-mentioned image to be annotated is specifically an image that needs to be annotated with data. The above-mentioned image to be annotated can usually include one or more objects to be annotated. When annotating an image to be annotated containing multiple objects to be annotated, image annotation can be performed for a target object to be annotated in the image to be annotated.

[0079] The above-mentioned object to be annotated is specifically an object in the image to be annotated that needs to be annotated by the user. For example, in an autonomous driving scenario, the above-mentioned object to be annotated may be a road that appears in the image. For another example, in a face recognition scenario, the above-mentioned object to be annotated may be the corresponding face area in the image.

[0080] It should be noted that the objects to be annotated are not fixed and can change according to the actual business scenario. For example, in the autonomous driving scenario, the object to be annotated in image A can be an obstacle or a road. In the face recognition scenario, the object to be annotated in image A can be a face.

[0081] The above-mentioned annotation operation is specifically an operation performed by the user on the image to be annotated.

[0082] In practical applications, the system can display the image to be annotated to the user through the interactive module (such as a terminal display). The user can perform the annotation operation in the image to be annotated according to actual annotation requirements, so as to outline the contour of the object to be annotated through the annotation operation, and the contour forms a connected area.

[0083] For example, in a face recognition scenario, the above annotation requirement is an annotation operation for a face to be annotated. At this time, the user can perform an annotation operation according to the edge of the face to be annotated in the image to be annotated displayed by the above system to outline the contour of the face.

[0084] The above-mentioned connected area may refer to an area formed by the contour line drawn by the user for the object to be annotated in the image to be annotated, wherein the connected area may be formed by the contour line itself or by the contour line and other lines in the image to be annotated.

[0085] For example, when the object to be labeled is a person, the connected area may be an area formed by a contour line drawn by a user for the person.

[0086] It should be noted that in different situations, the user needs to perform different annotation operations. For example, when part of the edge lines of the object to be annotated coincides with the boundary of the image to be annotated, the user can outline the part of the edge lines of the object to be annotated, so that the outlines outlined by the annotation operation and the boundary of the image to be annotated together form a connected area. For another example, when the object to be annotated does not include any edge lines that coincide with the boundary of the image to be annotated, the user can outline all the edge lines of the object to be annotated, so that the outlines outlined by the annotation operation form a connected area in the image to be annotated.

[0087] In some embodiments, when performing an image annotation operation, the system may display the image to be annotated to the user.

[0088] For example, the system may display the image to be annotated through a display interface.

[0089] After displaying the image to be annotated, the system may wait for the user to perform an annotation operation on the image to be annotated, and form a connected area through the contour lines drawn by the annotation operation.

[0090] For example, the above system may provide tools such as a brush for users to perform annotation operations.

[0091] Then in S102, the terminal device may provide the user with tools for annotation operation through the interaction module, such as a brush tool, etc. The user may use the tool to outline the edge line of the object to be annotated.

[0092] The system may determine the contour line drawn by the user using a brush according to an image scanning algorithm, and then the system may determine the connected area according to the contour line.

[0093] Optionally, the system may display the contour line drawn by the user to the user in real time in the image to be annotated.

[0094] After determining the connected region, the system may execute S104 to segment the image to be labeled based on the labeling operation to obtain a predicted segmentation result of the image to be labeled.

[0095] For example, the system may be equipped with an image segmentation algorithm. After determining the connected area, the system may perform predictive segmentation processing on the image to be labeled according to the image segmentation algorithm equipped therewith to obtain a predictive segmentation result of the image to be labeled.

[0096] After obtaining the predicted segmentation result, the system may execute S106 to obtain a labeling result of the image to be labeled based on the predicted segmentation result of the image to be labeled.

[0097] In some cases, the image may use the predicted segmentation result as the labeling result of the image to be labeled.

[0098] In some cases, the system performs one or more processing on the predicted segmentation result to obtain the labeling result of the image to be labeled. For example, the predicted segmentation result may be fine-tuned or corrected to obtain the labeling result of the image to be labeled.

[0099] It can be seen from the above technical solution that since the above image annotation system can generate a predicted segmentation result map corresponding to the above object to be annotated based on the annotation operation performed by the user on the image to be annotated, and then obtain the predicted segmentation result of the above image to be annotated according to the adjustment made by the user on the above predicted segmentation result map, and then obtain the annotation result of the above image to be annotated based on the above predicted segmentation result of the above image to be annotated, the image annotation system can assist manual annotation, reduce user workload, improve image annotation efficiency, and reduce manual operation costs.

[0100] In some embodiments, in order to improve the accuracy of image annotation, after obtaining the annotation result of the image to be annotated based on the predicted segmentation result of the image to be annotated, the system can also display the image annotation result to the user so that the user can make adjustments according to the displayed annotation result, thereby obtaining a more accurate image annotation result.

[0101] In practical applications, after obtaining the annotation result of the image to be annotated, the system may display the annotation result, and then adjust the annotation result according to the adjustment operation performed by the user on the displayed annotation result to obtain the final annotation result of the image to be annotated.

[0102] For example, after obtaining the annotation result of the image to be annotated, the system displays the annotation result to the user and provides the user with tools for adding, deleting, and repairing the image (e.g., polygonal area adding, deleting, and repairing tools, brushes, and erasers, etc.) for adjustment operations. The user can select various tools to adjust the area to be pre-annotated, so that the system can adjust the annotation result according to the user's adjustment data to obtain the final annotation result of the image to be annotated.

[0103] In some embodiments, after obtaining the annotation result of the image to be annotated based on the predicted segmentation result of the image to be annotated, the system can also display the image annotation result to the user so that the user can make adjustments based on the displayed annotation result, thereby obtaining a more accurate image annotation result.

[0104] In some embodiments of the present application, based on the user's annotation operation on the image to be annotated, a preliminary segmentation result can be obtained, wherein the preliminary segmentation can divide the image to be annotated into a foreground area, a background area and an edge area, and the image segmentation algorithm is used to predict the multiple pixels contained in the edge area to obtain a prediction result that each pixel in the multiple pixels belongs to the foreground area or the background area. Compared with the object contour outlined by the user's annotation operation, the image segmentation algorithm can more accurately predict the area to which the pixels in the edge area belong, thereby obtaining a more accurate image annotation result, reducing the annotation requirements of the annotator, and thus helping to improve the annotation efficiency.

[0105] In some embodiments, the auxiliary annotation algorithm used in the above-mentioned image annotation system may be the predictive segmentation algorithm proposed in this application.

[0106] The above-mentioned predictive segmentation algorithm refers to performing preliminary segmentation on the image to be labeled based on the labeling operation performed by the user on the image to be labeled to obtain a preliminary segmentation result. After obtaining the above-mentioned preliminary segmentation result, the image to be labeled is predicted and segmented by the image segmentation model based on the above-mentioned preliminary segmentation result to obtain the predicted segmentation result of the image to be labeled.

[0107] The preliminary segmentation result may include an initial foreground area, an initial background area, and an initial edge area between the initial foreground area and the initial background area of ​​the image to be labeled. The initial foreground area and the initial background area may be parts outside the contour line, and the contour line or the contour line and the area near the contour line constitute the initial edge area. In some embodiments, the initial foreground area and the initial background area are determined based on the connected area.

[0108] For example, the connected area is determined as the initial foreground area, and the area outside the connected area and the contour line is determined as the initial background area. For another example, the initial foreground area and the initial background area can be determined in a user-specified manner, but the embodiment of the present application does not limit this.

[0109] In some embodiments, the preliminary segmentation result may be displayed, for example, the preliminary segmentation result may be displayed in the image to be annotated.

[0110] In some embodiments, in order to improve user experience, in the preliminary segmentation result map corresponding to the preliminary segmentation result, the three regions can be represented by three colors. Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a preliminary segmentation result diagram shown in this application. Figure 2 As shown, the white area is the initial foreground area, the black area is the initial background area, and the gray area is the edge line area.

[0111] Here, it should be noted that the above preliminary segmentation result diagram is only for illustration. In actual situations, the display colors of the initial foreground area, the initial background area and the edge line area in the above preliminary segmentation result diagram can be set according to actual situations and are not limited here.

[0112] Specifically, when annotating an image to be annotated, the system may display the image to be annotated to the user.

[0113] See also Figure 3 , Figure 3 This is a schematic diagram of an image to be annotated according to the present application.

[0114] like Figure 3 As shown, the system can display the image to be annotated to the user through the interactive module (e.g., display interface). In the embodiment shown in the present application, it is assumed that the person in the image to be annotated is the object to be annotated, that is, the current annotation task is to annotate the person in the annotated image.

[0115] It should be noted that the above Figure 3 The above-mentioned interaction module is not displayed, and this application does not specifically limit the page elements of the above-mentioned interaction module.

[0116] After obtaining the image to be annotated, the user may perform an annotation operation on the image to be annotated.

[0117] exist Figure 3 In the example shown, when marking the connected domain, the user can use tools such as a brush provided by the system to roughly outline the contour line corresponding to the edge line of the object to be labeled in the image to be labeled.

[0118] In some cases, the object to be annotated is inside the image to be annotated (i.e., the object to be annotated does not include any edge line that coincides with the boundary of the image to be annotated). In this case, when outlining the contour line, the user can outline all the edge lines of the object to be annotated, so that the contour lines outlined by the above annotation operation form a connected area in the image to be annotated.

[0119] In some cases, when part of the edge line of the object to be annotated coincides with the boundary of the image to be annotated, when outlining the contour line, the user can outline the above part of the edge line of the object to be annotated, so that the contour line outlined by the above annotation operation and the boundary of the image to be annotated together form a connected area, thereby reducing the workload of the user in outlining the edge line and improving the efficiency of image annotation.

[0120] After the user completes the labeling operation (ie, completes the contour line outlining and forms a connected area), the system may execute S104 to segment the image to be labeled based on the labeling operation to obtain a predicted segmentation result of the image to be labeled.

[0121] In practical applications, the system may first perform preliminary segmentation on the image to be annotated based on the annotation operation to obtain a preliminary segmentation result, wherein the preliminary segmentation result includes an initial foreground area, an initial background area, and an initial edge area between the initial foreground area and the initial background area of ​​the image to be annotated.

[0122] For example, the system may execute a connected region search algorithm based on the contour lines drawn by the annotation operation and the connected regions, thereby obtaining a preliminary segmentation result corresponding to the image to be annotated.

[0123] Specifically, the above system can, by default, determine the area inside the connected area as the initial foreground area included in the above preliminary segmentation result, determine the area outside the connected area as the initial background area included in the above preliminary segmentation result, and determine the contour line outlined by the above labeling operation as the initial edge area between the above initial foreground area and the above initial background area.

[0124] It should be noted that in order to improve the accuracy of image annotation, in some embodiments, when the above-mentioned preliminary segmentation result is obtained according to the connected domain algorithm, the above-mentioned system may determine the above-mentioned contour line and the surrounding area of ​​the above-mentioned contour line as the initial edge area between the above-mentioned initial foreground area and the above-mentioned initial background area.

[0125] In some embodiments, in addition to the contour line as the initial edge region, the area near the contour line is also used as the initial edge region, which provides more useful information for subsequent image segmentation based on the initial edge region, thereby improving image annotation accuracy.

[0126] In some embodiments, in order to accurately locate the initial foreground area included in the preliminary segmentation result, the system may provide the user with an interactive process for determining the initial foreground area.

[0127] In practical applications, the system may display the first area and the second area into which the image to be annotated is divided by the contour line outlined by the annotation operation, wherein the first area and the second area may be the portion without the contour line or without the contour line and the area near the contour line. Then, the system may wait for the user to perform a selection operation on the first area and / or the second area. After the user completes the selection operation, the system may obtain the user's selection operation. The selection operation is used to select the initial foreground area and / or the initial background area from the first area and the second area to determine the initial foreground area and the initial background area included in the preliminary segmentation result.

[0128] See also Figure 4 , Figure 4 This is a schematic diagram of an image including a first area and a second area shown in the present application.

[0129] like Figure 4 As shown in the figure, the black line area is the connected area formed by the contour line drawn by the user for the image to be annotated. It should be noted that the display color and shape of the above edge line can be set according to actual needs and are not limited here. The inner area of ​​the above connected area can be the first area, and the outer area of ​​the above connected area can be the second area.

[0130] It should be noted that the first region and the second region are a pair of relative regions corresponding to the connected regions one by one, that is, when the user annotates multiple connected regions, the image to be annotated can be divided into multiple first regions and multiple second regions. Among the connected regions included in the image to be annotated, a connected region uniquely corresponding to any pair of the first region and the second region can always be found.

[0131] When showing users Figure 4 After viewing the image shown, the user may perform a selection operation. For example, the user may select the first area as the foreground area and / or the second area as the background area by clicking.

[0132] When the user has completed the selection, the system may use the foreground area and / or background area selected by the user as the foreground area included in the preliminary segmentation result.

[0133] It should be noted that, on the one hand, the above system can remind the user of the operations to be performed during the interaction with the user. On the other hand, the present application does not specifically limit the way in which the user selects the foreground. For example, in addition to the above examples, the user can also select the foreground by drawing a line or a circle.

[0134] See also Figure 5 , Figure 5This is a schematic diagram of an image after the user completes the selection operation shown in this application.

[0135] like Figure 5 As shown, the user clicks inside the connected area (the first area), so the system can determine the internal area of ​​the connected area (the dark gray area inside the frame surrounded by the black edge line) as the initial foreground area included in the preliminary segmentation result, and display the image selected by the user for the user to determine whether the foreground is selected correctly. It should be noted that the method of displaying the foreground selected by the user can be set according to the actual situation and is not limited here.

[0136] Since the user can select the foreground included in the preliminary segmentation result in the above intermediate image, the above system can accurately locate the foreground of the preliminary segmentation result according to the user's actual intention.

[0137] After obtaining the preliminary segmentation result, the system may perform predictive segmentation on the image to be labeled based on the preliminary segmentation result to obtain a predictive segmentation result of the image to be labeled.

[0138] In some embodiments, a predicted segmentation result may be obtained through an image segmentation model based on the preliminary segmentation result and the image to be annotated.

[0139] In some embodiments, the predicted segmentation result may be obtained through an image segmentation model based on the image to be labeled and information of a plurality of pixels belonging to the preliminary edge region in the preliminary segmentation result.

[0140] In some embodiments, a predicted segmentation result may be obtained through an image segmentation model based on the position information of each region in the image to be labeled and the preliminary segmentation result.

[0141] In some embodiments, the predicted segmentation result can be obtained through an image segmentation model based on the preliminary segmentation map corresponding to the preliminary segmentation result and the image to be annotated.

[0142] Among them, the above-mentioned predicted segmentation result can be expressed in a variety of forms, for example, it can be expressed in the form of a predicted segmentation image. For another example, different identifiers can be corresponded to the predicted segmentation results through the pre-set labeling rules, so as to express the above-mentioned predicted segmentation result in the form of identifiers. It can be understood that the scheme of expressing the above-mentioned predicted segmentation result by means of identifiers can refer to the scheme of expressing the above-mentioned predicted segmentation result in the form of a predicted segmentation result map. In this application, the above-mentioned predicted segmentation result is expressed in the form of a predicted segmentation result map as an example for explanation.

[0143] In some embodiments, the system may perform predictive segmentation on the preliminary segmentation result and the image to be labeled through an image segmentation model to obtain a predictive segmentation result; or,

[0144] The preliminary segmentation map corresponding to the preliminary segmentation result and the image to be labeled are input into the image segmentation model for processing to obtain the predicted segmentation result of the image to be labeled.

[0145] Among them, the above-mentioned image segmentation model can reasonably divide the above-mentioned initial edge line area into the foreground area and the background area based on the initial background area, the initial foreground area, and the initial edge line area information between the above-mentioned initial background area and the above-mentioned initial foreground area provided by the above-mentioned preliminary segmentation result.

[0146] After the initial edge line area is reasonably divided into a foreground area and a background area, the system may determine the foreground area reasonably divided from the initial edge and the initial foreground area included in the preliminary segmentation result as a pre-annotated area corresponding to the object to be annotated.

[0147] After determining the pre-annotated area, the system may generate a predicted segmentation result map based on the pre-annotated area and the image to be annotated to obtain a predicted segmentation result.

[0148] See also Figure 6 , Figure 6 This is a schematic diagram of the predicted segmentation result diagram shown in this application.

[0149] like Figure 6 As shown, the above-mentioned predicted segmentation result diagram displays the pre-annotated area corresponding to the object to be annotated obtained by the above-mentioned image segmentation algorithm based on the image to be annotated. The dark gray area (the area where the person is located) in the figure is the above-mentioned pre-annotated area. It should be noted that the way of displaying the above-mentioned pre-annotated area can be set according to the actual situation and is not limited here.

[0150] After obtaining the predicted segmentation result image, the system may obtain a labeling result of the image to be labeled based on the predicted segmentation result of the image to be labeled.

[0151] It can be seen from the above technical solution that the above system performs preliminary segmentation on the image to be annotated based on the annotation operation performed by the user on the image to be annotated, and obtains a preliminary segmentation result; wherein the above preliminary segmentation result includes the initial foreground area, the initial background area, and the initial edge area between the initial foreground area and the initial background area of ​​the image to be annotated. Then, based on the above preliminary segmentation result, the above system performs predictive segmentation on the image to be annotated, and obtains the predicted segmentation result of the image to be annotated. After obtaining the predicted segmentation result, the above system obtains the annotation result of the image to be annotated based on the predicted segmentation result of the image to be annotated.

[0152] Since the above-mentioned annotation results are obtained based on the above-mentioned preliminary segmentation results, and the above-mentioned preliminary segmentation results are determined based on the contour lines outlined by the user's annotation operation on the object to be annotated, and the above-mentioned contour lines, compared with the four extreme points in the extreme point annotation algorithm and the graffiti line segments in the graffiti annotation algorithm, can input more information useful for determining the preliminary segmentation results, therefore, the obtained predicted segmentation result map is more in line with the actual object to be annotated, thereby prompting the image annotation accuracy, reducing the user's subsequent adjustment workload, and improving the image annotation efficiency.

[0153] In some embodiments, in order to improve image segmentation accuracy, when obtaining the annotation result of the image to be annotated based on the predicted segmentation result of the image to be annotated, the predicted segmentation result can be corrected according to the preliminary segmentation result to obtain the annotation result of the image to be annotated.

[0154] In some embodiments, since the initial foreground area and the initial background area included in the preliminary segmentation results are relatively accurate, and the image segmentation model is mainly used to more accurately classify the pixels in the initial edge area, the above-mentioned predicted segmentation results can be corrected according to the initial foreground area and the initial background area included in the preliminary segmentation results to obtain a more accurate segmentation result.

[0155] In some embodiments, pixel points in the initial foreground area and the initial background area included in the preliminary segmentation result whose areas in the predicted segmentation result are inconsistent with the preliminary segmentation result can be corrected so that the area they belong to is the same as the preliminary segmentation result, thereby improving the image segmentation accuracy.

[0156] In some embodiments, when a first pixel point included in the initial foreground area does not belong to a foreground pixel point in the predicted segmentation result, the first pixel point may be corrected to be a foreground pixel point.

[0157] In some embodiments, when a second pixel point included in the initial background area does not belong to a background pixel point in the predicted segmentation result, the second pixel point may be corrected to a background pixel point.

[0158] The first pixel point may be any pixel point in the initial foreground area. In practical applications, the pixels in the initial foreground area may be used as the first pixel points for determination one by one.

[0159] The second pixel point may be any pixel point in the initial background area. In practical applications, the pixels in the initial background area may be used as the second pixel points for determination one by one.

[0160] Since the preliminary segmentation result includes relatively accurate initial foreground areas and initial background areas, a more accurate predicted segmentation result can be obtained by correcting the foreground areas and background areas included in the above predicted segmentation result according to the initial foreground areas and initial background areas included in the preliminary segmentation result.

[0161] In some embodiments, in order to simplify the workload of the user in outlining the edge line of the object to be annotated, the fit between the predicted segmentation result image and the object to be annotated is improved.

[0162] Before the user performs a marking operation, the system may provide the user with a brush line whose width is not less than a preset threshold.

[0163] The above-mentioned preset threshold value may be a value set by the developer according to the actual situation (eg, 10 mm), and is not particularly limited here.

[0164] When a thicker brush line is used for annotation operations, the time for the user to aim at the edge of the object to be annotated can be simplified, thereby simplifying the workload of the user to outline the edge line of the object to be annotated, and improving the fit between the predicted segmentation result map and the above-mentioned object to be annotated.

[0165] In some embodiments, the above system may provide brushes of various widths for users to choose according to actual situations, thereby improving user experience and system applicability.

[0166] In actual applications, before obtaining the annotation operation performed by the user on the image to be annotated, multiple brush configuration options are displayed for the user to select; wherein the multiple brush configuration options correspond to brush lines of different widths.

[0167] Since the edge line drawn by the brush is wider, the user does not need to be cautious when outlining the edge line of the object to be labeled. The user only needs to cover the actual edge line of the object to be labeled with the edge line drawn by the brush, which simplifies the user's workload of outlining the edge line of the object to be labeled.

[0168] The user can select a wider brush line from a variety of brush options provided by the above system for annotation operations.

[0169] Since the above-mentioned edge lines are relatively wide, the edge lines outlined by the user can easily cover the actual edge lines of the above-mentioned objects to be labeled. When generating pre-labeling results, it is convenient to accurately divide the edge lines between the foreground and the background included in the preliminary segmentation results into foreground or background, thereby improving the user experience and system applicability.

[0170] In some embodiments, in order to facilitate the user to adjust the above-mentioned predicted segmentation result map, when the above-mentioned annotation result is displayed to the user, the user can conveniently view the difference between the above-mentioned annotation result and the target object in the above-mentioned image to be annotated.

[0171] In practical applications, the system may display the annotation result in the image to be annotated, or display the image to be annotated and the annotation result in different display areas of the display interface, respectively, so that the user can view the difference between the annotation result and the target object in the image to be annotated.

[0172] Since the system can display the annotation result in the image to be annotated, or display the image to be annotated and the annotation result in different display areas of the display interface respectively, so that the user can view the difference between the annotation result and the target object in the image to be annotated, it is convenient for the user to understand the difference, thereby facilitating the user to adjust the annotation result.

[0173] In some embodiments, in order to further facilitate the user to adjust the predicted segmentation result map, when displaying the annotation result to the user, the system may highlight the difference between the annotation result and the target object in the image to be annotated.

[0174] In practical applications, the system may display the annotation result as a new layer overlay at the corresponding position of the layer where the image to be annotated is located; or,

[0175] Based on the annotation results, different regions of the image to be annotated are rendered to obtain and display the rendering results.

[0176] Since the above system highlights the above annotation result as a new layer covering the corresponding position of the layer where the above image to be annotated is located; or, based on the above annotation result, different areas of the above image to be annotated are rendered to obtain the rendering result and display the above pre-annotated area covering the above object to be annotated, therefore, the above system can highlight the difference between the above annotation result and the target object in the above image to be annotated to the user, thereby further facilitating the user to find the difference between the two, and further facilitating the user to adjust the above predicted segmentation result map.

[0177] In the above situation, in order to further facilitate the user to adjust the above predicted segmentation result map, when the above system obtains the final annotation result corresponding to the above object to be annotated according to the adjustment made by the user to the above predicted segmentation result map, the above system can obtain the final annotation area corresponding to the above object to be annotated according to the adjustment data adjusted by the user for the above difference. After determining the above final annotation area, the above system can obtain the final annotation result based on the above final annotation area.

[0178] In some embodiments, when the user adjusts the annotation result, the user may adjust the pre-annotated area to the final annotation area by simply adjusting the difference to obtain the final annotation result, thereby simplifying the user's adjustment operation workload and improving annotation efficiency.

[0179] See also Figure 7 , Figure 7 This is a schematic diagram of a labeling result graph corresponding to the labeling results shown in this application.

[0180] like Figure 7 As shown, there are at least two differences between the object to be annotated and the pre-annotated area. Difference 1: It can be determined through intuitive observation that the hair on the left side of the person in frame 1 is part of the object to be annotated, but it is not included in the pre-annotated area, so it needs to be adjusted. Difference 2: It can be determined through intuitive observation that the clothes of the person in frame 2 are partially incomplete, that is, the clothes of the person (part of the object to be annotated) are not included in the annotated area, so they need to be adjusted.

[0181] It should be noted that the above two differences are only for illustrating the embodiments and do not specifically limit the present application.

[0182] Regarding the above-mentioned difference 1, the hair part on the left side of the character in the picture does not belong to the pre-annotated area. Therefore, the user can use the polygon annotation tool provided by the above-mentioned system to re-annotate the border line of this area so that the above-mentioned hair part on the left side is added to the final annotation area corresponding to the object to be annotated.

[0183] Regarding the above difference 2, part of the left side of the clothes of the person in the picture does not belong to the pre-annotated area. Therefore, the user can use the brush tool provided by the above system to fill the part of the left side of the clothes so that the left corner area of ​​the clothes is added to the final annotation area corresponding to the object to be annotated.

[0184] Since the above-mentioned predicted segmentation result image intuitively shows the difference between the above-mentioned pre-annotated area and the above-mentioned object to be annotated, when the above-mentioned user adjusts the above-mentioned annotation result, he can adjust the above-mentioned pre-annotated area to the final annotation area by simply adjusting the above-mentioned difference to obtain the final annotation result, thereby simplifying the user's adjustment operation workload and improving the annotation efficiency.

[0185] In some embodiments, in order to improve the fit between the above-mentioned annotation results and the above-mentioned object to be annotated, the above-mentioned image segmentation model can be constructed using a deep learning network.

[0186] For example, the deep learning-based network may be a deep convolutional network or a dilated convolutional network.

[0187] See also Figure 8 , Figure 8 This is a flow chart of an image segmentation method shown in this application.

[0188] like Figure 8 As shown, the above method may include:

[0189] S802, encoding the image to be labeled and the preliminary segmentation map corresponding to the preliminary segmentation result to obtain an encoded feature map.

[0190] S804: Decode the encoded feature map to obtain a predicted segmentation result of the image to be labeled.

[0191] Since the above-mentioned image segmentation model adopts an image segmentation model constructed based on a deep learning network, in the process of inputting the preliminary segmentation map corresponding to the above-mentioned preliminary segmentation result and the above-mentioned image to be labeled into the image segmentation model for processing to obtain the predicted segmentation result of the above-mentioned image to be labeled, more accurate information can be obtained, thereby improving the fit between the above-mentioned predicted segmentation result and the object to be labeled.

[0192] In some embodiments, in order to reduce the amount of computation in the above-mentioned image segmentation model and improve the network convergence efficiency and computational efficiency, the above-mentioned image segmentation model can adopt a coding unit and a decoding unit architecture. When performing image segmentation through the image segmentation model, the above-mentioned preliminary segmentation result can be subjected to dimensionality reduction encoding to extract image features through the coding unit; and then decoded and restored to the original image size through the decoding unit, thereby reducing the amount of computation when extracting image features through dimensionality reduction encoding, and improving the network convergence efficiency and computational efficiency.

[0193] See also Fig. 9 , Fig. 9 This is a flow chart of an image segmentation method shown in this application.

[0194] like Fig. 9 As shown, when performing image segmentation based on the preliminary segmentation result, the above-mentioned image to be labeled and the above-mentioned preliminary segmentation result can be firstly taken as input and input into the encoding unit included in the image segmentation model for encoding to obtain an encoded feature map.

[0195] After obtaining the coded feature map, the coded feature map is input into the decoding unit included in the image segmentation model for decoding to obtain a predicted segmentation result corresponding to the object to be labeled.

[0196] In some embodiments, since the above-mentioned image segmentation model includes a coding unit that can perform dimensionality reduction operations on images, when image segmentation is performed through the image segmentation model, the above-mentioned preliminary segmentation result can be subjected to dimensionality reduction encoding through the coding unit to extract image features; and then decoded and restored to the original image size through the decoding unit, thereby reducing the amount of calculation when extracting image features through dimensionality reduction coding, thereby improving the network convergence efficiency and computing efficiency.

[0197] It should be noted that when the encoding unit and the decoding unit are trained, they can be trained separately or together, which is not limited here.

[0198] In some embodiments, after the encoded feature map is input into the decoding unit for decoding and a predicted segmentation result corresponding to the image to be labeled is obtained, the predicted segmentation result can be corrected according to the preliminary segmentation result to obtain a labeling result corresponding to the object to be labeled.

[0199] See also Fig.10 , Fig.10 This is a flow chart of an image segmentation method shown in this application.

[0200] like Fig.10 As shown, when performing image segmentation, the above-mentioned image to be labeled and the above-mentioned preliminary segmentation result can be first taken as input and input into the encoding unit included in the image segmentation model for encoding to obtain an encoded feature map.

[0201] After obtaining the coded feature map, the coded feature map is input into the decoding unit included in the image segmentation model for decoding to obtain a predicted segmentation result corresponding to the object to be labeled.

[0202] After obtaining the above-mentioned predicted segmentation result, the above-mentioned predicted segmentation result may be corrected according to the above-mentioned preliminary segmentation result to obtain a final labeling result corresponding to the above-mentioned object to be labeled.

[0203] Since the predicted segmentation result is corrected according to the preliminary segmentation result, the erroneous pixels in the predicted segmentation result that are inconsistent with the foreground area and the background area included in the preliminary segmentation result can be corrected, thereby improving the image segmentation accuracy.

[0204] In some embodiments, in order to further improve the accuracy of image segmentation, when the predicted segmentation result is corrected according to the preliminary segmentation result to obtain the final annotation result corresponding to the object to be annotated, when the first pixel point included in the initial foreground area does not belong to the foreground pixel point in the predicted segmentation result, the first pixel point can be corrected to a foreground pixel point; and / or, when the second pixel point included in the initial background area does not belong to the background pixel point in the predicted segmentation result, the second pixel point can be corrected to a background pixel point.

[0205] The first pixel point mentioned above is specifically any pixel point in the initial foreground area mentioned above.

[0206] In practical applications, when performing the above correction, the pixel points in the above initial foreground area can be used as the above first pixel points one by one.

[0207] The second pixel point is specifically any pixel point in the initial background area.

[0208] In practical applications, when performing the above correction, the pixel points in the above initial background area may be used as the above second pixel points one by one.

[0209] In this embodiment, since the preliminary segmentation result includes an accurate initial foreground area and an initial background area, a more accurate predicted segmentation result can be obtained by correcting the foreground area and the background area included in the above-mentioned predicted segmentation result according to the initial foreground area and the initial background area included in the preliminary segmentation result.

[0210] In some embodiments, in order to improve the robustness of the image segmentation model in performing image segmentation, the above-mentioned image segmentation model may include a plurality of parallel convolution branches.

[0211] At this time, when the image to be labeled and the preliminary segmentation result are input into the encoding unit for encoding to obtain the encoding feature map, the image to be labeled and the preliminary segmentation result can be fused to obtain a fused image. After obtaining the fused image, the fused image can be input into the plurality of parallel convolution branches for encoding to obtain the encoding feature map.

[0212] See also Fig.11 , Fig.11 This is a schematic diagram of the structure of a spatial pyramid pooling layer shown in this application.

[0213] like Fig.11 As shown, the input of the spatial pyramid pooling layer may be a feature map extracted after several convolutional layer operations. The spatial pyramid pooling layer may include three parallel pooling operation branches to pool the extracted feature map into three fixed image blocks.

[0214] Specifically, the extracted feature map can be divided into three equal parts, which are respectively input into the three parallel pooling operation branches for pooling operation to obtain three fixed image blocks. Then, the three image blocks are fused to obtain the encoded feature map.

[0215] It should be noted that the pooling operation branches included in the above-mentioned spatial pyramid pooling layer can be set according to actual business situations and are not particularly limited here.

[0216] On the one hand, since the spatial pyramid pooling layer includes parallel pooling operation branches, the extracted feature map is pooled into a fixed image block. Therefore, the spatial pyramid pooling layer can be used as a transition between the convolution layer and the fully connected layer. For preliminary segmentation results of different sizes, the encoded feature maps of the same size are extracted, so that the image segmentation model can adapt to preliminary segmentation results of various sizes, thereby improving the adaptability and robustness of the image segmentation model.

[0217] On the other hand, since the spatial pyramid pooling layer includes parallel pooling operation branches, multi-dimensional pooling can be performed on the extracted feature maps, thereby ensuring that the image segmentation model can extract corresponding image features for contents of different scales in the preliminary segmentation results, thereby improving the robustness of the image segmentation model for image segmentation.

[0218] In some embodiments, in order to further improve the robustness of the above-mentioned image segmentation model for image segmentation, the above-mentioned encoding unit may include a plurality of parallel dilated convolution branches; wherein, at least some of the multiple dilated convolution branches correspond to different dilated rates.

[0219] At this time, when performing encoding processing, the system can process the fused image separately through the multiple parallel dilated convolution branches included in the image segmentation model to obtain the processing results corresponding to the multiple dilated convolution branches. After obtaining the above-mentioned several processing results, the system can fuse the processing results corresponding to the multiple dilated convolution branches to obtain the above-mentioned encoding feature map.

[0220] See also Fig.12 , Fig.12 This is a schematic diagram of the structure of a coding unit shown in this application.

[0221] like Fig.12 As shown, the input of the encoding unit can be a fused image synthesized from the image to be annotated and the corresponding preliminary segmentation result. The encoding unit includes a plurality of first convolutional layers, a spatial pyramid pooling layer, and a second convolutional layer.

[0222] The output of the first convolutional layer is the input of the spatial pyramid pooling layer, the output of the spatial pyramid pooling layer is the input of the second convolutional layer, and the output of the second convolutional layer is the output of the encoding unit.

[0223] The first convolution layer can be used to extract features from the fused image. The first convolution layer can be a causal convolution layer, a dilated convolution layer, a depthwise separable convolution layer, etc., which are not particularly limited here.

[0224] The spatial pyramid pooling layer specifically includes a convolution layer with a dilation ratio of 1 and a convolution kernel of 1*1, a convolution layer with a dilation ratio of 6 and a convolution kernel of 3*3, a convolution layer with a dilation ratio of 12 and a convolution kernel of 3*3, a convolution layer with a dilation ratio of 18 and a convolution kernel of 3*3, and an image pooling layer. The output results of the above convolution layers and pooling layers are fused.

[0225] It should be noted that the atrous convolution layers and the number of pooling layers included in the spatial pyramid pooling layer, as well as the atrous rate and the convolution kernel size can be set according to actual needs and are not particularly limited here.

[0226] The second convolutional layer mentioned above can be used to adjust the number of channels.

[0227] In this example, the second convolutional layer may include several single-kernel convolutions.

[0228] When the fused image is input into the spatial pyramid pooling layer for encoding to obtain a coded feature map, the encoding unit may extract corresponding features of the fused image through the plurality of first convolutional layers to obtain a corresponding feature map.

[0229] After obtaining the corresponding feature map, the feature map can be divided into 5 identical parts and respectively input into the convolution layer and pooling layer included in the spatial pyramid pooling layer for processing to obtain the corresponding processing result maps.

[0230] After obtaining the above processing result graph, the above several processing result graphs can be fused, and the number of channels can be adjusted through the above second convolutional layer to obtain the above encoding feature graph.

[0231] Since the encoding unit may include multiple atrous convolution layers with different atrous rates, the image segmentation model may have receptive fields of different sizes through the spatial pyramid pooling layer, and may capture edge segmentation information of large objects and small objects at the same time, thereby ensuring that the image segmentation model may extract corresponding image features for contents of different scales in the preliminary segmentation results, and further improving the robustness of the image segmentation model for image segmentation.

[0232] In some embodiments, in order to reduce the amount of computation in the image segmentation model, the fused image may be reduced by a preset multiple.

[0233] In practical applications, before encoding the fused image, the system may reduce the fused image by a preset multiple to obtain the reduced fused image.

[0234] At this time, when the system encodes the fused image, it encodes the reduced fused image, thereby obtaining a coding feature map.

[0235] The preset multiple can be a reasonable multiple set based on experience, for example, 4 times, 8 times or 16 times.

[0236] For example, the system can reduce the fused image by 4 times through two convolutional layers (backbone convolutional network) to obtain a low-level feature map reduced by 4 times. Then the low-level feature map is input into the spatial pyramid pooling layer for processing to obtain a high-level feature map reduced by 16 times.

[0237] Since the fused image is reduced by 4 times before being input into the spatial pyramid pooling layer, and is further reduced by 4 times in the spatial pyramid pooling layer, the operations in the spatial pyramid pooling layer and the subsequent operations can reduce the amount of operations, thereby reducing the amount of operations in the image segmentation model and improving the image segmentation efficiency.

[0238] In some embodiments, in order to improve the robustness of the image segmentation model for image segmentation, when the above-mentioned encoded feature map is input into the decoding unit for decoding and the predicted segmentation result corresponding to the above-mentioned object to be labeled is obtained, the above-mentioned system can fuse the intermediate feature map generated in the above-mentioned encoding process with the above-mentioned encoded feature map to obtain a fused feature map. After obtaining the above-mentioned fused feature map, the above-mentioned system can decode the above-mentioned fused feature map to obtain the predicted segmentation result of the above-mentioned image to be labeled.

[0239] See also Fig.13 , Fig.13 This is a schematic diagram of the structure of a decoding unit shown in this application.

[0240] like Fig.13 As shown, the input of the above decoding unit is divided into two parts. The first part is the intermediate feature map generated by the encoding unit during the encoding process (for example, in some of the above embodiments, the low-level feature map generated by the backbone convolutional network is reduced by 4 times); the second part is the encoded feature map output by the encoding unit (for example, in some of the above embodiments, the high-level feature map is reduced by 16 times after being processed by the spatial pyramid pooling layer).

[0241] The decoding unit includes, in order of data processing, a third convolution layer, a first upsampling unit, a fusion unit, a fourth convolution layer, and a second upsampling unit.

[0242] The third convolutional layer mentioned above can be used to adjust the number of channels.

[0243] In this example, the third convolutional layer may include several single-kernel convolutions.

[0244] The first upsampling unit can determine the magnification factor according to the actual operation situation. The algorithm used by the upsampling unit can be a transposed convolution, bilinear interpolation, pooling and other algorithms, which are not particularly limited here. In this example, the first upsampling unit can use a bilinear interpolation method to magnify the above-mentioned encoding feature map by 4 times.

[0245] The fourth convolution layer can be used to adjust the number of channels. In this example, the fourth convolution layer can include a number of 3*3 convolution kernels.

[0246] The second upsampling unit can determine the magnification factor according to the actual operation situation. The algorithm used by the upsampling unit can be a transposed convolution, bilinear interpolation, pooling and other algorithms, which are not particularly limited here. In this example, the first upsampling unit can use a bilinear interpolation method to magnify the above-mentioned encoding feature map by 4 times.

[0247] When the encoded feature map is input into the decoding unit for decoding to obtain a predicted segmentation result corresponding to the object to be labeled, the intermediate feature map (first part input) can be adjusted in number of channels through the third convolutional layer to obtain an adjusted result.

[0248] For the above-mentioned coding feature map (second part of input), it can be passed through the above-mentioned first upsampling unit to obtain a first sampling processing result.

[0249] The channel adjustment result is obtained and fused with the first sampling processing result to obtain a fusion result.

[0250] After obtaining the above fusion result, the predicted segmentation result of the same size as the above preliminary segmentation result can be obtained through the above fourth convolution layer and the above second sampling processing result. Each pixel in the above predicted segmentation result is marked with a label; the above label indicates whether the pixel corresponding to the label is foreground or background.

[0251] In some embodiments, since the intermediate feature map (usually low-level features) generated during the encoding process is introduced during the decoding process, decryption is performed by fusing the low-level intermediate feature map with the high-level encoding feature map, thereby effectively utilizing multi-scale information and fusing high-level and low-level semantic features, thereby improving the robustness of the above-mentioned image segmentation model for image segmentation.

[0252] Corresponding to any of the above embodiments, the present application also proposes an image annotation device.

[0253] See also Fig.14 , Fig.14 This is a schematic diagram of the structure of an image annotation device shown in this application.

[0254] like Fig.14 As shown, the image annotation device 1400 may include: an acquisition module 1410, used to acquire the annotation operation performed by the user on the image to be annotated; wherein the contour lines outlined by the above-mentioned annotation operation form a connected area; a segmentation processing module 1420, used to perform segmentation processing on the above-mentioned image to be annotated based on the above-mentioned annotation operation to obtain a predicted segmentation result of the above-mentioned image to be annotated; and a annotation result obtaining module 1430, used to obtain the annotation result of the above-mentioned image to be annotated based on the above-mentioned predicted segmentation result of the above-mentioned image to be annotated.

[0255] In some of the illustrated embodiments, the contour lines drawn by the above-mentioned annotation operation form a connected area in the above-mentioned image to be annotated; or, the contour lines drawn by the above-mentioned annotation operation and the boundary of the above-mentioned image to be annotated jointly form a connected area.

[0256] In some of the illustrated embodiments, the segmentation processing module 1420 includes: a preliminary segmentation module, which is used to perform preliminary segmentation on the image to be labeled based on the labeling operation to obtain a preliminary segmentation result; wherein the preliminary segmentation result includes the initial foreground area, the initial background area and the initial edge area between the initial foreground area and the initial background area of ​​the image to be labeled; and a predictive segmentation module, which is used to perform predictive segmentation on the image to be labeled based on the preliminary segmentation result to obtain a predictive segmentation result of the image to be labeled.

[0257] In some of the illustrated embodiments, the preliminary segmentation module includes: a first display module for displaying the first area and the second area into which the image to be annotated is divided by the contour lines outlined by the annotation operation; an acquisition module 1410 for acquiring the user's selection operation; wherein the selection operation is used to select the initial foreground area and / or the initial background area from the first area and the second area.

[0258] In some of the illustrated embodiments, the preliminary segmentation module includes: a determination module that determines the contour line or the contour line and the vicinity of the contour line as an initial edge region between the initial foreground region and the initial background region.

[0259] In some of the illustrated embodiments, the above-mentioned prediction segmentation module includes: a first prediction segmentation submodule, which is used to predict the segmentation of the above-mentioned preliminary segmentation result and the above-mentioned image to be labeled through an image segmentation model to obtain a predicted segmentation result; or, a second prediction segmentation submodule, which is used to input the preliminary segmentation map corresponding to the above-mentioned preliminary segmentation result and the above-mentioned image to be labeled into the image segmentation model for processing to obtain the predicted segmentation result of the above-mentioned image to be labeled.

[0260] In some of the illustrated embodiments, the second prediction segmentation submodule includes: an encoding processing module, used to encode the preliminary segmentation map corresponding to the image to be labeled and the preliminary segmentation result to obtain a coding feature map; a decoding module, used to decode the coding feature map to obtain the prediction segmentation result of the image to be labeled.

[0261] In some of the illustrated embodiments, the above-mentioned encoding processing module includes: a first fusion module, used to fuse the above-mentioned image to be labeled with the preliminary segmentation map corresponding to the above-mentioned preliminary segmentation result to obtain a fused image; an encoding processing sub-module, used to encode the above-mentioned fused image to obtain a coding feature map.

[0262] In some of the illustrated embodiments, the encoding processing submodule is used to process the fused image respectively through multiple parallel dilated convolution branches included in the image segmentation model to obtain processing results corresponding to the multiple dilated convolution branches respectively; wherein at least some of the multiple dilated convolution branches correspond to different dilated rates; and the processing results corresponding to the multiple dilated convolution branches are fused to obtain the encoding feature map.

[0263] In some of the illustrated embodiments, the device 1400 further includes: an image reduction module, used to reduce the fused image by a preset multiple before encoding the fused image to obtain the reduced fused image; and the encoding processing submodule, used to encode the reduced fused image to obtain a coding feature map.

[0264] In some of the illustrated embodiments, the above-mentioned decoding processing module includes: a second fusion module, which is used to fuse the intermediate feature map generated in the above-mentioned encoding process with the above-mentioned encoding feature map to obtain a fused feature map; a decoding processing sub-module, which decodes the above-mentioned fused feature map to obtain the predicted segmentation result of the above-mentioned image to be labeled.

[0265] In some of the illustrated embodiments, the labeling result obtaining module 1430 includes: a first obtaining submodule 1440, used to use the predicted segmentation result as the labeling result of the image to be labeled; or, a second obtaining submodule 1450, used to correct the predicted segmentation result according to the preliminary segmentation result to obtain the labeling result of the image to be labeled.

[0266] In some of the illustrated embodiments, the second obtaining submodule 1450 includes: a first correction module, used to correct the first pixel point included in the initial foreground area to a foreground pixel point when the first pixel point included in the initial foreground area does not belong to the foreground pixel point in the predicted segmentation result; and / or, a second correction module, used to correct the second pixel point included in the initial background area to a background pixel point when the second pixel point included in the initial background area does not belong to the background pixel point in the predicted segmentation result.

[0267] In some of the illustrated embodiments, the device 1400 further includes: a second display module for displaying the annotation results; and an adjustment module for adjusting the annotation results according to adjustment operations performed by the user on the displayed annotation results to obtain the final annotation results of the image to be annotated.

[0268] In some of the illustrated embodiments, the second display module further includes: a first display submodule, used to display the annotation result in the image to be annotated, or a second display submodule, used to respectively display the image to be annotated and the annotation result in different display areas of the display interface, so that the user can view the difference between the annotation result and the target object in the image to be annotated.

[0269] In some of the illustrated embodiments, the first display submodule is used to: display the annotation results as a new layer overlay at a corresponding position of the layer where the image to be annotated is located; or, based on the annotation results, render different areas of the image to be annotated to obtain and display the rendering results.

[0270] In some of the illustrated embodiments, the apparatus 1400 further includes: a brush configuration option display module, configured to display a plurality of brush configuration options for the user to select before obtaining the annotation operation performed by the user on the image to be annotated; wherein the plurality of brush configuration options correspond to brush lines of different widths.

[0271] The embodiment of the image annotation device shown in this application can be applied to electronic devices. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, such as Fig.15 As shown, it is a structural schematic diagram of an electronic device shown in this application, except Fig.15 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0272] Please refer to Fig.15 An electronic device is shown, and the electronic device may include: a processor.

[0273] Memory used to store processor-executable instructions.

[0274] The processor implements the image annotation method shown in any of the above embodiments by running the above executable instructions.

[0275] The present application proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image annotation method shown in any of the above embodiments.

[0276] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0277] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0278] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0279] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0280] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that may include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0281] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The above processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0282] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0283] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0284] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work as above-mentioned in some combinations and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0285] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0286] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0287] The above is only a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. An image annotation method, characterized in that: The method comprises: Acquire a labeling operation performed by a user on the image to be labeled; wherein the contour lines drawn by the labeling operation form a connected area; Based on the labeling operation, segmenting the image to be labeled to obtain a predicted segmentation result of the image to be labeled, including: based on the labeling operation, performing preliminary segmentation on the image to be labeled to obtain a preliminary segmentation result; based on the preliminary segmentation result, performing predictive segmentation on the image to be labeled to obtain a predicted segmentation result of the image to be labeled; Based on the predicted segmentation result of the image to be labeled, obtaining a labeling result of the image to be labeled, including: correcting the predicted segmentation result according to the preliminary segmentation result to obtain a labeling result of the image to be labeled; Displaying the annotation result includes displaying the annotation result in the image to be annotated, that is, displaying the annotation result as a new layer overlaying a corresponding position of the layer where the image to be annotated is located; According to the adjustment operation performed by the user on the displayed annotation result, the annotation result is adjusted to obtain the final annotation result of the image to be annotated.

2. The method according to claim 1, characterized in that The contour lines drawn by the annotation operation form a connected area in the image to be annotated; or, The contour line drawn by the annotation operation and the boundary of the image to be annotated together form a connected area.

3. The method according to claim 1, characterized in that The preliminary segmentation result includes an initial foreground area, an initial background area, and an initial edge area between the initial foreground area and the initial background area of ​​the image to be labeled.

4. The method according to claim 3, characterized in that: The step of performing preliminary segmentation on the image to be labeled based on the labeling operation includes: Displaying the first area and the second area into which the image to be annotated is divided by the contour line outlined by the annotation operation; Acquire a selection operation of the user; wherein the selection operation is used to select the initial foreground area and / or the initial background area from the first area and the second area.

5. The method according to claim 3, characterized in that: The step of performing preliminary segmentation on the image to be labeled based on the labeling operation includes: The contour line or the contour line and the vicinity of the contour line are determined as an initial edge area between the initial foreground area and the initial background area.

6. The method according to claim 1, characterized in that The step of performing predictive segmentation on the image to be labeled based on the preliminary segmentation result to obtain the predictive segmentation result of the image to be labeled includes: Perform predictive segmentation on the preliminary segmentation result and the image to be labeled by using an image segmentation model to obtain a predictive segmentation result; or, The preliminary segmentation map corresponding to the preliminary segmentation result and the image to be labeled are input into an image segmentation model for processing to obtain a predicted segmentation result of the image to be labeled.

7. The method according to claim 6, characterized in that The step of inputting the preliminary segmentation map corresponding to the preliminary segmentation result and the image to be labeled into the image segmentation model for processing to obtain the predicted segmentation result of the image to be labeled includes: Performing encoding processing on the image to be labeled and the preliminary segmentation map corresponding to the preliminary segmentation result to obtain an encoded feature map; The encoded feature map is decoded to obtain a predicted segmentation result of the image to be labeled.

8. The method according to claim 7, characterized in that The encoding process of the image to be labeled and the preliminary segmentation result to obtain an encoding feature map includes: Fusing the image to be labeled with the preliminary segmentation map corresponding to the preliminary segmentation result to obtain a fused image; The fused image is encoded to obtain an encoded feature map.

9. The method according to claim 8, characterized in that The step of encoding the fused image to obtain a coding feature map includes: The fused image is processed respectively by a plurality of parallel dilated convolution branches included in the image segmentation model to obtain processing results respectively corresponding to the plurality of dilated convolution branches; wherein at least some of the plurality of dilated convolution branches correspond to different dilated rates; The processing results corresponding to multiple hole convolution branches are fused to obtain the encoding feature map.

10. The method according to claim 8 or 9, characterized in that: The method further comprises: Before encoding the fused image, reducing the fused image by a preset multiple to obtain the reduced fused image; The step of encoding the fused image to obtain a coding feature map includes: The reduced fused image is encoded to obtain an encoded feature map.

11. The method according to claim 7, characterized in that The decoding process of the encoded feature map to obtain the predicted segmentation result of the image to be labeled includes: The intermediate feature map generated in the encoding process is fused with the encoding feature map to obtain a fused feature map; The fused feature map is decoded to obtain a predicted segmentation result of the image to be labeled.

12. The method according to claim 3, characterized in that The step of correcting the predicted segmentation result according to the preliminary segmentation result to obtain the labeling result of the image to be labeled includes: When a first pixel point included in the initial foreground area does not belong to a foreground pixel point in the predicted segmentation result, correcting the first pixel point to a foreground pixel point; and / or, When the second pixel point included in the initial background area does not belong to the background pixel point in the predicted segmentation result, the second pixel point is corrected to a background pixel point.

13. The method according to claim 1, characterized in that The method further comprises: Before obtaining the annotation operation performed by the user on the image to be annotated, multiple brush configuration options are displayed for the user to select; wherein the multiple brush configuration options correspond to brush lines of different widths.

14. An image annotation device, characterized in that: The device comprises: An acquisition module is used to acquire the annotation operation performed by the user on the image to be annotated; wherein the contour lines drawn by the annotation operation form a connected area; A segmentation processing module, based on the labeling operation, performs segmentation processing on the image to be labeled to obtain a predicted segmentation result of the image to be labeled, specifically used to perform preliminary segmentation on the image to be labeled based on the labeling operation to obtain a preliminary segmentation result; based on the preliminary segmentation result, perform predictive segmentation on the image to be labeled to obtain a predicted segmentation result of the image to be labeled; A labeling result obtaining module, which obtains the labeling result of the image to be labeled based on the predicted segmentation result of the image to be labeled, and is specifically used to correct the predicted segmentation result according to the preliminary segmentation result to obtain the labeling result of the image to be labeled; A second display module is used to display the annotation result, including displaying the annotation result in the image to be annotated, and specifically to display the annotation result as a new layer overlay at a corresponding position of the layer where the image to be annotated is located; The adjustment module is used to adjust the annotation result according to the adjustment operation performed by the user on the displayed annotation result to obtain the final annotation result of the image to be annotated.

15. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the image annotation method described in any one of claims 1-13.

16. An electronic device, characterized in that: The device comprises: a processor; a memory for storing instructions executable by the processor; The processor is configured to call the executable instructions stored in the memory to implement the image annotation method described in any one of claims 1-13.

Images