Image processing device, image processing method and machine-readable storage medium

Abstract

The present disclosure relates to an image processing device, an image processing method and a machine-readable storage medium. The image processing device includes: an information extraction unit, which obtains image information about an input image and click information about a user's single click on a specific object among multiple objects on the input image; a first acquisition unit, which obtains a bounding box including a click point and an original mask corresponding to the bounding box based on the image information and the click information; a second acquisition unit, which uses the original mask and the bounding box to obtain an intermediate mask that highlights the specific object from the input image; a selection unit, which selects the best mask from multiple intermediate masks; and a labeling unit, which uses the best mask to label the edge of the specific object in the input image. Through the image processing device, the user can obtain the outline of the clicked object with only a single click.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing device, an image processing method and a machine-readable storage medium for interactive segmentation and annotation. Background Art

[0002] This section provides background information related to the present disclosure which is not necessarily prior art.

[0003] With the increase of data and the enhancement of computing power, deep learning has played a huge role in the field of image processing. Image annotation is an important step in deep learning tasks (such as object detection and image segmentation). The current annotation tools can be mainly divided into two categories. One is tools such as Labelme and Opensurface, which require the annotator to use the mouse to draw the outline of the target boundary little by little. Although this kind of annotation is accurate, it is very time-consuming, and different annotators may have different annotations for the same target, which brings challenges to subsequent model training.

[0004] On the other hand, tools such as ScribbleSup, Polygon-RNN, PolyRNN++, and CurveGCN require the annotator to input additional information. ScribbleSup requires the annotator to scribble on the target object. In Polygon-RNN, PolyRNN++, and CurveGCN, the bounding box containing the target is taken as input, and then the polygon vertices of the target object are generated as output. Compared with clicks, scribbles and bounding boxes are more complicated. In order to reduce the time and cost of annotation, this application uses user clicks as input for interactive annotation tools. Summary of the invention

[0005] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0006] The present disclosure aims to provide an image processing device, an image processing method and a machine-readable storage medium for interactive segmentation and annotation based on user clicks.

[0007] According to one aspect of the present disclosure, there is provided an image processing device, comprising: an information extraction unit, which obtains image information about an input image and click information about a user's single click on a specific object among multiple objects on the input image; a first acquisition unit, which obtains a bounding box including the click point and an original mask corresponding to the bounding box based on the image information and the click information; a second acquisition unit, which uses the original mask and the bounding box to obtain an intermediate mask that highlights the specific object from the input image; a selection unit, which selects an optimal mask from multiple intermediate masks; and a labeling unit, which uses the optimal mask to label the edge of the specific object in the input image.

[0008] According to another aspect of the present disclosure, there is provided an image processing method, comprising: acquiring image information about an input image and click information about a user's single click on a specific object among multiple objects on the input image; obtaining a bounding box including the click point and an original mask corresponding to the bounding box based on the image information and the click information; obtaining an intermediate mask that highlights the specific object from the input image using the original mask and the bounding box; and selecting an optimal mask from multiple intermediate masks, and marking the edge of the specific object in the input image using the optimal mask.

[0009] According to another aspect of the present disclosure, a machine-readable storage medium is provided, which carries a program product including machine-readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, enable the computer to perform the image processing method according to the present disclosure.

[0010] Using the image processing device, image processing method and machine-readable storage medium according to the present disclosure, a user can obtain the edge of a clicked object with only a single click. In addition, by executing the selection rule on the bounding box instead of executing the selection rule on multiple final outputs of the object annotation, the annotation efficiency and accuracy can be improved.

[0011] Further areas of applicability will become apparent from the description provided herein.The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure. In the drawings:

[0013] Figure 1 is a block diagram illustrating a structure of an image processing apparatus according to an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of labeling objects on an input image using an image processing device according to an embodiment of the present disclosure;

[0015] Figure 3 is a block diagram illustrating a structure of an image processing apparatus according to another embodiment of the present disclosure;

[0016] Figure 4 A block diagram illustrating a structure of an information extraction unit in an image processing apparatus according to an embodiment of the present disclosure;

[0017] Figure 5 is a block diagram illustrating a structure of a first acquisition unit in an image processing apparatus according to an embodiment of the present disclosure;

[0018] Figure 6 to Figure 7 A block diagram illustrating a bounding box selection principle of a first acquisition unit in an image processing device according to an embodiment of the present disclosure;

[0019] Figure 8 is a block diagram illustrating a structure of an image processing apparatus according to yet another embodiment of the present disclosure;

[0020] Fig. 9 A schematic diagram illustrating operations of a mask template acquisition unit and a contour template acquisition unit in an image processing device according to another embodiment of the present disclosure;

[0021] Fig.10 is a block diagram illustrating a structure of another first acquisition unit in the image processing apparatus according to an embodiment of the present disclosure;

[0022] Fig.11 is a block diagram illustrating a structure of a second acquisition unit in an image processing apparatus according to an embodiment of the present disclosure;

[0023] Fig.12 A flowchart illustrating an image processing method according to an embodiment of the present disclosure; and

[0024] Fig.13 is a block diagram of an exemplary structure of a general-purpose personal computer in which the image processing apparatus and method according to the embodiments of the present disclosure can be implemented.

[0025] Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown as examples in the drawings and are described in detail herein. It should be understood, however, that the description of specific embodiments herein is not intended to limit the present disclosure to the specific forms disclosed, but rather, the present disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present disclosure. It should be noted that throughout the several drawings, corresponding reference numerals indicate corresponding parts. DETAILED DESCRIPTION

[0026] Examples of the present disclosure will now be described more fully with reference to the accompanying drawings.The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0027] Example embodiments are provided so that the disclosure will be exhaustive and its scope will be fully conveyed to those skilled in the art. Numerous specific details such as examples of specific components, devices and methods are set forth to provide a detailed understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that the example embodiments may be implemented in many different forms without the use of specific details, none of which should be construed as limiting the scope of the present disclosure. In certain example embodiments, well-known processes, well-known structures and well-known techniques are not described in detail.

[0028] Combine the following Figure 1 To illustrate how the image processing device according to an embodiment of the present disclosure can obtain the edge of a clicked object by a user clicking on an object on an input image only once. It should be noted that the object can be various objects such as bottles, people, animals, etc. on the image, and the object is not limited to the examples shown in this application. The input of the image processing device according to the present application is the image to be segmented and annotated and the coordinates of the point clicked by the user, and the output is the segmentation annotation of the object to which the clicked point belongs.

[0029] Figure 1 FIG. 1 is a block diagram illustrating a structure of an image processing apparatus according to an embodiment of the present disclosure. Figure 1 As shown, the image processing apparatus 100 according to an embodiment of the present disclosure may include an information extracting unit 110 , a first acquiring unit 120 , a second acquiring unit 130 , a selecting unit 140 , and a labeling unit 150 .

[0030] The information extraction unit 110 may acquire image information about the input image and click information about a single click of a user on a specific object among a plurality of objects on the input image. In addition, the information extraction unit 110 may provide the image information and the click information to the first acquisition unit 120.

[0031] Further, the first acquisition unit 120 can obtain a bounding box including the click point and an original mask corresponding to the bounding box based on the image information and the click information. In addition, the first acquisition unit 120 can provide the bounding box including the click point and the original mask corresponding to the bounding box to the second acquisition unit 130.

[0032] Furthermore, the second acquisition unit 130 may obtain an intermediate mask that highlights the specific object from the input image using the original mask and the bounding box. In addition, the second acquisition unit 130 may provide the intermediate mask to the selection unit 140.

[0033] Furthermore, the selection unit 140 may select the best mask from the plurality of intermediate masks. For example, the selection unit 140 may select the intermediate mask with the highest score as the best mask based on the scores of the intermediate masks. In addition, the selection unit 140 may provide the selected best mask to the labeling unit 150.

[0034] Furthermore, the annotation unit 150 may use the optimal mask to mark the edge of the specific object in the input image. For example, the annotation unit 150 may extract the contour of the specific object in the optimal mask and mark the contour in the input image at a position corresponding to the specific object.

[0035] Thus, the image processing apparatus 100 according to an embodiment of the present disclosure can mark the edge (contour) of a specific object on the image according to a single click of the user on the specific object on the image. In addition, by executing the selection rule on the bounding box instead of executing the selection rule on multiple final outputs of the object annotation, the annotation efficiency and accuracy can be improved.

[0036] Figure 2 FIG. 1 is a schematic diagram of annotating objects on an input image using an image processing device according to an embodiment of the present disclosure. Figure 2 As shown in FIG. 1 , the input image includes multiple different bottles (objects). In the upper figure, the user clicks on one of the bottles. In the lower figure, the image processing device according to the embodiment of the present disclosure can segment and mark the edge (contour) of the bottle clicked by the user in the image. It should be noted that although Figure 2 The bottle clicked is shown to be segmented and marked in the form of dots and lines, but the form of segmentation and marking is not limited. For example, the segmentation and marking can also be performed in the form of squares and lines. In addition, Figure 2 The input images in are only examples, and although Figure 2 The input image in does not include a background, but the input image may include not only the multiple bottles shown, but also a background in which multiple bottles are placed, not shown.

[0037] In order to better understand the technical solution of the present disclosure, the image processing device of the present disclosure is described in more detail below.

[0038] Preferably, if Figure 3 As shown, the image processing apparatus 300 according to another embodiment of the present disclosure may further include a pre-processing unit 360 . Figure 3 The information extraction unit 310, the first acquisition unit 320, the second acquisition unit 330, the selection unit 340 and the labeling unit 350 shown in FIG. 3 correspond to Figure 1 The information extracting section 110 , the first acquiring section 120 , the second acquiring section 130 , the selecting section 140 and the labeling section 150 are shown, and thus the description thereof is omitted.

[0039] Specifically, the preprocessing unit 360 may be configured to resize and normalize the input image, and normalize the coordinates of the user's click. In addition, the preprocessing unit 360 provides the transformed input image and the click coordinates to the information extraction unit 310.

[0040] The information extraction unit 310 can obtain image information about the input image based on the input image provided by the preprocessing unit 360, and provide the image information about the input image and the obtained click information to the first acquisition unit 320. Figure 4 The structure of the information extraction unit is described in detail.

[0041] Figure 4 A block diagram illustrating a structure of an information extraction section in an image processing apparatus according to an embodiment of the present disclosure. Figure 4 The information extraction unit 400 shown in FIG. Figure 1 The information extraction unit 110 and Figure 3 The information extraction unit 310 shown in FIG. Figure 4 As shown, the information extraction unit 400 includes a first information extraction block 410 , a second information extraction block 420 and a third information extraction block 430 .

[0042] Specifically, the first information extraction block 410 may use ResNet101 to extract features of the input image.

[0043] Then, in the second information extraction block 420, a feature pyramid module may be used to generate feature maps of different scales as image information. In this way, the second information extraction block 420 may generate feature maps of different scales to detect objects of different sizes.

[0044] The third information extraction block 430 may obtain click information about a click point of a single click, such as coordinate information, based on a single click of a specific object by a user.

[0045] As an example, in Figure 4 In the first information extraction block 410 in FIG. 4 , C1 to C5 represent different levels of information of the image. For example, layers C1 to C5 may represent deeper information. For example, layer C1 may reflect the texture color of the image, while layer C5 may reflect the scene in the image, such as the image content. Of course, the image features extracted by the first information extraction block 410 are not limited to Figure 4 The C1 to C5 layers shown may also be other numbers of layer information or other layer information.

[0046] In addition, as an example, the second information extraction block 420 can generate feature maps of different scales through a feature pyramid based on different levels of information provided by the first information extraction block 410. For example, the second information extraction block 420 can generate multiple layers of feature maps P3 to P7 based on the information C3 to C5 provided by the first information extraction block 410. For example, feature map P5 can be generated based on information C5, that is, P5=f(C5), where f(C5) represents a functional transformation of information C5. For example, feature map P4 can be generated based on information C4 and feature map P5, that is, P4=f(C4)+g(P5), where f(C4) and g(P5) represent functional transformations of information C4 and feature map P5, respectively. For example, feature map P3 can be generated based on information C3 and feature map P4, that is, P3=f(C3)+g(P4), where f(C3) and g(P4) represent functional transformations of information C3 and feature map P4, respectively. Similarly, as Figure 4 As shown by the middle arrow, feature map P6 can be obtained based on feature map P5, and feature map P7 can be obtained based on feature map P6.

[0047] In addition, Figure 4 In the second information extraction block 420 in FIG. 4 , feature maps P3 to P7 represent feature maps of different scales. For example, feature map P3 is for small-scale objects, while feature map P7 is for large-scale objects. Of course, the feature image information extracted by the second information extraction block 420 is not limited to Figure 4 The feature maps P3 to P7 shown may also be feature maps of other scales or include feature maps of more different scales.

[0048] Thus, the information extraction unit 400 can provide the image information provided by the second information extraction block 420, such as the feature maps P3 to P7, and the click information provided by the third information extraction block 430, such as the coordinate information, to the first acquisition unit.

[0049] Combine the following Figure 5 A block diagram describing the structure of a first acquisition section in an image processing apparatus according to an embodiment of the present disclosure. Figure 5 The first acquisition unit 500 in corresponds to Figure 1 The first acquisition unit 120 and Figure 3 The first acquisition unit 320 shown in .

[0050] Figure 5 The first acquisition unit 500 may include a head prediction unit 510 and a mask template acquisition unit 520. According to the foregoing, the first acquisition unit 500 may acquire image information such as feature maps P3 to P7 and click information such as coordinate information of click points from the information extraction unit.

[0051] Specifically, the head prediction unit 510 can obtain information such as a mask coefficient, a contour coefficient, a bounding box, and a confidence level of the input image based on the image information and the click information. For example, the mask coefficient is related to the mask of the input image, and the contour coefficient is related to the contour of the object in the input image. Figure 6 and Figure 7 Describe the generation and selection of bounding boxes in detail.

[0052] In addition, the confidence level can represent the probability that an object in the input image belongs to a certain category. The probability that an object belongs to a certain category can refer to the probability that the object belongs to different categories of objects such as people, cats, dogs, bottles, etc., and can also refer to the probability that the object belongs to a subcategory of a certain category of objects. For example, for Figure 2 For an input image including multiple bottles in , the probability that an object belongs to a certain category can refer to the probability that the object belongs to a certain category among multiple categories of bottles. As an example, the confidence can be given by Figure 1 The selection unit 140 and Figure 3 The selection unit 340 in is used as the score of the intermediate mask, and thus used for selecting the best mask, that is, selecting the intermediate mask with the highest confidence as the best mask.

[0053] Further, the mask template acquisition unit 520 may obtain the mask template based on the image information. For example, the mask template acquisition unit 520 may obtain the mask template based on the feature map P3. It should be noted that the mask template in the present application may be a mask basis vector.

[0054] The first acquisition unit 500 may calculate the original mask of the input image based on the mask basis vectors obtained by the mask template acquisition unit 520 and the mask coefficients obtained by the head prediction unit 510. More specifically, the original mask of the input image may be calculated by multiplying the mask basis vectors by the mask coefficients.

[0055] The following will be combined Figure 2 , Figure 6 and Figure 7 The selection rule applied in the first acquisition unit 500 will be described in detail. Figure 6 to Figure 7 The present invention is a block diagram illustrating a bounding box selection principle of a first acquisition unit in an image processing apparatus according to an embodiment of the present disclosure.

[0056] For example, refer again to Figure 2 ,exist Figure 2 In the figure above, the user clicks on a bottle in the input image, thereby generating a click point on the input image. It is assumed here that the image information provided by the information extraction unit is a multi-layer feature map of the input image, such as Figure 4The first acquisition unit 500 may be configured to: generate a predetermined number of bounding boxes of different sizes for each pixel of each layer of the multi-layer feature map; and select a bounding box including the click point from the generated bounding boxes.

[0057] Specifically, Figure 6 As shown, the first acquisition unit 500 (the head prediction unit 510 therein) can generate 9 bounding boxes of different scales and sizes for each pixel of each layer of feature map. It should be understood that although Figure 6 Nine different bounding boxes are shown, but the number of generated bounding boxes is not limited to nine, and may be any other number. For each layer (P3, P4, P5, P6, P7) in the feature pyramid module, the first acquisition unit 500 will determine whether the generated bounding box contains the click point clicked by the user, and for the bounding box including the click point, the corresponding index will be recorded to select the corresponding other information, such as mask coefficient, silhouette coefficient, confidence and other information.

[0058] Figure 7 The selection process of the bounding box is shown. As an example, the size of feature map P3 is 69 pixels * 69 pixels, the size of feature map P4 is 35 pixels * 35 pixels, the size of feature map P5 is 18 pixels * 18 pixels, the size of feature map P6 is 9 pixels * 9 pixels, and the size of feature map P7 is 5 pixels * 5 pixels. Since 9 different bounding boxes are generated for each pixel, 69 * 69 * 9 = 42849 bounding boxes can be generated for feature map P3. Similarly, 11025 bounding boxes can be generated for feature map P4, 2916 bounding boxes can be generated for feature map P5, 729 bounding boxes can be generated for feature map P6, and 225 bounding boxes can be generated for feature map P7.

[0059] Next, the bounding boxes containing the user's click points are selected from all these bounding boxes. As an example, 140 bounding boxes containing click points can be selected for feature map P3, 145 bounding boxes containing click points can be selected for feature map P4, 147 bounding boxes containing click points can be selected for feature map P5, 150 bounding boxes containing click points can be selected for feature map P6, and 143 bounding boxes containing click points can be selected for feature map P7. Therefore, a total of 725 bounding boxes containing click points can be obtained, for example.

[0060] Thus, as an example, the first acquisition unit 500 can obtain 725 bounding boxes including the click point and the mask coefficients corresponding to the bounding boxes provided by the head prediction unit 510 and the mask template provided by the mask template acquisition unit 520, so as to obtain the corresponding original mask. In addition, the first acquisition unit 500 can provide the bounding box including the click point and the corresponding original mask to the second acquisition unit. It should be understood that the numbers listed in the above example are only illustrative and can also be other numbers.

[0061] It can be seen that the image processing device of the present application is an interactive segmentation and annotation device based on user clicks, which generates a mask for the object to which the click point belongs by adding selection rules. By executing the selection rules on the bounding box instead of executing the selection rules on multiple final outputs of the object annotation, the annotation efficiency and accuracy can be improved.

[0062] Furthermore, by adding selection rules, it is expected that the image processing apparatus according to the present application only focuses on the target containing the user click. Applying selection rules to the layer generating the bounding box can improve speed and accuracy compared to executing the selection rules on the final output of the segmentation and annotation apparatus.

[0063] Combine the following Figure 8 and Fig. 9 The structure of an image processing device according to still another embodiment of the present disclosure will be described. Figure 8 is a block diagram illustrating a structure of an image processing apparatus 800 according to yet another embodiment of the present disclosure. Fig. 9 The present invention illustrates the operations of a mask template acquisition unit and a contour template acquisition unit in an image processing device according to another embodiment of the present disclosure.

[0064] like Figure 8 As shown, the image processing device 800 according to another embodiment of the present disclosure may include an information extraction unit 810, a first acquisition unit 820, a second acquisition unit 830, a selection unit 840, a labeling unit 850, and a contour training unit 860. The first acquisition unit 820 may include a head prediction unit 821 and a mask template acquisition unit 822, and the contour training unit 860 may include a contour template acquisition unit 861, a contour image library 862, and a contour comparison unit 863.

[0065] It should be noted that Figure 8 The information extraction unit 810, the first acquisition unit 820, the second acquisition unit 830, the selection unit 840 and the labeling unit 850 correspond to Figure 1 The information extraction unit 110, the first acquisition unit 120, the second acquisition unit 130, the selection unit 140 and the labeling unit 150 shown in FIG. Figure 3The information extraction unit 310, the first acquisition unit 320, the second acquisition unit 330, the selection unit 340 and the labeling unit 350 shown in FIG. 3 are not shown in FIG. 3, and thus the description thereof will be appropriately omitted. Figure 8 The head prediction unit 821 and the mask template acquisition unit 822 shown in FIG. Figure 5 The head prediction unit 510 and the mask template acquisition unit 520 shown in FIG. 5 are not shown in FIG. 5 , so the description thereof will be appropriately omitted.

[0066] During the use of the contour training unit 860 , the information extraction unit 810 , the first acquisition unit 820 and the contour training unit 860 may constitute a training module to optimize the parameters of the image processing device 800 (especially some parameters of the mask template).

[0067] First, the information extraction unit 810 can obtain image information about the input image. Figure 4 The configuration of the information extraction section 400 is described in detail, which is also applicable to the information extraction section 810 .

[0068] Here, the information extraction unit 810 may provide the image information to the first acquisition unit 820 and the contour training unit 860. For example, the image information provided by the information extraction unit 810 may be Figure 4 Feature maps of different scales in , such as multi-layer feature maps P3 to P7.

[0069] In the first acquisition unit 820, Figure 5 Similar to the head prediction unit 510 in , the head prediction unit 821 can obtain information such as mask coefficients, contour coefficients and confidence levels based on image information.

[0070] In the first acquisition unit 820, Figure 5 Similar to the mask template acquisition unit 520 in , the mask template acquisition unit 822 can obtain the mask template such as the mask basis vector based on the image information. For example, the mask template acquisition unit 822 can obtain the mask template such as the mask basis vector based on the feature map P3.

[0071] The contour training unit 860 can obtain contour templates such as contour basis vectors, and learn and optimize mask coefficients, mask templates such as mask basis vectors, contour coefficients and contour templates such as contour basis vectors through existing contour image libraries of various objects to improve the edge accuracy of the object to be labeled.

[0072] Specifically, in the contour training unit 860, the contour template acquisition unit 861 may obtain the contour template based on the image information provided by the information extraction unit 810. For example, the contour template acquisition unit 861 may obtain the contour template such as the contour basis vector based on the feature map P3.

[0073] In addition, the contour training unit 860 can calculate the contour of the object in the input image based on the contour basis vector obtained by the contour template acquisition unit 861 and the contour coefficient obtained by the head prediction unit 821. More specifically, the contour of the object in the input image can be calculated by multiplying the contour basis vector with the contour coefficient. It should be noted that the calculated object contour here refers to the contours of all objects in the input image during the training process.

[0074] The contour comparison unit 863 can compare the calculated contour of the object with the contour of the same category object in the contour image library 862, and calculate comparison information such as a loss function. The contour image library 862 may include contour images of various objects. In addition, the contour comparison unit 863 can feed back the calculated comparison information to the contour template acquisition unit 861 and the head prediction unit 821 to optimize the mask coefficients, mask basis vectors, contour coefficients and contour basis vectors. Fig. 9 How the outline template acquisition unit 861 optimizes the mask template by comparing information is described in detail.

[0075] Finally, the mask template acquisition unit 822 stores the optimized parameters of the mask template, such as some parameters of the mask basis vector, for use in segmentation and labeling of objects of the same category.

[0076] It should be noted that the contour training unit 860 is used to optimize the image processing device according to the present disclosure, especially some parameters of the mask template to be obtained by the mask template acquisition unit in the image processing device. After determining these parameters of the optimized mask template, the contour training unit 860 is no longer needed when segmenting and annotating the input image including objects of the same category. In other words, for objects of different categories, the mask templates can be optimized separately by the contour training unit 860.

[0077] After the training module optimizes the parameters of the mask template, the information extraction unit 810, the first acquisition unit 820, the second acquisition unit 830, the selection unit 840 and the labeling unit 850 can constitute the image processing device according to the present application.

[0078] First, the information extraction unit 810 may obtain image information about the input image and click information about a single click of a user on a specific object among a plurality of objects on the input image. Figure 4 The configuration of the information extraction section 400 is described in detail, which is also applicable to the information extraction section 810 .

[0079] Here, the information extraction unit 810 provides the image information and the click information to the first acquisition unit 820. For example, the image information provided by the information extraction unit 810 may be Figure 4In the example, the feature maps of different scales in the multi-layer feature maps P3 to P7 are extracted. In addition, the click information provided by the information extraction unit 810 may be the coordinate information of the point clicked by the user.

[0080] and Figure 5 Similar to the head prediction unit 510 in FIG. 8 , the head prediction unit 821 can obtain information such as mask coefficients, contour coefficients, bounding boxes, and confidence levels based on image information and click information. Figure 5 The selection rules of the bounding box introduced above are also applicable to the head prediction unit 821 .

[0081] In addition, the mask template acquisition unit 822 may obtain the mask template, such as the mask basis vector, based on the image information and the optimized parameters of the mask template. For example, the mask template acquisition unit 822 may obtain the mask basis vector based on the feature map P3 and the optimized parameters of the mask template.

[0082] Thus, the first acquisition unit 820 can calculate the original mask of the input image based on the optimized mask basis vectors obtained by the mask template acquisition unit 822 and the mask coefficients obtained by the head prediction unit 821. More specifically, the original mask of the input image can be calculated by multiplying the mask basis vectors by the mask coefficients.

[0083] Furthermore, the second acquisition unit 830 may use the original mask and the bounding box to obtain an intermediate mask that highlights the specific object from the input image.

[0084] Furthermore, the selection unit 840 may select the best mask from the plurality of intermediate masks. For example, the selection unit 840 may use the confidence calculated by the head prediction unit 821 as the score of the intermediate mask, and select the intermediate mask with the highest score as the best mask. In addition, the selection unit 840 may provide the selected best mask to the labeling unit 850.

[0085] Furthermore, the labeling unit 850 may label the edge of a specific object in the input image using the optimal mask.

[0086] In this application, in order to meet the edge accuracy requirements of the annotation tool, a contour prediction branch (contour training unit) is added to enrich the features of the mask. In addition, due to the real-time requirements, the selection rule is applied to the layer that generates the bounding box instead of the output of the entire image processing device to avoid a large amount of calculation in the non-maximum suppression (NMS) process.

[0087] Fig. 9 The diagram illustrates operations of a mask template acquisition unit and a contour template acquisition unit in an image processing apparatus according to yet another embodiment of the present disclosure.

[0088] like Fig. 9As shown, on the mask template acquisition part side, the feature map P3 is provided to the mask template acquisition part as input, and then the mask template acquisition part can perform convolution operations C11 and C12 based on the feature map P3, and provide the results of the operations to the contour template acquisition part.

[0089] On the side of the contour template acquisition unit, the contour template acquisition unit can add the operation result provided by the mask template acquisition unit and the feature map P3 as the input of the contour template acquisition unit. Then, the contour template acquisition unit can perform convolution operations C21 and C22 on its input and provide the result of the operation to the mask template acquisition unit. Here, the comparison information calculated by the contour comparison unit 863 is provided to the convolution operation C21, so as to feed back the comparison result to the contour template acquisition unit.

[0090] Further, the mask template acquisition unit can add the operation result after the convolution operation C11 and the operation result after the convolution operation C22 provided by the contour template acquisition unit. After that, the mask template acquisition unit can upsample the added result. For example, before upsampling, the size of the input feature map and each operation result is 69 pixels * 69 pixels, and after upsampling, the size of each operation result is 138 pixels * 138 pixels. In addition, the mask template acquisition unit can perform a convolution operation C13 on the result after upsampling, and output the operation result after the convolution operation C13 as the output of the mask template acquisition unit.

[0091] The contour template acquisition unit can upsample the operation result after the convolution operation C21. Similarly, for example, before upsampling, the size of the input feature map and each operation result is 69 pixels * 69 pixels, and after upsampling, the size of each operation result is 138 pixels * 138 pixels. In addition, the contour template acquisition unit can perform a convolution operation C23 on the result after upsampling, and output the operation result after the convolution operation C23 as the output of the contour template acquisition unit.

[0092] It should be noted that Fig. 9 The convolution operations in different boxes shown in the figure can be the same operation algorithm, and the convolution operation in any box is not limited to only one convolution operation, and multiple convolution operations can be performed. The difference is that if there are fewer convolution operations in any box, the amount of operation is small, but the features obtained by the convolution operation may not be rich enough, and if there are relatively more convolution operations in any box, richer features can be obtained.

[0093] Thus, by fusing the features of the mask template acquisition unit and the contour template acquisition unit, the image processing device according to the present application can simultaneously learn the mask template and the contour template. The contour features extracted in the contour branch (contour training unit) are used to guide the prediction of the mask, which enriches the shape information and improves the edge accuracy.

[0094] Fig.10 1 is a block diagram illustrating the structure of another first acquisition unit in the image processing apparatus according to an embodiment of the present disclosure. The first acquisition unit 1000 may include a head prediction unit 1010, a bounding box removal unit 1030, and a mask template acquisition unit 1020. Figure 5 Compared with the first acquisition unit 500 shown in FIG. 1 , the first acquisition unit 1000 may further include a bounding box removal unit 1030 . Fig.10 The head prediction unit 1010 and the mask template acquisition unit 1020 shown in FIG. 1 correspond to Figure 5 The head prediction unit 510 and the mask template acquisition unit 520 shown in FIG. 1 are not shown in FIG. 1 , so the description thereof will be omitted appropriately. In addition, the first acquisition unit 1000 is the same as the first acquisition unit 500 and can be applied to Figure 1 , Figure 3 and Figure 8 In each image processing device shown.

[0095] Similar to the head prediction unit 510, the head prediction unit 1010 can obtain information such as mask coefficients, contour coefficients, bounding boxes and confidence levels based on image information and click information.

[0096] For example, Figure 7 As shown, the head prediction unit 1010 may generate 9 bounding boxes for each pixel of each feature map in the feature maps P3 to P7, and may select a bounding box including the click point from the bounding boxes. Figure 7 725 bounding boxes containing click points are selected.

[0097] The bounding box removal unit 1030 may be configured to perform non-maximum suppression processing on the bounding box including the click point to remove redundant bounding boxes. At the same time, the bounding box removal unit 1030 may record the index of the remaining bounding boxes to select corresponding other information, such as mask coefficients, contour coefficients, and confidence information.

[0098] For example, the bounding box removal unit 1030 may perform non-maximum suppression processing on the selected 725 bounding boxes containing click points to remove redundant bounding boxes, thereby obtaining reduced bounding boxes, such as dozens of bounding boxes, and corresponding mask coefficients.

[0099] Similar to the mask template acquisition unit 520 , the mask template acquisition unit 1020 may obtain a mask template, such as a mask basis vector, based on image information.

[0100] Thus, the first acquisition unit 1000 can calculate the original mask of the input image based on the mask basis vectors obtained by the mask template acquisition unit 1020 and the mask coefficients further filtered by the bounding box removal unit 1030. More specifically, the original mask of the input image can be calculated by multiplying the mask basis vectors by the mask coefficients.

[0101] The first acquisition unit 1000 may provide the filtered bounding box and the calculated corresponding original mask to the second acquisition unit.

[0102] Fig.11 FIG. 4 is a block diagram illustrating the structure of a second acquisition unit in an image processing apparatus according to an embodiment of the present disclosure. Fig.11 As shown, the second acquisition unit 1100 may include a noise removal unit 1110 and an intermediate mask acquisition unit 1120. The second acquisition unit 1100 may be applied to Figure 1 , Figure 3 and Figure 8 In each image processing device shown.

[0103] The second acquisition unit 1100 may receive, from the first acquisition unit, a bounding box including the click point (eg, a filtered bounding box including the click point) and an original mask corresponding to the bounding box.

[0104] Specifically, the noise removing unit 1110 may remove noise in other regions of the original mask except for the bounding box.

[0105] Furthermore, the intermediate mask acquisition unit 1120 may binarize the data of the original mask in the boundary frame of the original mask from which noise has been removed, thereby obtaining an intermediate mask.

[0106] Thus, the second acquisition unit 1100 can obtain an intermediate mask that highlights the specific object from the input image by using the original mask and the bounding box.

[0107] Combine the following Fig.12 An image processing method according to an embodiment of the present disclosure is described below.

[0108] like Fig.12 As shown, the image processing method according to the embodiment of the present disclosure starts at step S110. In step S110, image information about the input image and click information about a single click of a user on a specific object among multiple objects on the input image are acquired.

[0109] Next, in step S120, a bounding box including the click point and an original mask corresponding to the bounding box are obtained based on the image information and the click information.

[0110] Next, in step S130 , an intermediate mask that highlights the specific object from the input image is obtained using the original mask and the bounding box.

[0111] Next, in step S140 , an optimal mask is selected from a plurality of intermediate masks.

[0112] Next, in step S150, the edge of the specific object is marked in the input image using the optimal mask, after which the process ends.

[0113] According to an embodiment of the present disclosure, the image information is a multi-layer feature map of an input image, and the method also includes: generating a predetermined number of bounding boxes of different sizes for each pixel of each layer of the multi-layer feature map; and selecting a bounding box including a click point from the generated bounding boxes.

[0114] According to an embodiment of the present disclosure, the method further includes: performing non-maximum suppression processing on the selected bounding box including the click point to remove redundant bounding boxes.

[0115] According to an embodiment of the present disclosure, the method further includes: obtaining an original mask by calculating mask coefficients and a mask template (mask basis vector) based on image information.

[0116] According to an embodiment of the present disclosure, the method also includes: calculating the contour coefficient and the contour template (contour basis vector) based on the image information; during the training process, the mask coefficient, mask basis vector, contour coefficient and contour basis vector are learned and optimized through the existing contour image library of various objects to improve the edge accuracy of the object to be labeled.

[0117] According to an embodiment of the present disclosure, the step of obtaining the intermediate mask includes: removing noise in other areas of the original mask except for the bounding box and binarizing the data of the original mask in the bounding box, thereby obtaining the intermediate mask.

[0118] According to an embodiment of the present disclosure, the step of acquiring image information includes: using ResNet101 to extract features of an input image; and using a feature pyramid module to generate feature maps of different scales as image information.

[0119] According to an embodiment of the present disclosure, before acquiring the image information, the method further includes: resizing and normalizing the input image.

[0120] According to an embodiment of the present disclosure, the step of acquiring click information includes acquiring the coordinates of the user's click.

[0121] According to an embodiment of the present disclosure, the coordinates clicked by the user are normalized.

[0122] Thus, the image processing method according to the embodiment of the present disclosure can enable the user to obtain the edge of the clicked object with only a single click. In addition, by executing the selection rule on the bounding box instead of executing the selection rule on multiple final outputs of the object annotation, the annotation efficiency and accuracy can be improved.

[0123] Various specific implementations of the above steps of the image processing method according to the embodiment of the present disclosure have been described in detail above and will not be repeated here.

[0124] Obviously, each operation process of the image processing method according to the present disclosure can be implemented in the form of a computer executable program stored in various machine-readable storage media.

[0125] Moreover, the purpose of the present disclosure can also be achieved in the following manner: a storage medium storing the above-mentioned executable program code is directly or indirectly provided to a system or device, and a computer or a central processing unit (CPU) in the system or device reads out and executes the above-mentioned program code. In this case, as long as the system or device has the function of executing the program, the embodiment of the present disclosure is not limited to the program, and the program can also be in any form, for example, a target program, a program executed by an interpreter, or a script program provided to an operating system, etc.

[0126] The above-mentioned machine-readable storage media include, but are not limited to: various memories and storage units, semiconductor devices, disk units such as optical, magnetic and magneto-optical disks, and other media suitable for storing information.

[0127] In addition, the technical solution of the present disclosure can also be implemented by connecting a computer to a corresponding website on the Internet, downloading and installing the computer program code according to the present disclosure into the computer, and then executing the program.

[0128] Fig.13 is a block diagram of an exemplary structure of a general-purpose personal computer in which the image processing apparatus and method according to the embodiments of the present disclosure can be implemented.

[0129] like Fig.13 As shown, a CPU 1301 performs various processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage section 1308 to a random access memory (RAM) 1303. In the RAM 1303, data required when the CPU 1301 performs various processes and the like is also stored as needed. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output interface 1305 is also connected to the bus 1304.

[0130] The following components are connected to the input / output interface 1305: an input section 1306 (including a keyboard, a mouse, etc.), an output section 1307 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage section 1308 (including a hard disk, etc.), a communication section 1309 (including a network interface card such as a LAN card, a modem, etc.). The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 may also be connected to the input / output interface 1305 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 1310 as needed, so that a computer program read therefrom is installed in the storage section 1308 as needed.

[0131] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1311 .

[0132] It should be understood by those skilled in the art that such storage media is not limited to Fig.13 The removable medium 1311 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be the ROM 1302, a hard disk included in the storage section 1308, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0133] In the system and method of the present disclosure, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present disclosure. In addition, the steps of performing the above series of processes can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0134] Although the embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings, it should be understood that the embodiments described above are only used to illustrate the present disclosure and do not constitute a limitation of the present disclosure. For those skilled in the art, various modifications and changes can be made to the above embodiments without departing from the essence and scope of the present disclosure. Therefore, the scope of the present disclosure is limited only by the attached claims and their equivalent meanings.

[0135] Regarding the implementation methods including the above embodiments, the following additional notes are also disclosed:

[0136] Note 1. An image processing device, comprising:

[0137] an information extraction unit that acquires image information about an input image and click information about a single click of a user on a specific object among a plurality of objects on the input image;

[0138] A first acquisition unit, which obtains a bounding box including a click point and an original mask corresponding to the bounding box based on the image information and the click information;

[0139] a second acquisition unit, which uses the original mask and the bounding box to obtain an intermediate mask that highlights the specific object from the input image;

[0140] A selection unit that selects an optimal mask from a plurality of intermediate masks; and

[0141] A marking unit marks the edge of the specific object in the input image using the optimal mask.

[0142] Supplement 2. The image processing device according to Supplement 1, wherein the image information is a multi-layer feature map about the input image, and the first acquisition unit is configured to:

[0143] Generating a predetermined number of bounding boxes of different sizes for each pixel of each layer of feature maps in the multi-layer feature maps; and

[0144] From among the generated bounding boxes, select the bounding box that includes the click point.

[0145] Note 3. The image processing device according to Note 2, wherein the first acquisition unit is further configured to perform non-maximum suppression processing on the selected bounding box including the click point to remove redundant bounding boxes.

[0146] Supplement 4. The image processing device according to Supplement 1, wherein the first acquisition unit obtains the original mask by calculating mask coefficients and mask basis vectors based on the image information.

[0147] Supplement 5. The image processing device according to Supplement 4, wherein the first acquisition unit is further configured to: calculate a contour coefficient based on the image information, and

[0148] The image processing device also includes: a contour training unit, which obtains a contour basis vector, and learns and optimizes the mask coefficient, the mask basis vector, the contour coefficient and the contour basis vector through an existing contour image library of various objects to improve the edge accuracy of the object to be marked.

[0149] Note 6. The image processing device according to Note 1, wherein the second acquisition unit is configured to remove noise in other areas of the original mask except for the bounding box and binarize the data of the original mask in the bounding box, thereby obtaining the intermediate mask.

[0150] Note 7. The image processing device according to Note 1, wherein the information extraction unit is configured to: use ResNet101 to extract features of the input image; and use a feature pyramid module to generate feature maps of different scales as the image information.

[0151] Note 8. The image processing device according to Note 1 further includes: a preprocessing unit, which is configured to: resize and normalize the input image, and normalize the coordinates of the user's click.

[0152] Note 9. An image processing method, comprising:

[0153] acquiring image information about an input image and click information about a single click of a user on a specific object among a plurality of objects on the input image;

[0154] Obtaining a bounding box including the click point and an original mask corresponding to the bounding box based on the image information and the click information;

[0155] Obtaining an intermediate mask that highlights the specific object from the input image using the original mask and the bounding box; and

[0156] An optimal mask is selected from a plurality of intermediate masks, and the edge of the specific object is marked in the input image using the optimal mask.

[0157] Note 10. The method according to Note 9, wherein the image information is a multi-layer feature map of the input image, and the method further comprises:

[0158] Generating a predetermined number of bounding boxes of different sizes for each pixel of each layer of feature maps in the multi-layer feature maps; and

[0159] From among the generated bounding boxes, select the bounding box that includes the click point.

[0160] Note 11. The method according to Note 10 further includes:

[0161] The selected bounding box including the click point is subjected to non-maximum suppression processing to remove redundant bounding boxes.

[0162] Note 12. The method according to Note 9, wherein the original mask is obtained by calculating mask coefficients and mask basis vectors based on the image information.

[0163] Note 13. The method according to Note 12 further includes:

[0164] Calculate the contour coefficient and the contour basis vector based on the image information;

[0165] The mask coefficient, the mask basis vector, the contour coefficient and the contour basis vector are learned and optimized through an existing contour image library of various objects to improve the edge accuracy of the object to be marked.

[0166] Note 14. The method according to Note 9, wherein the step of obtaining the intermediate mask comprises: removing noise in other areas of the original mask except for the bounding box and binarizing the data of the original mask in the bounding box, thereby obtaining the intermediate mask.

[0167] Note 15. The method according to Note 9, wherein the step of obtaining the image information comprises:

[0168] ResNet101 is used to extract features of the input image; and

[0169] A feature pyramid module is used to generate feature maps of different scales as the image information.

[0170] Note 16. The method according to Note 9, wherein before acquiring the image information, it also includes: resizing and normalizing the input image.

[0171] Note 17. The method according to Note 9, wherein the step of obtaining the click information includes obtaining the coordinates of the user's click.

[0172] Note 18. The method according to Note 17, wherein the coordinates of the user's click are normalized.

[0173] Note 19. The method according to Note 9, wherein the edge of the specific object is marked in the input image by extracting the contour of the specific object from the optimal mask.

[0174] Note 20. A machine-readable storage medium carrying a program product including machine-readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, enable the computer to perform the image processing method described in Notes 9-19.

Claims

1. An image processing device, comprising: an information extraction unit that acquires image information about an input image and click information about a single click of a user on a specific object among a plurality of objects on the input image; A first acquisition unit, which obtains a bounding box including a click point and an original mask corresponding to the bounding box based on the image information and the click information; a second acquisition unit, which uses the original mask and the bounding box to obtain an intermediate mask that highlights the specific object from the input image; A selection unit that selects an optimal mask from a plurality of intermediate masks; and A marking unit marks the edge of the specific object in the input image using the optimal mask.

2. The image processing device according to claim 1, wherein: The image information is a multi-layer feature map of the input image, and the first acquisition unit is configured to: Generating a predetermined number of bounding boxes of different sizes for each pixel of each layer of feature maps in the multi-layer feature maps; and From among the generated bounding boxes, select the bounding box that includes the click point.

3. The image processing device according to claim 2, wherein: The first acquisition unit is further configured to perform a non-maximum suppression process on the selected bounding box including the click point to remove redundant bounding boxes.

4. The image processing device according to claim 1, wherein: The first acquisition unit obtains the original mask by calculating mask coefficients and mask basis vectors based on the image information.

5. The image processing device according to claim 4, wherein: The first acquisition unit is further configured to: calculate a contour coefficient based on the image information, and The image processing device also includes: a contour training unit, which obtains a contour basis vector, and learns and optimizes the mask coefficient, the mask basis vector, the contour coefficient and the contour basis vector through an existing contour image library of various objects to improve the edge accuracy of the object to be marked.

6. The image processing device according to claim 1, wherein: The second acquisition unit is configured to remove noise in other areas of the original mask except for the bounding box and binarize the data of the original mask in the bounding box, thereby obtaining the intermediate mask.

7. The image processing device according to claim 1, wherein: The information extraction unit is configured to: extract features of the input image using ResNet101; And a feature pyramid module is used to generate feature maps of different scales as the image information.

8. The image processing apparatus according to claim 1, further comprising: The preprocessing unit is configured to: resize and normalize the input image, and normalize the coordinates of the user's click.

9. An image processing method, comprising: acquiring image information about an input image and click information about a single click of a user on a specific object among a plurality of objects on the input image; Obtaining a bounding box including the click point and an original mask corresponding to the bounding box based on the image information and the click information; Obtaining an intermediate mask that highlights the specific object from the input image using the original mask and the bounding box; as well as An optimal mask is selected from a plurality of intermediate masks, and the edge of the specific object is marked in the input image using the optimal mask.

10. A machine-readable storage medium carrying a program product including machine-readable instruction codes stored therein, wherein: When the instruction code is read and executed by a computer, it can cause the computer to execute the image processing method according to claim 9.

Images