Image target object assisted labeling method based on point labeling

Abstract

The application discloses a kind of based on point mark image target object auxiliary marking method, comprising: a kind of image pre-segmentation method, semi-automatic generation target object mask method.Based on connected region segmentation algorithm, image is pre-segmented, and superpixel macroblock is obtained;Then the representative color of each superpixel macroblock and id are filled in the pixel points covered by it to generate pre-segmentation image and pre-segmentation mask respectively;Manual distinguishes target object on the pre-segmentation image of front interface and marks point, and the mapping relationship of superpixel macroblock id and target object id is automatically generated according to the marking point;Based on this mapping relationship, target object mask output is obtained, and the meta information of mark point is also output.The method for generating mask and marking point meta information that can be used for supervised image segmentation training in the application only needs simple manual marking, and provides sufficient data support for supervised learning model in image segmentation field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision image segmentation, in particular to an image target object auxiliary labeling method based on point labeling. BACKGROUND

[0002] With the increasing breakthroughs of Visual Transformer technology in the field of computer vision in recent years, super large-scale neural network models and super large-scale labeled image datasets have become increasingly necessary conditions in the field of computer vision. The labeling method based on point connection of software packages such as LabelImg draws a point connection on the edge of the target object in the image to depict the edge of the object. Its limitation is that for objects with slightly complex shapes, it may take hundreds of point connections to complete the labeling of a single target object in an image, increasing the difficulty of manual labeling. GrabCut is an effective image segmentation algorithm for extracting foreground objects from complex backgrounds, which has been integrated into the open source software package OpenCV known to all in the field of computer vision. However, the labeling method based on GrabCut also has limitations: first, it has a very high computational time complexity, and it consumes a huge amount of computation when processing high-definition large-size images; second, it usually needs to satisfy that the manually drawn frame contains a complete foreground image; third, the human-computer interaction image labeling method based on GrabCut improvement needs multiple manual interventions to recalibrate multiple ranges in order to modify the segmentation result.

[0003] Point labeling is a faster and more efficient image labeling method, but the prerequisite for point labeling is that the objects in the image have been segmented, so that the entire image object can be selected by labeling any pixel point on the image target object. Therefore, how to efficiently implement image segmentation or segmentation based on pixels is the key to reducing the cost of image manual labeling for auxiliary point labeling. Connected component analysis and hyper pixel are classic methods for image segmentation at the pixel scale, which do not require training of neural network models; and based on the connected component search algorithm, the computational time complexity is O(n). SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. The present application provides an image target object auxiliary labeling method based on point labeling, which can generate a target object mask in an image in a semi-automatic and only point labeling manner.

[0005] The present application provides an image pre-segmentation method, which comprises:

[0006] Step S01 establishes an empty mask of the same size as the original image, and the initial value of all elements in the mask is 0;

[0007] Step S02 sets the global maximum label value as 0;

[0008] Step S03 traverses all elements of the above array as the starting point of each nearest neighbor search, if the starting point is marked as zero, the current maximum label value + 1 is taken as the new label value of the starting point, and then the current connected region is expanded in all directions, otherwise the current starting point is skipped;

[0009] Step S04, when performing the connected region search algorithm, the similarity between the adjacent next pixel and the current pixel is calculated when propagating in each direction, if the similarity reaches the threshold, the propagation is continued, otherwise the direction propagation is stopped and the current pixel point is defined as a boundary point;

[0010] Step S05, when all propagation directions of a connected region are stopped, the connected region marked with the same label is called a superpixel connected region;

[0011] Step S06, optionally, regions with an area greater than a threshold s can be selected from the set of all superpixel connected regions as superpixel macroblocks, and all superpixel macroblocks are recompiled as superpixel macroblock ids or macroblock ids, or superpixel connected regions can be directly used as superpixel macroblocks, and connected region ids are used as macroblock ids;

[0012] Step S07, each superpixel macroblock takes the color with the most pixels in the macroblock as the representative color of the entire macroblock;

[0013] Step S08, all superpixel macroblocks together form a superpixel macroblock set of the current image;

[0014] Step S09, the labels of the superpixel macroblock set are integrated into the mask, that is, the macroblock id is filled into the 2d array of the same size as the image, and the default value of the label of the empty area is 0;

[0015] Step S10, the color of the superpixel macroblock set is integrated into a new image, that is, the macroblock representative color replaces the original color of the pixel in the new image;

[0016] Step S11, an empty mask is constructed, and all macroblocks in the superpixel macroblock set are traversed, and the coordinates of the pixels covered by each macroblock are filled into the corresponding coordinate positions of the empty mask to generate a pre-segmentation mask.

[0017] The application also provides a macroblock expansion method based on connected region search, comprising:

[0018] Step S12, all empty area pixels are traversed as starting points for performing connected region search, and in the search process, the pixels in the neighborhood are integrated into the surrounding macroblocks based on the steps S13 and S14.

[0019] The step S13 has a weighted mode of reference points: taking the pixel corresponding to the current empty point pixel position in the original image as a reference pixel, recording the pixel points in the neighborhood within the distance r on the weighted macro block id, but excluding the points whose similarity between the representative color of the macro block and the color of the reference pixel is lower than the set threshold;

[0020] The step S14 sorts the weights of the macro block ids in the neighborhood, takes the macro block id with the highest weight as the label of the current empty point, and takes the representative color of the macro block as the new color of the empty point;

[0021] The step S15 iterates the steps S12-S14 for n1 rounds;

[0022] The step S16 integrates the labels of the super-pixel macro block set into a new mask, and integrates the representative colors of the super-pixel macro block set into a new image.

[0023] The application also provides a semi-automatic generation method of target mask based on point labeling, comprising:

[0024] The step S17 loads the image and the super-pixel macro block mask in the computer front end, displays the image, and manually labels a point on each color block with different colors that can be distinguished by naked eyes on the image;

[0025] The step S18 finds the label (the label is the aforementioned super-pixel macro block id) on the corresponding coordinate position on the pre-segmentation mask by using the coordinates of the points on various target objects labeled by the front-end software or device, and establishes a mapping relationship of the super-pixel macro block id pointing to the target object id;

[0026] The step S19 collects the mapping relationship and outputs it into a mapping dictionary;

[0027] The step S20 collects the positions of all the labeled points and the target object ids, and outputs them into labeled point metadata;

[0028] The step S21 can be realized in a local html script or an online webpage service.

[0029] The step S22 can be a data file in any format such as json, xml, and yaml, or can directly enter the network online service platform backend in the form of a data stream;

[0030] The step S23 can be a data file in any format such as json, xml, and yaml, or can directly enter the network online service platform backend in the form of a data stream;

[0031] Step S24 constructs an empty mask as an initial mask, and assigns an initial value of a full image uniform background id to elements in the initial mask;

[0032] Step S25, for all superpixel macroblock ids found in the mapping dictionary, assigns a target object id to the corresponding superpixel macroblock covering the full coordinate position in the corresponding position of the new mask;

[0033] Step S26 outputs the initial mask as a target object mask;

[0034] Step S27, optionally, the output target object mask can be a json, xml, yaml, bitmap, etc. Any format data file, or directly in the form of a data stream into the network online service platform backend.

[0035] Additional aspects and advantages of the application will be described in part below, some will become apparent from the following description, or will be understood by those skilled in the art through practice of the application. The above general description and the following detailed description are only exemplary and explanatory, but not limiting the application. BRIEF DESCRIPTION OF DRAWINGS

[0036] The drawings incorporated into the specification and form part of the specification, which show embodiments consistent with the application, and together with the specification used to illustrate the technical solutions of the application.

[0037] Figure 1 A structural schematic diagram of an electronic device provided for the auxiliary labeling method embodiment of the application;

[0038] Figure 2 Flowchart of the auxiliary labeling method embodiment 1 of the application;

[0039] Figure 3 Flowchart of the auxiliary labeling method embodiment 2 of the application;

[0040] Figure 4 Flowchart of the pre-segmentation embodiment 3 of the embodiment of the application;

[0041] Figure 5 Flowchart of the macroblock expansion embodiment 4 of the embodiment of the application;

[0042] Figure 6 Flowchart of the semi-automatic generation of the target object mask method embodiment 5 of the application;

[0043] Figure 7 Visualized image of the image after labeling and the output mask of the embodiment 1 of the application;

[0044] Figure 8 Visualized image of the image after labeling and the output mask of the embodiment 2 of the application. DETAILED DESCRIPTION

[0045] The application will be further described in conjunction with the accompanying drawings and embodiments. The embodiments provided herein are only used to explain the application and not intended to limit the application. In addition, the embodiments provided herein are part of the embodiments for implementing the application, and the technical solutions of the embodiments of the application can be implemented in any combination without conflict.

[0046] It should be noted that in the embodiments of the application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that the method or device comprising a series of elements not only includes the elements explicitly listed, but also includes other elements not explicitly listed, or includes elements inherent in the implementation of the method or device. Without more limitations, the element defined by the sentence "comprising a" does not exclude the presence of other related elements (for example, steps in the method or units in the device, such as the unit can be part of the circuit, part of the processor, part of the program or software, etc.) in the method or device comprising the element.

[0047] The term "and / or" herein is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of the plurality or any combination of at least two of the plurality, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0048] Embodiments of the application can be applied in a computer system composed of terminals and servers, and can operate with many other general-purpose or special-purpose computing system environments or configurations. Here, the terminal can be a client, a handheld or laptop device, a tablet computer, a microprocessor-based system, a set-top box, a programmable consumer electronics, a network personal computer, a minicomputer system, etc., and the server can be a server computer system, a minicomputer system, a mainframe computer system, and a distributed cloud computing technology environment including any of the above systems, etc. The terminals, servers and other electronic devices can be described in the general context of executable instructions, such as program modules, executed by the computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0049] In practical applications, the foregoing steps S01 to S27 can be implemented by using a processor in an electronic device, and the processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), an FPGA, a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor.

[0050] Exemplarily, the to-be-processed image can be obtained from a local storage or a mobile device or a network, and the to-be-processed image can be in a format of a picture JPEG (Joint Photographic Experts GROUP), a bitmap BMP (Bitmap), a portable network graphics PNG (Portable Network Graphics), or an array file in a 2d space single channel (for example, a grayscale image) or a multi-channel (RGB, HSV, etc.), or other formats. It should be noted that, here, only the format and source of the to-be-processed image are exemplarily described, and the embodiment of the application does not limit the format and source of the to-be-processed image.

[0051] With the development of Artificial Intelligent technology, Computer Vision technology has made greater and greater breakthroughs and gradually entered the real application. After Convolution Neural Network (CNN), since 2019, Visual Transformer has become the most popular deep learning model in the field of computer vision. With the development of the model and the improvement of computing power, the field of computer vision has entered the era of super large scale neural network. Image segmentation is one of the main research directions in the field of computer vision, and at present it mainly relies on training large-scale or even super large-scale neural network model with a large number of images and their labels and mask data. Take SAM (Segment Anything Model) announced by Meta (formerly Facebook) in April 2023 as an example, it uses the ViT architecture model announced by Google Brain previously, and the largest ViT-H model reaches a super large scale of 390 million parameters, and in order to train the model, about 20 million high-definition images and their labels and target mask are used. In order to obtain so many image target masks, Meta company relies on its financial support and hires a large number of annotators to complete the preparation of the data set.

[0052] Mask, also known as mask, English name mask, in the field of computer vision refers to using selected graphics, images to partially block the image to be processed, to control the area of image processing. In the related technology, Labelme and LabelImg are early image annotation software in the field of computer vision. By clicking the mouse, draw a dotted line on the edge of the target object in the image, accurately draw the outline of the target object. The target object is segmented by a dotted line to obtain the target mask of the image. This annotation process requires fine and long manual annotation operation.

[0053] The annotation of images currently uses GrabCut or a pre-trained model or a combination of the two; it better realizes the separation of foreground and background, is an image semantic segmentation technology, and has been integrated into the OpenCV, an open source software toolkit well known in the computer vision field. To use GrabCut, only an image containing a target needs to be input, a box is drawn on the image to enclose the foreground object, and the foreground target can be separated from the background image. However, the main problem of the annotation method based on it is that it needs O(n^2) or O(nlogn) computational time complexity for an image containing n pixels; the target object must be within the box; and in the case that the foreground and background colors are similar or the box does not completely contain the target object, there will be obvious misclassification. Therefore, in the semi-automatic annotation method based on GrabCut, in order to eliminate this part of misclassification, multiple manual interventions are needed to re-label multiple ranges.

[0054] Four-connected search is a classic implementation of a connected region search algorithm, and the corresponding eight-connected search is also a plane search algorithm that expands from a search starting point to the surrounding. In order to meet the demand of a super large scale image mask data set in the field of computer vision, a more efficient image target annotation method has become a bottleneck. Therefore, in order to solve the above technical problems, an image target annotation method based on point annotation is implemented in the embodiment of the present application, so as to efficiently and semi-automatically generate an image target mask.

[0055] Figure 1 An electronic device 100 provided by the embodiment of the present application is shown, which includes a processor 101 and a memory 102. The processor and the memory can be connected through a bus or other means, Figure 1 The memory 102 as a non-transitory computer readable storage medium can be used to store non-transitory software programs and non-transitory computer executable programs. The memory 102 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state storage device. In addition, in some embodiments, the memory 102 can optionally include a memory 102 remotely arranged with respect to the processor, and these remote memories can be connected to the electronic device 100 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0056] Figure 2 The flowchart of the embodiment 1 of the present application is shown, and the flow includes:

[0057] First, a picture is taken out from an image data set.

[0058] Second step, image pre-segmentation, realized by superpixel region segmentation, specifically including: establishing a 2d array with the same scale as the original image, all elements initial value is 0, indicating that all elements are not included in any connected region, setting the global maximum label value as 0; traversing all element positions of the above array, if the label of the point is 0, then the current maximum label value + 1 is taken as the label value of the new connected region, and the color of the point is taken as the representative color of the connected region, then the point is taken as the search starting point and the connected region search algorithm is expanded in all directions; when a non-similar adjacent pixel is encountered in each direction, the propagation in the direction is stopped, and the current pixel position is recorded as the boundary, otherwise the adjacent pixel is included in the current connected region and the recursive propagation is continued; when all boundary pixel points of a connected domain stop continuing to propagate outward, the search returns to the root node along the search tree, that is, the search starting point of this search is returned; based on the selected superpixel connected region area threshold s, all superpixel connected regions are filtered, and superpixel connected regions with an area greater than the threshold s are selected as superpixel macroblocks, and the color with the most number in each superpixel macroblock is taken as the representative color of the entire macroblock; all superpixel macroblocks together form a superpixel macroblock set of the current image.

[0059] Third step, human-computer interaction labeling, specifically including: loading the pre-segmentation image and the pre-segmentation mask in the computer html webpage, manually drawing a point on the image target object as a labeling point, and the webpage automatically captures the pixel position of the labeling point on the image as the meta information of the labeling point; finding the label corresponding to the position in the pre-segmentation mask according to the pixel position of the labeling point, that is, the superpixel macroblock id; outputting the mapping dictionary of the superpixel macroblock id to the target object id; finding the mapping dictionary to replace the macroblock id in the pre-segmentation mask with the target object id, and replacing the id that cannot be replaced by searching with 0 as the background id, as the final output mask.

[0060] Fourth step, the output target object mask json file and the meta information json file of the labeling point are included in the labeling data set.

[0061] Figure 3 For the flowchart of embodiment 2 of the application, the flow includes:

[0062] First step, taking a picture from the image data set.

[0063] Second step, image pre-segmentation, is realized by superpixel region segmentation, specifically comprising: establishing a 2d array of the same size as the original image, all elements of which are initially 0, indicating that all elements are not included in any connected region, and setting the global maximum label value as 0; traversing all element positions of the above array, if the label of the point is 0, then the current maximum label value + 1 is taken as the label value of the new connected region, and the color of the point is taken as the representative color of the connected region, and then the point is taken as the search starting point to expand in all directions according to the connected region search algorithm; when a non-similar adjacent pixel is encountered in each direction, the propagation in the direction is stopped, and the current pixel position is recorded as the boundary, otherwise the adjacent pixel is included in the current connected region and the recursive propagation is continued; when all boundary pixels of a connected domain stop continuing to propagate outward, the search returns to the root node along the search tree, that is, the search starting point of this search; based on the selected superpixel connected region area threshold s, all superpixel connected regions are filtered, and superpixel connected regions with an area greater than the threshold s are selected as superpixel macroblocks, and the color with the most number in each superpixel macroblock is taken as the representative color of the entire macroblock; all superpixel macroblocks together form a superpixel macroblock set of the current image.

[0064] Third step, macroblock expansion, specifically comprising: traversing all pixels of the pre-segmentation image empty area, performing four-connected search based on the weighted reference point of the foregoing step S13; each time all empty pixels are traversed once, it is considered as a completed iteration, and the continuous iteration is not more than n=10 times; taking the updated image and the mask as a new image and a new mask.

[0065] Fourth step, human-computer interaction labeling, specifically comprising: loading the new image and the new mask in the computer html webpage, manually drawing a point on the image target object as a labeling point, and the webpage automatically capturing the pixel position of the labeling point on the image as the meta information of the labeling point; finding the label of the corresponding position in the new mask according to the pixel position of the labeling point, that is, the superpixel macroblock id; outputting the mapping dictionary of the superpixel macroblock id to the target object id; finding the mapping dictionary to replace the macroblock id in the pre-segmentation mask with the target object id, and replacing the id that cannot be found and replaced with 0 as the background id, as the final output mask.

[0066] Fifth step, the output target object mask json file and the meta information json file of the labeling point are included in the labeling data set.

[0067] Figure 4 For the flowchart of the pre-segmentation embodiment 3 of the embodiment of the application, specifically comprising:

[0068] First step, establish 2d array with the same scale as original image as empty mask, all elements initial value is 0, indicating that all elements are not in any connected region, set global maximum label value as 0, drop search pointer on the pixel of x=0 and y=0 position;

[0069] Second step, check whether the label value of the pixel where the current search pointer position is located is 0, if the label is not equal to 0, it indicates that the current pixel has been in the connected region corresponding to the label value in the earlier search, otherwise it indicates that a round of superpixel segmentation needs to be carried out with the pixel as the starting point of the search;

[0070] Fourth step, the superpixel connected region is filtered according to area, specifically including: based on the selected superpixel connected region area threshold s, the superpixel connected region with an area greater than the threshold s is selected from the set of all superpixel connected regions as a superpixel macroblock, and the number of colors in the macroblock is taken as the representative color of the entire macroblock;All superpixel macroblocks together form a superpixel macroblock set of the current image.

[0071] Fifth step, construct an empty mask with the same scale as the original image in x and y directions, all elements initial value is 0, traverse all superpixel macroblocks, fill the label of the pixel covered by the macroblock into the corresponding position in the pre-segmentation mask to obtain the output mask.

[0072] Sixth step: mark all pixels covered by the superpixel macroblock set as their respective representative colors, fill the representative color into a new image with the same frame as the original image, and the blank area not belonging to any macroblock takes the default value (0, 0, 0) of RGB pixel, to obtain the output image.

[0073] Figure 5 For the flowchart of embodiment 4 of the macroblock expansion of the embodiment of the application, this link traverses all blank pixels of the pre-segmentation image, specifically including:

[0074] First step, drop the pointer on the first pixel as the starting point of the search according to the plane arrangement order of the image;

[0075] Second step, check whether the current pixel point belongs to the blank area, which can be realized by comparing whether the current pixel RGB value is (0, 0, 0) or the label of the current pixel position in the pre-segmentation mask is 0;

[0076] Third step, find a pixel point in the neighborhood as a near neighbor point based on four-connected search;

[0077] Fourthly, the weighting is performed based on the weighting method with reference points in the step S13;

[0078] Fifthly, it is checked whether the current neighbor point is the last neighbor point in the neighborhood. If yes, the sixth step is performed. Otherwise, the third step is performed again.

[0079] Sixthly, it is checked whether the current search starting point is the last element in the image plane arrangement order. If yes, the seventh step is performed. Otherwise, the second step is performed again by moving to the next pixel point according to the pixel plane arrangement order.

[0080] Seventhly, the macro block id with the highest weight in the neighborhood is selected as the mask label of the vacancy point, and the color of the macro block is used to overwrite the color of the vacancy pixel point.

[0081] Eighthly, it is checked whether the iteration threshold is reached. If yes, the ninth step is performed. Otherwise, the first step is performed again.

[0082] Ninthly, a new image in jpeg format and a new mask in json format are output.

[0083] Figure 6 For the flowchart of the embodiment 5 of the semi-automatic target mask generation method of the application, it specifically includes:

[0084] Firstly, the target id is manually filled in the front-end interface, and the image and the super pixel macro block mask (i.e. the pre-segmentation mask / new mask generated in the previous step) are loaded and displayed locally.

[0085] Secondly, the points are manually marked on the distinguishable color blocks of the image target.

[0086] Thirdly, the computer finds the label (i.e. the super pixel macro block id) in the super pixel macro block mask at the same position as the marked point, and establishes the mapping relationship between the target id and the super pixel macro block id.

[0087] Fourthly, the computer collects all the marked points as the marked points, and the positions of the marked points and the target id are combined to form the meta information of the marked points.

[0088] Fifthly, the meta information of all the marked points is output in a json file.

[0089] Sixthly, all the mapping relationships are collected as a mapping dictionary and output in a json file.

[0090] Seventhly, a two-dimensional array with the same size as the super pixel macro block mask is constructed as an initial mask (i.e. the initialized target mask), and the initial value is all 0 representing the background id.

[0091] The eighth step is to traverse the label at each pixel position in the superpixel macroblock mask. If the label value is not 0 and the label value matches the superpixel macroblock id in the mapping dictionary, the value of the target id under the corresponding mapping relationship is overwritten in the corresponding position of the initial target mask. The ninth step is to obtain the target mask and output it in a json file.

[0092] The front-end tool used in this embodiment can be realized by writing an html webpage script and can be used by opening it in a modern computer web browser. There are two buttons in the front-end interface, which can be clicked by a mouse to respectively load an image and a pre-segmented mask json file from the local, and a window is provided for filling in the target id of this manual annotation task. The loaded image is displayed as a display image in the interface. The cursor moves on the displayed image, and a mouse left button click at any pixel point generates a motion capture event. The front-end tool records the current clicked pixel point coordinates and displays a small dot at the marked position on the display interface, as shown in Figure 6 、 7 The mouse right button click on the small dot on the display interface that is used to mark the annotation point deletes the annotation point. There is also a button in the blank part of the display interface to indicate that the annotation is complete, and a mouse click can output the current collected mapping relationship (in the form of a mapping dictionary json file) and the list of annotation points (in the form of an annotation point metadata json file).

[0093] It should be noted that the front-end tool used in this embodiment can also be used in mobile devices such as mobile phones and tablets, and touch clicks can replace the above-mentioned mouse left button clicks, and touch long presses can replace the above-mentioned mouse right button clicks.

[0094] Figure 7 The image and the output mask visualization image after annotation in embodiment 1 of the present application are divided into two parts, the upper half is the image displayed in the annotation interface, and the lower half is the display of the mask visualization after annotation. The Tomcat in the upper half of the image is the target object. In this image, the target object has an open arm, and if the GrabCut-based annotation method is used, the shape increases the number of foreground objects that will be boxed in when the entire Tomcat image is boxed. However, by using the point annotation of the present application, the place where the point is annotated is displayed as a small dot on the graph. The mask output by embodiment 1 is shown in the lower half of the figure, which is a binary grayscale image. The target object is highlighted and covers most of the target object area.

[0095] Figure 8For the labeled image and the mask visualization image produced by embodiment 2 of the present application, it is divided into two parts, the upper half is the image displayed in the labeling interface, and the lower half is the display of the mask visualization produced after labeling. Artificially mark points on the target object, and the marked points are displayed as small dots on the graph. The mask produced by embodiment 2 is shown in the lower half of the figure, which is a binary grayscale image. The target object is highlighted, covering more target object area, and the coverage range is close to complete. Figure 7

[0096] It should be noted that in the superpixel process disclosed in the art, the determination of similarity depends on the calculation of similarity measure, which can include Euclidean distance, texture difference in addition to color difference, which belongs to the method disclosed in the art. In particular, the color difference discrimination method of embodiments 1 and 2 takes the difference of the hue, contrast and brightness of two pixel colors in the HSV channel as the difference, respectively compared with the set threshold, the similarity is inversely proportional to the product of each difference, and the similarity is lower than the threshold. It is determined that the non-similar pixels cannot be merged, otherwise it is determined that the similar pixels can be merged. The above color resolution threshold can be optimized by debugging.

[0097] It should be noted that in all embodiments of the present application, the number of targets in the image to be processed is not limited, and the target in the image to be processed can be one or more. For example, in the case of a target image with a resolution of 1280*720, there can be multiple target objects in the image to be processed. Obviously, in the case of multiple targets in the image to be processed, the generated mask can more completely represent the position and shape of each target in the image.

[0098] The functional modules in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional module.

[0099] ​The integrated unit, if implemented in the form of a software function module and not sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments can essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in the embodiments. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0100] Specifically, the image auxiliary labeling method, the image pre-segmentation method, the macroblock expansion method, and the semi-automatic target mask generation method in the embodiments can be stored on a storage medium such as a disc, a hard disk, a U disk, etc. When the computer program instructions corresponding to any one of the image auxiliary labeling method, the image pre-segmentation method, the macroblock expansion method, and the semi-automatic target mask generation method are read or executed by an electronic device, the image auxiliary labeling method, the image pre-segmentation method, the macroblock expansion method, and the semi-automatic target mask generation method described in the foregoing embodiments are implemented.

[0101] In some embodiments, the functions of the apparatus provided by the embodiments of the present application or the modules included therein can be used to execute the methods described in the foregoing method embodiment, and the specific implementation can refer to the description of the foregoing method embodiment. For brevity, details are not repeated here.

[0102] The foregoing description of each embodiment tends to emphasize the differences between the embodiments, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0103] The features disclosed in the various method or product embodiments of the present application can be combined arbitrarily without conflict, to obtain new method or product embodiments.

[0104] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part which contributes to the prior art can be embodied in the form of software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a plurality of instructions to make a terminal (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) execute the method described in various embodiments of the present application.

[0105] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A point labeling-based image target object auxiliary labeling method, characterized in that The execution flow includes image pre-segmentation and semi-automatic target mask generation, and the specific steps include: A 2D array with the same size as the original image is established as an empty mask, and the maximum label value is set to 0; All elements of the array are traversed as starting points for each nearest neighbor search, and if the starting point is labeled as the initial value, the current maximum label value is incremented as the new label value of the starting point, and then the connected region search algorithm is used to expand the current connected region in all directions, otherwise the current starting point is skipped; When performing the connected region search algorithm, the similarity between the adjacent next pixel and the current pixel is calculated during the propagation in each direction, and if the similarity reaches the threshold, the propagation in the direction is continued, otherwise the propagation in the direction is stopped and the current pixel point is defined as a boundary point; When all propagation directions of a connected region are stopped, the connected region with the same label is called a superpixel connected region; From the set of all superpixel connected regions, regions with an area greater than a threshold are selected as superpixel macroblocks and their ids are recompiled as macroblock ids, or the superpixel connected regions are directly used as superpixel macroblocks and their ids are used as macroblock ids; The overall of all macroblocks or superpixel macroblocks is a superpixel macroblock set, and the color with the most number of pixels in each superpixel macroblock is taken as the representative color of the entire macroblock; The labels of the superpixel macroblock set are integrated into the mask; The colors of the superpixel macroblock set are integrated into a new image; An empty mask is constructed and filled with macroblock ids to generate a pre-segmentation mask; A user fills in the target id on the front-end interface, loads and displays the image and the pre-segmentation mask from the local; The user marks points on the distinguishable color blocks of the image target; The computer finds the label in the pre-segmentation mask at the same position as the marked point, which is the superpixel macroblock id, and establishes a mapping relationship between the target id and the superpixel macroblock id; The computer collects all the marked points as the marked points, and the positions of the marked points and the target id form the meta information of the marked points; The meta information of all the marked points is collected; All the mapping relationships are collected as a mapping dictionary; A 2D array with the same size as the pre-segmentation mask is constructed as an initial mask, i.e. an initialized target mask, and the initial value is 0 indicating the background id; The labels of each pixel position in the pre-segmentation mask are searched, and if the label is non-zero and matches the superpixel macroblock id in the mapping dictionary, the corresponding position in the initial target mask is overwritten with the value of the target id under the corresponding mapping relationship; The target mask is obtained.

2. A point labeling-based image target object auxiliary labeling method, characterized in that The execution flow of the macroblock expansion method also includes: A 2D array with the same size as the original image is established as an empty mask, and the maximum label value is set to 0; All elements of the array are traversed as starting points for each nearest neighbor search, and if the starting point is labeled as the initial value, the current maximum label value is incremented as the new label value of the starting point, and then the connected region search algorithm is used to expand the current connected region in all directions, otherwise the current starting point is skipped; In the execution of the connected region search algorithm, in the propagation in each direction, the similarity between the next pixel adjacent to the current pixel is calculated, and if the similarity reaches a threshold, the propagation continues, otherwise the propagation in this direction is stopped and the current pixel is defined as a boundary point; When all the propagation directions of a connected region are stopped, the connected region with the same label is called a superpixel connected region; From the set of all superpixel connected regions, the regions with an area greater than a threshold are selected as superpixel macroblocks and the id is recompiled as the macroblock id, or the superpixel connected region is directly taken as the superpixel macroblock and the connected region id is taken as the macroblock id; The whole of all the macroblocks or superpixel macroblocks is the superpixel macroblock set, and the color with the most number of pixels in each superpixel macroblock is taken as the representative color of the whole macroblock; The labels of the superpixel macroblock set are integrated into the mask; The colors of the superpixel macroblock set are integrated into the new image; An empty mask is constructed, and the macroblock id is filled in to generate a pre-segmentation mask; Step S12, all empty region pixel points are traversed as starting points for the execution of the connected region search, and the current empty point is integrated into the surrounding macroblock; Step S13, the pixel in the original image corresponding to the pixel position of the current empty point is taken as the reference pixel, and the pixels in the neighborhood within a certain distance are recorded on the weight of the macroblock id to which they belong, but the points whose similarity between the representative color of the belonging macroblock and the color of the reference pixel is lower than a certain threshold are excluded; Step S14, the weights of the macroblock ids in the neighborhood are sorted, the macroblock id with the highest weight is taken as the label of the current empty point, and the representative color of the macroblock is taken as the new color of the empty point; Step S15, iterate steps S12 to S14 for no less than 1 round; Step S16, the labels of the superpixel macroblock set are integrated into the new mask, and the representative colors of the superpixel macroblock set are integrated into the new image; A person fills in the target object id on the front-end interface, loads and displays the image and the new mask from the local; A person makes point marks on the distinguishable color blocks of the image target object; The computer finds the label in the new mask which is the same as the marked point, i.e. the superpixel macroblock id, and establishes the mapping relationship between the target object id and the superpixel macroblock id; The computer collects all the marked points as the marked points, and the position of the marked point and the target object id constitute the meta information of the marked point; Collect the meta information of all the marked points; Collect all the mapping relationships as the mapping dictionary; An initial mask of the same size as the new mask is constructed, i.e. the initial target object mask, and the initial value is all 0 representing the background id; each pixel position in the new mask is searched, if the label is non-zero and the label matches the superpixel macroblock id in the mapping dictionary, the corresponding position in the initial target object mask is overwritten with the value of the target object id under the corresponding mapping relationship; The target object mask is obtained.

3. An electronic device comprising: A memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that: when the processor executes the program, the point-marking-based image target object auxiliary labeling method of any one of claims 1 or 2 is realized.

4. A computer-readable storage medium storing computer-executable instructions, the computer-executable instructions comprising: The computer executable instructions are used to execute the point label based image target object auxiliary labeling method according to any one of claims 1 or 2. The computer executable instructions are used to execute the point label based image target object auxiliary labeling method according to any one of claims 1 or 2.

Images