Image annotation optimization method and device, electronic equipment and storage medium

Abstract

The invention discloses an image annotation optimization method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an original mask of a to-be-annotated image, and carrying out the edge point sampling of the original mask, and obtaining a sampling coordinate of the to-be-annotated image; performing mask prediction according to the to-be-labeled image and the sampling coordinates to obtain a prediction mask of the to-be-labeled image; and comparing the original mask with the predicted mask, and labeling the to-be-labeled image according to an obtained comparison result. According to the method, more accurate sampling point coordinates are obtained before mask prediction, a more refined mask is obtained, then the refined mask and an original mask are compared and labeled, a more accurate labeling result is obtained, and the labeling accuracy and quality are improved.

Description

Technical Field

[0001] The present application relates to the technical field of data annotation processing, and in particular to an image annotation optimization method, device, electronic device and storage medium. Background Art

[0002] Labeling data sets is an important step for deep learning, but labeling data is an extremely tedious task. To some extent, it can reduce the workload by first performing some rough labeling and then optimizing the labels. This optimization method can be completed with the help of semi-automatic labeling tools, that is, using the trained network model for labeling. However, the existing open source semi-automatic labeling tools are used on the premise that the model accuracy is as high as possible. If the detection results are inaccurate, it will increase the workload and make them unusable.

[0003] Based on the above problems, if you want to use the model for label optimization, then there are higher requirements for the accuracy of the model. A large-scale, accurately labeled data set used to train and ensure model performance is also required, but such models and data sets are lacking.

[0004] Therefore, how to improve the accuracy of data annotation optimization is extremely important. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image annotation optimization method, device, electronic device and storage medium to solve the technical problem of low quality of data annotation optimization in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a method for optimizing image annotation, including:

[0007] Acquire the original mask of the image to be annotated, and perform edge point sampling on the original mask to obtain the sampling coordinates of the image to be annotated;

[0008] Performing mask prediction according to the image to be labeled and the sampling coordinates to obtain a predicted mask of the image to be labeled;

[0009] The original mask is compared with the predicted mask, and the image to be labeled is labeled according to the comparison result obtained.

[0010] In a second aspect, an embodiment of the present application provides an image annotation optimization device, comprising:

[0011] A coordinate sampling module is used to obtain the original mask of the image to be labeled, and perform edge point sampling on the original mask to obtain the sampling coordinates of the image to be labeled;

[0012] A mask prediction module, used for performing mask prediction according to the image to be labeled and the sampling coordinates to obtain a predicted mask of the image to be labeled;

[0013] The image annotation module is used to compare the original mask with the predicted mask, and annotate the image to be annotated according to the comparison result.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in any one of the above-described methods for optimizing image annotation when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned methods for optimizing image annotation are implemented.

[0016] The embodiment of the present application provides an image annotation optimization method, device, electronic device and storage medium. When optimizing the annotation of image data, the original mask of the image to be annotated that is currently being optimized is determined, and edge point sampling processing is performed according to the original mask to obtain the sampling coordinates of the image to be annotated, and then mask prediction is performed according to the image to be annotated and the sampling coordinates to obtain the predicted mask of the image to be annotated. Finally, post-processing is performed to complete the annotation optimization of the image to be annotated, and the predicted mask and the original mask are compared, and then the image to be annotated is annotated according to the comparison result. It is achieved that when optimizing the annotation of the image, more accurate sampling point coordinates are obtained before mask prediction, and a more refined mask is obtained. Then, by comparing and annotating the refined mask with the original mask, a more accurate annotation result is obtained, thereby improving the quality and accuracy of the annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of an image annotation optimization method provided in an embodiment of the present application;

[0018] Figure 2 is a flowchart of the steps of obtaining sampling coordinates provided in an embodiment of the present application;

[0019] Figure 3 is a flowchart of steps for annotating an image to be annotated provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of an original mask after covering provided by an embodiment of the present application;

[0021] Figure 5is another flowchart of the steps of obtaining the labeled image to be labeled provided in an embodiment of the present application;

[0022] Figure 6 It is a schematic block diagram of annotating an image to be annotated provided in an embodiment of the present application;

[0023] Figure 7 It is a structural schematic diagram of an image annotation optimization device provided in an embodiment of the present application;

[0024] Figure 8 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0025] Fig. 9 This is another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] In related technologies, there are two methods used to optimize the annotation and improve the quality of existing annotated data. The first is human interaction, which can segment objects of any category, but requires manual correction and gradually optimizes the mask through iteration; the second is model automatic segmentation, which allows segmentation of specific object categories, but requires a large number of manually annotated labels for training.

[0030] Take the COCO dataset as an example. Since the COCO dataset was annotated a long time ago and it is a non-profit annotation, it is not annotated pixel by pixel, and it uses polygonal annotation. This makes the data annotation very rough. Even with the use of multiple data enhancement techniques, it is impossible to fundamentally improve its rough annotation. As a result, the model trained using the COCO dataset often finds it difficult to generate high-quality target segmentation results, further affecting the results of subsequent processing tasks. Therefore, how to effectively optimize the annotation and obtain more accurate and high-quality annotated data is very necessary for subsequent target segmentation processing.

[0031] In order to solve the technical problems existing in the related art, the present application embodiment provides an image annotation optimization method, see Figure 1 , Figure 1 It is a flow chart of an image annotation optimization method provided in an embodiment of the present application, and the method includes steps 101 to 104.

[0032] Step 101: obtain the original mask of the image to be labeled, and perform edge point sampling on the original mask to obtain the sampling coordinates of the image to be labeled.

[0033] In one embodiment, when optimizing the annotation of the roughly annotated data, an image to be annotated that needs to be optimized and processed is obtained, and the original mask of the image to be annotated is obtained at the same time, and then the sampling coordinates of the image to be annotated are obtained by sampling the edge points of the original mask.

[0034] It should be noted that in various embodiments of the present application, annotation optimization is performed on the data after coarse annotation. Generally speaking, the data after coarse annotation has completed the division of data categories. Taking image data as an example, the image after coarse annotation has completed the classification of the image. At the same time, the original mask can also be obtained when the coarse annotation of the image is completed.

[0035] Exemplarily, when optimizing the annotation of an image, an original mask of the image to be annotated is obtained, and then the original mask is processed in a certain manner, such as shrinking, and then sampling points are acquired after the processing is completed to obtain the coordinates of each sampling point.

[0036] Specifically, when processing to obtain the sampling coordinates of the image to be labeled, you can refer to Figure 2 , Figure 2 It is a flowchart of the steps of obtaining sampling coordinates provided in an embodiment of the present application, wherein the steps include steps 201 to 202.

[0037] Step 201, obtaining an original mask of the image to be annotated, and calculating the image gradient of the image to be annotated based on the original mask;

[0038] Step 202 , shrinking the original mask according to the image gradient, and sampling edge points of the shrunken original mask to obtain sampling coordinates of the image to be labeled.

[0039] When performing sampling processing, calculations are performed based on the original mask of the image to be labeled to obtain the image gradient of the image to be labeled, and then the original mask is shrunk using the image gradient. Finally, edge point sampling is performed on the original mask after the shrinkage processing to obtain the sampling coordinates of the image to be labeled.

[0040] Taking the COCO dataset as an example, due to its annotation characteristics, the original annotation of the data will have two problems: one is that the annotation boundary is fuzzy, and the other is that the annotation boundary will detect the actual edge of the object. Therefore, when determining the sampling point, it is necessary to process the original mask accordingly to make the obtained sampling point more accurate.

[0041] For example, during processing, the numpy library can be used to calculate the gradient of the original annotated image, and then the expansion algorithm can be applied to the obtained image map to achieve the effect of mask shrinkage. Then, on the shrunk original mask, the four edge points of the top, bottom, left and right are sampled to obtain the sampling point coordinates of the image to be annotated.

[0042] In practical applications, by shrinking the original mask, it is possible to effectively avoid sampling outside the boundary of the object or at blurred points on the edge. In addition, since the sampling points can surround the object, the object to be segmented can be more accurately located, and the boundary of the object can be segmented more accurately.

[0043] Step 102 : performing mask prediction according to the image to be labeled and the sampling coordinates to obtain a predicted mask of the image to be labeled.

[0044] In one embodiment, after sampling of the sampling point coordinates of the image to be annotated is completed, a mask prediction is performed according to the annotated image and the sampling coordinates to obtain a predicted mask corresponding to the image to be annotated.

[0045] It should be noted that, when performing mask prediction, the constructed segmentation model is used for mask prediction, wherein the constructed segmentation model may be a SAM model (Segment Anything Model), which is a segmentation model based on deep learning.

[0046] Exemplarily, during processing, by inputting the image to be labeled and the coordinates of the sampling points into the segmentation model used, a prediction mask of the image to be labeled can be obtained through the segmentation processing of the model, and the prediction mask can be a prediction mask of three levels: whole-part-sub-part.

[0047] Taking a vehicle image as an example, if the sampling point is selected on a car window, the overall level prediction mask corresponds to the entire vehicle, the partial level prediction mask corresponds to the door where the car window is located, and the sub-part level prediction mask corresponds to the car window itself. In other words, the prediction masks at different levels are mask predictions at different levels for objects in the image, and the prediction masks at different levels may all be used as the annotation results of the image to be annotated.

[0048] That is, when the predicted mask is obtained by processing the image to be labeled and the sampling coordinates, it includes: obtaining the classification label of the image to be labeled; performing mask prediction of different levels on the image to be labeled according to the classification label and the sampling coordinates to obtain the predicted mask of the image to be labeled, wherein the number of predicted masks is the same as the number of levels.

[0049] Specifically, when using the constructed model to predict the mask, the classification label of the image to be annotated is determined, and then the classification label and the sampling coordinates are input into the model for prediction processing to obtain predicted masks at different levels. There are corresponding associations between the predicted masks at different levels, such as the associations between the overall mask, partial mask and sub-partial mask described above.

[0050] Step 103 : Compare the original mask with the predicted mask, and annotate the image to be annotated according to the comparison result.

[0051] In one embodiment, after using the constructed related model to perform mask prediction and obtain predicted masks at different levels, the original mask is compared and analyzed with the predicted mask, and then the image to be labeled and optimized is completed according to the actual comparison and analysis results. In fact, the original mask is a rough annotation result of the image to be labeled. Since there may be inaccurate annotations, such as inaccurate boundaries, further analysis and comparison are required to determine how to label and optimize the image to be labeled.

[0052] Exemplarily, due to the defects of the original mask, the original mask is compared with the predicted mask for analysis, specifically including: calculating the intersection-and-union ratio of the original mask and the predicted mask, and determining whether the sampling coordinates are correct based on the intersection-and-union ratio; when it is determined that the sampling coordinates are correct, annotating the image to be annotated according to the predicted mask; when it is determined that the sampling coordinates are wrong, executing the steps: obtaining the original mask of the image to be annotated, and sampling the edge points of the original mask to obtain the sampling coordinates of the image to be annotated.

[0053] That is, when performing comparative analysis, the intersection-and-union ratio between the original mask and the predicted mask is calculated, and then the annotation and optimization of the image to be annotated is completed according to the intersection-and-union ratio. When calculating the intersection-and-union ratio, the intersection-and-union ratio of each predicted mask to the original mask is calculated, and then the sampling coordinates obtained in step S101 are determined to be accurate according to the intersection-and-union ratio. If it is determined to be accurate, the original mask is processed using the predicted mask to complete the annotation processing of the image to be annotated. If it is determined to be inaccurate, it is necessary to re-acquire the sampling points and predict the mask, that is, re-execute step 101.

[0054] Furthermore, when determining whether the sampling coordinates are accurate based on the calculated intersection-and-union ratio, this is achieved through size comparison, specifically including: obtaining the maximum intersection-and-union ratio in the intersection-and-union ratio, and comparing the maximum intersection-and-union ratio with a preset threshold; if the maximum intersection-and-union ratio is less than the preset threshold, determining that the sampling coordinates are wrong; if the maximum intersection-and-union ratio is greater than or equal to the preset threshold, determining that the sampling coordinates are correct.

[0055] In practical applications, for the predicted mask, as long as one of the conditions is met, the annotation processing can be performed. Therefore, when making comparisons, the maximum intersection-and-union ratio is determined in the calculated intersection-and-union ratios. When the maximum intersection-and-union ratio is greater than or equal to the preset threshold, it means that the sampling coordinates are accurate. At this time, the image to be annotated can be annotated based on the obtained predicted mask. When the maximum intersection-and-union ratio is less than the preset threshold, it means that the difference between the predicted mask and the original mask is too large. If annotation and optimization processing are still required at this time, the sampling coordinates will be re-determined and subsequent processing will be performed.

[0056] Further, when it is determined that the maximum intersection-over-union ratio in the prediction mask is greater than or equal to the preset threshold, the image to be labeled is labeled according to the prediction mask. Figure 3 , Figure 3 It is a flowchart of steps for annotating an image to be annotated provided in an embodiment of the present application, wherein the steps include step 3013 referring to step 302 .

[0057] Step 301, when it is determined that the sampling coordinates are correct, a prediction mask corresponding to the maximum intersection-and-union ratio is selected from the intersection-and-union ratios as a refinement mask;

[0058] Step 302: Cover the original mask according to the refined mask to obtain a labeled image to be labeled.

[0059] Specifically, when it is determined that the sampling coordinates are correct, the prediction mask corresponding to the maximum intersection-and-union ratio is selected as the refinement mask in the intersection-and-union ratios, and then the original mask is covered with the determined prediction mask to obtain the labeled image to be labeled.

[0060] That is, when performing the labeling process, the predicted mask corresponding to the maximum intersection-over-union ratio is selected as the refined mask for labeling the image to be labeled, and when performing the labeling process, the original mask is processed and the refined mask is covered on the original mask to complete the labeling optimization process.

[0061] However, in the actual covering process, the boundary of the object in the refined mask is usually inside the boundary of the object in the original mask, such as Figure 4 , in which case, after coverage is completed, the area between the two boundaries is an unlabeled area and is represented by Figure 4 It can be seen that the unlabeled area should not belong to the object itself. Therefore, after the coverage is completed, the original mask covered with the refined mask needs to be processed accordingly. For details, refer to Figure 5 , Figure 5 This is another flowchart of the step of obtaining the labeled image to be labeled provided in an embodiment of the present application, wherein the step includes step 501 to step 502.

[0062] Step 502, performing overlay processing on the original mask according to the refined mask to obtain the overlaid original mask;

[0063] Step 502, determining the missing area in the original mask after covering, and annotating each pixel in the missing area based on the proximity matching method to obtain an annotated image, wherein the missing area is composed of the boundary of the original mask and the boundary of the refined mask.

[0064] Specifically, after the original mask is covered with the refined mask, the lost area is determined in the covered original mask, wherein the lost area is an unlabeled area, and then the lost area is labeled according to the original mask by using a proximity matching method.

[0065] by Figure 4 Taking the scene shown in the figure as an example, when marking the lost area, the pixel points close to the lost area in the X area are selected for marking by using the proximity matching method, such as Figure 4 As shown in , pixel point a is marked using pixel point A, and pixel point b is marked using pixel point B.

[0066] After the unlabeled lost areas are labeled, the labeling and optimization processing of the image to be labeled is completed.

[0067] In one embodiment, referring to Figure 6 , Figure 6 : is a schematic diagram of a block diagram of annotating an image provided by an embodiment of the present application. The specific processing steps and processes are as follows:

[0068] First, calculate the image gradient of the original mask, use the image gradient to shrink the original mask, and sample the edge points on the shrunk original mask to obtain the sampling coordinates of the edge points. When sampling the edge points, the numpy library can be used to calculate the gradient of the original annotated image, and then the expansion algorithm is applied to the image gradient to achieve the effect of mask shrinkage, and then the four edge points of the upper, lower, left and right are sampled on the shrunk mask.

[0069] Then, the sampled edge point coordinates are sent to the SAM segmentation model to obtain three levels of prediction masks. Specifically, the constructed SAM model takes the original image (image to be annotated) and the sampling coordinates of the sampling points as input, and outputs three levels of prediction masks: whole-part-sub-part. In other words, three associated prediction masks will be obtained. Taking the vehicle image as an example, if the sampling point is selected on a car window, the mask at the whole level can correspond to the whole vehicle, the mask at the part level can correspond to the door where the window is located, and the mask at the sub-part level can correspond to the window itself.

[0070] Finally, after obtaining the three levels of prediction masks, the post-processing module decides whether to use the refined mask and which level of refined mask to use for subsequent processing. In fact, when the three levels of prediction masks are obtained based on the SAM model, they correspond to the three levels of target objects that the sampling points may represent. In order to obtain a prediction mask that matches the original mask, the three levels of prediction masks are calculated by IoU with the original mask respectively. If the maximum IoU value is less than the previously set threshold, it proves that the sampling point is selected incorrectly or the SAM model prediction is wrong; conversely, if the maximum IoU value is greater than the threshold, the prediction mask corresponding to the maximum IoU value is used as the final refinement result.

[0071] In addition, when the determined refined mask is overlaid on the original mask, there will be a problem of missing labels for pixels between the boundaries of the refined annotations and the boundaries of the original annotations. At this time, a neighboring pixel annotation matching operation is added in post-processing to assign the category labels of the outer neighboring pixels to the unlabeled pixels to solve this problem of missing labels.

[0072] In summary, the present application discloses an image annotation optimization method. When optimizing the annotation of image data, the original mask of the image to be annotated that is currently being optimized is determined, and edge point sampling is performed according to the original mask to obtain the sampling coordinates of the image to be annotated, and then mask prediction is performed according to the image to be annotated and the sampling coordinates to obtain the predicted mask of the image to be annotated. Finally, post-processing is performed to complete the annotation optimization of the image to be annotated, and the predicted mask and the original mask are compared, and then the image to be annotated is annotated according to the comparison result. It is achieved that when optimizing the annotation of the image, more accurate sampling point coordinates are obtained before mask prediction, and a more refined mask is obtained. Then, by comparing and annotating the refined mask with the original mask, a more accurate annotation result is obtained, thereby improving the quality and accuracy of the annotation.

[0073] According to the method described in the above embodiment, this embodiment will be further described from the perspective of an image annotation optimization device. The image annotation optimization device can be implemented as an independent entity or integrated into an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0074] See also Figure 7 , Figure 7 is a structural schematic diagram of an image annotation optimization device provided in an embodiment of the present application, such as Figure 7 As shown, the image annotation optimization device 700 provided in the embodiment of the present application includes:

[0075] The coordinate sampling module 701 is used to obtain the original mask of the image to be labeled, and perform edge point sampling on the original mask to obtain the sampling coordinates of the image to be labeled;

[0076] The mask prediction module 702 is used to perform mask prediction based on the image to be labeled and the sampling coordinates to obtain a predicted mask of the image to be labeled;

[0077] The image annotation module 703 is used to compare the original mask with the predicted mask, and annotate the image to be annotated according to the comparison result.

[0078] In one embodiment, the coordinate sampling module 701 is further used for:

[0079] Obtaining the original mask of the image to be annotated, and calculating the image gradient of the image to be annotated based on the original mask;

[0080] The original mask is shrunk according to the image gradient, and the edge points of the shrunk original mask are sampled to obtain the sampling coordinates of the image to be labeled.

[0081] In one embodiment, the mask prediction module 702 is further configured to:

[0082] Get the classification label of the image to be annotated;

[0083] Masks of different levels are predicted for the image to be labeled according to the classification labels and the sampling coordinates to obtain predicted masks of the image to be labeled, wherein the number of predicted masks is the same as the number of levels.

[0084] In one embodiment, the image annotation module 703 is further used to:

[0085] Calculate the intersection-and-union ratio of the original mask and the predicted mask, and determine whether the sampling coordinates are correct based on the intersection-and-union ratio;

[0086] When the sampling coordinates are determined to be correct, the image to be labeled is labeled according to the predicted mask;

[0087] When it is determined that the sampling coordinates are wrong, the steps are executed: obtaining the original mask of the image to be labeled, and performing edge point sampling on the original mask to obtain the sampling coordinates of the image to be labeled.

[0088] In one embodiment, the image annotation module 703 is further used to:

[0089] Obtaining a maximum intersection-and-union ratio among the intersection-and-union ratios, and comparing the maximum intersection-and-union ratio with a preset threshold;

[0090] If the maximum intersection-union ratio is less than a preset threshold, it is determined that the sampling coordinates are wrong;

[0091] If the maximum intersection-over-union ratio is greater than or equal to the preset threshold, it is determined that the sampling coordinates are correct.

[0092] In one embodiment, the image annotation module 703 is further used to:

[0093] When it is determined that the sampling coordinates are correct, the prediction mask corresponding to the maximum intersection-over-union ratio is selected as the refinement mask among the intersection-over-union ratios;

[0094] The original mask is covered according to the refined mask to obtain the labeled image to be labeled.

[0095] In one embodiment, the image annotation module 703 is further used to:

[0096] Covering the original mask according to the refined mask to obtain the covered original mask;

[0097] The lost area is determined in the covered original mask, and each pixel in the lost area is labeled based on the proximity matching method to obtain the labeled image to be labeled, wherein the lost area is composed of the boundary of the original mask and the boundary of the refined mask.

[0098] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0099] Also, see Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802. The processor 801 is electrically connected to the memory 802.

[0100] The processor 801 is the control center of the electronic device 800. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 800 and processes data by running or loading applications stored in the memory 802 and calling data stored in the memory 802, thereby monitoring the electronic device 800 as a whole.

[0101] In this embodiment, the processor 801 in the electronic device 800 will load the instructions corresponding to the processes of one or more applications into the memory 802 according to the following steps, and the processor 801 will run the application stored in the memory 802, thereby implementing any step in the image annotation optimization method provided in the above embodiment.

[0102] The electronic device 800 can implement the steps in any embodiment of the image annotation optimization method provided in the embodiments of the present application, and therefore can achieve the beneficial effects that can be achieved by any image annotation optimization method provided in the embodiments of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0103] See also Fig. 9 , Fig. 9 is another structural schematic diagram of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, Fig. 9 The specific structural block diagram of the electronic device provided in the embodiment of the present application is shown, and the electronic device can be used to implement the image annotation optimization method provided in the above embodiment. The electronic device 900 can be a mobile terminal such as a smart phone or a laptop computer.

[0104] The RF circuit 910 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 910 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 910 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0105] The memory 920 can be used to store software programs and modules, such as the program instructions / modules corresponding to the image annotation optimization method in the above-mentioned embodiment. The processor 980 executes various functional applications and the image annotation optimization method by running the software programs and modules stored in the memory 920.

[0106] The memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 920 may further include a memory remotely arranged relative to the processor 980, and these remote memories may be connected to the electronic device 900 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The input unit 930 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 930 may include a touch-sensitive surface 931 and other input devices 932. The touch-sensitive surface 931, also known as a touch display screen or a touchpad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 931), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 931 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 980, and can receive and execute commands sent by the processor 980. In addition, the touch-sensitive surface 931 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 931, the input unit 930 may further include other input devices 932. Specifically, the other input devices 932 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0108] The display unit 940 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 900, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 940 may include a display panel 941, and optionally, the display panel 941 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 931 may cover the display panel 941, and when the touch-sensitive surface 931 detects a touch operation on or near it, it is transmitted to the processor 980 to determine the type of the touch event, and then the processor 980 provides corresponding visual output on the display panel 941 according to the type of the touch event. Although in the figure, the touch-sensitive surface 931 and the display panel 941 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 931 and the display panel 941 can be integrated to implement input and output functions.

[0109] The electronic device 900 may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 941 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 900, they will not be repeated here.

[0110] The audio circuit 960, the speaker 961, and the microphone 962 can provide an audio interface between the user and the electronic device 900. The audio circuit 960 can transmit the electrical signal converted from the received audio data to the speaker 961, which is converted into a sound signal for output; on the other hand, the microphone 962 converts the collected sound signal into an electrical signal, which is received by the audio circuit 960 and converted into audio data, and then the audio data is output to the processor 980 for processing, and then sent to another terminal through the RF circuit 910, or the audio data is output to the memory 920 for further processing. The audio circuit 960 may also include an earplug jack to provide communication between an external headset and the electronic device 900.

[0111] The electronic device 900 can help users receive requests, send information, etc. through the transmission module 970 (such as a Wi-Fi module), which provides users with wireless broadband Internet access. Although the transmission module 970 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 900 and can be omitted as needed without changing the essence of the invention.

[0112] The processor 980 is the control center of the electronic device 900. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 920, and calling data stored in the memory 920, it executes various functions of the electronic device 900 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 980 may include one or more processing cores; in some embodiments, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 980.

[0113] The electronic device 900 also includes a power supply 990 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 980 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 990 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0114] Although not shown, the electronic device 900 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step of the image annotation optimization method provided in the above embodiment.

[0115] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0116] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium, which stores multiple instructions, and when the instructions are executed by the processor, any step in the image annotation optimization method provided in the above embodiments can be implemented.

[0117] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0118] Since the instructions stored in the storage medium can execute the steps in any embodiment of the image annotation optimization method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image annotation optimization method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0119] The above is a detailed introduction to an image annotation optimization method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can be made, and these improvements and modifications are also considered to be within the scope of protection of the present application.

Claims

1. A method for optimizing image annotation, characterized in that: include: Acquire the original mask of the image to be annotated, and perform edge point sampling on the original mask to obtain the sampling coordinates of the image to be annotated; Performing mask prediction according to the image to be labeled and the sampling coordinates to obtain a predicted mask of the image to be labeled; The original mask is compared with the predicted mask, and the image to be labeled is labeled according to the comparison result obtained.

2. The method according to claim 1, characterized in that The obtaining of the original mask of the image to be annotated and performing edge point sampling on the original mask to obtain the sampling coordinates of the image to be annotated includes: Acquire an original mask of the image to be annotated, and calculate the image gradient of the image to be annotated based on the original mask; The original mask is shrunk according to the image gradient, and edge points are sampled on the shrunk original mask to obtain sampling coordinates of the image to be labeled.

3. The method according to claim 1, characterized in that The step of performing mask prediction according to the image to be labeled and the sampling coordinates to obtain a predicted mask of the image to be labeled includes: Obtaining a classification label for the image to be labeled; Perform mask prediction of different levels on the image to be labeled according to the classification label and the sampling coordinates to obtain predicted masks of the image to be labeled, wherein the number of the predicted masks is the same as the number of the levels.

4. The method according to claim 1, characterized in that The comparing the original mask with the predicted mask and labeling the image to be labeled according to the comparison result includes: Calculating an intersection-and-union ratio between the original mask and the predicted mask, and determining whether the sampling coordinates are correct according to the intersection-and-union ratio; When it is determined that the sampling coordinates are correct, labeling the image to be labeled according to the prediction mask; When it is determined that the sampling coordinates are wrong, the steps are performed: obtaining an original mask of the image to be annotated, and performing edge point sampling on the original mask to obtain the sampling coordinates of the image to be annotated.

5. The method according to claim 4, characterized in that The determining whether the sampling coordinates are correct according to the intersection-and-union ratio includes: Obtaining a maximum intersection-and-union ratio among the intersection-and-union ratios, and comparing the maximum intersection-and-union ratio with a preset threshold; If the maximum intersection-over-union ratio is less than the preset threshold, it is determined that the sampling coordinates are wrong; If the maximum intersection-over-union ratio is greater than or equal to the preset threshold, it is determined that the sampling coordinates are correct.

6. The method according to claim 4, characterized in that When it is determined that the sampling coordinates are correct, labeling the image to be labeled according to the prediction mask includes: When it is determined that the sampling coordinates are correct, a prediction mask corresponding to a maximum intersection-over-union ratio is selected among the intersection-over-union ratios as a refinement mask; The original mask is covered according to the refined mask to obtain a labeled image to be labeled.

7. The method according to claim 6, characterized in that The step of covering the original mask according to the refined mask to obtain the labeled image to be labeled includes: Performing a covering process on the original mask according to the refined mask to obtain a covered original mask; A lost area is determined in the covered original mask, and each pixel in the lost area is labeled based on a proximity matching method to obtain a labeled image to be labeled, wherein the lost area is composed of a boundary of the original mask and a boundary of the refined mask.

8. An image annotation optimization device, characterized in that: include: A coordinate sampling module is used to obtain the original mask of the image to be labeled, and perform edge point sampling on the original mask to obtain the sampling coordinates of the image to be labeled; A mask prediction module, used for performing mask prediction according to the image to be labeled and the sampling coordinates to obtain a predicted mask of the image to be labeled; The image annotation module is used to compare the original mask with the predicted mask, and annotate the image to be annotated according to the comparison result.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images