Data annotation method and device, equipment and storage medium
By using detection and segmentation of target objects in the image, and comparing the output results to determine the target coordinate box, the problem of difficult to take into account both the labeling efficiency and accuracy in the prior art is solved, and efficient and accurate image data annotation is achieved.
Patent Information
- Application Number
- CN202311344865.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-17
- Publication Date
- 2025-04-18
AI Technical Summary
The prior art is difficult to take into account both the labeling efficiency and the accuracy of the labeling results in image data labeling. It usually requires a large amount of manual labeling to ensure accuracy, but this leads to low labeling efficiency.
The first detection model and the first segmentation model are used to detect and segment the target objects in the image. By comparing the output results of the two models, the target coordinate box is determined when the overlap range is greater than the preset range, ensuring the accuracy of the labeling results.
There is no need to perform additional training on the large model, saving model training time and improving labeling efficiency. At the same time, ensuring the accuracy of the labeling results by comparing the model output results.
Smart Images
Figure CN119851274A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data annotation technology, and in particular, relates to a data annotation method, device, equipment and storage medium. Background Art
[0002] Image data annotation is the process of labeling images in a dataset to train a machine learning model. Image data annotation can include annotation types such as image classification, object detection, object recognition, image segmentation, etc. For each annotation type, labeling different annotation objects can belong to different annotation tasks.
[0003] At present, image data is usually annotated through a specific task model. Specifically, for different annotation tasks, it is necessary to first obtain a large amount of manually annotated image data to train the initial model and obtain a specific task model. Then the specific task model is used for annotation.
[0004] In order to ensure the accuracy of the labeling results, a large amount of manually labeled data is usually required for training. Due to the low efficiency of manual labeling, the overall labeling efficiency of the specific task model is low. If less labeled data is used for training in order to improve the overall labeling efficiency of the specific task model, the accuracy of the labeling results will be reduced. Therefore, the labeling method using a specific task model cannot take into account both the labeling efficiency and the accuracy of the labeling results. Summary of the invention
[0005] The embodiments of the present application provide a data labeling method, apparatus, device and storage medium, which can both ensure the accuracy of the labeling results and improve the labeling efficiency.
[0006] In a first aspect, an embodiment of the present application provides a data labeling method, the method comprising:
[0007] Acquire a first image, wherein the first image includes a target object to be labeled;
[0008] Detecting the target object in the first image using a first detection large model to obtain a first coordinate frame including the target object, and segmenting the target object in the first image using a first segmentation large model to obtain a plurality of pixel points of the target object;
[0009] Determine a minimum circumscribed polygon of the plurality of pixels as a second coordinate frame including the target object;
[0010] When the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range, determining the first coordinate frame or the second coordinate frame as a target coordinate frame;
[0011] The target coordinate frame is determined as a labeling result corresponding to the target object.
[0012] In a possible implementation, when the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range, determining the first coordinate frame or the second coordinate frame as a target coordinate frame includes:
[0013] Using the Hungarian matching algorithm to match the first coordinate frame and the second coordinate frame to obtain a matching degree;
[0014] When the matching degree is greater than a preset matching degree, the first coordinate frame or the second coordinate frame is determined as a target coordinate frame.
[0015] In a possible implementation manner, after obtaining the second coordinate frame including the target object, the method further includes:
[0016] When the overlap range between the first coordinate frame and the second coordinate frame is not greater than the preset overlap range, obtaining a first probability corresponding to the first coordinate frame output by the first detection large model and a second probability corresponding to the second coordinate frame output by the first segmentation large model;
[0017] Determine a larger probability between the first probability and the second probability as a first target probability;
[0018] A coordinate frame corresponding to the first target probability is determined as a target coordinate frame.
[0019] In a possible implementation manner, after obtaining the second coordinate frame including the target object, the method further includes:
[0020] When the overlap range between the first coordinate frame and the second coordinate frame is not greater than the preset overlap range, receiving a target input for selecting the first coordinate frame or the second coordinate frame from a user;
[0021] In response to the target input, the coordinate box selected by the user is determined as a target coordinate box.
[0022] In a possible implementation manner, after determining the target coordinate frame, the method further includes:
[0023] Acquire a first data set, where the first data set is a training data set corresponding to a target large model, where the target large model is a large model corresponding to a second target probability or a large model corresponding to a coordinate box not selected by a user, and the second target probability is a smaller probability between a first probability corresponding to the first coordinate box and a second probability corresponding to the second coordinate box;
[0024] The target large model is iteratively optimized using the first data set.
[0025] In a possible implementation manner, before detecting the target object in the first image using the first detection large model, the method further includes:
[0026] Acquire a second data set, wherein the second data set includes a plurality of second images, and each of the second images includes the target object and its corresponding coordinate frame;
[0027] Using the second data set to train the first detection large model to obtain a second detection large model;
[0028] The detecting the target object in the first image by using the first detection model includes:
[0029] The target object in the first image is detected using the second detection model.
[0030] In a possible implementation manner, before segmenting the target object in the first image using the first large segmentation model, the method further includes:
[0031] Acquire a third data set, wherein the third data set includes a plurality of third images, and each of the third images includes the target object and a plurality of pixel points corresponding thereto;
[0032] Using the third data set to train the first segmentation large model to obtain a second segmentation large model;
[0033] The step of segmenting the target object in the first image by using the first segmentation model includes:
[0034] The target object in the first image is segmented using the second segmentation model.
[0035] In a second aspect, an embodiment of the present application provides a data labeling device, the device comprising:
[0036] A first acquisition module, used to acquire a first image, wherein the first image includes a target object to be labeled;
[0037] a model processing module, configured to detect the target object in the first image using the first detection large model to obtain a first coordinate frame including the target object, and to segment the target object in the first image using the first segmentation large model to obtain a plurality of pixel points of the target object;
[0038] A first determining module, configured to determine the minimum circumscribed polygon of the plurality of pixels as a second coordinate frame including the target object;
[0039] A second determination module, configured to determine the first coordinate frame or the second coordinate frame as a target coordinate frame when the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range;
[0040] The third determination module is used to determine the target coordinate frame as a labeling result corresponding to the target object.
[0041] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;
[0042] When the processor executes the computer program instructions, the processor implements any possible implementation method of the first aspect described above.
[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements a method in any possible implementation method of the first aspect described above.
[0044] In the data annotation method, device, equipment and storage medium of the embodiment of the present application, since the first detection large model and the first segmentation large model are both large models with strong generalization ability, for the annotation task corresponding to the target object, there is no need to perform additional training on the first detection large model and the first segmentation large model, and the first detection large model can be used to detect the target object in the first image to obtain a first coordinate frame including the target object, and the first segmentation large model can be used to segment the target object in the first image to obtain multiple pixel points of the target object. Since there is no need to perform additional training on the large model, the time of model training can be saved and the annotation efficiency can be improved. In addition, when the overlap range between the first coordinate frame and the second coordinate frame is greater than the preset overlap range, the first coordinate frame or the second coordinate frame is determined as the target coordinate frame, that is, by comparing the output results of the above two large models, any one of the output results is determined as the target coordinate frame when the two output results are close, so that the accuracy of the target coordinate frame can be guaranteed. In this way, by determining the target coordinate frame as the annotation result corresponding to the target object, the accuracy of the annotation result can be guaranteed. That is, through the embodiment of the present application, the accuracy of the annotation result can be guaranteed and the annotation efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0046] Figure 1 It is a flowchart of a data labeling method provided in an embodiment of the present application;
[0047] Figure 2 It is a flowchart of another data labeling method provided in an embodiment of the present application;
[0048] Figure 3 It is a structural schematic diagram of a data labeling device provided in an embodiment of the present application;
[0049] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0052] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0053] As described in the background technology section, in order to solve the problems of the prior art, the embodiments of the present application provide a data labeling method, apparatus, device and storage medium.
[0054] The following first introduces the data labeling method provided in the embodiment of the present application.
[0055] Figure 1 FIG. 1 is a flow chart of a data labeling method provided in an embodiment of the present application. Figure 1As shown, the data labeling method provided in the embodiment of the present application includes the following steps:
[0056] S110, acquiring a first image, where the first image includes a target object to be labeled;
[0057] S120, detecting the target object in the first image using the first detection large model to obtain a first coordinate frame including the target object, and segmenting the target object in the first image using the first segmentation large model to obtain a plurality of pixel points of the target object;
[0058] S130, determining a minimum circumscribed polygon of the plurality of pixels as a second coordinate frame including the target object;
[0059] S140, when the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range, determining the first coordinate frame or the second coordinate frame as a target coordinate frame;
[0060] S150: Determine the target coordinate frame as a labeling result corresponding to the target object.
[0061] In the data annotation method of the embodiment of the present application, since the first detection large model and the first segmentation large model are both large models with strong generalization ability, for the annotation task corresponding to the target object, there is no need to perform additional training on the first detection large model and the first segmentation large model, and the first detection large model can be used to detect the target object in the first image to obtain a first coordinate frame including the target object, and the first segmentation large model can be used to segment the target object in the first image to obtain multiple pixel points of the target object. Since there is no need to perform additional training on the large model, the time of model training can be saved and the annotation efficiency can be improved. In addition, when the overlap range between the first coordinate frame and the second coordinate frame is greater than the preset overlap range, the first coordinate frame or the second coordinate frame is determined as the target coordinate frame, that is, by comparing the output results of the above two large models, any one of the output results is determined as the target coordinate frame when the two output results are close, and the accuracy of the target coordinate frame can be guaranteed. In this way, by determining the target coordinate frame as the annotation result corresponding to the target object, the accuracy of the annotation result can be guaranteed. That is, through the embodiment of the present application, the accuracy of the annotation result can be guaranteed and the annotation efficiency can be improved.
[0062] The specific implementation methods of the above steps are introduced below.
[0063] In some embodiments, in S110, the target object may be any one of a person, an animal, a plant, and an object. The first image may include one target object or multiple target objects. In addition, the first image may include the entire target object or a portion of the target object, which is not limited here.
[0064] In some embodiments, in S120, the large model generally refers to a deep learning model whose parameter scale and / or training data scale reaches tens of millions or more, and has strong generalization ability. The detection large model can be a large model implemented based on the target detection algorithm. The target detection algorithm can detect objects such as people, animals, plants, and objects in the image to obtain the position of the object in the image. The detection large model may include Grounding DINO, OvarNet, etc. The first detection large model may be an open source detection large model that has not been additionally trained for the annotation task corresponding to the target object.
[0065] As an example, if the first image is input into the first detection model, the first detection model can detect each object in the first image, obtain a coordinate frame corresponding to each object, and output the object name of each object and its corresponding coordinate frame. If the first image and the name of the target object are input into the first detection model together, the first detection model can detect the target object in the first image, obtain a first coordinate frame corresponding to the target object, and output the first coordinate frame. The first coordinate frame can be the minimum coordinate frame that includes the target object.
[0066] In addition, the segmentation model can be a model based on an image segmentation algorithm. The image segmentation algorithm can be used to locate objects and boundaries (lines, curves, etc.) in an image and divide an image into multiple parts. The image segmentation algorithm can be executed at the pixel level, assigning each pixel in the image to a specific object. The segmentation model can include, for example, Segment Anything Model (SAM), fastSAM, mobile SAM, etc. The first segmentation model can be an open source segmentation model that has not been additionally trained for the annotation task corresponding to the target object.
[0067] As an example, if the first image is input into the first segmentation model, the first segmentation model can segment each object in the first image to obtain a segmentation heat map of the first image. In the segmentation heat map, each pixel can represent an object. If the first image and the name of the target object are input into the first segmentation model together, the first segmentation model can segment the target object in the first image to obtain multiple pixel points of the target object. For example, if the target object is a "dog" and the first image includes a "dog", the multiple pixel points of the "dog" may include [[233, 233], [233, 244]...[333, 333], [333, 334]].
[0068] Based on this, in order to improve the accuracy of the first coordinate frame obtained by detection, and thus improve the accuracy of the labeling result, in some embodiments, before the above S120, the following may also be included:
[0069] Acquire a second data set, the second data set including a plurality of second images, each of the second images including a target object and a corresponding coordinate frame;
[0070] Using the second data set to train the first detection model to obtain a second detection model;
[0071] Based on this, the above S120 may specifically include:
[0072] The target object in the first image is detected using the second detection model.
[0073] Here, the second data set may be a data set obtained by manual annotation. The number of second images in the second data set may be much smaller than the number of image data used to train the specific task model. Generally, manual annotation can ensure the accuracy of the annotation results. That is, the data quality in the second data set is relatively high. In addition, the second image may include one target object or multiple target objects. The second image may include the entire target object or a portion of the target object, which is not limited here.
[0074] In this way, for the labeling task corresponding to the target object, by using the second data set to perform additional training on the first detection large model to obtain the second detection large model, and using the second detection large model to detect the target object in the first image, the accuracy of the first coordinate frame obtained by detection can be improved, thereby improving the accuracy of the labeling result.
[0075] Based on this, in order to improve the accuracy of the multiple pixel points obtained by segmentation, and thus improve the accuracy of the labeling result, in some embodiments, before the above S120, the following may also be included:
[0076] Acquire a third data set, the third data set including a plurality of third images, each of the third images including a target object and a plurality of corresponding pixels;
[0077] Using the third data set to train the first segmentation large model to obtain a second segmentation large model;
[0078] Based on this, the above S120 may specifically include:
[0079] The target object in the first image is segmented using the second segmentation model.
[0080] Here, the third data set may be a data set obtained through manual annotation. The third data set may be the same as or different from the second data set, which is not limited here. Specifically, the number of third images in the third data set may be much smaller than the number of image data used to train a specific task model. Generally, manual annotation can ensure the accuracy of the annotation results. That is, the data quality in the third data set is relatively high. In addition, the third image may include one target object or multiple target objects. The third image may include the entire target object or a part of the target object, which is not limited here.
[0081] In this way, for the labeling task corresponding to the target object, by using the third data set to perform additional training on the first segmentation large model to obtain the second segmentation large model, and using the second segmentation large model to segment the target object in the first image, the accuracy of the multiple pixel points obtained by segmentation can be improved, thereby improving the accuracy of the labeling results.
[0082] In some embodiments, in S130, the minimum circumscribed polygon may be a minimum circumscribed rectangle. After obtaining the plurality of pixels, the minimum circumscribed polygon of the plurality of pixels may be determined by an existing method to obtain a second coordinate frame including the target object, which is not limited here.
[0083] In some embodiments, in S140 and S150, since the same first image is input into the first detection large model and the first segmentation large model respectively, the first coordinate frame and the second coordinate frame can be exactly the same for the same target object in the first image. Since there may be errors in the output result of the model, whether the first coordinate frame is the same as the second coordinate frame can be determined by presetting the overlap range. Among them, the preset overlap range can be a preset overlap range between the first coordinate frame and the second coordinate frame that can characterize the first coordinate frame and the second coordinate frame to be the same. If the actual overlap range between the first coordinate frame and the second coordinate frame is greater than the preset overlap range, it can be determined that the first coordinate frame is the same as the second coordinate frame. If the actual overlap range between the first coordinate frame and the second coordinate frame is less than or equal to the preset overlap range, it can be determined that the first coordinate frame is not the same as the second coordinate frame.
[0084] Based on this, in order to accurately determine whether the first coordinate frame is the same as the second coordinate frame, in some embodiments, the above S140 may specifically include:
[0085] The first coordinate frame and the second coordinate frame are matched using the Hungarian matching algorithm to obtain a matching degree;
[0086] When the matching degree is greater than a preset matching degree, the first coordinate frame or the second coordinate frame is determined as the target coordinate frame.
[0087] Here, the preset matching degree may be a preset matching degree between the first coordinate frame and the second coordinate frame that can characterize that the first coordinate frame and the second coordinate frame are identical. If the matching degree between the first coordinate frame and the second coordinate frame is greater than the preset matching degree, it can be determined that the first coordinate frame and the second coordinate frame are identical. If the matching degree between the first coordinate frame and the second coordinate frame is less than or equal to the preset matching degree, it can be determined that the first coordinate frame and the second coordinate frame are not identical.
[0088] In addition, as an example, since the first detection large model and the first segmentation large model are both large models with strong generalization capabilities, the output results of the model usually have a certain degree of accuracy. Therefore, if the first coordinate frame and the second coordinate frame are the same, it can be determined that the first coordinate frame and the second coordinate frame are both accurate. Therefore, any one of the first coordinate frame and the second coordinate frame can be determined as the target coordinate frame, and the target coordinate frame can be determined as the annotation result corresponding to the target object. Among them, determining the target coordinate frame as the annotation result corresponding to the target object can be to retain and store the first image including the target coordinate frame, or to retain and store the segmentation heat map including the target coordinate frame, which is not limited here. Specifically, in the case where the target coordinate frame is the first coordinate frame, the first image including the first coordinate frame can be directly retained and stored. In the case where the target coordinate frame is the second coordinate frame, the second coordinate frame in the segmentation heat map can be first mapped to the first image, and then the first image including the second coordinate frame can be retained and stored. It is also possible to directly retain and store the segmentation heat map including the second coordinate frame. It is also possible to simultaneously retain and store the first image including the second coordinate frame and the segmentation heat map including the second coordinate frame.
[0089] As another example, if the first coordinate frame is different from the second coordinate frame, it can be considered that one of the first coordinate frame and the second coordinate frame is accurate, while the other is inaccurate.
[0090] Based on this, in order to ensure the accuracy of the target coordinate frame, in some embodiments, after the above S130, the following steps may also be included:
[0091] When the overlap range between the first coordinate frame and the second coordinate frame is not greater than the preset overlap range, obtaining a first probability corresponding to the first coordinate frame output by the first detection large model and a second probability corresponding to the second coordinate frame output by the first segmentation large model;
[0092] Determine a larger probability of the first probability and the second probability as a first target probability;
[0093] A coordinate frame corresponding to the first target probability is determined as a target coordinate frame.
[0094] Here, when the first detection model outputs the first coordinate frame, it can also output the first probability corresponding to the first coordinate frame. In addition, when the first segmentation model outputs multiple pixels, it can also output the second probability corresponding to the multiple pixels. Since the second target frame is the minimum circumscribed polygon corresponding to the multiple pixels, the second probability and the second coordinate frame can correspond one to one.
[0095] After the first probability and the second probability are obtained, the magnitude relationship between the first probability and the second probability may be compared, and the larger probability may be determined as the first target probability.
[0096] In this way, by determining the coordinate frame corresponding to the first target probability as the target coordinate frame, the accuracy of the target coordinate frame can be guaranteed.
[0097] Based on this, in order to ensure the accuracy of the target coordinate frame, in some embodiments, after the above S130, the following steps may also be included:
[0098] When the overlap range between the first coordinate frame and the second coordinate frame is not greater than a preset overlap range, receiving a target input for selecting the first coordinate frame or the second coordinate frame from a user;
[0099] In response to the target input, the coordinate box selected by the user is determined as the target coordinate box.
[0100] Here, the target input may be an operation of the user selecting the first coordinate box or the second coordinate box through the annotation tool. The target input may include, for example, clicking a control corresponding to the coordinate box or inputting the name of the large model. In addition, the coordinate box selected by the user may be a relatively accurate coordinate box determined by the user himself.
[0101] In this way, by responding to the target input and determining the coordinate frame selected by the user as the target coordinate frame, the accuracy of the target coordinate frame can be guaranteed.
[0102] Based on this, in order to further improve the accuracy of the annotation results obtained by the subsequent annotation of the target large model, in some embodiments, after determining the target coordinate frame, the following steps may also be included:
[0103] Acquire a first data set, where the first data set is a training data set corresponding to a target large model, where the target large model is a large model corresponding to a second target probability or a large model corresponding to a coordinate box not selected by a user, and the second target probability is a smaller probability of a first probability corresponding to the first coordinate box and a second probability corresponding to the second coordinate box;
[0104] The target large model is iteratively optimized using the first data set.
[0105] Here, the first data set may include the annotated data obtained by manual annotation, and may also include the annotated data obtained by annotating the first detection large model and the first segmentation large model. Generally, manual annotation can ensure the accuracy of the annotation results. As mentioned above, the accuracy of the standard results can also be guaranteed by annotating the first detection large model and the first segmentation large model. Therefore, the data quality in the first data set is relatively high.
[0106] In addition, the target large model can be the first detection large model or the first segmentation large model. When the target large model is the first detection large model, the first data set may include multiple fourth images, and each fourth image may include the target object and its corresponding coordinate frame. The fourth image and the second image may be the same or different, which is not limited here. When the target large model is the first segmentation large model, the first data set may include multiple fifth images, and each fifth image may include the target object and its corresponding multiple pixel points. The fifth image and the third image may be the same or different, which is not limited here.
[0107] As an example, in the case where the target coordinate frame is determined by conditional screening (probability screening), if the coordinate frame corresponding to the first target probability is the first coordinate frame, the coordinate frame corresponding to the second target probability can be the second coordinate frame, that is, the large model corresponding to the second target probability can be the first segmentation large model. That is, the target large model can be the first segmentation large model. Therefore, the first data set can be used to iteratively optimize the first segmentation large model. If the coordinate frame corresponding to the first target probability is the second coordinate frame, the coordinate frame corresponding to the second target probability can be the first coordinate frame, that is, the large model corresponding to the second target probability can be the first detection large model. That is, the target large model can be the first detection large model. Therefore, the first data set can be used to iteratively optimize the first detection large model.
[0108] As another example, in the case where the target coordinate frame is determined by manual intervention, if the user selects the first coordinate frame, the target macromodel may be the first segmentation macromodel corresponding to the second coordinate frame. Therefore, the first segmentation macromodel may be iteratively optimized using the first data set. If the user selects the second coordinate frame, the target macromodel may be the first detection macromodel corresponding to the first coordinate frame. Therefore, the first detection macromodel may be iteratively optimized using the first data set.
[0109] In this way, for the labeling task corresponding to the target object, by iteratively optimizing the target large model using the first data set, the accuracy of the labeling results obtained by subsequent labeling of the target large model can be further improved.
[0110] In order to better describe the entire solution, some specific examples are given based on the above embodiments.
[0111] For example, Figure 2 As shown, by inputting the first image and the name of the target object into the first detection model, a first coordinate frame can be obtained. By inputting the first image and the name of the target object into the first segmentation model, a second coordinate frame can be obtained. By comparing the first coordinate frame and the second coordinate frame, an accurate target coordinate frame corresponding to the target object can be obtained, and the target coordinate frame can be determined as a labeling result corresponding to the target object. Among them, determining the target coordinate frame as a labeling result corresponding to the target object can be to retain and store the first image including the target coordinate frame, or to retain and store the segmentation heat map including the target coordinate frame. The first image including the target coordinate frame can be the fourth image. After obtaining the fourth image, the first detection model can be iteratively optimized using the labeling data set including the fourth image, and the labeling accuracy of the first detection model can be continuously improved. The segmentation heat map including the target coordinate frame can be the fifth image. After obtaining the fifth image, the first segmentation model can be iteratively optimized using the labeling data set including the fifth image, and the labeling accuracy of the first segmentation model can be continuously improved.
[0112] Based on the data annotation method provided in the above embodiment, the present application also provides a specific implementation of the data annotation device. Please refer to the following embodiment.
[0113] like Figure 3 As shown, the data labeling device 300 provided in the embodiment of the present application includes the following modules:
[0114] A first acquisition module 310 is used to acquire a first image, where the first image includes a target object to be labeled;
[0115] The model processing module 320 is used to detect the target object in the first image using the first detection model to obtain a first coordinate frame including the target object, and to segment the target object in the first image using the first segmentation model to obtain a plurality of pixel points of the target object;
[0116] A first determining module 330, configured to determine a minimum circumscribed polygon of a plurality of pixels as a second coordinate frame including a target object;
[0117] A second determination module 340 is used to determine the first coordinate frame or the second coordinate frame as a target coordinate frame when the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range;
[0118] The third determination module 350 is used to determine the target coordinate frame as a labeling result corresponding to the target object.
[0119] The data labeling device 300 is described in detail below, as follows:
[0120] In some embodiments, the second determining module 340 may specifically include:
[0121] A matching submodule, used for matching the first coordinate frame and the second coordinate frame using a Hungarian matching algorithm to obtain a matching degree;
[0122] The first determination submodule is used to determine the first coordinate frame or the second coordinate frame as the target coordinate frame when the matching degree is greater than a preset matching degree.
[0123] In some embodiments, the data annotation device 300 may further include:
[0124] A second acquisition module is used to obtain, after obtaining a second coordinate frame including the target object, a first probability corresponding to the first coordinate frame output by the first detection large model and a second probability corresponding to the second coordinate frame output by the first segmentation large model when the overlap range between the first coordinate frame and the second coordinate frame is not greater than a preset overlap range;
[0125] A fourth determination module, configured to determine a larger probability between the first probability and the second probability as a first target probability;
[0126] The fifth determination module is used to determine the coordinate frame corresponding to the first target probability as the target coordinate frame.
[0127] In some embodiments, the data annotation device 300 may further include:
[0128] A receiving module, configured to receive a target input from a user for selecting the first coordinate box or the second coordinate box when the overlap range between the first coordinate box and the second coordinate box is not greater than a preset overlap range;
[0129] The sixth determination module is used to determine the coordinate box selected by the user as the target coordinate box in response to the target input.
[0130] In some embodiments, the data annotation device 300 may further include:
[0131] A third acquisition module is used to acquire a first data set after determining the target coordinate frame, where the first data set is a training data set corresponding to the target large model, the target large model is a large model corresponding to the second target probability or a large model corresponding to a coordinate frame not selected by the user, and the second target probability is a smaller probability of a first probability corresponding to the first coordinate frame and a second probability corresponding to the second coordinate frame;
[0132] The optimization module is used to iteratively optimize the target large model using the first data set.
[0133] In some embodiments, the data annotation device 300 may further include:
[0134] A fourth acquisition module is used to acquire a second data set before detecting the target object in the first image using the first detection model, where the second data set includes a plurality of second images, each of which includes the target object and its corresponding coordinate frame;
[0135] A first training module is used to train the first detection model using the second data set to obtain a second detection model;
[0136] Based on this, the model processing module 320 may specifically include:
[0137] The detection submodule is used to detect the target object in the first image using the second detection model.
[0138] In some embodiments, the data annotation device 300 may further include:
[0139] a fifth acquisition module, used for acquiring a third data set before segmenting the target object in the first image using the first segmentation model, wherein the third data set includes a plurality of third images, each of which includes the target object and a plurality of pixel points corresponding thereto;
[0140] A second training module is used to train the first segmentation large model using a third data set to obtain a second segmentation large model;
[0141] Based on this, the model processing module 320 may specifically include:
[0142] The segmentation submodule is used to segment the target object in the first image using the second segmentation model.
[0143] In the data annotation method of the embodiment of the present application, since the first detection large model and the first segmentation large model are both large models with strong generalization ability, for the annotation task corresponding to the target object, there is no need to perform additional training on the first detection large model and the first segmentation large model, and the first detection large model can be used to detect the target object in the first image to obtain a first coordinate frame including the target object, and the first segmentation large model can be used to segment the target object in the first image to obtain multiple pixel points of the target object. Since there is no need to perform additional training on the large model, the time of model training can be saved and the annotation efficiency can be improved. In addition, when the overlap range between the first coordinate frame and the second coordinate frame is greater than the preset overlap range, the first coordinate frame or the second coordinate frame is determined as the target coordinate frame, that is, by comparing the output results of the above two large models, any one of the output results is determined as the target coordinate frame when the two output results are close, and the accuracy of the target coordinate frame can be guaranteed. In this way, by determining the target coordinate frame as the annotation result corresponding to the target object, the accuracy of the annotation result can be guaranteed. That is, through the embodiment of the present application, the accuracy of the annotation result can be guaranteed and the annotation efficiency can be improved.
[0144] Based on the data labeling method provided in the above embodiment, the embodiment of the present application also provides a specific implementation of the electronic device. Figure 4 A schematic diagram of an electronic device 400 provided in an embodiment of the present application is shown.
[0145] The electronic device 400 may include a processor 410 and a memory 420 storing computer program instructions.
[0146] Specifically, the processor 410 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0147] The memory 420 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 420 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 420 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 420 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 420 is a non-volatile solid-state memory.
[0148] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present application.
[0149] The processor 410 implements any one of the data labeling methods in the above embodiments by reading and executing computer program instructions stored in the memory 420.
[0150] In one example, the electronic device 400 may further include a communication interface 430 and a bus 440. Figure 4 As shown, the processor 410, the memory 420, and the communication interface 430 are connected via a bus 440 and communicate with each other.
[0151] The communication interface 430 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0152] Bus 440 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 440 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
[0153] Exemplarily, the electronic device 400 may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).
[0154] The electronic device can execute the data labeling method in the embodiment of the present application, thereby realizing the combination Figures 1 to 3 The data annotation method and device described are as follows.
[0155] In addition, in combination with the data labeling method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data labeling methods in the above embodiment is implemented.
[0156] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0157] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0158] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0159] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.
[0160] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A data labeling method, characterized in that: include: Acquire a first image, wherein the first image includes a target object to be labeled; Detecting the target object in the first image using a first detection large model to obtain a first coordinate frame including the target object, and segmenting the target object in the first image using a first segmentation large model to obtain a plurality of pixel points of the target object; Determine a minimum circumscribed polygon of the plurality of pixels as a second coordinate frame including the target object; When the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range, determining the first coordinate frame or the second coordinate frame as a target coordinate frame; The target coordinate frame is determined as a labeling result corresponding to the target object.
2. The method according to claim 1, characterized in that The step of determining the first coordinate frame or the second coordinate frame as a target coordinate frame when the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range includes: Using the Hungarian matching algorithm to match the first coordinate frame and the second coordinate frame to obtain a matching degree; When the matching degree is greater than a preset matching degree, the first coordinate frame or the second coordinate frame is determined as a target coordinate frame.
3. The method according to claim 1, characterized in that After obtaining the second coordinate frame including the target object, the method further includes: When the overlap range between the first coordinate frame and the second coordinate frame is not greater than the preset overlap range, obtaining a first probability corresponding to the first coordinate frame output by the first detection large model and a second probability corresponding to the second coordinate frame output by the first segmentation large model; Determine a larger probability between the first probability and the second probability as a first target probability; A coordinate frame corresponding to the first target probability is determined as a target coordinate frame.
4. The method according to claim 1, characterized in that After obtaining the second coordinate frame including the target object, the method further includes: When the overlap range between the first coordinate frame and the second coordinate frame is not greater than the preset overlap range, receiving a target input for selecting the first coordinate frame or the second coordinate frame from a user; In response to the target input, the coordinate box selected by the user is determined as a target coordinate box.
5. The method according to claim 3 or 4, characterized in that: After determining the target coordinate frame, the method further includes: Acquire a first data set, where the first data set is a training data set corresponding to a target large model, where the target large model is a large model corresponding to a second target probability or a large model corresponding to a coordinate box not selected by a user, and the second target probability is a smaller probability between a first probability corresponding to the first coordinate box and a second probability corresponding to the second coordinate box; The target large model is iteratively optimized using the first data set.
6. The method according to claim 1, characterized in that Before detecting the target object in the first image using the first detection model, the method further includes: Acquire a second data set, wherein the second data set includes a plurality of second images, and each of the second images includes the target object and its corresponding coordinate frame; Using the second data set to train the first detection large model to obtain a second detection large model; The detecting the target object in the first image by using the first detection model includes: The target object in the first image is detected using the second detection model.
7. The method according to claim 1, characterized in that Before segmenting the target object in the first image using the first segmentation model, the method further includes: Acquire a third data set, wherein the third data set includes a plurality of third images, and each of the third images includes the target object and a plurality of pixel points corresponding thereto; Using the third data set to train the first segmentation large model to obtain a second segmentation large model; The step of segmenting the target object in the first image by using the first segmentation model includes: The target object in the first image is segmented using the second segmentation model.
8. A data labeling device, characterized in that: The device comprises: A first acquisition module, used to acquire a first image, wherein the first image includes a target object to be labeled; a model processing module, configured to detect the target object in the first image using the first detection large model to obtain a first coordinate frame including the target object, and to segment the target object in the first image using the first segmentation large model to obtain a plurality of pixel points of the target object; A first determining module, configured to determine the minimum circumscribed polygon of the plurality of pixels as a second coordinate frame including the target object; A second determination module, configured to determine the first coordinate frame or the second coordinate frame as a target coordinate frame when the overlap range between the first coordinate frame and the second coordinate frame is greater than a preset overlap range; The third determination module is used to determine the target coordinate frame as a labeling result corresponding to the target object.
9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data labeling method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data labeling method according to any one of claims 1 to 7 is implemented.