Image data annotation method and device

Abstract

The embodiment of the present application provides a method and device for labeling image data, wherein the method includes: obtaining image data to be labeled; using a preset detection network model to determine a target area in the image data to be labeled, and labeling the position of the target area; using a preset classification network model to identify the image data at the position of the target area, and labeling the recognition result. This solution first uses a pre-trained detection network model to find and label the target area where the target object is located, and then uses a pre-trained classification network model to identify and label whether the image data in the target area is the target object, thereby solving the technical problems of poor accuracy and low efficiency in existing data labeling methods, and achieving the technical effect of being able to automatically label a large amount of image data efficiently and accurately.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method and device for labeling image data. Background Art

[0002] In the field of image recognition, it is usually necessary to first label the sample image data, and then use the labeled sample image data to learn and train the neural network model to obtain the neural network model for image recognition.

[0003] In order to obtain labeled sample image data for learning and training, most of the existing methods are to annotate the collected sample data by manual annotation. Specifically, the technicians rely on their own experience and understanding of the image content to frame the target objects in the image one by one and make corresponding annotations. As can be seen from the above, the existing image data annotation methods need to rely on the personal experience and understanding of the image content of the technicians during the implementation process, resulting in different technicians relying on different experiences and understandings of the image content. The labeled sample image data often has differences and is not objective enough. In addition, the existing methods use manual annotation, and the annotation speed is relatively slow. And because the number of images that usually need to be annotated is relatively large, for example, it may reach several T, which leads to errors in the implementation process of the existing methods. In summary, when the existing methods are implemented, there are often technical problems of poor accuracy and low efficiency.

[0004] Currently, no effective solution has been proposed to the above problems. Summary of the invention

[0005] The embodiments of the present application provide a method and device for labeling image data to solve the technical problems of poor accuracy and low efficiency in existing data labeling methods, and achieve the technical effect of automatically labeling a large amount of image data efficiently and accurately.

[0006] The present application provides a method for labeling image data, including:

[0007] Obtain image data to be annotated;

[0008] Determine the target area in the image data to be marked by using a preset detection network model, and mark the position of the target area;

[0009] The image data at the location of the target area is identified using a preset classification network model, and the identification result is marked.

[0010] In one embodiment, after acquiring the image data to be annotated, the method further includes:

[0011] Performing equalization processing on the image data to be labeled.

[0012] In one embodiment, performing equalization processing on the image data to be labeled includes:

[0013] Generating a histogram of image channel pixel values ​​according to the image data to be annotated;

[0014] Linearly expanding the channel pixel values ​​within a threshold range in the histogram of the channel pixel values ​​of the image;

[0015] According to the histogram of the pixel values ​​of the image channels after the linear expansion, the image data to be labeled after the equalization processing is determined.

[0016] In one embodiment, the image data at the location of the target area is identified using a preset classification network model, including:

[0017] extracting image data at the location of the target area;

[0018] The preset classification network model is used to perform target object recognition on the image data at the target area position to generate a confidence parameter, and the recognition result is determined based on the confidence parameter, wherein the confidence parameter is used to indicate the probability that the image data at the target area position is the target object.

[0019] In one embodiment, the preset detection network model is established in the following manner:

[0020] Acquire a first training sample, wherein the first training sample includes image data with a location of a target object marked thereon;

[0021] The first training sample is used to train and learn the first neural network model to obtain the preset detection network model.

[0022] In one embodiment, training and learning a first neural network model using the first training sample includes:

[0023] The first neural network model is trained and learned using the first training sample by means of transfer learning.

[0024] In one embodiment, the preset classification network model is established in the following manner:

[0025] Acquire a second training sample, wherein the second training sample includes image data labeled with a target object name;

[0026] The second neural network model is trained and learned using the second training sample to obtain the preset classification network model.

[0027] In one embodiment, the second neural network model includes an improved YOLOv3 model, wherein the improved YOLOv3 model is improved in the following manner:

[0028] In the case of multi-target detection, the number of regression classifiers in the YOLOv3 model is modified to multiple;

[0029] In the case of single target detection, the number of regression classifiers in the YOLOv3 model is modified to a single one.

[0030] In one embodiment, the target object includes a face or a license plate.

[0031] The present application also provides an image data annotation device, including:

[0032] An acquisition module, used for acquiring image data to be annotated;

[0033] A first labeling module, used to determine a target area in the image data to be labeled using a preset detection network model, and to label a position of the target area;

[0034] The second labeling module is used to identify the image data at the location of the target area using a preset classification network model and label the identification result.

[0035] In one embodiment, the device further includes an equalization processing module, which is used to perform equalization processing on the image data to be labeled.

[0036] In one embodiment, the first annotation module includes:

[0037] A first extraction unit, used to extract image data at the location of the target area;

[0038] The first labeling unit is used to use the preset classification network model to perform target object recognition on the image data at the target area position to generate a confidence parameter, and determine the recognition result according to the confidence parameter, wherein the confidence parameter is used to indicate the probability that the image data at the target area position is the target object.

[0039] In one embodiment, the apparatus further comprises a first establishing module, wherein the first establishing module comprises:

[0040] A first acquisition unit, configured to acquire a first training sample, wherein the first training sample includes image data marked with a location of a target object;

[0041] The first learning unit is used to train and learn the first neural network model using the first training sample to obtain the preset detection network model.

[0042] In one embodiment, the apparatus further includes a second establishing module, wherein the second establishing module includes:

[0043] A second acquisition unit, configured to acquire a second training sample, wherein the second training sample includes image data labeled with a target object name;

[0044] The second learning unit is used to train and learn the second neural network model using the second training sample to obtain the preset classification network model.

[0045] An embodiment of the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, are used to obtain image data to be labeled; determine a target area in the image data to be labeled using a preset detection network model, and mark the position of the target area; identify the image data at the position of the target area using a preset classification network model, and mark the identification result.

[0046] In an embodiment of the present application, a pre-trained detection network model is first used to find and mark the target area where the target object is located, and then a pre-trained classification network model is used to identify and mark whether the image data in the target area is the target object, thereby solving the technical problems of poor accuracy and low efficiency existing in the existing data labeling methods, and achieving the technical effect of automatically labeling a large amount of image data efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 is a processing flow chart of an image data annotation method provided in accordance with an embodiment of the present application;

[0049] Figure 2 is a structural diagram of a device for labeling image data provided according to an embodiment of the present application;

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device based on the image data annotation method provided in an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of a histogram of image channel pixel values ​​obtained by applying the image data annotation method and device provided in an embodiment of the present application in a scene example;

[0052] Figure 5 is a schematic diagram comparing an image after equalization processing and an image not subjected to equalization processing obtained by applying the image data annotation method and device provided in an embodiment of the present application in a scene example;

[0053] Figure 6 This is a schematic diagram of the structure of the YOLOv3 model without improvement in a scene example;

[0054] Figure 7 It is a schematic diagram of a labeled face image obtained by applying the image data labeling method and device provided in an embodiment of the present application in a scene example. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0056] Considering the existing image data annotation methods, they are often based on the technicians' experience and understanding of the image content, and the sample image data used for learning and training are annotated by manual annotation, that is, the technicians need to find and annotate the position, length, width, center point, etc. of the target object in the image frame by frame in each image data. Since the experience and understanding of the image content based on different technicians are often different, the results of annotation by different technicians are also different, and there is a certain subjective influence. And the amount of image data that needs to be annotated is usually large, for example, it may be as much as several T. At this time, if the manual annotation is performed by technicians alone, it is easy to make errors, and the annotation speed is relatively poor. Therefore, when the existing image data annotation methods are implemented, there are often technical problems such as poor annotation accuracy and low annotation efficiency. In view of the root cause of the above technical problems, this application considers whether it is possible to use computers and other equipment to automatically annotate a large amount of image data through deep learning. Specifically, we consider first using a pre-trained detection network model to find and mark the target area where the target object is located, and then using a pre-trained classification network model to identify and mark whether the image data in the target area is the target object, thereby solving the technical problems of poor accuracy and low efficiency in existing data labeling methods, and achieving the technical effect of automatically labeling large amounts of image data efficiently and accurately.

[0057] Based on the above thinking, this embodiment of the application provides a method for labeling image data. Figure 1 The processing flow chart of the image data annotation method provided in accordance with the embodiment of the present application is shown. The image data annotation method provided in the embodiment of the present application may include the following steps during specific implementation.

[0058] S11: Obtain image data to be annotated.

[0059] In this embodiment, the above-mentioned image data can be specifically understood as training sample data for learning and training neural network models such as target object recognition models (such as face recognition models). Of course, the target object recognition models listed above are only for better illustrating the implementation methods of this application. During specific implementation, the image annotation method provided in the embodiment of this application can also be extended to the annotation of image data in other types of scenes according to specific application scenarios. This application is not limited to this.

[0060] In this embodiment, the image data may specifically include data containing the target object and data not containing the target object. The target object may be determined according to a specific application scenario. For example, when learning and training a face recognition model, the target object may be a face, and accordingly, the image data may include image data containing a face and image data not containing a face. When learning and training a license plate recognition model, the target object may be a license plate, and accordingly, the image data may include image data containing a license plate and image data not containing a license plate.

[0061] In one embodiment, the image data to be annotated may be a photo or image data captured from a video. The present application does not limit the specific form of the image data.

[0062] In one embodiment, it is considered that the image data directly acquired is often rough. For example, many image data to be labeled are collected in a poor light environment (e.g., too bright or too dark), resulting in the target object in the obtained image data being not prominent or significant enough, which in turn affects the accuracy of subsequent recognition and labeling using a neural network model. Therefore, in this embodiment, in order to improve the image quality of the image data to be labeled and improve the robustness of the algorithm, after acquiring the image data to be labeled, the method may also include the following when implemented: performing equalization processing on the image data to be labeled.

[0063] In this embodiment, the above-mentioned equalization processing can be specifically understood as performing black and white balance on the image data, so that the image features of the target object in the image data are more obvious and clear, which is convenient for subsequent model extraction and analysis, and improves the image quality of the image data.

[0064] In one embodiment, the equalization processing of the image data to be labeled may include the following contents when implemented:

[0065] S1: generating a histogram of image channel pixel values ​​according to the image data to be annotated;

[0066] S2: linearly expanding the channel pixel values ​​within the threshold range in the histogram of the channel pixel values ​​of the image;

[0067] S3: Determine the image data to be labeled after equalization processing according to the histogram of the pixel values ​​of the image channels after linear expansion.

[0068] In this embodiment, the image data to be annotated may be first converted into a histogram of image channel pixel values, wherein the abscissa in the histogram of image channel pixel values ​​represents different channels, and the ordinate represents the pixel value corresponding to each channel.

[0069] In the present embodiment, the above-mentioned threshold region can be specifically understood as a plurality of channel regions where pixel values ​​are relatively concentrated in the histogram of the above-mentioned image channel pixel values. Specifically, the pixel threshold can be set according to the brightness of the image data, and the channel region where the pixel value in the histogram of the image channel pixel value is greater than or equal to the above-mentioned pixel threshold is determined as the above-mentioned threshold region. Of course, it should be noted that the above-mentioned pixel threshold can be flexibly set according to the specific situation and accuracy requirements of the image data. The specific value of the pixel threshold is not limited in this application.

[0070] In this embodiment, the linear expansion of the channel pixel values ​​within the threshold range in the histogram of the image channel pixel values ​​may specifically include: expanding the pixel interval of the channel pixel values ​​within the threshold range in the histogram of the image channel pixel values ​​to 0 to 225. This can make the image features of the target object in the original image data more prominent and clear.

[0071] In this embodiment, after obtaining the histogram of the pixel values ​​of the image channel after linear expansion, the histogram of the pixel values ​​of the image channel after linear expansion can be converted into image data, that is, the image data to be labeled after equalization processing is obtained. Then, the image data after equalization processing can be used as input data for recognition and labeling using the following neural network model that has been learned and trained.

[0072] S12: Determine the target area in the image data to be marked using a preset detection network model, and mark the position of the target area.

[0073] In this embodiment, the above-mentioned preset inspection network model can be specifically understood as a pre-learned and trained neural network model for detecting and identifying the area where the target area object is located.

[0074] In this embodiment, the above-mentioned target area can be specifically understood as the image area of ​​the target object in the image data that is identified and selected by the preset detection network model.

[0075] In one embodiment, the above-mentioned use of a preset detection network model to determine the target area in the image data to be annotated and mark the position of the target area may include the following contents during specific implementation: taking the image data to be annotated as input, and using a preset detection network model to detect the image data to be annotated to identify the target area where the target object is located in the image data to be annotated, and marking the above-mentioned target area with a marking box, and marking the position parameters of the target area (for example, the horizontal and vertical coordinates of the marked target area or the marking box, and the offset distance from the center point, etc.).

[0076] In this embodiment, the marking frame may be a rectangular frame for marking, or a circular frame, etc. The specific shape of the marking frame is not limited in this application.

[0077] In this implementation, the above-mentioned preset detection network model can be established and obtained in the following manner when it is implemented:

[0078] S1: Acquire a first training sample, wherein the first training sample includes image data marked with a location of a target object;

[0079] S2: Using the first training sample to train and learn the first neural network model to obtain the preset detection network model.

[0080] In this embodiment, the first training sample may specifically include only image data marked with the position of the target object, or may include both image data marked with the position of the target object and image data without marking. For example, a portion of the first training sample may be image data marked with the position of the target object, and the other portion may be image data without marking.

[0081] In this embodiment, the image data marked with the position of the target object included in the first training sample may be image data in which the area where the target object is located is marked with a label frame in advance. For example, for image data used for face recognition, the image data marked with the position of the target object included in the first training sample may be image data in which the position of the face in the image is framed with a label frame in advance.

[0082] In this embodiment, the image data marked with the position of the target object included in the above-mentioned first training sample can be obtained in the following manner: use labeling (an image processing software) to frame and mark the face area in the image data with a rectangular annotation box, and save the position coordinates and length and width parameters of the annotation box to generate an XML format file; then convert the XML format file into a CSV format file, thereby obtaining the image data marked with the position of the target object included in the above-mentioned first training sample.

[0083] In this embodiment, by using image data including image data marked with the location of the target object and image data without labels to learn and train the neural network, a neural network model that can automatically identify the location of the object in the target image data can be obtained, that is, the above-mentioned preset detection network model.

[0084] In one embodiment, in order to improve the recognition accuracy of the trained preset detection network model, before training, the first training sample can be equalized to obtain the first training sample after equalization to highlight the image features in the first training sample; then the first training sample after equalization is used for learning and training to obtain a detection network model with higher accuracy.

[0085] In this embodiment, the first neural network model can be a neural network model based on YOLO. Specifically, the YOLO can be understood as an object detection algorithm based on deep learning. The algorithm mainly processes the object detection problem into a regression problem, and can use a convolutional neural network structure to predict the object bounding box and category probability according to the input image. YOLO is different from R-CNN (Regions with CNN features) and Fast R-CNN (Fast Regions with CNN features) used in existing methods, and has the characteristics of fast processing speed, small error, and high accuracy.

[0086] In one embodiment, in order to improve the efficiency of learning and training the first neural network model, in a specific implementation, the above-mentioned use of the first training sample to train and learn the first neural network model may include: using the first training sample to train and learn the first neural network model through a transfer learning method. In this way, since the first neural network model and the detection network model to be established have certain similar features, in the process of learning and training, the parameters of the front layer in the first neural network model can be kept unchanged, and only some parameters of the back layer can be modified accordingly, and then the modified first neural network model can be used for training and learning, so as to converge faster to obtain a preset detection network model that meets the requirements.

[0087] S13: Using a preset classification network model to identify the image data at the location of the target area, and marking the identification result.

[0088] In this embodiment, the above-mentioned preset classification network model can be specifically understood as a pre-learned and trained neural network model for identifying and judging whether the image content at the target area position is the target object.

[0089] In one embodiment, in order to improve the efficiency and accuracy of recognition, in a specific implementation, the above-mentioned preset classification network model can be used to recognize only the image data at the location of the target area in the image to be labeled, obtain the recognition result of the target area, and mark the recognition result of the target area as the recognition result of the image to be labeled. In this way, it is possible to avoid recognizing image data where the target object does not exist in the image to be labeled, thereby improving the efficiency of recognition and avoiding interference of image data of non-target objects on recognition.

[0090] In one embodiment, the above-mentioned recognition of the image data at the location of the target area by using a preset classification network model may include the following contents during specific implementation:

[0091] S1: extracting image data at the location of the target area;

[0092] S2: Use the preset classification network model to perform target object recognition on the image data at the target area position to generate a confidence parameter, and determine the recognition result based on the confidence parameter, wherein the confidence parameter is used to indicate the probability that the image data at the target area position is the target object.

[0093] In this embodiment, in order to improve the recognition speed of the preset classification network model, only the image data at the position of the target area in the image data to be annotated can be extracted for recognition and annotation according to the position of the target area. For example, according to the position of the target area, the image data at the position of the target area in the image data to be annotated can be extracted as input, and the preset classification network model can be used to perform target object recognition.

[0094] In this embodiment, the above confidence parameter can be specifically understood as a probability for indicating whether the image data at the target area position is the target object. According to the above confidence parameter, it can be determined whether the image content at the target area in the image to be annotated is the target object, and then the corresponding recognition result is determined. Generally, the higher the value of the above confidence parameter, the greater the probability that the image content in the target area is the target object. During specific implementation, a threshold probability can be set according to the accuracy requirements to detect whether the value of the confidence parameter is greater than the threshold probability. If the value of the confidence parameter is greater than the above threshold probability, it is determined that the image content in the target area is the target object and the recognition is successful. If it is less than or equal to the above threshold probability, it is determined that the image content in the target area is not the target object and the recognition fails.

[0095] In one embodiment, the preset classification network model can be established and obtained in the following manner when it is implemented:

[0096] S1: Acquire a second training sample, wherein the second training sample includes image data labeled with a target object name;

[0097] S2: Using the second training sample to train the second neural network model to obtain the preset classification network model.

[0098] In this embodiment, the second training sample may specifically include only image data labeled with the target object name, or may include both image data labeled with the target object name and unlabeled image data. For example, a portion of the second training sample may be image data labeled with the target object name, and the other portion may be unlabeled image data.

[0099] In this embodiment, the image data labeled with the target object name included in the second training sample may be image data with the content name corresponding to the target object labeled in advance using identification information. For example, for image data used for face recognition, the image data labeled with the target object name included in the second training sample may be image data with the "face" label labeled in advance using identification information.

[0100] In this embodiment, the image data marked with the target object name included in the above-mentioned second training sample can be obtained in the following manner: use labeling (an image processing software) to frame and mark the face area in the image data with a rectangular annotation box, save the face label, and generate an XML format file; then convert the XML format file into a CSV format file, thereby obtaining the image data marked with the target object name included in the above-mentioned second training sample.

[0101] In this embodiment, by using image data labeled with the name of the target object and unlabeled image data to learn and train the neural network, a neural network model can be obtained that can automatically identify whether the image content in the target area belongs to the target object, that is, the above-mentioned preset classification network model.

[0102] In one embodiment, in order to improve the recognition accuracy of the trained preset classification network model, before training, the second training sample can be equalized to obtain the second training sample after equalization to highlight the image features in the second training sample; then the second training sample after equalization is used for learning and training to obtain a detection network model with higher accuracy.

[0103] In one embodiment, in order to improve the efficiency of learning and training the second neural network model, in specific implementation, the above-mentioned use of the second training sample to train and learn the second neural network model may include: using the second training sample to train and learn the second neural network model (or using the preset detection network model as the second neural network model) through a transfer learning method. In this way, since the second neural network model and the detection network model to be established have certain similar features, in the process of learning and training, the parameters of the front layer in the second neural network model can be kept unchanged, and only some parameters of the back layer can be modified accordingly, and then the modified second neural network model can be used for training and learning, so as to converge faster to obtain a preset detection network model that meets the requirements.

[0104] In one embodiment, the second neural network model may specifically include an improved YOLOv3 model, wherein the improved YOLOv3 model is improved in the following manner:

[0105] In the case of multi-target detection, the number of regression classifiers in the YOLOv3 model is modified to multiple;

[0106] In the case of single target detection, the number of regression classifiers in the YOLOv3 model is modified to a single one.

[0107] In this embodiment, the above-mentioned YOLOv3 model can be specifically understood as a version of a neural network model in the neural network model of YOLO. Of course, it should be noted that the YOLOv3 model listed above is only for better illustrating the implementation method of this application. During specific implementation, other types or versions of neural network models can also be used according to specific circumstances. This application does not limit this.

[0108] In the present embodiment, it is further considered that the above-mentioned preset classification model can only judge whether the image content in the target area is the target object during the specific implementation, that is, the above-mentioned preset classification model can be a two-classification model (can be used to realize single target detection). The commonly used YOLOv3 model is mostly designed for multi-classification. Therefore, the commonly used YOLOv3 model usually includes regression classifications of multiple dimensions (such as multiple logistic classification methods) and multiple regression classifiers (such as multiple logistic regression classifiers), resulting in a relatively complex structure of the YOLOv3 model, a relatively large occupation of resources and time during operation, and a relatively slow processing speed. For the application scenario of the present application, it is actually not necessary to use all the multiple regression classifiers in the original YOLOv3 model. Therefore, in order to simplify the YOLOv3 model and improve the efficiency of target object recognition based on the above-mentioned model, the YOLOv3 model can be improved in the following manner during the specific implementation to obtain a suitable second neural network model: the number of regression classifiers in the YOLOv3 model is modified to a single one.

[0109] In one embodiment, considering that the above-mentioned preset classification model can also make more complex judgments on the image content in the target area during specific implementation, for example, the above-mentioned preset classification model can also be a multi-classification model (which can be used to implement multi-target detection). At this time, in order to meet the requirements and obtain multiple target types, the YOLOv3 model can also be improved in the following manner to obtain a suitable second neural network model: the number of regression classifiers in the YOLOv3 model is modified to multiple. It should be noted that the specific number of the above-mentioned multiple regression classifiers can be determined according to the number of types of the multiple target types to be classified.

[0110] In this embodiment, during specific implementation, multiple regression classifiers can be modified into a single regression classifier by modifying the regression parameters in the YOLOv3 model, which can meet the accuracy requirements for identifying the target object, while simplifying the model structure, improving the determination efficiency, and avoiding waste of resources.

[0111] In one embodiment, the target object may specifically include a face or a license plate. Of course, it should be noted that the above-mentioned faces or license plates are only for better illustrating the implementation of the present application. In specific implementation, other image contents may also be determined as target objects according to specific scenarios and annotation requirements. This application does not limit this.

[0112] From the above description, it can be seen that the image data labeling method provided in the embodiment of the present application first uses a pre-trained detection network model to find and label the target area where the target object is located, and then uses a pre-trained classification network model to identify and label whether the image data in the target area is the target object, thereby avoiding manual labeling, thereby solving the technical problems of poor accuracy and low efficiency in the existing data labeling methods, and achieving the technical effect of being able to automatically label a large amount of image data efficiently and accurately; and by connecting the above-mentioned detection network model and the classification network model in series, and using the classification network model to only detect and judge whether the image content in the target area labeled by the detection network model is the target object, the processing efficiency and accuracy are further improved; and according to specific requirements, the YOLOv3 model is targetedly improved, redundant regression classifiers are deleted, the model structure is simplified, and the processing efficiency is improved; and the acquired image data to be labeled is also equalized in advance to highlight the image features of the target object in the image data, improve data quality, and further improve the accuracy of recognition and labeling.

[0113] Based on the same inventive concept, an image data annotation device is also provided in an embodiment of the present invention, as described in the following embodiment. Since the principle of solving the problem by the image data annotation device is similar to that of the image data annotation method, the implementation of the image data annotation device can refer to the implementation of the image data annotation method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements the predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable. Please refer to Figure 2 , is a composition structure diagram of an image data annotation device provided in an embodiment of the present application. The device may specifically include: an acquisition module 21, a first annotation module 22, and a second annotation module 23. The structure is described in detail below.

[0114] An acquisition module 21, which can be specifically used to acquire image data to be annotated;

[0115] The first labeling module 22 may be specifically used to determine the target area in the image data to be labeled using a preset detection network model, and label the position of the target area;

[0116] The second labeling module 23 can be specifically used to identify the image data at the location of the target area using a preset classification network model, and label the identification result.

[0117] In one embodiment, in order to improve the data quality of the acquired image data to be annotated and highlight the image features of the target object in the image data, the above-mentioned device may also include an equalization processing module during its specific implementation, which can be specifically used to perform equalization processing on the image data to be annotated.

[0118] In one embodiment, in order to perform equalization processing on the image data to be labeled, the equalization processing module may include the following structural units when implemented:

[0119] A first processing unit, specifically configured to generate a histogram of image channel pixel values ​​according to the image data to be annotated;

[0120] An expansion unit, which may be specifically used to linearly expand channel pixel values ​​within a threshold range in a histogram of channel pixel values ​​of the image;

[0121] The second processing unit may be specifically configured to determine the image data to be labeled after the equalization processing according to the histogram of the pixel values ​​of the image channels after the linear expansion.

[0122] In one embodiment, in order to use a preset classification network model to identify the image data at the location of the target area, the first labeling module 22 may include the following structural units when implemented:

[0123] A first extraction unit, specifically configured to extract image data at a location of the target area;

[0124] The first labeling unit can be specifically used to use the preset classification network model to perform target object recognition on the image data at the target area position to generate a confidence parameter, and determine the recognition result according to the confidence parameter, wherein the confidence parameter is used to indicate the probability that the image data at the target area position is the target object.

[0125] In one embodiment, in order to obtain the preset detection network model, the apparatus may further include a first establishment module, wherein the first establishment module may specifically include the following structural units:

[0126] A first acquisition unit may be specifically used to acquire a first training sample, wherein the first training sample includes image data marked with a location of a target object;

[0127] The first learning unit can be specifically used to train and learn the first neural network model using the first training sample to obtain the preset detection network model.

[0128] In this embodiment, the first training sample may specifically include only image data marked with the position of the target object, or may include both image data marked with the position of the target object and image data without marking. For example, a portion of the first training sample may be image data marked with the position of the target object, and the other portion may be image data without marking.

[0129] In one embodiment, when the first learning unit is implemented, it can be executed according to the following procedure: using the first training sample to train and learn the first neural network model through a transfer learning method.

[0130] In one embodiment, in order to obtain the preset classification network model, the above-mentioned device may further include a second establishment module when it is implemented, wherein the above-mentioned second establishment module may specifically include the following structural units:

[0131] A second acquisition unit may be specifically configured to acquire a second training sample, wherein the second training sample includes image data labeled with a target object name;

[0132] The second learning unit can be specifically used to train and learn the second neural network model using the second training sample to obtain the preset classification network model.

[0133] In this embodiment, the second training sample may specifically include only image data labeled with the target object name, or may include both image data labeled with the target object name and unlabeled image data. For example, a portion of the second training sample may be image data labeled with the target object name, and the other portion may be unlabeled image data.

[0134] The second neural network model may specifically include an improved YOLOv3 model, wherein the improved YOLOv3 model is improved in the following manner:

[0135] In the case of multi-target detection, the number of regression classifiers in the YOLOv3 model is modified to multiple;

[0136] In the case of single target detection, the number of regression classifiers in the YOLOv3 model is modified to a single one.

[0137] In the present embodiment, it is further considered that the above-mentioned preset classification model can only judge whether the image content in the target area is the target object during the specific implementation, that is, the above-mentioned preset classification model can be a two-classification model (can be used to realize single target detection). The commonly used YOLOv3 model is mostly designed for multi-classification. Therefore, the commonly used YOLOv3 model usually includes regression classifications of multiple dimensions (such as multiple logistic classification methods) and multiple regression classifiers (such as multiple logistic regression classifiers), resulting in a relatively complex structure of the YOLOv3 model, a relatively large occupation of resources and time during operation, and a relatively slow processing speed. For the application scenario of the present application, it is actually not necessary to use all the multiple regression classifiers in the original YOLOv3 model. Therefore, in order to simplify the YOLOv3 model and improve the efficiency of target object recognition based on the above-mentioned model, the YOLOv3 model can be improved in the following manner during the specific implementation to obtain a suitable second neural network model: the number of regression classifiers in the YOLOv3 model is modified to a single one.

[0138] In one embodiment, considering that the above-mentioned preset classification model can also make more complex judgments on the image content in the target area during specific implementation, for example, the above-mentioned preset classification model can also be a multi-classification model (which can be used to implement multi-target detection). At this time, in order to meet the requirements and obtain multiple target types, the YOLOv3 model can also be improved in the following manner to obtain a suitable second neural network model: the number of regression classifiers in the YOLOv3 model is modified to multiple. It should be noted that the specific number of the above-mentioned multiple regression classifiers can be determined according to the number of types of the multiple target types to be classified.

[0139] In one embodiment, the target object may specifically include a face or a license plate, etc. Of course, it should be noted that the target objects listed above are only for better illustrating the implementation of the present application. During specific implementation, other content objects may be selected as the target objects according to specific application scenarios, and the image data annotation method provided in the embodiment of the present application may be extended and applied to other types of image annotation scenarios. This application does not limit this.

[0140] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] It should be noted that the systems, devices, modules or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, in this specification, the above devices are described in various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0142] In addition, in this specification, adjectives such as first and second may only be used to distinguish one element or action from another element or action, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be interpreted as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0143] The image data labeling device provided in the embodiment of the present application uses a pre-trained detection network model through a first labeling module to find and label the target area where the target object is located, and then uses a pre-trained classification network model through a second labeling module to identify and label whether the image data in the target area is the target object, thereby avoiding manual labeling, thereby solving the technical problems of poor accuracy and low efficiency in the existing data labeling methods, and achieving the technical effect of being able to automatically label a large amount of image data efficiently and accurately; and by connecting the above-mentioned detection network model and the classification network model in series, and using the classification network model to only detect and judge whether the image content in the target area labeled by the detection network model is the target object, the processing efficiency and accuracy are further improved; and according to specific needs, the YOLOv3 model is targetedly improved, redundant regression classifiers are deleted, the model structure is simplified, and the processing efficiency is improved; and the equalization processing module is used to perform equalization processing on the acquired image data to be labeled in advance to highlight the image features of the target object in the image data, improve data quality, and further improve the accuracy of recognition and labeling.

[0144] The present application also provides an electronic device. Figure 3As shown, a schematic diagram of the structure of an electronic device based on the image data annotation method provided in an embodiment of the present application is shown, and the electronic device may specifically include an input device 31, a processor 32, and a memory 33. Among them, the input device 31 can be specifically used for image data to be annotated. The processor 32 can be specifically used to determine the target area in the image data to be annotated using a preset detection network model, and mark the position of the target area; use a preset classification network model to identify the image data at the position of the target area, and mark the identification result. The memory 33 can be specifically used to store the image data to be annotated input through the input device 31, the intermediate data generated during the processing of the processor 32, and related program instructions.

[0145] In this embodiment, the input device may specifically be one of the main devices for information exchange between the user and the computer system. The input device may include a keyboard, a mouse, a camera, a scanner, a light pen, a handwriting input board, a voice input device, etc.; the input device is used to input the original data and the program for processing these numbers into the computer. The input device can also obtain and receive data transmitted from other modules, units, and devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of a computer-readable medium, a logic gate, a switch, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable logic controller, and an embedded microcontroller, etc., such as a microprocessor or a processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor. The memory may specifically be a memory device used to store information in modern information technology. The memory may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function without a physical form is also called a memory, such as a RAM, a FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, a TF card, etc.

[0146] In this embodiment, the functions and effects specifically realized by the electronic device can be explained in comparison with other embodiments and will not be described in detail here.

[0147] In an embodiment of the present application, a computer storage medium based on an image data labeling method is also provided, wherein the computer storage medium stores computer program instructions, which, when executed, implement: obtaining image data to be labeled; determining a target area in the image data to be labeled using a preset detection network model, and marking the position of the target area; identifying the image data at the position of the target area using a preset classification network model, and marking the identification result.

[0148] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD) or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0149] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments and will not be repeated here.

[0150] In a specific implementation scenario example, the image data annotation method and device provided in the embodiment of the present application are used to automatically annotate face image data used to train a face recognition model. The specific implementation process can be found in the following content.

[0151] S1: Perform equalization preprocessing on the image (i.e., the image data to be labeled).

[0152] In this embodiment, the image can be pre-processed to achieve a balanced white-black balance, making the image features of the corresponding target object more prominent. The specific method at the image level is as follows: First of all, it should be noted that the pixel value of each channel of the image should actually be distributed in the range of 0-255. Therefore, the pixel value of each channel can be counted to obtain the above-mentioned histogram (i.e., the histogram of the pixel value of the image channel). For details, please refer to Figure 4 The diagram shown is a schematic diagram of a histogram of image channel pixel values ​​obtained by applying the image data annotation method and device provided by the embodiment of the present application in a scene example. In fact, the pixel values ​​will appear in the corresponding interval (i.e., the threshold area), and then these pixel intervals are linearly expanded to 0-255, that is, the histogram is equalized. Then, the image to be annotated after the equalization processing is obtained according to the histogram after the equalization processing, which can be referred to Figure 5 The diagram shown is a comparison diagram of an image after equalization and an image not after equalization obtained by applying the image data annotation method and device provided in an embodiment of the present application in a scene example. The image quality of the image after equalization is significantly improved, and the image features of the target object in the image are more prominent and clear.

[0153] S2: Establish and train a target detection model (i.e., a preset detection network model) and a target classification model (i.e., a preset classification network model).

[0154] In this embodiment, it is necessary to first label some sample data in the training samples used to train the target detection model and the target classification model. Specifically, the labelimg tool can be used to directly frame the corresponding target: the face (i.e., the target object), and save it to generate the corresponding xml format. Among them, the location, length and width information of the target are specifically saved. The xml file will then be converted into a csv file for subsequent model training.

[0155] In this embodiment, when choosing which neural network to use as the basic network of the above-mentioned target detection model and target classification model, a comparative analysis is performed on common detection neural network models such as FasterRCNN and YOLO. In order to take into account the corresponding speed advantages, the YOLO series is considered to be used as the basic model and is optimized and modified accordingly. Among them, the reasons why the YOLO series is used are as follows: First, YOLO is a unified network. YOLO does not display the process of obtaining region proposals. Although RPN and fast rcnn share convolutional layers in Faster R-CNN, the RPN network and fast rcnn network need to be repeatedly trained during the model training process. Compared with the "look twice" (candidate box extraction and classification) of the R-CNN series, YOLO only needs Look Once. Therefore, YOLO is more efficient. Secondly, YOLO is unified into a regression problem, while R-CNN divides the detection results into two parts for solution: object category (classification problem) and object position, i.e., bounding box (regression problem).

[0156] In this implementation, for the transfer learning target classification model, combined with the specific scenario requirements, this example also improves and optimizes the original YOLOv3 model for face detection accordingly. In the original unimproved YOLOv3 model, its specific structure can be found in Figure 6 The structure diagram of the unmodified YOLOv3 model in a scene example is shown. After several convolution, pooling, residual and other operations, and after multiple scale operations, the image is regressed by logistic classifiers in multiple dimensions to obtain the corresponding detection target. This multiple logistic classifier method has a good effect on the detection tasks of multiple targets, but for the current single target (only face) task in this example, there is obviously a corresponding waste of efficiency. In response to this situation, the YOLOv3 model in this example has been modified accordingly, and the multiple logistic classifier methods have been adjusted to a single logistic regression classifier. In this way, the detection efficiency can be improved while achieving the corresponding detection accuracy.

[0157] In this implementation, it should be added that, compared with ordinary training tasks, transfer learning is also a process of minimizing cross entropy. For example, in tensorflow, when the loss in tensorboard is observed to have decreased and converged and no longer decreases, the model training can be basically judged to be completed. It can be used for verification, first looking at the efficiency.

[0158] In this embodiment, when it is implemented, in the detection task, the information of the training data is the position information and a larger image, where the detection target will be a part of the detection image and will be framed by the detection frame. However, for the classification task, in the face classification task, each training set is basically the whole face, or non-face, and the face or non-face part will occupy the vast majority of the picture.

[0159] S3: Use the object detection model and the object classification model to frame and label the faces in the image.

[0160] In this embodiment, two transfer learning models can be obtained through the above steps. One is the target detection model; the other is the target classification model. Then, the picture that needs to be labeled (i.e., the image data to be labeled) can be pre-processed by equalization. Then, the target detection model is used to detect the pre-processed picture to frame the position of the face. Furthermore, considering that the above-mentioned target detection model is still a model obtained by transfer learning, its detection accuracy may not reach the corresponding accuracy rate, so the detected face image can be cut out; then the cut-out face image is binary classified (face, non-face classification) using the target classification model. Finally, the classification result (i.e., recognition result) is determined and labeled. Among them, the classification result is a face, corresponding to a high-confidence labeling result; the classification result is a non-face, corresponding to a low-confidence labeling result. Please refer to Figure 7 The diagram shown is a schematic diagram of a labeled facial image obtained by applying the image data labeling method and device provided in an embodiment of the present application in a scene example.

[0161] By comparing and analyzing the image data annotation method in this example with the existing image data annotation methods, it can be seen that: in this example, an automated overall solution for annotating data is adopted, which greatly reduces the workload of manual annotation and greatly improves efficiency; a serial inspection method for detection tasks and classification tasks is also used. The data to be annotated will first obtain the target data through the detection network, and then the target data will be extracted and verified by the classification network to see if it is correct data. This double-check method is very helpful in improving the accuracy of the annotated data; YOLOv3 is also used as the basic network of the detection network, and the network is modified accordingly for specific face detection tasks, and the last redundant logistic classifier is cut off, which significantly improves the execution efficiency of the detection network; the original data used are also pre-processed before the detection and classification tasks. This image equalization processing is more conducive to the prominence of the image target itself, and has very good algorithm robustness for faces appearing in overly bright and dark environments.

[0162] Through the above-mentioned scenario examples, the image data labeling method and device provided in the embodiments of the present application are verified. By first using a pre-trained detection network model to find and label the target area where the target object is located, and then using a pre-trained classification network model to identify and label whether the image data in the target area is the target object, the technical problems of poor accuracy and low efficiency existing in the existing data labeling methods are indeed solved, and the technical effect of automatically labeling a large amount of image data efficiently and accurately is achieved.

[0163] Although different specific embodiments are mentioned in the content of this application, this application is not limited to the situations described in the industry standards or embodiments, etc. Some industry standards or slightly modified implementation plans based on the implementation of the custom methods or embodiments can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above embodiments. The embodiments of data acquisition, processing, output, judgment methods, etc. after these modifications or deformations can still fall within the scope of the optional implementation plans of this application.

[0164] Although the present application provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "includes", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0165] The devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by dividing them into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules. The device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0166] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0167] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0168] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0169] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0170] Although the present application has been described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present application without departing from the spirit of the present application. It is intended that the attached embodiments include these modifications and changes without departing from the present application.

Claims

1. A method for labeling image data, characterized in that: include: Obtain image data to be annotated; Determine the target area in the image data to be marked by using a preset detection network model, and mark the position of the target area; Using a preset classification network model to identify the image data at the location of the target area, and marking the identification result; The preset classification network model is established in the following manner: Acquire a second training sample, wherein the second training sample includes image data labeled with a target object name; Using the second training sample to train and learn the second neural network model to obtain the preset classification network model; The second neural network model includes an improved YOLOv3 model, wherein the improved YOLOv3 model is improved in the following manner: In the case of multi-target detection, the number of regression classifiers in the YOLOv3 model is modified to multiple; In the case of single target detection, the number of regression classifiers in the YOLOv3 model is modified to a single one.

2. The method according to claim 1, characterized in that After acquiring the image data to be annotated, the method further includes: Performing equalization processing on the image data to be labeled.

3. The method according to claim 2, characterized in that Performing equalization processing on the image data to be labeled, including: Generating a histogram of image channel pixel values ​​according to the image data to be annotated; Linearly expanding the channel pixel values ​​within a threshold range in the histogram of the channel pixel values ​​of the image; According to the histogram of the pixel values ​​of the image channels after the linear expansion, the image data to be labeled after the equalization processing is determined.

4. The method according to claim 1, characterized in that: Using a preset classification network model to identify the image data at the location of the target area includes: extracting image data at the location of the target area; The preset classification network model is used to perform target object recognition on the image data at the target area position to generate a confidence parameter, and the recognition result is determined based on the confidence parameter, wherein the confidence parameter is used to indicate the probability that the image data at the target area position is the target object.

5. The method according to claim 1, characterized in that The preset detection network model is established in the following manner: Acquire a first training sample, wherein the first training sample includes image data with a location of a target object marked thereon; The first training sample is used to train and learn the first neural network model to obtain the preset detection network model.

6. The method according to claim 5, characterized in that Using the first training sample to train a first neural network model includes: The first neural network model is trained and learned using the first training sample by means of transfer learning.

7. A device for labeling image data, characterized in that: include: An acquisition module, used for acquiring image data to be annotated; A first labeling module, used to determine a target area in the image data to be labeled using a preset detection network model, and to label a position of the target area; A second labeling module is used to identify the image data at the location of the target area using a preset classification network model and label the identification result; The device further includes a second establishing module, wherein the second establishing module includes: A second acquisition unit, configured to acquire a second training sample, wherein the second training sample includes image data labeled with a target object name; A second learning unit, used to train and learn a second neural network model using the second training sample to obtain the preset classification network model; The second neural network model includes an improved YOLOv3 model, wherein the improved YOLOv3 model is improved in the following manner: In the case of multi-target detection, the number of regression classifiers in the YOLOv3 model is modified to multiple; In the case of single target detection, the number of regression classifiers in the YOLOv3 model is modified to a single one.

8. The device according to claim 7, characterized in that The device also includes an equalization processing module, which is used to perform equalization processing on the image data to be labeled.

9. The device according to claim 7, characterized in that The first marking module includes: A first extraction unit, used to extract image data at the location of the target area; The first labeling unit is used to use the preset classification network model to perform target object recognition on the image data at the target area position to generate a confidence parameter, and determine the recognition result according to the confidence parameter, wherein the confidence parameter is used to indicate the probability that the image data at the target area position is the target object.

10. The device according to claim 7, characterized in that The device further includes a first establishing module, wherein the first establishing module includes: A first acquisition unit, configured to acquire a first training sample, wherein the first training sample includes image data marked with a location of a target object; The first learning unit is used to train and learn the first neural network model using the first training sample to obtain the preset detection network model.

11. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the steps of the method according to any one of claims 1 to 6 are implemented.

Images