Visual data labeling method and apparatus, electronic device, storage medium, and product

Abstract

Disclosed in embodiments of the present invention are a visual data labeling method and apparatus, an electronic device, a storage medium, and a product. The method comprises: determining unlabeled first visual data and unlabeled second visual data; determining a first component from a segmentation result of the first visual data; in response to a labeling operation on the first visual data, labeling the first visual data; segmenting a second component from the second visual data; determining a difference between a first pixel distribution of the first component in a color mode and a second pixel distribution of the second component in the color mode; and labeling the second visual data on the basis of the difference. Automatic labeling can be performed on visual data to automatically generate labels, thereby reducing the complexity and workload of the labeling process, and improving the quality of trained artificial intelligence models. In addition, training samples for retraining artificial intelligence models can be conveniently combined, thereby improving the accuracy of the models.

Description

Method, apparatus, electronic device, storage medium and product for labeling visual data TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision (CV), and in particular to a method, apparatus, electronic device, storage medium and product for labeling visual data. BACKGROUND

[0002] Computer vision is the simulation of biological vision using computers and related devices, which usually includes using cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to generate images suitable for human observation or transmission to instruments.

[0003] In recent years, artificial intelligence (AI) technology has developed rapidly and improved. However, in various use cases of computer vision based on artificial intelligence, a large amount of labeled data is usually required to train an artificial intelligence model.

[0004] Currently, manual data labeling is required. This is both time-consuming and expensive in terms of cost.

[0005] SUMMARY

[0006] Embodiments of the present application provide a method, apparatus, electronic device, storage medium and product for labeling visual data.

[0007] In a first aspect, a method for labeling visual data is provided. The method comprises:

[0008] determining unlabeled first visual data and unlabeled second visual data;

[0009] determining a first component from a segmentation result of the first visual data;

[0010] labeling the first visual data in response to a labeling operation on the first visual data;

[0011] segmenting a second component from the second visual data;

[0012] determining a difference between a first pixel distribution of the first component in a color mode and a second pixel distribution of the second component in the color mode;

[0013] labeling the second visual data based on the difference.

[0014] In a second aspect, an apparatus for labeling visual data is provided. The apparatus comprises:

[0015] a first determining module configured to determine unannotated first visual data and unannotated second visual data;

[0016] a first segmenting module configured to determine a first component from a segmentation result of the first visual data;

[0017] a first annotating module configured to annotate the first visual data in response to an annotation operation on the first visual data;

[0018] a second segmenting module configured to segment a second component from the second visual data;

[0019] a second determining module configured to determine a difference between a first pixel distribution of the first component in a color mode and a second pixel distribution of the second component in the color mode;

[0020] a second annotating module configured to annotate the second visual data based on the difference.

[0021] In a third aspect, an electronic device is provided. The electronic device includes:

[0022] a processor;

[0023] a memory configured to store executable instructions of the processor;

[0024] the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for annotating visual data according to any one of the preceding aspects.

[0025] In a fourth aspect, a computer-readable storage medium is provided, which stores computer instructions. The computer instructions are executed by a processor to implement the method for annotating visual data according to any one of the preceding aspects.

[0026] In a fifth aspect, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the method for annotating visual data according to any one of the preceding aspects.

[0027] It can be seen from the technical solution that in the embodiment of the present application, the unlabeled first visual data and the unlabeled second visual data are determined; the first component is determined from the segmentation result of the first visual data; the first visual data is labeled in response to the labeling operation of the first visual data; the second component is segmented from the second visual data; the difference between the first pixel distribution of the first component in the color mode and the second pixel distribution of the second component in the color mode is determined; and the second visual data is labeled based on the difference. As can be seen, the embodiment of the present application does not need to provide additional data (such as domain-specific knowledge or a three-dimensional model of an object), can perform automatic labeling on visual data to automatically generate labels, reduces the complexity and workload of the labeling process, and can also improve the quality of the trained artificial intelligence model. Moreover, the training samples for retraining the visual data labeling model can be conveniently combined, which is conducive to improving the accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0028] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, so that the above and other features and advantages of the present application can be more clearly understood by those skilled in the art.

[0029] FIG. 1 is an exemplary flowchart of a method for labeling visual data according to an embodiment of the present application.

[0030] FIG. 2 is an exemplary schematic diagram of labeling first visual data as reference data according to an embodiment of the present application.

[0031] FIG. 3 is an exemplary schematic diagram of a process for labeling visual data according to an embodiment of the present application.

[0032] FIG. 4 is a schematic diagram of generating training samples for retraining a visual data labeling model according to an embodiment of the present application.

[0033] FIG. 5 is an exemplary structural diagram of an apparatus for labeling visual data according to an embodiment of the present application.

[0034] FIG. 6 is an exemplary structural diagram of an electronic device according to an embodiment of the present application.

[0035] In the drawings, the reference signs are as follows: DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the following embodiments will further illustrate the present application in detail.

[0037] For clarity and conciseness in description, the solutions of the present application are described below by describing several representative embodiments. A large number of details in the embodiments are only used to help understand the solutions of the present application. However, it is obvious that the technical solutions of the present application can not be limited to these details. In order to avoid unnecessary obscurity of the solutions of the present application, some embodiments are not described in detail, but only a framework is given. In the following, "comprising" means "comprising but not limited to", and "according to" means "at least according to, but not limited to only according to". Due to the language habits of Chinese, when the quantity of a component is not specifically indicated in the following, it means that the component can be one or more, or can be understood as at least one.

[0038] Currently, manual annotation of data is needed to train artificial intelligence models. However, manual annotation of data has at least the following disadvantages:

[0039] (1) Low reliability: humans can intentionally or unintentionally make mistakes, especially in intensive and repetitive tasks. For visual use cases, annotation usually includes two types of label information: the first type is object classification; the second type is region boundary. Image classification models usually only use the first type of label information, segmentation models usually only use the second type of label information; object detection models usually use both types of label information. Since the human eye is difficult to distinguish the pixel-level difference of the boundary box, it is difficult to correct the wrong label information of the region boundary.

[0040] (2) Low efficiency: manual annotation is less efficient than running an algorithm that creates label annotations. Moreover, when people feel tired, the annotation efficiency further decreases over time.

[0041] Currently, some algorithms for automatically generating annotated images have appeared. The main process of the algorithm includes: importing a three-dimensional (3D) model of an object, and fully simulating environmental parameters such as lighting conditions and camera positioning, and using a software program to generate (that is, render) a synthetic image. Since there is a three-dimensional model of the object, the software program can generate accurate classes for image classification models or generate boundary boxes accurate to the pixel level for object detection models. However, in some scenarios, users want to use real images instead of synthetic images to train artificial intelligence models. Even, some artificial intelligence scientists are skeptical about the performance of models trained on synthetic data. Usually, they expect to train artificial intelligence models based on real image data, or use a hybrid mode to use synthetic images and real images together. In addition, when artificial intelligence scientists find that the performance of the model is not satisfactory on the production line, they will collect some image data from the production environment to retrain the model. In this case, real image data is usually used to retrain the model.

[0042] The embodiment of the present application provides a technical scheme for automatically labeling visual data without manual labeling and being significantly different from the above-mentioned synthetic data mode, and no matter whether the labeled data is used for an image classification model or a target detection model, etc., the visual data does not need to be labeled by using a model of an object. In the embodiment of the present application, without providing additional data (for example, field-specific knowledge or a 3D model of an object), automatic labeling is performed on real visual data to automatically generate labels, the complexity and workload of the labeling process are reduced, and the quality of the trained artificial intelligence model can be improved.

[0043] The above disclosure details the technical defects existing in the prior art, the causes leading to the technical defects, and the thinking and analysis process of overcoming the technical defects. Actually, the cognition of the above technical defects is not the common knowledge in the field, but the novel discovery of the applicant in the research. In addition, the cause tracing of the technical defects and the thinking and analysis process of overcoming the technical defects are also the gradual analysis results of the applicant in the actual research process, and are not the common knowledge in the field.

[0044] FIG. 1 is an exemplary flowchart of a method for labeling visual data according to an embodiment of the present application. As shown in FIG. 1, the method 100 includes:

[0045] Step 101: determining unlabeled first visual data and unlabeled second visual data.

[0046] In an embodiment: the unlabeled first visual data and the unlabeled second visual data are real visual data obtained by shooting, rather than virtual visual data (for example, synthetic images) rendered by using a three-dimensional model. For example, the first visual data and the second visual data can be any image frame in a series of visual data (for example, a video stream) obtained by shooting an object in time sequence. Depending on different specific application scenarios, the object can be a workpiece in an industrial site, a scanned object (for example, a patient) in a medical site, various monitoring targets (for example, the shooting object of a camera) in a daily life scene, etc. Moreover, depending on different shooting modes, the specific format of the first visual data and the second visual data can include: a static two-dimensional image (for example, an RGB image, a BMP image, etc.), a static three-dimensional image, a dynamic two-dimensional image (for example, a GIF image), a dynamic three-dimensional image, a video stream, a panoramic image, a laser radar point cloud, navigation information or a medical image, etc.

[0047] For example, in one embodiment, the first visual data is a reference image of a workpiece captured by an industrial camera, and the second visual data is any image in a real-time image stream of the workpiece captured by the industrial camera. For another example, in one embodiment, the first visual data is a reference X-ray image of a detection object obtained by performing X-ray photography on the detection object by an X-ray device, and the second visual data is any X-ray image in an X-ray image stream obtained by performing X-ray photography on the detection object by the X-ray device.

[0048] The unannotated first visual data is the reference data to be manually annotated. The first visual data can be selected by a user from a series of visual data (e.g., a video stream) captured in time sequence for the object, or randomly selected from the series of visual data (e.g., a video stream) based on a random algorithm. The unannotated second visual data is the visual data to be annotated. For example, the second visual data can be any visual data in the series of visual data different from the first visual data.

[0049] Step 102: determining the first component from the segmentation result of the first visual data.

[0050] Here, the segmentation result containing one or more components can be segmented from the first visual data in various ways. For example, the first visual data is input into an artificial intelligence model for segmenting components to obtain all components contained in the first visual data. Moreover, the first component that is helpful for annotating the first visual data is determined from the segmentation result of the first visual data (e.g., based on the category of the first component, the category of the first visual data can be determined).

[0051] For example, in one embodiment, the segmentation result (containing all components) of the first visual data is displayed in a graphical user interface. Then, based on the selection operation triggered by the user on the graphical user interface, a first component (usually the most prominent and important part, which is helpful for annotating the first visual data) is selected from the segmentation result. In the subsequent process, the first component can be compared with the components of other visual data to automatically annotate the other visual data.

[0052] For example, the AI model for segmenting the component can be implemented as a Segment Anything Model (SAM). The SAM is a deep learning-based image segmentation model. It is a promptable segmentation system that can generalize to unfamiliar objects and images without additional training, allowing it to "cut out" any object in any image. The SAM model allows users to instruct the model to perform specific tasks through simple text input. In addition, the SAM model is trained on a large data corpus containing millions of images and billions of masks, allowing it to "cut out" any object in any image without human annotation or manually specified masks. The network structure of the SAM model is similar to the encoder-decoder structure of U-Net, where the encoder part is composed of multiple convolutional layers and pooling layers to extract image features, and the decoder part is composed of multiple deconvolutional layers and up-sampling layers to restore the feature map to the original image size and generate a segmentation result.

[0053] The above exemplary describes a typical model for segmenting a component from first visual data, and those skilled in the art can appreciate that this description is only exemplary and does not limit the protection scope of the embodiments of the present application.

[0054] Step 103: In response to the labeling operation on the first visual data, label the first visual data.

[0055] For example, in response to the labeling operation on the first visual data received via the graphical user interface, label the first visual data.

[0056] For example: after all components are segmented based on the SAM model, the user displays all components on the graphical user interface, and then selects the first component from all components based on the selection operation triggered by the user. Moreover, the user manually labels the first visual data based on the category of the first component. For example, when the user observes that all components segmented based on the SAM model include a programmable logic controller (PLC) that can label the first visual data, the user selects the PLC as the first component from all components, and determines that the label of the first visual data is "PLC".

[0057] FIG. 2 is an exemplary schematic diagram of labeling the first visual data as reference data according to an embodiment of the present application.

[0058] In FIG. 2, after the first visual data 10 is input into the SAM model, the segmentation result 11 is displayed on the graphical user interface. It can be seen that the segmentation result 11 contains multiple components, including, specifically, a background region 13, a PLC 14, and a screen 15. The user observes the segmentation result 11 in the graphical user interface, confirms that the classification of the first visual data 10 is PLC, and thus selects the PLC 14 as the first component and performs manual labeling (e.g., manually inputs “PLC”) on the first visual data 10. The labeling result (i.e., the label) of the first visual data 10 is “PLC”, and the first component is the PLC 14.

[0059] Step 104: segmenting a second component from second visual data.

[0060] Here, the unlabeled second visual data is visual data that needs to be automatically labeled. For example, the second visual data can be any visual data in the same visual data series (e.g., the same visual data stream) as the first visual data, different from the first visual data. The second component can be any component in the segmentation result of the second visual data.

[0061] Similarly, the second component can be segmented from the second visual data in various ways. For example, the second visual data is input into an artificial intelligence model (e.g., a SAM model) for segmenting components to obtain the second component contained in the second visual data. The number of second components can be one or more. Preferably, the artificial intelligence model used to segment the first visual data in step 102 is the same as the artificial intelligence model used to segment the second visual data in step 104.

[0062] Step 105: determining the difference between the first pixel distribution of the first component in the color mode and the second pixel distribution of the second component in the color mode.

[0063] A color mode is an algorithm for representing colors in the digital world. In the digital world, in order to represent various colors, people usually divide colors into several components. Due to the difference in color forming principle, there is a difference in the way of generating colors between color devices such as displays, projectors, and scanners, which directly synthesize colors by using color light, and printing devices such as printers and printing machines, which rely on the use of pigments.

[0064] In one embodiment, the color mode includes RGB color mode, CMYK color mode, HSB color mode, Lab color mode, index color mode, bitmap color mode, and grayscale color mode, etc. The typical color models are exemplarily described below.

[0065] (1) RGB color mode

[0066] The RGB color mode is the most widely used color mode and is supported by many browsers and displays. The RGB color mode has three color components (or named channels), R (red), G (green), and B (blue). These three components specify an intensity value for each pixel between 0 (black) and 255 (white). For example: a bright red color has a value of 246 for the R component, 20 for the G component, and 50 for the B component. When all three components have the same value, the result is a neutral gray color. When all three components have a value of 255, the result is pure white; when all three components have a value of 0, the result is pure black. RGB images use three colors or three channels to reproduce colors on a screen.

[0067] (2) CMYK color mode

[0068] The CMYK color mode is mainly used for printing. This mode specifies a percentage value for each pixel for each type of ink. The ink color percentage specified for the brightest highlight colors is relatively low, while the percentage specified for darker shades is relatively high.

[0069] (3) Indexed color mode

[0070] The indexed color mode generates an 8-bit image file with a maximum of 256 colors. When converting an RGB mode image to indexed color, a color lookup table will be constructed to store and index the colors in the image. If a certain color in the original image does not appear in the table, the closest color will be selected or a simulated color will be used to simulate the existing color. Indexed color can reduce file size by limiting the palette while maintaining sufficient visual quality.

[0071] (4) Grayscale color mode

[0072] The grayscale color mode uses 256 levels of gray to represent an image. In this color mode, the brightness value of each pixel in the image is between 0 (black) and 255 (white). Although the grayscale color mode may not represent color-rich images as well as the RGB color mode, it can also be used in some cases due to the smaller size of grayscale images.

[0073] The above exemplary description provides a typical example of color modes, and those skilled in the art can realize that the description is only exemplary and is not intended to limit the protection scope of the embodiments of the present application.

[0074] Here, the pixel distribution refers to the statistics of the values of each pixel in the image. Preferably, the pixel distribution can be implemented as a pixel probability distribution. The first pixel distribution of the first component can be a single pixel distribution corresponding to a single channel, or multiple pixel distributions corresponding to multiple channels. Similarly, the second pixel distribution of the second component can be a single pixel distribution corresponding to a single channel, or multiple pixel distributions corresponding to multiple channels.

[0075] In one embodiment, the step 105 comprises: determining a pixel distribution of the first component in each channel of the color mode; determining a pixel distribution of the second component in each channel of the color mode; determining a difference between the pixel distribution of the first component in each channel and the pixel distribution of the corresponding channel of the second component; combining the multiple differences corresponding to the multiple channels of the color mode into a difference group. The channel includes at least one of: a color channel; a chroma channel; a hue channel; a saturation channel; a brightness channel; an achromatic channel, etc.

[0076] Taking the RGB color mode as an example, the color of each pixel can be converted into an RGB value. The RGB value is always in the range of 0 to 255. By first calculating the probability of each value between 0 and 255, and then calculating the RGB value respectively, the RGB distribution can be obtained. More specifically: first, determine the respective pixel distribution in the R channel, G channel and B channel of the second component. Then, determine the respective pixel distribution in the R channel, G channel and B channel of the first component. Next, determine the difference between the pixel distribution in the R channel of the second component and the pixel distribution in the R channel of the first component, the difference between the pixel distribution in the G channel of the second component and the pixel distribution in the G channel of the first component, and the difference between the pixel distribution in the B channel of the second component and the pixel distribution in the B channel of the first component. Then, combine the three differences into a difference group.

[0077] In one embodiment, the step 105 comprises: determining a dispersion between the first pixel distribution and the second pixel distribution; determining the difference based on the dispersion. For example, the dispersion can be directly determined as the difference, or a predetermined mathematical operation is performed on the dispersion, and the result of the mathematical operation is determined as the difference. In one embodiment, the dispersion includes: KL divergence; JS divergence; dispersion based on an entropy function; dispersion determined based on a cross-entropy loss function; dispersion based on Kolmogorov-Smirnov test, etc.

[0078] In one embodiment, an example of calculating JS divergence (JS Divergence) is as follows:

[0079] KL Divergence:

[0080] JS Divergence:

[0081] wherein: P is the second component; Q is the first component, P(X) is the probability of each pixel in the second component corresponding to each pixel value x on the horizontal axis; Q(X) is the probability of each pixel in the first component corresponding to each pixel value x in the horizontal axis; and KL Divergence is the KL divergence. From the properties of JS divergence, the smaller the JS divergence, the smaller the difference between the two distributions.

[0082] Step 106: based on the difference value, labeling the second visual data.

[0083] Here, when the difference value is small enough (such as compared with a predetermined threshold to determine whether it is small enough), it can be determined that the second component is similar enough to the first component, and it can be determined that the second component contained in the second visual data and the first component contained in the first visual data belong to the same classification in essence, and therefore the labeling result of the second visual data can be automatically determined based on the labeling result of the first visual data (i.e. the classification label of the second visual data is the same as the classification label of the first visual data). When the difference value is not small enough, it can be determined that the second component is not similar enough to the first component, and it can be determined that the second component contained in the second visual data and the first component contained in the first visual data belong to different classifications in essence, and therefore the labeling result of the second visual data cannot be automatically determined based on the labeling result of the first visual data, and the second visual data can be determined as unlabeled data.

[0084] In an embodiment, when the number of second components is multiple, as long as the difference value of one second component and the first component is small enough, the labeling result of the second visual data can be automatically determined based on the labeling result of the first visual data (i.e. the classification label of the second visual data is the same as the classification label of the first visual data).

[0085] In one embodiment, step 106 comprises: based on a comparison result of at least one difference value in the difference set and a predetermined first threshold, labeling the second visual data; wherein: when the at least one difference value is greater than or equal to the first threshold, determining that the second visual data is unlabeled; or when the at least one difference value is less than the first threshold, labeling the second visual data based on the labeling result of the first visual data, which means that the labeling result of the first visual data is determined as the labeling result of the second visual data. For example, the value of the first threshold ranges from 0 to 1. Preferably, the first threshold is adjustable. Here, the second visual data is directly labeled based on the at least one difference value extracted from the difference set.

[0086] In one embodiment, the method further comprises: comparing the similarity of the shape of the first component and the shape of the second component; and when the similarity is less than a predetermined second threshold, determining that the second visual data is unlabeled. For example, when it is determined based on the similarity of the shape that the first component is not similar to any second component, the second visual data is directly determined to be unlabeled, thereby saving the computational complexity of calculating the pixel distribution.

[0087] In one embodiment, step 106 comprises: based on at least one difference value in the difference set, determining at least one transformation value; based on a comparison result between the at least one transformation value and a third threshold, labeling the second visual data; wherein: when the at least one transformation value is greater than or equal to the third threshold, determining that the second visual data is unlabeled; or when the at least one transformation value is less than the third threshold, labeling the second visual data based on the labeling result of the first visual data, which means that the labeling result of the first visual data is determined as the labeling result of the second visual data. The transformation value can be determined in various ways according to the difference set. For example, the average of all difference values in the difference set can be determined as the transformation value. Alternatively, the average of part of the difference values in the difference set can be determined as the transformation value. Here, labeling the second visual data based on the comparison result of the transformation value determined based on the difference set can provide more options.

[0088] In some specific cases, the second visual data cannot be accurately labeled based on the difference value or the transformation value. For example, the pixel distribution of the first component and the second component can be the same, but the patterns of the two can be completely different, so based on the above method, the second visual data can be incorrectly labeled.

[0089] In one embodiment, after labeling the second visual data based on the difference value or the transformed value (i.e., determining the labeling result of the first visual data as the labeling result of the second visual data), it is further evaluated whether the labeling result of the second visual data is correct based on a comparison of the pixel distribution of the sub-block image extracted from the second component part with the pixel distribution in the corresponding sub-block image extracted from the first component part. For example, the extracted sub-block image can have any ratio of 1 / 16, 1 / 8, 1 / 4, 1 / 2, etc. of the original image.

[0090] In one embodiment, the method comprises: extracting a first sub-block image from the first component part; extracting a second sub-block image corresponding to the first sub-block image from the second component part; determining the pixel distribution of the first sub-block image in the color mode; determining the pixel distribution of the second sub-block image in the color mode; determining the difference between the pixel distribution of the first sub-block image in the color mode and the pixel distribution of the second sub-block image in the color mode (named as a second difference value). After the labeling of the second visual data based on the labeling result of the first visual data according to the difference between the first pixel distribution of the first component part in the color mode and the second pixel distribution of the second component part in the color mode: when the second difference value is greater than or equal to a predetermined fourth threshold value, the second visual data is updated as unlabeled; or when the second difference value is less than the fourth threshold value, the labeling result of the second visual data is kept. Therefore, after the labeling of the second visual data based on the labeling result of the first visual data when the difference between the first pixel distribution and the second pixel distribution is less than the first threshold value, if the second visual data is determined as unlabeled based on the second difference value, the second visual data is finally determined as unlabeled.

[0091] In one embodiment, it comprises: combining the second visual data and the labeling result of the second visual data as a training sample for training an image classification model. In another embodiment, a bounding box of the second component part is generated; the second visual data, the bounding box and the labeling result of the second visual data are combined as a training sample for training a target detection model. It can be seen that the embodiments of the present application can automatically generate multiple types of training samples for training an image classification model or a target detection model.

[0092] In one embodiment, comprising: obtaining unannotated third visual data; inputting the third visual data into the trained visual data annotation model to output a first annotation result of the third visual data by the visual data annotation model; segmenting a third component from the third visual data; determining a difference between a first pixel distribution of the first component in the color mode and a third pixel distribution of the third component in the color mode; determining a second annotation result of the third visual data based on the difference between the first pixel distribution of the first component in the color mode and the third pixel distribution of the third component in the color mode; when the first annotation result is different from the second annotation result, in response to a trigger operation, determining a third annotation result of the third visual data; and combining the third visual data and the third annotation result as a training sample for retraining the visual data annotation model. Therefore, the training sample for retraining the visual data annotation model provided by the embodiment of the present application embodies the error-prone characteristics of the visual data annotation model, which is particularly beneficial for retraining the visual data annotation model.

[0093] In one embodiment, in response to the trigger operation, determining the third annotation result of the third visual data comprises: in response to a selection operation on the first annotation result and the second annotation result, selecting one of the first annotation result and the second annotation result as the third annotation result; or, in response to an annotation operation on the third component, annotating the third annotation result. Therefore, the correct third annotation result can be selected from the first annotation result and the second annotation result. Moreover, when neither the first annotation result nor the second annotation result is ideal, the correct third annotation result can be manually annotated, which is beneficial for retraining the visual data annotation model.

[0094] FIG. 3 is an exemplary schematic diagram of a process of annotating visual data according to an embodiment of the present application. The process of annotating visual data comprises:

[0095] Step 1: Perform preparation of the dataset. A set of unannotated visual data is collected and stored in the unannotated image database 27 of the storage 20. The collection method can be various. For example, obtained from other databases, real-time shooting according to the camera, imported from third-party software providing real images or synthetic images, etc. The processor 21 can extract basic information about the dataset and initialize the embedding metadata of each image (such as tracking the size of the dataset and whether the image is annotated, etc.).

[0096] Step 2: The user provides a reference image

[0097] Once the image dataset is received, the user can start the automatic annotation of the data. First, the backend service 22 executed by the processor 22 selects a reference image 10 from the unannotated image database 27 of the memory 20 (e.g., randomly selects an unannotated image) and performs a decision process 28 on the reference image. In the decision process 28, it is decided whether the reference image 10 is an unannotated image. If it is an unannotated image (corresponding to the “Y” branch), it is sent to the segmentation model 29, otherwise (corresponding to the “N” branch), it is sent to the annotated image database 31. In the segmentation model 29 (e.g., a SAM module), all the components in the reference image 10 are segmented. The reference image 10 and the segmentation result 11 (containing all the segmented components) of the reference image 10 are displayed in the first interface 25 of the graphical user interface 23. Based on the selection operation of the user 24 on the graphical user interface 23, the user 24 selects a component from the segmentation result 11 as a reference region. After selecting the reference region, the user 24 also inputs the name / category of the reference region (i.e., labels the reference image 10) in the graphical user interface 23 so that the backend service 22 can identify similar regions in other images.

[0098] Step 3: Label auto-generation

[0099] Then, the backend service 22 executed by the processor 22 obtains an unannotated image from the unannotated image database 27 of the memory 20 that is different from the reference image 10 and performs the decision process 28 on the unannotated image. In the decision process 28, it is decided whether the unannotated image is an unannotated image. If it is an unannotated image (corresponding to the “Y” branch), it is sent to the segmentation model 29, otherwise (corresponding to the “N” branch), it is sent to the annotated image database 31. The segmentation model 29 performs segmentation on the unannotated image and compares the pixel distribution of each component in the unannotated image with the pixel distribution of the reference region using the processing process 30 to find the component similar to the reference region in the reference image. Optionally, the user 24 can also choose whether to check whether the area size and the area shape between the reference region and the component of the unannotated image are the same, which allows the user 24 to freely choose whether to expect different object proportions and tilt angles when taking photos.

[0100] For similar components in the unlabeled images, the processing procedure 30 can automatically generate labels for the images. If the user expects to train an image classification model with the automatically labeled visual data, the processing procedure 30 will label the unlabeled images containing similar pixel distribution of components as the same class as the reference image. On the other hand, if the user expects to train an object detection model with the automatically labeled visual data, the processing procedure 30 will draw bounding boxes in the reference regions of the reference image and draw bounding boxes on the components in the unlabeled images with similar pixel distribution as the reference regions, and label the unlabeled images as the same class as the reference image. Once the images are labeled, they will be labeled as labeled in the embedding metadata and will no longer be segmented and relabeled to reduce runtime costs. The labeled visual data is stored into the labeled image database 31.

[0101] Step 4: Verify and review the dataset

[0102] Here, the user 24 can view the labeling results 261 for image classification and the labeling results 262 for object detection on the second interface 26 of the graphical user interface 23 and decide whether to relabel. In addition, unlabeled visual data can be presented on the second interface 26 for the user 24 to decide whether to manually label the images or delete the images.

[0103] In the output procedure 32, the labeled images 33 or the labeling files 34, etc. can be output from the labeled image database 31. Moreover, the labeled images 33 and the labeling files 34 can be output to an external database 35, etc.

[0104] In some cases, the user has already deployed a trained artificial intelligence model and wants to improve the model performance while the model is working. However, the user needs to prepare labeled images for retraining. The embodiments of the present application can also help the user to accelerate the labeling process and the retraining process.

[0105] FIG. 4 is a schematic diagram of generating training samples for a retraining visual data labeling model according to an embodiment of the present application.

[0106] In FIG. 4, the unlabeled visual data 41, 42…4N is input into the trained visual data labeling model 50 to perform automatic labeling on the unlabeled visual data 41, 42…4N by the trained visual data labeling model, to obtain the corresponding labeling results 51, 52…5N. Among them: the labeling result 51 is the labeling result of the visual data 41; the labeling result 52 is the labeling result of the visual data 42. By analogy, the labeling result 5N is the labeling result of the visual data 4N.

[0107] Further, based on the method 100 of labeling visual data shown in FIG. 1, the unlabeled visual data 41, 42…4N are respectively executed automatic labeling, and the corresponding labeling results 61, 62…6N are obtained. Among them: the labeling result 61 is the labeling result of the visual data 41; the labeling result 62 is the labeling result of the visual data 42. Similarly, the labeling result 6N is the labeling result of the visual data 4N.

[0108] In the comparison process 60, each pair of labeling results in the labeling results 51, 52…5N and the labeling results 61, 62…6N are compared. For example, whether the labeling result 51 is the same as the labeling result 61 is compared; whether the labeling result 52 is the same as the labeling result 62 is compared; similarly, whether the labeling result 5N is the same as the labeling result 6N is compared. Among them: when the labeling results of any pair are different, a labeling result is selected from the labeling results of this pair as the correct labeling result based on user triggering, or the correct labeling result is manually labeled by the user. For example, when the labeling result 51 is different from the labeling result 61, the user can specify one correct labeling result from the labeling result 51 and the labeling result 61, or the user inputs another correct labeling result.

[0109] In the combination process 70: the correct labeling result specified by the user or the correct labeling result input by the user, and the corresponding visual data, are combined into training samples for retraining the visual data labeling model 50. For example, when the labeling result 51 is different from the labeling result 61, the user specifies the labeling result 51 as the correct labeling result, and then the unlabeled visual data 41 and the labeling result 51 (as the label of the unlabeled visual data 41) are combined into training samples for retraining the visual data labeling model 50. For another example, when the labeling result 52 is different from the labeling result 62, the user is not satisfied with the two labeling results, and inputs a labeling result 76 by himself, and then the unlabeled visual data 42 and the labeling result 76 (as the label of the unlabeled visual data 42) are combined into training samples for retraining the visual data labeling model 50.

[0110] FIG. 5 is an exemplary structural diagram of an apparatus for labeling visual data according to an embodiment of the present application. As shown in FIG. 5, the apparatus 500 for labeling visual data comprises: a first determining module 501 configured to determine un-labeled first visual data and un-labeled second visual data; a first segmenting module 502 configured to determine a first component from a segmentation result of the first visual data; a first labeling module 503 configured to label the first visual data in response to a labeling operation on the first component; a second segmenting module 504 configured to segment a second component from the second visual data; a second determining module 505 configured to determine a difference between a first pixel distribution of the first component in a color mode and a second pixel distribution of the second component in the color mode; and a second labeling module 506 configured to label the second visual data based on the difference.

[0111] In one embodiment, the second determining module 505 is configured to determine a pixel distribution of the first component in each channel of the color mode; determine a pixel distribution of the second component in each channel of the color mode; determine a difference between the pixel distribution of the first component in each channel and the pixel distribution of the second component in the corresponding channel; and combine a plurality of differences corresponding to a plurality of channels of the color mode into a difference group.

[0112] In one embodiment, the second determining module 505 is configured to determine a dispersion between the first pixel distribution and the second pixel distribution; and determine the difference based on the dispersion.

[0113] In one embodiment, the second labeling module 506 is configured to label the second visual data based on a comparison result of at least one difference in the difference group and a predetermined first threshold; and wherein: when the at least one difference is greater than or equal to the first threshold, determine that the second visual data is un-labeled; or when the at least one difference is less than the first threshold, label the second visual data based on a labeling result of the first visual data.

[0114] In one embodiment, the second labeling module 506 is configured to combine the second visual data and a labeling result of the second visual data into a training sample for training an image classification model; or generate a bounding box of the second component; and combine the second visual data, the bounding box and the labeling result of the second visual data into a training sample for training a target detection model.

[0115] In an embodiment, the sample generation module 507 is configured to obtain unannotated third visual data; input the third visual data into the trained visual data annotation model to output a first annotation result of the third visual data by the visual data annotation model; segment a third component from the third visual data; determine a difference between a first pixel distribution of the first component in the color mode and a third pixel distribution of the third component in the color mode; determine a second annotation result of the third visual data based on the difference between the first pixel distribution of the first component in the color mode and the third pixel distribution of the third component in the color mode; when the first annotation result is different from the second annotation result, determine a third annotation result of the third visual data in response to a trigger operation; and combine the third visual data and the third annotation result as a training sample for retraining the visual data annotation model.

[0116] In an embodiment, determining the third annotation result of the third visual data in response to the trigger operation includes: selecting one of the first annotation result and the second annotation result as the third annotation result in response to a selection operation on the first annotation result and the second annotation result; and annotating the third annotation result in response to an annotation operation on the third component.

[0117] In an embodiment, the second segmentation module 504 is configured to compare a similarity between a shape of the first component and a shape of the second component; and determine that the second visual data is unannotated when the similarity is less than a second preset threshold.

[0118] In an embodiment, the first visual data is a reference image of a workpiece captured by an industrial camera, and the second visual data is a real-time image of the workpiece captured by the industrial camera.

[0119] An electronic device having a processor-memory architecture is also provided in an embodiment of the present application. FIG. 6 is an exemplary block diagram of an electronic device having a processor-memory architecture according to an embodiment of the present application. As shown in FIG. 6, the electronic device 600 includes a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601, which, when executed by the processor 601, implements any of the above methods of annotating visual data. The memory 602 can be embodied as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a programmable program read-only memory (PROM), or the like. The processor 601 can be embodied as one or more central processing units or one or more field programmable gate arrays, which integrate one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core can be embodied as a CPU or a MCU or a DSP, or the like.

[0120] It should be noted that not all steps and modules in the above processes and structural diagrams are necessary, and some steps or modules can be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only a functional division adopted for ease of description. In actual implementation, one module can be implemented by multiple modules, and the functions of multiple modules can be implemented by the same module. These modules can be located in the same device or in different devices.

[0121] The hardware module in each embodiment can be implemented mechanically or electronically. For example, a hardware module can include a dedicated, permanently configured, and permanent circuit or logic device, such as a special-purpose processor, for example, an FPGA or an ASIC, to perform specific operations. The hardware module can also include a programmable logic device or circuit, such as a general-purpose processor or other programmable processor, which is temporarily configured by software to perform specific operations. As to whether to implement the hardware module mechanically or by a dedicated, permanent circuit or by a temporarily configured circuit (such as by software), it can be determined according to cost and time considerations.

[0122] The above description is only a preferred embodiment of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for labeling visual data, characterized in that, include: Identify (101) the unlabeled first visual data and the unlabeled second visual data; From the segmentation results of the first visual data, determine (102) the first component; In response to the annotation operation on the first visual data, the first visual data is annotated (103); The second component is segmented (104) from the second visual data; Determine (105) the difference between the first pixel distribution of the first component in the color mode and the second pixel distribution of the second component in the color mode; Based on the difference, the second visual data is labeled (106).

2. The method according to claim 1, characterized in that, The determination (105) of the difference between the first pixel distribution of the first component in the color mode and the second pixel distribution of the second component in the color mode includes: Determine the pixel distribution of the first component in each channel of the color mode; Determine the pixel distribution of the second component in each channel of the color mode; Determine the difference between the pixel distribution of the first component in each channel and the pixel distribution of the second component in the corresponding channel; Multiple differences corresponding to multiple channels of the color mode are combined into a difference group.

3. The method according to claim 1, characterized in that, Determining the difference between the first pixel distribution of the first component in the color mode and the second pixel distribution of the second component in the color mode (105) includes: Determine the dispersion between the first pixel distribution and the second pixel distribution; The difference is determined based on the dispersion.

4. The method according to claim 2, characterized in that, The annotation (106) of the second visual data based on the difference includes: The second visual data is labeled based on the comparison result of at least one difference in the difference group with a predetermined first threshold; in: When the at least one difference is greater than or equal to the first threshold, the second visual data is determined to be unlabeled; or When the at least one difference is less than the first threshold, the second visual data is labeled based on the labeling result of the first visual data.

5. The method according to claim 1, characterized in that, include: The second visual data and the annotation results of the second visual data are combined into training samples for training the image classification model. or Generate the bounding box of the second component; combine the second visual data, the bounding box, and the annotation result of the second visual data into training samples for training the object detection model.

6. The method according to any one of claims 1-5, characterized in that, include: Obtain unlabeled third-person visual data; The third visual data is input into a trained visual data annotation model, so that the visual data annotation model outputs a first annotation result for the third visual data; The third component is segmented from the third visual data; Determine the difference between the first pixel distribution of the first component in the color mode and the third pixel distribution of the third component in the color mode; The second annotation result of the third visual data is determined based on the difference between the first pixel distribution of the first component in the color mode and the third pixel distribution of the third component in the color mode. When the first annotation result is different from the second annotation result, a trigger operation is initiated to determine the third annotation result of the third visual data; The third visual data and the third annotation result are combined to form training samples for retraining the visual data annotation model.

7. The method according to claim 6, characterized in that, The step of determining the third annotation result of the third visual data in response to the triggering operation includes: In response to a selection operation between the first annotation result and the second annotation result, one of the first annotation result and the second annotation result is selected as the third annotation result; or In response to the annotation operation for the third component, the third annotation result is annotated.

8. The method according to any one of claims 1-5, characterized in that, include: Compare the similarity between the shape of the first component and the shape of the second component; When the similarity is less than a preset second threshold, the second visual data is determined to be unlabeled.

9. The method according to any one of claims 1-5, characterized in that, The first visual data is a reference image of the workpiece taken using an industrial camera, and the second visual data is a real-time image of the workpiece taken using the industrial camera.

10. A device for labeling visual data, characterized in that, include: The first determining module (501) is used to determine the unlabeled first visual data and the unlabeled second visual data; The first segmentation module (502) is used to determine a first component from the segmentation result of the first visual data; The first annotation module (503) is used to annotate the first visual data in response to the annotation operation on the first visual data; The second segmentation module (504) is used to segment a second component from the second visual data; The second determining module (505) is used to determine the difference between the first pixel distribution of the first component in the color mode and the second pixel distribution of the second component in the color mode; The second annotation module (506) is used to annotate the second visual data based on the difference.

11. An electronic device, characterized in that, include: Processor (601); Memory (602) for storing executable instructions of the processor (601); The processor (601) is configured to read the executable instructions from the memory (602) and execute the executable instructions to implement the method of annotating visual data according to any one of claims 1-9.

12. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, they implement the method for annotating visual data as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method for annotating visual data as described in any one of claims 1-9.

Images