Image annotation method, device and electronic equipment

Abstract

The present application relates to the field of deep learning technology, and provides an image annotation method, device and electronic device, the method comprising: obtaining aligned color images and depth images; using a preset trained target detection model to detect the color image, and obtaining label information and key point coordinate information corresponding to the color image, wherein the label information is the category information of the target; generating an annotation file corresponding to the depth image based on the alignment relationship according to the label information and key point coordinate information corresponding to the color image; converting the depth image into a pseudo-color image, and correcting the annotation file corresponding to the depth image based on the pseudo-color image of the depth image, to obtain a corrected annotation data set. This embodiment improves the efficiency of image annotation.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an image annotation method, device and electronic device. Background Art

[0002] In deep learning, a large number of images with image annotation information are needed as data sets for deep learning and training, but data annotation is a very tedious and time-consuming task.

[0003] Currently, most research institutions or companies purchase existing datasets or use existing public datasets to obtain datasets, but this approach is not conducive to the research of specific scenarios or specific target detection. In addition, they also outsource data annotation to annotation companies to obtain datasets, which makes the cost of deep learning research very high.

[0004] At present, data annotation is mostly done manually, and annotators need to perform a large number of repetitive judgments and operations to annotate images. This inevitably requires a lot of manpower and time costs, reducing the efficiency of data set optimization and model building. Summary of the invention

[0005] In view of this, the embodiments of the present application provide an image annotation method, device, and electronic device, which can solve one or more technical problems in the related art.

[0006] In a first aspect, an embodiment of the present application provides an image annotation method, comprising:

[0007] Get aligned color image and depth image;

[0008] Detecting the color image using a preset trained target detection model to obtain label information and key point coordinate information corresponding to the color image, wherein the label information is the category information of the target;

[0009] Generate a annotation file corresponding to the depth image based on the label information and key point coordinate information corresponding to the color image and the alignment relationship;

[0010] The depth image is converted into a pseudo-color image, and based on the pseudo-color image of the depth image, the annotation file corresponding to the depth image in the same group is corrected to obtain a corrected annotation data set.

[0011] In this embodiment, on the one hand, a trained public model is used to perform reasoning detection on color images, and a labeling file is generated according to the detection results, which reduces manual labeling and greatly improves the labeling efficiency of the data set. On the other hand, the depth image is converted into a pseudo-color image to facilitate users to recheck the labeling file, improve the rechecking efficiency, and also improve the accuracy of the labeling, and obtain a data set with higher confidence. On the other hand, the labeling file is applied to the depth image, and a labeling method for quickly labeling the depth image is realized, and then training and learning are carried out, which can quickly carry out research on the depth image and promote the development of 3D related technologies.

[0012] As an implementation of the first aspect, converting the depth image into a pseudo-color image includes:

[0013] Acquire a chromaticity diagram, wherein the chromaticity diagram includes a mapping relationship between color values ​​and pixel values;

[0014] Normalize the depth image to obtain a normalized image corresponding to the depth image;

[0015] Map the normalized image into a pseudo-color image according to the chromaticity diagram.

[0016] As an implementation of the first aspect, acquiring the aligned color image and depth image includes:

[0017] The pixels of the color image and the pixels of the depth image are aligned one by one.

[0018] As an implementation of the first aspect, generating a label file corresponding to the depth image based on the label information and key point coordinate information corresponding to the color image and the alignment relationship includes:

[0019] Based on the alignment relationship between the color image and the depth image, the label information and key point coordinate information corresponding to the color image are converted into annotation information of the depth image, and written into an annotation file in a preset format to obtain an annotation file corresponding to the depth image.

[0020] As an implementation manner of the first aspect, the annotation file also includes image information in the depth image, wherein the image information includes the length, width, channel, path and image name of the image.

[0021] As an implementation of the first aspect, the revised annotated data set includes: using the pseudo-color image to calibrate the label information and the coordinate information included in the annotation file one by one to obtain the revised annotated data set.

[0022] As an implementation of the first aspect, the using the pseudo-color image to perform one-to-one proofreading on the label information and the coordinate information included in the annotation file includes:

[0023] The annotation file is corrected by using an annotation tool to obtain a corrected annotation data set, wherein the annotation tool corrects the annotation file by determining whether label information and coordinate information included in the annotation file displayed on the pseudo-color image are correct.

[0024] In a second aspect, an embodiment of the present application provides an image annotation device, including:

[0025] An acquisition module, used for acquiring color images and depth images;

[0026] An alignment module, configured to align the color image with the depth image;

[0027] A detection module, used to detect the color image using a preset trained target detection model to obtain label information and key point coordinate information corresponding to the color image, wherein the label information is the category information of the target;

[0028] A file generation module, configured to generate a labeling file corresponding to the depth image according to the label information and key point coordinate information corresponding to the color image and based on the alignment relationship;

[0029] A conversion module, used for converting the depth image into a pseudo-color image;

[0030] The annotation module is used to correct the annotation file corresponding to the depth image based on the pseudo-color image of the depth image to obtain a corrected annotation data set.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the image annotation method as described in the first aspect or any implementation manner of the first aspect are implemented.

[0032] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image annotation method described in the first aspect or any implementation manner of the first aspect are implemented.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps of the image annotation method described in the first aspect or any implementation of the first aspect.

[0034] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0037] Figure 2 This is a schematic diagram of an implementation flow of an image annotation method provided by an embodiment of the present application;

[0038] Figure 3 This is a schematic diagram of a specific implementation process of step S140 in an image annotation method provided in an embodiment of the present application;

[0039] Figure 4 This is a schematic diagram of the implementation process of another image annotation method provided by an embodiment of the present application;

[0040] Figure 5 is a structural schematic diagram of an image annotation device provided by an embodiment of the present application;

[0041] Figure 6 It is a structural schematic diagram of another image annotation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0043] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0044] "One embodiment" or "some embodiments" described in the specification of this application means that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the sentences "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0045] Furthermore, in the description of the present application, “plurality” means two or more.

[0046] It should also be understood that, unless otherwise clearly specified or limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0047] The current manual data annotation method is time-consuming, labor-intensive, costly, and inefficient. In addition, most data annotation methods on the market are suitable for color images, and rarely consider data annotation for depth images.

[0048] Therefore, an embodiment of the present application provides an image annotation method that can realize rapid annotation of images, and further can realize rapid annotation of depth images to obtain a depth image dataset with image annotation information, thereby facilitating research on depth images and promoting the development of three-dimensional (3D) related technologies.

[0049] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0050] Figure 1 The schematic diagram of the structure of an electronic device provided in one embodiment of the present application. The electronic device includes but is not limited to computers, tablets, laptops, netbooks and other electronic devices, and the present embodiment of the application does not impose any restrictions on the specific type of electronic device.

[0051] In some embodiments of the present application, Figure 1 As shown, the electronic device may include one or more processors 10 ( Figure 1Only one is shown in the figure), a memory 11 and a computer program 12 stored in the memory 11 and executable on one or more processors 10, for example, a program for image annotation. When one or more processors 10 execute the computer program 12, the various steps in the image annotation method embodiment described later can be implemented. Alternatively, when one or more processors 10 execute the computer program 12, the functions of various modules / units in various image annotation device embodiments described later can be implemented.

[0052] Exemplarily, the computer program 12 may be divided into one or more modules / units, one or more modules / units are stored in the memory 11 and executed by the processor 10 to complete the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 12 in the processing unit. For example, the computer program 12 may be divided into the following modules. The specific functions of each module are as follows:

[0053] An acquisition module, used for acquiring a color image and a depth image;

[0054] Alignment module, used to align color image and depth image;

[0055] A detection module, used to detect the color image using a preset trained target detection model to obtain label information and key point coordinate information corresponding to the color image;

[0056] A file generation module, configured to generate a labeling file corresponding to the depth image according to the label information and key point coordinate information corresponding to the color image and based on the alignment relationship;

[0057] A conversion module, used for converting the depth image into a pseudo-color image;

[0058] The annotation module is used to correct the annotation files corresponding to the depth images in the same group based on the pseudo-color images of the depth images to obtain a corrected annotation data set.

[0059] Those skilled in the art will understand that Figure 1 The electronic device is merely an example of an electronic device and does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0060] The processor 10 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0061] The memory 11 may be an internal storage unit of the processing unit, such as a hard disk or memory of the processing unit. The memory 11 may also be an external storage device of the processing unit, such as a plug-in hard disk equipped on the processing unit, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 may also include both an internal storage unit of the processing unit and an external storage device. The memory 11 is used to store computer programs and other programs and data required by the processing unit. The memory 11 may also be used to temporarily store data that has been output or is to be output.

[0062] An embodiment of the present application further provides another preferred embodiment of an electronic device. In this embodiment, the electronic device includes one or more processors, and the one or more processors are used to execute the following program modules stored in the memory:

[0063] An acquisition module, used for acquiring a color image and a depth image;

[0064] Alignment module, used to align color image and depth image;

[0065] A detection module, used to detect the color image using a preset trained target detection model to obtain label information and key point coordinate information corresponding to the color image;

[0066] A file generation module, configured to generate a labeling file corresponding to the depth image according to the label information and key point coordinate information corresponding to the color image and based on the alignment relationship;

[0067] A conversion module, used for converting the depth image into a pseudo-color image;

[0068] The annotation module is used to correct the annotation files corresponding to the depth images in the same group based on the pseudo-color images of the depth images to obtain a corrected annotation data set.

[0069] Figure 2 1 is a schematic diagram of an implementation flow of an image annotation method provided in an embodiment of the present application. The image annotation method in this embodiment is applicable to situations where depth images need to be annotated. The image annotation method in this embodiment can be executed by an electronic device. As an example and not a limitation, the image annotation method can be executed by Figure 1 The electronic device in the illustrated embodiment is implemented.

[0070] like Figure 2 As shown, the image annotation method may include: step S110 to step S150.

[0071] S110, acquiring aligned color image and depth image.

[0072] The color image and the depth image are images of the same scene at the same shooting angle. For the same scene, at least one set of color images and depth images at the same shooting angle can be shot. By changing the shooting scene, a large amount of image data can be obtained. Some of the images in different groups include the target object (such as a human body), while others do not.

[0073] In some implementations, the color image and the depth image are captured by a camera, which may be the same camera or different cameras.

[0074] As a non-limiting example, the electronic device activates a camera and captures multiple sets of color images and depth images of multiple scenes through the camera.

[0075] Furthermore, each group of color images and depth images is aligned to obtain an aligned depth image or an aligned color image.

[0076] In some implementations, the depth image may be aligned with each pixel of the color image based on the color image of the same group to obtain an aligned depth image.

[0077] In some embodiments of the present application, before obtaining the annotation file corresponding to the depth image, it is necessary to align each corresponding pixel of the acquired depth image and color image. More specifically, a color image of a scene can be obtained by a color camera, and a depth image of the same scene can be obtained by another depth camera. Therefore, the color image and the depth image have different viewing angles. In order to improve the accuracy of manual re-inspection and thus improve the confidence of the data set, in these embodiments, the depth image and the color image can be further aligned to obtain the transformation matrix of the corresponding pixels, and then the aligned depth image and color image are obtained. In these embodiments, preferably, the color camera and the depth camera are arranged adjacent to each other. The closer the positions of the two are, the closer the fields of view captured by the receiving ends or imaging ends of the two are. In this way, the color image and the depth image have more corresponding pixels during the alignment process, thereby further increasing the accuracy of the data set.

[0078] In some other embodiments of the present application, the color image and the depth image are captured by different acquisition modules of the same camera, and the color image and the depth image can also be aligned, which is not limited here.

[0079] S120, using a preset trained target detection model to detect the color image, and obtain label information (label) and key point coordinate information corresponding to the color image; wherein the label information is the category information of the target.

[0080] Among them, the preset trained target detection model can be a public model, which is a model obtained by deep learning training based on a public color image data set. The preset trained human body detection model is deployed in the memory of the electronic device in advance and can be called when used. It should be noted that the preset trained target detection model may include but is not limited to yolov5, faster RCNN, SSD, yolov4 and other models, and the embodiment of the present application does not limit the target detection model.

[0081] As a non-limiting example, the target detection model is a human body detection model, the human body detection model can be a two-classification model, and the label information can include a human body (e.g., "person") or a non-human body. It should be understood that if the label information corresponding to a color image is a human body, the key point coordinate information corresponding to the color image is specific joint point coordinate information; if the label information corresponding to a color image is a non-human body, the key point coordinate information corresponding to the color image is empty or does not exist.

[0082] As another non-limiting example, the target detection model is a model with more than three categories, and the label information may include human body, animal, object, etc. It should be understood that if the label information corresponding to a color image is human body, the key point coordinate information corresponding to the color image is specific joint point coordinate information; if the label information corresponding to a color image is animal or object, etc., the key point coordinate information corresponding to the color image is empty or does not exist.

[0083] It should be noted that in models with more than three categories, the category information of the target can be further subdivided, and the embodiments of the present application are not limited to this.

[0084] It should also be noted that, for the convenience of describing this embodiment, the human body is used as an example of a candidate object, that is, a target. It should be understood that in other embodiments, the target may include an object or an animal.

[0085] More generally, the object detection model is used to detect the color image to obtain the label information and key point coordinate information corresponding to the color image. In other words, the object detection model obtains the label information on the one hand and the coordinate information of the box on the other hand.

[0086] S130, generating a labeling file corresponding to the depth image according to the label information and key point coordinate information corresponding to the color image and based on the alignment relationship.

[0087] In one embodiment, when a depth image captured by a camera is obtained, the depth image may carry image information, and the image information includes but is not limited to a combination of one or more of the length, width, channel, path, and image name of the image. According to the label information and key point coordinate information corresponding to the color image obtained in step S120, the annotation information of the depth image is generated based on the alignment relationship between the depth image and the color image, and combined with the image information of the depth image, a annotation file corresponding to the depth image is generated.

[0088] In some embodiments, label information, key point coordinate information and other corresponding image information corresponding to the depth image are written into a preset format annotation file to obtain an annotation file.

[0089] In some implementations, the preset format annotation file may include an XML annotation file of VOC, a TXT annotation file of YOLO, or a JSON annotation file of COCO. The present application embodiment does not specifically limit the format of the annotation file.

[0090] S140, converting the depth image into a pseudo-color image.

[0091] For the original depth image, it is difficult for the human eye to identify the target and perceive the change of depth. Therefore, the depth image is converted into a pseudo-color image to achieve better visualization effect and facilitate subsequent manual re-inspection.

[0092] In some embodiments, Figure 3 As shown, converting the depth image into a pseudo-color image includes steps S141 to S143.

[0093] S141, obtaining a chromaticity map (colormap), the chromaticity map including a mapping relationship between color values ​​and pixel values.

[0094] In the embodiment of the present application, there are many kinds of chromaticity diagrams, and a chromaticity diagram suitable for the application scenario can be selected. In the embodiment of the present application, the chromaticity diagram is selected to be a chromaticity diagram that can largely distinguish the target object from the background.

[0095] In some implementations, a chromaticity map suitable for the target object detection scenario is selected, and the chromaticity map may be selected as colormap_jet.

[0096] In some implementations, the color value may be an RGB value.

[0097] S142, normalizing the depth image to obtain a normalized image corresponding to the depth image;

[0098] The pixel values ​​in the depth image are normalized to a range of 0 to 255, and the range of 0 to 255 can be a closed interval [0, 8000]. That is, the original range of each pixel value in the depth image is 0 to 8000, and through the normalization operation, these pixel value ranges are normalized to a range of 0 to 255.

[0099] As a non-limiting example, if the maximum pixel value in a depth image is a, and the depth value of a pixel point j in the depth image is z, then the pixel point j is normalized to between [0,225], then:

[0100]

[0101] Among them, G(j) represents the normalized value of pixel j.

[0102] S143, mapping the normalized image into a pseudo-color image according to the chromaticity diagram.

[0103] Since the chromaticity diagram includes the correspondence between color values ​​and pixel values, the color value corresponding to the pixel value of each pixel in the normalized image (i.e., the value after normalizing the depth value) can be determined according to the chromaticity diagram, thereby mapping the normalized image into a pseudo-color image.

[0104] In another embodiment, Figure 4As shown, after obtaining the pseudo-color image, step S160 is further included: a one-to-one correspondence between the annotation file corresponding to the depth image and the pseudo-color image of the depth image is made, so as to facilitate subsequent revision of the annotation text, which is not limited here.

[0105] S150, using the pseudo-color image of the depth image to correct the annotation file corresponding to the depth image, to obtain a corrected annotation data set.

[0106] More specifically, the label information and the coordinate information included in the annotation file are checked one by one using a pseudo-color image to obtain a corrected annotation data set, that is, a corrected annotation depth image data set.

[0107] In some other embodiments, step S150 also includes: using a labeling tool to correct the labeling file to obtain a corrected labeling data set, wherein the labeling tool corrects the labeling file by determining whether the label information and coordinate information included in the labeling file displayed on the pseudo-color image are correct.

[0108] After the color image is automatically annotated, the position of the box (i.e., the key point coordinate information) may be inaccurate, missing, or mislabeled, which may cause similar problems in the annotation file corresponding to the depth image. Therefore, in some embodiments, the annotation box can be corrected by the annotation tool to improve the accuracy of the annotation box. For the depth image aligned with the color image, since the visualization effect of the depth image is poor, manual re-inspection using pseudo-color images corresponding to each depth image with good visualization effect can allow users to perceive the target quickly and friendly, facilitate users to perform manual re-inspection, improve the efficiency and accuracy of re-inspection, and obtain a depth image dataset with high annotation accuracy.

[0109] In one embodiment, the annotation tool determines whether the information in the annotation file is correct by displaying the label information and coordinate information included in the annotation file on the pseudo-color image, so as to facilitate the user (i.e., the annotation personnel) to conduct a recheck. The user can check the label information and the human coordinate information. When an error is found, the user can input manual recheck data through an external device such as a microphone, a mouse, a keyboard, etc. The annotation tool corrects the label information and the coordinate information of the minimum circumscribed rectangle in the annotation file according to the manual recheck data input by the user, thereby obtaining a corrected annotation file. Furthermore, after the manual recheck, an accurately annotated data set will be obtained, that is, the annotated depth image is converted into an annotated pseudo-color image, and a data set including annotated depth image and annotated pseudo-color image is generated, which is not limited here.

[0110] It should be noted that if the user manually rechecks and finds that the annotation information corresponding to the annotation file does not need to be modified, the original annotation information will be retained.

[0111] In some embodiments, one or more pairs of one-to-one corresponding annotation information and pseudo-color images in the annotation files may be displayed on the same screen.

[0112] As a non-limiting example, the labeling tool is labelimg software, and labelimg software is used to perform batch labeling correction on pseudo-color images in which each frame corresponds to the labeling file.

[0113] Specifically, after the user opens the labelimg software, the main things that need to be checked are the label information and body coordinate information in the label file. For image information, the LabelImg software will automatically read the image information of the label file and the pseudo-color image. If there is a discrepancy between the image information in the label file and the pseudo-color image, the pseudo-color image will be automatically modified and written.

[0114] In the embodiments of the present application, on the one hand, a trained public model is used to perform reasoning detection on color images, and a labeling file is generated according to the detection results, thereby reducing manual labeling and greatly improving the labeling efficiency of the data set. On the other hand, converting the depth image into a pseudo-color image facilitates users to recheck the labeling file, improves the rechecking efficiency, and also improves the accuracy of the labeling, and obtains a data set with higher confidence. On the other hand, the labeling file is applied to the depth image, and a labeling method for quickly labeling the depth image is realized, which is convenient for labeling on the depth image and then training and learning, and can quickly carry out research on the depth image and promote the development of 3D related technologies.

[0115] It should be noted that Figure 2 The step numbers of the illustrated embodiment should not be interpreted as limiting the time sequence of the steps. It should be understood that in some other embodiments, the order of the steps can be swapped according to the logical relationship between the steps without affecting the implementation of the present solution. As a non-limiting example, step S140 is performed at any time after step S110 and before step S150.

[0116] Corresponding to the above image annotation method, an embodiment of the present application further provides an image annotation device. For details not described in detail in the image annotation device, please refer to the description of the above method.

[0117] Figure 5 FIG. 1 is a schematic diagram of a structure of an image annotation device provided by an embodiment of the present application. As an example, the image annotation device may be configured to Figure 1The electronic device shown. The image annotation device comprises: an acquisition module 61, an alignment module 62, a detection module 63, a file generation module 64, a conversion module 65 and an annotation module 66, wherein:

[0118] An acquisition module 61, used for acquiring color images and depth images;

[0119] An alignment module 62, configured to align the color image and the depth image;

[0120] A detection module 63 is used to detect the color image using a preset trained target detection model to obtain label information and key point coordinate information corresponding to the color image;

[0121] A file generating module 64, configured to generate a labeling file corresponding to the depth image according to the label information and key point coordinate information corresponding to the color image and based on the alignment relationship;

[0122] A conversion module 65, configured to convert the depth image into a pseudo-color image;

[0123] The annotation module 66 is used to correct the annotation files corresponding to the same group of the depth images based on the pseudo-color images of the depth images to obtain a corrected annotation data set.

[0124] In one embodiment, the acquisition module 61 can be used to acquire depth images and color images of multiple scenes. More specifically, the acquisition module 61 includes two cameras, a color camera and a depth camera. The color camera is used to capture color images of a scene, and the depth camera is used to capture depth images of the same scene. The color image can be, for example, an RGB image. Depth cameras include but are not limited to cameras based on time-of-flight (TOF) such as indirect time-of-flight (iToF) or direct time-of-flight (dToF), binocular vision, or structured light. The color camera and the depth camera are as close as possible to reduce the parallax between the two and improve the accuracy of the results.

[0125] In some other embodiments, the acquisition module 61 includes only one camera, and the acquisition module 61 includes a depth acquisition unit and a color acquisition unit, and the depth acquisition unit and the color acquisition unit are used to capture a depth image and a color image of each scene, respectively, which is not limited here.

[0126] In order to collect a sufficient amount of image data for deep learning, the scene can be changed multiple times, and for each scene, at least one set of depth images and color images can be obtained, so that sample image data in multiple different scenes can be obtained.

[0127] In some embodiments, the conversion module 65 is specifically used to:

[0128] Obtain a chromaticity diagram, the chromaticity diagram including a mapping relationship between color values ​​and pixel values;

[0129] Normalize the depth image to obtain a normalized image corresponding to the depth image;

[0130] Each normalized image is mapped into a pseudo-color image according to the chromaticity diagram.

[0131] In some embodiments, acquiring the chromaticity diagram includes: acquiring a chromaticity diagram of a target detection scene.

[0132] In some embodiments, the file generation module 64 is specifically used to:

[0133] The label information and key point coordinate information corresponding to the depth image, as well as the image information of the depth image, are written into a labeling file in a preset format to obtain a labeling file corresponding to the depth image.

[0134] In some embodiments, the annotated data set includes: converting the annotated depth image into an annotated pseudo-color image to obtain an annotated depth image data set.

[0135] In some embodiments, the preset trained target detection model is a publicly trained target detection model.

[0136] See also Figure 6 , Figure 6 6 is a schematic diagram of another image annotation device provided in an embodiment of the present application. The image annotation device includes: an acquisition module 61, an alignment module 62, a detection module 63, a file generation module 64, a conversion module 65, a correspondence module 67 and an annotation module 66. Figure 5 The same modules as those in the illustrated device will not be described in detail here.

[0137] The corresponding module 67 is used to make a one-to-one correspondence between the annotation file corresponding to each depth image and the pseudo-color image of the same group of depth images.

[0138] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0139] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned image annotation method embodiments when executing the computer program.

[0140] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various image annotation method embodiments can be implemented.

[0141] An embodiment of the present application provides a computer program product. When the computer program product is executed on an electronic device, the electronic device can implement the steps in each of the above-mentioned image annotation method embodiments.

[0142] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0144] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0148] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image annotation method, characterized in that: include: Get aligned color image and depth image; Detecting the color image using a preset trained target detection model to obtain label information and key point coordinate information corresponding to the color image, wherein the label information is the category information of the target; Generate a label file corresponding to the depth image based on the alignment relationship according to the label information and key point coordinate information corresponding to the color image; Converting the depth image into a pseudo-color image, and correcting the annotation file corresponding to the depth image based on the pseudo-color image of the depth image to obtain a corrected annotation data set, including: Using the pseudo color image to check the label information and the coordinate information included in the annotation file one by one to obtain the corrected annotation data set, further comprising: The annotation file is corrected by using an annotation tool to obtain the corrected annotation data set, wherein the annotation tool corrects the annotation file by determining whether label information and coordinate information included in the annotation file displayed on the pseudo-color image are correct.

2. The image annotation method according to claim 1, characterized in that: The converting the depth image into a pseudo-color image comprises: Acquire a chromaticity diagram, wherein the chromaticity diagram includes a mapping relationship between color values ​​and pixel values; Normalizing the depth image to obtain a normalized image corresponding to the depth image; The normalized image is mapped into a pseudo-color image according to the chromaticity diagram.

3. The image annotation method according to claim 1, characterized in that: The step of obtaining the aligned color image and depth image comprises: The pixels of the color image and the pixels of the depth image are aligned one by one.

4. The image annotation method according to claim 3, characterized in that: The step of generating a label file corresponding to the depth image based on the label information and key point coordinate information corresponding to the color image and the alignment relationship includes: Based on the alignment relationship between the color image and the depth image, the label information and key point coordinate information corresponding to the color image are converted into annotation information of the depth image, and written into an annotation file in a preset format to obtain an annotation file corresponding to the depth image.

5. The image annotation method according to claim 4, characterized in that: The annotation file also includes image information of the depth image, wherein the image information includes the length, width, channel, path and image name of the image.

6. An image annotation device, characterized in that: include: An acquisition module, used for acquiring color images and depth images; An alignment module, configured to align the color image and the depth image; A detection module, used to detect the color image using a preset trained target detection model to obtain label information and key point coordinate information corresponding to the color image, wherein the label information is the category information of the target; A file generation module, configured to generate a labeling file corresponding to the depth image according to the label information and key point coordinate information corresponding to the color image and based on the alignment relationship; A conversion module, used for converting the depth image into a pseudo-color image; A labeling module, used to correct the labeling file corresponding to the depth image based on the pseudo-color image of the depth image to obtain a corrected labeling data set, including: Using the pseudo color image to check the label information and the coordinate information included in the annotation file one by one to obtain the corrected annotation data set, further comprising: The annotation file is corrected by using an annotation tool to obtain the corrected annotation data set, wherein the annotation tool corrects the annotation file by determining whether label information and coordinate information included in the annotation file displayed on the pseudo-color image are correct.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image annotation method according to any one of claims 1 to 5 is implemented.

8. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image annotation method according to any one of claims 1 to 5 is implemented.

Images