Image labeling method and apparatus, and electronic device and storage medium

Abstract

Disclosed in the embodiments of the present disclosure are an image labeling method and apparatus, and an electronic device and a storage medium. The method comprises: labeling each infrared image by means of a grayscale map labeling model, so as to obtain a first pseudo label of each infrared image, wherein the grayscale map labeling model is constructed on the basis of a grayscale image converted from each color image and label data of each color image, and each infrared image and each color image constitute a pair of images; on the basis of a homologous-point correspondence between the grayscale image converted from each color image and a corresponding infrared image, transferring the label data of each color image to the corresponding infrared image, so as to obtain a second pseudo label of each infrared image; and on the basis of the first pseudo label and the second pseudo label, determining label data of each infrared image.

Description

Image annotation method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese invention patent application with application number 202311415461.0 filed on October 27, 2023 and titled “A method, device, electronic device and storage medium for image annotation”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and more particularly to an image annotation method, apparatus, electronic device, and storage medium. Background Art

[0003] Currently, some scenarios require both image acquisition devices capable of capturing color (Red, Green, Blue, RGB) and infrared (Infrared Radiation, IR) images. Therefore, algorithms for both RGB and IR images need to be deployed. To provide these algorithms, a large number of RGB and IR images must be annotated simultaneously. This results in long annotation cycles and high costs.

[0004] Summary of the Invention

[0005] Embodiments of the present disclosure provide an image annotation method, apparatus, electronic device, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides an image annotation method, comprising: annotating each infrared image through a grayscale annotation model to obtain a first pseudo-label for each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; according to the correspondence relationship of the same-name points from the grayscale image converted from each color image to the corresponding infrared image, the label data of each color image is migrated to the corresponding infrared image to obtain a second pseudo-label for each infrared image; and according to the first pseudo-label and the second pseudo-label, the label data of each infrared image is determined.

[0007] In a second aspect, an embodiment of the present disclosure further provides an image annotation device, comprising: a model annotation module, configured to annotate each infrared image through a grayscale annotation model to obtain a first pseudo-label for each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein the each infrared image and the each color image constitute a paired image; a label migration module, configured to migrate the label data of each color image to the corresponding infrared image based on the correspondence relationship of the same-name points from the grayscale image converted from the each color image to the corresponding infrared image, to obtain a second pseudo-label for each infrared image; and a label determination module, configured to determine the label data of each infrared image based on the first pseudo-label and the second pseudo-label.

[0008] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image annotation method as described in any one of the embodiments of the present disclosure.

[0009] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to execute the image annotation method as described in any one of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0011] FIG1 is a schematic diagram of a flow chart of an image annotation method provided by an embodiment of the present disclosure;

[0012] FIG2 is a flow chart of a process for labeling label data of each color image in an image labeling method provided by an embodiment of the present disclosure;

[0013] FIG3 is a flowchart of an image annotation method provided by an embodiment of the present disclosure;

[0014] FIG4 is a flow chart of a process for determining correspondences of homonymous points in an image annotation method provided by an embodiment of the present disclosure;

[0015] FIG5 is a schematic diagram of feature sub-matching in an image annotation method provided by an embodiment of the present disclosure;

[0016] FIG6 is a schematic structural diagram of an image annotation device provided by an embodiment of the present disclosure;

[0017] FIG7 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0020] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions. In addition, during the collection and use of data, the user's personal information will not be collected or processed in a manner that does not reveal the user's identity.

[0024] Figure 1 is a flow chart illustrating an image annotation method provided by an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to labeling paired color and infrared images. The method can be performed by an image annotation device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer.

[0025] As shown in FIG1 , the image annotation method provided by this embodiment may include steps S110 , S120 , S130 , and S140 .

[0026] S110 , annotating each infrared image using a grayscale image annotation model to obtain a first pseudo label for each infrared image.

[0027] In the disclosed embodiments, the grayscale annotation model is constructed based on grayscale images converted from each color image and the label data for each color image; wherein each infrared image and each color image constitute a paired image. In some embodiments, the color image can be understood as a true color image, i.e., a (Red, Green, Blue, RGB) image.

[0028] In applications such as vehicle-mounted cameras and security cameras, cameras can integrate both color and infrared (IR) image acquisition devices to enable full-time, multi-detail image capture. In some embodiments, the color and infrared image acquisition devices can be considered to be closely integrated within the camera, with acquisition timestamps substantially aligned. Therefore, each color image and infrared image captured by the camera can be considered a paired image.

[0029] In this embodiment, the label data for color images and infrared images can be considered as graphic label data within the image, such as object outline label data, rectangular frame label data, etc. For example, in some implementations, the image annotation method can be applied to the field of image detection; in particular, the label data for color and infrared images can include detection frame data of target objects, thereby enabling image detection frame labeling, laying the foundation for the development and optimization of image detection algorithms for color and infrared images.

[0030] Since color images are relatively easy to annotate, they can be annotated first to obtain their label data. In some embodiments, the label data for color images can include manually annotated true label data and high-quality pseudo-label data. However, due to slight deviations in the integration locations and acquisition timestamps of the color and infrared image acquisition devices, the paired color and infrared images may differ, making it generally impossible to transfer the label data of the color image to the infrared image intact.

[0031] Because grayscale images converted from color images are similar to corresponding infrared images, in the disclosed embodiments, each color image can first be converted to a grayscale image based on existing conversion methods. For example, a weighted average of the three-channel components of the color image can be performed to convert the grayscale image. Then, at least one grayscale image annotation model can be constructed using the label data of each grayscale image and the color image. Finally, based on the constructed grayscale image model, pseudo-labels can be applied to each unlabeled infrared image to obtain a first pseudo-label for each infrared image.

[0032] In some embodiments, the grayscale image annotation model may include an existing large image processing model. For example, the grayscale image annotation model may include a detection algorithm model with relatively good performance, such as the Cascade Region Convolutional Neural Network (Cascade R-CNN) with the Vision Transformer (ViT) as the backbone network; it may also include a semi-supervised detection algorithm model, such as a SOFT-teacher model. The processing results of these existing large image processing models are very accurate and can meet the high-quality requirements of pseudo-label marking.

[0033] In some optional implementations, annotating each infrared image through a grayscale annotation model to obtain a first pseudo-label for each infrared image may include: annotating each infrared image through at least two grayscale annotation models; when the annotation results of the two grayscale annotation models are consistent, using the annotation result as the first pseudo-label for the corresponding infrared image.

[0034] In these optional implementations, the consistency of the labeling results of at least two grayscale image annotation models can be considered to mean that the at least two grayscale image annotation models annotate the same amount of graphic label data, with positions and sizes within a preset error range. By constructing two or more grayscale image annotation models using the label data of each grayscale image and the corresponding color image, the labeling results of each infrared image can be mutually corrected using the two or more grayscale image annotation models. The labeling results that are consistent are used as the first pseudo-labels for the corresponding infrared images, thereby selecting high-quality pseudo-label data for the infrared images.

[0035] In addition, other strategies can be used to mutually correct the labeling results of each infrared image using two or more grayscale image annotation models. For example, if the labeling results of grayscale image annotation models that meet a preset ratio are consistent, the labeling result of the grayscale image model that meets the preset ratio can be used as the first pseudo-label for the corresponding infrared image, and so on. This is not an exhaustive list.

[0036] S120 , based on the correspondence relationship between the grayscale images converted from the color images and the corresponding infrared images, migrate the label data of each color image to the corresponding infrared image to obtain a second pseudo label for each infrared image.

[0037] In the embodiment of the present disclosure, step S110 and step S120 do not have a strict time sequence relationship.

[0038] Because the grayscale image converted from a color image and the corresponding infrared image contain similar image content, the grayscale image and infrared image corresponding to the same color image often have many points of the same name. In some embodiments, points of the same name can be considered to be pixels representing the same object point in at least two images.

[0039] In this embodiment, based on existing feature-based image matching methods, the pixel position transformation relationship between the grayscale image converted from the color image and the same-name points on the infrared image corresponding to the color image can be determined and used as the same-name point correspondence relationship. In some embodiments, the feature-based image matching method includes, for example, an image matching method based on deep learning feature extraction, and an image graph matching method based on traditional feature extraction. Traditional feature extraction methods include, for example, Scale-Invariant Feature Transform (SIFT).

[0040] In some embodiments, the correspondence between the key points of the grayscale image converted from each color image and the corresponding infrared image can be used as the inter-image graphical correspondence between each color image and the corresponding infrared image. Furthermore, based on this correspondence, the label data of each color image can be mapped to the corresponding infrared image to correct any label deviations between the color and infrared images, thereby obtaining a more accurate second pseudo-label for each infrared image.

[0041] S130 : Determine label data of each infrared image according to the first pseudo label and the second pseudo label.

[0042] In the disclosed embodiment, the first pseudo-label marked by the grayscale image annotation model and the second pseudo-label migrated by the label data based on image matching can be combined to correct each other and select high-quality infrared image annotation results.

[0043] For example, in some optional implementations, determining the label data of each infrared image based on the first pseudo-label and the second pseudo-label may include: when the first pseudo-label and the second pseudo-label are consistent, using any label of the first pseudo-label and the second pseudo-label as the label data of the corresponding infrared image.

[0044] In these optional implementations, the first and second pseudo-labels are identical, which can be considered to mean that the first and second pseudo-labels contain the same amount of graphic label data, and their positions and sizes are within a preset error range. If the first and second pseudo-labels are identical, it can be considered that the label data for the infrared image has been obtained with high accuracy. In this case, either the first or second pseudo-label can be used as the label data for the corresponding infrared image.

[0045] In addition, the first pseudo-label and the second pseudo-label can be used to determine the label data of the infrared image based on other strategies. For example, the first pseudo-label can be used as the main label data, and the second pseudo-label can be used as the auxiliary label data for correction. If the graphic label data in the first pseudo-label is consistent with at least part of the graphic label data in the second pseudo-label, the first pseudo-label can be used as the label data of the corresponding infrared image, and so on. This is not an exhaustive list.

[0046] The technical solution of the embodiment of the present disclosure is to annotate each infrared image through a grayscale image annotation model to obtain a first pseudo label for each infrared image; wherein, the grayscale image annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein, each infrared image and each color image constitute a paired image; according to the correspondence relationship of the same-name points from the grayscale image converted from each color image to the corresponding infrared image, the label data of each color image is migrated to the corresponding infrared image to obtain a second pseudo label for each infrared image; according to the first pseudo label and the second pseudo label, the label data of each infrared image is determined.

[0047] For paired color images and infrared images, the grayscale image converted from the color image has similarity with the corresponding infrared image. On this basis, by converting the color image into a grayscale image and using the label data of the grayscale image and the color image to construct a grayscale image annotation model, the trained grayscale image annotation model can achieve relatively accurate annotation of the infrared image and obtain a first pseudo-label. In addition, based on the image matching relationship between the color image and the infrared image (i.e., the correspondence relationship between the same-name points), the label data of the color image is migrated to the corresponding infrared image, and a relatively accurate second pseudo-label can also be obtained. Finally, the first pseudo-label and the second pseudo-label can be combined to correct each other to obtain high-quality label data for each infrared image. By completing the automatic annotation of infrared images based on the label data of the color image, it is possible to not only improve the annotation efficiency, shorten the annotation cycle, and reduce the annotation cost, but also ensure the annotation quality.

[0048] The present embodiment can be combined with the various optional solutions in the image annotation method provided in the above embodiments. The image annotation method provided in this embodiment provides a detailed description of the labeling process of color image label data. It can use a small amount of true label data of color images to complete the annotation of the remaining color images, which can further improve the annotation efficiency, shorten the annotation cycle, and reduce the annotation cost.

[0049] FIG2 is a flow chart illustrating the process of labeling the label data of each color image in an image labeling method provided by an embodiment of the present disclosure. As shown in FIG2 , the process of labeling the label data of each color image in the image labeling method provided by this embodiment may include steps S210, S220, and S230.

[0050] S210: Construct at least two color image annotation models based on the color images with true label data in each color image.

[0051] In this embodiment, partial images can be sampled from each color image, and true label data for these sampled images can be obtained through manual annotation. Furthermore, at least two color map annotation models can be constructed based on these sampled images and their true label data. In some embodiments, the at least two color map annotation models can also include existing large image processing models, such as the Cascade R-CNN model and the SOFT-teacher model, which are not exhaustive here.

[0052] S220 , annotating the remaining color images that do not have true label data in each color image by constructing at least two color image annotation models.

[0053] In some embodiments, two color image annotation models may be used to perform pseudo-labeling on the remaining color images that are not manually annotated (ie, do not have true label data) in each color image.

[0054] S230: Determine label data for the remaining color images based on the annotation results of at least two color image annotation models.

[0055] In some embodiments, the annotation results of different color image annotation models can be used to perform mutual corrections to select high-quality label data for color images. In some implementations, determining the label data for the remaining color images based on the annotation results of at least two color image annotation models can include: if the annotation results of the two color image annotation models are consistent, using the annotation results as the label data for the remaining color images. Alternatively, it can also include: if the annotation results of the color image annotation models that meet a preset ratio are consistent, using the annotation results of the color image model that meets the preset ratio as the label data for the remaining color images, etc., which are not exhaustive here.

[0056] For example, Figure 3 is a flowchart of an image annotation method provided by an embodiment of the present disclosure. Referring to Figure 3 , the image annotation method may include steps S310, S320, S330, S340, S350, S360, S370, and S380.

[0057] S310 , extracting frames from the color video and infrared video captured by the camera to obtain pairs of color images and infrared images.

[0058] S320: Manually label some color images to obtain true label data.

[0059] S330. Construct at least two color image annotation models based on the color image with true label data.

[0060] S340: annotate the remaining color images that do not have true label data in each color image by constructing at least two color image annotation models, and correct each other based on the annotation results to obtain label data of each color image.

[0061] S350: Convert the color image into a grayscale image, and construct at least two grayscale image annotation models using the label data of each grayscale image and color image.

[0062] S360: Annotate each infrared image using at least two grayscale image annotation models, and correct each other based on the annotation results to obtain a first pseudo label for each infrared image.

[0063] S370 , based on the correspondence between the grayscale images converted from the color images and the corresponding infrared images, migrate the label data of each color image to the corresponding infrared image to obtain a second pseudo label for each infrared image.

[0064] S380: Combining the first pseudo label marked by the grayscale image annotation model and the second pseudo label migrated based on the image matching label data to correct each other and output high-quality label data of the infrared image.

[0065] Furthermore, during the image annotation process, unlabeled infrared and color images can be manually annotated to obtain their true label data. Experimental verification shows that only 20%-30% of the color images need to be annotated to complete the labeling of the remaining color images and unlabeled infrared images. This significantly reduces the annotation cost and shortens the annotation cycle, while also enabling the acquisition of a large amount of high-quality pseudo-labeled data. Furthermore, the percentage of unlabeled infrared images can be reduced to approximately 10%, enabling rapid response to algorithm development and optimization work on infrared images.

[0066] The technical solution of the embodiment of the present disclosure provides a detailed description of the labeling process of the label data of the color image. It can use a small amount of true label data of the color image to complete the labeling of the remaining color images, which can further improve the labeling efficiency, shorten the labeling cycle, and reduce the labeling cost. The image labeling method provided by the embodiment of the present disclosure and the image labeling method provided by the above embodiment belong to the same disclosed concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0067] The embodiments of this disclosure can be combined with the various optional solutions in the image annotation method provided in the above embodiments. The image annotation method provided in this embodiment provides a detailed description of the process for determining the correspondence between homonymous points. It can determine the affine transformation matrix of the homonymous points from the grayscale image converted from the color image to the infrared image, and use it as the correspondence between the homonymous points to achieve label migration from the color image to the infrared image.

[0068] Figure 4 is a flow chart of a process for determining the correspondence between homonymous points in an image annotation method provided by an embodiment of the present disclosure. As shown in Figure 4, the process for determining the correspondence between homonymous points in the image annotation method provided by this embodiment may include steps S410, S420, and S430.

[0069] S410 , extracting a first feature descriptor from the grayscale image converted from each color image and a second feature descriptor from the corresponding infrared image.

[0070] In this embodiment, each color image can be converted into a grayscale image based on existing conversion methods. A first feature descriptor for the grayscale image and a second feature descriptor for the corresponding infrared image can be obtained based on a deep learning model. In some embodiments, the first feature descriptor and the second feature descriptor can be considered to be feature points of objects in the image, such as corners.

[0071] S420: Match the first feature descriptor and the second feature descriptor, and use the matched first feature descriptor and second feature descriptor as points of the same name.

[0072] In this embodiment, the first feature descriptor and the second feature descriptor can be matched based on an existing matching method (e.g., based on an Open CV tool). In some embodiments, the matched first feature descriptor and the second feature descriptor can be considered to be the same object points (i.e., points with the same name) in the grayscale image and the color image.

[0073] S430 , determining an affine transformation matrix from each color image to the corresponding infrared image based on the pixel coordinates of the points of the same name in the color image and the corresponding infrared image.

[0074] In this embodiment, the pixel coordinates of the first feature descriptor in the grayscale image are the same as the pixel coordinates in the color image. The pixel coordinates of each matching first feature descriptor and second feature descriptor on their respective images can be obtained as the pixel coordinates of each point of the same name in the color image and the corresponding infrared image. Furthermore, the affine transformation matrix from the color image to the infrared image can be determined based on an existing solution algorithm (for example, based on an Open CV tool). According to the affine transformation matrix, the label data of the color image can be mapped to the infrared image. For example, the pixel position of each point in the label data of the color image can be calculated with the affine transformation matrix to obtain the pixel position corresponding to each point in the infrared image, and then the label data of the infrared image can be obtained.

[0075] In some optional implementations, matching the first feature descriptor and the second feature descriptor may include: dividing each color image and the corresponding infrared image into image blocks in the same manner; and matching the first feature descriptor in each image block of each color image with the second feature descriptor of the same image block in the corresponding infrared image.

[0076] For example, Figure 5 is a schematic diagram of feature sub-matching in an image annotation method provided by an embodiment of the present disclosure. Referring to Figure 5 , each color image and corresponding infrared image can be divided into 2×2 image blocks, resulting in image blocks 1 through 4 for the color image and image blocks 1' through 4' for the infrared image; where image block 1 corresponds to image block 1', image block 2 corresponds to image block 2', image block 3 corresponds to image block 3', and image block 4 corresponds to image block 4'. Matching of the first feature descriptor and the second feature descriptor can be performed within the corresponding image blocks.

[0077] In these optional implementations, by dividing each color image and the corresponding infrared image into image blocks in the same way, and matching the first feature descriptor and the second feature descriptor in the corresponding image blocks, it is possible to determine the same-name points in each corresponding image block, thereby avoiding the same-name points from being concentrated in partial areas of the color image and the infrared image, improving the accuracy of the affine transformation matrix, and being more conducive to the accurate migration of the label data of the color image to the infrared image.

[0078] The technical solution of the disclosed embodiment describes in detail the process of determining the correspondence between homonymous points. It can determine the affine transformation matrix of the homonymous points from the grayscale image converted from the color image to the infrared image and use this as the correspondence between the homonymous points to achieve label migration from the color image to the infrared image. The image annotation method provided in the disclosed embodiment and the image annotation method provided in the above embodiment are based on the same disclosed concept. Technical details not fully described in this embodiment can be referred to in the above embodiment. The same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0079] Figure 6 is a schematic diagram of the structure of an image tagging device provided by an embodiment of the present disclosure. The image tagging device provided by this embodiment is suitable for labeling a pair of color images and infrared images.

[0080] As shown in Figure 6, the image annotation device provided by the embodiment of the present disclosure may include: a model annotation module 610, which is used to annotate each infrared image through a grayscale image annotation model to obtain a first pseudo label for each infrared image; wherein the grayscale image annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; a label migration module 620, which is used to migrate the label data of each color image to the corresponding infrared image based on the correspondence relationship of the same-name points from the grayscale image converted from each color image to the corresponding infrared image, to obtain a second pseudo label for each infrared image; a label determination module 630, which is used to determine the label data of each infrared image based on the first pseudo label and the second pseudo label.

[0081] In some optional implementations, the model annotation module can be used to: annotate each infrared image using at least two grayscale annotation models; when the annotation results of the two grayscale annotation models are consistent, the annotation result is used as the first pseudo label of the corresponding infrared image.

[0082] In some optional implementations, the image annotation device may also include: a color image annotation module, used to annotate the label data of each color image based on the following process: constructing at least two color image annotation models based on the color images with true label data in each color image; annotating the remaining color images in each color image that do not have true label data through the at least two constructed color image annotation models; and determining the label data of the remaining color images based on the annotation results of the at least two color image annotation models.

[0083] In some optional implementations, the image annotation device may further include: a correspondence determination module, used to determine the correspondence relationship of the same-name points based on the following process: extracting the first feature descriptor in the grayscale image converted from each color image and the second feature descriptor in the corresponding infrared image; matching the first feature descriptor and the second feature descriptor, and using the matched first feature descriptor and second feature descriptor as the same-name points; and determining the affine transformation matrix from each color image to the corresponding infrared image based on the pixel coordinates of the same-name points in the color image and the corresponding infrared image respectively.

[0084] In some optional implementations, the correspondence determination module can be used to: divide each color image and the corresponding infrared image into image blocks in the same manner; and match the first feature descriptor in each image block of each color image with the second feature descriptor of the same image block in the corresponding infrared image.

[0085] In some optional implementations, the label determination module may be configured to: when the first pseudo label is consistent with the second pseudo label, use any one of the first pseudo label and the second pseudo label as label data corresponding to the infrared image.

[0086] In some optional implementations, the method is applied to the field of image detection, wherein the label data of the infrared image includes detection frame data of the target object.

[0087] The image annotation device provided in the embodiments of the present disclosure can execute the image annotation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0089] Reference is now made to FIG7 , which illustrates a schematic diagram of the structure of an electronic device (e.g., a terminal device or server in FIG7 ) 700 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG7 is merely an example and should not limit the functionality or scope of use of the embodiments of the present disclosure.

[0090] As shown in Figure 7, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0091] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 7 shows the electronic device 700 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may be implemented or present instead.

[0092] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the image annotation method of the embodiment of the present disclosure are performed.

[0093] The electronic device provided by the embodiment of the present disclosure and the image annotation method provided by the above embodiment belong to the same disclosed concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0094] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the image annotation method provided in the above embodiment is implemented.

[0095] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0096] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0097] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0098] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the image annotation method provided by the above embodiment: annotating each infrared image through a grayscale annotation model to obtain a first pseudo label for each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; according to the correspondence relationship of the same-name points from the grayscale image converted from each color image to the corresponding infrared image, the label data of each color image is migrated to the corresponding infrared image to obtain a second pseudo label for each infrared image; and according to the first pseudo label and the second pseudo label, the label data of each infrared image is determined.

[0099] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the names of the units and modules do not, in certain circumstances, limit the units and modules themselves.

[0102] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and the like.

[0103] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] According to one or more embodiments of the present disclosure, an image annotation method is provided, which includes: annotating each infrared image through a grayscale annotation model to obtain a first pseudo-label for each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein the each infrared image and the each color image constitute a paired image; according to the correspondence relationship of the same-name points from the grayscale image converted from the each color image to the corresponding infrared image, the label data of the each color image is migrated to the corresponding infrared image to obtain a second pseudo-label for each infrared image; and according to the first pseudo-label and the second pseudo-label, the label data of each infrared image is determined.

[0105] According to one or more embodiments of the present disclosure, an image annotation method is provided, which also includes: in some optional implementations, annotating each infrared image through a grayscale annotation model to obtain a first pseudo label of each infrared image, including: annotating each infrared image through at least two grayscale annotation models; when the annotation results of the two grayscale annotation models are consistent, using the annotation results as the first pseudo label of the corresponding infrared image.

[0106] According to one or more embodiments of the present disclosure, an image annotation method is provided, which also includes: in some optional implementations, the annotation process of the label data of each color image includes: constructing at least two color image annotation models based on the color images with true label data among the color images; annotating the remaining color images without true label data among the color images through the at least two constructed color image annotation models; and determining the label data of the remaining color images based on the annotation results of the at least two color image annotation models.

[0107] According to one or more embodiments of the present disclosure, an image annotation method is provided, which also includes: in some optional implementations, the process of determining the correspondence relationship of the same-name points includes: extracting the first feature descriptor in the grayscale image converted from each color image and the second feature descriptor in the corresponding infrared image; matching the first feature descriptor and the second feature descriptor, and using the matched first feature descriptor and second feature descriptor as the same-name points; and determining the affine transformation matrix from each color image to the corresponding infrared image based on the pixel coordinates of the same-name points in the color image and the corresponding infrared image, respectively.

[0108] According to one or more embodiments of the present disclosure, an image annotation method is provided, which also includes: in some optional implementations, matching the first feature descriptor and the second feature descriptor includes: dividing the color images and the corresponding infrared images into image blocks in the same manner; matching the first feature descriptor in each image block of the color images with the second feature descriptor of the same image block in the corresponding infrared image.

[0109] According to one or more embodiments of the present disclosure, an image annotation method is provided, which also includes: in some optional implementations, determining the label data of each infrared image based on the first pseudo-label and the second pseudo-label, including: when the first pseudo-label is consistent with the second pseudo-label, using any label of the first pseudo-label and the second pseudo-label as the label data of the corresponding infrared image.

[0110] According to one or more embodiments of the present disclosure, an image annotation method is provided, further comprising: in some optional implementations, being applied to the field of image detection; wherein the label data of the infrared image includes detection box data of the target object.

[0111] According to one or more embodiments of the present disclosure, an image annotation device is provided, which includes: a model annotation module, used to annotate each infrared image through a grayscale annotation model to obtain a first pseudo label for each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein the each infrared image and the each color image constitute a paired image; a label migration module, used to migrate the label data of each color image to the corresponding infrared image according to the correspondence relationship of the same-name points from the grayscale image converted from the each color image to the corresponding infrared image, to obtain a second pseudo label for each infrared image; and a label determination module, used to determine the label data of each infrared image based on the first pseudo label and the second pseudo label.

[0112] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0113] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0114] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An image annotation method, comprising: Annotating each infrared image through a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on a grayscale image converted from each color image and label data of each color image; wherein each infrared image and each color image constitute a paired image; According to the correspondence relationship between the grayscale images converted from the color images and the corresponding infrared images, the label data of the color images are transferred to the corresponding infrared images to obtain the second pseudo labels of the infrared images; The label data of each infrared image is determined according to the first pseudo label and the second pseudo label.

2. The method according to claim 1, wherein: The step of labeling each infrared image by using a grayscale image labeling model to obtain a first pseudo label of each infrared image includes: Annotating each of the infrared images by using at least two grayscale image annotation models; When the annotation results of the two grayscale image annotation models are consistent, the annotation results are used as the first pseudo label of the corresponding infrared image.

3. The method according to claim 1, wherein: The labeling process of the label data of each color image includes: Constructing at least two color image annotation models according to the color images having true label data among the color images; By constructing the at least two color image annotation models, annotating the remaining color images without true label data in the color images; Based on the annotation results of the at least two color image annotation models, label data of the remaining color images are determined.

4. The method according to claim 1, wherein: The process of determining the correspondence relationship of the same-name points includes: Extracting a first feature descriptor in the grayscale image converted from each color image and a second feature descriptor in the corresponding infrared image; Matching the first feature descriptor and the second feature descriptor, and taking the matched first feature descriptor and the second feature descriptor as points of the same name; Based on the pixel coordinates of the same-name points in the color image and the corresponding infrared image, an affine transformation matrix from each color image to the corresponding infrared image is determined.

5. The method according to claim 4, wherein: The matching the first feature descriptor with the second feature descriptor includes: Dividing each color image and the corresponding infrared image into image blocks in the same manner; The first feature descriptor in each image block of each color image is matched with the second feature descriptor of the same image block in the corresponding infrared image.

6. The method according to claim 1, wherein: The step of determining the label data of each infrared image according to the first pseudo label and the second pseudo label includes: When the first pseudo label is consistent with the second pseudo label, the first pseudo label and the second pseudo label are Any tag in the tag is used as the tag data of the corresponding infrared image.

7. The method according to any one of claims 1 to 6, wherein: Applied to the field of image detection; wherein the label data of the infrared image includes detection frame data of the target object.

8. An image annotation device, comprising: A model annotation module, used for annotating each infrared image by a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; A label migration module, used for migrating the label data of each color image to the corresponding infrared image according to the correspondence relationship of the same-name points between the grayscale image converted from each color image and the corresponding infrared image, so as to obtain a second pseudo label of each infrared image; The label determination module is used to determine the label data of each infrared image according to the first pseudo label and the second pseudo label.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image annotation method as described in any one of claims 1 to 7.

10. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image annotation method according to any one of claims 1 to 7 when executed by a computer processor.

Images