An image labeling method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose an image labeling method and device, electronic equipment and storage medium, wherein the method comprises: labeling each infrared image through a grayscale image labeling model to obtain first pseudo labels of the infrared images; wherein the grayscale image labeling model is constructed based on grayscale images converted from each color image and label data of the color images; wherein the infrared images and the color images constitute paired images; according to the correspondence relationship between the same points of the grayscale images converted from the color images and the corresponding infrared images, the label data of the color images is migrated to the corresponding infrared images to obtain second pseudo labels of the infrared images; and according to the first pseudo labels and the second pseudo labels, the label data of the infrared images is determined. The labeling efficiency can be improved, the labeling period can be shortened, and the labeling cost can be reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to an image labeling method and device, electronic equipment and storage medium. BACKGROUND

[0002] Currently, in some scenarios, image acquisition devices capable of acquiring color images (Red Green Blue, RGB) and image acquisition devices capable of acquiring infrared images (Infrared Radiation, IR) need to be set up at the same time, and therefore the algorithm capabilities of RGB images and IR images need to be deployed at the same time. In order to provide algorithm capabilities, a large number of RGB images and IR images need to be labeled at the same time. This will result in a long labeling period and high labeling cost. SUMMARY

[0003] Embodiments of the present disclosure provide an image labeling method and device, electronic equipment and storage medium, which can improve labeling efficiency, shorten labeling period and reduce labeling cost.

[0004] In a first aspect, embodiments of the present disclosure provide an image labeling method, comprising:

[0005] labeling each infrared image by using a grayscale image labeling model to obtain first pseudo labels of the infrared images; wherein the grayscale image labeling model is constructed based on grayscale images converted from each color image and label data of the color images; wherein the infrared images and the color images constitute paired images;

[0006] migrating the label data of the color images to corresponding infrared images according to a same-name point correspondence relationship from the grayscale images converted from the color images to the corresponding infrared images, to obtain second pseudo labels of the infrared images;

[0007] determining label data of the infrared images according to the first pseudo labels and the second pseudo labels.

[0008] In a second aspect, embodiments of the present disclosure further provide an image labeling device, comprising:

[0009] a model labeling module configured to label each infrared image by using a grayscale image labeling model to obtain first pseudo labels of the infrared images; wherein the grayscale image labeling model is constructed based on grayscale images converted from each color image and label data of the color images; wherein the infrared images and the color images constitute paired images;

[0010] a label migration module configured to migrate label data of the color images to corresponding infrared images according to a same-name point correspondence relationship from the grayscale images converted from the color images to the corresponding infrared images, to obtain second pseudo-labels of the infrared images;

[0011] a label determination module configured to determine label data of the infrared images according to the first pseudo-labels and the second pseudo-labels.

[0012] In a third aspect, the embodiments of the present disclosure further provide an electronic device, which comprises:

[0013] one or more processors;

[0014] a storage device configured to store one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the image labeling method according to any of the embodiments of the present disclosure.

[0016] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to perform the image labeling method according to any of the embodiments of the present disclosure.

[0017] The technical solution of the embodiments of the present disclosure labels each infrared image by using a grayscale image labeling model to obtain first pseudo-labels of the infrared images; the grayscale image labeling model is constructed based on grayscale images converted from color images and label data of the color images; the color images and the infrared images constitute paired images; label data of the color images is migrated to corresponding infrared images according to a same-name point correspondence relationship from the grayscale images converted from the color images to the corresponding infrared images, to obtain second pseudo-labels of the infrared images; and label data of the infrared images is determined according to the first pseudo-labels and the second pseudo-labels.

[0018] For pairs of color images and infrared images, the color image converted into a grayscale image has similarity with the corresponding infrared image. On this basis, by converting the color image into a grayscale image and using the label data of the grayscale image and the color image to construct a grayscale image labeling model, the trained grayscale image labeling model can realize relatively accurate labeling of the infrared image, obtaining first pseudo-labels. In addition, according to the image matching relationship (i.e. the corresponding relationship of the same name points) of the color image to the infrared image, the label data of the color image can be transferred to the corresponding infrared image, and second pseudo-labels can also be obtained. Finally, the first pseudo-labels and the second pseudo-labels can be combined to correct each other, obtaining high-quality label data of each infrared image. Through automatic labeling of the infrared image based on the label data of the color image, not only can the labeling efficiency be improved, the labeling period be shortened, and the labeling cost be reduced, but also the labeling quality can be guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other features, advantages, and aspects of the present disclosure will become more apparent as various embodiments of the present disclosure are described in conjunction with the following detailed description, taken in conjunction with the accompanying drawings. Throughout the drawings, like or similar reference numerals refer to like or similar elements. It should be understood that the drawings are schematic and elements and features are not necessarily to scale.

[0020] Figure 1 A flowchart of an image labeling method provided by an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a labeling process of label data of each color image in an image labeling method provided by an embodiment of the present disclosure;

[0022] Figure 3 A flowchart of an image labeling method provided by an embodiment of the present disclosure;

[0023] Figure 4 A flowchart of a determination process of a same name point corresponding relationship in an image labeling method provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of feature sub-matching in an image labeling method provided by an embodiment of the present disclosure;

[0025] Figure 6 A structural schematic diagram of an image labeling apparatus provided by an embodiment of the present disclosure;

[0026] Figure 7 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term "comprising" and variations thereof as used herein are open-ended, that is "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related terms are defined in the following description.

[0030] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0032] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws, regulations and provisions.

[0033] Figure 1 A flowchart of an image labeling method provided by an embodiment of the present disclosure is shown. The embodiment of the present disclosure is applicable to the case of labeling paired color images and infrared images. The method can be performed by an image labeling device, which can be implemented in the form of software and / or hardware, and can be configured in an electronic device, such as a computer.

[0034] As shown in Figure 1 The image labeling method provided by the embodiment can include:

[0035] S110, label each infrared image by a grayscale image labeling model to obtain a first pseudo label of each infrared image.

[0036] In the embodiments of the present disclosure, the grayscale image annotation model is constructed based on the grayscale images converted from the color images and the label data of the color images. The pairs of images are formed by the infrared images and the color images. The color image can be understood as a true color image, i.e., a Red Green Blue (RGB) image.

[0037] In the application scenarios of vehicle-mounted cameras, security cameras, etc., the camera can integrate color image and infrared image (Infrared Radiation, IR) acquisition devices to realize image acquisition in all time periods and with multiple details. The color image and infrared image acquisition devices can be considered to be integrated close to each other in the camera, and the acquisition timestamps are basically aligned, so the color images and infrared images acquired by the camera can be considered as pairs of images.

[0038] In the embodiments, the label data of the color images and the label data of the infrared images can be considered as in-image graphic label data, such as object contour box label data, rectangular box label data, etc. For example, in some implementations, the image annotation method can be applied to the image detection field. The label data of the color images and the infrared images can include detection box data of target objects. Thus, the detection box labeling of the images can be realized, laying a foundation for the research and optimization of image detection algorithms for color images and infrared images.

[0039] Since the color image is relatively low in labeling difficulty, the color image can be labeled first to obtain its label data. The label data of the color image can include manually labeled true label data and high-quality pseudo label data. Since the integrated positions and acquisition timestamps of the color image and infrared image acquisition devices have slight deviations, the pairs of color images and infrared images also have differences, and the label data of the color image cannot be directly migrated to the infrared image.

[0040] Since the grayscale image converted from the color image has similarity with the corresponding infrared image, in the embodiments of the present disclosure, the color images can be first converted into grayscale images based on the existing conversion method, for example, the three-channel components of the color image can be weighted and averaged to convert into a grayscale image. Then, at least one grayscale image annotation model can be constructed using the label data of the grayscale images and the color images. Finally, the unlabeled infrared images can be pseudo-labeled based on the constructed grayscale image model to obtain the first pseudo label of each infrared image.

[0041] The grayscale image labeling model can include an existing image processing large model. For example, the grayscale image labeling model can include a currently optimal detection algorithm model, such as a cascade region convolutional neural network model (Cascade R-CNN) with a vision transformer (ViT) as a backbone network; and can also include a semi-supervised detection algorithm model, such as a teacher model (SOFT-teacher). The processing result accuracy of the existing image processing large model is very high, and can meet the high quality requirement of pseudo-label labeling.

[0042] In some optional implementations, labeling each infrared image by using the grayscale image labeling model to obtain the first pseudo label of each infrared image can include: labeling each infrared image by using at least two grayscale image labeling models; and in a case where labeling results of the two grayscale image labeling models are consistent, taking the labeling result as the first pseudo label of the corresponding infrared image.

[0043] In these optional implementations, the labeling results of the at least two grayscale image labeling models are consistent, which means that the number of graphic label data labeled by the at least two grayscale image labeling models is the same, and the position and size are within a preset error range. By using the label data of each grayscale image and the corresponding color image to construct two or more grayscale image labeling models, the labeling results of each infrared image by using the two or more grayscale image labeling models can be corrected, and the labeling result in the case where the labeling results are consistent is taken as the first pseudo label of the corresponding infrared image, so that the pseudo label data of the high-quality infrared image can be selected.

[0044] In addition, the labeling results of each infrared image by using the two or more grayscale image labeling models can also be corrected based on other strategies, for example, in a case where the labeling results of grayscale image labeling models satisfying a preset proportion are consistent, the labeling result of the grayscale image labeling model satisfying the preset proportion is taken as the first pseudo label of the corresponding infrared image, and the like, which will not be enumerated here.

[0045] S120, migrating the label data of each color image to the corresponding infrared image according to the same point correspondence relationship from the grayscale image converted from each color image to the corresponding infrared image, to obtain the second pseudo label of each infrared image.

[0046] In the embodiments of the present disclosure, the step S110 and the step S120 do not have a strict time sequence relationship.

[0047] Since the gray-scale image converted from the color image is close to the image content contained in the corresponding infrared image, there are usually many homonymic points in the gray-scale image and the infrared image corresponding to the same color image. Among them, the homonymic points can be considered as pixel points representing the same object point in at least two images.

[0048] In this embodiment, the pixel position transformation relationship of the homonymic points from the gray-scale image converted from the color image to the corresponding infrared image can be determined based on an existing feature-based image matching method, and is taken as the homonymic point correspondence. The feature-based image matching method includes, for example, an image matching method based on deep learning feature extraction, and also includes, for example, an image matching method based on traditional feature extraction. The traditional feature extraction method includes, for example, Scale-Invariant Feature Transform (SIFT) and the like.

[0049] The homonymic point correspondence from the gray-scale image converted from each color image to the corresponding infrared image can be taken as the image pattern correspondence between each color image and the corresponding infrared image. Furthermore, based on the homonymic point correspondence, the label data of each color image can be mapped into the corresponding infrared image to correct the label deviation existing in the color image and the infrared image, so as to obtain a more accurate second pseudo label of each infrared image.

[0050] S130, determining the label data of each infrared image according to the first pseudo label and the second pseudo label.

[0051] In the embodiments of the present disclosure, the first pseudo label labeled by the gray-scale image labeling model and the second pseudo label obtained by label data migration based on image matching can be corrected with each other to select a high-quality labeling result of the infrared image.

[0052] For example, in some optional implementations, determining the label data of each infrared image according to the first pseudo label and the second pseudo label can include: in the case where the first pseudo label is consistent with the second pseudo label, taking any label in the first pseudo label and the second pseudo label as the label data of the corresponding infrared image.

[0053] In these optional implementations, the consistency of the first pseudo label and the second pseudo label can be considered as that the number of pattern label data in the first pseudo label and the second pseudo label is the same, and the position and size are within a preset error range. If the first pseudo label is consistent with the second pseudo label, it can be considered that the label data of the infrared image with higher accuracy is obtained. At this time, any pseudo label in the first pseudo label and the second pseudo label can be taken as the label data of the corresponding infrared image.

[0054] In addition, the label data of the infrared image can also be determined based on other strategies using the first pseudo label and the second pseudo label. For example, the first pseudo label is the main label data, the second pseudo label is the label data for auxiliary correction, in the case where the graphic label data in the first pseudo label is consistent with at least part of the graphic label data in the second pseudo label, the first pseudo label is taken as the label data of the corresponding infrared image, and the like, which are not listed here.

[0055] The technical scheme of the embodiment of the present disclosure labels each infrared image by using a grayscale image labeling model to obtain the first pseudo label of each infrared image; wherein the grayscale image labeling model is constructed based on the grayscale images converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a pair of images; the label data of each color image is migrated to the corresponding infrared image according to the correspondence relationship between the same name points from the grayscale image converted from each color image to the corresponding infrared image to obtain the second pseudo label of each infrared image; and the label data of each infrared image is determined according to the first pseudo label and the second pseudo label.

[0056] For the pair of color images and infrared images, the grayscale image converted from the color image has similarity with the corresponding infrared image. On this basis, by converting the color image into a grayscale image and constructing a grayscale image labeling model using the label data of the grayscale image and the color image, the trained grayscale image labeling model can realize relatively accurate labeling of the infrared image to obtain the first pseudo label. In addition, according to the image matching relationship (i.e. the correspondence relationship between the same name points) from the color image to the infrared image, the label data of the color image can also be migrated to the corresponding infrared image to obtain a relatively accurate second pseudo label. Finally, the first pseudo label and the second pseudo label can be combined to correct each other to obtain high-quality label data of each infrared image. By completing the automatic labeling of the infrared image based on the label data of the color image, not only can the labeling efficiency, the labeling period and the labeling cost be improved, but also the labeling quality can be guaranteed.

[0057] The embodiment of the present disclosure can be combined with each optional scheme in the image labeling method provided in the above-mentioned embodiments. The image labeling method provided in the present embodiment describes the labeling process of the label data of the color image in detail, which can complete the labeling of the remaining color images using a small amount of true label data of the color images, and can further improve the labeling efficiency, shorten the labeling period and reduce the labeling cost.

[0058] Figure 2 A flowchart of the labeling process of the label data of each color image in an image labeling method provided by the embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the labeling process of the label data of each color image in the image labeling method provided by the present embodiment can include: Figure 2 ​

[0059] S210, constructing at least two color image annotation models according to the color images with true label data in each color image.

[0060] In this embodiment, part of the images can be sampled from each color image, and the true label data of these sampled images can be obtained by manual annotation. Furthermore, at least two color image annotation models can be constructed based on the sampled images and their true label data. The at least two color image annotation models can also include existing image processing large models, such as Cascade R-CNN model, SOFT-teacher model, etc., which are not exhaustively listed here.

[0061] S220, labeling the remaining color images without true label data in each color image by using the constructed at least two color image annotation models.

[0062] The two color image annotation models can be used to pseudo-label the remaining color images without manual annotation (i.e., without true label data) in each color image.

[0063] S230, determining the label data of the remaining color images based on the labeling results of the at least two color image annotation models.

[0064] The labeling results of different color image annotation models can be used to correct each other, and the label data of high-quality color images can be selected. In some implementations, determining the label data of the remaining color images based on the labeling results of the at least two color image annotation models can include: in the case that the labeling results of the two color image annotation models are consistent, the labeling results are taken as the label data of the remaining color images. Or, it can also include: in the case that the labeling results of a color image annotation model satisfying a preset proportion are consistent, the labeling results of the color image model satisfying the preset proportion are taken as the label data of the remaining color images, etc., which are not exhaustively listed here.

[0065] Exemplarily, Figure 3 A flowchart of an image annotation method provided by an embodiment of the present disclosure.

[0066] Referring to Figure 3 , the flow of the image annotation method can include:

[0067] S310, video frame extraction is performed on the color video and infrared video captured by the camera to obtain pairs of color images and infrared images;

[0068] S320, part of the color images are manually annotated to obtain their true label data;

[0069] S330, at least two color image annotation models are constructed according to the color images with true label data.

[0070] S340: annotating the remaining color images without true label data among the color images by using the at least two color image annotation models constructed, and correcting each other based on the annotation results to obtain label data for each color image;

[0071] S350, converting the color image into a grayscale image, and constructing at least two grayscale image annotation models using the label data of each grayscale image and the color image;

[0072] S360: annotate each infrared image using at least two grayscale image annotation models, and mutually correct the annotation results to obtain a first pseudo label for each infrared image;

[0073] S370, migrating the label data of each color image to the corresponding infrared image based on the correspondence between the grayscale images converted from each color image and the corresponding infrared image, to obtain a second pseudo label for each infrared image;

[0074] S380: Combining the first pseudo label marked by the grayscale image annotation model and the second pseudo label migrated based on the image matching label data to correct each other and output high-quality label data of the infrared image.

[0075] Furthermore, during the image annotation process, unlabeled infrared and color images can be manually annotated to obtain their true label data. Experimental verification shows that only 20%-30% of the color images need to be annotated to complete the labeling of the remaining color images and unlabeled infrared images. This significantly reduces the annotation cost and shortens the annotation cycle, while also enabling the acquisition of a large amount of high-quality pseudo-labeled data. Furthermore, the percentage of unlabeled infrared images can be reduced to approximately 10%, enabling rapid response to algorithm development and optimization work on infrared images.

[0076] The technical solution of the embodiment of the present disclosure provides a detailed description of the labeling process of the label data of the color image. It can use a small amount of true label data of the color image to complete the labeling of the remaining color images, which can further improve the labeling efficiency, shorten the labeling cycle, and reduce the labeling cost. The image labeling method provided by the embodiment of the present disclosure and the image labeling method provided by the above embodiment belong to the same disclosed concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0077] The embodiments of the present disclosure can be combined with the optional schemes of the image labeling method provided in the above embodiments. The image labeling method provided in the present embodiment describes in detail the determination process of the corresponding relationship of the homonymous points, can determine the affine transformation matrix of the homonymous points from the gray image converted from the color image to the infrared image, and take the affine transformation matrix as the corresponding relationship of the homonymous points, so as to realize the label migration from the color image to the infrared image.

[0078] Figure 4 FIG. 1 is a flowchart of the determination process of the corresponding relationship of the homonymous points in the image labeling method provided in the present disclosure. As shown in FIG. 1, the determination process of the corresponding relationship of the homonymous points in the image labeling method provided in the present embodiment can include: Figure 4

[0079] S410, extracting a first feature descriptor in each gray image converted from a color image and a second feature descriptor in a corresponding infrared image.

[0080] In the present embodiment, each color image can be converted into a gray image based on an existing conversion method. The first feature descriptor of the gray image and the second feature descriptor of the corresponding infrared image can be obtained based on a deep learning model. The first feature descriptor and the second feature descriptor can be considered as feature points of objects in the image, such as corner points.

[0081] S420, matching the first feature descriptor and the second feature descriptor, and taking the matched first feature descriptor and the second feature descriptor as homonymous points.

[0082] In the present embodiment, the matching of the first feature descriptor and the second feature descriptor can be performed based on an existing matching method (for example, based on the Open CV tool). The matched first feature descriptor and the second feature descriptor can be considered as the same object points (i.e., homonymous points) in the gray image and the color image.

[0083] S430, determining an affine transformation matrix from each color image to the corresponding infrared image based on the pixel coordinates of the homonymous points in the color image and the corresponding infrared image.

[0084] ​In this embodiment, the pixel coordinates of the first feature descriptor in the grayscale image are the same as the pixel coordinates of the first feature descriptor in the color image. The pixel coordinates of each matched first feature descriptor and second feature descriptor on the image to which they belong can be obtained as the pixel coordinates of each corresponding point in the color image and the corresponding infrared image, respectively. Furthermore, the affine transformation matrix of the color image to the infrared image can be determined based on an existing solving algorithm (for example, based on the Open CV tool). According to the affine transformation matrix, the label data of the color image can be mapped to the infrared image. For example, the pixel positions of each point in the label data of the color image can be operated with the affine transformation matrix to obtain the pixel positions of each corresponding point in the infrared image, and thus the label data of the infrared image can be obtained.

[0085] In some optional implementations, matching the first feature descriptors and the second feature descriptors can include: dividing each color image and the corresponding infrared image into image blocks in the same manner; and matching the first feature descriptors in each image block of each color image with the second feature descriptors in the same image block of the corresponding infrared image.

[0086] For example, Figure 5 An exemplary schematic diagram of feature sub-matching in an image labeling method provided by an embodiment of the present disclosure. Referring to Figure 5 Each color image and the corresponding infrared image can be divided into 2x2 image blocks to obtain image block 1-image block 4 of the color image and image block 1'-image block 4' of the infrared image; wherein image block 1 corresponds to image block 1', image block 2 corresponds to image block 2', image block 3 corresponds to image block 3', and image block 4 corresponds to image block 4'. The matching of the first feature descriptors and the second feature descriptors can be performed in the corresponding image blocks.

[0087] In these optional implementations, by dividing each color image and the corresponding infrared image into image blocks in the same manner and matching the first feature descriptors and the second feature descriptors in the corresponding image blocks, it can be determined that there is a corresponding point in each image block, thereby avoiding the concentration of corresponding points in part of the color image and the infrared image, improving the accuracy of the affine transformation matrix, and facilitating the accurate migration of the label data of the color image to the infrared image.

[0088] The technical solution of the embodiment of the present disclosure is described in detail, which can determine the affine transformation matrix of the homonymous points from the color image converted gray image to the infrared image, and take it as the corresponding relationship of the homonymous points, so as to realize the label migration from the color image to the infrared image. The image labeling method provided by the embodiment of the present disclosure belongs to the same disclosure concept as the image labeling method provided by the above embodiment. The technical details not described in detail in the present embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in the present embodiment and the above embodiment.

[0089] Figure 6 The structure diagram of an image labeling device provided by the embodiment of the present disclosure is shown. The image labeling device provided by the embodiment is suitable for the case of labeling the paired color image and infrared image.

[0090] As shown in Figure 6 The image labeling device provided by the embodiment of the present disclosure can include:

[0091] The model labeling module 610 is configured to label each infrared image by using a gray image labeling model to obtain the first pseudo label of each infrared image. The gray image labeling model is constructed based on the gray image converted from each color image and the label data of each color image. Each infrared image and each color image form a pair of images.

[0092] The label migration module 620 is configured to migrate the label data of each color image to the corresponding infrared image according to the homonymous point correspondence from the gray image converted from each color image to the corresponding infrared image, to obtain the second pseudo label of each infrared image.

[0093] The label determination module 630 is configured to determine the label data of each infrared image according to the first pseudo label and the second pseudo label.

[0094] In some optional implementations, the model labeling module can be configured to:

[0095] Label each infrared image by using at least two gray image labeling models.

[0096] In the case where the labeling results of the two gray image labeling models are consistent, the labeling result is taken as the first pseudo label of the corresponding infrared image.

[0097] In some optional implementations, the image labeling device can further include:

[0098] The color image labeling module is configured to label the label data of each color image based on the following process:

[0099] constructing at least two color image annotation models according to the color images with true label data in each color image;

[0100] annotating the remaining color images without true label data in each color image through the constructed at least two color image annotation models;

[0101] determining the label data of the remaining color images based on the annotation results of the at least two color image annotation models.

[0102] In some optional implementation manners, the image annotation apparatus can further include:

[0103] a corresponding relationship determination module configured to determine the corresponding relationship of the homonymous points based on the following process:

[0104] extracting the first feature descriptors in the converted gray images of each color image and the second feature descriptors in the corresponding infrared images;

[0105] matching the first feature descriptors and the second feature descriptors, and taking the matched first feature descriptors and second feature descriptors as the homonymous points;

[0106] determining the affine transformation matrix from each color image to the corresponding infrared image based on the pixel coordinates of the homonymous points in the color image and the corresponding infrared image.

[0107] In some optional implementation manners, the corresponding relationship determination module can be configured to:

[0108] performing image block division on each color image and the corresponding infrared image in the same manner;

[0109] matching the first feature descriptors in each image block of each color image with the second feature descriptors in the same image block of the corresponding infrared image.

[0110] In some optional implementation manners, the label determination module can be configured to:

[0111] in the case where the first pseudo label is consistent with the second pseudo label, taking any label in the first pseudo label and the second pseudo label as the label data of the corresponding infrared image.

[0112] In some optional implementation manners, the application is applied to the image detection field; wherein the label data of the infrared image includes the detection frame data of the target object.

[0113] The image annotation apparatus provided by the embodiments of the present disclosure can execute the image annotation method provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0114] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0115] Reference below Figure 7 , which shows an electronic device (eg Figure 7 The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0116] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0117] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0118] In particular, according to embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from the network through the communication apparatus 709, or installed from the storage apparatus 708, or installed from the ROM 702. When the computer program is executed by the processing apparatus 701, the above-mentioned functions defined in the image labeling method of the embodiments of the present disclosure are executed.

[0119] The electronic device provided by the embodiments of the present disclosure belongs to the same disclosure concept as the image labeling method provided by the above-mentioned embodiments, and the technical details not described in detail in the present embodiment can be referred to the above-mentioned embodiments, and the present embodiment has the same beneficial effects as the above-mentioned embodiments.

[0120] The embodiments of the present disclosure provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the image labeling method provided by the above-mentioned embodiments.

[0121] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take any of a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including, but not limited to, wire, cable, RF (radio frequency), or the like, or any suitable combination of the foregoing.

[0122] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0123] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device described above.

[0124] The computer-readable medium described above carries one or more programs that, when executed by the electronic device described above, cause the electronic device to:

[0125] The first pseudo label of each infrared image is obtained by labeling each infrared image through a grayscale image labeling model, wherein the grayscale image labeling model is constructed based on the grayscale images converted from the color images and the label data of the color images; wherein each infrared image and each color image constitute a pair of images; the label data of each color image is migrated to the corresponding infrared image according to the correspondence between the same points of the grayscale images converted from the color images and the corresponding infrared images, to obtain the second pseudo label of each infrared image; and the label data of each infrared image is determined according to the first pseudo label and the second pseudo label.

[0126] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ as well as conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0127] The flow and block diagrams in the drawings show architectural, functional, and operational representations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0128] The units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the names of the units and modules do not constitute a limitation on the units and modules themselves.

[0129] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that can be used include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), System on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0130] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] According to one or more embodiments of the present disclosure, an image labeling method is provided, the method comprising:

[0132] labeling each infrared image by a grayscale image labeling model to obtain first pseudo labels of the infrared images; wherein the grayscale image labeling model is constructed based on grayscale images converted from the color images and label data of the color images; wherein the infrared images and the color images constitute pairs of images;

[0133] migrating the label data of the color images to corresponding infrared images according to a same-name point correspondence relationship from grayscale images converted from the color images to corresponding infrared images, to obtain second pseudo labels of the infrared images;

[0134] determining label data of the infrared images according to the first pseudo labels and the second pseudo labels.

[0135] According to one or more embodiments of the present disclosure, an image labeling method is provided, further comprising:

[0136] In some optional implementations, the grayscale image labeling model is used to label each infrared image to obtain a first pseudo label of the infrared image, including:

[0137] The at least two grayscale image labeling models are used to label each infrared image.

[0138] When the labeling results of the two grayscale image labeling models are consistent, the labeling result is taken as the first pseudo label of the corresponding infrared image.

[0139] According to one or more embodiments of the present disclosure, an image labeling method is provided, further comprising:

[0140] In some optional implementations, the labeling process of the label data of each color image includes:

[0141] At least two color image labeling models are constructed according to the color images with true label data.

[0142] The remaining color images without true label data are labeled by using the constructed at least two color image labeling models.

[0143] Based on the labeling results of the at least two color image labeling models, the label data of the remaining color images is determined.

[0144] According to one or more embodiments of the present disclosure, an image labeling method is provided, further comprising:

[0145] In some optional implementations, the determination process of the corresponding relationship of the same name points includes:

[0146] The first feature descriptor in the grayscale image converted from the color image and the second feature descriptor in the corresponding infrared image are extracted.

[0147] The first feature descriptor and the second feature descriptor are matched, and the matched first feature descriptor and the second feature descriptor are taken as the same name points.

[0148] Based on the pixel coordinates of the same name points in the color image and the corresponding infrared image respectively, an affine transformation matrix from the color image to the corresponding infrared image is determined.

[0149] According to one or more embodiments of the present disclosure, an image labeling method is provided, further comprising:

[0150] In some optional implementations, the matching the first feature descriptor and the second feature descriptor comprises:

[0151] The color images and the corresponding infrared images are divided into image blocks in the same manner.

[0152] The first feature descriptor in each image block of the color images is matched with the second feature descriptor in the same image block of the corresponding infrared images.

[0153] According to one or more embodiments of the present disclosure, an image labeling method is provided, further comprising:

[0154] In some optional implementations, the determining the label data of the infrared images according to the first pseudo label and the second pseudo label comprises:

[0155] In a case where the first pseudo label is consistent with the second pseudo label, any one of the first pseudo label and the second pseudo label is taken as the label data of the corresponding infrared image.

[0156] According to one or more embodiments of the present disclosure, an image labeling method is provided, further comprising:

[0157] In some optional implementations, the method is applied to the field of image detection; and the label data of the infrared images comprises detection frame data of a target object.

[0158] According to one or more embodiments of the present disclosure, an image labeling device is provided, which comprises:

[0159] A model labeling module is configured to label each infrared image by using a grayscale image labeling model to obtain a first pseudo label of the infrared image; the grayscale image labeling model is constructed based on grayscale images converted from color images and label data of the color images; the infrared images and the color images form paired images.

[0160] A label migration module is configured to migrate the label data of the color images to corresponding infrared images according to a same-name point correspondence relationship from grayscale images converted from the color images to the corresponding infrared images, to obtain a second pseudo label of the infrared image.

[0161] A label determining module is configured to determine label data of the infrared images according to the first pseudo label and the second pseudo label.

[0162] The above description merely illustrates the preferred embodiments of the disclosure and a principle for applying the technologies. It is understood by those skilled in the art that the disclosed scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be formed.

[0163] Further, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are included for the purpose of providing a thorough disclosure, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0164] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. An image annotation method, characterized in that: include: Annotating each infrared image using a grayscale annotation model to obtain a first pseudo label for each infrared image; wherein the grayscale annotation model is constructed based on grayscale images converted from each color image and label data of each color image; wherein each infrared image and each color image constitute a paired image; Migrating the label data of each color image to the corresponding infrared image based on the correspondence between the grayscale images converted from the color images and the corresponding infrared images to obtain a second pseudo label for each infrared image; The label data of each infrared image is determined according to the first pseudo label and the second pseudo label.

2. The method according to claim 1, characterized in that The step of labeling each infrared image using a grayscale image labeling model to obtain a first pseudo label for each infrared image includes: Annotating each of the infrared images using at least two grayscale annotation models; When the annotation results of the two grayscale image annotation models are consistent, the annotation results are used as the first pseudo label of the corresponding infrared image.

3. The method according to claim 1, characterized in that The labeling process of the label data of each color image includes: Constructing at least two color image annotation models according to the color images having true label data among the color images; Annotating the remaining color images without true label data among the color images by constructing the at least two color image annotation models; Based on the annotation results of the at least two color image annotation models, label data of the remaining color images are determined.

4. The method according to claim 1, wherein The process of determining the correspondence relationship of the homonymous points includes: Extracting a first feature descriptor from the grayscale image converted from each color image and a second feature descriptor from the corresponding infrared image; Matching the first feature descriptor and the second feature descriptor, and taking the matched first feature descriptor and the second feature descriptor as points of the same name; Based on the pixel coordinates of the points of the same name in the color image and the corresponding infrared image, an affine transformation matrix from the color image to the corresponding infrared image is determined.

5. The method according to claim 4, characterized in that The matching the first feature descriptor and the second feature descriptor includes: Dividing each color image and the corresponding infrared image into image blocks in the same manner; The first feature descriptor in each image block of each color image is matched with the second feature descriptor of the same image block in the corresponding infrared image.

6. The method according to claim 1, characterized in that The determining of the label data of each infrared image according to the first pseudo label and the second pseudo label includes: In the case that the first pseudo label is consistent with the second pseudo label, any one of the first pseudo label and the second pseudo label is used as label data of the corresponding infrared image.

7. The method according to any one of claims 1 to 6, characterized in that: Applied to the field of image detection; wherein, the label data of the infrared image includes detection frame data of the target object.

8. An image annotation device, characterized in that: include: a model annotation module, configured to annotate each infrared image using a grayscale annotation model to obtain a first pseudo label for each infrared image; wherein the grayscale annotation model is constructed based on grayscale images converted from each color image and label data of each color image; wherein each infrared image and each color image constitute a paired image; a label migration module for migrating the label data of each color image to the corresponding infrared image based on the correspondence between the grayscale images converted from the color images and the corresponding infrared images, thereby obtaining a second pseudo label for each infrared image; The label determination module is configured to determine label data of each infrared image according to the first pseudo label and the second pseudo label.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image annotation method according to any one of claims 1 to 7.

10. A storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform the image annotation method according to any one of claims 1 to 7.

Images