Image labeling method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image annotation method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the annotation of each infrared image through a gray-scale map annotation model, and obtaining a first pseudo tag of each infrared image; wherein the grey-scale map labeling model is constructed based on a grey-scale image converted from each color image and label data of each color image; wherein each infrared image and each color image form a paired image; according to a homonymy point corresponding relation from the gray scale image converted from each color image to the corresponding infrared image, migrating the label data of each color image to the corresponding infrared image to obtain a second pseudo label of each infrared image; and determining label data of each infrared image according to the first pseudo label and the second pseudo label. The labeling efficiency can be improved, the labeling period is shortened, and the labeling cost is reduced.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to an image annotation method, device, electronic device, and storage medium. Background Art

[0002] At present, in some scenarios, it is necessary to set up image acquisition devices that can collect color images (Red Green Blue, RGB) and image acquisition devices that can collect infrared images (Infrared Radiation, IR) at the same time, so it is necessary to deploy algorithm capabilities for RGB images and IR images at the same time. In order to provide algorithm capabilities, a large number of RGB images and IR images need to be annotated at the same time. This will lead to a long annotation cycle and high annotation costs. Summary of the invention

[0003] The embodiments of the present disclosure provide an image annotation method, device, electronic device, and storage medium, which can improve annotation efficiency, shorten annotation cycle, and reduce annotation cost.

[0004] In a first aspect, an embodiment of the present disclosure provides an image annotation method, comprising:

[0005] Annotating each infrared image through a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on a grayscale image converted from each color image and label data of each color image; wherein each infrared image and each color image constitute a paired image;

[0006] According to the correspondence relationship between the grayscale images converted from the color images and the corresponding infrared images, the label data of the color images are transferred to the corresponding infrared images to obtain the second pseudo labels of the infrared images;

[0007] The label data of each infrared image is determined according to the first pseudo label and the second pseudo label.

[0008] In a second aspect, the present disclosure also provides an image annotation device, including:

[0009] A model annotation module, used for annotating each infrared image by a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image;

[0010] A label migration module, used for migrating the label data of each color image to the corresponding infrared image according to the correspondence relationship of the same-name points between the grayscale image converted from each color image and the corresponding infrared image, so as to obtain a second pseudo label of each infrared image;

[0011] The label determination module is used to determine the label data of each infrared image according to the first pseudo label and the second pseudo label.

[0012] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the image annotation method as described in any one of the embodiments of the present disclosure.

[0016] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the image annotation method as described in any one of the embodiments of the present disclosure.

[0017] The technical solution of the embodiment of the present disclosure is to annotate each infrared image through a grayscale image annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale image annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; according to the correspondence relationship of the same-name points from the grayscale image converted from each color image to the corresponding infrared image, the label data of each color image is migrated to the corresponding infrared image to obtain a second pseudo label of each infrared image; according to the first pseudo label and the second pseudo label, the label data of each infrared image is determined.

[0018] For a pair of color images and infrared images, the grayscale image converted from the color image has similarity with the corresponding infrared image. On this basis, by converting the color image into a grayscale image and using the label data of the grayscale image and the color image to construct a grayscale image annotation model, the trained grayscale image annotation model can achieve a relatively accurate annotation of the infrared image and obtain a first pseudo-label. In addition, according to the image matching relationship between the color image and the infrared image (i.e., the corresponding relationship of the same name points), the label data of the color image is migrated to the corresponding infrared image, and a relatively accurate second pseudo-label can also be obtained. Finally, the first pseudo-label and the second pseudo-label can be combined to correct each other to obtain high-quality label data for each infrared image. By completing the automatic annotation of infrared images based on the label data of the color image, not only can the annotation efficiency be improved, the annotation cycle be shortened, and the annotation cost be reduced, but also the annotation quality can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart of an image annotation method provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of a flow chart of a process of labeling label data of each color image in an image labeling method provided in an embodiment of the present disclosure;

[0022] Figure 3 A flowchart of an image annotation method provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a process of determining a correspondence relationship between points of the same name in an image annotation method provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of feature sub-matching in an image annotation method provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of an image annotation device provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] Figure 1 The present invention is a flowchart of an image annotation method provided by an embodiment of the present invention. The present invention is applicable to the case of labeling a pair of color images and infrared images. The method can be performed by an image annotation device, which can be implemented in the form of software and / or hardware, and can be configured in an electronic device, such as a computer.

[0034] like Figure 1 As shown, the image annotation method provided in this embodiment may include:

[0035] S110 , annotating each infrared image using a grayscale image annotation model to obtain a first pseudo label for each infrared image.

[0036] In the disclosed embodiment, the grayscale image annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image. The color image can be understood as a true color image, i.e., a (Red Green Blue, RGB) image.

[0037] In application scenarios such as vehicle-mounted cameras and security cameras, cameras can integrate color image and infrared image (IR) acquisition devices to achieve full-time, multi-detail image acquisition. Among them, the color image and infrared image acquisition devices can be considered to be integrated in the camera at close locations, and the acquisition timestamps are basically aligned, so each color image and each infrared image acquired by the camera can be considered as a pair of images.

[0038] In this embodiment, the label data of the color image and the label data of the infrared image can be considered as graphic label data in the image, such as object outline frame label data, rectangular frame label data, etc. For example, in some implementations, the image annotation method can be applied to the field of image detection; wherein the label data of the color image and the infrared image can include the detection frame data of the target object. Thus, the detection frame labeling of the image can be realized, laying a foundation for the research and development and optimization of the image detection algorithm of the color image and the infrared image.

[0039] Since color images are less difficult to annotate, we can first annotate color images to obtain their label data. The label data of color images can include manually annotated true label data and high-quality pseudo label data. Since the integration position and acquisition timestamp of the acquisition devices of color images and infrared images are slightly offset, the paired color images and infrared images are also different, and the label data of color images cannot usually be migrated to infrared images intact.

[0040] Since the grayscale image converted from the color image has similarities with the corresponding infrared image, in the embodiment of the present disclosure, each color image can first be converted into a grayscale image based on an existing conversion method. For example, the three-channel components of the color image can be weighted averaged to convert it into a grayscale image. Then, at least one grayscale image annotation model can be constructed using the label data of each grayscale image and color image. Finally, pseudo-labeling can be performed on each unlabeled infrared image based on the constructed grayscale image model to obtain the first pseudo-label of each infrared image.

[0041] Among them, the grayscale image annotation model may include an existing large image processing model. Exemplarily, the grayscale image annotation model may include a detection algorithm model with relatively good current results, such as the Cascade Region Convolutional Neural Network (Cascade R-CNN) with Vision Transformer (ViT) as the backbone network; it may also include a semi-supervised detection algorithm model, such as a teacher model (SOFT-teacher). The processing results of these existing large image processing models are very accurate and can meet the high quality requirements of pseudo-label marking.

[0042] In some optional implementations, annotating each infrared image through a grayscale annotation model to obtain a first pseudo label for each infrared image may include: annotating each infrared image through at least two grayscale annotation models; when the annotation results of the two grayscale annotation models are consistent, using the annotation result as the first pseudo label of the corresponding infrared image.

[0043] In these optional implementations, the labeling results of at least two grayscale image labeling models are consistent, which can be considered that the number of graphic label data labeled by at least two grayscale image labeling models is the same, and the position and size are within the preset error range. By using the label data of each grayscale image and the corresponding color image, more than two grayscale image labeling models are constructed, and the labeling results of each infrared image can be mutually corrected by using the more than two grayscale image labeling models. The labeling results when the labeling results are consistent are used as the first pseudo-label of the corresponding infrared image, so that high-quality pseudo-label data of the infrared image can be selected.

[0044] In addition, based on other strategies, more than two grayscale annotation models can be used to correct the labeling results of each infrared image. For example, when the labeling results of the grayscale annotation model that meets the preset ratio are consistent, the labeling result of the grayscale image model that meets the preset ratio can be used as the first pseudo-label of the corresponding infrared image, and so on. The examples are not exhaustive here.

[0045] S120 , according to the correspondence relationship between the like-name points of the grayscale images converted from the color images and the corresponding infrared images, migrate the label data of each color image to the corresponding infrared image to obtain a second pseudo label of each infrared image.

[0046] In the disclosed embodiment, step S110 and step S120 do not have a strict timing relationship.

[0047] Since the grayscale image converted from a color image has similar image contents to the corresponding infrared image, there are usually many points of the same name in the grayscale image and infrared image corresponding to the same color image. The points of the same name can be considered as pixels representing the same object point in at least two images.

[0048] In this embodiment, based on the existing feature-based image matching method, the pixel position transformation relationship between the grayscale image converted from the color image and the same-name points on the infrared image corresponding to the color image can be determined and used as the same-name point correspondence relationship. Among them, the feature-based image matching method includes, for example, an image matching method based on deep learning feature extraction, and an image graphic matching method based on traditional feature extraction. Among them, the traditional feature extraction method includes, for example, Scale-Invariant Feature Transform (SIFT) and the like.

[0049] The correspondence between the same-name points of the grayscale image converted from each color image and the corresponding infrared image can be used as the graphic correspondence between the color images and the corresponding infrared image. Furthermore, based on the same-name point correspondence, the label data of each color image can be mapped to the corresponding infrared image to correct the label deviation between the color image and the infrared image, and obtain a more accurate second pseudo label for each infrared image.

[0050] S130 : Determine label data of each infrared image according to the first pseudo label and the second pseudo label.

[0051] In the disclosed embodiment, the first pseudo-label marked by the grayscale image annotation model and the second pseudo-label migrated based on the image matching label data can be combined to correct each other and select a high-quality infrared image annotation result.

[0052] For example, in some optional implementations, determining the label data of each infrared image based on the first pseudo-label and the second pseudo-label may include: when the first pseudo-label is consistent with the second pseudo-label, using any label of the first pseudo-label and the second pseudo-label as the label data of the corresponding infrared image.

[0053] In these optional implementations, the first pseudo-label and the second pseudo-label are consistent, which can be considered to be the same number of graphic label data in the first pseudo-label and the second pseudo-label, and the position and size are within the preset error range. If the first pseudo-label is consistent with the second pseudo-label, it can be considered that the label data of the infrared image with higher accuracy is obtained. At this time, any pseudo-label of the first pseudo-label and the second pseudo-label can be used as the label data of the corresponding infrared image.

[0054] In addition, the label data of the infrared image can be determined using the first pseudo label and the second pseudo label based on other strategies. For example, the first pseudo label is used as the main label data, the second pseudo label is used as the auxiliary corrected label data, and when the graphic label data in the first pseudo label is consistent with at least part of the graphic label data in the second pseudo label, the first pseudo label is used as the label data of the corresponding infrared image, etc., which are not exhaustive here.

[0055] The technical solution of the disclosed embodiment is to annotate each infrared image through a grayscale image annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale image annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; according to the correspondence relationship of the same-name points from the grayscale image converted from each color image to the corresponding infrared image, the label data of each color image is migrated to the corresponding infrared image to obtain a second pseudo label of each infrared image; according to the first pseudo label and the second pseudo label, the label data of each infrared image is determined.

[0056] For a pair of color images and infrared images, the grayscale image converted from the color image has similarity with the corresponding infrared image. On this basis, by converting the color image into a grayscale image and using the label data of the grayscale image and the color image to construct a grayscale image annotation model, the trained grayscale image annotation model can achieve a relatively accurate annotation of the infrared image and obtain a first pseudo-label. In addition, according to the image matching relationship between the color image and the infrared image (i.e., the corresponding relationship of the same name points), the label data of the color image is migrated to the corresponding infrared image, and a relatively accurate second pseudo-label can also be obtained. Finally, the first pseudo-label and the second pseudo-label can be combined to correct each other to obtain high-quality label data for each infrared image. By completing the automatic annotation of infrared images based on the label data of the color image, not only can the annotation efficiency be improved, the annotation cycle be shortened, and the annotation cost be reduced, but also the annotation quality can be guaranteed.

[0057] The various optional solutions in the image annotation method provided in the embodiment of the present disclosure and the above embodiment can be combined. The image annotation method provided in this embodiment describes the annotation process of the label data of the color image in detail, and can use a small amount of true label data of the color image to complete the annotation of the remaining color images, which can further improve the annotation efficiency, shorten the annotation cycle, and reduce the annotation cost.

[0058] Figure 2 The figure is a flow chart of the labeling process of the label data of each color image in an image labeling method provided by an embodiment of the present disclosure. Figure 2 As shown, in the image annotation method provided in this embodiment, the annotation process of the label data of each color image may include:

[0059] S210. Construct at least two color image annotation models according to the color images with true label data in each color image.

[0060] In this embodiment, partial images can be sampled from each color image, and the true label data of these sampled images can be obtained by manual annotation. Furthermore, at least two color map annotation models can be constructed based on these sampled images and their true label data. Among them, at least two color map annotation models can also include existing large image processing models, such as Cascade R-CNN model, SOFT-teacher model, etc., which are not exhaustive here.

[0061] S220: annotate the remaining color images that do not have true label data in each color image by constructing at least two color image annotation models.

[0062] Among them, two color image annotation models can be used to perform pseudo-labeling on the remaining color images that are not manually annotated (ie, do not have true label data) in each color image.

[0063] S230: Determine label data of the remaining color images based on the annotation results of at least two color image annotation models.

[0064] Among them, the annotation results of different color image annotation models can be used to make mutual corrections to select high-quality label data of color images. In some implementations, based on the annotation results of at least two color image annotation models, the label data of the remaining color images is determined, which can include: when the annotation results of the two color image annotation models are consistent, the annotation results are used as the label data of the remaining color images. Alternatively, it can also include: when the annotation results of the color image annotation models that meet the preset ratio are consistent, the annotation results of the color image model that meets the preset ratio are used as the label data of the remaining color images, etc., which are not exhaustive here.

[0065] For example, Figure 3 A flowchart of an image annotation method provided by an embodiment of the present disclosure.

[0066] See also Figure 3 ,The process of the image annotation method may include:

[0067] S310, extracting frames from the color video and infrared video captured by the camera to obtain pairs of color images and infrared images;

[0068] S320, manually labeling part of the color image to obtain its true label data;

[0069] S330, constructing at least two color image annotation models according to the color image with true label data;

[0070] S340, annotating the remaining color images without true label data in each color image by constructing at least two color image annotation models, and correcting each other based on the annotation results to obtain label data of each color image;

[0071] S350, converting the color image into a grayscale image, and constructing at least two grayscale image annotation models using the label data of each grayscale image and color image;

[0072] S360, annotating each infrared image using at least two grayscale image annotation models, and correcting each other based on the annotation results to obtain a first pseudo label for each infrared image;

[0073] S370, according to the correspondence relationship between the grayscale images converted from the color images and the corresponding infrared images, migrate the label data of each color image to the corresponding infrared image to obtain a second pseudo label of each infrared image;

[0074] S380, combining the first pseudo label marked by the grayscale image annotation model and the second pseudo label migrated based on the image matching label data to correct each other, and outputting high-quality label data of the infrared image.

[0075] In addition, in the above image annotation process, the unannotated infrared images and color images can be manually annotated to obtain their true label data. Experimental verification shows that only 20%-30% of the color images need to be annotated to achieve labeling of the remaining color images and unannotated infrared images, which greatly reduces the annotation cost and shortens the annotation cycle, while being able to obtain a large amount of high-quality pseudo-labeled annotation data. In addition, the number of unannotated infrared images can be reduced to about 10%, which can quickly respond to the research and development and optimization of algorithms on infrared images.

[0076] The technical solution of the disclosed embodiment describes in detail the labeling process of the label data of the color image, and can use a small amount of true label data of the color image to complete the labeling of the remaining color images, which can further improve the labeling efficiency, shorten the labeling cycle, and reduce the labeling cost. The image labeling method provided by the disclosed embodiment and the image labeling method provided by the above embodiment belong to the same disclosed concept, and the technical details not described in detail in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0077] The various optional solutions in the image annotation method provided in the embodiment of the present disclosure and the above embodiment can be combined. The image annotation method provided in this embodiment describes in detail the process of determining the correspondence relationship of the same-name points, and can determine the affine transformation matrix of the same-name points from the grayscale image converted from the color image to the infrared image, and use it as the correspondence relationship of the same-name points to realize the label migration from the color image to the infrared image.

[0078] Figure 4 The following is a flow chart of a process for determining the correspondence relationship of points of the same name in an image annotation method provided by an embodiment of the present disclosure. Figure 4 As shown, in the image annotation method provided in this embodiment, the process of determining the correspondence relationship of the same-name points may include:

[0079] S410, extracting a first feature descriptor in the grayscale image converted from each color image and a second feature descriptor in the corresponding infrared image.

[0080] In this embodiment, each color image can be converted into a grayscale image based on an existing conversion method. A first feature descriptor of the grayscale image and a second feature descriptor of the corresponding infrared image can be obtained based on a deep learning model. The first feature descriptor and the second feature descriptor can be considered as feature points of an object in the image, such as corner points.

[0081] S420: Match the first feature descriptor and the second feature descriptor, and use the matched first feature descriptor and second feature descriptor as points of the same name.

[0082] In this embodiment, the first feature descriptor and the second feature descriptor can be matched based on an existing matching method (for example, based on an Open CV tool), wherein the matched first feature descriptor and the second feature descriptor can be considered to be the same object points (i.e., points with the same name) in the grayscale image and the color image.

[0083] S430 , determining an affine transformation matrix from each color image to the corresponding infrared image based on the pixel coordinates of the same-name points in the color image and the corresponding infrared image.

[0084] In this embodiment, the pixel coordinates of the first feature descriptor in the grayscale image are the same as its pixel coordinates in the color image. The pixel coordinates of each matching first feature descriptor and second feature descriptor on the image to which they belong can be obtained as the pixel coordinates of each point of the same name in the color image and the corresponding infrared image. Furthermore, the affine transformation matrix from the color image to the infrared image can be determined based on an existing solution algorithm (for example, based on the Open CV tool). According to the affine transformation matrix, the label data of the color image can be mapped to the infrared image. For example, the pixel position of each point in the label data of the color image can be calculated with the affine transformation matrix to obtain the pixel position corresponding to each point in the infrared image, and then the label data of the infrared image can be obtained.

[0085] In some optional implementations, matching the first feature descriptor and the second feature descriptor may include: dividing each color image and the corresponding infrared image into image blocks in the same manner; matching the first feature descriptor in each image block of each color image with the second feature descriptor of the same image block in the corresponding infrared image.

[0086] For example, Figure 5 A schematic diagram of feature sub-matching in an image annotation method provided by an embodiment of the present disclosure. Figure 5 , each color image and the corresponding infrared image can be divided into 2×2 image blocks, obtaining image blocks 1 to 4 of the color image and image blocks 1' to 4' of the infrared image; wherein image block 1 corresponds to image block 1', image block 2 corresponds to image block 2', image block 3 corresponds to image block 3', and image block 4 corresponds to image block 4'. The first feature descriptor and the second feature descriptor can be matched in the corresponding image blocks.

[0087] In these optional implementations, by dividing each color image and the corresponding infrared image into image blocks in the same way, and matching the first feature descriptor and the second feature descriptor in the corresponding image blocks, it is possible to determine the same-name points in each corresponding image block, thereby avoiding the same-name points from being concentrated in partial areas of the color image and the infrared image, improving the accuracy of the affine transformation matrix, and being more conducive to the accurate migration of the label data of the color image to the infrared image.

[0088] The technical solution of the disclosed embodiment describes in detail the process of determining the correspondence of homonymous points, and can determine the affine transformation matrix of the homonymous points from the grayscale image converted from the color image to the infrared image, and use it as the correspondence of the homonymous points to achieve label migration from the color image to the infrared image. The image annotation method provided by the disclosed embodiment and the image annotation method provided by the above embodiment belong to the same disclosed concept, and the technical details not described in detail in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0089] Figure 6 The schematic diagram of the structure of an image annotation device provided by an embodiment of the present disclosure is as follows. The image annotation device provided by this embodiment is suitable for labeling a pair of color images and infrared images.

[0090] like Figure 6 As shown, the image annotation device provided by the embodiment of the present disclosure may include:

[0091] The model annotation module 610 is used to annotate each infrared image through a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image;

[0092] The label migration module 620 is used to migrate the label data of each color image to the corresponding infrared image according to the correspondence relationship of the same-name points between the grayscale image converted from each color image and the corresponding infrared image, so as to obtain a second pseudo label of each infrared image;

[0093] The label determination module 630 is used to determine the label data of each infrared image according to the first pseudo label and the second pseudo label.

[0094] In some optional implementations, the model annotation module can be used to:

[0095] Annotating each infrared image by at least two grayscale image annotation models;

[0096] When the annotation results of the two grayscale image annotation models are consistent, the annotation results are used as the first pseudo label of the corresponding infrared image.

[0097] In some optional implementations, the image annotation apparatus may further include:

[0098] The color image annotation module is used to annotate the label data of each color image based on the following process:

[0099] Constructing at least two color image annotation models according to the color images with true label data in each color image;

[0100] By constructing at least two color image annotation models, the remaining color images without true label data in each color image are annotated;

[0101] Based on the annotation results of at least two color image annotation models, label data of the remaining color images are determined.

[0102] In some optional implementations, the image annotation apparatus may further include:

[0103] The corresponding relationship determination module is used to determine the corresponding relationship of the same-name points based on the following process:

[0104] Extracting a first feature descriptor in the grayscale image converted from each color image and a second feature descriptor in the corresponding infrared image;

[0105] The first feature descriptor and the second feature descriptor are matched, and the matched first feature descriptor and the second feature descriptor are used as points of the same name;

[0106] Based on the pixel coordinates of the same-name points in the color image and the corresponding infrared image, an affine transformation matrix from each color image to the corresponding infrared image is determined.

[0107] In some optional implementations, the corresponding relationship determination module may be used to:

[0108] Divide each color image and the corresponding infrared image into image blocks in the same manner;

[0109] The first feature descriptor in each image block of each color image is matched with the second feature descriptor of the same image block in the corresponding infrared image.

[0110] In some optional implementations, the tag determination module may be used to:

[0111] When the first pseudo label is consistent with the second pseudo label, any label of the first pseudo label and the second pseudo label is used as label data of the corresponding infrared image.

[0112] In some optional implementations, the method is applied to the field of image detection, wherein the label data of the infrared image includes detection frame data of the target object.

[0113] The image annotation device provided in the embodiments of the present disclosure can execute the image annotation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0114] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0115] Reference below Figure 7 , which shows an electronic device (eg, Figure 7 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0116] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0117] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0118] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the image annotation method of the embodiment of the present disclosure are executed.

[0119] The electronic device provided by the embodiment of the present disclosure and the image annotation method provided by the above embodiment belong to the same disclosed concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0120] The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image annotation method provided by the above embodiment is implemented.

[0121] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0122] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0123] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0124] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0125] Each infrared image is annotated by a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; according to the correspondence relationship of the same-name points from the grayscale image converted from each color image to the corresponding infrared image, the label data of each color image is transferred to the corresponding infrared image to obtain a second pseudo label of each infrared image; according to the first pseudo label and the second pseudo label, the label data of each infrared image is determined.

[0126] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0127] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the units and modules do not, in some cases, limit the units and modules themselves.

[0129] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), etc.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] According to one or more embodiments of the present disclosure, a method for annotating an image is provided, the method comprising:

[0132] Annotating each infrared image through a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on a grayscale image converted from each color image and label data of each color image; wherein each infrared image and each color image constitute a paired image;

[0133] According to the correspondence relationship between the grayscale images converted from the color images and the corresponding infrared images, the label data of the color images are transferred to the corresponding infrared images to obtain the second pseudo labels of the infrared images;

[0134] The label data of each infrared image is determined according to the first pseudo label and the second pseudo label.

[0135] According to one or more embodiments of the present disclosure, there is provided an image annotation method, further comprising:

[0136] In some optional implementations, labeling each infrared image using a grayscale image labeling model to obtain a first pseudo label for each infrared image includes:

[0137] Annotating each of the infrared images by using at least two grayscale image annotation models;

[0138] When the annotation results of the two grayscale image annotation models are consistent, the annotation results are used as the first pseudo label of the corresponding infrared image.

[0139] According to one or more embodiments of the present disclosure, there is provided an image annotation method, further comprising:

[0140] In some optional implementations, the labeling process of the label data of each color image includes:

[0141] Constructing at least two color image annotation models according to the color images having true label data among the color images;

[0142] By constructing the at least two color image annotation models, annotating the remaining color images without true label data in the color images;

[0143] Based on the annotation results of the at least two color image annotation models, label data of the remaining color images are determined.

[0144] According to one or more embodiments of the present disclosure, there is provided an image annotation method, further comprising:

[0145] In some optional implementations, the process of determining the correspondence relationship of the homonymous points includes:

[0146] Extracting a first feature descriptor in the grayscale image converted from each color image and a second feature descriptor in the corresponding infrared image;

[0147] Matching the first feature descriptor and the second feature descriptor, and taking the matched first feature descriptor and the second feature descriptor as points of the same name;

[0148] Based on the pixel coordinates of the same-name points in the color image and the corresponding infrared image, an affine transformation matrix from each color image to the corresponding infrared image is determined.

[0149] According to one or more embodiments of the present disclosure, there is provided an image annotation method, further comprising:

[0150] In some optional implementations, matching the first feature descriptor with the second feature descriptor includes:

[0151] Dividing each color image and the corresponding infrared image into image blocks in the same manner;

[0152] The first feature descriptor in each image block of each color image is matched with the second feature descriptor of the same image block in the corresponding infrared image.

[0153] According to one or more embodiments of the present disclosure, there is provided an image annotation method, further comprising:

[0154] In some optional implementations, determining the label data of each infrared image according to the first pseudo label and the second pseudo label includes:

[0155] When the first pseudo label is consistent with the second pseudo label, any label of the first pseudo label and the second pseudo label is used as label data of the corresponding infrared image.

[0156] According to one or more embodiments of the present disclosure, there is provided an image annotation method, further comprising:

[0157] In some optional implementations, the invention is applied to the field of image detection; wherein the label data of the infrared image includes detection frame data of the target object.

[0158] According to one or more embodiments of the present disclosure, there is provided an image annotation device, the device comprising:

[0159] A model annotation module, used for annotating each infrared image by a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image;

[0160] A label migration module, used for migrating the label data of each color image to the corresponding infrared image according to the correspondence relationship of the same-name points between the grayscale image converted from each color image and the corresponding infrared image, so as to obtain a second pseudo label of each infrared image;

[0161] The label determination module is used to determine the label data of each infrared image according to the first pseudo label and the second pseudo label.

[0162] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0163] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0164] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image annotation method, characterized in that: include: Annotating each infrared image through a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on a grayscale image converted from each color image and label data of each color image; wherein each infrared image and each color image constitute a paired image; According to the correspondence relationship between the grayscale images converted from the color images and the corresponding infrared images, the label data of the color images are transferred to the corresponding infrared images to obtain the second pseudo labels of the infrared images; The label data of each infrared image is determined according to the first pseudo label and the second pseudo label.

2. The method according to claim 1, characterized in that The step of labeling each infrared image by using a grayscale image labeling model to obtain a first pseudo label of each infrared image includes: Annotating each of the infrared images by using at least two grayscale image annotation models; When the annotation results of the two grayscale image annotation models are consistent, the annotation results are used as the first pseudo label of the corresponding infrared image.

3. The method according to claim 1, characterized in that The labeling process of the label data of each color image includes: Constructing at least two color image annotation models according to the color images having true label data among the color images; By constructing the at least two color image annotation models, annotating the remaining color images without true label data in the color images; Based on the annotation results of the at least two color image annotation models, label data of the remaining color images are determined.

4. The method according to claim 1, characterized in that: The process of determining the correspondence relationship of the same-name points includes: Extracting a first feature descriptor in the grayscale image converted from each color image and a second feature descriptor in the corresponding infrared image; Matching the first feature descriptor and the second feature descriptor, and taking the matched first feature descriptor and the second feature descriptor as points of the same name; Based on the pixel coordinates of the same-name points in the color image and the corresponding infrared image, an affine transformation matrix from each color image to the corresponding infrared image is determined.

5. The method according to claim 4, characterized in that The matching the first feature descriptor with the second feature descriptor includes: Dividing each color image and the corresponding infrared image into image blocks in the same manner; The first feature descriptor in each image block of each color image is matched with the second feature descriptor of the same image block in the corresponding infrared image.

6. The method according to claim 1, characterized in that The step of determining the label data of each infrared image according to the first pseudo label and the second pseudo label includes: When the first pseudo label is consistent with the second pseudo label, any label of the first pseudo label and the second pseudo label is used as label data of the corresponding infrared image.

7. The method according to any one of claims 1 to 6, characterized in that: Applied to the field of image detection; wherein the label data of the infrared image includes detection frame data of the target object.

8. An image annotation device, characterized in that: include: A model annotation module, used for annotating each infrared image by a grayscale annotation model to obtain a first pseudo label of each infrared image; wherein the grayscale annotation model is constructed based on the grayscale image converted from each color image and the label data of each color image; wherein each infrared image and each color image constitute a paired image; A label migration module, used for migrating the label data of each color image to the corresponding infrared image according to the correspondence relationship of the same-name points between the grayscale image converted from each color image and the corresponding infrared image, so as to obtain a second pseudo label of each infrared image; The label determination module is used to determine the label data of each infrared image according to the first pseudo label and the second pseudo label.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image annotation method as described in any one of claims 1 to 7.

10. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image annotation method according to any one of claims 1 to 7 when executed by a computer processor.

Images