Image labeling method and device, equipment and storage medium

Abstract

The invention discloses an image annotation method and device, equipment and a storage medium. The method comprises the steps that a first image is acquired, and the first image comprises a plurality of objects to be labeled; performing image segmentation on the first image by using the first large segmentation model to obtain a segmented thermodynamic diagram, the segmented thermodynamic diagram comprising a plurality of pixel regions, and each pixel region corresponding to one to-be-labeled object; cutting the first image according to the plurality of pixel regions to obtain a second image corresponding to each to-be-labeled object; for each second image, carrying out image-text matching on the second image and a preset category label by utilizing a first image-text matching large model; and determining the preset category label matched with the second image as a labeling result of the to-be-labeled object in the second image. According to the image annotation method provided by the embodiment of the invention, the to-be-annotated objects of multiple categories can be annotated.

Description

Technical Field

[0001] The present application belongs to the field of data annotation technology, and in particular, relates to an image annotation method, device, equipment and storage medium. Background Art

[0002] Image annotation is the process of labeling images in a dataset to train a machine learning model. Image annotation can include image classification, object detection, object recognition, image segmentation and other annotation types. For each annotation type, labeling different categories of objects to be labeled can belong to different annotation tasks.

[0003] At present, image annotation is usually performed through a specific task model. Specifically, for different annotation tasks, it is necessary to first obtain a large amount of manually annotated image data to train the initial model and obtain a specific task model. Then the specific task model is used for annotation. Since the specific task model is trained for a single annotation task, it can only annotate objects of a certain category. Summary of the invention

[0004] The embodiments of the present application provide an image annotation method, apparatus, device, and storage medium, which can annotate objects to be annotated of multiple categories.

[0005] In a first aspect, an embodiment of the present application provides an image annotation method, the method comprising:

[0006] Acquire a first image, wherein the first image includes a plurality of objects to be labeled, and the plurality of objects to be labeled correspond to a plurality of categories;

[0007] Performing image segmentation on the first image using a first large segmentation model to obtain a segmentation heat map, wherein the segmentation heat map includes a plurality of pixel regions, each of which corresponds to one of the objects to be labeled;

[0008] Clipping the first image according to the multiple pixel regions to obtain a second image corresponding to each of the objects to be labeled;

[0009] For each of the second images, use the first image-text matching large model to perform image-text matching on the second image and a preset category label;

[0010] A preset category label matching the second image is determined as a labeling result of the object to be labeled in the second image.

[0011] In a possible implementation manner, the cropping the first image according to the multiple pixel regions includes:

[0012] Mapping the plurality of pixel regions into the first image to obtain a plurality of mapping regions;

[0013] The first image is cropped according to the plurality of mapping areas.

[0014] In a possible implementation, there are multiple preset category labels, and using the first large image-text matching model to perform image-text matching on the second image and the preset category labels includes:

[0015] Using the first image-text matching large model, respectively perform image-text matching on the second image and the plurality of preset category labels to obtain a plurality of similarities;

[0016] The step of determining the preset category label matching the second image as the labeling result of the object to be labeled in the second image includes:

[0017] When the similarity is greater than a preset threshold, determining that the second image matches the preset category label;

[0018] In the case where there are multiple similarities greater than the preset threshold, determining a target similarity with the largest value among the multiple similarities greater than the preset threshold;

[0019] The preset category label corresponding to the target similarity is determined as the labeling result of the object to be labeled in the second image.

[0020] In a possible implementation manner, after obtaining the multiple similarities, the method further includes:

[0021] When the multiple similarities are all less than the preset threshold, obtaining a first data set, wherein the first data set includes multiple third images and their corresponding segmentation heat maps;

[0022] The first segmentation model is iteratively optimized using the first data set.

[0023] In a possible implementation manner, after obtaining the multiple similarities, the method further includes:

[0024] When the multiple similarities are all smaller than the preset threshold, obtaining a second data set, wherein the second data set includes multiple fourth images and their corresponding category labels;

[0025] The first image-text matching large model is iteratively optimized using the second data set.

[0026] In a possible implementation manner, before performing image segmentation on the first image using the first large segmentation model, the method further includes:

[0027] Acquire a third data set, wherein the third data set includes a plurality of fifth images and their corresponding segmentation heat maps;

[0028] Using the third data set to train the first segmentation large model to obtain a second segmentation large model;

[0029] The step of performing image segmentation on the first image by using the first segmentation large model comprises:

[0030] The first image is segmented using the second segmentation model.

[0031] In a possible implementation, before performing image-text matching on the second image and the preset category label using the first image-text matching large model, the method further includes:

[0032] Acquire a fourth data set, where the fourth data set includes a plurality of sixth images and corresponding category labels;

[0033] Using the fourth data set to train the first image-text matching large model to obtain a second image-text matching large model;

[0034] The performing image-text matching on the second image and the preset category label by using the first image-text matching large model includes:

[0035] The second image and the preset category label are matched by using the second image-text matching large model.

[0036] In a second aspect, an embodiment of the present application provides an image annotation device, the device comprising:

[0037] A first acquisition module, configured to acquire a first image, wherein the first image includes a plurality of objects to be labeled, and the plurality of objects to be labeled correspond to a plurality of categories;

[0038] A segmentation module, configured to perform image segmentation on the first image using a first segmentation model to obtain a segmentation heat map, wherein the segmentation heat map includes a plurality of pixel regions, each of which corresponds to one of the objects to be labeled;

[0039] A cropping module, configured to crop the first image according to the plurality of pixel regions to obtain a second image corresponding to each of the objects to be labeled;

[0040] A matching module, configured to perform image-text matching between the second image and a preset category label using the first image-text matching large model for each of the second images;

[0041] A determination module is used to determine a preset category label that matches the second image as a labeling result of the object to be labeled in the second image.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0043] When the processor executes the computer program instructions, the processor implements any possible implementation method of the first aspect described above.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements a method in any possible implementation method of the first aspect described above.

[0045] The image annotation method, device, equipment and storage medium of the embodiment of the present application can segment the first image into multiple second images including the objects to be annotated by using the first segmentation large model to obtain a segmentation heat map including multiple pixel areas, and crop the first image according to the multiple pixel areas to obtain a second image corresponding to each object to be annotated. By determining the preset category label matching the second image as the annotation result of the object to be annotated in the second image for each second image, the annotation result corresponding to each of the multiple objects to be annotated in the first image can be obtained. Since multiple objects to be annotated correspond to multiple categories, by annotating the first image together with the first segmentation large model and the first image-text matching large model, objects to be annotated in multiple categories can be annotated. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 It is a flowchart of an image annotation method provided in an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of an image annotation method provided in an embodiment of the present application;

[0049] Figure 3 is a structural schematic diagram of an image annotation device provided in an embodiment of the present application;

[0050] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0053] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0054] As described in the background technology section, in order to solve the problems of the prior art, the embodiments of the present application provide an image annotation method, apparatus, device and storage medium.

[0055] The following first introduces the image annotation method provided in the embodiment of the present application.

[0056] Figure 1 FIG. 1 is a flow chart of an image annotation method provided by an embodiment of the present application. Figure 1 As shown, the image annotation method provided in the embodiment of the present application includes the following steps:

[0057] S110, acquiring a first image, wherein the first image includes a plurality of objects to be labeled, and the plurality of objects to be labeled correspond to a plurality of categories;

[0058] S120, using the first large segmentation model to perform image segmentation on the first image to obtain a segmentation heat map, where the segmentation heat map includes a plurality of pixel regions, each pixel region corresponding to an object to be labeled;

[0059] S130, cropping the first image according to the plurality of pixel regions to obtain a second image corresponding to each object to be labeled;

[0060] S140: for each second image, use the first image-text matching large model to perform image-text matching on the second image and a preset category label;

[0061] S150: Determine a preset category label that matches the second image as a labeling result of the object to be labeled in the second image.

[0062] The image annotation method of the embodiment of the present application can segment the first image into multiple second images including the objects to be annotated by using the first segmentation large model to obtain a segmentation heat map including multiple pixel areas, and crop the first image according to the multiple pixel areas to obtain a second image corresponding to each object to be annotated. By determining the preset category label matching the second image as the annotation result of the object to be annotated in the second image for each second image, the annotation results corresponding to each of the multiple objects to be annotated in the first image can be obtained. Since multiple objects to be annotated correspond to multiple categories, by annotating the first image together with the first segmentation large model and the first image-text matching large model, objects to be annotated in multiple categories can be annotated.

[0063] The specific implementation methods of the above steps are introduced below.

[0064] In some embodiments, in S110, the category corresponding to the object to be labeled may belong to any one of a person, an animal, a plant, or an object. Taking animals as an example, the category corresponding to the object to be labeled may be any one of "cat", "dog", "bird", or "fish". In the first image, one object to be labeled may correspond to one category, and one category may correspond to at least one object to be labeled. For example, if the first image includes three objects to be labeled, the three objects to be labeled may correspond to one category, two categories, or three categories.

[0065] In some embodiments, in S120, the large model generally refers to a deep learning model whose parameter scale and / or training data scale reaches tens of millions or more, and has strong generalization ability. The segmentation large model can be a large model implemented based on an image segmentation algorithm. The image segmentation algorithm can be used to locate objects and boundaries (lines, curves, etc.) in an image, divide an image into multiple parts, and obtain a segmentation heat map. Specifically, the image segmentation algorithm can be executed at the pixel level, assigning each pixel in the image to a specific object. Multiple pixels can form a pixel area corresponding to the specific object. That is, each pixel area can correspond to a specific object. The first segmentation large model can be an open source segmentation large model that has not been additionally trained. Since there is no need to perform additional training on the first segmentation large model, the first segmentation large model can be used to perform image segmentation on the first image to obtain a segmentation heat map, thereby saving model training time.

[0066] As an example, after the first image is input into the first segmentation model, the first segmentation model can segment each object to be labeled in the first image to obtain a segmentation heat map of the first image. In the segmentation heat map, each pixel area can represent an object to be labeled.

[0067] In actual situations, the segmentation heat map may not be accurate. For example, a pixel area may include two objects to be labeled. If a pixel area includes two objects to be labeled, the image corresponding to the pixel area may not be labeled later, affecting the accuracy of the labeling result. Therefore, in order to improve the accuracy of the segmentation heat map and thus improve the accuracy of the labeling result, in some embodiments, before the above S120, the following may also be included:

[0068] Acquire a third data set, where the third data set includes a plurality of fifth images and their corresponding segmentation heat maps;

[0069] Using the third data set to train the first segmentation large model to obtain a second segmentation large model;

[0070] Based on this, the above S120 may specifically include:

[0071] The first image is segmented using the second segmentation model.

[0072] Here, the third data set may be a data set obtained through manual annotation. The number of fifth images in the third data set may be much smaller than the number of image data used to train a specific task model. Generally, manual annotation can ensure the accuracy of the annotation results. That is, the data quality in the third data set is relatively high. In addition, each fifth image may include multiple objects to be annotated. For each object to be annotated, the fifth image may include the entire object to be annotated, or may include a part of the object to be annotated, which is not limited here. The fifth image may be the same as or different from the first image.

[0073] In this way, by using the third data set to perform additional training on the first segmentation large model to obtain the second segmentation large model, and using the second segmentation large model to segment the first image, the accuracy of the segmentation heat map can be improved, thereby improving the accuracy of the annotation results.

[0074] In some embodiments, in S130, since the segmentation heat map is obtained by performing image segmentation on the first image, the first image may include multiple regions that are the same as the multiple pixel regions in the segmentation heat map. Therefore, by cropping the first image according to the multiple pixel regions, the first image may be cropped into multiple second images. Each of the second images may include an object to be labeled.

[0075] Based on this, in order to ensure the accuracy of the cropped second image, in some embodiments, the above S130 may specifically include:

[0076] Mapping a plurality of pixel regions into the first image to obtain a plurality of mapping regions;

[0077] The first image is cropped according to the plurality of mapping regions.

[0078] Here, since the first image and the segmentation heat map have the same size, by mapping multiple pixel regions into the first image, it is possible to ensure that the multiple regions in the first image are the same as the multiple regions in the segmentation heat map, thereby ensuring the accuracy of the cropped second image.

[0079] In some embodiments, in S140, the preset category label may be a preset category name. The preset category label may include, for example, "cat", "dog", "bird", "fish", etc. The preset category label may be one or more, which is not limited here. In the case where there are multiple preset category labels, the multiple preset category labels may include all categories to which the multiple objects to be labeled in the first image belong.

[0080] In addition, the image-text matching large model can be a large model implemented based on the image-text matching algorithm. The image-text matching algorithm can match images and texts to obtain the similarity between the two. Among them, the text can be a category name text. That is, if the image matches the text, it can be determined that the image belongs to the category corresponding to the text. The first image-text matching large model can be an open source image-text matching large model that has not undergone additional training. Since there is no need to perform additional training on the first image-text matching large model, the first image-text matching large model can be used to perform image-text matching on the second image and the preset category label, thereby saving model training time.

[0081] As an example, after the second image and the preset category label are input into the first image-text matching model, the first image-text matching model can perform category matching on the second image and the preset category label, and obtain and output the similarity between the second image and the preset category label. The similarity can represent the probability that the object to be annotated in the second image belongs to the category corresponding to the preset category label.

[0082] Based on this, in order to improve the accuracy of image-text matching and thus improve the accuracy of the annotation results, in some embodiments, before the above S140, the following may also be included:

[0083] Acquire a fourth data set, where the fourth data set includes a plurality of sixth images and corresponding category labels;

[0084] Using the fourth data set to train the first image-text matching large model to obtain a second image-text matching large model;

[0085] Based on this, the above S140 may specifically include:

[0086] The second image and the preset category label are matched by using the second image-text matching large model.

[0087] Here, the fourth data set may be a data set obtained through manual annotation. The number of sixth images in the fourth data set may be much smaller than the number of image data used to train a specific task model. Generally, manual annotation can ensure the accuracy of the annotation results. That is, the data quality in the fourth data set is relatively high. In addition, each sixth image may include an object to be annotated and its corresponding category label. Specifically, the sixth image may include the entire object to be annotated, or may include a part of the object to be annotated, which is not limited here. The sixth image may be the same as or different from the second image.

[0088] In this way, by using the fourth data set to perform additional training on the first image-text matching model to obtain the second image-text matching model, and using the second image-text matching model to perform image-text matching on the second image and the preset category label, the accuracy of image-text matching can be improved, thereby improving the accuracy of the labeling results.

[0089] In some embodiments, in S150, the first image-text matching model may output the similarity between the second image and the preset category label. On this basis, the second image and the preset category label match may be such that the similarity is greater than a preset threshold. The preset threshold may be a preset similarity threshold.

[0090] As an example, in the case where there are multiple preset category labels, for each preset category label, if the similarity is greater than a preset threshold, it can be determined that the object to be annotated in the second image belongs to the category corresponding to the preset category label. That is, the preset category label can be determined as the annotation result corresponding to the object to be annotated in the second image. That is, the preset category label can be determined as the category label of the second image. If there are multiple similarities greater than the preset threshold, a preset category label corresponding to the similarity greater than the preset threshold can be selected from the similarities greater than the preset threshold as the category label of the second image. If the similarity is less than or equal to the preset threshold, it can be determined that the object to be annotated in the second image does not belong to the category corresponding to the preset category label. In this way, after each second image is annotated, the annotation results corresponding to each of the objects to be annotated in the multiple second images can be determined as the annotation results corresponding to each of the multiple objects to be annotated in the first image. That is, multiple category labels corresponding to the first image can be obtained.

[0091] Based on this, when there are multiple preset category labels, in order to further improve the accuracy of the labeling result, in some embodiments, the above S140 may specifically include:

[0092] Using the first image-text matching large model, image-text matching is performed on the second image and multiple preset category labels respectively to obtain multiple similarities;

[0093] Based on this, the above S150 may specifically include:

[0094] When the similarity is greater than a preset threshold, determining that the second image matches the preset category label;

[0095] In the case where there are multiple similarities greater than the preset threshold, determining a target similarity with the largest value among the multiple similarities greater than the preset threshold;

[0096] The preset category label corresponding to the target similarity is determined as the labeling result of the object to be labeled in the second image.

[0097] Here, if there are m preset category labels, m similarities can be obtained after image-text matching. If there are n (n≤m) similarities greater than the preset threshold, the n similarities can be sorted from large to small, and the similarity ranked first can be determined as the target similarity with the largest value. The maximum target similarity can indicate that the category of the object to be labeled in the second image is most likely to be the preset category label corresponding to the target similarity. Therefore, the preset category label corresponding to the target similarity can be determined as the category label of the second image, and the labeling result of the object to be labeled in the second image can be obtained.

[0098] In this way, by determining the preset category label corresponding to the target similarity with the largest value as the labeling result of the object to be labeled in the second image, the accuracy of the labeling result can be further improved.

[0099] If the m preset category labels include all categories to which the multiple objects to be annotated in the first image belong, and the m similarities are all less than the preset threshold, it can be determined that the image segmentation of the first segmentation model is inaccurate and / or the image-text matching of the first image-text matching model is inaccurate. Based on this, in order to further improve the accuracy of subsequent image segmentation of the first segmentation model and thus improve the accuracy of the annotation results, in some embodiments, after obtaining the multiple similarities as described above, the following may also be included:

[0100] When the multiple similarities are all less than a preset threshold, obtaining a first data set, the first data set including a plurality of third images and their corresponding segmentation heat maps;

[0101] The first segmentation model is iteratively optimized using the first data set.

[0102] Here, the first data set may include annotated data obtained by manual annotation. Generally, manual annotation can ensure the accuracy of the annotation results. Therefore, the data quality in the first data set is relatively high. In addition, each third image may include multiple objects to be annotated. For each object to be annotated, the third image may include the entire object to be annotated, or may include a part of the object to be annotated, which is not limited here. The third image may be the same as or different from the fifth image.

[0103] In this way, by iteratively optimizing the first segmentation large model using the first data set, the accuracy of subsequent image segmentation of the first segmentation large model can be further improved, thereby improving the accuracy of the labeling results.

[0104] Based on this, in order to further improve the accuracy of subsequent image-text matching of the first image-text matching large model, and thus improve the accuracy of the annotation results, in some embodiments, after obtaining the multiple similarities, the following steps may also be included:

[0105] When the multiple similarities are all less than the preset threshold, obtaining a second data set, the second data set including a plurality of fourth images and their corresponding category labels;

[0106] The second data set is used to iteratively optimize the first image-text matching model.

[0107] Here, the second data set may include the annotated data obtained by manual annotation. Generally, manual annotation can ensure the accuracy of the annotation results. Therefore, the data quality in the second data set is relatively high. In addition, each fourth image may include an object to be annotated. Specifically, the fourth image may include the entire object to be annotated, or may include a part of the object to be annotated, which is not limited here. The fourth image may be the same as or different from the sixth image.

[0108] In this way, by iteratively optimizing the first large image-text matching model using the second data set, the accuracy of subsequent image segmentation of the first large image-text matching model can be further improved, thereby improving the accuracy of the annotation results.

[0109] In order to better describe the entire solution, some specific examples are given based on the above embodiments.

[0110] For example, Figure 2As shown, after the first image is input into the first segmentation model, the first segmentation model can perform image segmentation on the first image to obtain a segmentation heat map. The first image can be clipped according to the multiple pixel areas in the segmentation heat map to obtain a second image corresponding to each object to be labeled. For each second image, after the second image and the preset category label are input into the first image-text matching model, the first image-text matching model can perform image-text matching on the second image and the preset category label to obtain the similarity between the second image and the preset category label, and determine the preset category label with a similarity greater than a preset threshold as the labeling result of the object to be labeled in the second image. In the case where there are multiple similarities greater than the preset threshold, the preset category label corresponding to the similarity with the largest value among the multiple similarities can be determined as the labeling result of the object to be labeled in the second image. In this way, by repeating the above steps, the labeling results corresponding to each of the multiple objects to be labeled in the first image can be obtained. Compared with the current specific task model that can only label objects to be labeled of a certain category, through the embodiment of the present application, objects to be labeled belonging to different categories can be labeled separately.

[0111] Based on the image annotation method provided in the above embodiment, the present application also provides a specific implementation of the image annotation device. Please refer to the following embodiment.

[0112] like Figure 3 As shown, the image annotation device 300 provided in the embodiment of the present application includes the following modules:

[0113] A first acquisition module 310 is used to acquire a first image, wherein the first image includes a plurality of objects to be labeled, and the plurality of objects to be labeled correspond to a plurality of categories;

[0114] A segmentation module 320, configured to perform image segmentation on the first image using the first segmentation model to obtain a segmentation heat map, wherein the segmentation heat map includes a plurality of pixel regions, each pixel region corresponding to an object to be labeled;

[0115] A cropping module 330, configured to crop the first image according to a plurality of pixel regions to obtain a second image corresponding to each object to be labeled;

[0116] A matching module 340 is used to perform image-text matching between each second image and a preset category label using the first image-text matching large model;

[0117] The determination module 350 is configured to determine a preset category label that matches the second image as a labeling result of the object to be labeled in the second image.

[0118] The image annotation device 300 is described in detail below, as follows:

[0119] In some embodiments, the trimming module 330 may specifically include:

[0120] A mapping submodule, used for mapping a plurality of pixel regions into a first image to obtain a plurality of mapping regions;

[0121] The cropping submodule is used to crop the first image according to the multiple mapping areas.

[0122] In some embodiments, there are multiple preset category labels. Based on this, the matching module 340 may specifically include:

[0123] A first matching submodule is used to perform image-text matching on the second image and a plurality of preset category labels respectively using the first image-text matching large model to obtain a plurality of similarities;

[0124] Based on this, the determination module 350 may specifically include:

[0125] A first determination submodule, used to determine that the second image matches a preset category label when the similarity is greater than a preset threshold;

[0126] A second determination submodule is used to determine a target similarity with the largest value among the multiple similarities greater than the preset threshold when there are multiple similarities greater than the preset threshold;

[0127] The third determination submodule is used to determine the preset category label corresponding to the target similarity as the labeling result of the object to be labeled in the second image.

[0128] In some embodiments, the determination module 350 may further include:

[0129] A first acquisition submodule, configured to acquire a first data set when the multiple similarities are all less than the preset threshold, wherein the first data set includes multiple third images and their corresponding segmentation heat maps;

[0130] The first iterative submodule is used to iteratively optimize the first segmentation model using the first data set.

[0131] In some embodiments, the determination module 350 may further include:

[0132] A second acquisition submodule, configured to acquire a second data set when the multiple similarities are all less than a preset threshold, wherein the second data set includes multiple fourth images and their corresponding category labels;

[0133] The second iterative submodule is used to iteratively optimize the first image-text matching large model using the second data set.

[0134] In some embodiments, the image annotation apparatus 300 may further include:

[0135] A second acquisition module is used to acquire a third data set before performing image segmentation on the first image using the first segmentation model, wherein the third data set includes a plurality of fifth images and their corresponding segmentation heat maps;

[0136] A first training module is used to train the first segmentation large model using a third data set to obtain a second segmentation large model;

[0137] Based on this, the segmentation module 320 may specifically include:

[0138] The segmentation submodule is used to perform image segmentation on the first image using the second segmentation model.

[0139] In some embodiments, the image annotation apparatus 300 may further include:

[0140] A third acquisition module is used to acquire a fourth data set before performing image-text matching on the second image and the preset category label using the first image-text matching large model, wherein the fourth data set includes a plurality of sixth images and their corresponding category labels;

[0141] A second training module is used to train the first image-text matching large model using the fourth data set to obtain a second image-text matching large model;

[0142] Based on this, the matching module 340 may specifically include:

[0143] The second matching submodule is used to perform image-text matching on the second image and the preset category label using the second image-text matching large model.

[0144] The image annotation device of the embodiment of the present application can segment the first image into multiple second images including the objects to be annotated by using the first segmentation large model to obtain a segmentation heat map including multiple pixel areas, and crop the first image according to the multiple pixel areas to obtain a second image corresponding to each object to be annotated. By determining the preset category label matching the second image as the annotation result of the object to be annotated in the second image for each second image, the annotation result corresponding to each of the multiple objects to be annotated in the first image can be obtained. Since multiple objects to be annotated correspond to multiple categories, by annotating the first image together with the first segmentation large model and the first image-text matching large model, objects to be annotated in multiple categories can be annotated.

[0145] Based on the image annotation method provided in the above embodiment, the embodiment of the present application also provides a specific implementation of the electronic device. Figure 4 A schematic diagram of an electronic device 400 provided in an embodiment of the present application is shown.

[0146] The electronic device 400 may include a processor 410 and a memory 420 storing computer program instructions.

[0147] Specifically, the processor 410 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0148] The memory 420 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 420 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 420 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 420 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 420 is a non-volatile solid-state memory.

[0149] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present application.

[0150] The processor 410 implements any one of the image annotation methods in the above embodiments by reading and executing computer program instructions stored in the memory 420 .

[0151] In one example, the electronic device 400 may further include a communication interface 430 and a bus 440. Figure 4 As shown, the processor 410, the memory 420, and the communication interface 430 are connected via a bus 440 and communicate with each other.

[0152] The communication interface 430 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0153] Bus 440 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 440 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0154] Exemplarily, the electronic device 400 may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).

[0155] The electronic device can execute the image annotation method in the embodiment of the present application, thereby realizing the combination Figures 1 to 3 The image annotation method and device are described.

[0156] In addition, in combination with the image annotation method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image annotation methods in the above embodiment is implemented.

[0157] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0158] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0159] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0160] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0161] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An image annotation method, characterized in that: include: Acquire a first image, wherein the first image includes a plurality of objects to be labeled, and the plurality of objects to be labeled correspond to a plurality of categories; Performing image segmentation on the first image using a first large segmentation model to obtain a segmentation heat map, wherein the segmentation heat map includes a plurality of pixel regions, each of which corresponds to one of the objects to be labeled; Clipping the first image according to the multiple pixel regions to obtain a second image corresponding to each of the objects to be labeled; For each of the second images, use the first image-text matching large model to perform image-text matching on the second image and a preset category label; A preset category label matching the second image is determined as a labeling result of the object to be labeled in the second image.

2. The method according to claim 1, characterized in that The step of clipping the first image according to the plurality of pixel regions comprises: Mapping the plurality of pixel regions into the first image to obtain a plurality of mapping regions; The first image is cropped according to the plurality of mapping areas.

3. The method according to claim 1, characterized in that There are multiple preset category labels, and using the first image-text matching large model to perform image-text matching on the second image and the preset category labels includes: Using the first image-text matching large model, respectively perform image-text matching on the second image and the plurality of preset category labels to obtain a plurality of similarities; The step of determining the preset category label matching the second image as the labeling result of the object to be labeled in the second image includes: When the similarity is greater than a preset threshold, determining that the second image matches the preset category label; In the case where there are multiple similarities greater than the preset threshold, determining a target similarity with the largest value among the multiple similarities greater than the preset threshold; The preset category label corresponding to the target similarity is determined as the labeling result of the object to be labeled in the second image.

4. The method according to claim 3, characterized in that After obtaining the multiple similarities, the method further includes: When the multiple similarities are all less than the preset threshold, obtaining a first data set, wherein the first data set includes multiple third images and their corresponding segmentation heat maps; The first segmentation model is iteratively optimized using the first data set.

5. The method according to claim 3, characterized in that: After obtaining the multiple similarities, the method further includes: When the multiple similarities are all smaller than the preset threshold, obtaining a second data set, wherein the second data set includes multiple fourth images and their corresponding category labels; The first image-text matching large model is iteratively optimized using the second data set.

6. The method according to claim 1, characterized in that Before performing image segmentation on the first image using the first segmentation large model, the method further includes: Acquire a third data set, wherein the third data set includes a plurality of fifth images and their corresponding segmentation heat maps; Using the third data set to train the first segmentation large model to obtain a second segmentation large model; The step of performing image segmentation on the first image by using the first segmentation large model comprises: The first image is segmented using the second segmentation model.

7. The method according to claim 1, characterized in that Before performing image-text matching on the second image and the preset category label using the first image-text matching large model, the method further includes: Acquire a fourth data set, where the fourth data set includes a plurality of sixth images and corresponding category labels; Using the fourth data set to train the first image-text matching large model to obtain a second image-text matching large model; The performing image-text matching on the second image and the preset category label by using the first image-text matching large model includes: The second image and the preset category label are matched by using the second image-text matching large model.

8. An image annotation device, characterized in that: The device comprises: A first acquisition module, configured to acquire a first image, wherein the first image includes a plurality of objects to be labeled, and the plurality of objects to be labeled correspond to a plurality of categories; A segmentation module, used to perform image segmentation on the first image using a first segmentation model to obtain a segmentation heat map, wherein the segmentation heat map includes a plurality of pixel regions, each of which corresponds to one of the objects to be labeled; A cropping module, configured to crop the first image according to the plurality of pixel regions to obtain a second image corresponding to each of the objects to be labeled; A matching module, configured to perform image-text matching between the second image and a preset category label using the first image-text matching large model for each of the second images; A determination module is used to determine a preset category label that matches the second image as a labeling result of the object to be labeled in the second image.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image annotation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image annotation method according to any one of claims 1 to 7 is implemented.

Images