Image automatic labeling method and system, electronic device and storage medium

Abstract

The application provides an image automatic labeling method, system, electronic equipment and storage medium. The image automatic labeling method comprises the following steps: obtaining a to-be-detected picture and a target picture; using a pre-trained image labeling model to generate a target label corresponding to the target picture in the to-be-detected picture; and the target in the target picture is a target existing or not existing in a training set of the image labeling model. The application can accurately detect and automatically label any target, can realize high-quality automatic labeling regardless of whether the target appears in the training set of the image labeling model, greatly improves the labeling efficiency, accuracy and comprehensiveness, and provides better support for research and application in the related field.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to an image automatic labeling method and system, an electronic device and a storage medium. BACKGROUND

[0002] Image labeling refers to automatically or manually extracting target information in an image by analyzing and understanding the image to be labeled, and generating a corresponding labeling result. This technology has a wide application prospect in the fields of image classification, image search and image retrieval.

[0003] However, the existing image labeling technology still has certain limitations. For example, for newly added detection categories, manual labeling requires a lot of manpower and material resources. In addition, a cross-modal target detection model trained based on image-text pairs can automatically label the target with high quality through a prompt word, i.e., a relevant description of the target to be labeled. However, this kind of model can only accurately label the targets that have appeared in the training set. If the target to be labeled has never appeared in the training set, it cannot be accurately labeled, or even cannot be automatically labeled. This limits the application of the existing technology in dealing with unknown targets. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide an image automatic labeling method, system, electronic device and storage medium, which can solve the technical problem that the prior art cannot accurately detect and automatically label any target.

[0005] In a first aspect, the application provides an image automatic labeling method, comprising: obtaining a to-be-detected picture and a target picture; and generating a target label corresponding to the target picture in the to-be-detected picture by using a pre-trained image labeling model; the target in the target picture is a target that exists or does not exist in the training set of the image labeling model.

[0006] In an implementation form of the first aspect, the generating of the target label corresponding to the target picture in the to-be-detected picture by using the pre-trained image labeling model comprises:

[0007] extracting a first feature map from the to-be-detected picture and a second feature map from the target picture;

[0008] generating a to-be-detected picture feature vector and a target picture feature vector based on the first feature map and the second feature map;

[0009] generating a query embedding vector based on the to-be-detected picture feature vector and the target picture feature vector;

[0010] generate a target label corresponding to the target picture on the picture to be detected based on the query embedding vector.

[0011] In an implementation form of the first aspect, the target label comprises a target detection box.

[0012] In an implementation form of the first aspect, generating a picture to be detected feature vector and a target picture feature vector based on the first feature map and the second feature map comprises:

[0013] performing cross-attention calculation on the first feature map and the second feature map to generate a cross-processed feature map;

[0014] performing target feature extraction and representation learning on the cross-processed feature map to generate the target picture feature vector;

[0015] performing picture to be detected feature extraction and representation learning on the cross-processed feature map to generate the picture to be detected feature vector.

[0016] In an implementation form of the first aspect, generating a query embedding vector based on the picture to be detected feature vector and the target picture feature vector comprises:

[0017] capturing associated information inside the target picture feature vector to generate a query vector;

[0018] performing cross-attention calculation on the query vector and the picture to be detected feature vector to generate the query embedding vector.

[0019] In an implementation form of the first aspect, generating a target label corresponding to the target picture on the picture to be detected based on the query embedding vector comprises:

[0020] performing linear transformation and mapping on the query embedding vector to predict the position and size of the target detection box.

[0021] In a second aspect, the present application provides an image automatic labeling system, comprising:

[0022] an acquisition module configured to acquire a picture to be detected and a target picture;

[0023] a labeling module configured to generate a target label corresponding to the target picture in the picture to be detected by using a pre-trained image labeling model; a target in the target picture is a target existing or not existing in a training set of the image labeling model.

[0024] In an implementation form of the second aspect, the image labeling model comprises a feature extraction module, an encoder, a decoder and a prediction module.

[0025] The feature extraction module is configured to extract a first feature map from the to-be-detected picture and a second feature map from the target picture.

[0026] The encoder is configured to generate a to-be-detected picture feature vector and a target picture feature vector based on the first feature map and the second feature map.

[0027] The decoder is configured to generate a query embedding vector based on the to-be-detected picture feature vector and the target picture feature vector.

[0028] The prediction module is configured to generate a target label corresponding to the target picture on the to-be-detected picture based on the query embedding vector.

[0029] In a third aspect, the present application provides an electronic device, comprising a processor and a memory.

[0030] The memory is configured to store a computer program.

[0031] The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the method described in any one of the above aspects.

[0032] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method described in any one of the above aspects.

[0033] As described above, the image automatic labeling method, system, electronic device and storage medium of the present application have the following beneficial effects:

[0034] (1) It can accurately detect and automatically label any target, that is, whether the target appears in the training set of the image labeling model or not, high-quality automatic labeling can be realized;

[0035] (2) It greatly improves the efficiency, accuracy and comprehensiveness of labeling, and provides better support for research and application in related fields. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart showing the image automatic labeling method of the present application in an embodiment is shown.

[0037] Figure 2 A flowchart showing the image automatic labeling method of the present application in another embodiment is shown.

[0038] Figure 3 A structural schematic diagram showing the image automatic labeling system of the present application in an embodiment is shown.

[0039] Figure 4Fig. 1 shows a schematic diagram of an image automatic labeling system according to an embodiment of the present application.

[0040] Figure 5 Fig. 2 shows a flow chart of an electronic device according to an embodiment of the present application.

[0041] Element Number Description

[0042] 1 Acquisition Module

[0043] 2 Labeling Module

[0044] 21 Feature Extraction Module

[0045] 22 Encoder

[0046] 23 Decoder

[0047] 24 Prediction Module

[0048] 3 Memory

[0049] 4 Processor DETAILED DESCRIPTION

[0050] The present application is described in greater detail by the following specific examples. Other advantages and benefits of the present application will become apparent to those skilled in the art upon consideration of the disclosure herein. The present application can be implemented or applied in other different embodiments and with various modifications not departing from the spirit of the present application. The details in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] It should be noted that the drawings provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the drawings, not the number, shape and size of the components in actual implementation. The shape, number and ratio of the components in actual implementation can be arbitrarily changed, and the layout of the components can be more complex.

[0052] In addition, the description of "first", "second" and the like in the present application is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the technical features indicated or implicitly indicating the number of technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of ordinary skilled in the art. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the scope of protection claimed by the present application.

[0053] The embodiments of the present application provide an image automatic labeling method, system, electronic device and storage medium. The corresponding application scenarios of the technical solutions include, but are not limited to, image search engines, image management systems, e-commerce platforms and intelligent album applications, etc. The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0054] As shown in FIG. 1, the embodiments of the present application provide an image automatic labeling method, including steps S1 and S2. Figure 1

[0055] Step S1, obtaining a to-be-detected picture and a target picture.

[0056] Specifically, the to-be-detected picture refers to a picture that needs to be subjected to target detection, i.e., a picture in which a target object is expected to be recognized and located. The to-be-detected picture of the present application can be any picture, which may, for example, contain one or more target objects.

[0057] The target picture refers to a picture used as a reference. The target picture of the present application can be a picture containing a target object or a picture containing only a background. The present application does not limit the content of the target picture.

[0058] Step S2, generating a target label corresponding to the target picture in the to-be-detected picture by using a pre-trained image labeling model.

[0059] When training the image labeling model, an image training set containing labeled targets is needed, which is composed of a group of images and target labels corresponding to the images.

[0060] In an embodiment of the present application, the target label includes a target detection box. In other embodiments, the target label can also be a target category, a target pose label, a target attribute label, a target emotion label and a target key point label, etc. In actual applications, more types of target labels can be defined according to specific needs, which are not limited herein.

[0061] In the present application, the target in the target picture is a target that exists or does not exist in the training set of the image labeling model. This means that the image automatic labeling method has the ability to label known targets and unknown targets, and can be applied to a wider range of application scenarios.

[0062] For the target existing in the training set, the pre-trained image labeling model can be used to generate a target label corresponding to the target picture according to the existing target label. This can help users quickly label a large number of pictures of known targets.

[0063] ​For targets not present in the training set, the generalization ability of image annotation models can be leveraged to infer potential targets in the target image based on existing samples in the training set and features learned by the model, and generate corresponding target annotations. This helps users annotate new images, improving the efficiency and accuracy of annotation.

[0064] like Figure 2 As shown, in one embodiment of this application, generating target annotations corresponding to the target image in the image to be detected using a pre-trained image annotation model includes steps S21 to S24. Step S21: Extracting a first feature map from the image to be detected and extracting a second feature from the target image.

[0065] Specifically, the image to be detected is input into the first backbone network, and the first feature map is extracted from the image using the first backbone network. The target image is input into the second backbone network, and the second feature map is extracted from the target image using the second backbone network.

[0066] In this application, the first backbone network is a network structure based on a convolutional neural network (CNN). CNN is a deep learning model that learns features in an image through components such as convolutional layers, pooling layers, and fully connected layers. Convolutional layers can effectively capture local features in an image, and higher-level features can be extracted through multi-layer stacking.

[0067] Similarly, the second backbone network is also a CNN-based network. The difference is that the second backbone network accepts the target image as input and performs feature extraction on the target image; while the first backbone network accepts the image to be detected as input and performs feature extraction on the image to be detected.

[0068] It should be noted that, in addition to CNN-based networks, other Transformer-based networks that can achieve feature extraction, such as ViT, are also applicable to this application.

[0069] Step S22: Generate the feature vector of the image to be detected and the feature vector of the target image based on the first feature map and the second feature map.

[0070] In one embodiment of this application, generating a feature vector for the image to be detected and a feature vector for the target image based on the first feature map and the second feature map includes:

[0071] Step S221: Perform cross-attention calculation on the first feature map and the second feature map to generate a cross-processed feature map.

[0072] Specifically, the relevance between each position in the first feature map and each position in the second feature map is measured by calculating attention weights. The attention weights represent the importance of each position in the to-be-detected picture to the target picture. This process can be achieved by calculating the similarity between the two feature maps, for example, using dot product attention, bilinear attention or other similarity measurement methods.

[0073] According to the calculated attention weights, each position in the first feature map is weighted and summed with all positions in the second feature map. The purpose of this process is to fuse the relevant information of the target picture into the feature representation of the to-be-detected picture, thereby generating a cross-processed feature map.

[0074] In this implementation, by introducing cross-attention calculation, the semantic association between the to-be-detected picture and the target picture can be better captured, and the performance of the image annotation model can be improved.

[0075] Step S222, target feature extraction and representation learning are performed on the cross-processed feature map to generate the target picture feature vector.

[0076] Specifically, the first feed-forward neural network (FNN) is used to perform linear transformation and non-linear activation function processing on the cross-processed feature map to generate the target picture feature vector.

[0077] In this application, the first feed-forward neural network is a three-layer perceptron structure composed of a ReLU (Rectified Linear Unit) activation function, a hidden layer and a linear mapping. The target picture feature vector output by the first feed-forward neural network includes a first query vector (Q1), a first key vector (K1) and a first value vector (V1); wherein Q1 contains information about the target picture, K1 is the key prompt information corresponding to the target picture; V1 is the original capture of the target picture, which is not affected by attention.

[0078] Step S223, target feature extraction and representation learning are performed on the cross-processed feature map to generate the target picture feature vector.

[0079] Specifically, the second feed-forward neural network is used to perform linear transformation and non-linear activation function processing on the cross-processed feature map to generate the to-be-detected picture feature vector.

[0080] The second feedforward neural network structure is similar to the structure of the first feedforward neural network, except that the two capture different information from the cross-processed feature map, and thus the output vectors are different. In this application, the second feedforward neural network outputs a to-be-detected picture feature vector, which includes a second key vector (K2) and a second value vector (V2); K2 is the key prompt information corresponding to the to-be-detected picture; V2 is the original capture of the to-be-detected picture and is not affected by attention.

[0081] Step S23, generating a query embedding vector based on the to-be-detected picture feature vector and the target picture feature vector.

[0082] In an embodiment of the present application, generating a query embedding vector based on the to-be-detected picture feature vector and the target picture feature vector includes:

[0083] Step S231, capturing the associated information inside the target picture feature vector to generate a query vector.

[0084] Specifically, the self-attention mechanism (Self Attention) is used to capture the associated information inside the target picture feature vector to generate a query vector (Q2).

[0085] The Self Attention is a special attention mechanism that is better at discovering associated semantic features between feature vectors. The Attention mechanism uses the following calculation formula:

[0086]

[0087] wherein d k is the dimension of the key vector (K).

[0088] In this application, the calculation steps of the Self Attention mechanism include:

[0089] (1) calculating the dot product of Q1 and K1 to generate an Attention Score;

[0090] (2) dividing the Attention Score by the square root of the dimension of K1, so as to obtain a stable gradient;

[0091] (3) using the Softmax function to normalize to obtain a weight matrix, so that the numerical value is distributed between (0, 1);

[0092] (4) weighting and summing the weight matrix and V1 to generate an output vector; and using the output vector as a new query vector Q2 for subsequent operations.

[0093] Step S232, cross attention calculation is performed between the query vector and the picture feature vector to be detected to generate a query embedding vector.

[0094] It should be noted that the cross attention calculation process of step S232 is basically the same as step S221, which will not be repeated here. Unlike step S221, the vectors participating in the cross attention calculation in step S232 include the query vector (Q2) and the second key vector (K2) and the second value vector (V2) in the picture feature vector to be detected, wherein the second key vector (K2) and the second value vector (V2) form a key-value pair (K2-V2).

[0095] It should be noted that the number of query embedding vectors can be defined as needed, for example, the generated query embedding vector can be 256, or 512, etc.

[0096] Step S24, based on the query embedding vector, a target label corresponding to the target picture is generated on the picture to be detected.

[0097] In an embodiment of the present application, generating a target label corresponding to the target picture on the picture to be detected based on the query embedding vector includes: performing linear transformation and mapping on the query embedding vector to predict the position and size of the target detection frame. This process can be completed by using a third feedforward neural network.

[0098] It should be noted that the final predicted target label of the present application corresponds to the number of query embedding vectors, for example, when the number of query embedding vectors is 20, then 20 target frames can be predicted at different positions through the third feedforward neural network.

[0099] Further, each target frame will be labeled with a probability value of the existence of the target, for example, when the probability value is 0, it means that the current target frame does not contain the target; when the probability value is 1, it means that the current target frame contains the target.

[0100] The protection scope of the image automatic labeling method described in the embodiments of the present application is not limited to the execution order of the steps listed in the embodiments, and any scheme realized by adding, replacing or replacing steps of the prior art according to the principles of the present application is included in the protection scope of the present application.

[0101] As shown in Figure 3 The embodiment provides an image automatic labeling system including an acquisition module and a labeling module.

[0102] The acquisition module is used to acquire a picture to be detected and a target picture.

[0103] Specifically, the image to be detected refers to an image for which object detection is required, that is, an image from which it is hoped to identify and locate target objects. The image to be detected in this application can be any image, for example, it may contain one or more target objects.

[0104] The target image refers to an image used as a reference. The target image in this application can be an image containing the target object or an image containing only the background. This application does not limit the content of the target image.

[0105] The annotation module is used to generate target annotations corresponding to the target image in the image to be detected using a pre-trained image annotation model.

[0106] Specifically, when training an image annotation model, a training set of images containing labeled targets is required. This training set consists of a set of images and corresponding target annotations for those images.

[0107] In one embodiment of this application, the target annotation includes a target detection box. In other embodiments, the target annotation may also be a target category, target pose annotation, target attribute annotation, target sentiment annotation, and target keypoint annotation, etc. In practical applications, more types of target annotations can be defined according to specific needs, and are not limited here.

[0108] In this application, the target in the target image refers to a target that exists or does not exist in the training set of the image annotation model. This means that the automatic image annotation system has the ability to annotate both known and unknown targets, and can be applied to a wider range of application scenarios.

[0109] For targets present in the training set, the automatic image annotation system can utilize a pre-trained image annotation model to generate corresponding target annotations for the target images based on existing target annotations. This helps users quickly annotate a large number of images containing known targets.

[0110] For targets not present in the training set, the automatic image annotation system can leverage the generalization ability of the image annotation model to infer the possible targets in the target image based on existing samples in the training set and the features learned by the model, and generate corresponding target annotations. This helps users annotate new images, improving the efficiency and accuracy of annotation.

[0111] like Figure 4 As shown, in one embodiment of this application, the image annotation model includes a feature extraction module 21, an encoder 22, a decoder 23, and a prediction module 24.

[0112] The feature extraction module 21 is used to extract a first feature map from the image to be detected and to extract a second feature map from the target image.

[0113] Specifically, the feature extraction module 21 comprises a first backbone network and a second backbone network.

[0114] The first backbone network is configured to input the to-be-detected picture and extract the first feature map from the to-be-detected picture.

[0115] The second backbone network is configured to input the target picture and extract the second feature map from the target picture.

[0116] In the present application, the first backbone network is a CNN-based network structure. CNN is a deep learning model that learns features in images through components such as convolutional layers, pooling layers, and fully connected layers. The convolutional layer can effectively capture local features in images, and through multi-layer stacking, higher-level features can be extracted.

[0117] Similarly, the second backbone network is also a CNN-based network, the difference being that the second backbone network accepts the target picture as input and performs feature extraction on the target picture; while the first backbone network accepts the to-be-detected picture as input and performs feature extraction on the to-be-detected picture.

[0118] It should be noted that in addition to CNN-based networks, other Transformer-based networks that can achieve feature extraction functions, such as ViT, are also applicable to the present application.

[0119] The encoder 22 is configured to generate a to-be-detected picture feature vector and a target picture feature vector based on the first feature map and the second feature map.

[0120] In an embodiment of the present application, the encoder 22 comprises a first cross-attention module, a first feedforward neural network, and a second feedforward neural network.

[0121] The first cross-attention module is configured to perform cross-attention calculation on the first feature map and the second feature map to generate a cross-processed feature map.

[0122] Specifically, the first cross-attention module performing cross-attention calculation on the first feature map and the second feature map comprises:

[0123] 1) Calculate the correlation between each position in the first feature map and each position in the second feature map by calculating attention weights.

[0124] The attention weight represents the importance of each position in the to-be-detected picture to the target picture. This step can be implemented by calculating the similarity between the two feature maps, for example, using dot product attention, bilinear attention or other similarity measurement methods.

[0125] 2) According to the calculated attention weight, each position in the first feature map is weighted and summed with all positions in the second feature map.

[0126] The purpose of this step is to fuse the relevant information of the target picture into the feature representation of the to-be-detected picture, thereby generating a cross-processed feature map.

[0127] In this implementation, by introducing cross-attention calculation, the encoder 22 can better capture the associated information between the to-be-detected picture and the target picture, thereby improving the performance of the image labeling model.

[0128] The first feed-forward neural network (FNN) is used to perform linear transformation and nonlinear activation function processing on the cross-processed feature map to generate the target picture feature vector.

[0129] The second feed-forward neural network is used to perform linear transformation and nonlinear activation function processing on the cross-processed feature map to generate the to-be-detected picture feature vector.

[0130] In this application, the first feed-forward neural network is a three-layer perceptron structure composed of a ReLU (Rectified Linear Unit) activation function, a hidden layer and a linear mapping. The target picture feature vector output by the first feed-forward neural network includes a first query vector (Q1), a first key vector (K1) and a first value vector (V1); wherein Q1 contains the information of the target picture, K1 is the key prompt information corresponding to the target picture; V1 is the original capture of the target picture, which is not affected by attention.

[0131] The structure of the second feed-forward neural network is similar to that of the first feed-forward neural network, except that the information captured from the cross-processed feature map by the two is different, so the output vectors are also different. In this application, the second feed-forward neural network outputs a to-be-detected picture feature vector, which includes a second key vector (K2) and a second value vector (V2); wherein K2 is the key prompt information corresponding to the to-be-detected picture; V2 is the original capture of the to-be-detected picture, which is not affected by attention.

[0132] In other embodiments, more than one encoder 22 can be stacked, i.e. the output of one encoder is the input of the next encoder. The number of encoders is not limited in the present application.

[0133] The decoder 23 is configured to generate a query embedding vector based on the image feature vector to be detected and the target image feature vector.

[0134] In an embodiment of the present application, the decoder 23 comprises a self-attention module and a second cross-attention module.

[0135] The self-attention module is configured to capture the internal correlation information of the target image feature vector to generate a query vector (Q2).

[0136] Specifically, the self-attention module adopts a self-attention mechanism (Self Attention). The Self Attention is a special attention mechanism that is better at discovering associated semantic features between feature vectors.

[0137] The Attention mechanism adopts the following calculation formula:

[0138]

[0139] wherein d k is the dimension of the key vector (K).

[0140] In the present application, the calculation steps of the Self Attention mechanism include:

[0141] (1) calculating the dot product of Q1 and K1 to generate an Attention Score.

[0142] (2) dividing the Attention Score by the square root of the dimension of K1, so as to obtain a stable gradient.

[0143] (3) using a Softmax function to normalize to obtain a weight matrix, so that the numerical value is distributed between (0, 1).

[0144] (4) weighting and summing the weight matrix and V1 to generate an output vector.

[0145] (5) inputting the output vector as a new query vector Q2 to the next module.

[0146] The second cross-attention module is configured to perform cross-attention calculation on the query vector (Q2) and the image feature vector to be detected to generate the query embedding vector.

[0147] Specifically, the processing logic of the second cross-attention module is basically the same as that of the first cross-attention module, and will not be repeated here. Unlike the first cross-attention module, the input of the second cross-attention module includes the query vector (Q2) generated by the self-attention module and the second key vector (K2) and the second value vector (V2) in the feature vector of the image to be detected.

[0148] It should be noted that the number of query embedding vectors can be defined according to the requirements. For example, the generated query embedding vectors can be 256 or 512, etc.

[0149] In other embodiments, there may be more than one decoder 23, which can be stacked in multiple layers, that is, the output of one decoder serves as the input of the next decoder. This application does not limit the number of decoders.

[0150] The prediction module 24 is used to generate target annotations corresponding to the target image on the image to be detected based on the query embedding vector.

[0151] In one embodiment of this application, the prediction module 24 includes a third feedforward neural network.

[0152] Specifically, the third feedforward neural network includes a regression branch.

[0153] The regression branch is used to perform a linear transformation and mapping on the query embedding vector to predict the position and size of the target detection box.

[0154] It should be noted that the target labels predicted in this application correspond to the number of query embedding vectors. For example, when the number of query embedding vectors is 20, the third feedforward neural network can predict 20 target boxes at different locations.

[0155] Furthermore, each target box will be marked with a probability value indicating the presence of the target. For example, a probability value of 0 indicates that the current target box does not contain the target at all, while a probability value of 1 indicates that the current target box contains the target.

[0156] The automatic image annotation system described in this application can implement the automatic image annotation method described in this application. However, the implementation device of the automatic image annotation method described in this application includes, but is not limited to, the structure of the automatic image annotation system listed in this embodiment. All structural modifications and substitutions of the prior art made based on the principles of this application are included within the protection scope of this application.

[0157] like Figure 5 As shown, this embodiment provides an electronic device, including a processor and a memory.

[0158] The memory is configured to store a computer program.

[0159] The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the image automatic labeling method described above.

[0160] In several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules / units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, device or module or unit indirect coupling or communication connection, which can be electrical, mechanical or other forms.

[0161] The modules / units described as separate components can or can not be physically separated, and the components displayed as modules / units can or can not be physical modules, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the modules / units can be selected to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in each embodiment of the present application can be integrated in a processing module, or each module / unit can be physically separated, or two or more modules / units can be integrated in one module / unit.

[0162] Those of ordinary skill in the art should further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, each example has been described in the above description in terms of its general functionality. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0163] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the image automatic labeling method. Those skilled in the art can understand that all or part of the steps of the method described in the above embodiment can be instructed by a program to complete the processor, and the program can be stored in a computer readable storage medium. The storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid state disk, a magnetic tape, a floppy disk, an optical disc and any combination thereof. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center and the like, which includes one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)) or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0164] The embodiment of the present application can also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiment of the present application are generated. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave and the like) mode.

[0165] When the computer program product is executed by a computer, the computer executes the method described in the foregoing method embodiment. The computer program product can be a software installation package, and when the foregoing method is needed, the computer program product can be downloaded and executed on the computer.

[0166] In summary, the present application can accurately detect and automatically label any target, and can achieve high-quality automatic labeling whether the target appears in the training set of the image labeling model or not. The efficiency, accuracy and comprehensiveness of labeling are greatly improved, and better support is provided for the research and application in related fields.

[0167] The above embodiments are only illustrative of the principles of the present application and its effects, and are not intended to limit the present application. Any modification or change made by any person skilled in the art without departing from the spirit and scope of the present application shall be covered by the claims of the present application.

Claims

1. An automatic image annotation method, characterized in that, include: Obtain the image to be detected and the target image; the image to be detected refers to the image for which target detection needs to be performed; The target image refers to the image used as a reference; A pre-trained image annotation model is used to generate target annotations corresponding to the target image in the image to be detected; The target in the target image is a target that does not exist in the training set of the image annotation model; Generating target annotations corresponding to the target image in the image to be detected using a pre-trained image annotation model includes: A first feature map is extracted from the image to be detected, and a second feature map is extracted from the target image; the first feature map and the second feature map are used to perform cross-attention calculation to capture the semantic relationship between the image to be detected and the target image. Generate feature vectors for the image to be detected and feature vectors for the target image based on the first feature map and the second feature map; A query embedding vector is generated based on the feature vector of the image to be detected and the feature vector of the target image; Based on the query embedding vector, a target annotation corresponding to the target image is generated on the image to be detected.

2. The method according to claim 1, characterized in that, The target annotation includes the target detection box.

3. The method according to claim 1, characterized in that, Generating feature vectors for the image to be detected and feature vectors for the target image based on the first feature map and the second feature map includes: Cross-attention calculation is performed on the first feature map and the second feature map to generate a cross-processed feature map; The cross-processed feature map is subjected to target feature extraction and representation learning to generate the target image feature vector; The cross-processed feature map is subjected to feature extraction and representation learning to generate the feature vector of the image to be detected.

4. The method according to claim 1, characterized in that, Generating a query embedding vector based on the feature vector of the image to be detected and the feature vector of the target image includes: Capture the correlation information within the feature vector of the target image to generate a query vector; The query vector and the feature vector of the image to be detected are cross-attention calculated to generate the query embedding vector.

5. The method according to claim 1, characterized in that, Generating target annotations corresponding to the target image on the image to be detected based on the query embedding vector includes: The query embedding vector is linearly transformed and mapped to predict the position and size of the target detection box.

6. An automatic image annotation system, characterized in that, include: The acquisition module is used to acquire the image to be detected and the target image; the image to be detected refers to the image for which target detection needs to be performed. The target image refers to the image used as a reference; The annotation module is used to generate target annotations corresponding to the target image in the image to be detected using a pre-trained image annotation model. The target in the target image is a target that does not exist in the training set of the image annotation model; The image annotation model includes a feature extraction module, an encoder, a decoder, and a prediction module; The feature extraction module is used to extract a first feature map from the image to be detected and a second feature map from the target image; the first feature map and the second feature map are used to perform cross-attention calculation to capture the semantic relationship between the image to be detected and the target image. The encoder is used to generate feature vectors of the image to be detected and feature vectors of the target image based on the first feature map and the second feature map; The decoder is used to generate a query embedding vector based on the feature vector of the image to be detected and the feature vector of the target image; The prediction module is used to generate target annotations corresponding to the target image on the image to be detected based on the query embedding vector.

7. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory to cause the electronic device to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 5.

Images