Image annotation method and device

Abstract

The embodiment of the present application provides an image annotation method and device, which relates to the field of data processing technology. The method includes: obtaining at least one text feature corresponding to multiple target retrieval texts; obtaining image features corresponding to the image to be annotated; converting the at least one text feature and the image feature into features of the same type; calculating the similarity score between the at least one text feature and the image feature; obtaining the text features whose similarity score is greater than a preset threshold, and obtaining corresponding keywords based on at least one of the text features, and annotating the image to be annotated. The embodiment of the present application is used to improve the accuracy when generating image tags.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an image annotation method and device. Background Art

[0002] With the rapid development of modern multimedia technology and network technology and the rise of social media, the characteristics of image presentation can be marked by labeling images. Labeling services have enriched the text description information of images to a great extent.

[0003] Existing image annotation methods mainly obtain labels that match the current image scene by identifying the image. However, due to the different complexity of the scenes, when faced with more complex scenes, the above methods can only obtain a single level of information from the image modality, and often obtain some labels that do not match the current image scene, resulting in low accuracy in labeling images. Summary of the invention

[0004] In view of this, an embodiment of the present application provides an image annotation method and device for improving the accuracy when generating image labels.

[0005] In a first aspect, an embodiment of the present application provides an image annotation method, comprising:

[0006] Obtaining at least one text feature corresponding to a plurality of target retrieval texts;

[0007] Obtain image features corresponding to the image to be annotated;

[0008] converting the at least one text feature and the image feature into features of the same type;

[0009] Calculating a similarity score between the at least one text feature and the image feature;

[0010] The text features whose similarity scores are greater than a preset threshold are obtained, and corresponding keywords are obtained according to the text features to annotate the image to be annotated.

[0011] As an optional implementation of the embodiment of the present application, the obtaining of at least one text feature corresponding to the multiple target search texts includes:

[0012] Inputting the multiple target search texts into a language model to obtain text features corresponding to the multiple target search texts respectively;

[0013] The language model is a model obtained by training a first machine learning model based on a first sample data set, and the first sample data set includes a plurality of sample retrieval texts and sample text features corresponding to the plurality of sample retrieval texts.

[0014] As an optional implementation of the embodiment of the present application, before inputting the multiple target search texts into the language model and obtaining the text features respectively corresponding to the multiple target search texts, the method further includes:

[0015] Preprocessing is performed on a plurality of preset search texts to obtain a plurality of target search texts corresponding to the plurality of preset search texts.

[0016] As an optional implementation of the embodiment of the present application, the obtaining of image features corresponding to the image to be annotated includes:

[0017] Inputting the image to be annotated into an image model to obtain the image features;

[0018] The image model is a model obtained by training a second machine learning model based on a second sample data set, and the second sample data set includes multiple sample images and sample image features corresponding to the multiple sample images.

[0019] As an optional implementation of the embodiment of the present application, before inputting the image to be annotated into the image model to obtain the image features, the method further includes:

[0020] Perform image enhancement processing on the image to be labeled.

[0021] As an optional implementation of the embodiment of the present application, after obtaining at least one text feature corresponding to multiple target search texts and obtaining image features corresponding to the image to be annotated, the method further includes:

[0022] Converting the at least one text feature from a first bit depth to a second bit depth; wherein the first bit depth is greater than the second bit depth; converting the image feature from a third bit depth to a fourth bit depth; wherein the third bit depth is greater than the fourth bit depth;

[0023] The second bit depth is the same as the first bit depth.

[0024] As an optional implementation of the embodiment of the present application, the calculating the similarity score between the at least one text feature and the image feature includes:

[0025] A cosine similarity algorithm is used to respectively obtain the similarity between the at least one text feature and the image feature.

[0026] In a second aspect, an embodiment of the present application provides an image annotation device, including:

[0027] A text feature acquisition unit, used to acquire at least one text feature corresponding to a plurality of target search texts;

[0028] An image feature acquisition unit, used to acquire image features corresponding to the image to be annotated;

[0029] a conversion unit, configured to convert the at least one text feature and the image feature into features of the same type;

[0030] a calculation unit, configured to calculate a similarity score between the at least one text feature and the image feature;

[0031] The labeling unit is used to obtain the text features whose similarity scores are greater than a preset threshold, and obtain corresponding keywords according to at least one of the text features to label the image to be labeled.

[0032] As an optional implementation of an embodiment of the present application, the text feature acquisition module is specifically used to input the multiple target retrieval texts into a language model to obtain text features corresponding to the multiple target retrieval texts respectively; wherein the language model is a model obtained by training a first machine learning model based on a first sample data set, and the first sample data set includes multiple sample retrieval texts and sample text features corresponding to the multiple sample retrieval texts.

[0033] As an optional implementation of the embodiment of the present application, the text feature acquisition module is also used to pre-process multiple preset search texts to obtain multiple target search texts corresponding to the multiple preset search texts.

[0034] As an optional implementation of the embodiment of the present application, the image feature acquisition module is specifically used to input the image to be annotated into an image model to obtain the image features; wherein the image model is a model obtained by training a second machine learning model based on a second sample data set, and the second sample data set includes multiple sample images and sample image features corresponding to the multiple sample images.

[0035] As an optional implementation of an embodiment of the present application, the conversion unit is also used to convert the at least one text feature from a first bit depth to a second bit depth; the first bit depth is greater than the second bit depth; convert the image feature from a third bit depth to a fourth bit depth; the third bit depth is greater than the fourth bit depth; wherein the second bit depth is the same as the first bit depth.

[0036] As an optional implementation of the embodiment of the present application, the calculation unit is specifically used to adopt a cosine similarity algorithm to respectively obtain the similarity between the at least one text feature and the image feature.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the image annotation method described in any one of the above embodiments when executing the computer program.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image annotation method described in any of the above embodiments.

[0039] In a fifth aspect, an embodiment of the present application provides a vehicle, comprising: the image annotation device described in the second aspect or the electronic device described in the third aspect.

[0040] Compared with the prior art, the technical solution provided by the disclosed embodiment has the following advantages: The image annotation method provided by the embodiment of the present application is specifically: obtaining at least one text feature corresponding to multiple target retrieval texts; obtaining image features corresponding to the image to be annotated; converting the at least one text feature and the image feature into features of the same type; calculating the similarity between the at least one text feature and the image feature; obtaining the text feature whose similarity is greater than a preset threshold, and obtaining corresponding keywords based on the text feature to annotate the image to be annotated. The embodiment of the present application obtains features in both text and image modes, and then converts the features in both modes to the same preset type to obtain the similarity between the at least one text feature and the image feature, and then obtains the retrieval text corresponding to the text vector whose similarity is greater than the preset threshold, so as to generate corresponding keywords, and then annotates the image to be annotated. Compared with the prior art that can only obtain a single level of information from the image modality, the embodiment of the present application combines the features of the two modalities of retrieval text and image. In more complex scenarios, the image can be targetedly labeled in combination with the retrieval text. Therefore, the real-time example of the present application can combine multiple modalities to achieve labeling of images of complex scenes, thereby improving the accuracy of generating image labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 One of the step flow charts of the image annotation method provided in the embodiment of the present application;

[0044] Figure 2 The second step flow chart of the image annotation method provided in the embodiment of the present application;

[0045] Figure 3 The third step flow chart of the image annotation method provided in the embodiment of the present application;

[0046] Figure 4 A schematic diagram of the structure of an image annotation device provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0050] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0051] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0052] Based on the above content, the present application embodiment provides an image annotation method, referring to Figure 1 As shown, the image annotation method includes the following steps S101 to S104:

[0053] S101. Retrieve text according to a target and obtain at least one text feature.

[0054] The image annotation method provided in the embodiment of the present application is mainly applied to the vehicle surrounding environment image collected for the vehicle, and the scene recognition is performed in combination with the pre-set search text to annotate the vehicle surrounding environment image with the corresponding label. However, since the present application is mainly a method for annotating the vehicle surrounding environment image, the multiple target search texts set are all about the vehicle surrounding environment, so that the image label corresponding to the vehicle surrounding image can be better obtained, and the technical solution provided by the present application can not only be used to annotate the vehicle surrounding environment image, but also can be applied to any other scene, not limited to the scene provided in the embodiment of the present application, for example, it can be applied to the shopping platform to annotate the object image provided to the user, etc., and the present application does not make any limitation to this.

[0055] In some embodiments, the target retrieval text may be obtained through a cloud language model. Specifically, the usage scenario of the current image annotation method may be used as the input of the cloud language model, and the cloud language model obtains the retrieval text that conforms to the current scenario through reasoning as the target retrieval text; wherein the cloud language model is obtained by training a machine model using a large amount of text data; for example, when the usage scenario of the image annotation method is vehicle driving, the retrieval text may be: "Are there pedestrians crossing the road in the picture" or "Does the picture describe a congested traffic scene" and other questions used to evaluate the specific conditions of the current vehicle surrounding environment.

[0056] S102: Obtain image features corresponding to the image to be annotated.

[0057] In an embodiment of the present application, when labeling the vehicle surrounding environment image, the vehicle side will obtain the vehicle surrounding video through the image acquisition device set around the vehicle, and then use the key frames in the surrounding environment video as the image to be labeled at a fixed frequency.

[0058] In some embodiments, the method for obtaining the image features corresponding to the image to be annotated may be to obtain the image features through a related image feature extraction method.

[0059] S103: Convert the at least one text feature and the image feature into features of the same preset type.

[0060] In some embodiments, since text and images correspond to two different modes in media information, the text features obtained through text and the image features obtained through images will also be two different types. In order to facilitate the subsequent calculation of the similarity scores between the text features and the image features, it is necessary to convert the at least one text feature and the image feature into features of the same preset type for calculation.

[0061] The modality refers to some ways of expressing or perceiving things. Each source or form of information can be called a modality. For example, people have touch, hearing, vision, and smell; the media of information include voice, video, text, etc.; there are various sensors, such as radar, infrared, accelerometer, etc. Each of the above can be called a modality.

[0062] In the embodiment of the present application, the image is labeled based on two modalities, image and text, which is a multimodal labeling method. Specifically, it expresses or perceives an object from the perspective of multiple modalities. Multimodality can be divided into homogeneous modalities and heterogeneous modalities. Homogeneous modalities can be understood as pictures taken by two cameras of the same object; heterogeneous modalities can be understood as the relationship between pictures and texts in the embodiment of the present application.

[0063] In some embodiments, the method for converting the at least one text feature and the image feature into features of the same preset type may be to input the at least one text feature and the image feature into a multimodal decoder for processing; exemplarily, a lightweight ViT model (Vision Transformer, a model that applies Transformer to image classification) may be used as a multimodal decoder. The function of the multimodal decoder is to align the features in the image modality and the features in the text modality to the same feature dimension and perform feature comparison.

[0064] S104: Calculate a similarity score between the at least one text feature and the image feature to be annotated.

[0065] In some embodiments, after obtaining the at least one text feature and the image feature under the same preset mode, a one-to-one comparison is performed, specifically, each of the at least one text feature is used to calculate with the image feature to obtain a corresponding similarity score. Specifically, the similarity score can be obtained by the cosine similarity method, and can also be calculated by feature similarity calculation methods such as Euclidean distance and standardized Euclidean distance.

[0066] S105: Acquire the text features whose similarity scores are greater than a preset threshold, and acquire corresponding keywords according to at least one of the text features to annotate the image to be annotated.

[0067] In some embodiments, in the above step S104, the similarity scores between the at least one text feature and the target image text are obtained, and based on the similarity scores and the preset threshold, the text features with similarity scores greater than the preset threshold can be obtained. Then, the corresponding keywords can be obtained based on the text features, and the search keywords can be used as labels for the image to be annotated and output.

[0068] Specifically, the method of obtaining the corresponding keyword based on at least one of the text features can be based on a pre-set corpus, and the corpus includes a preset correspondence between text features and keywords. When obtaining the text feature corresponding to the search text, the keywords in the search text are first extracted, and then the corresponding text feature is obtained based on the preset correspondence between the text features and the keywords in the corpus. Similarly, when obtaining the keyword corresponding to the text feature, it can be obtained based on the preset correspondence between the text features and the keywords in the corpus.

[0069] It should be noted that when the similarity score is greater than the preset threshold, it means that the corresponding text feature and image feature are matched, that is, the text feature is consistent with the semantics presented by the image feature.

[0070] Compared with the prior art, the technical solution provided by the disclosed embodiment has the following advantages: The image annotation method provided by the embodiment of the present application is specifically: obtaining at least one text feature corresponding to multiple target retrieval texts; obtaining image features corresponding to the image to be annotated; converting the at least one text feature and the image feature into features of the same type; calculating the similarity between the at least one text feature and the image feature; obtaining the text feature whose similarity is greater than a preset threshold, and obtaining corresponding keywords based on the text feature to annotate the image to be annotated. The embodiment of the present application obtains features in both text and image modes, and then converts the features in both modes to the same preset type to obtain the similarity between the at least one text feature and the image feature, and then obtains the retrieval text corresponding to the text vector whose similarity is greater than the preset threshold, so as to generate corresponding keywords, and then annotates the image to be annotated. Compared with the prior art that can only obtain a single level of information from the image modality, the embodiment of the present application combines the features of the two modalities of retrieval text and image. In more complex scenarios, the image can be targetedly labeled in combination with the retrieval text. Therefore, the real-time example of the present application can combine multiple modalities to achieve labeling of images of complex scenes, thereby improving the accuracy of generating image labels.

[0071] As an extension and refinement of the above embodiment, refer to Figure 2 As shown, the present application also provides an image annotation method, which includes the following steps S201 to S205:

[0072] S201 : Preprocess a plurality of preset search texts to obtain a plurality of target search texts corresponding to the plurality of preset search texts.

[0073] In some embodiments, the purpose of preprocessing the multiple preset search texts is to adjust the preset search texts to be more in line with the current image annotation scenario. This is because the multiple preset search texts are set by a large amount of sample data and have a certain degree of randomness when in use. Therefore, preprocessing is required during the reasoning process. Specifically, by parsing, part-of-speech analysis, synonym analysis and replacement, etc., the multiple preset search texts are analyzed and rewritten to obtain multiple target search texts corresponding to the multiple preset search texts that are more in line with the current image annotation scenario; at the same time, for the large language model in the embodiment of the present application, since the preset search texts are rewritten, the preset search text library is further made more comprehensive, and the text training set is also expanded.

[0074] Exemplarily, when the current image annotation scene is an autonomous driving scene, it is necessary to annotate the image of the vehicle's surrounding environment. By preprocessing the preset search text, the preset search text is made more consistent with the description of the autonomous driving scene, thereby improving the matching degree of the generated label.

[0075] S202: Input the multiple target search texts into a language model to obtain text features corresponding to the multiple target search texts respectively.

[0076] The language model is a model obtained by training a first machine learning model based on a first sample data set, and the first sample data set includes a plurality of sample retrieval texts and sample text features corresponding to the plurality of sample retrieval texts.

[0077] In some embodiments, the first machine learning model can be a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a fully connected neural network (FCN) model, etc. The embodiments of the present application are not specifically limited here, and the functions of the above embodiments can be realized.

[0078] In some embodiments, the language model can be a large language model. Specifically, a large language model (LLM) is an artificial intelligence model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and the like. LLMs are characterized by their large scale and contain billions of parameters, which help them learn complex patterns in language data. These models are typically based on deep learning architectures such as transformers, which helps them achieve impressive performance on various NLP tasks.

[0079] In some embodiments, the language model can be deployed in the cloud, and the cloud generates text features, which are transmitted to the vehicle side, and further integrated and compared with the image features generated by the vehicle side to obtain the similarity. Specifically, the language model can obtain the text features by generating a scene word list, wherein the scene word list includes texts of various scene keywords and their corresponding text features; then, according to the keywords in the current search text, the corresponding search text features can be obtained from the scene word list.

[0080] In some embodiments, prompt engineering is also used for training and reasoning of large language models. When the task is more complex, it is necessary to guide the model in a way that the model can understand to improve the accuracy of the model.

[0081] Specifically, the embodiments of the present application use soft-prompt engineering: in a deep network model, learnable parameters are used to provide additional prompt information for input samples. Implicit prompts are usually specially designed structures in multimodal models, which help improve the model effect.

[0082] For example, during the training process, additional prompt information can be set as information about the autonomous driving scenario, so that the model will focus on keywords about the autonomous driving scenario to improve the model accuracy.

[0083] In some embodiments, image enhancement processing language is also used. The training of large language models refers to processing certain degraded image features, such as edges, contours, contrast, etc. through a certain image processing method to improve the visual effect of the image, increase the clarity of the image, or highlight some "useful" information in the image, compress other "useless" information, and convert the image into a form that is more suitable for computer analysis and processing.

[0084] In an embodiment of the present application, a vehicle obtains a video of the surrounding environment through an image acquisition device arranged around the vehicle. The vehicle uses the key frames in the surrounding environment video as the image to be labeled at a fixed frequency, and then uses an image enhancement method to optimize the image to be labeled. Specifically, the image enhancement processing can use random rotation, cropping, deformation and other methods to make the image features learned by the model more generalized.

[0085] S203: Input the image to be annotated into an image model to obtain the image features.

[0086] The image model is a model obtained by training a second machine learning model based on a second sample data set, and the second sample data set includes multiple sample images and sample image features corresponding to the multiple sample images.

[0087] In some embodiments, the second machine learning model can refer to the description of the first machine learning model shown in the above step S202, which will not be repeated here.

[0088] In some embodiments, the image model can be any encoder in the prior art that can extract features from an image. Specifically, obtaining the image features mainly includes three steps: feature extraction, feature encoding, and feature generation. The feature extraction mainly uses image processing techniques such as scale-invariant feature transform (SIFT), (SURF), Histogram of Oriented Gradient (HOG), etc. to extract image features. The feature encoding mainly encodes the extracted features, such as Bag of Words (BOW), VGG (Visual Geometry Group), etc. The feature generation mainly converts the encoded features into numerical vectors as image features.

[0089] In some embodiments, image enhancement processing is also used in the training of the image model. Specifically, image enhancement processing is to process certain degraded image features in the image to be annotated, such as edges, contours, contrast, etc., through a certain image processing method to improve the visual effect of the image, improve the clarity of the image, or highlight certain "useful" information in the image, compress other "useless" information, and convert the image into a form more suitable for computer analysis and processing. Among them, the image enhancement processing can use random rotation, cropping, deformation and other methods to make the image features learned by the model more generalized, and also improve the accuracy of subsequent acquisition of image features using the image model.

[0090] S204: Convert the at least one text feature and the image feature into features of the same preset type.

[0091] S205: Calculate a similarity score between the at least one text feature and the image feature.

[0092] In some embodiments, the specific implementation method of calculating the similarity score between the at least one text feature and the image feature may be:

[0093] A cosine similarity algorithm is used to respectively obtain the similarity between the at least one text feature and the image feature.

[0094] The features in the embodiments of the present application are represented by vectors. For two vectors, the cosine of the angle between them can be used to represent the degree of their approximation, referring to the following formula:

[0095]

[0096] in, and Respectively represent vectors corresponding to two different features. In this embodiment of the present application and The vector of the text feature and the vector of the image feature may be represented.

[0097] Specifically, the cosine value of the angle between two vectors is the cosine similarity of the two vectors. The above formula is directly derived from the definition of the inner product. The numerator is the inner product of the two vectors, and the denominator is the product of the two vector moduli (lengths). Through the formula, it can be intuitively believed that it eliminates the influence of vector length to a certain extent, and the cosine similarity reflects the difference in direction.

[0098] Exemplarily, when the angle is 0, the two vectors are in the same direction, which is equivalent to the highest similarity, and the cosine value is 1. When the angle is 90°, the two vectors are perpendicular, the cosine is 0, and when the angle is 180°, the two vectors are in opposite directions, and the cosine is -1. Cosine similarity is widely used for comparison: such as face comparison and voice comparison, to quickly determine the similarity of two pictures or two sounds, and then determine whether they are from the same person. When an image or sound sample has n-dimensional features, we can regard it as an n-dimensional vector. When two samples are compared using cosine similarity, the cosine value of the angle between the two n-dimensional vectors is measured. In particular, when we have a trained classification neural network, in theory, the input of this neural network in the last fully connected layer is the feature extracted by the neural network (because the last fully connected layer completes the classification task with these data). Cosine similarity is not an algorithm. After finding the cosine similarity, we can know how to define the threshold (value greater than how much is considered to be from the same class). It is often necessary to test all data sets with different threshold values ​​in turn, and select the best value as the preset threshold.

[0099] S206: Acquire the text features whose similarity scores are greater than a preset threshold, and acquire corresponding keywords according to at least one of the text features to annotate the image to be annotated.

[0100] In some embodiments, after obtaining corresponding keywords according to at least one of the text features and annotating the image to be annotated, the process further includes uploading the annotated image to other preset systems or applications to facilitate the implementation of other tasks.

[0101] As an extension and refinement of the above embodiment, after obtaining at least one text feature corresponding to multiple target search texts and obtaining image features corresponding to the image to be annotated in the above embodiment, the method further includes the following steps 1 and 2:

[0102] Step 1: Convert the at least one text feature from a first bit depth to a second bit depth.

[0103] The first bit depth is greater than the second bit depth.

[0104] Step 2: Perform precision conversion on the image features to obtain lightweight image features.

[0105] The second bit depth is the same as the fourth bit depth.

[0106] In some embodiments, most solutions use floating-point 32-bit algorithms to train the model when acquiring the text features and the image features through the model. After the model is put into use, FP32 (single-precision floating point) features will also be acquired. However, the technical solution of the present application is to complete the main retrieval task in the vehicle. The computing resources on the vehicle side are often limited by the memory bandwidth. If the FP32 features are still used, it will not work properly and affect the computing progress. Therefore, this can move data faster and maximize computing resources. The embodiment of the present application adopts precision conversion of the at least one text feature and the image feature, converting FP32 into half-precision features of int8 (half-precision floating point) to reduce the resource size occupied by the text feature, while retaining the characteristics of the original text feature. The reduction in precision may allow better use of cache and reduce memory bottlenecks.

[0107] In some embodiments, step S101 can be executed in the vehicle computer or in the cloud server. If it is executed in the cloud server, at least one acquired text feature needs to be sent to the vehicle computer, and the vehicle computer performs subsequent steps. In step 105, after acquiring the text feature with a similarity greater than a preset threshold and acquiring corresponding keywords based on the text feature, the vehicle computer annotates the image to be annotated, and then uploads the annotated image to the cloud server, thereby completing the image crowdsourcing collection work. It should be noted that the traditional image crowdsourcing collection work is to upload all the images collected by the vehicle computer to the cloud server, and the cloud server will filter the images that meet the requirements after receiving the images. Using the above method, the vehicle computer can directly select the images that need to be uploaded to the cloud server based on the received text features, which can effectively reduce the amount of data transmission between the vehicle computer and the cloud server, making the cost of crowdsourcing data collection lower, and also reducing the data reading and writing and processing volume of the vehicle computer and the cloud server.

[0108] In some embodiments, the present application also provides an overall architecture diagram of the image annotation method, referring to Figure 5 As shown, it includes: a text preprocessing module 31, a cloud-side text encoding unit 32, and a vehicle-side real-time retrieval unit 33.

[0109] Specifically: the text preprocessing module 31 is used to preprocess a plurality of preset search texts to obtain a plurality of target search texts corresponding to the plurality of preset search texts.

[0110] The cloud text encoding unit 32 includes: an implicit prompt module 321, a language model 322, and a quantization precision conversion module 323;

[0111] The implicit prompt module 321 is used for training and reasoning the language model, guiding the model in a way that the model can understand, and improving the accuracy of the model.

[0112] The language model 322 is used to input the multiple target search texts into the language model to obtain text features corresponding to the multiple target search texts respectively.

[0113] The precision conversion module 323 is used to convert the at least one text feature from a first bit depth to a second bit depth; the first bit depth is greater than the second bit depth.

[0114] The vehicle-side real-time retrieval unit 33 includes: an image model 331, a quantization accuracy conversion module 332, a feature mode alignment module 333, a comparison and scoring module 334 and a judgment module 335;

[0115] The image model 341 is used to input the image to be annotated into the image model to obtain the image features.

[0116] The precision conversion module 342 is used to perform precision conversion on the image features to obtain lightweight image features.

[0117] The feature modality alignment module 343 is used to convert the at least one text feature and the image feature into features of the same type.

[0118] The comparison and scoring module 344 is used to calculate a similarity score between the at least one text feature and the image feature.

[0119] The judgment module 345 is used to obtain the text features whose similarity scores are greater than a preset threshold, and obtain corresponding keywords according to at least one of the text features to annotate the image to be annotated.

[0120] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present invention further provides an image annotation device, which corresponds to the above method embodiment. For ease of reading, this embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the image annotation device in this embodiment can correspond to all the contents in the above method embodiment.

[0121] The present application embodiment provides an image annotation device, Figure 4 is a structural schematic diagram of the image annotation device, such as Figure 4 As shown, the image annotation device 400 includes:

[0122] A text feature acquisition unit 401 is used to acquire at least one text feature corresponding to a plurality of target search texts;

[0123] An image feature acquisition unit 402 is used to acquire image features corresponding to the image to be annotated;

[0124] A conversion unit 403, configured to convert the at least one text feature and the image feature into features of the same type;

[0125] A calculation unit 404, configured to calculate a similarity score between the at least one text feature and the image feature;

[0126] The labeling unit 405 is used to obtain the text features whose similarity scores are greater than a preset threshold, and obtain corresponding keywords according to at least one of the text features to label the image to be labeled.

[0127] As an optional implementation of the embodiment of the present application, the text feature acquisition module 401 is specifically used to input the multiple target retrieval texts into a language model to obtain text features corresponding to the multiple target retrieval texts respectively; wherein the language model is a model obtained by training a first machine learning model based on a first sample data set, and the first sample data set includes multiple sample retrieval texts and sample text features corresponding to the multiple sample retrieval texts.

[0128] As an optional implementation of the embodiment of the present application, the text feature acquisition module 401 is also used to pre-process multiple preset search texts to obtain multiple target search texts corresponding to the multiple preset search texts.

[0129] As an optional implementation of the embodiment of the present application, the image feature acquisition module 402 is specifically used to input the image to be annotated into an image model to obtain the image features; wherein the image model is a model obtained by training a second machine learning model based on a second sample data set, and the second sample data set includes multiple sample images and sample image features corresponding to the multiple sample images.

[0130] As an optional implementation of an embodiment of the present application, the conversion unit 403 is also used to convert the at least one text feature from a first bit depth to a second bit depth; the first bit depth is greater than the second bit depth; convert the image feature from a third bit depth to a fourth bit depth; the third bit depth is greater than the fourth bit depth; wherein the second bit depth is the same as the first bit depth.

[0131] As an optional implementation of the embodiment of the present application, the calculation unit is specifically used to adopt a cosine similarity algorithm to respectively obtain the similarity between the at least one text feature and the image feature.

[0132] As an optional implementation of an embodiment of the present application, the image annotation device also includes a conversion unit, which is specifically used to convert the at least one text feature from a first bit depth to a second bit depth; the first bit depth is greater than the second bit depth; the image feature is precision converted to obtain a lightweight image feature; wherein the at least one text feature and the image feature have the same precision.

[0133] As an optional implementation of the embodiment of the present application, the conversion unit 403 is specifically used to: input the at least one text feature and the image feature into a multimodal decoder to obtain the at least one text feature and the image feature.

[0134] As an optional implementation of the embodiment of the present application, the calculation unit 404 is specifically configured to respectively obtain the similarity between the at least one text feature and the image feature using a cosine similarity algorithm.

[0135] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device provided in this embodiment includes: a memory 501 and a processor 502, wherein the memory 501 is used to store a computer program; and the processor 502 is used to execute the image annotation method provided in the above embodiment when executing the computer program.

[0136] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the image annotation method provided in the above embodiment.

[0137] Based on the same inventive concept, an embodiment of the present application further provides a vehicle, which includes the image annotation device provided by the above embodiment or the electronic device provided by the above embodiment.

[0138] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0139] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0140] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0141] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image annotation method, characterized in that: include: Obtaining at least one text feature corresponding to a plurality of target retrieval texts; Obtain image features corresponding to the image to be annotated; converting the at least one text feature and the image feature into features of the same type; Calculating the similarity between the at least one text feature and the image feature; The text features whose similarity is greater than a preset threshold are obtained, and corresponding keywords are obtained according to the text features to annotate the image to be annotated.

2. The method according to claim 1, characterized in that The obtaining of at least one text feature corresponding to the plurality of target search texts includes: Inputting the multiple target search texts into a language model to obtain text features corresponding to the multiple target search texts respectively; The language model is a model obtained by training a first machine learning model based on a first sample data set, and the first sample data set includes multiple sample retrieval texts and sample text features corresponding to the multiple sample retrieval texts.

3. The method according to claim 2, characterized in that Before inputting the plurality of target search texts into the language model and obtaining text features respectively corresponding to the plurality of target search texts, the method further includes: Preprocessing is performed on a plurality of preset search texts to obtain target search texts corresponding to the preset search texts. A plurality of target search texts corresponding to the plurality of preset search texts are obtained.

4. The method according to claim 1, characterized in that: The obtaining of image features corresponding to the image to be annotated includes: Inputting the image to be annotated into an image model to obtain the image features; Among them, the image model is a model obtained by training a second machine learning model based on a second sample data set, and the second sample data set includes multiple sample images and sample image features corresponding to the multiple sample images.

5. The method according to claim 1, characterized in that After obtaining at least one text feature corresponding to a plurality of target search texts and obtaining an image feature corresponding to the image to be annotated, the method further includes: Converting the at least one text feature from a first bit depth to a second bit depth; wherein the first bit depth is greater than the second bit depth; converting the image feature from a third bit depth to a fourth bit depth; wherein the third bit depth is greater than the fourth bit depth; The second bit depth is the same as the fourth bit depth.

6. The method according to claim 2, characterized in that The calculating the similarity score between the at least one text feature and the image feature comprises: A cosine similarity algorithm is used to respectively obtain the similarity between the at least one text feature and the image feature.

7. An image annotation device, characterized in that: include: A text feature acquisition unit, used to acquire at least one text feature corresponding to a plurality of target search texts; An image feature acquisition unit, used to acquire image features corresponding to the image to be annotated; a conversion unit, configured to convert the at least one text feature and the image feature into features of the same type; a calculation unit, configured to calculate a similarity score between the at least one text feature and the image feature; The labeling unit is used to obtain the text features whose similarity scores are greater than a preset threshold, and obtain corresponding keywords according to at least one of the text features to label the image to be labeled.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the image annotation method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the image annotation method according to any one of claims 1 to 6.

10. A vehicle, characterized in that: include: The image annotation device according to claim 7 or the electronic device according to claim 8.

Images