Image labeling method and apparatus

Abstract

The application discloses an image labeling method and device, and belongs to the technical field of computer vision. After a computer device acquires a training image set corresponding to a target image task and a language description text set corresponding to the training image set, a target image labeling model is called to determine the predicted label of each training sample image. The predicted label of the training sample image is obtained based on the feature matching result of the image feature corresponding to the training sample image and the language feature corresponding to each language description text in the language description text set by the target image labeling model. Then, the target image labeling model is trained according to the error between the real label and the predicted label of the plurality of training sample images in the training image set until the target image labeling model converges. The application transforms the image task into a matching task of image features and language features, greatly improves the initial labeling performance of the image labeling model, and thus improves the image labeling efficiency.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image annotation method and apparatus. Background Technology

[0002] When developing image-based artificial intelligence (AI) applications, human intervention is required to annotate each training image individually. Since successful AI models require thousands or even millions of accurately annotated training images, image annotation tasks are typically time-consuming and costly.

[0003] Intelligent image annotation is one of the most practical technologies for developing image-oriented AI applications. It leverages AI algorithms to quickly and automatically annotate images to be annotated from a small number of already labeled images. With intelligent image annotation technology, users can save significant image annotation costs. However, improving the efficiency of image annotation is a pressing issue that needs to be addressed when adopting intelligent image annotation technology. Summary of the Invention

[0004] This application provides an image annotation method and apparatus that can improve image annotation efficiency.

[0005] Firstly, an image annotation method is provided. This method is used on a computer device. The method includes: acquiring a training image set corresponding to a target image task and a set of corresponding language description texts. The training image set includes multiple training sample images. Each training sample image is labeled with a ground truth label. The set of language description texts includes multiple language description texts. Each language description text corresponds one-to-one with a multi-class label corresponding to the training image set. Each language description text describes the semantics of one class of labels among the multi-class labels. A target image annotation model corresponding to the target image task is invoked to determine the predicted label for each training sample image in the training image set. The predicted label for the training sample image is obtained by the target image annotation model based on feature matching results between the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set. Based on the error between the ground truth labels and predicted labels of the multiple training sample images in the training image set, the target image annotation model is trained until it converges. The target image annotation model is used to determine the annotation label for the image to be annotated under the target image task.

[0006] The image features corresponding to the training sample images are those obtained by feature extraction from the training sample images. The language features corresponding to the language description text are those obtained by feature extraction from the language description text. Model convergence can be achieved when the model's loss value is less than a preset threshold, or when the weight change of the model in two consecutive iterations is less than a preset threshold, or when the number of model iterations reaches a preset number.

[0007] In this application, the computer device acquires the set of linguistic description text corresponding to the training image set, introduces prior linguistic association knowledge of the annotation task into the training sample images, and transforms the image annotation task into a matching task between image features and linguistic features in the image annotation model. This can significantly improve the initial annotation performance of the image annotation model, increase the first-round annotation accuracy of the image annotation model when the number of initial training sample images is small, effectively reduce the number of training rounds of the image annotation model, and thus improve the efficiency of image annotation.

[0008] Optionally, after the target image annotation model converges, the computer device can also invoke the target image annotation model to determine the predicted labels of multiple validation sample images in the validation image set. The validation image set includes multiple validation sample images, each of which is labeled with a ground truth label. The computer device determines the annotation accuracy of the target image annotation model based on the ground truth labels and predicted labels of the multiple validation sample images. When the annotation accuracy of the target image annotation model does not reach a preset threshold, the computer device performs one or more model training processes until the annotation accuracy of the target image annotation model reaches the preset threshold. The model training process includes: invoking the target image annotation model to determine the predicted labels of multiple images to be annotated and the confidence level of the predicted labels of the images to be annotated; obtaining difficult-to-annotate images from the multiple images to be annotated whose predicted label confidence level is lower than the confidence level threshold; outputting the difficult-to-annotate images and their predicted labels for manual correction; and, in response to receiving the manual annotation results for the difficult-to-annotate images, adding the difficult-to-annotate images as new training sample images to the training image set to obtain an updated training image set. The target image annotation model is invoked to determine the predicted label for each training sample image in the updated training image set. The predicted label for each training sample image is obtained based on the feature matching results between the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set corresponding to the updated training image set. Based on the error between the true and predicted labels of multiple training sample images in the updated training image set, the target image annotation model is trained until it converges again.

[0009] Optionally, when the annotation accuracy of the target image annotation model reaches a preset threshold, the computer device will use the predicted label of the image to be annotated determined by the target image annotation model as the annotation label of the image to be annotated.

[0010] Optionally, the target image annotation model includes a first feature extraction layer, a second feature extraction layer, and a feature matching layer. The first feature extraction layer includes an image feature output terminal and a language feature output terminal. The feature matching layer includes an image feature input terminal and a language feature input terminal. The image feature output terminal is connected to the input terminal of the second feature extraction layer. The language feature output terminal is connected to the language feature input terminal. The output terminal of the second feature extraction layer is connected to the image feature input terminal. The process by which the computer device calls the target image annotation model corresponding to the target image task to determine the predicted label for each training sample image in the training image set includes: the computer device extracts features from each language description text in the language description text set through the first feature extraction layer to obtain a language feature set. The language feature set includes multiple sets of language features, and each set of language features is the language feature corresponding to one language description text in the language description text set. For each training sample image in the training image set, the computer device extracts features from the training sample image through a first feature extraction layer to obtain the global image features corresponding to the training sample image. Then, a second feature extraction layer extracts features from the global image features to obtain the target image features. These target image features are associated with the target image task. A feature matching layer performs feature matching between the target image features and each group of language features in the language feature set, obtaining the feature matching results. The feature matching results include the feature similarity between the target image features and each group of language features in the language feature set. The label described by the target language description text is used as the predicted label for the training sample image. The target language description text is the language description text corresponding to the group of language features in the language feature set that has the highest feature similarity to the target image features.

[0011] Optionally, the target image annotation model also includes a supervision module, which has a loss function matched to the target image task. The input of the supervision module is connected to the output of the feature matching layer. The computer device trains the target image annotation model based on the error between the ground truth labels and predicted labels of multiple training sample images in the training image set. This training includes: the computer device calculating the loss value of the loss function based on the ground truth labels and predicted labels of multiple training sample images in the training image set through the supervision module; and back-transmitting the gradient information of the loss function to the second feature extraction layer to adjust the network parameters from the second feature extraction layer to the feature matching layer.

[0012] Optionally, an image annotation model framework is pre-stored in the computer device. The image annotation model framework includes a first feature extraction layer, a downstream task model head, and a feature matching layer. The image feature output of the first feature extraction layer is connected to the input of the downstream task model head. The output of the downstream task model head is connected to the image feature input of the feature matching layer. The downstream task model head includes multiple feature extraction layers corresponding one-to-one with multiple image tasks. The second feature extraction layer is the feature extraction layer in the downstream task model head corresponding to the target image task. Specifically, the image feature output of the first feature extraction layer is configured to connect to one feature extraction layer in the downstream task model head at a time.

[0013] In this application, by designing a downstream task model head within the image annotation model framework, the framework can adapt to diverse downstream needs. This allows for the construction of corresponding image annotation models simply by selecting different feature extraction layers from the downstream task model head. Since multiple image tasks can share the same image annotation model framework, there is no need to design separate image annotation models for each task. This achieves unified and standardized management of intelligent image annotation, reducing development and maintenance complexity and thus lowering technical costs.

[0014] Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-trained model. This visual language pre-trained model can extract features from the input image to obtain image features, and it can also extract features from the input language description text to obtain language features.

[0015] Optionally, the computer device acquires the set of language description texts corresponding to the training image set in the following manner: In response to receiving a start command for a target image task, the computer device displays a template setting prompt, which prompts the user to set a language description template corresponding to the target image task. For each label corresponding to the training image set, the computer device generates a language description text based on the set language description template and the label.

[0016] This application provides a human-computer interaction interface, allowing users to manually set the language description template corresponding to the image task. This can improve the accuracy of the semantic expression of the label by the language description text, further improve the accuracy of the language features extracted based on the language description text, and thus improve the accuracy of the image annotation model.

[0017] Optionally, the computer device may display the language description text after generating it.

[0018] In this application, the computer device displays language description text, allowing the user to view whether the generated language description text accurately expresses the meaning of the label, so that the user can adjust the language description template.

[0019] Optionally, the computer device may also display multiple image tasks, with the target image task being one of the multiple image tasks. In response to detecting a selection operation for the target image task, the computer device determines that a start command for the target image task has been received.

[0020] Optionally, the multiple image tasks include, but are not limited to, one or more of image classification, object detection, or action recognition.

[0021] Secondly, an image annotation apparatus is provided. The apparatus includes multiple functional modules that interact to implement the methods described in the first aspect and its various embodiments. The multiple functional modules can be implemented using software, hardware, or a combination of both, and can be arbitrarily combined or divided based on specific implementations.

[0022] Thirdly, a computer device is provided, including: a processor and a memory;

[0023] The memory is used to store computer programs, the computer programs including program instructions;

[0024] The processor is used to invoke the computer program to implement the methods described in the first aspect and its various embodiments.

[0025] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored thereon, which, when executed by a processor, implement the methods described in the first aspect and its various embodiments.

[0026] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the methods described in the first aspect and its various embodiments.

[0027] In a sixth aspect, a chip is provided, the chip including programmable logic circuitry and / or program instructions, which, when the chip is running, implement the methods described in the first aspect and its various embodiments. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of an image annotation model framework provided in an embodiment of this application;

[0029] Figure 2 This is a flowchart illustrating an image annotation method provided in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of a display interface provided in an embodiment of this application;

[0031] Figure 4This is another schematic diagram of a display interface provided in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of the structure of a target image annotation model provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the architecture involved in an image annotation method provided in an embodiment of this application;

[0034] Figure 7 This is a schematic diagram of the structure of an image annotation device provided in an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of another image annotation device provided in an embodiment of this application;

[0036] Figure 9 This is a schematic diagram of the hardware structure of an image annotation device provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0038] Intelligent image annotation is a technology that uses AI algorithms to automatically annotate unannotated images based on a small number of already labeled images. With intelligent image annotation, users can reduce image data annotation costs by 50% to 90%. Currently, the basic process of intelligent image annotation includes the following five steps.

[0039] Step 1: Train an AI model for the corresponding image task based on the training image set, which includes multiple training sample images labeled with real labels.

[0040] Step 2: Based on the AI ​​model, reason about the image to be labeled to obtain the predicted label (also known as the pseudo label).

[0041] Step 3: Select valuable samples and their predicted labels from the inferred images according to the confidence strategy for manual correction and annotation.

[0042] Step 4: Add the manually corrected and labeled images to the training image set as new training sample images, and re-optimize the AI ​​model based on the updated training image set.

[0043] Step 5: Repeat the iterative optimization of steps 1-4 until a high-accuracy AI model is obtained. Then, use the AI ​​model for inference on all images to be labeled, output the predicted labels of the images to be labeled, and use the final output predicted labels as the final labels of the corresponding images to be labeled. This gives the intelligent image labeling results.

[0044] However, current intelligent image annotation technologies suffer from low accuracy in the first round of model training when the number of initial training sample images (images labeled with real labels) is small. The reliability of the predicted labels obtained based on AI model inference is also poor, making it difficult for the image screening process based on confidence strategy (step 3 above) to find valuable samples. This leads to a large number of model training rounds and manual correction annotation rounds, resulting in low efficiency of current image annotation.

[0045] Based on this, this application provides an image annotation method. First, a computer device acquires a training image set corresponding to an image task and a set of corresponding language description texts. The training image set includes multiple training sample images labeled with real labels. The set of language description texts includes multiple language description texts. These multiple language description texts correspond one-to-one with the multiple labels corresponding to the training image set; that is, the number of language description texts in the language description text set is the same as the number of label categories of the training sample images in the training image set. Each language description text is used to describe the semantics of one type of label corresponding to the training image set. Then, the computer device calls the image annotation model corresponding to the image task to determine the predicted label for each training sample image in the training image set. Specifically, the predicted label is obtained by the image annotation model based on the feature matching results between the image features corresponding to the training sample images and the language features corresponding to each language description text in the language description text set. Finally, the computer device trains the image annotation model corresponding to the image task based on the error between the real labels and predicted labels of the multiple training sample images in the training image set until the image annotation model converges. The finally trained image annotation model with an annotation accuracy reaching a preset threshold is used to determine the annotation label for the image to be annotated under the image task. This application's embodiments introduce prior linguistic association knowledge for the annotation task into the training sample images by obtaining the language description text set corresponding to the training image set. Furthermore, the image annotation task is transformed into a matching task between image features and language features within the image annotation model. This significantly improves the initial annotation performance of the image annotation model, enhancing the first-round annotation accuracy when the number of initial training sample images is small, effectively reducing the number of training rounds, and thus improving image annotation efficiency.

[0046] Furthermore, image-based AI application development involves a wide variety of tasks. These tasks include, but are not limited to, image classification, object detection, and action recognition. Currently, for each image task, a separate AI model needs to be designed and trained to produce an image annotation model capable of automatically annotating images for that task. Designing a separate AI model for each image task incurs high development and maintenance costs.

[0047] Optionally, embodiments of this application provide an image annotation model framework applicable to multiple image tasks. This image annotation model framework includes a first feature extraction layer, a downstream task model head, and a feature matching layer. For example, Figure 1 This is a schematic diagram of an image annotation model framework provided in an embodiment of this application. For example... Figure 1 As shown, the first feature extraction layer includes an image feature output terminal m1 and a language feature output terminal m2. The feature matching layer includes an image feature input terminal n1 and a language feature input terminal n2. The image feature output terminal m1 of the first feature extraction layer is connected to the input terminal of the downstream task model head. The language feature output terminal m2 of the first feature extraction layer is connected to the language feature input terminal n2 of the feature matching layer. The output terminal of the downstream task model head is connected to the image feature input terminal n1 of the feature matching layer. The image tasks corresponding to the downstream task model head include image classification, object detection, and action recognition.

[0048] The first feature extraction layer is used to extract features from the input language description text to obtain the corresponding language features. The first feature extraction layer is also used to extract features from the input image to obtain the corresponding global image features. Since the global image features obtained by the first feature extraction layer can reflect the global features of the entire image, different image tasks can share the first feature extraction layer. Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-trained model. The visual language pre-trained model is a model trained based on large-scale visual image data and corresponding language descriptions; the training method of the visual language pre-trained model will not be elaborated here in this embodiment.

[0049] The downstream task model head includes multiple feature extraction layers, each corresponding to a specific image task. Each feature extraction layer in the downstream task model head is used to extract image features for its corresponding image task; that is, the image features extracted by the feature extraction layer in the downstream task model head are associated with the corresponding image task. For example, the feature extraction layer corresponding to the image classification task in the downstream task model head is used to extract image region features containing the classified object. The feature extraction layer corresponding to the object detection task in the downstream task model head is used to extract image region features containing the detected object and the location features of the image region containing the detected object. The feature extraction layer corresponding to the action recognition task in the downstream task model head is used to extract image region features related to the object to be recognized. The image feature output of the first feature extraction layer is configured to connect to one feature extraction layer in the downstream task model head each time. When using the image annotation model framework, the corresponding feature extraction layer can be selected in the downstream task model head according to the image task to be executed to construct the image annotation model corresponding to that image task. Then, when using the constructed image annotation model, the selected feature extraction layer further extracts features from the global image features output by the first feature extraction layer and outputs the finally extracted image features to the feature matching layer.

[0050] The feature matching layer is used to perform feature matching between input image features and language features to obtain the feature similarity between the image features and language features. Since the function of the feature matching layer is the same across different image tasks, a unified feature matching layer can be constructed for multiple image tasks. Subsequently, during model training, the network parameters of the feature matching layer can be automatically adjusted according to the actual image task.

[0051] In this embodiment, by designing a downstream task model head within the image annotation model framework, the framework can adapt to diverse downstream needs. The corresponding image annotation model can be constructed simply by selecting different feature extraction layers from the downstream task model head. Since multiple image tasks can share the same image annotation model framework, there is no need to design separate image annotation models for each task. This achieves unified and standardized management of intelligent image annotation, reducing development and maintenance complexity and thus lowering technical costs.

[0052] The technical solution provided in this application will be described in detail below from multiple perspectives, including application scenarios, methodologies, software devices, and hardware devices.

[0053] The following are examples illustrating the application scenarios of embodiments of this application.

[0054] The image annotation method provided in this application is used in a computer device. This computer device can be a server, a server cluster consisting of several servers, or a cloud computing center. For example, the image annotation method can be applied to a cloud computing development platform as a web service. Alternatively, the image annotation method can also be applied to a server, where users interact with functions through a software user interface (UI).

[0055] The method flow of the embodiments of this application is illustrated below.

[0056] For example, Figure 2 This is a flowchart illustrating an image annotation method provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0057] Step 201: The computer device acquires the training image set corresponding to the target image task and the language description text set corresponding to the training image set.

[0058] The training image set includes multiple training sample images, each labeled with a ground truth label. These ground truth labels can be manually labeled. The language description text set includes multiple language description texts, each corresponding one-to-one with a different label class in the training image set. That is, the number of language description texts in the language description text set is the same as the number of label classes in the training image set. Each language description text describes the semantics of one of the multiple label classes.

[0059] Alternatively, the target image task can be image classification, object detection, or action recognition.

[0060] For example, the target image task is image classification, specifically classifying animal images. The training image set includes two classes of training sample images: one class with the ground truth label "dog" and the other with the ground truth label "cat," meaning the training image set corresponds to two labels. The corresponding set of descriptive text for this training image set includes two descriptive texts: one describing the presence of a dog in the image, and the other describing the presence of a cat. For example, the format of the descriptive text could be "a photo of {}", "a xx (xx is an adjective) {}", or "this is a {}", where the "{}" are used to add the ground truth labels of the training sample images.

[0061] For example, the target image task is object detection, specifically detecting litter, which requires detecting both litter and pedestrians picking up litter. The training image set includes two types of training sample images: one type contains only litter, with the ground truth label being "litter"; the other type contains pedestrians picking up litter, with the ground truth labels including both litter and pedestrians. Thus, the training image set corresponds to two types of labels. The corresponding language description text set for this training image set includes two language description texts: one describing the presence of litter in the image, and the other describing the presence of a pedestrian. For example, the two language description texts could be "there are bags of {garbage}" and "a {person} on the road," respectively. The format of the language description text for the object detection task can be "detect: {}", "there is {} on the xx (xx is a noun)" or "{}, which is xx (xx is an adjective)," where the {} are used to add the ground truth labels of the training sample images. It's worth noting that, in addition to the ground truth labels, training sample images for object detection tasks may also be labeled with ground truth (GT) locations. These GT locations reflect the region of the detected target within the image. GT locations are typically rectangular boxes.

[0062] For example, the target image task is action recognition, specifically recognizing a person's disinfection action. The training image set includes three types of training sample images: one type with the ground truth label "washing feet," another with the ground truth label "washing hands," and the third with the ground truth label "disinfecting." Thus, the training image set corresponds to three types of labels. The corresponding set of descriptive text for this training image set includes three descriptive texts: one describing the person washing feet, another describing the person washing hands, and the third describing the person disinfecting. For example, the descriptive text format could be "the man is {}", "human action of {}", or "this is {}, a frame of action," with the ground truth labels from the training sample images added within the curly braces {}.

[0063] Optionally, one implementation of the computer device acquiring the set of language description text corresponding to the training image set includes the following steps 2011 to 2012.

[0064] In step 2011, in response to receiving a start command for the target image task, the computer device displays a template setting prompt, which prompts the user to set the language description template corresponding to the target image task.

[0065] Optionally, the template setting prompt may include one or more alternative semantic description templates for the target image task. And / or, the template setting prompt may include a custom control that allows the user to input the semantic description template themselves. For example, if the target image task is image classification, Figure 3 This is a schematic diagram of a display interface provided in an embodiment of this application. This display interface is a template setting interface. For example... Figure 3 As shown, the display interface A includes a language description template option A1 corresponding to the target image task and a custom control A2. Language description template option A1 includes two language description templates: one is "a photo of {}", and the other is "this is a {}". Custom control A2 includes an input box, an add option, and a confirm option. When the computer device detects a selection operation for the add option, it uses the input content in the input box as a language description template corresponding to the target image task and reopens the input box for the user to add a new language description template. When the computer device detects a selection operation for the confirm option, it uses the input content in the input box as the language description template corresponding to the target image task and ends the template customization setting process. When setting a language description template, the user can choose a language description template provided by the computer device, or define their own language description template, or combine both. That is, the language description template for an image task ultimately determined by the user can include the language description template provided by the computer device and / or the user-defined language description template.

[0066] Optionally, after obtaining the user-defined language description template, the computer device can further fine-tune the language description template according to the specific image task being performed, so that the language description template can better match the current image task, thereby improving the accuracy of the subsequently generated language description text in describing the semantics of the labels. For example, if the current image task being performed by the computer device is to identify the category of flowers in an image, and the user-defined language description template is "a photo of {}", the computer device can fine-tune this language description template to obtain the language description template "a flower photo of {}", to more accurately express that this is a task of classifying flowers.

[0067] Optionally, before the computer device displays the template setting prompt, the computer device displays multiple image tasks, with the target image task being one of these multiple image tasks. In response to detecting a selection operation on the target image task, the computer device determines that it has received a start command for the target image task. For example, Figure 4This is a schematic diagram of another display interface provided in an embodiment of this application. This display interface is an image task startup interface. For example... Figure 4 As shown, display interface B includes image task options, which include three image tasks: image classification, object detection, and action recognition. For example, if the target image task is image classification, when the computer device detects a selection operation for image classification through display interface B, the computer device can display something like this: Figure 3 The display interface shown is A.

[0068] In step 2012, for each label corresponding to the training image set, the computer device generates a language description text based on the set language description template and the label.

[0069] For example, if the target image task is image classification, and the training image set includes two types of training sample images, one type of training sample images has a true label of "dog" and the other type of training sample images has a true label of "cat", and the user sets the language description template as "a photo of {}", then the computer device will generate two language description texts, namely "a photo of {dog}" and "a photo of {cat}".

[0070] This application provides a human-computer interaction interface, allowing users to manually set the language description template corresponding to the image task. This can improve the accuracy of the semantic expression of the label by the language description text, further improve the accuracy of the language features extracted based on the language description text, and thus improve the accuracy of the image annotation model.

[0071] Optionally, after generating the language description text, the computer device can also display the language description text. By displaying the language description text, the computer device allows the user to check whether the generated language description text accurately expresses the meaning of the label, so that the user can adjust the language description template.

[0072] Alternatively, if the user already knows all the labels corresponding to the training image set, the user can also set a language description text for each label to obtain a set of language description texts corresponding to the training image set.

[0073] Step 202: The computer device calls the target image annotation model corresponding to the target image task to determine the predicted label of each training sample image in the training image set.

[0074] The predicted labels for the training sample images are obtained by the target image annotation model based on the feature matching results between the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set.

[0075] Optionally, Figure 5This is a schematic diagram of the structure of a target image annotation model provided in an embodiment of this application. For example... Figure 5 As shown, the target image annotation model includes a first feature extraction layer, a second feature extraction layer, and a feature matching layer. The first feature extraction layer includes an image feature output terminal m1 and a language feature output terminal m2. The feature matching layer includes an image feature input terminal n1 and a language feature input terminal n2. The image feature output terminal m1 of the first feature extraction layer is connected to the input terminal of the second feature extraction layer. The language feature output terminal m2 of the first feature extraction layer is connected to the language feature input terminal n2 of the feature matching layer. The output terminal of the second feature extraction layer is connected to the image feature input terminal n1 of the feature matching layer.

[0076] Optionally, the computer device pre-stores an image annotation model framework, which may be, for example, as... Figure 1 As shown. The second feature extraction layer mentioned above is the feature extraction layer in the downstream task model header that corresponds to the target image task. Optionally, in response to receiving a start command for the target image task, the computer device selects the feature extraction layer corresponding to the target image task in the downstream task model header to construct the target image annotation model.

[0077] Optionally, combined with, for example Figure 5 The target image annotation model shown above, the implementation process of step 202 may include steps 2021 to 2022.

[0078] In step 2021, the computer device extracts features from each language description text in the language description text set through the first feature extraction layer to obtain a language feature set.

[0079] The language feature set includes multiple sets of language features, each set representing the language feature corresponding to one language description text in the language description text set. Optionally, the language description text set includes M language description texts, and the language feature corresponding to the m-th language description text is denoted as L. m Then the set of language features can be represented as: L = {L1, L2, ..., L...} m , ..., L M}. Where M is an integer greater than 1, 1≤m≤M. The first feature extraction layer extracts features from each language description text in the input language description text set and outputs the resulting language feature set to the feature matching layer.

[0080] In step 2022, for each training sample image in the training image set, the computer device executes the label prediction process to obtain the predicted label for each training sample image.

[0081] The label prediction process includes the following steps S1 to S4.

[0082] In step S1, the computer device extracts features from the training sample image through the first feature extraction layer to obtain the global image features corresponding to the training sample image.

[0083] Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-trained model. Global image features are used to reflect the global features of the entire image and are independent of the image task. After extracting features from the input training sample image, the first feature extraction layer outputs the obtained global image features to the second feature extraction layer.

[0084] In step S2, the computer device extracts features from the global image through the second feature extraction layer to obtain the target image features, which are then associated with the target image task.

[0085] Optionally, the second feature extraction layer employs a feature extraction algorithm matched to the target image task to extract features from the global image features. The second feature extraction layer is primarily used to extract feature sets with strong category relevance from the global image features. For example, if the target image task is image classification or action recognition, the second feature extraction layer is used to extract image region features containing the classified object or the object to be recognized. As another example, if the target image task is object detection, the second feature extraction layer is used to extract image region features containing the detected object and the location features of the image region containing the detected object. The location features of the image region containing the detected object can be represented by the predicted bounding box position. The predicted bounding box is typically a rectangle. After extracting features from the input global image features, the second feature extraction layer outputs the obtained target image features to the feature matching layer.

[0086] In step S3, the computer device performs feature matching between the target image features and each group of language features in the language feature set through the feature matching layer to obtain the feature matching result, which includes the feature similarity between the target image features and each group of language features in the language feature set.

[0087] Optionally, the feature similarity between image features and language features can be cosine similarity or Euclidean similarity, etc.

[0088] Optionally, the target image task is image classification or action recognition. For training sample images in the training image set, the image features output by the second feature extraction layer are F, where i is a positive integer. The language feature set L = {L1, L2, ..., L...} m , ..., L M}

[0089] The feature matching result obtained by the feature matching layer based on the input linguistic feature set and the target image features can be expressed as: Logits = func_similarity(F, L). Here, func_similarity is a function that calculates the feature similarity between the image features and each set of linguistic features in the linguistic feature set.

[0090] Optionally, the target image task is object detection. For training sample images in the training image set, the second feature extraction layer can extract N region features, which can be represented as: O F ={O1, O2, ..., O n , ..., O N}. Where N is an integer greater than 1, 1≤n≤N. The language feature set L={L1, L2, …, L m , ..., L M The feature matching result obtained by the feature matching layer based on the input language feature set and the target image features can be expressed as: Logits = O F L T L T Let L be the transpose of L. The feature matching result includes the feature similarity between each region feature and each set of language features.

[0091] In step S4, the computer device uses the labels described by the target language description text as the predicted labels of the training sample images. The target language description text is the language description text corresponding to the set of language features in the language feature set that has the highest feature similarity to the target image features.

[0092] For object detection tasks, an image may contain multiple objects to be detected. For each object in the image, the computer device uses the set of linguistic features with the highest similarity to the image features of that object, corresponding to the linguistic description text, as the predicted label for that object. Finally, the predicted labels of all objects in the image are used as the predicted label for the entire image. For example, if the task is to detect trash and pedestrians carrying trash from an image, if the image only contains trash, the predicted label is "trash." If the image contains pedestrians carrying trash, the predicted label includes both "trash" and "pedestrian."

[0093] Optionally, step S4 can be performed by the feature matching layer. That is, after performing feature matching on image and language features to obtain the feature matching result, the feature matching layer determines the predicted label of the training sample image based on the feature matching result, and finally outputs the predicted label of the training sample image. Alternatively, step S4 can also be performed by other modules in the image annotation model (e.g., the supervision module) based on the output result of the feature matching layer. Alternatively, step S4 can be determined separately by a computer device based on the output result of the feature matching layer. In the latter two cases, after the feature matching layer performs feature matching on image and language features to obtain the feature matching result, it can directly output the feature matching result.

[0094] Step 203: The computer equipment trains the target image annotation model based on the error between the true labels and predicted labels of multiple training sample images in the training image set until the target image annotation model converges.

[0095] Optionally, please continue to see Figure 5 The target image annotation model also includes a supervision module, which contains a loss function tailored to the target image task. The input of the supervision module is connected to the output of the feature matching layer. See also... Figure 5 The feature matching layer can output feature matching results to the supervision module, which then determines the predicted labels of the training sample images based on the input feature matching results.

[0096] Optionally, the implementation process of step 203 above may include steps 2031 to 2032.

[0097] In step 2031, the computer device calculates the loss value of the loss function based on the real and predicted labels of multiple training sample images in the training image set through the supervision module.

[0098] Optionally, the target image task is image classification or action recognition. The loss function set in the supervision module can then be expressed as: loss = func_classify(Logits, G). Here, G is the true label of the training sample image. The meaning of Logits refers to the relevant definition in step S3 above when the target image task is image classification or action recognition. func_classify is a function that calculates the classification loss based on the label, such as the cross-entropy loss function, focal loss (a hard sample mining loss) function, or its variants.

[0099] Optionally, the target image task is object detection. Then, the loss function set in the supervision module consists of two parts: classification loss and localization loss. The classification loss function can be expressed as: loss_cls = func_classify(Logits, Tc ), where T c To determine the true labels for the training sample images, assuming the training sample images contain K detection targets, then T c ={C1, ..., C k C K}. K is a positive integer, 1≤k≤K. The meaning of Logits refers to the relevant definition in step S3 above when the target image task is object detection. The localization loss function can be expressed as: loss_loc=func_iou(O B T B ), among which, O B For the N region features extracted by the second feature extraction layer, the N predicted box positions are respectively given. B The output from the second feature extraction layer can be sent to the supervision module. B The positions of the K ground truth bounding boxes corresponding to the K detected targets in the training sample image can be represented as: T B ={Box1, ..., Box} k Box K `func_iou` is a function that calculates the intersection over union (IOU) between the predicted bounding box and the ground truth bounding box, such as the generalized IOU (GIOU) and complete IOU (IOU) loss functions. In object detection tasks, the loss function can be the sum of the loss value of the classification loss function and the loss value of the localization loss function.

[0100] In step 2032, the computer device transmits the gradient information of the loss function back to the second feature extraction layer to adjust the network parameters from the second feature extraction layer to the feature matching layer.

[0101] Alternatively, step 2032 above can be replaced by: the computer device transmitting the gradient information of the loss function back to the first feature extraction layer to adjust the network parameters from the first feature extraction layer to the feature matching layer. That is, the first feature extraction layer can remain unchanged during model training, or it can be fine-tuned during model training based on the actual image task.

[0102] In one round of model training, the computer repeatedly trains the image annotation model based on the same training image set (i.e., continuously adjusts the network parameters) until the loss function converges, thus obtaining the converged image annotation model for that round of training. Convergence of the loss function can be defined as the loss value reaching a preset value.

[0103] It is worth noting that during model training, a supervision module can be set in the image annotation model. After the model training is completed, since the supervision module no longer plays a role in reverse tuning of network parameters, the supervision module in the image annotation model can be deleted or retained.

[0104] Steps 201 to 203 above describe one round of training of the target image annotation model. After completing one round of training, the computer device can further verify the accuracy of the target image annotation model. If the accuracy of the target image annotation model reaches the preset requirement, the computer device stops training the target image annotation model and uses the finally trained target image annotation model to determine the annotation label of the image to be annotated under the target image task. If the accuracy of the target image annotation model does not reach the preset requirement, the computer device continues to train the target image annotation model for a new round. This process is repeated iteratively to update the target image annotation model until the accuracy of the target image annotation model reaches the preset requirement. For the specific implementation process, please refer to steps 204 to 207.

[0105] Step 204: After the target image annotation model converges, the computer device calls the target image annotation model to determine the predicted labels of multiple validation sample images in the validation image set.

[0106] The verification image set includes multiple verification sample images, each labeled with a real label. The real labels of the verification sample images can be manually labeled. The predicted labels of the verification sample images are obtained by the target image annotation model based on the feature matching results between the image features corresponding to the verification sample image and the language features corresponding to each language description text in the language description text set. The method by which the target image annotation model predicts and determines the predicted labels of the verification sample images can refer to the method for determining the predicted labels of the training sample images in step 202 above, and will not be repeated here in this embodiment.

[0107] Optionally, the validation image set and the training image set have no overlap. The validation image set can be a fixed set of images, meaning the validation sample images in the validation image set remain unchanged. For example, if the intelligent annotation dataset includes 1000 images, of which 100 images are labeled with real labels and the remaining 900 images are images to be labeled, then the 50 images labeled with real labels can be used as validation sample images to obtain the validation image set. The other 50 images labeled with real labels can be used as training sample images to obtain the initial training image set.

[0108] Step 205: The computer device determines the annotation accuracy of the target image annotation model based on the real and predicted labels of multiple verification sample images.

[0109] For example, if the validation image set includes 50 validation sample images, and the true labels of 30 of these validation sample images are the same as the predicted labels, then the annotation accuracy of the target image annotation model is 60%.

[0110] Step 206: When the annotation accuracy of the target image annotation model does not reach the preset threshold, the computer device performs one or more model training processes until the annotation accuracy of the target image annotation model reaches the preset threshold.

[0111] In this embodiment of the application, when there are few initial training sample images, the images to be labeled can be used together for model training. Combining an active learning strategy, difficult examples in the images to be labeled are selected and manually corrected and labeled. These manually corrected and labeled images are then added to the training image set as new training sample images, thereby expanding the size of the training image set and performing a new round of model training to improve model accuracy. The above model training process includes steps 2061 to 2066.

[0112] In step 2061, the computer device invokes the target image annotation model to determine the predicted labels of multiple images to be annotated and the confidence level of the predicted labels of the images to be annotated.

[0113] In step 2062, the computer device acquires images from multiple images to be labeled that are difficult to label, where the confidence level of the predicted label is lower than the confidence level threshold.

[0114] In step 2063, the computer device outputs the difficult-to-label image and its predicted label for manual correction.

[0115] In step 2064, in response to receiving the manual annotation results for the difficult-to-annotate images, the computer device adds the difficult-to-annotate images as new training sample images to the training image set, thereby obtaining an updated training image set.

[0116] The manual annotation results include the ground truth labels for images that are difficult to annotate manually. Steps 2061 to 2064 above constitute the active learning process in the model training process, i.e., the part involving human intervention, which is an iterative learning process in which a suitable candidate set is selected by computer equipment for manual annotation.

[0117] In step 2065, the computer device invokes the target image annotation model to determine the predicted label for each training sample image in the updated training image set. The predicted label for the training sample image is obtained based on the feature matching results between the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set corresponding to the updated training image set.

[0118] Optionally, if the labels corresponding to the updated training image set change, the computer device re-acquires the language description text set corresponding to the updated training image set. For specific implementation methods, please refer to steps 2011 to 2012 above. The embodiments of this application will not be repeated here.

[0119] In step 2066, the computer device trains the target image annotation model based on the error between the true labels and predicted labels of multiple training sample images in the updated training image set, until the target image annotation model converges again.

[0120] The implementation process of step 2066 can refer to the implementation process of step 203 above, and will not be repeated here in the embodiments of this application.

[0121] Step 207: When the annotation accuracy of the target image annotation model reaches the preset threshold, the computer device will call the predicted label of the image to be annotated determined by the target image annotation model as the annotation label of the image to be annotated.

[0122] Optionally, the computer device can further output annotation labels for all images to be annotated.

[0123] For example, Figure 6 This is a schematic diagram of the architecture involved in an image annotation method provided in an embodiment of this application. For example... Figure 6 As shown, this architecture includes a data storage medium, a processor, and an interactive UI. The data storage medium stores the intelligent annotation dataset, which includes labeled images and images to be labeled. The processor can be a central processing unit (CPU) or a graphics processing unit (GPU). The processor runs and trains the image annotation model, which includes a cascaded visual-language pre-trained model, a downstream task model head, and a language-image feature matching layer. The visual-language pre-trained model is also connected to the language-image feature matching layer. The interactive UI allows users to add language descriptions, including setting language description templates and batch adding language description text. The interactive UI also outputs annotation results, such as displaying the predicted labels of the images to be labeled by the image annotation model and presenting active learning / hard example mining results for users to manually correct annotations, etc. The downstream task model head includes, but is not limited to, image tasks such as image classification, object detection, and action recognition.

[0124] In summary, the image annotation method provided in this application embodiment obtains the language description text set corresponding to the training image set, introduces prior language association knowledge of the annotation task into the training sample images, and transforms the image task into a matching task of image features and language features in the image annotation model. This can significantly improve the initial annotation performance of the image annotation model, increase the first-round annotation accuracy of the image annotation model when the number of initial training sample images is small, effectively reduce the number of training rounds of the image annotation model, and thus improve the efficiency of image annotation. In addition, this application embodiment can also provide an image annotation model framework applicable to multiple image tasks. By designing downstream task model heads in the image annotation model framework, the image annotation model framework can adapt to diverse downstream needs. Only different feature extraction layers in the downstream task model heads need to be selected to construct the corresponding image annotation model. Since multiple image tasks can share the image annotation model framework, it is not necessary to design corresponding image annotation models for each image task separately, realizing unified and standardized management of intelligent image annotation, reducing the complexity of development and maintenance, and thus reducing technical costs.

[0125] The order of steps in the image annotation method provided in this application can be adjusted appropriately, and steps can be added or removed as needed. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. For example, displaying information on a computer device can be done by the computer device displaying information on its own display interface, or by the computer device sending information to other display devices for display.

[0126] The following describes an example of a virtual device in an embodiment of this application.

[0127] For example, Figure 7 This is a schematic diagram of the structure of an image annotation device provided in an embodiment of this application. For example... Figure 7 As shown, the image annotation device 700 includes: an acquisition module 701, a determination module 702, and a training module 703.

[0128] The acquisition module 701 is used to acquire the training image set corresponding to the target image task and the language description text set corresponding to the training image set. The training image set includes multiple training sample images, each of which is labeled with a real label. The language description text set includes multiple language description texts, each of which corresponds one-to-one with the multi-class labels corresponding to the training image set. Each language description text is used to describe the semantics of one of the multi-class labels.

[0129] The determination module 702 is used to call the target image annotation model corresponding to the target image task to determine the predicted label of each training sample image in the training image set. The predicted label of the training sample image is obtained by the target image annotation model based on the feature matching results of the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set.

[0130] The training module 703 is used to train the target image annotation model based on the error between the true labels and predicted labels of multiple training sample images in the training image set until the target image annotation model converges. The target image annotation model is used to determine the annotation label of the image to be annotated under the target image task.

[0131] Optionally, the determining module 702 is further configured to, after the target image annotation model converges, call the target image annotation model to determine the predicted labels of multiple validation sample images in the validation image set. The validation image set includes multiple validation sample images, each of which is labeled with a ground truth label. The determining module 702 is further configured to determine the annotation accuracy of the target image annotation model based on the ground truth labels and predicted labels of the multiple validation sample images. The training module 703 is further configured to, when the annotation accuracy of the target image annotation model does not reach a preset threshold, perform one or more model training processes until the annotation accuracy of the target image annotation model reaches the preset threshold.

[0132] The model training process includes: calling the target image annotation model to determine the predicted labels and confidence levels of multiple images to be labeled. Retrieving hard-labeled images from the multiple images to be labeled whose predicted label confidence levels are below a certain threshold. Outputting the hard-labeled images and their predicted labels for manual correction. Responding to the received manual annotation results for the hard-labeled images, adding them as new training sample images to the training image set, resulting in an updated training image set. Calling the target image annotation model to determine the predicted label for each training sample image in the updated training image set. The predicted label for each training sample image is obtained based on the feature matching results between the image features corresponding to the training sample image and the linguistic features corresponding to each linguistic description text in the updated training image set. Training the target image annotation model based on the errors between the true and predicted labels of multiple training sample images in the updated training image set until the target image annotation model converges again.

[0133] Optionally, the determining module 702 is further configured to, when the labeling accuracy of the target image labeling model reaches a preset threshold, use the predicted label of the image to be labeled determined by the target image labeling model as the labeling label of the image to be labeled.

[0134] Optionally, the target image annotation model includes a first feature extraction layer, a second feature extraction layer, and a feature matching layer. The first feature extraction layer includes an image feature output terminal and a language feature output terminal. The feature matching layer includes an image feature input terminal and a language feature input terminal. The image feature output terminal is connected to the input terminal of the second feature extraction layer, and the language feature output terminal is connected to the language feature input terminal. The output terminal of the second feature extraction layer is connected to the image feature input terminal. The determining module 702 is used to: extract features from each language description text in the language description text set through the first feature extraction layer to obtain a language feature set. The language feature set includes multiple sets of language features, and each set of language features corresponds to the language feature of one language description text in the language description text set. For each training sample image in the training image set, features are extracted from the training sample image through the first feature extraction layer to obtain the global image features corresponding to the training sample image. Features are then extracted from the global image features through the second feature extraction layer to obtain the target image features. The target image features are associated with the target image task. The target image features are matched with each group of language features in the language feature set through the feature matching layer to obtain the feature matching results. The feature matching results include the feature similarity between the target image features and each group of language features in the language feature set. The labels described by the target language description text are used as the predicted labels of the training sample images. The target language description text is the language description text corresponding to the group of language features in the language feature set that has the highest feature similarity with the target image features.

[0135] Optionally, the target image annotation model also includes a supervision module, which has a loss function matched to the target image task. The input of the supervision module is connected to the output of the feature matching layer. The training module 703 is used to: calculate the loss value of the loss function based on the real and predicted labels of multiple training sample images in the training image set using the supervision module; and backpropagate the gradient information of the loss function to the second feature extraction layer to adjust the network parameters from the second feature extraction layer to the feature matching layer.

[0136] Optionally, the computer device pre-stores an image annotation model framework, which includes a first feature extraction layer, a downstream task model head, and a feature matching layer. The image feature output end is connected to the input end of the downstream task model head, and the output end of the downstream task model head is connected to the image feature input end of the feature matching layer. The downstream task model head includes multiple feature extraction layers that correspond one-to-one with multiple image tasks. The second feature extraction layer is the feature extraction layer in the downstream task model head that corresponds to the target image task. The image feature output end is configured to connect to one of the feature extraction layers in the downstream task model head each time.

[0137] Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-trained model.

[0138] Optionally, such as Figure 8 As shown, the image annotation device 700 also includes a display module 704.

[0139] Optionally, the display module 704 is configured to display a template setting prompt in response to receiving a start command for the target image task. The template setting prompt is used to prompt the user to set a language description template corresponding to the target image task. The acquisition module 701 is configured to generate a language description text for each category label corresponding to the training image set, based on the set language description template and the label.

[0140] Optionally, the display module 704 is also configured to display the language description text after it has been generated.

[0141] Optionally, the display module 704 is further configured to display multiple image tasks, with the target image task being one of the multiple image tasks. The acquisition module 701 is configured to determine, in response to detecting a selection operation on the target image task, that a start command for the target image task has been received.

[0142] Optionally, the multiple image tasks include one or more of image classification, object detection, or action recognition.

[0143] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0144] The following provides examples illustrating the basic hardware structures involved in the embodiments of this application.

[0145] For example, Figure 9 This is a schematic diagram of the hardware structure of an image annotation device provided in an embodiment of this application. For example... Figure 9 As shown, the image annotation device 900 includes a processor 901 and a memory 902, and the memory 901 and memory 902 are connected via a bus 903. Figure 9 The processor 901 and memory 902 are described independently. Alternatively, the processor 901 and memory 902 are integrated together. Alternatively, Figure 9 The image annotation device 900 in the image annotation device is any computer device with computing capabilities.

[0146] The memory 902 is used to store computer programs, including the operating system and program code. The memory 902 can be various types of storage media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), flash memory, optical storage, registers, optical disc storage, disk storage, or other magnetic storage devices.

[0147] The processor 901 is a general-purpose processor or a special-purpose processor. The processor 901 may be a single-core processor or a multi-core processor. The processor 901 includes at least one circuit to execute the image annotation method described above in the embodiments of this application.

[0148] Optionally, the image annotation apparatus 900 also includes a network interface 904, which is connected to the processor 901 and the memory 902 via a bus 903. The network interface 904 enables the image annotation apparatus 900 to communicate with other devices.

[0149] Optionally, the image annotation device 900 also includes an input / output (I / O) interface 905, which is connected to the processor 901 and the memory 902 via a bus 903. The processor 901 can receive input commands or data through the I / O interface 905. The I / O interface 905 is used for connecting the image annotation device 900 to input devices, such as a keyboard or mouse. Optionally, in some possible scenarios, the network interface 904 and the I / O interface 905 are collectively referred to as a communication interface.

[0150] Optionally, the image annotation device 900 further includes a display 906, which is connected to the processor 901 and the memory 902 via a bus 903. The display 906 can be used to display intermediate and / or final results generated by the processor 901 executing the above methods, such as displaying image tasks, template setting prompts, and descriptive text. In one possible implementation, the display 906 is a touch screen to provide a human-computer interaction interface.

[0151] Wherein, bus 903 can be any type of communication bus used to interconnect the internal devices of image annotation apparatus 900. For example, a system bus. This embodiment of the application illustrates the interconnection of the aforementioned devices inside image annotation apparatus 900 via bus 903 as an example. Optionally, the aforementioned devices inside image annotation apparatus 900 may communicate with each other using connection methods other than bus 903, such as interconnecting the aforementioned devices inside image annotation apparatus 900 via internal logic interfaces.

[0152] The aforementioned devices can be disposed on separate chips, or at least partially or entirely on the same chip. Whether to dispose of the devices independently on different chips or integrate them on one or more chips often depends on the needs of the product design. This application does not limit the specific implementation of the aforementioned devices.

[0153] Figure 9 The image annotation device 900 shown is merely an example. In its implementation, the image annotation device 900 includes other components, which will not be listed here. Figure 9 The image annotation device 900 shown can perform intelligent annotation of images by executing all or part of the steps of the method provided in the above embodiments.

[0154] This application embodiment also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement... Figure 2 The image annotation method shown.

[0155] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the following: Figure 2 The image annotation method shown.

[0156] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0157] In the embodiments of this application, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0158] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0159] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the image data involved in this application were all obtained with full authorization.

[0160] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of this application should be included within the protection scope of this application.

Claims

1. An image annotation method, characterized in that, For use in a computer device, the method includes: Obtain the training image set corresponding to the target image task and the language description text set corresponding to the training image set. The training image set includes multiple training sample images, each of which is labeled with a real label. The language description text set includes multiple language description texts, each of which corresponds one-to-one with the multi-class labels corresponding to the training image set. Each language description text is used to describe the semantics of one of the multi-class labels. The target image annotation model corresponding to the target image task is invoked to determine the predicted label of each training sample image in the training image set. The predicted label of the training sample image is obtained by the target image annotation model based on the feature matching results of the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set. The target image annotation model is trained based on the error between the true and predicted labels of multiple training sample images in the training image set until the target image annotation model converges. The target image annotation model is used to determine the annotation label of the image to be annotated under the target image task.

2. The method according to claim 1, characterized in that, After the target image annotation model converges, the method further includes: The target image annotation model is invoked to determine the predicted labels of multiple validation sample images in the validation image set, wherein the validation image set includes multiple validation sample images, and each of the multiple validation sample images is annotated with a real label; The labeling accuracy of the target image labeling model is determined based on the true and predicted labels of the multiple verification sample images. When the annotation accuracy of the target image annotation model does not reach the preset threshold, one or more model training processes are performed until the annotation accuracy of the target image annotation model reaches the preset threshold. The model training process includes: The target image annotation model is invoked to determine the predicted labels of multiple images to be labeled and the confidence level of the predicted labels of the images to be labeled; From the plurality of unlabeled images, obtain the images that are difficult to label, whose predicted label confidence is lower than the confidence threshold; Output the difficult-to-label image and its predicted label for manual correction; In response to receiving the manual annotation results for the difficult-to-annotate image, the difficult-to-annotate image is added as a new training sample image to the training image set to obtain an updated training image set; The target image annotation model is invoked to determine the predicted label of each training sample image in the updated training image set. The predicted label of the training sample image is obtained based on the feature matching results between the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set corresponding to the updated training image set. The target image annotation model is trained based on the error between the true and predicted labels of multiple training sample images in the updated training image set until the target image annotation model converges again.

3. The method according to claim 2, characterized in that, The method further includes: When the annotation accuracy of the target image annotation model reaches the preset threshold, the predicted label of the image to be annotated determined by the target image annotation model will be used as the annotation label of the image to be annotated.

4. The method according to any one of claims 1 to 3, characterized in that, The target image annotation model includes a first feature extraction layer, a second feature extraction layer, and a feature matching layer. The first feature extraction layer includes an image feature output terminal and a language feature output terminal. The feature matching layer includes an image feature input terminal and a language feature input terminal. The image feature output terminal is connected to the input terminal of the second feature extraction layer, and the language feature output terminal is connected to the language feature input terminal. The output terminal of the second feature extraction layer is connected to the image feature input terminal. The step of calling the target image annotation model corresponding to the target image task to determine the predicted label for each training sample image in the training image set includes: The first feature extraction layer extracts features from each language description text in the language description text set to obtain a language feature set. The language feature set includes multiple sets of language features, and each set of language features is a language feature corresponding to a language description text in the language description text set. For each training sample image in the training image set, The first feature extraction layer extracts features from the training sample image to obtain the global image features corresponding to the training sample image. The second feature extraction layer extracts features from the global image to obtain target image features, which are then associated with the target image task. The feature matching layer performs feature matching between the target image features and each group of language features in the language feature set, respectively, to obtain feature matching results. The feature matching results include the feature similarity between the target image features and each group of language features in the language feature set. The labels described by the target language descriptive text are used as the predicted labels of the training sample images. The target language descriptive text is the language descriptive text corresponding to the set of language features in the language feature set that has the highest feature similarity to the target image features.

5. The method according to claim 4, characterized in that, The target image annotation model further includes a supervision module, which has a loss function matched to the target image task. The input of the supervision module is connected to the output of the feature matching layer. Training the target image annotation model based on the error between the true and predicted labels of multiple training sample images in the training image set includes: The supervision module calculates the loss value of the loss function based on the true and predicted labels of multiple training sample images in the training image set. The gradient information of the loss function is transmitted in reverse to the second feature extraction layer to adjust the network parameters from the second feature extraction layer to the feature matching layer.

6. The method according to claim 4, characterized in that, The computer device pre-stores an image annotation model framework, which includes a first feature extraction layer, a downstream task model head, and a feature matching layer. The image feature output end is connected to the input end of the downstream task model head, and the output end of the downstream task model head is connected to the image feature input end of the feature matching layer. The downstream task model head includes multiple feature extraction layers that correspond one-to-one with multiple image tasks. The second feature extraction layer is the feature extraction layer in the downstream task model head that corresponds to the target image task. The image feature output end is configured to connect to one feature extraction layer in the downstream task model head each time.

7. The method according to claim 4, characterized in that, The first feature extraction layer is implemented by a pre-trained visual language pre-trained model.

8. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the language description text set corresponding to the training image set includes: In response to receiving a start command for the target image task, a template setting prompt is displayed, which prompts the user to set a language description template corresponding to the target image task; For each label corresponding to the training image set, a language description text is generated based on the set language description template and the label.

9. The method according to claim 8, characterized in that, The method further includes: After the language description text is generated, the language description text is displayed.

10. The method according to claim 8, characterized in that, The method further includes: Displaying multiple image tasks, wherein the target image task is one of the multiple image tasks; In response to detecting a selection operation for the target image task, it is determined that a start command for the target image task has been received.

11. The method according to claim 6 or 10, characterized in that, The multiple image tasks include one or more of image classification, object detection, or action recognition.

12. An image annotation device, characterized in that, For a computer device, the means includes: The acquisition module is used to acquire a training image set corresponding to the target image task and a language description text set corresponding to the training image set. The training image set includes multiple training sample images, each of which is labeled with a real label. The language description text set includes multiple language description texts, each of which corresponds one-to-one with a multi-class label corresponding to the training image set. Each language description text is used to describe the semantics of one of the multi-class labels. The determination module is used to call the target image annotation model corresponding to the target image task to determine the predicted label of each training sample image in the training image set. The predicted label of the training sample image is obtained by the target image annotation model based on the feature matching results of the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set. The training module is used to train the target image annotation model based on the error between the true labels and predicted labels of multiple training sample images in the training image set until the target image annotation model converges. The target image annotation model is used to determine the annotation label of the image to be annotated under the target image task.

13. The apparatus according to claim 12, characterized in that, The determining module is further configured to, after the target image annotation model converges, call the target image annotation model to determine the predicted labels of multiple verification sample images in the verification image set, wherein the verification image set includes multiple verification sample images, and each of the multiple verification sample images is labeled with a real label; The determining module is further configured to determine the labeling accuracy of the target image labeling model based on the real labels and predicted labels of the plurality of verification sample images; The training module is also used to perform one or more model training processes when the annotation accuracy of the target image annotation model does not reach the preset threshold, until the annotation accuracy of the target image annotation model reaches the preset threshold. The model training process includes: The target image annotation model is invoked to determine the predicted labels of multiple images to be labeled and the confidence level of the predicted labels of the images to be labeled; From the plurality of unlabeled images, obtain the images that are difficult to label, whose predicted label confidence is lower than the confidence threshold; Output the difficult-to-label image and its predicted label for manual correction; In response to receiving the manual annotation results for the difficult-to-annotate image, the difficult-to-annotate image is added as a new training sample image to the training image set to obtain an updated training image set; The target image annotation model is invoked to determine the predicted label of each training sample image in the updated training image set. The predicted label of the training sample image is obtained based on the feature matching results between the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set corresponding to the updated training image set. The target image annotation model is trained based on the error between the true and predicted labels of multiple training sample images in the updated training image set until the target image annotation model converges again.

14. The apparatus according to claim 13, characterized in that, The determining module is further configured to, when the labeling accuracy of the target image labeling model reaches the preset threshold, use the predicted label of the image to be labeled determined by the target image labeling model as the labeling label of the image to be labeled.

15. The apparatus according to any one of claims 12 to 14, characterized in that, The target image annotation model includes a first feature extraction layer, a second feature extraction layer, and a feature matching layer. The first feature extraction layer includes an image feature output terminal and a language feature output terminal. The feature matching layer includes an image feature input terminal and a language feature input terminal. The image feature output terminal is connected to the input terminal of the second feature extraction layer. The language feature output terminal is connected to the language feature input terminal. The output terminal of the second feature extraction layer is connected to the image feature input terminal. The determining module is used for: The first feature extraction layer extracts features from each language description text in the language description text set to obtain a language feature set. The language feature set includes multiple sets of language features, and each set of language features is a language feature corresponding to a language description text in the language description text set. For each training sample image in the training image set, The first feature extraction layer extracts features from the training sample image to obtain the global image features corresponding to the training sample image. The second feature extraction layer extracts features from the global image to obtain target image features, which are then associated with the target image task. The feature matching layer performs feature matching between the target image features and each group of language features in the language feature set, respectively, to obtain feature matching results. The feature matching results include the feature similarity between the target image features and each group of language features in the language feature set. The labels described by the target language descriptive text are used as the predicted labels of the training sample images. The target language descriptive text is the language descriptive text corresponding to the set of language features in the language feature set that has the highest feature similarity to the target image features.

16. The apparatus according to claim 15, characterized in that, The target image annotation model further includes a supervision module, which has a loss function matched to the target image task. The input of the supervision module is connected to the output of the feature matching layer. The training module is used for: The supervision module calculates the loss value of the loss function based on the true and predicted labels of multiple training sample images in the training image set. The gradient information of the loss function is transmitted in reverse to the second feature extraction layer to adjust the network parameters from the second feature extraction layer to the feature matching layer.

17. The apparatus according to claim 15, characterized in that, The computer device pre-stores an image annotation model framework, which includes a first feature extraction layer, a downstream task model head, and a feature matching layer. The image feature output end is connected to the input end of the downstream task model head, and the output end of the downstream task model head is connected to the image feature input end of the feature matching layer. The downstream task model head includes multiple feature extraction layers that correspond one-to-one with multiple image tasks. The second feature extraction layer is the feature extraction layer in the downstream task model head that corresponds to the target image task. The image feature output end is configured to connect to one feature extraction layer in the downstream task model head each time.

18. The apparatus according to claim 15, characterized in that, The first feature extraction layer is implemented by a pre-trained visual language pre-trained model.

19. The apparatus according to any one of claims 12 to 14, characterized in that, The device further includes: a display module; The display module is configured to display a template setting prompt in response to receiving a start command for the target image task. The template setting prompt is used to prompt the user to set the language description template corresponding to the target image task. The acquisition module is used to generate a language description text for each type of label corresponding to the training image set, based on the set language description template and the label.

20. The apparatus according to claim 19, characterized in that, The display module is further configured to display the language description text after it has been generated.

21. The apparatus according to claim 19, characterized in that, The display module is also used to display multiple image tasks, wherein the target image task is one of the multiple image tasks; The acquisition module is configured to determine, in response to detecting a selection operation for the target image task, that a start command for the target image task has been received.

22. The apparatus according to claim 17 or 21, characterized in that, The multiple image tasks include one or more of image classification, object detection, or action recognition.

23. A computer device, characterized in that, include: Processor and memory; The memory is used to store computer programs, the computer programs including program instructions; The processor is used to invoke the computer program to implement the image annotation method as described in any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a processor, implement the image annotation method as described in any one of claims 1 to 11.

25. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the image annotation method as described in any one of claims 1 to 11.

Images