Image labeling method and device, electronic equipment and storage medium

Abstract

The application relates to an image labeling method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a to-be-labeled image, calling a multi-modal large model to perform global understanding on the to-be-labeled image, and identifying the potential semantic label of the to-be-labeled image; extracting the detection box coordinates corresponding to the potential semantic label according to the potential semantic label; determining the target category label corresponding to the potential semantic label based on a preset category mapping relationship; generating a standard labeled image according to the target category label and the detection box coordinates; and labeling the image for training a target image detection model. Through the above method, a unified label naming system is constructed, the labels can be mapped, the problem of one object having multiple names in the image labeling process is solved, and the uniformity of data set label naming is significantly improved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and artificial intelligence, and in particular to image annotation methods, devices, electronic devices and storage media. Background Technology

[0002] In the field of deep learning object detection, such as the YOLO algorithm (an object detection algorithm in the field of computer vision), the model performance is highly dependent on large-scale, high-quality labeled datasets. However, traditional manual labeling methods are inefficient and extremely costly.

[0003] To address this issue, relevant techniques typically employ pre-trained, fixed-category detection models on publicly available datasets to perform forward inference on newly input, unlabeled images, outputting bounding boxes and corresponding labels based on the model's built-in category system. However, this approach results in inconsistent label names generated by different models or different versions of label naming systems, leading to inconsistent dataset naming.

[0004] In summary, the existing automatic labeling solutions suffer from inconsistent label naming, making it difficult to meet the needs of actual engineering projects. Summary of the Invention

[0005] To address or partially address the problems existing in related technologies, this application provides an image annotation method, apparatus, electronic device, and storage medium, which constructs a unified label naming system, enables label mapping, solves the problem of multiple names for one object in the image annotation process, and significantly improves the uniformity of dataset label naming.

[0006] The first aspect of this application provides an image annotation method, including: The image to be labeled is obtained, and a multimodal large model is invoked to perform a global understanding of the image to be labeled, and the potential semantic labels of the image to be labeled are identified. Extract the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label; Based on a preset category mapping relationship, the target category label corresponding to the potential semantic label is determined; A standard labeled image is generated based on the target category label and the coordinates of the detection box; the labeled image is used to train the target image detection model.

[0007] In a preferred embodiment, extracting the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label includes: The latent semantic labels are input into the trained general detection model; The general detection model outputs the coordinates of the corresponding detection box based on the latent semantic labels.

[0008] In a preferred embodiment, determining the target category label corresponding to the latent semantic label based on a preset category mapping relationship includes: A pre-defined category mapping relationship between the latent semantic tags and the target category tags is established based on a large language model; The target category label corresponding to the latent semantic label is determined based on the latent semantic label and the preset category mapping relationship.

[0009] In a preferred embodiment, the latent semantic tags include redundant tags and / or biased tags; the method further includes: Add the mapping relationship between the redundant labels and / or deviation labels and the target category labels to the preset category mapping relationship; The target category label for the redundant label and / or the deviation label is determined based on the preset category mapping relationship.

[0010] In a preferred embodiment, the step of invoking a multimodal large model to perform a global understanding of the image to be labeled and identifying the latent semantic labels of the image to be labeled includes: Define prompt words, and construct a multimodal message list based on the prompt words and the image to be labeled; The multimodal big model performs a global understanding based on the multimodal message list and returns the latent semantic labels of the image to be labeled.

[0011] In a preferred embodiment, the target image detection model is trained through the following process: A training dataset is constructed based on the image to be labeled, and the training dataset is input into the target image detection model to generate predicted labels; A loss function is constructed based on the predicted label and the target category label in the labeled image corresponding to the image to be labeled; The parameters in the target image detection model are adjusted according to the loss function to train the target image detection model; The target image detection model is trained when the loss function value of the loss function is stable.

[0012] In a preferred embodiment, adjusting the parameters in the target image detection model according to the loss function includes: Calculate the derivative of the loss function with respect to the parameters; New parameters are generated based on the current parameters and the derivative, and the current parameters are updated using the new parameters.

[0013] A second aspect of this application provides an image annotation apparatus, comprising: The recognition module is used to acquire the image to be labeled, call the multimodal large model to perform global understanding of the image to be labeled, and identify the potential semantic labels of the image to be labeled. The extraction module is used to extract the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label; The mapping module is used to determine the target category label corresponding to the potential semantic label based on a preset category mapping relationship; The generation module is used to generate a standard labeled image based on the target category label and the detection box coordinates; the labeled image is used to train the target image detection model.

[0014] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0015] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0016] The technical solution provided in this application may include the following beneficial effects: This application provides an image annotation method, including: acquiring an image to be annotated; calling a multimodal large model to perform global understanding of the image to be annotated and identifying latent semantic labels (LSCs) of the image; extracting the coordinates of the detection boxes corresponding to the LSCs based on the LSCs; determining the target category label corresponding to the LSCs based on a preset category mapping relationship; generating a standard annotation image based on the target category label and the detection box coordinates; and using the annotation image to train a target image detection model. Through this method, a unified label naming system is constructed, enabling label mapping and solving the problem of multiple names for one object during image annotation, significantly improving the uniformity of dataset label naming.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0018] The above and other objects, features and advantages of this application will become more apparent from the following description of exemplary embodiments of this application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components.

[0019] Figure 1 This is a schematic flowchart illustrating the image annotation method in an embodiment of this application; Figure 2 This is another schematic flowchart illustrating the image annotation method in an embodiment of this application; Figure 3a This is the original image to be labeled shown in the embodiments of this application; Figure 3b The image shown in the embodiments of this application includes potential semantic tags; Figure 3c This is an image including a target category label shown in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the image annotation device shown in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0020] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] In the field of deep learning object detection, such as the YOLO algorithm (an object detection algorithm in the field of computer vision), the model performance is highly dependent on large-scale, high-quality labeled datasets. However, traditional manual labeling methods are inefficient and extremely costly.

[0024] To address this issue, relevant techniques typically employ pre-trained, fixed-category detection models on publicly available datasets to perform forward inference on newly input, unlabeled images, outputting bounding boxes and corresponding labels based on the model's built-in category system. However, this approach results in inconsistent label names generated by different models or different versions of label naming systems, leading to inconsistent dataset naming.

[0025] In summary, the existing automatic labeling solutions suffer from inconsistent label naming, making it difficult to meet the needs of actual engineering projects.

[0026] To address the aforementioned issues, this application provides an image annotation method that constructs a unified label naming system, enabling label mapping and resolving the problem of multiple names for one object during image annotation, thus significantly improving the uniformity of dataset label naming.

[0027] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating the image annotation method in an embodiment of this application.

[0029] See Figure 1 The method includes: Step 110: Obtain the image to be labeled, call the multimodal large model to perform global understanding of the image to be labeled, and identify the potential semantic labels of the image to be labeled.

[0030] In this embodiment of the application, the images to be labeled can be stored in the same folder in advance. Before labeling, the original image dataset to be labeled can be constructed based on the original images to be labeled. During the labeling process, in order to ensure the quality of the labeling and the diversity of the data, and considering the complexity of the scene, if it is necessary to understand the labels involved in the complex image, especially the main labels, the images to be labeled can be obtained one by one from the original image dataset.

[0031] Multimodal large models refer to models trained by combining multimodal information such as text, images, videos, and audio. They represent a cutting-edge direction in artificial intelligence technology and are considered the "brain" of embodied intelligence. In this embodiment, a multimodal large model is invoked to perform a global understanding of the image to be labeled, identify potential semantic tags in the image, and store them in a temporary document. Compared to ordinary image recognition models, which identify tags with coarser granularity (e.g., only shampoo or soap labels), multimodal large models can identify finer-grained tags, such as "brand A shampoo," "brand B shampoo," etc.

[0032] Step 120: Extract the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label.

[0033] For any latent semantic label, its position in the image to be labeled can be extracted. The position of the object corresponding to the latent semantic label in the image can be represented by the coordinates of the detection box. In one example, for the latent semantic label "soap", the extracted coordinates of the "soap" detection box are [[22,135],[454,567]]. This means that these coordinates represent the position coordinates in a coordinate system established with the top-left corner of the image to be labeled as the origin. Here, [22,135] represents the pixel coordinates of the top-left corner of the detection box with the latent semantic label "soap" in the image to be labeled, and [454,567] represents the pixel coordinates of the bottom-right corner of the detection box with the latent semantic label "soap" in the image to be labeled. The size and position of the detection box can be determined based on the pixel coordinates of the top-left and bottom-right corners of the detection box.

[0034] Step 130: Based on the preset category mapping relationship, determine the target category label corresponding to the potential semantic label.

[0035] To ensure label consistency, this application embodiment can set relevant preset category mapping relationships according to the application scenario. For example, for animal detection scenarios, target category labels such as ["monkey", "elephant", "bird", "cat", "dog", ...] can be set. In this application embodiment, the preset category mapping relationship can adopt a coverage relationship, mapping the identified latent semantic labels to coarse-grained target category labels. For example, if the identified latent semantic labels are tabby cat, orange cat, and tufted cat, they can all be mapped to the target category label "cat". In one example, the preset category mapping relationship can be in the form of a tree diagram.

[0036] Step 140: Generate a standard labeled image based on the target category label and the coordinates of the detection box; the labeled image is used to train the target image detection model.

[0037] Standard labeled images are generated based on the target category labels and detection box coordinates. These labeled images include the category and coordinates of the identified objects. After all the original images in the original image dataset have been labeled, a labeling file can be generated from the labeled images. The labeling file can be a TXT (text) document or a JSON (JS key-value pair data) document. The labeling file can be used to train the target image detection model. The target image detection model in this embodiment is a high-concurrency, low-parameter model applicable to different scenarios, saving computational resources. In one example, the target image detection model can be YOLOv8, but other models can also be used; this application does not impose any restrictions.

[0038] This application provides an image annotation method, including: acquiring an image to be annotated; calling a multimodal large model to perform global understanding of the image to be annotated and identifying latent semantic labels (LSCs) of the image; extracting the coordinates of the detection boxes corresponding to the LSCs based on the LSCs; determining the target category label corresponding to the LSCs based on a preset category mapping relationship; generating a standard annotation image based on the target category label and the detection box coordinates; and using the annotation image to train a target image detection model. Through this method, a unified label naming system is constructed, enabling label mapping and solving the problem of multiple names for one object during image annotation, significantly improving the uniformity of dataset label naming.

[0039] Figure 2 This is another schematic diagram of the image annotation method shown in the embodiments of this application.

[0040] Step 210: Obtain the image to be labeled, call the multimodal large model to perform global understanding of the image to be labeled, and identify the potential semantic labels of the image to be labeled.

[0041] In this embodiment, a multimodal large model is invoked to perform a global understanding of the image to be labeled, identify the latent semantic labels in the image, and store them in a temporary document. Compared with ordinary image recognition models, ordinary image recognition models identify labels with coarser granularity. For example, ordinary image recognition models may only identify labels with granularity such as shampoo and soap in the image, while the multimodal large model can identify labels with finer granularity, such as brand A shampoo and brand B shampoo.

[0042] In one example, refer to Figure 3a The image is the original image to be labeled. By using a multimodal large model to perform a global understanding of the image, the potential semantic labels of objects such as computer monitors, keyboards, mouse pads, plates, coffee cups, flowers, vases, and headphones can be identified.

[0043] In an optional embodiment of this application, step 210 includes: Define cue words and construct a multimodal message list based on the cue words and the image to be labeled; The multimodal large model performs global understanding based on a multimodal message list and returns latent semantic labels for the images to be labeled.

[0044] When calling a multimodal large model, you can predefine prompts, such as: "You are now a data analysis assistant, used to identify which products are contained in the image. Just answer what products are in the image, do not give redundant answers. The result must be returned in the form of a Python (a programming language) list, for example: ["computer", "keyboard"]. If no product is in the image, return an empty list: []".

[0045] A multimodal message list can be constructed based on prompt words and the image to be labeled. The multimodal large model performs global understanding based on this message list. A code example of global understanding using the multimodal large model is as follows: # 1. Model Initialization # Load a large multimodal model (e.g., Qwen3-VL) and configure the corresponding inference parameters. model = load_multimodal_model("qwen3-vl") # 2. Define the System Prompt # Define the model's roles, task boundaries, and strict output format requirements. SYSTEM_PROMPT = """ You are now a data analysis assistant, tasked with identifying the products contained in images.

[0046] Simply answer what items are shown in the picture; do not provide redundant answers.

[0047] The result must be returned as a Python list, for example: ["computer", "keyboard"].

[0048] If no product is found in the image, an empty list is returned: [].

[0049] """ # 3. Prepare input and call the model # Receive externally input image links image_url = get_user_input_image_url() # Building a multimodal message list (Messages) # User input including system settings and image links messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": [ {"type": "image", "image_url": image_url}, {"type": "text", "text": "Please analyze the product in the image."} ]} ] # Call the model for inference # It is recommended to set temperature=0 or a lower value to ensure the stability of the output format. response = model.chat(messages=messages, temperature=0.0) # 4. Return the answer provided by the multimodal model. # Extracting text content generated by the model raw_answer = response.get_text() # (Optional) Post-processing: Parse the list of strings into actual Python List objects # Prevent model output from containing markdown code block markers (such as ```python ... ```) parsed_result = parse_string_to_list(raw_answer) # Return the final result to the business layer, for example: ['shoes', 'jacket'] return parsed_result Finally, the potential semantic labels of the image to be labeled are returned.

[0050] Step 220: Extract the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label.

[0051] For any latent semantic label, its position in the image to be labeled can be extracted. The position of the object corresponding to the latent semantic label in the image can be represented by the coordinates of the detection box. In one example, for the latent semantic label "soap", the extracted coordinates of the "soap" detection box are [[22,135],[454,567]]. This means that these coordinates represent the position coordinates in a coordinate system established with the top-left corner of the image to be labeled as the origin. Here, [22,135] represents the pixel coordinates of the top-left corner of the detection box with the latent semantic label "soap" in the image to be labeled, and [454,567] represents the pixel coordinates of the bottom-right corner of the detection box with the latent semantic label "soap" in the image to be labeled. The size and position of the detection box can be determined based on the pixel coordinates of the top-left and bottom-right corners of the detection box.

[0052] In an optional embodiment of this application, step 220 includes: Input the latent semantic labels into the trained general detection model; The general detection model outputs the coordinates of the corresponding detection box based on the latent semantic labels.

[0053] In this embodiment, a pre-trained general detection model can be invoked. Latent semantic labels are input into the general detection model, and its detection capabilities are utilized to output the coordinates of the detection boxes corresponding to the latent semantic labels. However, the general detection model has a large number of parameters, relatively slow inference speed, and high deployment cost. Therefore, in this embodiment, the general detection model is not directly used for unified annotation of the images to be labeled; rather, its detection capabilities are utilized.

[0054] Figure 3b For images that include latent semantic labels. Figure 3b The image only shows a portion of the identified latent semantic labels, such as "computer monitor" for a computer in the image to be labeled, "earphones" for headphones, and "flower" for flowers, etc. In addition, the image includes the confidence scores of the latent semantic labels; for example, "computer monitor" has a confidence score of 0.86, "earphones" has a confidence score of 0.41, and "flower" has a confidence score of 0.56. The confidence score represents the score of the general detection model's judgment. The closer it is to 1, the more accurate the model's judgment is; conversely, the lower the confidence score, the more doubtful the model is. Generally, the last step in a deep learning detection model includes a softmax (normalization) operation, which gives the model the probability that the object in the detection box belongs to each category. The confidence score is calculated by dividing the highest predicted probability by the sum of the predicted probabilities of all categories. The aforementioned categories refer to the categories covered by the general detection model. For example, if general detection model A is used to detect animals, it may cover 200 animal categories.

[0055] This application leverages the understanding capabilities of multimodal large models and general detection models to support open-domain recognition, enabling the labeling of special objects that the target image detection model has never seen before. For example, a typical image detection model trained for animal detection can only detect animals but not plants. However, this application, by leveraging the understanding capabilities of multimodal large models and general detection models, can label plants.

[0056] Step 230: Establish a pre-defined category mapping relationship between latent semantic labels and target category labels based on the large language model.

[0057] In this embodiment, a pre-defined category mapping relationship between the model output category and the target category label can be established using a Large Language Model (LLM). Users can construct this relationship themselves, input it into the Large Language Model for modification and adjustment, or construct it directly through the LLM. For example, for animal detection scenarios, target category labels ["monkey", "elephant", "bird", "cat", "dog", ...] can be set. In this embodiment, the pre-defined category mapping relationship can adopt a coverage relationship, mapping the identified latent semantic labels to coarse-grained target category labels. For example, if the identified latent semantic labels are tabby cat, orange cat, and tufted cat, they can all be mapped to the target category label "cat". In one example, the pre-defined category mapping relationship can be in the form of a tree diagram. Referring to Table 1, which is a schematic table of pre-defined category mapping relationships, the... Figure 3a The objects in the map are mapped into 7 categories: Table 1

[0058] Here, the original label is the latent semantic label, and the mapped target category is the target category label.

[0059] Step 240: Determine the target category label corresponding to the latent semantic label based on the latent semantic label and the preset category mapping relationship.

[0060] The target category label corresponding to a latent semantic label can be determined based on the latent semantic label and the predefined category mapping relationship. In one example, in Figure 3b According to the mapping logic in Table 1, the latent semantic label "computer monitor" and the latent semantic label "earphones" can be mapped to the target category label "Electronics," and the latent semantic label "flower" can be mapped to the target category label "Decoration." See also... Figure 3c The images include target category labels, where the target category label for computers is "Electronics" with a confidence score of 0.86, for headphones it is "Electronics" with a confidence score of 0.41, and for flowers it is "Decoration" with a confidence score of 0.56. The confidence scores are calculated using a general detection model.

[0061] In an optional embodiment of this application, the method further includes: Add a mapping relationship between redundant labels and / or bias labels and the target category labels in the preset category mapping relationship; The target category label for redundant labels and / or biased labels is determined based on the preset category mapping relationship.

[0062] Furthermore, if redundant or biased labels are found during mapping, a large language model can be used to add a mapping relationship between the redundant or biased labels and the target category label in the preset category mapping relationship. For example, if the target category label is "clothes" and the potential semantic label of "down jacket" is identified, but there is no previous mapping between "clothes" and "down jacket", it can be determined that "down jacket" is a redundant label. A large language model can be used to add a mapping relationship between "down jacket" and "clothes" in the preset category mapping relationship. Based on the preset category mapping relationship, the target category label of "down jacket" is determined to be "clothes", thus mapping "down jacket" to "clothes".

[0063] Step 250: Generate a standard labeled image based on the target category label and the coordinates of the detection box; the labeled image is used to train the target image detection model.

[0064] Standard labeled images are generated based on the target category labels and detection box coordinates. These labeled images include the category and coordinates of the identified objects in the image. After all the original images in the original image dataset have been labeled, a label file can be generated from the labeled images. The label file can be a TXT or JSON document. The following is a code example for labeling image00001: The content of the image00001.json file is: { "version": "3.3.9", "flags": {}, "shapes": [ { "kie_linking": [], "label": "Webcam", "score": null, "points": [ [ 27.77777777777715, 448.888888888889 ], [ 1390.0, 448.888888888889 ], [ 1390.0, 1766.6666666666665 ], [ 27.77777777777715, 1766.6666666666665 ] ], "group_id": null, "description": "", "difficult": false, "shape_type": "rectangle", "flags": {}, "attributes": {} } ], "imagePath": "_no1045101_4.jpeg", "imageData": null, "imageHeight": 1920, "imageWidth": 1440, "description": ""} Annotation files can be generated by labeling images, and these files can be used to train a target image detection model. The target image detection model in this embodiment is a high-concurrency model applicable to different scenarios with a small number of parameters, thus saving computational resources.

[0065] In an optional embodiment of this application, the target image detection model is trained through the following process: A training dataset is constructed based on the images to be labeled, and the training dataset is input into the target image detection model to generate predicted labels; Construct a loss function based on the predicted label and the target category label in the corresponding labeled image of the image to be labeled; The parameters in the target image detection model are adjusted according to the loss function in order to train the target image detection model; The target image detection model is trained when the loss function value stabilizes.

[0066] This application embodiment can train a target image detection model using an image to be labeled and labeled images. First, a training dataset is constructed based on the image to be labeled. The training dataset is then input into the target image detection model to generate predicted labels. For example, for the input image to be labeled... (i.e., input values), which are then used by the target image detection model to generate predicted labels. Image to be labeled The target category label generated by the image annotation method in this application embodiment is: (i.e., real labels).

[0067] The loss function L can be constructed based on the predicted label and the target category label in the corresponding labeled image of the image to be labeled. In one example, the loss function L can be calculated by the following equation (1): (1) Where i represents the i-th image to be labeled, and N represents the total number of images to be labeled. This represents the true label of the i-th image to be labeled. This represents the predicted label for the i-th image to be labeled.

[0068] By calculating the loss function value, the parameters in the target image detection model can be adjusted based on the loss function value. After adjustments are made, the image to be labeled is input into the target image detection model again to generate the next round of predicted labels, and the loss function value is recalculated. This process is repeated until the calculated loss function value stabilizes around a certain value, and the target image detection model training is complete.

[0069] In an optional embodiment of this application, adjusting the parameters in the target image detection model according to the loss function includes: Calculate the derivative of the loss function with respect to the parameters; Generate new parameters based on the current parameters and their derivatives, and update the current parameters using the new parameters.

[0070] When adjusting the parameters in the target image detection model, gradient descent can be used to continuously update the parameters. In one example, the idea of ​​chain rule differentiation can be used, with real labels. Predicted tags Input values (i.e., the image to be labeled) is used to represent the derivative of the loss function L with respect to the parameter W. See equation (2) below: (2) in, This can represent the magnitude of the change in the loss function L when the parameter W changes slightly. During training, the goal is to make the loss function L smaller and smaller. If a larger parameter W results in a larger loss function L, then parameter W needs to be reduced; conversely, if a larger parameter W results in a smaller loss function L, then parameter W needs to be increased. Therefore, parameter W needs to be continuously updated.

[0071] During the parameter update process, it is possible to update the parameters based on the current parameters. and derivative Calculate new parameters See equation (3) below: (3) in, It is the learning rate, used to control the size of parameter updates. After the new parameters are calculated, they can be used to replace the current parameters to update the current parameters.

[0072] This application provides an image annotation method that constructs a unified label naming system, enabling label mapping and solving the problem of multiple names for one object during image annotation. This significantly improves the uniformity of dataset label naming, automates the entire process, and allows technicians to implement it without complex programming, resulting in stable annotated image files.

[0073] Corresponding to the aforementioned application function implementation method embodiments, this application also provides an image annotation device, an electronic device, and corresponding embodiments.

[0074] Figure 4 This is a schematic diagram of the structure of the image annotation device shown in the embodiments of this application.

[0075] See Figure 4 The device includes: The recognition module 410 is used to acquire the image to be labeled, call the multimodal large model to perform global understanding of the image to be labeled, and identify the potential semantic labels of the image to be labeled. Extraction module 420 is used to extract the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label; The mapping module 430 is used to determine the target category label corresponding to the latent semantic label based on the preset category mapping relationship; The generation module 440 is used to generate a standard labeled image based on the target category label and the coordinates of the detection box; the labeled image is used to train the target image detection model.

[0076] In an optional embodiment of this application, the extraction module 420 includes: The input submodule is used to input latent semantic labels into the trained general detection model; The output submodule is used by the general detection model to output the coordinates of the corresponding detection box based on the latent semantic labels.

[0077] In an optional embodiment of this application, the mapping module 430 includes: The mapping relationship submodule is used to establish a pre-defined category mapping relationship between latent semantic labels and target category labels based on a large language model; The first target category label submodule is used to determine the target category label corresponding to the latent semantic label based on the latent semantic label and the preset category mapping relationship.

[0078] In an optional embodiment of this application, the latent semantic tags include redundant tags and / or biased tags; the device further includes: Add a module to add a mapping relationship between redundant labels and / or bias labels and target category labels in a preset category mapping relationship; The second target category label module is used to determine the target category label of redundant labels and / or deviation labels according to the preset category mapping relationship.

[0079] In an optional embodiment of this application, the identification module 410 includes: The construction submodule is used to define prompt words and build a multimodal message list based on the prompt words and the image to be labeled; The Global Understanding submodule is used for global understanding of multimodal large models based on a list of multimodal messages, and returns the latent semantic labels of the images to be labeled.

[0080] In an optional embodiment of this application, the device further includes: The training dataset module is used to construct a training dataset based on the images to be labeled. The training dataset is then input into the target image detection model to generate predicted labels. The loss function module is used to construct a loss function based on the predicted label and the target category label in the corresponding labeled image of the image to be labeled. The parameter adjustment module is used to adjust the parameters in the target image detection model according to the loss function in order to train the target image detection model; The judgment module is used to determine whether the target image detection model has completed training when the loss function value of the loss function is stable.

[0081] In an optional embodiment of this application, the parameter adjustment module includes: The derivative calculation submodule is used to calculate the derivative of the loss function with respect to the parameters; The update submodule is used to generate new parameters based on the current parameters and their derivatives, and then update the current parameters using these new parameters.

[0082] This application provides an image annotation device that constructs a unified label naming system, enabling label mapping and solving the problem of multiple names for one object during image annotation. This significantly improves the uniformity of dataset label naming, automates the entire process, and allows technicians to implement it without complex programming, resulting in stable annotated image files.

[0083] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0084] Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0085] See Figure 5 The electronic device 500 includes a memory 510 and a processor 520.

[0086] The processor 520 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0087] Memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 520 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 510 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 510 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0088] The memory 510 stores executable code, which, when processed by the processor 520, can cause the processor 520 to execute part or all of the methods described above.

[0089] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0090] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) that, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0091] This application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the method described above.

[0092] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image annotation method, characterized in that, The method includes: The image to be labeled is obtained, and a multimodal large model is invoked to perform a global understanding of the image to be labeled, and the potential semantic labels of the image to be labeled are identified. Extract the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label; Based on a preset category mapping relationship, the target category label corresponding to the potential semantic label is determined; A standard labeled image is generated based on the target category label and the coordinates of the detection box; the labeled image is used to train the target image detection model.

2. The method according to claim 1, characterized in that, The step of extracting the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label includes: The latent semantic labels are input into the trained general detection model; The general detection model outputs the coordinates of the corresponding detection box based on the latent semantic labels.

3. The method according to claim 1, characterized in that, The step of determining the target category label corresponding to the latent semantic label based on a preset category mapping relationship includes: A pre-defined category mapping relationship between the latent semantic tags and the target category tags is established based on a large language model; The target category label corresponding to the latent semantic label is determined based on the latent semantic label and the preset category mapping relationship.

4. The method according to claim 3, characterized in that, The latent semantic labels include redundant labels and / or biased labels; the method further includes: Add the mapping relationship between the redundant labels and / or deviation labels and the target category labels to the preset category mapping relationship; The target category label for the redundant label and / or the deviation label is determined based on the preset category mapping relationship.

5. The method according to claim 1, characterized in that, The step of calling a multimodal large model to perform a global understanding of the image to be labeled and identifying the latent semantic labels of the image to be labeled includes: Define prompt words, and construct a multimodal message list based on the prompt words and the image to be labeled; The multimodal big model performs a global understanding based on the multimodal message list and returns the latent semantic labels of the image to be labeled.

6. The method according to claim 1, characterized in that, The target image detection model is trained through the following process: A training dataset is constructed based on the image to be labeled, and the training dataset is input into the target image detection model to generate predicted labels; A loss function is constructed based on the predicted label and the target category label in the labeled image corresponding to the image to be labeled; The parameters in the target image detection model are adjusted according to the loss function to train the target image detection model; The target image detection model is trained when the loss function value of the loss function is stable.

7. The method according to claim 6, characterized in that, The step of adjusting the parameters in the target image detection model according to the loss function includes: Calculate the derivative of the loss function with respect to the parameters; New parameters are generated based on the current parameters and the derivative, and the current parameters are updated using the new parameters.

8. An image annotation device, characterized in that, The device includes: The recognition module is used to acquire the image to be labeled, call the multimodal large model to perform global understanding of the image to be labeled, and identify the potential semantic labels of the image to be labeled. The extraction module is used to extract the coordinates of the detection box corresponding to the latent semantic label based on the latent semantic label; The mapping module is used to determine the target category label corresponding to the potential semantic label based on a preset category mapping relationship; The generation module is used to generate a standard labeled image based on the target category label and the detection box coordinates; the labeled image is used to train the target image detection model.

9. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

Images