Data labeling generation method and apparatus, storage medium, and electronic device

Abstract

The application provides a data annotation generation method and device, a storage medium and an electronic device. The method comprises the following steps: determining at least one candidate region image from to-be-processed image data; calling a description generation model to generate a region description text of each candidate region image in the at least one candidate region image; calling a picture-text matching model to determine a picture-text similarity score of each candidate region image; determining a gating index value of each candidate region image based on the picture-text similarity score of each candidate region image; determining a target region image set from the at least one candidate region image based on the gating index value of each candidate region image; generating a target data annotation of each target region image in the target region image set; and adding the target data annotation of each target region image to a target data annotation set. The embodiment of the application can improve the efficiency of data annotation generation while reducing the cost.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, and in particular to a data annotation generation method, apparatus, storage medium, and electronic device. Background Technology

[0002] Currently, generating high-quality, fine-grained data annotations for massive image data faces significant challenges, especially in low-altitude scenarios (such as drone inspections, security monitoring, and agricultural plant protection) where targets exhibit drastic scale changes, unique perspectives (e.g., overhead shots), complex backgrounds, and numerous specialized sub-concepts. For instance, in power line inspections, it's necessary to annotate components such as insulator strings, towers, and hardware, along with their defect conditions (e.g., "damaged" or "corroded"). However, current technologies typically rely on manual annotation, leading to high costs and low efficiency. Therefore, a satisfactory solution for conveniently generating data annotations while reducing costs and improving efficiency remains elusive. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data annotation generation method, apparatus, storage medium, and electronic device to solve the problems of high cost and low efficiency caused by manual annotation methods in related technologies. In other words, embodiments of the present invention can conveniently generate target data annotations for images of various target regions through description generation models and image-text matching models, which can effectively improve the efficiency of data annotation generation while reducing costs, and can effectively improve the accuracy of target data annotation through gating index values, and can effectively ensure the consistency of data annotation through a unified automatic generation method.

[0004] According to one aspect of the present invention, a data annotation generation method is provided, the method comprising:

[0005] Acquire image data to be processed, and determine at least one candidate region image from the image data to be processed;

[0006] The description generation model is invoked to generate region description text for each candidate region image in the at least one candidate region image;

[0007] The image-text matching model is invoked to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the region description text of the corresponding candidate region image. The image-text similarity score of a candidate region image is determined based on the model output matching score of the corresponding candidate region image generated by the image-text matching model.

[0008] Based on the image-text similarity score of each candidate region image, a gating index value is determined for each candidate region image; and based on the gating index value of each candidate region image, a target region image set is determined from the at least one candidate region image.

[0009] Generate target data annotations for each target region image in the target region image set, wherein the target data annotation for a region image includes the region description text of the corresponding region image; and add the target data annotations for each target region image to the target data annotation set;

[0010] The method further includes:

[0011] For any candidate region image among the at least one candidate region image, perform text parsing on the region description text of any candidate region image to obtain a standard candidate annotation set for any candidate region image;

[0012] The open vocabulary verification model is invoked to output the confidence scores of each standard candidate annotation in the standard candidate annotation set of any candidate region image. The confidence score of a standard candidate annotation of any candidate region image refers to the confidence score of the corresponding standard candidate annotation for the existence of any candidate region image.

[0013] The step of determining the gating index value of each candidate region image based on the image-text similarity score of each candidate region image includes:

[0014] Based on the image-text similarity score of any candidate region image and the confidence level of each standard candidate label of any candidate region image, the gating index value of any candidate region image is determined.

[0015] According to another aspect of the present invention, a data annotation generation apparatus is provided, the apparatus comprising:

[0016] The acquisition unit is used to acquire the image data to be processed;

[0017] A processing unit is configured to determine at least one candidate region image from the image data to be processed;

[0018] The processing unit is also used to call the description generation model to generate region description text for each candidate region image in the at least one candidate region image;

[0019] The processing unit is further configured to invoke the image-text matching model to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the region description text of the corresponding candidate region image. The image-text similarity score of a candidate region image is determined based on the model output matching score of the corresponding candidate region image generated by the image-text matching model.

[0020] The processing unit is further configured to determine the gating index value of each candidate region image based on the image-text similarity score of each candidate region image; and to determine the target region image set from the at least one candidate region image based on the gating index value of each candidate region image.

[0021] The processing unit is further configured to generate target data annotations for each target region image in the target region image set, wherein the target data annotation for a region image includes the region description text of the corresponding region image; and to add the target data annotations for each target region image to the target data annotation set;

[0022] The processing unit is also used for:

[0023] For any candidate region image among the at least one candidate region image, perform text parsing on the region description text of any candidate region image to obtain a standard candidate annotation set for any candidate region image;

[0024] The open vocabulary verification model is invoked to output the confidence scores of each standard candidate annotation in the standard candidate annotation set of any candidate region image. The confidence score of a standard candidate annotation of any candidate region image refers to the confidence score of the corresponding standard candidate annotation for the existence of any candidate region image.

[0025] When determining the gating index value of each candidate region image based on the image-text similarity score of each candidate region image, the processing unit is specifically used for:

[0026] Based on the image-text similarity score of any candidate region image and the confidence level of each standard candidate label of any candidate region image, the gating index value of any candidate region image is determined.

[0027] According to another aspect of the present invention, an electronic device is provided, the electronic device including a processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the methods mentioned above.

[0028] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods mentioned above.

[0029] This invention, after acquiring image data to be processed, identifies at least one candidate region image from the image data; and invokes a description generation model to generate region description text for each candidate region image in the at least one candidate region image. Then, an image-text matching model is invoked to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image indicates the similarity between the corresponding candidate region image and its corresponding region description text. Based on this, a gating index value for each candidate region image is determined based on its image-text similarity score; and a set of target region images is determined from the at least one candidate region image based on its gating index value. Further, target data annotations for each target region image in the target region image set are generated, whereby the target data annotation for a region image includes the corresponding region description text; and the target data annotations for each target region image are added to the target data annotation set. As can be seen, the embodiments of the present invention can conveniently generate target data annotations for images of various target regions by describing the generation model and the image-text matching model, which can effectively improve the efficiency of data annotation generation while reducing costs, and can effectively improve the accuracy of target data annotation by using gating index values, and can effectively ensure the consistency of data annotation by using a unified automatic generation method. Attached Figure Description

[0030] Further details, features, and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A flowchart illustrating a data annotation generation method according to an exemplary embodiment of the present invention is shown;

[0032] Figure 2 A flowchart illustrating another data annotation generation method according to an exemplary embodiment of the present invention is shown;

[0033] Figure 3 A flowchart illustrating yet another data annotation generation method according to an exemplary embodiment of the present invention is shown;

[0034] Figure 4 A flowchart illustrating another data annotation generation method according to an exemplary embodiment of the present invention is shown;

[0035] Figure 5 A schematic diagram of a closed-loop update according to an exemplary embodiment of the present invention is shown;

[0036] Figure 6 A schematic block diagram of a data annotation generation apparatus according to an exemplary embodiment of the present invention is shown;

[0037] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0038] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0039] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0040] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0041] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0042] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0043] It should be noted that the execution subject of the data annotation generation method provided in this embodiment of the invention can be one or more electronic devices, and this invention does not limit this; wherein, the electronic device can be a terminal (i.e., a client) or a server. Therefore, when the execution subject includes multiple electronic devices, and among the multiple electronic devices includes at least one terminal and at least one server, the data annotation generation method provided in this embodiment of the invention can be jointly executed by the terminal and the server. Accordingly, the terminal mentioned herein may include, but is not limited to: smartphones, tablets, laptops, desktop computers, smartwatches, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server mentioned herein can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0044] Optionally, the data annotation generation method provided in this embodiment of the invention can be applied to any data annotation generation scenario (also known as an image data annotation scenario), and this embodiment of the invention does not limit it. For example, the data annotation generation scenario can be a drone inspection scenario, a security monitoring scenario, an agricultural plant protection scenario, etc.

[0045] Based on the above description, this embodiment of the invention proposes a data annotation generation method, which can be executed by the aforementioned electronic device (terminal or server); or, the data annotation generation method can be executed jointly by the terminal and the server. For ease of explanation, the following description will use the execution of the data annotation generation method by an electronic device as an example; such as Figure 1 As shown, the data annotation generation method may include the following steps S101-S105:

[0046] S101, acquire the image data to be processed, and determine at least one candidate region image from the image data to be processed.

[0047] Optionally, the image data to be processed can be any image data, and the embodiments of the present invention do not limit this; for example, the image data to be processed can be any image, any video frame, etc.

[0048] In embodiments of the present invention, the methods for acquiring the image data to be processed may include, but are not limited to, the following:

[0049] The first acquisition method: The electronic device stores at least one image data to be labeled in its own storage space. In this case, the electronic device can sequentially take each image data to be labeled from the at least one image data to be labeled as an image data to be processed, so as to acquire the image data to be processed.

[0050] The second method of acquisition: Electronic devices can receive images sent by image acquisition devices (such as drones) and use the received images as image data to be processed, thereby acquiring the image data to be processed.

[0051] The third method of acquisition: Electronic devices can also obtain image data download links and use these links to download image data, thereby using the downloaded image data as image data to be processed, and so on.

[0052] Optionally, when determining at least one candidate region image from the image data to be processed, the electronic device can locate the candidate region to be labeled from the image data to be processed. Optionally, the electronic device can determine the region positioning data of the image data to be processed, and determine at least one candidate region image (also referred to as a region cropping image) from the image data to be processed based on the region positioning data. Optionally, the region positioning data may include region image indication data of each candidate region image in at least one candidate region image; optionally, the region image indication data of a candidate region image may include, but is not limited to, at least one of the following: the region identifier of the corresponding candidate region image (also referred to as a unique region identifier), the bounding box coordinate information (i.e., region geometric information), the segmentation mask polygon, and the identifier of the original image to which it belongs (also referred to as the image identifier, which can be the image identifier of the image data to which the region image belongs, such as the image data to be processed), etc.; the embodiments of the present invention do not limit this. For example, an electronic device can determine at least one candidate region image from the image data to be processed based on the bounding box coordinate information of each candidate region image, i.e., crop out at least one candidate region image; or, it can determine at least one candidate region image from the image data to be processed based on the segmentation mask polygons of each candidate region image, and so on; the embodiments of the present invention do not limit this. Optionally, the region identifier and the original image identifier can be the image name or the image number, etc., the embodiments of the present invention do not limit this.

[0053] Optionally, when determining the region localization data of the image data to be processed, the electronic device may invoke a target detection model to generate at least one region geometric information of the image data to be processed (i.e., including the region geometric information of each candidate region image). In other words, the image data to be processed can be input into the target detection model, and the target detection model can output at least one region geometric information of the image data to be processed. Based on this, at least one region geometric information can be added to the region localization data, that is, the region geometric information of any candidate region image can be added to the region image indication data of any candidate region image. And / or, the electronic device may invoke an image segmentation model to generate at least one segmentation mask polygon of the image data to be processed (which may include the segmentation mask polygons of each candidate region image). In other words, the image data to be processed can be input into the image segmentation model, and the image segmentation model can output at least one segmentation mask polygon of the image data to be processed. Based on this, at least one segmentation mask polygon can be added to the region localization data, that is, the segmentation mask polygon of any candidate region image can be added to the region image indication data of any candidate region image. And / or, the electronic device can also determine the region identifier and the original image identifier (which can be the image identifier of the image data to be processed) of any candidate region image, and add the region identifier and the original image identifier of any candidate region image to the region image indication data of any candidate region image, etc.; this embodiment of the invention does not limit this. Optionally, the electronic device can generate the region identifier of each candidate region image according to a preset region identifier generation rule; optionally, the preset region identifier generation rule can be set according to experience or actual needs, and this embodiment of the invention does not limit this. Optionally, the image identifier of the image data to be processed can be directly obtained (such as already stored in its own storage space, or the image identifier of the corresponding acquired image can be received when the acquired image is received, etc.), or it can be generated according to the preset image identifier generation rule, and this embodiment of the invention does not limit this; optionally, the preset image identifier generation rule can be set according to experience or actual needs, and this embodiment of the invention does not limit this.

[0054] Optionally, the object detection model described above can be any pre-trained object detection model, and this embodiment of the invention does not limit this; for example, the object detection model can be a YOLO (You Only Look Once, also known as an object detection algorithm) model, or a DETR series (Detection Transformer, an object detection model based on Transformer (an attention mechanism architecture)) model, etc. Optionally, the image segmentation model described above can be any pre-trained image segmentation model, and this embodiment of the invention does not limit this; for example, the image segmentation model can be a SAM (SegmentAnything Model) model, or a YOLACT (You Only Look At Coefficients, a real-time instance segmentation) model, etc.

[0055] Optionally, the electronic device may include a region candidate generation module. In this case, the electronic device can determine at least one candidate region image from the image data to be processed through the region candidate generation module, thereby generating at least one candidate region image. In this embodiment of the invention, a candidate region image may be a region image of a candidate region, that is, an image of a candidate region cropped from the image data to be processed. Optionally, the image size of a candidate region image may be the same as the image size of the image data to be processed. The pixel values ​​of the pixels located within the corresponding candidate region in the candidate region image of a candidate region are the same as the pixel values ​​of the corresponding pixels in the image data to be processed, and the pixel values ​​of the pixels located outside the corresponding candidate region may be zero.

[0056] S102, Invoke the description generation model to generate region description text for each candidate region image in at least one candidate region image.

[0057] Optionally, the generative model can be described as a multimodal large model. It should be noted that the generative model can be any multimodal large model, and the embodiments of the present invention do not limit it. For example, the generative model can be a GPT-4V (GPT-4 with Vision, or multimodal GPT-4) model, or a Qwen-VL (Qwen Visual Language Model) model, etc. Among them, GPT-4 can be represented as Generative Pre-trained Transformer 4.

[0058] Based on this, the electronic device can obtain image description information of the image data to be processed, also known as context information, that is, the context information of the original image where a candidate region image is located, which can ensure the interpretation of the scene. Optionally, the image acquisition device can record acquisition information when acquiring images and use the acquisition information as the image description information of the acquired image, and send the image description information of the acquired image to the electronic device. In this case, when the electronic device receives the acquired image and the image description information of the acquired image, it can use the received acquired image as the image data to be processed, and the received image description information of the acquired image as the image description information of the image data to be processed; or, the electronic device stores the image description information of the image data to be processed in its own storage space, and the electronic device can obtain the image description information of the image data to be processed from its own storage space; or, the electronic device can also call the original image description information generation model to generate the image description information of the image data to be processed, that is, input the image data to be processed into the original image description information generation model, so as to output the image description information of the image data to be processed through the original image description information generation model, etc.; the embodiments of the present invention do not limit this. Optionally, the original image description information generation model can be any large language model, and the embodiments of the present invention do not limit this.

[0059] Furthermore, the electronic device can invoke a description generation model to generate region description text for each candidate region image based on image description information and each candidate region image in at least one candidate region image. Optionally, for any candidate region image in at least one candidate region image, the electronic device can invoke a description generation model to generate region description text for any candidate region image based on image description information and any candidate region image. For example, the electronic device can input image description information and any candidate region image into the description generation model to output region description text for any candidate region image through the description generation model, thereby generating region description text for any candidate region image.

[0060] Optionally, the electronic device can also determine the description generation prompt text (also known as the prompt word template) and call the description generation model to generate a region description text for any candidate region image based on the image description information, any candidate region image, and the description generation prompt text. For example, the image description information, any candidate region image, and the description generation prompt text can be input into the description generation model to output the region description text for any candidate region image. Optionally, the description generation prompt text can be set according to experience or actual needs, and this embodiment of the invention does not limit this. For example, the description generation prompt text can be used to control the output style, length, and focus; for example, the description generation prompt text could be: Please describe the category, color, and state of the main objects in this area in a short phrase, etc.

[0061] Based on this, embodiments of the present invention can obtain accurate and concise region description text by describing the generation model.

[0062] Optionally, the electronic device may also include a description generation module. In this case, the electronic device can use the description generation module to call a description generation model to generate region description text for each candidate region image in at least one candidate region image.

[0063] S103, call the image-text matching model to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the region description text of the corresponding candidate region image.

[0064] Optionally, the electronic device may also include an image-text matching scoring module. In this case, the electronic device can use the image-text matching scoring module to call the image-text matching model and determine the image-text similarity score of each candidate region image, such as... Figure 2 As shown. Optionally, the image-text similarity score can also be called a scalar similarity score or image-text matching score, etc. Optionally, the image-text matching score module can be the first automatic quality inspection line, aiming to evaluate the semantic consistency between the generated descriptive phrase (i.e., region description text) and the content of the candidate region image as a whole, that is, to determine the semantic similarity between any candidate region image and any candidate region image's region description text. Optionally, the image-text matching model can be any pre-trained image-text matching model, and this embodiment of the invention does not limit it; for example, the image-text matching model can be a CLIP (Contrastive Language-Image Pre-training) model, or a BLIP (Bootstrapping Language-Image Pre-training) model, etc.

[0065] Optionally, for any candidate region image among at least one candidate region image, the electronic device may invoke an image-text matching model to generate a model output matching score for any candidate region image based on the candidate region image and its region description text. For example, the electronic device may input the candidate region image and its region description text into the image-text matching model to generate a model output matching score for any candidate region image; or, the electronic device may determine the image features of any candidate region image (e.g., feature extraction from the candidate region image using a convolutional neural network to obtain the image features of the candidate region image), and input the image features and region description text into the image-text matching model to generate a model output matching score for any candidate region image, etc.; this embodiment of the invention does not limit this.

[0066] For example, taking any candidate region image as the i-th candidate region image among at least one candidate region image as an example, the model output matching score for any candidate region image can be shown in Equation 1.1:

[0067] Formula 1.1

[0068] Among them, match_score i Let σ(.) represent the model output matching score of the i-th candidate region image (which can be any candidate region image here), and let σ(.) represent the activation function (such as the Sigmoid activation function, where σ(x) = 1 / (1+e^(-1 / 2)). -x (This can normalize the output to the [0,1] interval), W m The weight matrix representing the learned data (i.e., the weight matrix of the image-text matching model, W) m ∈R d×d d×d represents the dimension of the matrix (d rows and d columns), where d is any integer), φ(I i ) can represent the visual features extracted by the image encoder (i.e., the visual features of any candidate region image, I) i It can be any candidate region image or any image feature of the candidate region image, φ(I i )∈R d (where d is the dimension of the visual feature vector), ψ(T) i ) can represent the text features extracted by the text encoder (i.e., the text features of the region description text of any candidate region image, T) i The region description text can be provided for any candidate region image, ψ(T) i )∈R d (where d is the dimension of the text feature vector), b mThis can be a bias term (which here can be the bias term in the image-text matching model). It should be noted that the embodiments of the present invention do not limit the specific model structure of the image-text matching model, that is, they do not limit the weight matrix and bias term in the image-text matching model. In other words, the weight matrix and bias term here can be set as needed, such as being obtained by training the image-text matching model, etc.

[0069] In one implementation, the electronic device can output a matching score for the model of any candidate region image, which can be used as the image-text similarity score of any candidate region image.

[0070] In another embodiment, the electronic device can also perform optical character recognition (OCR) on any candidate region image to obtain the optical character recognition result of any candidate region image. For example, optical character recognition can be performed through a CRNN (Convolutional Recurrent Neural Network) model or a CTC (Connectionist Temporal Classification) model, etc. The embodiments of the present invention do not limit this. Based on the region description text of any candidate region image and the optical character recognition result, the optical character recognition score of any candidate region image can be determined. Thus, based on the model output of any candidate region image, the matching score and the optical character recognition score can be determined to determine the image-text similarity score of any candidate region image.

[0071] Optionally, when determining the optical character recognition score of any candidate region image based on the region description text and optical character recognition results of any candidate region image, the electronic device can call the optical character recognition scoring model to determine the optical character recognition score of any candidate region image based on the region description text and optical character recognition results of any candidate region image. That is, the region description text and optical character recognition results of any candidate region image can be input into the optical character recognition scoring model to output the optical character recognition score of any candidate region image through the optical character recognition scoring model. The optical character recognition scoring model can be any large language model, and this embodiment of the invention does not limit it. Alternatively, the electronic device can determine the standard candidate annotation set of the region description text of any candidate region image (that is, determine the standard candidate annotation set of any candidate region image based on the region description text of any candidate region image), and determine the similarity (such as the cosine similarity between vectors, etc.) between each standard candidate annotation in the standard candidate annotation set of any candidate region image and each extracted entity in the optical character recognition results of any candidate region image. Thus, the optical character recognition score of any candidate region image is determined based on the similarity between each standard candidate annotation in the standard candidate annotation set of any candidate region image and each extracted entity in the optical character recognition results of any candidate region image. Optionally, the optical character recognition score for any candidate region image can be: the number of similarity-matching annotations in the standard candidate annotation set of any candidate region image / the total number of standard candidate annotations in the standard candidate annotation set of any candidate region image. A similarity-matching annotation can refer to a standard candidate annotation whose maximum similarity to each extracted entity in the optical character recognition result of any candidate region image is greater than the optical matching similarity threshold. Alternatively, the optical character recognition score for any candidate region image can be: the number of similarity-matching annotations in the standard candidate annotation set of any candidate region image / the total number of standard candidate annotations in the standard candidate annotation set of any candidate region image × a preset optical matching total score, etc. This embodiment of the invention does not limit this. Optionally, both the optical matching similarity threshold and the preset optical matching total score can be set according to experience or actual needs, and this embodiment of the invention does not limit this. Based on this, this embodiment of the invention can verify the authenticity of the region description text through optical character recognition scoring, etc. The extracted entities can also be called text entities or entity words, etc.

[0072] Optionally, when determining the image-text similarity score of any candidate region image based on the model output matching score and optical character recognition score of any candidate region image, the electronic device can perform a weighted summation of the model output matching score and optical character recognition score of any candidate region image to obtain the image-text similarity score of any candidate region image; or, the electronic device can also perform a compliance assessment of the region description text of any candidate region image according to a preset scoring rule set to obtain a rule score of any candidate region image, thereby performing a weighted summation of the model output matching score, optical character recognition score, and rule score of any candidate region image to obtain the image-text similarity score of any candidate region image, and so on; the embodiments of the present invention do not limit this. Optionally, the preset scoring rule set may be set according to experience or actual needs, and the embodiments of the present invention do not limit this; for example, the preset scoring rule set may include, but is not limited to, at least one of the following: prohibited word filtering rules (such as whether the evaluation area description text does not include prohibited words), length constraint rules (such as whether the length of the evaluation area description text is less than a preset length threshold, the preset length threshold may be set according to experience or actual needs), and entity integrity rules (such as whether the entities in the evaluation area description text meet the integrity conditions, the integrity conditions may be set according to experience or actual needs), etc. Optionally, when evaluating the compliance of the region description text of any candidate region image according to a preset scoring rule set to obtain a rule score for any candidate region image, the rule score of the region description text of any candidate region image under each scoring rule in the preset scoring rule set can be determined, and the sum of the rule scores of the region description text of any candidate region image under each scoring rule can be used as the rule score of any candidate region image; alternatively, the number of compliance scoring rules for the region description text of any candidate region image can be determined from the preset scoring rule set (compliance scoring rules can be scoring rules that allow the region description text of any candidate region image to pass the compliance evaluation, i.e., scoring rules that the region description text of any candidate region image conforms to), and the rule score of any candidate region image can be determined based on the number of compliance scoring rules and the total number of scoring rules in the preset scoring rule set. For example, the rule score of any candidate region image can be: the number of compliance scoring rules / the total number of scoring rules in the preset scoring rule set × the total preset rule evaluation score, etc.; this embodiment of the invention does not limit this. Optionally, the total preset rule evaluation score can be set according to experience or actual needs, and this embodiment of the invention does not limit this.Optionally, for any scoring rule in the preset scoring rule set, if the region description text of any candidate region image passes the compliance assessment of any scoring rule, the first rule score can be used as the rule score of the region description text of any candidate region image under any scoring rule; if the region description text of any candidate region image fails the compliance assessment of any scoring rule, the second rule score can be used as the rule score of the region description text of any candidate region image under any scoring rule; Optionally, both the first rule score and the second rule score can be set according to experience or according to actual needs, and the embodiments of the present invention do not limit this.

[0073] For example, taking any candidate region image as the i-th candidate region image, the electronic device can use Formula 1.2 to perform a weighted summation of the model output matching score, optical character recognition score, and rule score for any candidate region image to obtain the image-text similarity score for any candidate region image:

[0074] Equation 1.2

[0075] Wherein, α + β + γ = 1, and α, β, and γ are weight coefficients greater than or equal to 0. Optionally, these weight coefficients can be set according to experience or actual needs, or determined according to weight setting rules; this embodiment of the invention does not limit this. For example, the weight setting rule can be that β is zero when the optical character recognition result is empty. For instance, the weight coefficients corresponding to any candidate region image can be determined according to whether the optical character recognition result of any candidate region image is empty, so that based on the weight coefficients corresponding to any candidate region image, the model outputs a matching score, an optical character recognition score, and... The rule-based scores are weighted and summed to obtain the image-text similarity score for any candidate region image, and so on. For example, if the optical character recognition result of any candidate region image is empty, each weight coefficient in the first weight coefficient combination can be used as the corresponding weight coefficient for any candidate region image, and β in the first weight coefficient combination is zero. If the optical character recognition result of any candidate region image is not empty, each weight coefficient in the second weight coefficient combination can be used as the corresponding weight coefficient for any candidate region image. Both the first and second weight coefficient combinations can be set according to experience or actual needs, and this embodiment of the invention does not limit this. Correspondingly, OCR_score i The rule_score represents the optical character recognition score for any candidate region image. iThis can represent the rule score for any candidate region image. Based on this, embodiments of the present invention can integrate multi-source evidence, combining global semantic matching (i.e., model output matching score), local text verification (i.e., optical character recognition score), and business rule constraints (i.e., rule score) to define an enhanced image-text consistency score as the image-text similarity score. This can effectively enhance the robustness of the score, that is, effectively enhance the accuracy of the image-text similarity score. In other words, embodiments of the present invention can integrate optical character recognition results (i.e., if there is text in the candidate region, it can verify whether the region description text mentions it) and / or a preset set of scoring rules to correct the model output matching score, so as to form a more stable first quality inspection result (i.e., image-text similarity score).

[0076] In another embodiment, the electronic device can also evaluate the compliance of the region description text of any candidate region image according to a preset set of scoring rules, obtain the rule score of any candidate region image, and then perform a weighted sum of the model output matching score and the rule score of any candidate region image to obtain the image-text similarity score of any candidate region image, and so on.

[0077] It should be understood that the image-text similarity score of any candidate region image reflects the global relevance (i.e., similarity) between the region description text of any candidate region image and the candidate region image itself. Based on this, the lower the image-text similarity score of any candidate region image, the more likely it is that the region description text of any candidate region image may be seriously misleading or deviating.

[0078] Based on this, the image-text similarity score of a candidate region image is determined by the model output matching score of the corresponding candidate region image generated by the image-text matching model.

[0079] S104, based on the image-text similarity score of each candidate region image, determine the gating index value of each candidate region image; and based on the gating index value of each candidate region image, determine the target region image set from at least one candidate region image.

[0080] S105, generate target data annotations for each target region image in the target region image set, wherein the target data annotation of a region image includes the region description text of the corresponding region image; and add the target data annotations of each target region image to the target data annotation set.

[0081] Optionally, the target data annotation set may include all target data annotations obtained by performing the data annotation generation method; for example, when there are multiple image data to be processed, the target data annotations of each target region image in each image data to be processed can be added to the target data annotation set. Optionally, the electronic device may also send the target data annotation set to any downstream task, or output the target data annotation set, etc.

[0082] This invention, after acquiring image data to be processed, identifies at least one candidate region image from the image data; and invokes a description generation model to generate region description text for each candidate region image in the at least one candidate region image. Then, an image-text matching model is invoked to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image indicates the similarity between the corresponding candidate region image and its corresponding region description text. Based on this, a gating index value for each candidate region image is determined based on its image-text similarity score; and a set of target region images is determined from the at least one candidate region image based on its gating index value. Further, target data annotations for each target region image in the target region image set are generated, whereby the target data annotation for a region image includes the corresponding region description text; and the target data annotations for each target region image are added to the target data annotation set. As can be seen, the embodiments of the present invention can conveniently generate target data annotations for images of various target regions by describing the generation model and the image-text matching model, which can effectively improve the efficiency of data annotation generation while reducing costs, and can effectively improve the accuracy of target data annotation by using gating index values, and can effectively ensure the consistency of data annotation by using a unified automatic generation method.

[0083] Based on the above description, this embodiment of the invention also proposes a more specific data annotation generation method. Accordingly, this data annotation generation method can be executed by the aforementioned electronic device (terminal or server); or, the data annotation generation method can be executed jointly by the terminal and the server. For ease of explanation, the following description will use the execution of this data annotation generation method by an electronic device as an example; please refer to [link to previous text]. Figure 3 The data annotation generation method may include the following steps S301-S308:

[0084] S301, acquire the image data to be processed, and determine at least one candidate region image from the image data to be processed.

[0085] S302, Invoke the description generation model to generate region description text for each candidate region image in at least one candidate region image.

[0086] S303, call the image-text matching model to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the region description text of the corresponding candidate region image.

[0087] S304, for any candidate region image in at least one candidate region image, perform text parsing on the region description text of any candidate region image to obtain a standard candidate annotation set for any candidate region image.

[0088] Optionally, the electronic device may also include an entity extraction and normalization module. In this case, the electronic device can use the entity extraction and normalization module to perform text parsing (also known as deep parsing) on ​​the region description text of each candidate region image to obtain a set of standard candidate annotations for each candidate region image.

[0089] Optionally, the electronic device can invoke a text segmentation model to perform text segmentation on the region description text of any candidate region image, obtaining the segmentation result of any candidate region image. The region description text of any candidate region image can be input into the text segmentation model to output the segmentation result of the region description text of any candidate region image. Optionally, the text segmentation model can be any large language model; this embodiment of the invention does not limit this. Optionally, the text segmentation model can be used for word segmentation and part-of-speech analysis to identify entity words such as nouns and adjectives (also called entities). Based on this, the electronic device can determine the standard candidate annotation set of any candidate region image based on the segmentation result of any candidate region image, thereby realizing text parsing of the region description text of any candidate region image to obtain the standard candidate annotation set of any candidate region image.

[0090] In one implementation, when determining a standard candidate annotation set for any candidate region image based on the word segmentation result of any candidate region image, the electronic device may use the word segmentation result of any candidate region image as the standard candidate annotation set for any candidate region image.

[0091] In another implementation, the electronic device can perform word segmentation conversion processing on the word segmentation results of any candidate region image according to preset word segmentation processing rules, to obtain a standard candidate annotation set for any candidate region image, thereby realizing the word segmentation conversion processing of the word segmentation results. Optionally, the preset word segmentation processing rules can be set according to experience or actual needs, and this embodiment of the invention does not limit this; for example, the preset word segmentation processing rules may include, but are not limited to, at least one of the following: word form restoration (such as restoring plurals, past tenses, etc. to their basic forms), synonym mapping (such as mapping professional terms such as "insulator damage" and "insulator self-explosion" to the standard word "insulator_damage", etc.), stop word filtering (such as filtering words in the stop word list), black / white list constraints (such as filtering irrelevant or sensitive words), and deduplication processing, etc. For example, the text parsing process may sequentially include: word segmentation and part-of-speech analysis, synonym mapping, stop word filtering, black / white list constraints, and deduplication processing, etc.

[0092] Based on this, embodiments of the present invention can extract all key entities (i.e., initial candidate annotations) from the region description text of any candidate region image using a text segmentation model, and transform them into a structured, standardized set of standard candidate annotations. In other words, embodiments of the present invention can output a stable, unambiguous set of standard candidate annotations through text parsing, which can provide a clear query target for the next stage of fine-grained entity verification.

[0093] S305, call the open vocabulary verification model, and output the confidence of each standard candidate annotation in the standard candidate annotation set of any candidate region image. The confidence of a standard candidate annotation of any candidate region image refers to the confidence of the corresponding standard candidate annotation for any candidate region image.

[0094] Correspondingly, the confidence level of a standard candidate label for any candidate region image can refer to: the confidence level that the corresponding standard candidate label supports in describing any candidate region image, that is, the confidence level that the corresponding standard candidate label exists in any candidate region image, that is, the confidence level that the corresponding standard candidate label exists in the candidate region indicated by any candidate region image; in other words, the confidence level that the object indicated by the corresponding standard candidate label exists in the candidate region indicated by any candidate region image.

[0095] Optionally, the electronic device may also include an open-vocabulary entity verification module. In this case, the electronic device can use the open-vocabulary entity verification module to call the open-vocabulary verification model and output the confidence scores of each standard candidate label in the standard candidate label set of any candidate region image. In this embodiment of the invention, the open-vocabulary entity verification module can constitute a second, and more granular, automatic quality inspection line, aiming to verify one by one whether each entity label (i.e., standard candidate label) in the standard candidate label set of any candidate region image truly exists in the corresponding candidate region, that is, whether it truly exists in the corresponding candidate region image.

[0096] Optionally, the open vocabulary verification model can be any open vocabulary detection model (such as the Open-World Localization with Vision Transformer (OWLViT) model, the Marrying DINO with GroundedPre-Training for Open-Set Object Detection (Grounding DINO) model, etc.), or any open vocabulary classification model (such as the Contrastive Language-Image Pre-training model, etc.). This embodiment of the invention does not limit this.

[0097] Based on this, for each standard candidate label (also known as a candidate entity) in the standard candidate label set of any candidate region image, an open vocabulary verification model can be used to query within any candidate region image (i.e., the candidate region indicated by any candidate region image) to output the confidence score of the existence of each standard candidate label. For example, an electronic device can input the standard candidate label set of any candidate region image and any candidate region image into the open vocabulary verification model to output the confidence score of each standard candidate label.

[0098] Optionally, the electronic device can also determine the open-vocabulary entity verification result (hereinafter referred to as entity verification result) of each standard candidate annotation based on the confidence level of each standard candidate annotation. Optionally, for any standard candidate annotation in the set of standard candidate annotations for any candidate region image, the electronic device can determine the open-vocabulary entity verification result of any standard candidate annotation according to the preset open-vocabulary entity verification rules and the confidence level of any standard candidate annotation; or, the confidence level of any standard candidate annotation can be used as the open-vocabulary entity verification result of any standard candidate annotation, etc.; the embodiments of the present invention do not limit this. Optionally, the preset open vocabulary entity verification rules can be set according to experience or actual needs, and this embodiment of the invention does not limit this. For example, the preset open vocabulary entity verification rules may include at least one confidence threshold. For instance, when the confidence of any standard candidate tag is greater than or equal to a first confidence threshold, the open vocabulary entity verification result of any standard candidate tag can be determined as a hit; when the confidence of any standard candidate tag is less than the first confidence threshold but greater than a second confidence threshold, the open vocabulary entity verification result of any standard candidate tag can be determined as uncertain (or pending); when the confidence of any standard candidate tag is less than or equal to the second confidence threshold, the open vocabulary entity verification result of any standard candidate tag can be determined as a miss, and so on. Optionally, both the first confidence threshold and the second confidence threshold can be set according to experience or actual needs, and this embodiment of the invention does not limit this. In this case, the open vocabulary entity verification result can be divided into "hit," "uncertain," and "miss." The first confidence threshold is greater than the second confidence threshold.

[0099] For example, suppose the region description text of any candidate region image is "hot spot exists on the photovoltaic panel", and the standard candidate label set of any candidate region image is ["photovoltaic panel", "hot spot"]. Then, the electronic device can use the open vocabulary entity verification module to verify whether there is a "photovoltaic panel" in any candidate region image (i.e., the candidate region indicated by any candidate region image) (object existence, such as the confidence of "photovoltaic panel" can be output by the open vocabulary verification model), and whether there is a bright region representing a "hot spot" (attribute / defect existence, such as the confidence of "hot spot" can be output by the open vocabulary verification model), and so on.

[0100] Optionally, the electronic device can use an open vocabulary entity verification module to aggregate the open vocabulary entity verification results of all entities in any candidate region image (i.e., the open vocabulary entity verification results of each standard candidate annotation), forming a second quality inspection report (also known as a second quality inspection result) for any candidate region image. This directly reveals the specific illusionary costs that may exist in the region description text of any candidate region image, such as revealing the missing standard candidate annotations in any candidate region image. Based on this, using the open vocabulary verification model to perform pixel-level or region-level existence verification of entities (i.e., annotations) extracted from the region description text is a key technology to ensure the authenticity of annotations.

[0101] S306. Based on the image-text similarity score of any candidate region image and the confidence level of each standard candidate annotation of any candidate region image, determine the gating index value of any candidate region image.

[0102] Optionally, the electronic device may also include a quality gating and dual-track output module. In this case, the electronic device can determine the gating index value of any candidate region image based on the image-text similarity score of any candidate region image and the confidence level of each standard candidate annotation of any candidate region image, so as to determine the target region image set from at least one candidate region image, and then add the target data annotations of each target region image to the target data annotation set, and so on. Optionally, the quality gating and dual-track output module can be the decision center for automatic quality inspection, and can be responsible for integrating the first quality inspection result and the second quality inspection result to perform quality gating.

[0103] In one implementation, the electronic device can determine the key entity hit rate based on the confidence levels of each standard candidate annotation in any candidate region image; and determine the gating index value of any candidate region image based on the image-text similarity score and the key entity hit rate. Optionally, the key entity hit rate can be the ratio of the number of hit standard candidate annotations in the standard candidate annotation set of any candidate region image to the total number of standard candidate annotations in the standard candidate annotation set of any candidate region image, where the hit standard candidate annotations can refer to standard candidate annotations with a confidence level greater than or equal to a first confidence threshold.

[0104] For example, when the image-text similarity score of any candidate region image is greater than or equal to the consistency score threshold, and the key entity hit rate is greater than or equal to the key entity hit rate threshold, the first gate value can be used as the gate index value of any candidate region image; when the image-text similarity score of any candidate region image is less than the consistency score threshold, or the key entity hit rate is less than the key entity hit rate threshold, the second gate value can be used as the gate index value of any candidate region image. Optionally, both the first gate value and the second gate value can be set according to experience or actual needs, and this embodiment of the invention does not limit this; for example, the first gate value can be 1, the second gate value can be 0, etc. For example, at this time, the electronic device can use the gate function shown in Formula 2.1 to determine the gate index value of any candidate region image:

[0105] Equation 2.1

[0106] In Formula 2.1, when determining the gating index value of any candidate region image, G can represent the gating index value of any candidate region image, and s can represent the image-text similarity score (i.e., Score) of any candidate region image. i ), θ s This can represent the consistency score threshold, θ r Let represent the key entity hit rate threshold, 'e' represent the standard candidate annotation (i.e., candidate entity), 'E' represent the set of standard candidate annotations for any candidate region image, '|E|' represent the total number of standard candidate annotations in the set of standard candidate annotations for any candidate region image, and 'H(e)' represent the hit indicator function for the standard candidate annotation 'e': H(e) = I[conf(e) ≥ θ e ]; where θ e I[.] can be the first confidence threshold (also known as the preset entity confidence threshold), I[.] can be an indicator function (i.e., it is 1 if the confidence is greater than or equal to the first confidence threshold, and 0 if the confidence is less than the first confidence threshold), and conf(e) can represent the confidence of the standard candidate label e.

[0107] Based on this, when the image-text similarity score of any candidate region image is greater than or equal to the consistency score threshold, and the key entity hit rate is greater than or equal to the key entity hit rate threshold, it can be judged as high-quality annotation, thus passing the gating. Correspondingly, if the gating index value of any candidate region image is equal to the first gating value (e.g., 1), then any candidate region image can be determined to have passed the gating; if the gating index value of any candidate region image is equal to the second gating value (e.g., 0), then any candidate region image can be determined to have failed the gating.

[0108] In another implementation, the electronic device can use the average confidence score among the confidence scores of each standard candidate label of any candidate region image as the average entity confidence score (also known as the average entity confidence score of any candidate region image); and determine the gating index value of any candidate region image based on the average entity confidence score and the image-text similarity score of any candidate region image. For example, for differentiable training, in order to transform discrete, non-differentiable quality gating decisions into continuous, differentiable probabilistic forms, enabling the entire system to perform end-to-end gradient backpropagation and optimization, embodiments of the present invention can introduce a soft-gating probability, which is used as the gating index value of any candidate region image. The soft-gating probability can be expressed as shown in Formula 2.2:

[0109] Equation 2.2

[0110] In Formula 2.2, when determining the gating index value of any candidate region image, p can represent the gating index value of any candidate region image, r can represent the average entity confidence (which is the sum of the confidence values ​​of each standard candidate label / the total number of standard candidate labels in the standard candidate label set of any candidate region image), and w s The weight w can represent the difference in the consistency score between the text and images. r The weights that can represent the average entity confidence difference (i.e., w) s w r (each can be a weight), b g This can be a gated bias term. Optional, w s w r and b g All settings can be based on experience or actual needs, and this embodiment of the invention does not limit them; for example, the gating bias term can also be a bias term in the open vocabulary verification model, which can be set according to the model training results of the open vocabulary verification model, and so on.

[0111] In this scenario, if the gating index value of any candidate region image is greater than or equal to the gating index threshold, then any candidate region image can be determined to have passed the gating; if the gating index value of any candidate region image is less than the gating index threshold, then any candidate region image can be determined to have failed the gating. Alternatively, if the gating index value of any candidate region image is greater than or equal to the gating index threshold, and the image-text similarity score of any candidate region image is greater than or equal to the consistency score threshold, and the key entity hit rate is greater than or equal to the key entity hit rate threshold, then any candidate region image can be determined to have passed the gating; otherwise, any candidate region image can be determined to have failed the gating, and so on. Optionally, the gating index threshold can be set according to experience or actual needs, and this embodiment of the invention does not limit this.

[0112] Optionally, in other embodiments, for any candidate region image among at least one candidate region image, the electronic device may also use the image-text similarity score of any candidate region image as a gating index value for that candidate region image. Accordingly, if the gating index value of any candidate region image is greater than or equal to the gating index threshold, then it can be determined that any candidate region image has passed the gating; if the gating index value of any candidate region image is less than the gating index threshold, then it can be determined that any candidate region image has not passed the gating, and so on. In this case, the candidate region image with the larger image-text similarity score (i.e., the candidate region image whose gating index value is greater than or equal to the gating index threshold) can be determined to have passed the gating.

[0113] S307, based on the gating index value of each candidate region image, determine the target region image set from at least one candidate region image.

[0114] Optionally, the electronic device can determine whether any candidate region image passes the gate based on the gate index value of any candidate region image. For example, if the gate index value is greater than or equal to the gate index threshold, it can be determined that the image has passed the gate, and so on.

[0115] In one implementation, if any candidate region image passes the gating, then any candidate region image can be used as a target region image in the target region image set, that is, any candidate region image can be added to the target region image set.

[0116] In another implementation, if any candidate region image passes the gating, the confidence levels of each standard candidate annotation of the candidate region image can be used to perform a miss filter on the standard candidate annotation set of the candidate region image, resulting in a target entity annotation set for the candidate region image. Consistency cross-validation is then performed between the region description text of the candidate region image and the target entity annotation set. When any candidate region image passes the consistency cross-validation (i.e., the region description text of the candidate region image and the target entity annotation set are consistent), the candidate region image is included as a target region image in the target region image set. Based on this, the target entity annotation set of any candidate region image can include all standard candidate annotations in the standard candidate annotation set of the candidate region image, except for the miss standard candidate annotations. The miss standard candidate annotations can be standard candidate annotations with a confidence level less than or equal to a second confidence threshold. Therefore, a target entity annotation set can be obtained by filtering out the miss standard candidate annotations from the corresponding standard candidate annotation set, effectively ensuring the authenticity and reliability of the annotations.

[0117] Optionally, when performing consistency cross-validation (also known as dual-track cross-validation) on the region description text and target entity annotation set of any candidate region image, the electronic device can invoke a consistency check model to perform consistency cross-validation on the region description text and target entity annotation set of any candidate region image, and obtain a consistency cross-validation result. The consistency cross-validation result can be used to indicate whether any candidate region image passes the consistency cross-validation. Optionally, the consistency check model can be any large language model, and this embodiment of the invention does not limit it; for example, the consistency cross-validation result can be a consistency cross-validation score. In this case, when the consistency cross-validation result is greater than a preset consistency cross-validation score threshold, it can be determined that any candidate region image passes the consistency cross-validation; when the consistency cross-validation result is less than or equal to the preset consistency cross-validation score threshold, it can be determined that any candidate region image fails the consistency cross-validation; or, the consistency cross-validation result can be a consistency cross-validation pass flag (to indicate that consistency cross-validation has been passed) or a consistency cross-validation fail flag (to indicate that consistency cross-validation has failed), etc. Optionally, both the consistency cross-validation pass flag and the consistency cross-validation fail flag can be set according to experience or actual needs, and this embodiment of the invention does not limit them. Based on this, embodiments of the present invention can use consistency cross-validation to check whether the region description text of any candidate region image completely covers all key entities in the target entity annotation set of any candidate region image, and to check whether the target entity annotation set of any candidate region image faithfully reflects the core information of the region description text of any candidate region image.

[0118] Optionally, if any candidate region image fails consistency cross-validation (i.e., the region description text of any candidate region image is inconsistent with the target entity annotation set), that candidate region image may not be included as a target region image in the target region image set. Optionally, in this case, any candidate region image may be marked as requiring manual review, or the image data of any candidate region image to be reviewed may be added to the image data set to be reviewed for manual review, such as... Figure 4 As shown; or, the description generation model can be called again to generate supplementary region description text for any candidate region image, so as to achieve the supplementary generation of region description text for any candidate region image through the supplementary region description text of any candidate region image, until any candidate region image is added to the target region image set, and so on.

[0119] Correspondingly, if any candidate region image fails the gating, then that candidate region image will not be included as a target region image in the target region image set; however, the image data to be reviewed for candidate region images that fail the gating can be added to the image data set to be reviewed. In other words, if any candidate region image fails the gating, the image data to be reviewed for any candidate region image can be added to the image data set to be reviewed.

[0120] Optionally, the image data to be verified for a candidate region image may include, but is not limited to, at least one of the following: the corresponding candidate region image, the region description text of the corresponding candidate region image, a set of standard candidate annotations, image-text similarity scores and open vocabulary entity verification results, a set of target entity annotations, region identifiers and the identifiers of the original image to which it belongs, etc.; the embodiments of the present invention do not limit this. It should be understood that when the image data to be verified for a candidate region image includes a set of target entity annotations for the corresponding candidate region image, if the set of target entity annotations for the corresponding candidate region image is not obtained, the electronic device may also perform a miss filter on the standard candidate annotation set of the corresponding candidate region image to obtain the set of target entity annotations for the corresponding candidate region image, and add it to the image data to be verified for the corresponding candidate region image.

[0121] Furthermore, the electronic device can transfer the set of image data to be reviewed to manual processing. For example, the electronic device can display the set of image data to be reviewed in a review interface, allowing users (such as administrators or auditors) to review the set of image data through the review interface. For instance, users can edit and correct regional description text and standard candidate annotation sets through the review interface. Optionally, the electronic device can display the set of image data to be reviewed in the review interface when a query operation to be reviewed is detected. It should be noted that the specific execution method of the query operation to be reviewed in this embodiment of the invention is not limited.

[0122] Optionally, the electronic device may also include a manual review module. In this case, the electronic device can display the set of image data to be reviewed in the review interface through the manual review module, thereby providing final quality assurance and knowledge input. Based on this, embodiments of the present invention allow users (such as reviewers) to conduct reviews through a visual interface (such as a review interface); the visual interface can simultaneously display candidate region images, automatically generated region description text and standard candidate annotations, as well as detailed evidence from two quality checks (such as low score prompts, highlighting of entities that did not meet the criteria), etc. (that is, displaying the image data to be reviewed); correspondingly, the user can edit and correct the region description text and the set of standard candidate annotations, and can select predefined error reasons (such as "model illusion", "incomplete description"), etc.

[0123] Optionally, after approval, the correction result can be saved as a "gold standard" (e.g., as a label), and the sample can be marked as a "manually corrected sample". Optionally, for any image data to be reviewed in the image data set to be reviewed, after reviewing the image data to be reviewed according to the review operation for any image data to be reviewed to obtain a correction sample of any image data to be reviewed, the correction sample of any image data to be reviewed can be added to the correction sample set (also called the correction sample pool). Optionally, the correction sample of any image data to be reviewed may include, but is not limited to, at least one of the following: the error correction score of the candidate region image indicated by any image data to be reviewed (also called the error correction score label), the error correction entity verification result (also called the error correction entity verification result label, such as 1 or 0 if it is a confidence label), the error correction annotation set (also called the error correction annotation set label), the gating decision indication information, the gating decision label, etc.; the embodiments of the present invention do not limit this.

[0124] Based on this, embodiments of the present invention can achieve efficient human-machine collaboration through intelligent quality gating. In other words, through configurable quality gating strategies, embodiments of the present invention can automatically filter out high-confidence annotation results (i.e., target data annotations) for direct output, submitting only a small number of low-quality or conflicting samples for manual review. This significantly reduces the workload of manual review (typically covering more than 80% of automatic annotations), achieving an optimal balance between automation efficiency and human accuracy, especially suitable for low-altitude fields requiring extensive professional knowledge. For example, the seamless integration of the collaborative workflow between generation, dual quality checks, and gating is the core of achieving efficient and high-quality automatic annotation.

[0125] S308, Generate target data annotations for each target region image in the target region image set. The target data annotation for a region image includes the region description text of the corresponding region image; and add the target data annotations for each target region image to the target data annotation set.

[0126] The target data annotation can be the final annotation result. Optionally, the target data annotation of any target region image may include, but is not limited to, at least one of the following: region description text of any target region image, target entity annotation set, region identifier, original image identifier, etc., and the embodiments of the present invention do not limit this.

[0127] Based on this, when the target data annotation of any target region image includes both the region description text and the target entity annotation set, it can be output in parallel in a dual-track manner, providing downstream tasks with both structured annotations and natural language descriptions. In this case, the embodiments of the present invention can output data that simultaneously contains natural language descriptions (i.e., region description text) and structured labels (i.e., target entity annotation sets), both originating from the same process and ensuring semantic consistency through consistency cross-validation. This dual-track data is easy for humans to understand and also beneficial for machine learning models, improving the practical value and reliability of data annotation, thus making the information more complete and reliable.

[0128] Optionally, the electronic device may also include a sample pool and a closed-loop update module. In this case, the electronic device can also update the gating threshold parameters and / or model parameters in the model to be trained through the sample pool and the closed-loop update module. Optionally, the embodiments of the present invention may also provide a data annotation generation system. In this case, the data annotation generation method proposed in the embodiments of the present invention can also be executed by the data annotation generation system. That is, the electronic device used to execute the data annotation generation method can be an electronic device equipped with a data annotation generation system. Then, the electronic device can execute the data annotation generation method proposed in the embodiments of the present invention through the data annotation generation system. Based on this, the sample pool and the closed-loop update module can be the core for realizing the self-evolution of the data annotation generation system. Optionally, the electronic device may also include a low-score sample pool and / or a conflict sample pool, etc. The embodiments of the present invention do not limit this. Among them, the low-score sample pool can be used to store samples with image-text similarity scores less than the consistency score threshold (such as including corresponding candidate region images, image-text similarity scores, gating judgment indication information, etc.), and the conflict sample pool can be used to store samples of candidate region images that have not passed consistency cross-validation and / or samples with key entity hit rates less than the key entity hit rate threshold, etc. Based on this, the sample pool and closed-loop update module can be used to maintain multiple specialized sample pools.

[0129] Optionally, the electronic device can also acquire a first training region image data set. A first training region image data set may include gating decision indication information for a first training region image. Based on the gating decision indication information for each first training region image in the first training region image data set, it can determine whether each first training region image is a region image that passes the gating error judgment or a region image that fails the gating correct judgment, thereby determining the number of region images that pass the gating error judgment and the number of region images that fail the gating correct judgment from the first training region image data set. Then, based on the number of region images that pass the gating error judgment and the number of region images that fail the gating correct judgment, the gating threshold parameter can be updated. The gating threshold parameter is used to determine whether a region image passes the gating. Optionally, the gating threshold parameter may include, but is not limited to, at least one of the following: consistency score threshold and key entity hit rate threshold, etc., which are not limited in this embodiment of the invention. Optionally, a first training region image data set may also include, but is not limited to, a first training region image, etc., which are not limited in this embodiment of the invention.

[0130] Optionally, the first training region image data set may include multiple error-correcting samples from the error-correcting sample set (e.g., any error-correcting sample can be used as a first training region image data); and / or, it may include first training region image data determined from newly added error-correcting samples (i.e., error-correcting samples not used to determine the first training region image data) in the error-correcting sample set; and / or, it may also include training samples of multiple target region images in the target region image set, such as constructing training samples of any target region image as a first training region image data; and / or, it may also include first training region image data determined from the low-scoring sample pool, etc.; the embodiments of the present invention do not limit this. Optionally, the gating determination indication information of a first training region image may include the gating index value of the corresponding first training region image, and may also include the gating determination result identifier of the corresponding first training region image (which can be used to indicate whether the gating is passed), and may also include the image-text similarity score and key entity hit rate of the corresponding first training region image (which can be used to determine whether the gating is passed), etc.; the embodiments of the present invention do not limit this. Optionally, the gate control determination result identifier can be a pass gate control identifier (which can be used to indicate that the gate control has been passed) or a fail gate control identifier (which can be used to indicate that the gate control has not been passed); optionally, both the pass gate control identifier and the fail gate control identifier can be set according to experience or actual needs, and the embodiments of the present invention do not limit this.

[0131] Among them, the area image that was incorrectly gated and passed can refer to the area image that was incorrectly gated and passed (also known as a false positive sample), and the area image that was correctly gated and failed can refer to the area image that was correctly gated and failed (also known as a true negative sample).

[0132] Optionally, when determining whether each first training region image is a region image that has passed the gate control error or failed the gate control correctly based on the gate control determination indication information of each first training region image in the first training region image data set, the electronic device can determine the gate control determination label of any first training region image in the first training region image data set (i.e., the first training region image indicated by any first training region image data). This label can be used to indicate whether the gate control has been passed. When the gate control determination indication information of any first training region image indicates that the first training region image has passed the gate control, and the gate control determination label of any first training region image indicates that the first training region image has failed the gate control, the first training region image can be determined to be a region image that has passed the gate control error. When the gate control determination indication information of any first training region image indicates that the first training region image has failed the gate control, and the gate control determination label of any first training region image indicates that the first training region image has failed the gate control, the first training region image can be determined to be a region image that has failed the gate control correctly.

[0133] Optionally, when determining the gate control label for any first training region image, the electronic device can determine the gate control label for any first training region image from the first training region image data where the first training region image is located. That is, at this time, a first training region image data may also include the gate control label for the corresponding first training region image. In this case, the gate control label may be a manually labeled label; or, the electronic device may also determine the gate control label for any first training region image according to the gate control label determination method, etc. The embodiments of the present invention do not limit this. It should be noted that the gate control label determination method may be set according to experience or actual needs, and the embodiments of the present invention do not limit this. For example, when the gate control index value is calculated by formula 2.1 to determine whether the gate control is passed, the gate control label determination method may refer to the method of determining whether the gate control is passed by calculating the gate control index value by formula 2.2. That is, at this time, the gate control label may be determined by formula 2.2, etc.

[0134] Optionally, the electronic device can also determine, based on the gating judgment indication information of each first training region image in the first training region image dataset, whether each first training region image is a region image that failed the gating error judgment or a region image that passed the gating judgment correctly, so as to determine the number of region images that failed the gating error judgment and the number of region images that passed the gating judgment correctly from the first training region image dataset. Based on this, for any first training region image in the first training region image dataset, when the gating judgment indication information of any first training region image indicates that any first training region image failed the gating, and the gating judgment label of any first training region image indicates that any first training region image passed the gating, it can be determined that any first training region image is a region image that failed the gating error judgment; when the gating judgment indication information of any first training region image indicates that any first training region image passed the gating, and the gating judgment label of any first training region image indicates that any first training region image passed the gating, it can be determined that any first training region image is a region image that passed the gating judgment correctly. Among them, the area image that failed the gating error can refer to the area image that failed the gating error (also known as a false negative sample), and the area image that passed the gating correctly can refer to the area image that passed the gating correctly (also known as a true positive sample).

[0135] Therefore, when updating the gating threshold parameter based on the number of images that passed the gating erroneous judgment and the number of images that failed the gating correctly, the electronic device can use the number of images that passed the gating erroneous judgment (also known as the number of false positive samples), the number of images that failed the gating correctly (also known as the number of true negative samples), and the number of images that passed the gating correctly (also known as the number of true positive samples) to update the gating threshold parameter. For example, the electronic device can use Formula 2.3 to update the gating threshold parameter based on the number of images that passed the gating erroneous judgment and the number of images that failed the gating correctly:

[0136] Equation 2.3

[0137] Where t can be the iteration round (i.e., the number of iterations), FPR t It can represent the false alarm rate (which can be equal to FP). t / (FP t +TN t ), FP t TN can represent the number of regions of images that have passed through the gate erroneous detection. t It can represent the number of images in the region that were correctly identified as not passing the gate; FNR t This can represent the false negative rate (which can be equal to FN).t / (FN t +TP t )), FN t TP can represent the number of regions of images that failed the gating error detection. t η1 can represent the number of regions of images that were correctly identified by the gating system; η1 can represent the learning rate for adjusting the gating threshold parameter; FPR target It can represent the preset target false alarm rate, FNR target The preset target false positive rate can be represented. Optionally, the gating threshold parameter adjustment learning rate, the preset target false positive rate, and the preset target false negative rate can all be set according to experience or actual needs, and this embodiment of the invention does not limit this.

[0138] Optionally, the electronic device can iteratively execute the above-described process of acquiring the first training region image data set to update the gating threshold parameter. For example, the first training region image data set can be acquired at preset gating threshold parameter update intervals to update the gating threshold parameter; alternatively, the first training region image data set can be acquired to update the gating threshold parameter when the number of incremental samples (i.e., the number of newly added samples) in the error correction sample pool is detected to be greater than a preset incremental sample threshold, and so on. This embodiment of the invention does not limit this. Optionally, both the preset gating threshold parameter update time and the preset incremental sample threshold can be set based on experience or actual needs, and this embodiment of the invention does not limit this.

[0139] Based on this, embodiments of the present invention can optimize the gating threshold parameter by analyzing the image data set of the first training region (e.g., by analyzing samples in the error correction sample pool), so that the gating threshold parameter is updated through a feedback control mechanism.

[0140] Optionally, the electronic device can also acquire a second training region image data set. A second training region image data set includes: a text-image similarity score and a training score label for a second training region image. Based on each second training region image data set in the second training region image data set, a text-image matching loss value can be determined. Then, based on the text-image matching loss value, a model loss value can be determined. This optimizes the model parameters in the model to be trained by reducing the model loss value. The model to be trained includes at least one of the following: a description generation model and a text-image matching model. Optionally, a second training region image may be the same as or different from a first training region image; this is not limited in this embodiment. Optionally, the second training region image data set may include all error-correcting samples in the error-correcting sample set (i.e., second training region image data determined from the error-correcting sample set), and may also include newly added error-correcting samples in the error-correcting sample set (here, error-correcting samples not used to determine the second training region image data, or second training region image data determined from newly added error-correcting samples in the error-correcting sample set), etc.; this is not limited in this embodiment. Optionally, the training score label of a second training region image can be the error correction score of the corresponding second training region image.

[0141] Optionally, when determining the image-text matching loss value based on the image data of each second training region in the second training region image dataset, the electronic device may use the image-text similarity score and training score label of each second training region image in the second training region image dataset to determine the image-text matching loss value. For example, the image-text matching loss value can be calculated based on the difference between the image-text similarity score and the training score label of each second training region image; alternatively, cross-entropy loss can be calculated on the image-text similarity score and the training score label of each second training region image to obtain the image-text matching loss value (where the training score label can be 0 (i.e., image-text mismatch) or 1 (i.e., image-text match), and the image-text similarity score can be a value within the range of 0 to 1), etc.; this embodiment of the invention does not limit this.

[0142] Optionally, when determining the model loss value based on the image-text matching loss value, the electronic device can use the image-text matching loss value as the model loss value; or, it can calculate the entity verification loss value based on the open entity verification results and training entity verification result labels (such as error correction entity verification result labels) of each second training region image. Then, the model loss value can be determined based on the image-text matching loss value and the entity verification loss value. In this case, a second training region image data can also include the open entity verification results and training entity verification result labels of the corresponding second training region image; or, the electronic device can also use the region description text and training entity annotation set labels of each second training region image. The consistency loss value is determined by the signature, and then the model loss value is determined based on the image-text matching loss value, entity verification loss value, and consistency loss value. At this time, the image data of a second training region may also include the region description text of the corresponding second training region image and the training entity annotation set label. The training entity annotation set label of a second training region image may be the target entity annotation set or the error correction annotation set of the corresponding second training region image, etc. Alternatively, the electronic device may also determine the diversity loss value and perform a weighted summation of the image-text matching loss value, entity verification loss value, consistency loss value, and diversity loss value to obtain the model loss value, etc. The embodiments of the present invention do not limit this.

[0143] Optionally, when calculating the entity verification loss value based on the open entity verification results and training entity verification result labels of each second training region image, the entity verification loss value can be calculated based on the difference between the open entity verification results and training entity verification result labels of each second training region image, such that the open entity verification result can be the confidence level; or, cross-entropy loss can be calculated on the open entity verification results and training entity verification result labels of the second training region image to obtain the entity verification loss value, etc.; the embodiments of the present invention do not limit this.

[0144] Optionally, when determining the consistency loss value based on the region description text and training entity annotation set labels of each second training region image, the electronic device can input the region description text of each second training region image into the semantic alignment model. The semantic alignment model then outputs the predicted probability data of each second training region image (the predicted probability data of a training region image may include the probability of each word (i.e., token) in the training entity annotation set labels of the corresponding training region image). The consistency loss value is then calculated using the predicted probability data of each second training region image, such as calculating the cross-entropy loss using the predicted probability data of each second training region image. Optionally, the semantic alignment model can be any large language model, such as a description generation model.

[0145] Optionally, when determining the diversity loss value, the repetition penalty can be determined based on the region description text of each second training region image (i.e., the penalty term can be calculated based on the repetition count of n-tuples (such as single characters, double words, and triple characters, where n is a positive integer) in the region description text; for example, when the repetition count of a word element is 3, the penalty for that word element can be 2, and the repetition penalty can be the product of the sum of the penalties for all word elements and the penalty coefficient), and the repetition penalty can be used as the diversity loss value; or, the diversity loss value can be determined based on the standard candidate annotation set and the training entity annotation set labels of each second training region image, such as the reciprocal or negative number of the sum of the total cosine similarity between the standard candidate annotation set and the training entity annotation set labels of each second training region image, etc.; the embodiments of the present invention do not limit this. For example, for any second training region image in the second training region image dataset, and any standard candidate label in the standard candidate label set of any second training region image, the electronic device can calculate the cosine similarity between any standard candidate label and each entity label in the training entity label set of any second training region image, and take the maximum cosine similarity among the cosine similarities between any standard candidate label and each entity label in the training entity label set of any second training region image as the cosine similarity of any standard candidate label, and thus take the sum of the cosine similarities of each standard candidate label of each second training region image as the total cosine similarity, and so on.

[0146] For example, electronic devices can use Formula 2.4 to determine the model loss value:

[0147] Equation 2.4

[0148] Among them, L total L can represent the model loss value. match This can represent the image-text matching loss value (which can be used to optimize the scoring accuracy of the image-text matching scoring module, i.e., to optimize the accuracy of the image-text matching model), L entity This can represent the entity validation loss value (which can be used to optimize the entity validation module's ability to determine entity existence, i.e., to optimize the open lexicon validation model), L consistency The consistency loss value can be represented (and can be used to ensure semantic alignment between the region description text (i.e., the description phrase) and the annotation set), L diversityThe diversity loss value can be represented (and can be used as a regularization term to encourage diversity in generated results, such as encouraging diversity in region description text generation). λ1, λ2, and λ3 can be corresponding weight hyperparameters (i.e., weight coefficients) to balance the importance of each loss term; optionally, each weight hyperparameter here can be set empirically or according to actual needs, and this embodiment of the invention does not limit this. Optionally, the parameters of the model to be trained may include a description generation model, an image-text matching model, and an open vocabulary validation model, etc.

[0149] For example, taking the second training region image dataset as an example where the dataset is selected from the error correction sample pool, the electronic device can use Formula 2.5 to optimize the model parameters in the model to be trained in the direction of reducing the model loss value:

[0150] Equation 2.5

[0151] Where, θ model It can represent the model parameters (i.e., the model parameter vector) in the model to be trained. D can represent the gradient of the loss function L with respect to the model parameters. pool η2 can represent the image data set of the second training region, and η2 can represent the model parameter learning rate; optionally, the model parameter learning rate can be set according to experience or actual needs, and this embodiment of the invention does not limit this. Based on this, this embodiment of the invention can use the gradient descent method to update the model parameters.

[0152] Optionally, the electronic device can iteratively execute the above-described acquisition of the second training region image data set to optimize the model parameters in the model to be trained. For example, the second training region image data set can be acquired at preset model parameter update intervals to optimize the model parameters in the model to be trained; or the second training region image data set can be acquired when the number of incremental samples (i.e., the number of newly added samples) in the error correction sample pool is detected to be greater than a preset incremental sample threshold to optimize the model parameters in the model to be trained, etc. This embodiment of the invention does not limit this. Optionally, the preset model parameter update interval can be set according to experience or actual needs, and this embodiment of the invention does not limit this. Optionally, the preset model parameter update interval can be the same as or different from the preset gate threshold parameter update interval, and this embodiment of the invention does not limit this.

[0153] Based on this, the data annotation system (i.e., the electronic device can use the data annotation system) periodically or after accumulating sufficient data, uses a sample pool to drive closed-loop updates. Optionally, the electronic device can update the data according to an update strategy; for example, such as... Figure 5As shown, the update strategy may include, but is not limited to, at least one of the following: adjusting the quality inspection strategy, updating the dictionary resources, and fine-tuning the model parameters, etc., which are not limited in this embodiment of the invention. Specifically, adjusting the quality inspection strategy can be used to adjust the gating threshold parameters (i.e., adjusting the threshold of quality gating) and / or the weights of the fusion strategy (i.e., the weight coefficients for calculating the image-text similarity score, such as adjusting according to preset weight coefficient adjustment rules or adjusting according to weight coefficient adjustment operations, etc.). Updating the dictionary resources may include updating the thesaurus resources such as the synonym mapping table and black / white lists (e.g., periodically updating the thesaurus mapping table by adding new synonym mappings, etc.). Fine-tuning the model parameters can be used for incremental training or fine-tuning of the model to be trained so that the model to be trained learns from errors. Optionally, the preset weight coefficient adjustment rules may be set according to experience or actual needs, which are not limited in this embodiment of the invention. Based on this, the updated quality inspection strategy, resources, and model parameters can be applied to subsequent batches of data annotation generation processes, thereby achieving iterative enhancement of the overall annotation performance of the data annotation generation system.

[0154] In summary, the embodiments of this invention establish a complete feedback loop of "problem discovery - sample collection - strategy / model update." The data annotation generation system can continuously learn from manual error correction and automatic quality inspection failures, constantly optimizing its generation, quality inspection, and decision-making capabilities. This allows the annotation quality to continuously improve with use, possessing long-term evolutionary capabilities and gradually adapting to the complex changes in low-altitude scenarios (such as gradually adapting to the professional terminology and scenario characteristics of the low-altitude field). This effectively achieves closed-loop iterative enhancement, making the data annotation generation system more accurate with use. Optionally, the data annotation generation system may include, but is not limited to, at least one of the following: a region candidate generation module, a description generation module, a text-image matching scoring module, an entity extraction and normalization module, an open-vocabulary entity verification module, a quality gating and dual-track output module, a manual review module, and a sample pool and closed-loop update module, etc. The embodiments of this invention do not limit this.

[0155] Optionally, the electronic device can also record complete quality metadata (such as text-image similarity score, verification result, model version, decision threshold, etc.) for the data annotation of each region image, forming an immutable chain of evidence. This not only supports the auditing and traceability of annotation results, but also provides a data foundation for problem localization and analysis in closed-loop updates, so as to achieve full-process traceability of quality metadata and evidence chains. Furthermore, the entire process of this embodiment only requires input images, without relying on expensive multi-sensor data or preset closed category systems. Utilizing powerful pre-trained multimodal models and open vocabulary models, it can handle a constant stream of new low-altitude concepts (such as novel defects), with strong generalization ability and low deployment cost.

[0156] Based on this, the data annotation generation method provided in this invention has broad application potential in multimodal dataset construction, computer vision, artificial intelligence, and other fields, especially in low-altitude scenarios requiring large-scale, high-quality region-text pairs of data; such as low-altitude power inspection (which can automatically annotate key components such as insulators, towers, and fittings in aerial videos and their defects (such as breakage, corrosion, and foreign object hanging), and build a high-precision defect detection dataset), low-altitude photovoltaic operation and maintenance (which can perform regional-level annotation on photovoltaic panel images taken by drones, generating defect descriptions such as "hot spots," "cracks," and "occlusions," and realize automated fault diagnosis data production for photovoltaic power stations), low-altitude agricultural monitoring (which can annotate diseased areas, pest areas, and growth status on crop images collected by plant protection drones, providing data support for precision pesticide application), low-altitude security and transportation (which can provide fine-grained descriptions of vehicles, pedestrian flow, and abnormal behaviors (such as illegal parking and gatherings) in urban low-altitude monitoring images, improving the level of intelligent urban governance), and so on. Accordingly, the embodiments of the present invention can greatly reduce the construction cost and cycle of high-quality multimodal datasets, and provide a reliable data infrastructure for the development of the next generation of low-altitude multimodal AI (Artificial Intelligence) models.

[0157] This invention, after acquiring image data to be processed, can determine at least one candidate region image from the image data; call a description generation model to generate region description text for each candidate region image in the at least one candidate region image; and call an image-text matching model to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the corresponding candidate region image's region description text. Further, for any candidate region image in the at least one candidate region image, the region description text of any candidate region image can be parsed to obtain a set of standard candidate annotations for any candidate region image; and an open vocabulary verification model can be called to output the confidence level of each standard candidate annotation in the set of standard candidate annotations for any candidate region image. The confidence level of a standard candidate annotation for any candidate region image refers to the confidence level of the corresponding standard candidate annotation for the existence of any candidate region image; thus, based on the image-text similarity score of any candidate region image and the confidence levels of each standard candidate annotation of any candidate region image, a gating index value for any candidate region image can be determined. Based on this, a set of target region images can be determined from at least one candidate region image based on the gating index value of each candidate region image. This allows for the generation of target data annotations for each target region image in the target region image set. The target data annotation for a region image includes the corresponding region description text. The target data annotations for each target region image are then added to the target data annotation set. As can be seen, this embodiment of the invention can effectively suppress illusions through dual automatic quality checks. It combines "global image-text consistency scoring" (i.e., image-text similarity score) and "fine-grained open-vocabulary entity verification" (i.e., confidence level) to form a dual quality check mechanism. This not only controls the overall descriptive relevance but also deeply verifies the authenticity of each entity in the region description text. In particular, it fundamentally suppresses the illusion problem of multimodal large models in low-altitude technical terms, significantly improving annotation accuracy.

[0158] Based on the description of the relevant embodiments of the data annotation generation method above, this invention also proposes a data annotation generation device, which can be a computer program (including program code) running in an electronic device; such as Figure 6 As shown, the data annotation generation apparatus may include an acquisition unit 601 and a processing unit 602. The data annotation generation apparatus can perform... Figure 1 or Figure 3 The data annotation generation method shown, i.e., the data annotation generation device, can operate the above-mentioned unit:

[0159] Acquisition unit 601 is used to acquire image data to be processed;

[0160] Processing unit 602 is used to determine at least one candidate region image from the image data to be processed;

[0161] The processing unit 602 is further configured to call the description generation model to generate region description text for each candidate region image in the at least one candidate region image;

[0162] The processing unit 602 is further configured to invoke the image-text matching model to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the region description text of the corresponding candidate region image. The image-text similarity score of a candidate region image is determined based on the model output matching score of the corresponding candidate region image generated by the image-text matching model.

[0163] The processing unit 602 is further configured to determine the gating index value of each candidate region image based on the image-text similarity score of each candidate region image; and to determine the target region image set from the at least one candidate region image based on the gating index value of each candidate region image.

[0164] The processing unit 602 is further configured to generate target data annotations for each target region image in the target region image set, wherein the target data annotation for a region image includes the region description text of the corresponding region image; and add the target data annotations for each target region image to the target data annotation set;

[0165] The processing unit 602 can also be used for:

[0166] For any candidate region image among the at least one candidate region image, perform text parsing on the region description text of any candidate region image to obtain a standard candidate annotation set for any candidate region image;

[0167] The open vocabulary verification model is invoked to output the confidence scores of each standard candidate annotation in the standard candidate annotation set of any candidate region image. The confidence score of a standard candidate annotation of any candidate region image refers to the confidence score of the corresponding standard candidate annotation for the existence of any candidate region image.

[0168] When determining the gating index value of each candidate region image based on the image-text similarity score of each candidate region image, the processing unit 602 can specifically be used for:

[0169] Based on the image-text similarity score of any candidate region image and the confidence level of each standard candidate label of any candidate region image, the gating index value of any candidate region image is determined.

[0170] In one implementation, the description generation model is a multimodal large model; when the processing unit 602 calls the description generation model to generate region description text for each candidate region image in the at least one candidate region image, it can be specifically used for:

[0171] Obtain the image description information of the image data to be processed;

[0172] The description generation model is invoked to generate region description text for each candidate region image based on the image description information and each candidate region image in the at least one candidate region image.

[0173] In another implementation, when the processing unit 602 calls the image-text matching model to determine the image-text similarity score of each candidate region image, it can specifically be used to:

[0174] The image-text matching model is invoked to generate a model output matching score for any candidate region image based on the candidate region image and the region description text of the candidate region image.

[0175] Optical character recognition is performed on any candidate region image to obtain the optical character recognition result of the candidate region image; and based on the region description text of the candidate region image and the optical character recognition result, the optical character recognition score of the candidate region image is determined.

[0176] Based on the model output matching score and optical character recognition score of any candidate region image, the image-text similarity score of any candidate region image is determined.

[0177] In another embodiment, when the processing unit 602 determines the target region image set from the at least one candidate region image based on the gating index value of each candidate region image, it may specifically be used to:

[0178] Based on the gating index value of any candidate region image, determine whether any candidate region image passes the gating;

[0179] If any candidate region image passes the gating, then the confidence scores of each standard candidate label of the candidate region image are used to perform a miss filter on the standard candidate label set of the candidate region image to obtain the target entity label set of the candidate region image. Then, consistency cross-validation is performed on the region description text and the target entity label set of the candidate region image. When any candidate region image passes the consistency cross-validation, the candidate region image is taken as a target region image in the target region image set.

[0180] If any candidate region image fails the gating, then that candidate region image will not be included as a target region image in the target region image set; wherein, the image data of the candidate region image that fails the gating can be added to the image data set to be reviewed.

[0181] In another embodiment, the acquisition unit 601 can also be used for:

[0182] Obtain a set of image data for a first training region, wherein each set of image data for a first training region includes gating decision indication information for that first training region image;

[0183] Processing unit 602 can also be used for:

[0184] Based on the gating judgment indication information of each first training region image in the first training region image data set, determine whether each first training region image is a region image that passes the gating error judgment or a region image that fails the gating correct judgment, so as to determine the number of region images that pass the gating error judgment and the number of region images that fail the gating correct judgment from the first training region image data set.

[0185] Based on the number of region images that passed the gate erroneously and the number of region images that failed the gate correctly, the gate threshold parameter is updated. The gate threshold parameter is used to determine whether a region image passes the gate.

[0186] In another embodiment, the acquisition unit 601 can also be used for:

[0187] Obtain a set of image data for the second training region. A set of image data for the second training region includes: the image-text similarity score and the training score label of the second training region image.

[0188] Processing unit 602 can also be used for:

[0189] Based on the image data of each second training region in the second training region image dataset, determine the image-text matching loss value;

[0190] Based on the image-text matching loss value, the model loss value is determined; and the model parameters in the model to be trained are optimized in the direction of reducing the model loss value; wherein, the model to be trained includes at least one of the following: the description generation model and the image-text matching model.

[0191] According to one embodiment of the present invention, Figure 6Each unit in the data annotation generation device shown can be individually or entirely merged into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effect of the embodiments of the present invention. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, any data annotation generation device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0192] According to another embodiment of the present invention, it is possible to perform operations such as those described above by running on a general-purpose electronic device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 1 or Figure 3 The computer program (including program code) involved in each step of the corresponding method shown, to construct such... Figure 6 The data annotation generation apparatus shown herein, and the data annotation generation method for implementing embodiments of the present invention, are described. The computer program may be stored on, for example, a computer storage medium, loaded onto the aforementioned electronic device via the computer storage medium, and run therein.

[0193] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method according to an embodiment of the present invention.

[0194] An exemplary embodiment of the present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0195] An exemplary embodiment of the present invention also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of the present invention.

[0196] refer to Figure 7The present invention will now be described in the form of a structural block diagram of an electronic device 700 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0197] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0198] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0199] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the data annotation generation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to perform the data annotation generation method by any other suitable means (e.g., by means of firmware).

[0200] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0201] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0202] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0204] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0205] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0206] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A data annotation generation method, characterized in that, include: Acquire image data to be processed, and determine at least one candidate region image from the image data to be processed; The description generation model is invoked to generate region description text for each candidate region image in the at least one candidate region image; The image-text matching model is invoked to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the region description text of the corresponding candidate region image. The image-text similarity score of a candidate region image is determined based on the model output matching score of the corresponding candidate region image generated by the image-text matching model. Based on the image-text similarity score of each candidate region image, a gating index value is determined for each candidate region image; and based on the gating index value of each candidate region image, a target region image set is determined from the at least one candidate region image. Generate target data annotations for each target region image in the target region image set, wherein the target data annotation for a region image includes the region description text of the corresponding region image; and add the target data annotations for each target region image to the target data annotation set; The method further includes: For any candidate region image among the at least one candidate region image, perform text parsing on the region description text of any candidate region image to obtain a standard candidate annotation set for any candidate region image; The open vocabulary verification model is invoked to output the confidence scores of each standard candidate annotation in the standard candidate annotation set of any candidate region image. The confidence score of a standard candidate annotation of any candidate region image refers to the confidence score of the corresponding standard candidate annotation for the existence of any candidate region image. The step of determining the gating index value of each candidate region image based on the image-text similarity score of each candidate region image includes: Based on the image-text similarity score of any candidate region image and the confidence level of each standard candidate label of any candidate region image, the gating index value of any candidate region image is determined.

2. The method according to claim 1, characterized in that, The description generation model is a multimodal large model; the invocation of the description generation model to generate region description text for each candidate region image in the at least one candidate region image includes: Obtain the image description information of the image data to be processed; The description generation model is invoked to generate region description text for each candidate region image based on the image description information and each candidate region image in the at least one candidate region image.

3. The method according to claim 1 or 2, characterized in that, The step of calling the image-text matching model to determine the image-text similarity score of each candidate region image includes: The image-text matching model is invoked to generate a model output matching score for any candidate region image based on the candidate region image and the region description text of the candidate region image. Optical character recognition is performed on any candidate region image to obtain the optical character recognition result of the candidate region image; and based on the region description text of the candidate region image and the optical character recognition result, the optical character recognition score of the candidate region image is determined. Based on the model output matching score and optical character recognition score of any candidate region image, the image-text similarity score of any candidate region image is determined.

4. The method according to claim 1 or 2, characterized in that, The step of determining a set of target region images from the at least one candidate region image based on the gating index values ​​of each candidate region image includes: Based on the gating index value of any candidate region image, determine whether any candidate region image passes the gating; If any candidate region image passes the gating, then the confidence scores of each standard candidate label of the candidate region image are used to perform a miss filter on the standard candidate label set of the candidate region image to obtain the target entity label set of the candidate region image. Then, consistency cross-validation is performed on the region description text and the target entity label set of the candidate region image. When any candidate region image passes the consistency cross-validation, the candidate region image is taken as a target region image in the target region image set. If any candidate region image fails the gating, then that candidate region image will not be included as a target region image in the target region image set; wherein, the image data of the candidate region image that fails the gating can be added to the image data set to be reviewed.

5. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain a set of image data for a first training region, wherein each set of image data for a first training region includes gating decision indication information for that first training region image; Based on the gating judgment indication information of each first training region image in the first training region image data set, determine whether each first training region image is a region image that passes the gating error judgment or a region image that fails the gating correct judgment, so as to determine the number of region images that pass the gating error judgment and the number of region images that fail the gating correct judgment from the first training region image data set. Based on the number of region images that passed the gate erroneously and the number of region images that failed the gate correctly, the gate threshold parameter is updated. The gate threshold parameter is used to determine whether a region image passes the gate.

6. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain a set of image data for the second training region. A set of image data for the second training region includes: the image-text similarity score and the training score label of the second training region image. Based on the image data of each second training region in the second training region image dataset, determine the image-text matching loss value; Based on the image-text matching loss value, the model loss value is determined; and the model parameters in the model to be trained are optimized in the direction of reducing the model loss value; wherein, the model to be trained includes at least one of the following: the description generation model and the image-text matching model.

7. A data annotation generation device, characterized in that, The device includes: The acquisition unit is used to acquire the image data to be processed; A processing unit is configured to determine at least one candidate region image from the image data to be processed; The processing unit is also used to call the description generation model to generate region description text for each candidate region image in the at least one candidate region image; The processing unit is further configured to invoke the image-text matching model to determine the image-text similarity score of each candidate region image. The image-text similarity score of a candidate region image is used to indicate the similarity between the corresponding candidate region image and the region description text of the corresponding candidate region image. The image-text similarity score of a candidate region image is determined based on the model output matching score of the corresponding candidate region image generated by the image-text matching model. The processing unit is further configured to determine the gating index value of each candidate region image based on the image-text similarity score of each candidate region image; and to determine the target region image set from the at least one candidate region image based on the gating index value of each candidate region image. The processing unit is further configured to generate target data annotations for each target region image in the target region image set, wherein the target data annotation for a region image includes the region description text of the corresponding region image; and to add the target data annotations for each target region image to the target data annotation set; The processing unit is also used for: For any candidate region image among the at least one candidate region image, perform text parsing on the region description text of any candidate region image to obtain a standard candidate annotation set for any candidate region image; The open vocabulary verification model is invoked to output the confidence scores of each standard candidate annotation in the standard candidate annotation set of any candidate region image. The confidence score of a standard candidate annotation of any candidate region image refers to the confidence score of the corresponding standard candidate annotation for the existence of any candidate region image. When determining the gating index value of each candidate region image based on the image-text similarity score of each candidate region image, the processing unit is specifically used for: Based on the image-text similarity score of any candidate region image and the confidence level of each standard candidate label of any candidate region image, the gating index value of any candidate region image is determined.

8. An electronic device, characterized in that, include: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images