Industrial image automatic annotation methods, apparatus, equipment and storage media

Abstract

This application provides an automatic industrial image annotation method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology. The method acquires an image to be annotated, inputs the image into a visual encoder for feature encoding to obtain image features, acquires a set of domain terms related to the industrial image, obtains a query vector based on an initial noise vector, and obtains key and value vectors based on the image features. The query vector, key vector, and value vector are input into a diffusion model, and iterative feature processing is performed at multiple time steps using the domain terminology set as a constraint to obtain a text latent vector. The text latent vector and the domain terminology set are input into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated. The domain terminology set is used as a constraint throughout the entire annotation generation process, ensuring that the annotation text always revolves around the specific industrial scenario, which can significantly improve the accuracy and standardization of industrial image annotation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods, apparatus, devices and storage media for automatic annotation of industrial images. Background Technology

[0002] With the rapid development of smart manufacturing, industrial IoT, and edge computing, the demand for visual perception and intelligent analysis in industrial settings is constantly increasing, leading to the collection and storage of massive amounts of image data, including equipment operating status, operational behavior, safety hazards, and abnormal events. How to efficiently and accurately perform semantic understanding and text annotation of industrial images has become a key issue in promoting smart manufacturing and industrial digital transformation.

[0003] Related technologies utilize diffusion probability models to progressively model complex data distributions through a "noise-denoising" process, capturing intricate image-text correspondences and high-level semantic features for industrial image annotation. However, these diffusion models focus on general image scenarios and lack knowledge-guided mechanisms tailored to industrial tasks. This results in insufficient understanding of equipment names, operational details, and operating procedures, leading to inadequate adaptability to specific domains such as industrial scenarios. Furthermore, the generated language still lags significantly in terms of professionalism, terminology standardization, and engineering usability. Summary of the Invention

[0004] The main objective of this application is to propose an automatic annotation method, apparatus, device, and storage medium for industrial images, thereby improving the accuracy and standardization of industrial image annotation.

[0005] To achieve the above objectives, a first aspect of this application proposes an automatic industrial image annotation method, comprising:

[0006] Obtain the image to be labeled, and input the image to be labeled into a visual encoder for feature encoding to obtain image features;

[0007] Obtain a set of domain terms related to the industrial image, obtain an initial noise vector, obtain a query vector based on the initial noise vector, obtain a key vector and a value vector based on the image features, input the query vector, the key vector, and the value vector into a diffusion model, and perform iterative feature processing at multiple time steps with the set of domain terms as constraints to obtain a text latent vector;

[0008] The text latent vector and the domain terminology set are input into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated.

[0009] In some embodiments, the domain terminology set includes at least domain-specific terms and general terms. The step of inputting the text latent vector and the domain terminology set into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated includes:

[0010] Based on the text latent vector, obtain the hidden state vector corresponding to each position;

[0011] Calculate the matching score between the hidden state vector and the word vector of each word;

[0012] The priority coefficient corresponding to the domain word is obtained based on the hidden state vector, and the priority coefficient is multiplied by the corresponding matching score to update the matching score;

[0013] Calculate the probability value of each word at the position based on all the matching scores, determine the candidate word corresponding to the position based on the probability value, and obtain the image annotation text based on all the candidate words.

[0014] In some embodiments, obtaining the priority coefficient corresponding to the domain word based on the hidden state vector includes:

[0015] Obtain the priority weight and priority bias, calculate the product of the priority weight and the hidden state vector, add the priority bias to obtain the priority median value, and calculate the activation value corresponding to the priority median value.

[0016] Obtain the upper limit and lower limit of the coefficient, calculate the difference between the upper limit and the lower limit, calculate the product of the activation value and the difference, and add the lower limit to obtain the priority coefficient.

[0017] In some embodiments, the step of inputting the query vector, the key vector, and the value vector into a diffusion model, and performing iterative feature processing at multiple time steps using the domain terminology set as constraints to obtain text latent vectors includes:

[0018] Generate a term embedding sequence corresponding to the domain term set, and generate a term key vector and a term value vector based on the term embedding sequence;

[0019] Obtain the query vector corresponding to the current time step; perform cross-attention calculation on the query vector, the key vector, and the value vector to obtain an image context vector; perform cross-attention calculation on the query vector, the term key vector, and the term value vector to obtain a term context vector; fuse the image context vector and the term context vector to obtain an output vector; calculate the prediction mean and prediction covariance based on the output vector; calculate the query vector corresponding to the next time step based on the prediction mean and prediction covariance; iterate multiple times; and obtain the text latent vector based on the last query vector.

[0020] In some embodiments, the training process of the diffusion model includes at least:

[0021] Obtain the image sample features and initial latent vector corresponding to the sample image. Add Gaussian noise to the initial latent vector according to the time step to obtain the noise data corresponding to each time step. The sample image contains the corresponding sample annotation text.

[0022] Starting from the total time step, the noise data, the current time step, and the image sample features are input into the diffusion model to calculate attention and obtain an iterative latent vector. The iterative mean and iterative covariance are calculated based on the iterative latent vector. The noise data corresponding to the next time step is calculated based on the iterative mean and iterative covariance. This process is repeated multiple times. The last iterative latent vector is used as the predicted latent vector. Text is generated based on the domain terminology set to obtain the predicted text.

[0023] A first loss value is calculated based on the predicted latent vector and the initial latent vector. A second loss value is calculated based on the sample labeled text and the predicted text. A total loss value is calculated based on the first loss value and the second loss value. The model parameters corresponding to the diffusion model are adjusted based on the total loss value until training ends, and the trained diffusion model is obtained.

[0024] In some embodiments, obtaining the initial latent vector corresponding to the sample image includes:

[0025] The sample labeled text is input into a text encoder for feature encoding to obtain a sample feature vector, and the sample feature vector is scaled to obtain a scale feature vector;

[0026] The image sample features are projected to the same dimension as the latent vector of the sample feature vector to obtain the image dimension features. Through at least one repeated expansion, a visual condition matrix with the same text length as the sample labeled text is obtained. The visual condition matrix is ​​multiplied by the gating coefficient and then added to the scale feature vector to obtain the initial latent vector.

[0027] In some embodiments, the training process of the visual encoder includes at least:

[0028] Obtain visual training images and corresponding text descriptions, and take each matching pair of visual training images and text descriptions as a positive sample pair, and take other unmatched pairs of visual training images and text descriptions as negative sample pairs;

[0029] The visual feature vector corresponding to the visual training image is obtained using the visual encoder, and the text feature vector corresponding to the text description is obtained using the text encoder. The similarity between each positive sample pair and the negative sample pair is calculated based on the visual feature vector and the text feature vector, respectively.

[0030] Based on the similarity, a contrastive learning loss value is calculated, and the model parameters corresponding to the visual encoder are adjusted at least according to the contrastive learning loss value to obtain the trained visual encoder.

[0031] In some embodiments, calculating the contrastive learning loss value based on the similarity includes:

[0032] For each of the visual training images, obtain the corresponding positive image similarity value and the first similarity value corresponding to each of the text feature vectors, obtain the image similarity value based on the first similarity value, and obtain the first loss value based on the positive image similarity value and the image similarity value;

[0033] For each of the text descriptions, a second similarity value corresponding to each of the visual feature vectors is obtained, a text similarity value is obtained based on the second similarity value, and a second loss value is obtained based on the image positive similarity value and the text similarity value;

[0034] The contrastive learning loss value is calculated based on the first loss value and the second loss value.

[0035] To achieve the above objectives, a second aspect of this application provides an automatic industrial image annotation device, comprising:

[0036] Image encoding module: used to acquire the image to be labeled, input the image to be labeled into the visual encoder for feature encoding, and obtain image features;

[0037] The diffusion generation module is used to obtain a set of domain terms related to the industrial image, obtain an initial noise vector, obtain a query vector based on the initial noise vector, obtain a key vector and a value vector based on the image features, input the query vector, the key vector, and the value vector into the diffusion model, and perform iterative feature processing at multiple time steps with the set of domain terms as constraints to obtain the text latent vector.

[0038] Decoding module: Used to input the text latent vector and the domain term set into the text decoder for decoding, so as to obtain the image annotation text corresponding to the image to be annotated.

[0039] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0040] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0041] The industrial image automatic annotation method, apparatus, device, and storage medium proposed in this application involve acquiring an image to be annotated, inputting the image into a visual encoder for feature encoding to obtain image features, acquiring a set of domain terms related to the industrial image, obtaining an initial noise vector, obtaining a query vector based on the initial noise vector, obtaining a key vector and a value vector based on the image features, inputting the query vector, key vector, and value vector into a diffusion model, and performing iterative feature processing at multiple time steps using the domain terminology set as a constraint to obtain a text latent vector, and inputting the text latent vector and the domain terminology set into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated. This application embodiment utilizes a set of domain terms strongly related to the industrial image as a constraint guide throughout the annotation generation process, ensuring that the annotation text always revolves around the specific industrial scenario and avoiding generalized expressions that are detached from the industrial context. Specifically, a visual encoder extracts image features from the image to be labeled, converting them into key and value vectors. Simultaneously, a query vector is generated based on an initial noise vector. These three vectors, along with the time step, are input into a diffusion model for iterative feature processing. Under the constraint of a domain terminology set, the diffusion model prioritizes descriptions containing precise equipment names and operational details, progressively optimizing the text latent vectors to continuously correct them towards conforming to industry terminology standards and accurately reflecting the actual content of the image, thus reducing labeling bias. Furthermore, the text decoder's decoding process also operates within the domain terminology set, forcibly prioritizing the use of standardized industry terminology and avoiding colloquial or non-standardized expressions. Therefore, the embodiments of this application can significantly improve the accuracy and standardization of industrial image labeling. Attached Figure Description

[0042] Figure 1 This is a flowchart of the automatic industrial image annotation method provided in the embodiments of this application.

[0043] Figure 2A flowchart illustrating the training process of the visual encoder provided in this application embodiment.

[0044] Figure 3 This is a flowchart of calculating the contrastive learning loss value based on similarity, provided in an embodiment of this application.

[0045] Figure 4 This is a flowchart provided in this application embodiment, which inputs query vector, key vector, and value vector into a diffusion model, performs iterative feature processing at multiple time steps using a set of domain terms as constraints, and obtains text latent vectors.

[0046] Figure 5 This is a flowchart provided in this application embodiment of which inputs text latent vectors and a set of domain terms into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated.

[0047] Figure 6 This is a schematic diagram illustrating the reasoning process of the automatic industrial image annotation method provided in the embodiments of this application.

[0048] Figure 7 This is a flowchart of the training process of the diffusion model provided in the embodiments of this application.

[0049] Figure 8 This is a flowchart of obtaining the initial latent vector corresponding to the sample image provided in the embodiments of this application.

[0050] Figure 9 This is a schematic diagram illustrating the training process of the diffusion model provided in this application embodiment.

[0051] Figure 10 This is a structural block diagram of an industrial image automatic annotation device provided in another embodiment of this application.

[0052] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0056] First, let's analyze some of the terms used in this application:

[0057] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0058] With the rapid development of intelligent manufacturing, the Industrial Internet of Things (IIoT), and edge computing, the demand for visual perception and intelligent analysis in industrial settings is constantly increasing, leading to the collection and storage of massive amounts of image data. These images contain rich information about production activities, including equipment operating status, operational behaviors, safety hazards, and abnormal events, serving as a crucial source of information for intelligent monitoring and analysis of industrial systems. How to efficiently and accurately perform semantic understanding and text annotation of industrial images has become a key issue in promoting intelligent manufacturing and industrial digital transformation. Especially in complex scenarios such as metallurgy, construction, and manufacturing, the automated analysis of visual data and the generation of descriptive text are of great significance for improving operators' situational awareness and reducing operational risks.

[0059] In practical applications, initial image annotation primarily relied on manual methods. Typically, professionals observed the image content and filled in corresponding text descriptions. This method was inefficient, time-consuming, costly, and heavily dependent on expert experience. Furthermore, it was difficult to achieve rapid and standardized processing when dealing with large-scale image data. In addition, differences in terminology and expression styles among different personnel affected the consistency and reusability of annotation results.

[0060] Therefore, in recent years, automatic image annotation technology has gradually combined computer vision and natural language generation methods. A common approach employs an encoder-decoder architecture: using convolutional neural networks to extract image features, and then using recurrent neural networks to generate text descriptions word by word. However, these models still belong to sequence decoding models, and their application in industrial scenarios faces the following challenges: 1. Insufficient ability to extract complex features: Industrial environments commonly have complex backgrounds, occlusions, dust, and strong reflections, resulting in low image quality and severely impacting the model's ability to extract complex features; 2. One-sided semantic representation and lack of contextual consistency: Word-by-word decoding is prone to redundancy or semantic jumps, making it difficult to maintain the overall coherence and logical consistency of the generated text; 3. Inaccurate use of technical terms: Lack of industrial domain knowledge leads to non-standard use of terminology in equipment names, process flows, and operational behaviors; 4. Insufficient stability of text generation: Sequence decoding mechanisms are sensitive to input and lack robustness in scenarios with high semantic diversity and structural uncertainty.

[0061] Therefore, related technologies introduce diffusion probability models, which use a "noise-denoising" process to progressively model complex data distributions, capturing complex image-text correspondences and high-level semantic features for industrial image annotation. However, these diffusion models focus on general image scenarios and lack knowledge-guided mechanisms for industrial tasks. This results in insufficient understanding of equipment names, operating details, and operational procedures. The semantic alignment between image details and generated text remains imprecise, with omissions or deviations in description. Consequently, their adaptability to specific domains such as industrial scenarios is insufficient, and the generated language still lags significantly in terms of professionalism, terminology standardization, and engineering usability.

[0062] It is evident that both encoder-decoder architectures and diffusion models have significant shortcomings in automatic industrial image annotation, particularly lacking the ability to integrate industrial domain knowledge, making it difficult to guarantee the accuracy, professionalism, and consistency of the generated text. Therefore, introducing knowledge enhancement mechanisms not only helps improve the model's semantic understanding and detail capture capabilities in complex industrial scenarios but also significantly improves the professional standardization and engineering usability of the generated text, thereby effectively solving the core challenges in current automatic industrial image annotation.

[0063] Based on this, embodiments of this application provide an automatic industrial image annotation method, apparatus, device, and storage medium. Throughout the annotation generation process, a set of domain terms strongly related to the industrial image is used as a constraint guide, ensuring that the annotation text always revolves around the specific industrial scenario and avoids generalized expressions detached from the industrial context. Specifically, image features of the image to be annotated are extracted by a visual encoder and converted into key vectors and value vectors. Simultaneously, a query vector is generated based on the initial noise vector. These three elements, along with the time step, are input into a diffusion model for iterative feature processing. Under the constraint of the domain terminology set, the diffusion model prioritizes expressions containing precise equipment names and operational details, gradually optimizing the text latent vector and continuously correcting it towards conforming to industrial terminology standards and fitting the actual content of the image, reducing annotation deviation. Furthermore, the text decoder's decoding process is also performed within the domain terminology set, forcibly prioritizing the use of standardized industrial terms and avoiding colloquial or non-standardized expressions. Therefore, embodiments of this application can significantly improve the accuracy and standardization of industrial image annotation.

[0064] This application provides an automatic industrial image annotation method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the automatic industrial image annotation method in this application is described.

[0065] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0066] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0067] The industrial image automatic annotation method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting automatic industrial image annotation, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. This industrial image automatic annotation method can be executed by the terminal or the server, or by the terminal and the server working together.

[0068] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or a network, etc., and this embodiment does not impose any limitations.

[0069] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] The following describes an automatic industrial image annotation method in the embodiments of this application.

[0071] Figure 1 This is an optional flowchart of the industrial image automatic annotation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 110 to 130. It is also understood that this embodiment... Figure 1 The order of steps 110 to 130 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0072] Step 110: Obtain the image to be labeled, input the image to be labeled into the visual encoder for feature encoding, and obtain the image features.

[0073] In one embodiment, the image to be labeled is an industrial scene image, including equipment operating status, work behavior, safety hazards, and abnormal events in a specific scene. Since images acquired in industrial production environments often have cluttered backgrounds, large variations in lighting, and are obscured by dust or smoke, directly using general visual features would lead to reduced accuracy in downstream generation. Therefore, in this embodiment, the visual encoder introduces a contrastive learning mechanism based on industrial domain knowledge for domain-adaptive fine-tuning, making it more sensitive to the core semantics of the industrial environment during feature extraction.

[0074] In one embodiment, the core objective of the visual encoder is to transform an image into a feature vector that can be aligned with the semantics of the text. Therefore, through a training process, the visual encoder gradually learns the association patterns between the image and the text. (Refer to...) Figure 2 , Figure 2 The flowchart of the training process of the visual encoder provided in the embodiments of this application includes at least the following steps:

[0075] Step 210: Obtain the visual training images and their corresponding text descriptions. Treat each pair of matching visual training images and text descriptions as positive sample pairs, and treat other mismatched visual training images and text descriptions as negative sample pairs.

[0076] In one embodiment, a large number of visual training images and their corresponding text descriptions are acquired. These text descriptions can be descriptions of the features, states, functions, etc., of objects in the images, and are used together as training data for the visual encoder. Then, matching visual training images and text descriptions are used as positive sample pairs, representing the correct correspondence between the visual training images and text descriptions. Next, visual training images and non-matching text descriptions are manually selected to form negative sample pairs, which are used to allow the visual encoder to learn the differences between images and text.

[0077] Step 220: Use the visual encoder to obtain the visual feature vector corresponding to the visual training image, and use the text encoder to obtain the text feature vector corresponding to the text description. Calculate the similarity of each positive sample pair and negative sample pair based on the visual feature vector and the text feature vector respectively.

[0078] In one embodiment, the first six layers of the BERT model can be selected as the text encoder for the training phase of the visual encoder, while the last six layers of the BERT model serve as the corresponding text decoder for the subsequent inference phase. Alternatively, the visual encoder can be the BLIP model. It is understood that the text encoder can be selected based on the actual situation, such as other pre-trained language models like RoBERTa and ELECTRA, or multimodal semantic alignment models like CLIP and ALIGN, to achieve mapping and semantic representation between visual and text spaces. The text decoder can be other visual backbone networks, such as ViT, Swin Transformer, and ConvNeXt, or models obtained using optimization strategies such as distillation learning and multi-task learning to enhance the model's ability to extract complex features from industrial images. This is only an illustration of the text encoder and text decoder and does not imply any limitation on them.

[0079] In one embodiment, a visual encoder is used. Visual training images The pixels are processed layer by layer to extract key features of the visual training image, such as edges, textures, and shapes, ultimately resulting in a visual feature vector that can represent the image. This process can be represented as:

[0080]

[0081] The text encoder encodes the words or characters in the text description c, captures the semantic information of the text description, and generates the corresponding text feature vector.

[0082] Then, for each positive and negative sample pair, the corresponding similarity is obtained by calculating the distance or similarity metric between the visual feature vector and the text feature vector, such as cosine similarity or Euclidean distance. Here, the similarity between the visual and text feature vectors can be calculated in matrix form. Taking the i-th visual feature vector and the j-th text feature vector as an example, their similarity is expressed as:

[0083]

[0084] in, This represents the normalized feature of the i-th visual feature vector. This represents the normalized feature of the j-th text feature vector. This represents the similarity between the i-th visual feature vector and the j-th text feature vector. This represents a temperature coefficient set according to actual conditions, used to control the smoothness of the distribution. It can be understood that when i=j, it corresponds to a positive sample pair, and the rest are negative sample pairs.

[0085] Step 230: Calculate the contrastive learning loss value based on similarity, and adjust the model parameters corresponding to the visual encoder at least according to the contrastive learning loss value to obtain the trained visual encoder.

[0086] In one embodiment, reference is made to Figure 3 , Figure 3 This is a flowchart of calculating the contrastive learning loss value based on similarity, provided in an embodiment of this application, which specifically includes the following steps:

[0087] Step 310: For each visual training image, obtain the corresponding positive image similarity value and the first similarity value corresponding to each text feature vector, obtain the image similarity value based on the first similarity value, and obtain the first loss value based on the positive image similarity value and the image similarity value.

[0088] In one embodiment, taking the i-th visual training image as an example, its corresponding image positive similarity value is The first similarity value corresponding to the j-th text feature vector is The image similarity value obtained based on the first similarity value is Therefore, the first loss value obtained based on the positive image similarity value and the image similarity value is expressed as:

[0089]

[0090] Step 320: For each text description, obtain the second similarity value corresponding to each visual feature vector, obtain the text similarity value based on the second similarity value, and obtain the second loss value based on the image positive similarity value and the text similarity value.

[0091] In one embodiment, taking the j-th text description as an example, its second similarity value with the i-th visual feature vector is... The text similarity value obtained based on the second similarity value is Therefore, the second loss value is obtained based on the image positive similarity value and the text similarity value, and is expressed as:

[0092]

[0093] Step 330: Calculate the contrastive learning loss value based on the first loss value and the second loss value.

[0094] In one embodiment, the contrastive learning loss value calculated based on the first loss value and the second loss value is expressed as: :

[0095]

[0096] Then, the model parameters corresponding to the visual encoder are adjusted based on at least the contrastive learning loss value to obtain the trained visual encoder.

[0097] Therefore, by comparing the learning loss values, the similarity of positive samples can be maximized and the similarity of negative samples can be minimized, thereby prompting the visual encoder to approach the corresponding domain description and avoid irrelevant descriptions in the feature space. The visual encoder obtained after training can effectively capture fine-grained features and key semantic information in industrial images. It not only retains general visual recognition capabilities but also has enhanced perception capabilities for key information such as the unique structure and location of industrial scenes, thereby improving cross-modal alignment accuracy and the accuracy of subsequent image annotation text generation.

[0098] After the visual encoder is trained, the image to be labeled I can be input into the visual encoder for feature encoding to obtain image features v.

[0099] Step 120: Obtain a set of domain terms related to industrial images, obtain an initial noise vector, obtain a query vector based on the initial noise vector, obtain a key vector and a value vector based on image features, input the query vector, key vector, and value vector into the diffusion model, and perform iterative feature processing at multiple time steps with the set of domain terms as constraints to obtain the text latent vector.

[0100] In one embodiment, during the generation stage of automatic industrial image annotation, the descriptive text not only needs to be semantically fluent, but also must strictly include specialized vocabulary in fields such as process flow, equipment names, operating procedures, and safety terms; otherwise, it may result in missing information or misleading information. Therefore, this application embodiment introduces an industrial terminology constraint mechanism to construct a set of domain-specific terms, ensuring both the professionalism and accuracy of the generated results.

[0101] Specifically, the domain terminology set includes at least domain-specific terms and general terms. First, a domain terminology set is constructed, containing domain-specific terms derived from specific industrial standard documents, process manuals, and historical annotation data, covering key information such as common machinery, production processes, and safety measures. Then, a general raw vocabulary is obtained. The words in this thesaurus are general terms. Finally, the domain terminology set is compared with the original thesaurus. Merge to form an enhanced vocabulary The enhanced vocabulary is used as the final set of domain terms. This domain terminology set enables the annotation process to use industry-specific terminology at the lexical level. Through this enhancement process, the embodiments of this application not only help improve the semantic understanding and detail capture capabilities of relevant models in complex industrial scenarios, but also significantly improve the professional standardization and engineering usability of the generated text, thereby effectively solving the core challenges in current automatic industrial image annotation.

[0102] Then, we need to obtain the initial noise vector, which is pure Gaussian noise, denoted as... Where T is the total time step. This represents the unit covariance matrix, which follows a normal distribution with a mean of 0 and a variance of 1. It needs to be processed by a diffusion model to obtain the final noise-free data. Next, we obtain the query weights W that have been learned in the diffusion model. Q The query weights are multiplied by the initial noise vector to obtain the query vector, while the learned key weights W are also obtained. K Sum weight W V Image features v are compared with key weights W respectively. K Sum weight W V Multiplying them yields the key vector and the value vector.

[0103] In one embodiment, reference is made to Figure 4 , Figure 4 This application provides a flowchart illustrating how query vectors, key vectors, and value vectors are input into a diffusion model, and how a domain terminology set is used as a constraint to perform iterative feature processing at multiple time steps to obtain text latent vectors. The flowchart specifically includes the following steps:

[0104] Step 410: Generate the term embedding sequence corresponding to the domain term set, and generate term key vector and term value vector based on the term embedding sequence.

[0105] In one embodiment, in order to use the domain terminology set as a constraint during the diffusion process, it is necessary to generate a term embedding sequence corresponding to the domain terminology set. , where the term embedding sequence It is constructed from the word vectors corresponding to all words in the domain terminology set. Then, the term embedding sequence is... Respectively with bond weight W K Sum weight W V Multiplying them yields the corresponding term key vector and term value vector.

[0106] Step 420: Obtain the query vector corresponding to the current time step. Perform cross-attention calculation on the query vector, key vector, and value vector to obtain the image context vector. Perform cross-attention calculation on the query vector, term key vector, and term value vector to obtain the term context vector. Fuse the image context vector and term context vector to obtain the output vector. Calculate the prediction mean and prediction covariance based on the output vector. Calculate the query vector corresponding to the next time step based on the prediction mean and prediction covariance. Iterate multiple times and obtain the text latent vector based on the last query vector.

[0107] In one embodiment, the diffusion model iterates according to time steps, with a preset number of time steps T. In the first iteration, t=T. At this time, the input data processed by the diffusion model is the initial noise vector, and then the query vector is obtained based on the initial noise vector. The diffusion model can consist of 12 Transformer encoder blocks, with a cross-modal cross-attention module embedded in each layer, enabling it to fully integrate the key vector K from the visual encoder with the time step t during the denoising process. It is understood that the diffusion model can be any model with cross-modal generative capabilities, including but not limited to: Transformer-based autoregressive generative models, variational autoencoder-based generative models, or generative adversarial network-based generative models, to achieve automatic annotation from industrial images to text. This embodiment is only illustrative and does not represent a limitation.

[0108] In other words, during each iteration, the diffusion model processes the input data through 12 Transformer encoder layers to obtain an output vector. This output vector is then fed into two fully connected layers. The predicted mean and predicted covariance are calculated based on the output vector, and the predicted mean and predicted covariance are used to generate the input data for the next time step (t=T-1). This input data is then compared with the query weight W. QThe multiplication yields the updated query vector. The process is iterated T times to obtain the processed data at time step t=0, which is then used as the text latent vector. The specific iterative process is as follows:

[0109]

[0110] in, This represents the input data corresponding to time step t. This represents the input data corresponding to time step t. and These are abstract functions of the diffusion model parameterization, used to represent the predicted mean and predicted covariance at time step t, respectively. This is sampling noise.

[0111] The process of obtaining the output vector using the diffusion model is described in detail below.

[0112] First, cross-attention is calculated between the query vector and the key and value vectors to obtain the image context vector. Next, cross-attention is calculated between the query vector and the term key and value vectors to obtain the term context vector. Then, the results of these two cross-attention calculations are fused, that is, the image context vector and the term context vector are fused to obtain the output vector. This fusion can be achieved through weighted summation, element-wise multiplication, or attention-weighted fusion, etc., and this embodiment does not limit the specific method used.

[0113] In the above process, the query vector is cross-attentioned with key / value vectors from different sources, and then the output vector is obtained by fusion. This enables the alignment of visual features with semantic terms, allowing the diffusion model to capture both the visual context of the image and the semantic context of the technical terms, thus avoiding the problem of visual-semantic disconnect in the generated results.

[0114] Step 130: Input the text latent vector and the set of domain terms into the text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated.

[0115] In one embodiment, once the text latent vectors are obtained, the decoding stage can begin. (See also...) Figure 5 , Figure 5 This is a flowchart provided in this application embodiment of the process of inputting text latent vectors and a set of domain terms into a text decoder for decoding to obtain image annotation text corresponding to the image to be annotated. The flowchart specifically includes the following steps:

[0116] Step 510: Obtain the hidden state vector corresponding to each position based on the text latent vector.

[0117] In one embodiment, the text latent vector contains global semantic information of the entire image-annotated text, and is a matrix containing the semantics of the entire sentence. Therefore, for each word at each position in the image-annotated text, the vector at the corresponding position in the text latent vector matrix is ​​obtained as the hidden state vector. For example, the hidden state vector corresponding to the i-th position is... In addition, the process of extracting the hidden state vector needs to retain the temporal information of each position to ensure that the hidden state vector can accurately correspond to the specific words in the image-annotated text, which facilitates the subsequent matching process.

[0118] Step 520: Calculate the matching score between the hidden state vector and the word vector of each word.

[0119] In one embodiment, for each word in the domain terminology set, its corresponding word vector is obtained, and then the matching score between the hidden state vector and the word vector of each word is calculated. Taking the i-th position as an example, its hidden state vector... domain terminology set The matching score between the word vectors of the u-th word is represented as: .

[0120] Step 530: Obtain the priority coefficient corresponding to the domain word based on the hidden state vector, multiply the priority coefficient by the corresponding matching score, and update the matching score.

[0121] In one embodiment, to improve the matching priority of domain terms in the domain terminology set, for each domain term, such as the first one in the domain terminology set... These words are domain terms, and their word vector transposes are... Therefore, the priority coefficient is obtained based on the hidden state vector. and priority coefficient With corresponding matching score Multiply the results to update the matching score.

[0122] Therefore, the updated matching score is expressed as:

[0123]

[0124] Among them, priority coefficients greater than 1 are weighted before softmax calculation.

[0125] In one embodiment, to avoid the diffusion model forgetting term constraints during long sequence generation, a dynamic gating mechanism is introduced. The priority coefficients corresponding to domain words are obtained based on the hidden state vector. Specifically, this includes: obtaining priority weights and priority biases; calculating the product of the priority weights and the hidden state vectors; adding the priority biases to obtain the priority intermediate value; calculating the activation value corresponding to the priority intermediate value; obtaining the upper limit and lower limit of the coefficients; calculating the difference between the upper limit and the lower limit of the coefficients; calculating the product of the activation value and the difference; adding the lower limit of the coefficients to obtain the priority coefficients.

[0126] Taking the i-th hidden state vector as an example, the activation function is expressed as follows: Among them, the priority weight is calculated. and hidden state vector product In addition to priority bias Get the preferred median value And calculate the activation value corresponding to the preferred median value to obtain the upper limit of the coefficient. Lower limit of coefficient Calculate the difference between the upper and lower limits of the coefficient. Calculate the product of the activation value and the difference, and add the lower limit of the coefficient. Obtain the priority coefficient , is represented as:

[0127]

[0128] in, for Functions, Priority and preference bias All are learnable parameters, and the upper limit of the coefficients... Lower limit of coefficient It can be set according to the actual situation. This adaptive calculation of priority coefficients ensures that each domain word appears in a semantically reasonable context, rather than being inserted arbitrarily.

[0129] Step 540: Calculate the probability value of each word at the position based on all matching scores, determine the candidate word corresponding to the position based on the probability value, and obtain the image annotation text based on all candidate words.

[0130] In one embodiment, for the i-th position, the word with the highest probability value is selected as the correct word. Therefore, the probability value of word w being the correct word at the i-th position is expressed as:

[0131]

[0132] Then, all candidate words are obtained in order to form the corresponding image annotation text.

[0133] In one embodiment, it is assumed that the latent text vector at t=0 is This text latent vector not only contains key information corresponding to the semantics of the input image, but also incorporates constraints from industry-specific terminology, thus ensuring professionalism. Next, The input is fed into the text decoder to obtain the predicted image annotation text. This process is represented as follows:

[0134]

[0135] in, Represents a text decoder, L represents the image annotation text, D represents the text length, and D represents the dimension of the hidden state vector.

[0136] This application's embodiments introduce an industrial terminology constraint mechanism from a domain data set during the decoding stage. Priority coefficients are adjusted during probability distribution calculation, ensuring the text decoder prioritizes candidate words containing technical terms such as process, equipment, and safety. This not only improves the professionalism and consistency of the image annotation text but also avoids the terminology omissions and inaccuracies often found in general models in industrial applications. The final image annotation text... It has the following characteristics: the semantics are highly consistent with the input image to be labeled, and it contains accurate process, equipment and safety terms. The text is fluent and clearly structured, and can be directly used in industrial application scenarios such as production process monitoring, equipment maintenance records, risk assessment and safety review.

[0137] In one embodiment, during the inference phase, instead of relying on any real text input, an initial noise vector is obtained directly from random Gaussian noise. A trained diffusion model is then used to progressively denoise and reconstruct the latent text vector under the constraints of visual features and terminology in the industrial image, until the image-annotated text is generated. (Refer to...) Figure 6 , Figure 6 This is a schematic diagram of the reasoning process of the automatic industrial image annotation method provided in this application embodiment. First, the image to be annotated shown in the figure is input into a visual encoder for feature encoding to obtain image features. Then, a domain terminology set and an initial noise vector are input into a diffusion model. The domain terminology set is used as a constraint to perform iterative feature processing at multiple time steps to obtain a text latent vector. Next, the text latent vector is input into a text decoder for decoding, and the image annotation text corresponding to the image to be annotated is obtained based on the domain terminology set.

[0138] Next, the training process of the diffusion model in the embodiments of this application will be described.

[0139] In one embodiment, reference is made to Figure 7 , Figure 7 This is a flowchart of the training process of the diffusion model provided in this application embodiment, which specifically includes the following steps:

[0140] Step 710: Obtain the image sample features and initial latent vector corresponding to the sample image. Add Gaussian noise to the initial latent vector according to the time step to obtain the noise data corresponding to each time step. The sample image contains the corresponding sample annotation text.

[0141] In one embodiment, for a sample image, it is input into a trained visual encoder for feature extraction to obtain the corresponding image sample features. Then obtain the initial latent vector, referring to... Figure 8 , Figure 8 This is a flowchart of obtaining the initial latent vector corresponding to a sample image provided in an embodiment of this application, specifically including the following steps:

[0142] Step 810: Input the sample labeled text into the text encoder for feature encoding to obtain the sample feature vector, and scale the sample feature vector to obtain the scale feature vector.

[0143] In one embodiment, the text latent space is used as a bridge between the image space and the text space to achieve efficient semantic conversion from image to text. Assume the sample labeled text is... ,in, For text length, For the first One word, This represents the sample feature vector obtained by the text encoder. Represented as:

[0144]

[0145] in, Let the dimension of the latent vector be... Indicates the first The latent vectors are semantic vectors at each position, so the sample feature vectors retain the necessary syntactic structure and local semantic information. The latent vectors here can be regarded as the vectors corresponding to each word in the sample labeled text.

[0146] Next, to ensure that the text latent space has an appropriate semantic density, thus preserving both fine-grained differences and global semantic consistency in cross-modal mapping, we can normalize the sample feature vector position by position, and then multiply the normalization result by a learnable scaling coefficient to perform scaling adjustment, obtaining the scaling feature vector, expressed as:

[0147]

[0148] in, To achieve position-by-position normalization, These are learnable scaling coefficients. For the normalized result, This is the scale eigenvector.

[0149] Step 820: Project the image sample features to the same dimension as the latent vector of the sample feature vector to obtain the image dimensional features. Through at least one repeated expansion, obtain the visual condition matrix with the same text length as the sample labeled text. Multiply the visual condition matrix and the gating coefficients, and add the scale feature vector to obtain the initial latent vector.

[0150] In one embodiment, image sample features Projecting onto the same dimension as the sample feature vector yields the image dimensional features. , is represented as:

[0151]

[0152] in, Represents the dimension transformation matrix. This represents the dimension corresponding to the image sample features.

[0153] Since the sample labeled text is a matrix containing multiple latent vectors, the image dimensional features are... By repeatedly expanding the text according to its length, a visual condition matrix with the same length as the sample labeled text is obtained. Then, the visual condition matrix and the gating coefficients are multiplied together, and the scale eigenvectors are added to obtain the initial latent vector. , is represented as:

[0154]

[0155] in, The learnable gating coefficients are used, and the initial latent vectors are enhanced text latent vectors aligned with the visual condition matrix. This mechanism ensures high compatibility in dimensionality, distribution, and semantics between the mapping paths of sample images, text latent space, and sample labeled text. This allows the diffusion model to be modeled in a continuous space with a reasonable structure and appropriate semantic density, thereby improving the accuracy, stability, and efficiency of generating text descriptions from industrial images.

[0156] In one embodiment, the diffusion model, through a progressive process of forward "noise addition" and reverse "denoising," is able to capture fine-grained semantic changes while maintaining global semantic consistency, thereby generating high-quality industrial image descriptions.

[0157] The first step is the forward noise addition process, starting from the initial latent vector. Initially, Gaussian noise is gradually added at each time step to obtain the noise data corresponding to each time step. The resulting sequence is represented as follows:

[0158]

[0159] in, Used to control the injection rate of noise. For the corresponding time step, is the unit covariance matrix. When Approaching the maximum total time step At that time, the corresponding noise data The sequence approximates a standard Gaussian distribution, completely losing its original semantic information. This process transforms the generation task into the inverse process of gradually recovering ordered semantics. Understandably, for each time step, the corresponding noisy data can be directly obtained from the sequence.

[0160] Step 720: Starting from the total time step, input the noise data, the current time step, and image sample features into the diffusion model to calculate the attention and obtain the iterative latent vector. Calculate the iterative mean and iterative covariance based on the iterative latent vector. Calculate the noise data corresponding to the next time step based on the iterative mean and iterative covariance. Repeat this process multiple times, using the last iterative latent vector as the predicted latent vector. Then, generate text from the predicted latent vector based on the domain terminology set to obtain the predicted text.

[0161] In one embodiment, during the reverse denoising process, starting from time step t=T, the process iterates gradually to t=0. During each iteration, noise data, the current time step, and image sample features are input into a diffusion model for attention calculation to obtain an iterative latent vector. For example, in the inference process, the corresponding query vector is obtained using the noise data, and the corresponding key and value vectors are obtained using the image sample features. Based on cross-attention calculation, the iterative latent vector corresponding to the current time step is obtained. The iterative mean and iterative covariance are calculated based on the iterative latent vector, and the noise data corresponding to the next time step is calculated based on the iterative mean and iterative covariance. This process is repeated multiple times, and the last iterative latent vector is used as the predicted latent vector. This process can be represented as:

[0162]

[0163] in, Represents the predicted latent vector. This represents the denoising process of the diffusion model. Understandably, during training, there's no need to use a domain-specific terminology set for guidance, which improves training efficiency.

[0164] Then, similar to the reasoning process, text is generated from the predicted latent vectors based on the domain terminology set to obtain the predicted text.

[0165] Step 730: Calculate the first loss value based on the predicted latent vector and the initial latent vector, calculate the second loss value based on the sample labeled text and the predicted text, calculate the total loss value based on the first loss value and the second loss value, and adjust the model parameters corresponding to the diffusion model according to the total loss value until the training is completed, and obtain the trained diffusion model.

[0166] In one embodiment, in order to simultaneously optimize the reconstruction accuracy of latent vectors and the quality of the final generated text, the total loss value includes a first loss value indicating the accuracy of inverse denoising and a second loss value indicating the accuracy of text generation.

[0167] Specifically, the first loss value calculated based on the predicted latent vector and the initial latent vector is expressed as:

[0168]

[0169] in, This indicates the mean square error.

[0170] The second loss value, obtained by calculating the cross-entropy based on the labeled and predicted texts, is expressed as:

[0171]

[0172] in, This represents the latent vector corresponding to the word at position i in the prediction latent vector. This represents the candidate word predicted at the i-th position of the predicted text. The probability of being the correct word.

[0173] Therefore, the total loss value Represented as:

[0174]

[0175] in, This represents a tradeoff coefficient used to balance semantic restoration and text generation quality.

[0176] In one embodiment, during the specific training process, the total number of time steps is fixed at 1. Noise variance parameter from Linear interpolation up to 0.02 was used, with training employing the AdamW optimizer, a batch size of 48, and an initial learning rate of [missing information]. The warmup ratio is 0.2, the weight decay is 0.01, and gradient clipping is performed, with a maximum norm of 1.0. The total loss value can be calculated at each time step iteration, and the model parameters of the diffusion model can be adjusted based on the total loss value until training is complete, resulting in a trained diffusion model.

[0177] In one embodiment, reference is made to Figure 9 , Figure 9 This is a schematic diagram illustrating the training process of the diffusion model provided in this application embodiment. Specifically, sample images and corresponding sample labeled text are acquired. The sample labeled text is then input into a text encoder for feature encoding to obtain sample feature vectors, and image sample features corresponding to the sample images are obtained based on a visual encoder. The visual encoder incorporates a contrastive learning process during training. After obtaining the sample feature vectors, scale adjustment is performed to obtain scale feature vectors. The image sample features are then projected to the same dimension as the latent vector of the sample feature vectors to obtain image dimensional features. Through at least one repeated expansion, a visual condition matrix with the same text length as the sample labeled text is obtained. The visual condition matrix is ​​multiplied by the gating coefficients and then added to the scale feature vector to obtain the initial latent vector.

[0178] Next, the noise data (generating query vectors), the current time step, and image sample features (generating key vectors and value vectors) are input into the diffusion model to calculate attention and obtain iterative latent vectors. The iterative mean and iterative covariance are calculated based on the iterative latent vectors. The noise data corresponding to the next time step is calculated based on the iterative mean and iterative covariance. After multiple iterations, the last iterative latent vector is used as the predicted latent vector. The predicted latent vector is then used to generate text based on the domain terminology set to obtain the predicted text.

[0179] Understandably, the diffusion module does not participate in computation during the forward noise addition stage. However, in the reverse denoising stage, the diffusion model receives noisy data, image sample features, and time steps. Through multi-layer self-attention and cross-attention mechanisms, it gradually recovers the noise-free predictive latent vector, achieving a semantic mapping from image visual features to text description. This significantly reduces the generation difficulty and improves the smoothness of semantic transitions and global consistency. Finally, the predicted text is obtained using a text decoder. This mechanism ensures that the diffusion module has both the ability to model global semantic dependencies and the ability to dynamically focus on key regions and details in industrial images, thereby achieving high-precision cross-modal generative modeling in the semantic latent space.

[0180] As seen above, by performing the diffusion process in the semantic latent space, the diffusion model avoids the high complexity problem of direct generation in the discrete word space and achieves a smoother semantic transition in the continuous space. It not only maintains cross-modal semantic consistency from image to text but also leverages the global modeling capabilities and noise robustness of the diffusion model to achieve high-precision semantic parsing and specialized description of complex industrial images, improving the accuracy and stability of automatic industrial image annotation.

[0181] Therefore, the automatic industrial image annotation method proposed in this application employs a domain-adaptive visual encoder to extract features from industrial images. It utilizes a visual encoder optimized through contrastive learning to extract corresponding image features and aligns them cross-modally with the text latent space. During the text generation stage, a conditional diffusion process involving forward noise addition and reverse denoising is performed in the continuous latent space obtained by fusing image semantic information and text latent representation. Visual conditions are introduced through cross-modal cross-attention, achieving a semantically coherent generative modeling process and effectively addressing the semantic jumps and structural instability issues that easily occur during word-by-word generation. Simultaneously, a domain terminology set is introduced as an industrial terminology dictionary during text encoding and decoding to constrain word selection and provide semantic guidance. This ensures accuracy and standardization in equipment names, process flows, and operational behaviors, significantly improving the professionalism and domain adaptability of the generated text. The final image annotation text possesses semantic consistency and professionalism and can be directly used in scenarios such as production management, quality inspection, and risk monitoring. Through the above improvements, the embodiments of this application can generate semantically accurate, logically coherent, and terminologically standardized descriptive text in industrial image automatic annotation tasks, greatly reducing the workload and cost of manual annotation, and demonstrating good engineering application value in scenarios such as industrial quality inspection, production monitoring, and safety inspection.

[0182] The technical solution provided in this application involves acquiring an image to be labeled, inputting the image into a visual encoder for feature encoding to obtain image features, acquiring a set of domain terms related to the industrial image, obtaining an initial noise vector, obtaining a query vector based on the initial noise vector, obtaining a key vector and a value vector based on the image features, inputting the query vector, key vector, and value vector into a diffusion model, and performing iterative feature processing at multiple time steps using the set of domain terms as constraints to obtain a text latent vector, and inputting the text latent vector and the set of domain terms into a text decoder for decoding to obtain the image annotation text corresponding to the image to be labeled. This application embodiment utilizes a set of domain terms strongly related to the industrial image as constraints throughout the entire annotation generation process, ensuring that the annotation text always revolves around a specific industrial scenario and avoiding generalized expressions that deviate from the industrial context. Specifically, a visual encoder extracts image features from the image to be labeled, converting them into key and value vectors. Simultaneously, a query vector is generated based on an initial noise vector. These three vectors, along with the time step, are input into a diffusion model for iterative feature processing. Under the constraint of a domain terminology set, the diffusion model prioritizes descriptions containing precise equipment names and operational details, progressively optimizing the text latent vectors to continuously correct them towards conforming to industry terminology standards and accurately reflecting the actual content of the image, thus reducing labeling bias. Furthermore, the text decoder's decoding process also operates within the domain terminology set, forcibly prioritizing the use of standardized industry terminology and avoiding colloquial or non-standardized expressions. Therefore, the embodiments of this application can significantly improve the accuracy and standardization of industrial image labeling.

[0183] This application also provides an automatic industrial image annotation device that can implement the above-described automatic industrial image annotation method, see reference. Figure 10 The device includes:

[0184] Image encoding module 1010: Used to acquire the image to be labeled, input the image to be labeled into the visual encoder for feature encoding, and obtain image features.

[0185] The diffusion generation module 1020 is used to obtain a set of domain terms related to industrial images, obtain an initial noise vector, obtain a query vector based on the initial noise vector, obtain a key vector and a value vector based on image features, input the query vector, key vector, and value vector into the diffusion model, and perform iterative feature processing at multiple time steps with the set of domain terms as constraints to obtain the text latent vector.

[0186] Decoding module 1030: Used to input the text latent vector and domain term set into the text decoder for decoding, so as to obtain the image annotation text corresponding to the image to be annotated.

[0187] The specific implementation of the industrial image automatic annotation device in this embodiment is basically the same as the specific implementation of the industrial image automatic annotation method described above, and will not be repeated here.

[0188] This application also provides an electronic device, including:

[0189] At least one memory;

[0190] At least one processor;

[0191] At least one program;

[0192] The program is stored in a memory, and the processor executes the at least one program to implement the industrial image automatic annotation method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0193] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0194] The processor 1101 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0195] The memory 1102 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and called and executed by the processor 1101 using the industrial image automatic annotation method of the embodiments of this application.

[0196] Input / output interface 1103 is used to implement information input and output;

[0197] The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0198] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0199] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0200] This application embodiment also provides a storage medium that stores a computer program, which, when executed by a processor, implements the above-described automatic industrial image annotation method.

[0201] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0202] The industrial image automatic annotation method, apparatus, device, and storage medium proposed in this application involve acquiring an image to be annotated, inputting the image into a visual encoder for feature encoding to obtain image features, acquiring a set of domain terms related to the industrial image, obtaining an initial noise vector, obtaining a query vector based on the initial noise vector, obtaining a key vector and a value vector based on the image features, inputting the query vector, key vector, and value vector into a diffusion model, and performing iterative feature processing at multiple time steps using the domain terminology set as a constraint to obtain a text latent vector, and inputting the text latent vector and the domain terminology set into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated. This application embodiment utilizes a set of domain terms strongly related to the industrial image as a constraint guide throughout the annotation generation process, ensuring that the annotation text always revolves around the specific industrial scenario and avoiding generalized expressions that are detached from the industrial context. Specifically, a visual encoder extracts image features from the image to be labeled, converting them into key and value vectors. Simultaneously, a query vector is generated based on an initial noise vector. These three vectors, along with the time step, are input into a diffusion model for iterative feature processing. Under the constraint of a domain terminology set, the diffusion model prioritizes descriptions containing precise equipment names and operational details, progressively optimizing the text latent vectors to continuously correct them towards conforming to industry terminology standards and accurately reflecting the actual content of the image, thus reducing labeling bias. Furthermore, the text decoder's decoding process also operates within the domain terminology set, forcibly prioritizing the use of standardized industry terminology and avoiding colloquial or non-standardized expressions. Therefore, the embodiments of this application can significantly improve the accuracy and standardization of industrial image labeling.

[0203] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0204] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0206] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0207] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0208] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0209] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0210] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0212] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0213] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An industrial image automatic labeling method, characterized in that, The method comprises the following steps: acquiring a to-be-labeled image, inputting the to-be-labeled image into a visual encoder for feature encoding to obtain image features; acquiring a domain term set related to the industrial image, the domain term set including at least domain words and general words, acquiring an initial noise vector, obtaining a query vector according to the initial noise vector, obtaining a key vector and a value vector according to the image features, inputting the query vector, the key vector and the value vector into a diffusion model to generate a term embedding sequence corresponding to the domain term set, and generating a term key vector and a term value vector according to the term embedding sequence; acquiring the query vector corresponding to the current time step, performing cross-attention calculation on the query vector, the key vector and the value vector to obtain an image context vector, performing cross-attention calculation on the query vector, the term key vector and the term value vector to obtain a term context vector, fusing the image context vector and the term context vector to obtain an output vector, calculating a predicted mean and a predicted covariance according to the output vector, calculating the query vector corresponding to the next time step according to the predicted mean and the predicted covariance, iteratively multiple times, and obtaining a text latent vector according to the last query vector; acquiring a hidden state vector corresponding to each position based on the text latent vector; calculating a matching score between the hidden state vector and a word vector of each word, acquiring a priority weight and a priority bias, calculating the product of the priority weight and the hidden state vector, adding the priority bias to obtain a priority intermediate value, and calculating an activation value corresponding to the priority intermediate value; acquiring an upper limit of a coefficient and a lower limit of the coefficient, calculating the difference between the upper limit of the coefficient and the lower limit of the coefficient, calculating the product of the activation value and the difference, adding the lower limit of the coefficient to obtain a priority coefficient, and multiplying the priority coefficient by the corresponding matching score to update the matching score; based on all the matching scores, calculating a probability value of each word in the position, determining a candidate word corresponding to the position according to the probability value, and obtaining an image labeling text according to all the candidate words.

2. The industrial image automatic labeling method of claim 1, wherein, The training process of the diffusion model at least includes: acquiring an image sample feature and an initial latent vector corresponding to a sample image, adding Gaussian noise to the initial latent vector according to time steps to obtain noise data corresponding to each time step, and the sample image containing a corresponding sample labeling text; starting from a total time step, inputting the noise data, the current time step and the image sample feature into the diffusion model to perform attention calculation to obtain an iteration latent vector, calculating an iteration mean and an iteration covariance according to the iteration latent vector, calculating the noise data corresponding to the next time step according to the iteration mean and the iteration covariance, iteratively multiple times, taking the last iteration latent vector as a predicted latent vector, and performing text generation on the predicted latent vector based on the domain term set to obtain a predicted text. A first loss value is calculated according to the predicted latent vector and the initial latent vector, a second loss value is calculated based on the sample labeled text and the predicted text, a total loss value is calculated according to the first loss value and the second loss value, and the model parameters corresponding to the diffusion model are adjusted according to the total loss value until the training is completed, thereby obtaining the trained diffusion model.

3. The industrial image automatic labeling method according to claim 2, characterized in that, An initial latent vector corresponding to a sample image is obtained, including: The sample labeled text is input into a text encoder for feature encoding to obtain a sample feature vector, and the sample feature vector is scaled to obtain a scaled feature vector; The image sample feature is projected to the same dimension as the latent vector of the sample feature vector to obtain an image dimension feature, and through at least one repeated expansion, a visual condition matrix consistent with the text length of the sample labeled text is obtained. The visual condition matrix is multiplied by a gating coefficient and then added to the scaled feature vector to obtain the initial latent vector.

4. The industrial image automatic labeling method according to any one of claims 1 to 3, characterized in that, The training process of the visual encoder at least includes: Obtaining visual training images and corresponding text descriptions, taking each pair of matched visual training images and text descriptions as a positive sample pair, and taking other unmatched visual training images and text descriptions as a negative sample pair; Using the visual encoder to obtain a visual feature vector corresponding to the visual training image, using a text encoder to obtain a text feature vector corresponding to the text description, and calculating the similarity of each positive sample pair and negative sample pair according to the visual feature vector and the text feature vector, respectively; Based on the similarity, a contrast learning loss value is calculated, and the model parameters corresponding to the visual encoder are adjusted at least according to the contrast learning loss value to obtain the trained visual encoder.

5. The industrial image automatic labeling method according to claim 4, characterized in that, The contrast learning loss value is calculated based on the similarity, including: For each visual training image, an image positive similarity value corresponding to each text feature vector is obtained, an image similarity value is obtained according to the first similarity value, and a first loss value is obtained according to the image positive similarity value and the image similarity value; For each text description, a second similarity value corresponding to each visual feature vector is obtained, a text similarity value is obtained according to the second similarity value, and a second loss value is obtained according to the image positive similarity value and the text similarity value; The contrast learning loss value is calculated according to the first loss value and the second loss value.

6. An industrial image automatic labeling apparatus characterized by comprising: Including: An image encoding module is configured to obtain a to-be-labeled image, input the to-be-labeled image into a visual encoder for feature encoding, and obtain an image feature; The diffusion generation module is configured to obtain a domain term set related to the industrial image, the domain term set including at least domain words and general words, obtain an initial noise vector, obtain a query vector according to the initial noise vector, obtain a key vector and a value vector according to the image feature, input the query vector, the key vector and the value vector into a diffusion model, generate a term embedding sequence corresponding to the domain term set according to the diffusion model, and generate a term key vector and a term value vector according to the term embedding sequence; obtain the query vector corresponding to the current time step, perform cross-attention calculation on the query vector, the key vector and the value vector to obtain an image context vector, perform cross-attention calculation on the query vector, the term key vector and the term value vector to obtain a term context vector, fuse the image context vector and the term context vector to obtain an output vector, calculate a predicted mean and a predicted covariance according to the output vector, calculate the query vector corresponding to the next time step according to the predicted mean and the predicted covariance, perform multiple iterations, and obtain a text latent vector according to the last query vector; The decoding module is configured to obtain a hidden state vector corresponding to each position based on the text latent vector; Calculate a matching score between the hidden state vector and a word vector of each word; Obtain a priority weight and a priority bias, calculate the product of the priority weight and the hidden state vector, add the priority bias to obtain a priority intermediate value, and calculate an activation value corresponding to the priority intermediate value; Obtain an upper limit of a coefficient and a lower limit of the coefficient, calculate the difference between the upper limit of the coefficient and the lower limit of the coefficient, calculate the product of the activation value and the difference, add the lower limit of the coefficient to obtain a priority coefficient, multiply the priority coefficient by the corresponding matching score, and update the matching score; calculate a probability value of each word in the position based on all the matching scores, determine a candidate word corresponding to the position according to the probability value, and obtain an image annotation text according to all the candidate words.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the industrial image automatic annotation method of any one of claims 1 to 5 when executing the computer program.

8. A storage medium storing a computer program, characterized by The computer program is executed by the processor to implement the industrial image automatic annotation method of any one of claims 1 to 5.

Images