Industrial image automatic labeling method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an industrial image automatic labeling method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining a to-be-labeled image, inputting the to-be-labeled image into a visual encoder for feature coding to obtain image features, obtaining a domain term set related to an industrial image, obtaining a query vector according to an initial noise vector, obtaining a key vector and a value vector according to the image features, and inputting the query vector, the key vector and the value vector into a diffusion model. And taking the domain term set as a constraint condition to perform iterative feature processing of multiple time steps to obtain a text latent vector, and inputting the text latent vector and the domain term set into a text decoder for decoding to obtain an image annotation text corresponding to the to-be-annotated image. In the whole annotation generation process, a domain term set is used as constraint guidance, it is ensured that an annotation text is always expanded around a specific industrial scene, and the accuracy and normalization of industrial image annotation can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an industrial image automatic labeling method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of intelligent manufacturing, industrial Internet of Things and edge computing, the demand for visual perception and intelligent analysis in industrial sites is increasing, and a large amount of image data including device operating status, job behavior, safety hazards and abnormal events is collected and stored. How to efficiently and accurately understand the semantics and text labeling of industrial images has become a key problem to promote the development of intelligent manufacturing and industrial digital transformation.

[0003] In related technologies, diffusion probability models are used to model complex data distribution step by step through a "noise-removing" process, capture complex image-text correspondence and high-level semantic features, and label industrial images. However, these diffusion models focus on general image scenarios and lack knowledge guidance mechanisms for industrial tasks, resulting in insufficient understanding of device names, working conditions, and operation steps, and thus insufficient adaptability to specific fields such as industrial scenarios, and there is still a large gap in the professionalism, terminology standardization, and engineering usability of the generated language. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose an industrial image automatic labeling method, device, equipment and storage medium to improve the labeling accuracy and standardization of industrial images.

[0005] To achieve the above purpose, the first aspect of the embodiments of the present application proposes an industrial image automatic labeling method, comprising: obtaining a to-be-labeled image, inputting the to-be-labeled image into a visual encoder for feature encoding to obtain image features; obtaining a domain term set related to the industrial image, obtaining an initial noise vector, obtaining a query vector according to the initial noise vector, obtaining a key vector and a value vector according to the image features, inputting the query vector, the key vector and the value vector into a diffusion model, and performing iterative feature processing at multiple time steps with the domain term set as a constraint condition to obtain a text latent vector; inputting the text latent vector and the domain term set into a text decoder for decoding to obtain an image labeling text corresponding to the to-be-labeled image.

[0006] In some embodiments, the domain term set includes at least domain words and general words, and the inputting the text latent vector and the domain term set into the text decoder for decoding to obtain the image labeling text corresponding to the to-be-labeled image comprises: obtaining a hidden state vector corresponding to each position based on the text latent vector; calculating a matching score between the hidden state vector and a word vector of each of the words; obtaining a priority coefficient corresponding to the domain word according to the hidden state vector, and updating the matching score by multiplying the priority coefficient and the corresponding matching score; calculating a probability value of each of the words at the position based on all the matching scores, determining a candidate word corresponding to the position according to the probability value, and obtaining the image annotation text according to all the candidate words.

[0007] In some embodiments, the obtaining a priority coefficient corresponding to the domain word according to the hidden state vector comprises: obtaining a priority weight and a priority bias, calculating a product of the priority weight and the hidden state vector, adding the priority bias to obtain a priority intermediate value, and calculating an activation value corresponding to the priority intermediate value; obtaining an upper limit of the coefficient and a lower limit of the coefficient, calculating a difference value between the upper limit of the coefficient and the lower limit of the coefficient, calculating a product of the activation value and the difference value, and adding the lower limit of the coefficient to obtain the priority coefficient.

[0008] In some embodiments, the inputting the query vector, the key vector and the value vector into a diffusion model, and performing iterative feature processing at multiple time steps with the set of domain terms as constraint conditions to obtain a text latent vector comprises: generating a term embedding sequence corresponding to the set of domain terms, and generating a term key vector and a term value vector according to the term embedding sequence; obtaining the query vector corresponding to the current time step, performing cross-attention calculation on the query vector, the key vector and the value vector to obtain an image context vector, performing cross-attention calculation on the query vector, the term key vector and the term value vector to obtain a term context vector, fusing the image context vector and the term context vector to obtain an output vector, calculating a predicted mean and a predicted covariance according to the output vector, calculating the query vector corresponding to the next time step according to the predicted mean and the predicted covariance, performing multiple iterations, and obtaining the text latent vector according to the last query vector.

[0009] In some embodiments, the training process of the diffusion model at least comprises: obtaining an image sample feature and an initial latent vector corresponding to a sample image, adding Gaussian noise to the initial latent vector according to time steps to obtain noise data corresponding to each time step, and the sample image contains a corresponding sample annotation text; From the total time step, input the noise data, the current time step and the image sample feature into the diffusion model for attention calculation to obtain an iteration latent vector, calculate an iteration mean and an iteration covariance according to the iteration latent vector, calculate the noise data corresponding to the next time step according to the iteration mean and the iteration covariance, iterate multiple times, take the last iteration latent vector as a predicted latent vector, and perform text generation on the predicted latent vector based on the domain term set to obtain a predicted text; Calculate a first loss value according to the predicted latent vector and the initial latent vector, calculate a second loss value based on the sample labeled text and the predicted text, calculate a total loss value according to the first loss value and the second loss value, and adjust the model parameters corresponding to the diffusion model according to the total loss value until the training is completed to obtain the trained diffusion model.

[0010] In some embodiments, an initial latent vector corresponding to a sample image is obtained, comprising: Input the sample labeled text into a text encoder for feature encoding to obtain a sample feature vector, and perform scale adjustment on the sample feature vector to obtain a scale feature vector; Project the image sample feature to the same dimension as the latent vector of the sample feature vector to obtain an image dimension feature, obtain a visual condition matrix consistent with the text length of the sample labeled text through at least one repeated expansion, multiply the visual condition matrix and a gating coefficient, and then add the scale feature vector to obtain the initial latent vector.

[0011] In some embodiments, the training process of the visual encoder at least includes: Obtain a visual training image and a corresponding text description, take each pair of matched visual training image and text description as a positive sample pair, and take other unmatched visual training image and text description as a negative sample pair; Obtain a visual feature vector corresponding to the visual training image by using the visual encoder, obtain a text feature vector corresponding to the text description by using a text encoder, and calculate the similarity of each positive sample pair and negative sample pair according to the visual feature vector and the text feature vector, respectively; Calculate a contrast learning loss value based on the similarity, and adjust the model parameters corresponding to the visual encoder according to at least the contrast learning loss value to obtain the trained visual encoder.

[0012] In some embodiments, the contrast learning loss value is calculated based on the similarity, comprising: For each of the visual training images, an image positive similarity value corresponding to each of the text feature vectors is obtained, an image similarity value is obtained according to the first similarity value, and a first loss value is obtained according to the image positive similarity value and the image similarity value; For each of the text descriptions, a second similarity value corresponding to each of the visual feature vectors is obtained, a text similarity value is obtained according to the second similarity value, and a second loss value is obtained according to the image positive similarity value and the text similarity value. The contrast learning loss value is calculated according to the first loss value and the second loss value.

[0013] To achieve the above object, a second aspect of the embodiment of the present application proposes an industrial image automatic labeling device, comprising: An image encoding module is configured to obtain a to-be-labeled image, input the to-be-labeled image into a visual encoder for feature encoding, and obtain an image feature. A diffusion generation module is configured to obtain a domain term set related to the industrial image, obtain an initial noise vector, obtain a query vector according to the initial noise vector, obtain a key vector and a value vector according to the image feature, input the query vector, the key vector and the value vector into a diffusion model, and perform iterative feature processing at multiple time steps with the domain term set as a constraint condition to obtain a text latent vector. A decoding module is configured to input the text latent vector and the domain term set into a text decoder for decoding to obtain an image labeling text corresponding to the to-be-labeled image.

[0014] To achieve the above object, a third aspect of the embodiment of the present application proposes an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0015] To achieve the above object, a fourth aspect of the embodiment of the present application proposes a storage medium, which is a storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0016] The industrial image automatic labeling method, device, equipment and storage medium provided by the embodiment of the present application, by obtaining a to-be-labeled image, inputting the to-be-labeled image into a visual encoder for feature encoding, obtaining image features, obtaining a domain term set related to the industrial image, obtaining an initial noise vector, obtaining a query vector according to the initial noise vector, obtaining a key vector and a value vector according to the image features, inputting the query vector, the key vector and the value vector into a diffusion model, performing iterative feature processing of multiple time steps with the domain term set as a constraint condition, obtaining a text latent vector, inputting the text latent vector and the domain term set into a text decoder for decoding, and obtaining an image labeling text corresponding to the to-be-labeled image. The embodiment of the present application uses the domain term set strongly related to the industrial image as a constraint guide throughout the whole labeling generation process, ensures that the labeling text always revolves around a specific industrial scene, and avoids generalization expressions that deviate from the industrial context. Specifically, the image features of the to-be-labeled image are extracted by the visual encoder, and are converted into a key vector and a value vector. At the same time, a query vector is generated based on an initial noise vector. The three are input into a diffusion model together with a time step for iterative feature processing. Under the constraint of the domain term set, the diffusion model will preferentially select expressions containing precise equipment names and working condition details, gradually optimize the text latent vector, and make it continuously corrected in the direction of conforming to the industrial term specification and fitting the actual image content, thereby reducing the labeling deviation. In addition, the decoding process of the text decoder is also performed within the range of the domain term set, which preferentially selects standardized industrial terms and avoids colloquial and non-standard expressions. Therefore, the embodiment of the present application can significantly improve the accuracy and standardization of industrial image labeling. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of an industrial image automatic labeling method provided by the embodiment of the present application.

[0018] Figure 2 is a flowchart of the training process of the visual encoder provided by the embodiment of the present application.

[0019] Figure 3 is a flowchart of calculating a contrast learning loss value based on similarity provided by the embodiment of the present application.

[0020] Figure 4 is a flowchart of inputting the query vector, the key vector and the value vector into the diffusion model, performing iterative feature processing of multiple time steps with the domain term set as a constraint condition, and obtaining a text latent vector provided by the embodiment of the present application.

[0021] Figure 5 is a flowchart of inputting the text latent vector and the domain term set into the text decoder for decoding, and obtaining an image labeling text corresponding to the to-be-labeled image provided by the embodiment of the present application.

[0022] Figure 6is a reasoning process schematic diagram of the industrial image automatic labeling method provided by an embodiment of the present application.

[0023] Figure 7 is a training process flowchart of the diffusion model provided by an embodiment of the present application.

[0024] Figure 8 is a flowchart of obtaining an initial latent vector corresponding to a sample image provided by an embodiment of the present application.

[0025] Figure 9 is a training process principle diagram of the diffusion model provided by an embodiment of the present application.

[0026] Figure 10 is a structural block diagram of an industrial image automatic labeling device provided by another embodiment of the present application.

[0027] Figure 11 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed descriptions will be given to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0029] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0031] First, the terms involved in the present application are analyzed: Artificial Intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; Artificial intelligence is a branch of computer science, artificial intelligence tries to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence, the research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0032] With the rapid development of intelligent manufacturing, industrial Internet of Things and edge computing, the demand for visual perception and intelligent analysis in industrial field is increasing, and a large amount of image data is collected and stored. These images contain rich production activity information, including equipment running state, operation behavior, safety hidden danger and abnormal event, which are important information sources for realizing intelligent monitoring and analysis of industrial system. How to efficiently and accurately understand the semantics of industrial images and text annotation has become a key problem to promote the development of intelligent manufacturing and industrial digital transformation. Especially in complex scenes such as metallurgy, construction and manufacturing, automatically analyzing visual data and generating descriptive text is of great significance to improve the situational awareness ability of operators and reduce the risk level of operation.

[0033] In practical applications, the initial image annotation mainly relies on manual methods. Usually, professional personnel observe the image content and fill in the corresponding text description. This method is low in efficiency, time-consuming and high in cost, and is seriously dependent on expert experience. Moreover, in the face of large-scale image data, it is difficult to realize rapid and standardized processing. In addition, different personnel have differences in the use of terms and expression style, which affects the consistency and reusability of the annotation results.

[0034] Therefore, in recent years, image automatic annotation technology has gradually combined computer vision and natural language generation methods. The common approach is to use an encoder-decoder architecture: image features are extracted using a convolutional neural network, and then a recurrent neural network is used to generate text word by word. However, these models are still sequence decoding models, and their application in industrial scenarios still faces the following challenges: 1. Insufficient complex feature extraction capability: complex backgrounds, occlusions, dust, and strong reflections are common in industrial scenes, resulting in low-quality industrial scene images, which severely affects the model's ability to extract complex features; 2. One-sided semantic representation, lack of contextual consistency: word-by-word decoding is prone to redundancy or semantic jumps, making it difficult to maintain the overall coherence and logical consistency of the generated text; 3. Inaccurate use of professional terminology: lack of support from industrial domain knowledge leads to non-standard use of terminology in device names, process flow, and operation behavior; 4. Insufficient text generation stability: the sequence decoding mechanism is sensitive to input, and lacks robustness in scenarios with high semantic diversity and structural uncertainty.

[0035] Therefore, in related technologies, diffusion probability models are introduced to model complex data distributions step by step through the "noise-removal" process, capturing complex image-text correspondence and high-level semantic features, and annotating industrial images. However, these diffusion models focus on general image scenarios and lack knowledge-guided mechanisms for industrial tasks, resulting in insufficient understanding of device names, operating conditions, and operation steps, and the semantic alignment between image details and generated text is still not rigorous enough, with description omissions or deviations. Therefore, the adaptability to specific domains such as industrial scenarios is insufficient, and the generated language still has a large gap in terms of professionalism, terminology standardization, and engineering usability.

[0036] It can be seen that both encoder-decoder architectures and diffusion models have obvious shortcomings in industrial image automatic annotation, especially the lack of industrial domain knowledge fusion capabilities, making it difficult to ensure the accuracy, professionalism, and consistency of the generated text. Therefore, the introduction of knowledge enhancement mechanisms not only helps to improve the model's semantic understanding and detail capturing ability in complex industrial scenarios, but also significantly improves the professional standardization and engineering usability of the generated text, effectively solving the core problems in current industrial image automatic annotation.

[0037] Based on this, the embodiment of the application provides an industrial image automatic labeling method, device, equipment and storage medium. In the whole labeling generation process, the field term set strongly related to the industrial image is used as a constraint guide to ensure that the labeling text always revolves around specific industrial scenes and avoids generalization expressions that deviate from the industrial context. Specifically, the image features of the image to be labeled are extracted by a visual encoder, which are converted into key vectors and value vectors. At the same time, a query vector is generated based on an initial noise vector. The three are input into a diffusion model together with a time step for iterative feature processing. Under the constraint of the field term set, the diffusion model will preferentially select expressions containing precise equipment names and working condition details, and gradually optimize the text latent vector to continuously correct it in the direction of conforming to the industrial term specification and fitting the actual image content, thereby reducing the labeling deviation. In addition, the decoding process of the text decoder is also carried out within the range of the field term set, which preferentially selects standardized industrial terms to avoid colloquial and non-standard expressions. Therefore, the embodiment of the application can significantly improve the accuracy and standardization of industrial image labeling.

[0038] The embodiment of the application provides an industrial image automatic labeling method, device, equipment and storage medium, which is specifically described as follows. First, the industrial image automatic labeling method in the embodiment of the application is described.

[0039] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0040] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning.

[0041] The industrial image automatic labeling method provided by the embodiments of the present application relates to the technical field of artificial intelligence. The industrial image automatic labeling method provided by the embodiments of the present application can be applied to a terminal, can be applied to a server side, and can also be a computer program running in the terminal or the server side. For example, the computer program can be a native program or a software module in an operating system; can be a native application (Application, APP), that is, a program that needs to be installed in an operating system to run, such as a client supporting industrial image automatic labeling, that is, a program that can run only by being downloaded into a browser environment; and can also be an applet that can be embedded into any APP. In summary, the above computer program can be any form of application program, module or plug-in. The terminal communicates with the server through a network. The industrial image automatic labeling method can be executed by the terminal or the server, or cooperatively executed by the terminal and the server.

[0042] In some embodiments, the terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart watch or the like. The server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms; or a service node in a blockchain system, the service nodes in the blockchain system form a peer-to-peer (Peer To Peer, P2P) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server can be connected through communication connection modes such as Bluetooth, universal serial bus (Universal Serial Bus, USB) or network, which are not limited in the embodiments.

[0043] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0044] An industrial image automatic labeling method in an embodiment of the application is described below.

[0045] Figure 1 is an optional flowchart of an industrial image automatic labeling method provided by an embodiment of the application, Figure 1 The method in can include, but is not limited to, steps 110 to 130. It can be understood that the embodiment does not make specific limitations on the order of steps 110 to 130 in Figure 1 Steps 110 to 130 in can be adjusted in order, or some steps can be reduced or added according to actual needs.

[0046] Step 110: Obtain an image to be labeled, input the image to be labeled into a visual encoder for feature encoding, and obtain image features.

[0047] In an embodiment, the image to be labeled is an industrial scene image, including device running state, work behavior, safety hazards, and abnormal events in a specific scene. Since the images collected in an industrial production environment usually have problems such as cluttered background, large illumination variation, dust or smoke shielding, and the like, if general visual features are directly used, the downstream generation precision will be reduced. Therefore, in the embodiment of the application, a contrast learning mechanism of industrial field knowledge is introduced into the visual encoder for field adaptive fine-tuning, so that the visual encoder can be more sensitive to the core semantics of the industrial environment when extracting features.

[0048] In an embodiment, the core goal of the visual encoder is to convert an image into a feature vector that can be aligned with text semantics, so that the visual encoder gradually learns the association rules between images and texts through a training process. Referring to Figure 2 , Figure 2 is a flowchart of a training process of the visual encoder provided by an embodiment of the application, including at least the following steps: Step 210: Obtain visual training images and corresponding text descriptions, and match each pair of visual training images and text descriptions as positive sample pairs, and other unmatched visual training images and text descriptions as negative sample pairs.

[0049] In an embodiment, a large number of visual training images and their corresponding text descriptions are obtained, which can be descriptions of the characteristics, state, function, etc. of the objects in the images, collectively serving as training data for the visual encoder. Then, the matched visual training images and text descriptions are taken as positive sample pairs, representing the correct correspondence between the visual training images and the text descriptions. Negative sample pairs are artificially selected from visual training images and unmatched text descriptions to allow the visual encoder to learn the differences between images and text.

[0050] Step 220: Obtain the visual feature vector corresponding to the visual training image using the visual encoder, and obtain the text feature vector corresponding to the text description using the text encoder, and calculate the similarity of each positive sample pair and negative sample pair based on the visual feature vector and the text feature vector.

[0051] In an embodiment, the first 6 layers of the BERT model can be selected as the text encoder for the training phase of the visual encoder, and the last 6 layers of the BERT model can be selected as the corresponding text decoder for the subsequent inference phase. In addition, the visual encoder can use the BLIP model. It can be understood that the text encoder here can be selected according to actual conditions, such as other pre-trained language models, such as RoBERTa, ELECTRA, etc., or multi-modal semantic alignment models, such as CLIP, ALIGN, etc., to realize the mapping and semantic representation between the visual space and the text space. The text decoder can be other visual backbone networks, such as ViT, Swin Transformer, ConvNeXt, etc., or models obtained by using distillation learning, multi-task learning, etc. to enhance the model's ability to extract complex features of industrial images. Here, only the text encoder and text decoder are illustrated, and they are not limited to this.

[0052] In an embodiment, the visual encoder is used to process the pixels of the visual training image layer by layer to extract the key features of the visual training image, such as edges, textures, shapes, etc., and finally obtain a visual feature vector that can represent the image. This process is represented as:

[0053] The text encoder encodes the words or characters in the text description c, captures the semantic information of the text description, and generates a corresponding text feature vector.

[0054] Then, for each positive sample pair and negative sample pair, a corresponding similarity is obtained by calculating the distance or similarity measure between the visual feature vector and the text feature vector, such as cosine similarity, Euclidean distance, etc. Here, the visual feature vector and the text feature vector can be calculated in the form of a matrix to calculate the similarity. Taking the i th visual feature vector and the j th text feature vector as an example, the similarity is represented as:

[0055] wherein, denotes the normalized feature of the i th visual feature vector, denotes the normalized feature of the j th text feature vector, denotes the similarity of the i th visual feature vector and the j th text feature vector, denotes the temperature coefficient set according to the actual situation, used to control the distribution smoothness. It can be understood that when i = j, it corresponds to a positive sample pair, and the rest are negative sample pairs.

[0056] Step 230: Calculate the contrast learning loss value based on the similarity, and at least adjust the model parameters of the visual encoder according to the contrast learning loss value to obtain the trained visual encoder.

[0057] In an embodiment, referring to Figure 3 , Figure 3 is a flowchart provided by the embodiment of the present application for calculating the contrast learning loss value based on the similarity, and specifically includes the following steps: Step 310: For each visual training image, obtain the corresponding image positive similarity value and the first similarity value corresponding to each text feature vector, obtain the image similarity value according to the first similarity value, and obtain the first loss value according to the image positive similarity value and the image similarity value.

[0058] In an embodiment, taking the i th visual training image as an example, the corresponding image positive similarity value is , the first similarity value corresponding to the j th text feature vector is , the image similarity value obtained according to the first similarity value is , and the first loss value obtained according to the image positive similarity value and the image similarity value is represented as:

[0059] Step 320: For each text description, obtain the second similarity value corresponding to each visual feature vector, obtain the text similarity value according to the second similarity value, and obtain the second loss value according to the image positive similarity value and the text similarity value.

[0060] In one embodiment, taking the j-th text description as an example, its second similarity value with the i-th visual feature vector is... The text similarity value obtained based on the second similarity value is Therefore, the second loss value is obtained based on the image positive similarity value and the text similarity value, and is expressed as:

[0061] Step 330: Calculate the contrastive learning loss value based on the first loss value and the second loss value.

[0062] In one embodiment, the contrastive learning loss value calculated based on the first loss value and the second loss value is expressed as: :

[0063] Then, the model parameters corresponding to the visual encoder are adjusted based on at least the contrastive learning loss value to obtain the trained visual encoder.

[0064] Therefore, by comparing the learning loss values, the similarity of positive samples can be maximized and the similarity of negative samples can be minimized, thereby prompting the visual encoder to approach the corresponding domain description and avoid irrelevant descriptions in the feature space. The visual encoder obtained after training can effectively capture fine-grained features and key semantic information in industrial images. It not only retains general visual recognition capabilities but also has enhanced perception capabilities for key information such as the structure and location specific to industrial scenes, thereby improving cross-modal alignment accuracy and the accuracy of subsequent image annotation text generation.

[0065] After the visual encoder is trained, the image to be labeled I can be input into the visual encoder for feature encoding to obtain image features v.

[0066] Step 120: Obtain a set of domain terms related to industrial images, obtain an initial noise vector, obtain a query vector based on the initial noise vector, obtain a key vector and a value vector based on image features, input the query vector, key vector, and value vector into the diffusion model, and perform iterative feature processing at multiple time steps with the set of domain terms as constraints to obtain the text latent vector.

[0067] In one embodiment, during the generation stage of automatic industrial image annotation, the descriptive text not only needs to be semantically fluent, but also must strictly include specialized vocabulary in fields such as process flow, equipment names, operating procedures, and safety terms; otherwise, it may result in missing information or misleading information. Therefore, this application embodiment introduces an industrial terminology constraint mechanism to construct a set of domain-specific terms, ensuring both the professionalism and accuracy of the generated results.

[0068] Specifically, the words in the domain term set include at least domain words and general words. First, a domain term set is constructed, the words in the set are domain words, and the domain words are derived from specific industrial field standard documents, process manuals, and historical annotation data, etc., and can cover common key information such as mechanical equipment, production process, and safety measures. Then, a general original word table is obtained , the words in the word table are general words. Finally, the domain term set and the original word table are merged to form an enhanced word table , and the enhanced word table is taken as the final domain term set. The domain term set obtained in this way enables the annotation process to have the ability to use industrial proprietary vocabulary at the lexical level. Through this enhancement process, the embodiments of the present application not only help to improve the semantic understanding and detail capturing ability of the related model in the complex industrial scene, but also significantly improve the professional standardization and engineering usability of the generated text, thereby effectively solving the core problem in the current industrial image automatic annotation.

[0069] Then an initial noise vector is needed, the initial noise vector is pure Gaussian noise, and is expressed as , wherein T is the total time step, , represents a unit covariance matrix, and is a normal distribution with a mean of 0 and a variance of 1, and needs to be processed by the diffusion model to obtain the final noiseless data. Next, the learned query weight W Q is obtained in the diffusion model. K , the query weight is multiplied by the initial noise vector to obtain the query vector, and the learned key weight W V and the value weight W V are obtained. K , the image feature v is multiplied by the key weight W K and the value weight W V to obtain the key vector and the value vector.

[0070] In an embodiment, referring to Figure 4 , Figure 4 is a flowchart provided by the embodiments of the present application, which inputs the query vector, the key vector, and the value vector into the diffusion model, iteratively processes the features for multiple time steps with the domain term set as the constraint condition, and obtains the text latent vector, and specifically includes the following steps: Step 410: generating a term embedding sequence corresponding to the domain term set, and generating a term key vector and a term value vector according to the term embedding sequence.

[0071] In an embodiment, in order to use the domain term set as a constraint condition in the diffusion process, a term embedding sequence corresponding to the domain term set needs to be generated , wherein the term embedding sequence is constructed by the word vectors corresponding to all the words in the domain term set. Then, the term embedding sequence respectively, to obtain the corresponding term key vector and term value vector. K and value weight W V respectively, to obtain the corresponding term key vector and term value vector.

[0072] Step 420: Obtain the query vector corresponding to the current time step, perform cross-attention calculation on the query vector, the key vector and the value vector to obtain the image context vector, perform cross-attention calculation on the query vector, the term key vector and the term value vector to obtain the term context vector, fuse the image context vector and the term context vector to obtain the output vector, calculate the predicted mean and the predicted covariance according to the output vector, calculate the query vector corresponding to the next time step according to the predicted mean and the predicted covariance, iterate multiple times, and obtain the text latent vector according to the last query vector.

[0073] In an embodiment, the diffusion model iterates according to time steps, and the total number of time steps T is a preset value. In the first iteration, t = T, at this time, the input data processed by the diffusion model is the initial noise vector, and then the query vector is obtained according to the initial noise vector. The diffusion model can be composed of 12 layers of Transformer encoder blocks, and a cross-modal cross-attention module is embedded in each layer, so that it can fully fuse the key vector K from the visual encoder and the time step t in the denoising process. It can be understood that the diffusion model can be any cross-modal generation model, including but not limited to: a Transformer-based autoregressive generation model, a variational autoencoder-based generation model, or a generative adversarial network-based generation model, to realize the automatic labeling function from industrial images to text. This embodiment is only an example and does not represent a limitation.

[0074] That is, in each iteration process, the diffusion model obtains an output vector after processing the input data through the 12-layer Transformer encoder block. Then the output vector is sent into two fully connected layers respectively, the predicted mean and the predicted covariance are calculated according to the output vector, and the input data corresponding to the next time step (that is, t = T-1) is generated using the predicted mean and the predicted covariance. After multiplying the input data and the query weight W Q , the updated query vector is obtained. Through T times of iteration process, the processing data corresponding to the time step t = 0 is obtained as the text latent vector. The specific iteration process is represented as:

[0075] wherein, represents the input data corresponding to the time step t, represents the input data corresponding to the time step t, and is an abstract function parameterized by the diffusion model, respectively representing the predicted mean and the predicted covariance for time step t, is the sampling noise.

[0076] The process of obtaining the output vector by the diffusion model is described in detail below.

[0077] First, cross-attention calculation between the query vector and the key vector and the value vector is performed, and the image context vector is obtained by cross-attention calculation between the query vector and the key vector and the value vector. Next, cross-attention calculation between the query vector and the term key vector and the term value vector is performed, and the term context vector is obtained by cross-attention calculation between the query vector and the term key vector and the term value vector. Then, the results of the two cross-attention calculations are fused, that is, the image context vector and the term context vector are fused to obtain the output vector. The fusion here can be weighted summation, element-wise multiplication or attention weighted fusion, etc., which is not limited in the embodiment.

[0078] In the above process, cross-attention calculation is performed between the query vector and the key / value vector of different sources, and the output vector is obtained by fusion, which can realize alignment of visual features and semantic terms, so that the diffusion model can capture both the visual context of the image and the semantic context of the professional term, avoiding the problem of visual semantic disconnection in the generated result.

[0079] Step 130: input the text latent vector and the domain term set into the text decoder to obtain the image annotation text corresponding to the image to be annotated.

[0080] In an embodiment, after obtaining the text latent vector, the decoding stage can be entered. Referring to Figure 5 , Figure 5 is a flowchart provided by the embodiment of the present application for inputting the text latent vector and the domain term set into the text decoder to obtain the image annotation text corresponding to the image to be annotated, and specifically includes the following steps: Step 510: obtain the hidden state vector corresponding to each position based on the text latent vector.

[0081] In an embodiment, the text latent vector contains global semantic information of the entire image annotation text, which is a matrix containing the semantic information of an entire sentence. Therefore, for each position word in the image annotation text, the vector at the corresponding position in the matrix of the text latent vector is obtained as the hidden state vector. For example, the hidden state vector corresponding to the i-th position is In addition, the process of extracting the hidden state vector needs to preserve the timing information of each position to ensure that the hidden state vector can accurately correspond to the specific words in the image annotation text, facilitating the subsequent matching process.

[0082] Step 520: Calculate the matching score between the hidden state vector and the word vector of each term.

[0083] In an embodiment, for each term in the domain term set, its corresponding word vector is obtained, and then the matching score between the hidden state vector and the word vector of each term is calculated. Taking the i-th position as an example, its hidden state vector is and the word vector of the u-th term in the domain term set is represented as .

[0084] Step 530: Obtain the priority coefficient corresponding to the domain word according to the hidden state vector, multiply the priority coefficient with the corresponding matching score, and update the matching score.

[0085] In an embodiment, in order to improve the matching priority of the domain words in the domain term set, for each domain word, such as the i-th term in the domain term set, the transpose of its word vector is , and thus the priority coefficient corresponding to the domain word is obtained according to the hidden state vector , and the priority coefficient is multiplied with the corresponding matching score to update the matching score.

[0086] Therefore, the updated matching score is represented as:

[0087] wherein the priority coefficient is greater than 1, and the weighting is performed before the softmax calculation.

[0088] In an embodiment, in order to avoid the forgetting of the term constraint in the long sequence generation of the diffusion model, a dynamic gating mechanism is introduced, the priority coefficient corresponding to the domain word is obtained according to the hidden state vector, and specifically includes: obtaining the priority weight and the priority bias, calculating the product of the priority weight and the hidden state vector, adding the priority bias to obtain the priority intermediate value, calculating the activation value corresponding to the priority intermediate value, obtaining the upper limit of the coefficient and the lower limit of the coefficient, calculating the difference value of the upper limit of the coefficient and the lower limit of the coefficient, calculating the product of the activation value and the difference value, and adding the lower limit of the coefficient to obtain the priority coefficient.

[0089] Taking the i-th hidden state vector as an example, the activation function is represented as , wherein the product of the priority weight and the hidden state vector is calculated , the priority bias is added to obtain the priority intermediate value , the activation value corresponding to the priority intermediate value is calculated, the upper limit of the coefficient and the lower limit of the coefficient​ the difference between the coefficient upper limit and the coefficient lower limit the product of the activation value and the difference, plus the coefficient lower limit the priority coefficient is obtained is expressed as:

[0090] wherein, is a function, the priority weight and the priority bias are all learnable parameters, while the coefficient upper limit and the coefficient lower limit can be set according to actual situations. In this way of adaptively calculating the priority coefficient, it can be ensured that each field word appears in a semantically reasonable context position, rather than being inserted mechanically.

[0091] Step 540: Calculate the probability value of each word in the position based on all the matching scores, determine the candidate word corresponding to the position according to the probability value, and obtain the image annotation text according to all the candidate words.

[0092] In an embodiment, for the ith position, the word corresponding to the maximum probability value is selected as the correct word, and thus the probability value of the word w as the correct word of the ith position is expressed as:

[0093] Then, all the candidate words are obtained in sequence to form the corresponding image annotation text.

[0094] In an embodiment, it is assumed that the text latent vector at iteration t=0 is The text latent vector not only contains key information corresponding to the input image semantics, but also fuses the constraints of industrial field terms, thereby ensuring professionalism. Then, is input into the text decoder to obtain the predicted image annotation text, and this process is expressed as:

[0095] wherein, denotes the text decoder, denotes the image annotation text, L denotes the text length, and D denotes the dimension of the hidden state vector.

[0096] The embodiment of the present application introduces an industry term constraint mechanism of the field data set in the decoding stage, adjusts the priority coefficient when generating the probability distribution calculation, makes the text decoder preferentially output the candidate words containing professional terms such as process, equipment and safety, not only improves the professionalism and consistency of the image annotation text, but also avoids the problems of term missing and inaccurate expression of the general model in industrial application. The final image annotation text has the following characteristics: high consistency with the semantics of the input image to be annotated, and contains accurate process, equipment and safety terms, the text is fluent and the structure is clear, which can be directly used in industrial application scenarios such as production process monitoring, equipment maintenance record, risk assessment and safety review.

[0097] In an embodiment, in the inference stage, without relying on any real text input, an initial noise vector is obtained directly from random Gaussian noise, and the trained diffusion model is used to gradually denoise and restore the text latent vector under the visual features of the industrial image and the term constraint condition, until the image annotation text is generated. Referring to Figure 6 , Figure 6 is a schematic diagram of the inference process of the industrial image automatic annotation method provided by the embodiment of the present application. First, the image to be annotated shown in the figure is input into a visual encoder to obtain image features, then the field term set and the initial noise vector are input into the diffusion model, and the field term set is used as a constraint condition for iterative feature processing at multiple time steps to obtain a text latent vector. Next, the text latent vector is input into a text decoder for decoding, and the image annotation text corresponding to the image to be annotated is obtained based on the field term set.

[0098] Next, the training process of the diffusion model in the embodiment of the present application is described.

[0099] In an embodiment, referring to Figure 7 , Figure 7 is a training process flowchart of the diffusion model provided by the embodiment of the present application, specifically including the following steps: Step 710: Obtain image sample features and initial latent vectors corresponding to sample images, add Gaussian noise in the initial latent vectors according to time steps to obtain noise data corresponding to each time step, and the sample images contain corresponding sample annotation texts.

[0100] In an embodiment, for a sample image, the image is input into a trained visual encoder for feature extraction to obtain corresponding image sample features . Then, an initial latent vector is obtained, referring to Figure 8 , Figure 8 is a flowchart provided by the embodiment of the present application for obtaining initial latent vectors corresponding to sample images, specifically including the following steps: Step 810: Input the sample labeled text into the text encoder for feature encoding to obtain the sample feature vector, and scale the sample feature vector to obtain the scale feature vector.

[0101] In one embodiment, the text latent space is used as a bridge between the image space and the text space to achieve efficient semantic conversion from image to text. Assume the sample labeled text is... ,in, For text length, For the first One word, This represents the sample feature vector obtained by the text encoder. Represented as:

[0102] in, Let the dimension of the latent vector be... Indicates the first The latent vectors are semantic vectors at each position, so the sample feature vectors retain the necessary syntactic structure and local semantic information. The latent vectors here can be regarded as the vectors corresponding to each word in the sample labeled text.

[0103] Next, to ensure that the text latent space has an appropriate semantic density, thus preserving both fine-grained differences and global semantic consistency in cross-modal mapping, we can normalize the sample feature vector position by position, and then multiply the normalization result by a learnable scaling coefficient to perform scaling adjustment, obtaining the scaling feature vector, expressed as:

[0104] in, To achieve position-by-position normalization, These are learnable scaling coefficients. For the normalized result, This is the scale eigenvector.

[0105] Step 820: Project the image sample features to the same dimension as the latent vector of the sample feature vector to obtain the image dimensional features. Through at least one repeated expansion, obtain the visual condition matrix with the same text length as the sample labeled text. Multiply the visual condition matrix and the gating coefficients, and add the scale feature vector to obtain the initial latent vector.

[0106] In one embodiment, image sample features Projecting onto the same dimension as the sample feature vector yields the image dimensional features. , represented as:

[0107] in, denotes a dimension conversion matrix, is a dimension corresponding to the image sample feature.

[0108] Since the sample annotation text is a matrix containing multiple latent vectors, the image dimension feature is repeated and expanded according to the text length to obtain a visual condition matrix consistent with the text length of the sample annotation text. Then the visual condition matrix is multiplied by the gating coefficient, and the scale feature vector is added to obtain the initial latent vector , denoted as:

[0109] wherein, is a learnable gating coefficient, and the initial latent vector is an enhanced text latent vector aligned with the visual condition matrix. Through this mechanism, the mapping path between the sample image, the text latent space and the sample annotation text can be highly compatible in terms of dimension, distribution and semantics, so that the diffusion model can be modeled in a continuous space with reasonable structure and appropriate semantic density, thereby improving the generation accuracy, stability and efficiency from industrial images to text descriptions.

[0110] In an embodiment, the diffusion model can capture fine-grained semantic changes while maintaining global semantic consistency through a gradual process of forward "noise addition" and backward "noise removal", thereby generating high-quality industrial image descriptions.

[0111] First, the forward noise addition process starts with the initial latent vector , and gradually adds Gaussian noise at each time step to obtain noise data corresponding to each time step, which forms a sequence denoted as:

[0112] wherein, is used to control the injection rate of noise, is the corresponding time step, is a unit covariance matrix. When approaches the maximum total time step , the corresponding noise data approaches the standard Gaussian distribution and completely loses the original semantic information. This process converts the generation task into an inverse process of gradually restoring ordered semantics. It can be understood that for each time step, the corresponding noise data can be directly obtained from the sequence.

[0113] Step 720: from the total time step, input the noise data, the current time step and the image sample feature into the diffusion model to perform attention calculation to obtain an iteration latent vector, calculate an iteration mean and an iteration covariance according to the iteration latent vector, calculate noise data corresponding to the next time step according to the iteration mean and the iteration covariance, iterate multiple times, take the last iteration latent vector as a predicted latent vector, and perform text generation on the predicted latent vector based on the domain term set to obtain a predicted text.

[0114] In an embodiment, in the reverse denoising process, starting from time step t=T and gradually iterating to t=0, in the iteration process at each time step, input the noise data, the current time step and the image sample feature into the diffusion model to perform attention calculation to obtain an iteration latent vector. As in the inference process, here the noise data is used to obtain a corresponding query vector, the image sample feature is used to obtain a corresponding key vector and a value vector, and based on cross-attention calculation, an iteration latent vector corresponding to the current time step is obtained. According to the iteration latent vector, an iteration mean and an iteration covariance are calculated, and noise data corresponding to the next time step is calculated according to the iteration mean and the iteration covariance. Multiple iterations are performed, and the last iteration latent vector is taken as a predicted latent vector. This process can be represented as:

[0115] wherein, represents the predicted latent vector, represents the denoising process of the diffusion model. It can be understood that in the training process, the domain term set does not need to be guided, which can improve the training efficiency.

[0116] Then, as in the inference process, text generation is performed on the predicted latent vector based on the domain term set to obtain a predicted text.

[0117] Step 730: calculate a first loss value according to the predicted latent vector and the initial latent vector, calculate a second loss value based on the sample labeled text and the predicted text, calculate a total loss value according to the first loss value and the second loss value, and adjust the model parameters corresponding to the diffusion model according to the total loss value until the training is completed, to obtain a trained diffusion model.

[0118] In an embodiment, in order to optimize the reconstruction accuracy of the latent vector and the quality of the finally generated text at the same time, the total loss value includes a first loss value indicating the reverse denoising accuracy and a second loss value indicating the text generation accuracy.

[0119] Specifically, the first loss value calculated according to the predicted latent vector and the initial latent vector is represented as:

[0120] wherein, represents the calculation of the mean square error.

[0121] And the second loss value calculated based on the sample annotation text and the predicted text is represented as:

[0122] wherein, represents the latent vector of the word corresponding to the i-th position in the predicted latent vector, represents the candidate word predicted by the i-th position of the predicted text is the probability of the correct word.

[0123] Therefore, the total loss value is represented as:

[0124] wherein, represents a trade-off coefficient for balancing semantic restoration and text generation quality In an embodiment, in a specific training process, the total time step is fixed as , the noise variance parameter is linearly interpolated from 0.02, the training uses the AdamW optimizer, the batch size is 48, the initial learning rate is 0.0001, the warmup ratio is 0.2, the weight decay is 0.01, and gradient clipping is performed with a maximum norm of 1.0. The corresponding total loss value can be calculated in each iteration process of each time step, and then the model parameters of the diffusion model are adjusted according to the total loss value until the training is completed, and the trained diffusion model is obtained.

[0125] In an embodiment, referring to Figure 9 , Figure 9 is the training process schematic diagram of the diffusion model provided by the embodiments of the present application. Specifically, a sample image and a corresponding sample annotation text are obtained, and then the sample annotation text is input into a text encoder for feature encoding to obtain a sample feature vector, and an image sample feature corresponding to the sample image is obtained based on a visual encoder. The visual encoder introduces a contrast learning process in the training process. After obtaining the sample feature vector, a scale adjustment is performed to obtain a scale feature vector, and the image sample feature is projected to the same dimension as the latent vector of the sample feature vector to obtain an image dimension feature. Through at least one repeated expansion, a visual condition matrix consistent with the text length of the sample annotation text is obtained. After multiplying the visual condition matrix and a gating coefficient, the scale feature vector is added to obtain an initial latent vector.

[0126] Next, the noise data (generate the query vector), the current time step and the image sample feature (generate the key vector and the value vector) are input into the diffusion model for attention calculation to obtain the iteration latent vector, the iteration mean and the iteration covariance are calculated according to the iteration latent vector, the noise data corresponding to the next time step is calculated according to the iteration mean and the iteration covariance, the last iteration latent vector is taken as the prediction latent vector after multiple iterations, and the prediction text is obtained by text generation based on the domain term set.

[0127] It can be understood that, in the forward noise adding stage, the diffusion module does not participate in the operation, and in the backward noise removing stage, the diffusion model receives the noise data, the image sample feature and the time step, gradually recovers the noise-free prediction latent vector through the multi-layer self-attention and cross-attention mechanism, realizes the semantic mapping from the image visual feature to the text description, significantly reduces the generation difficulty, and improves the smoothness and global consistency of the semantic transition. Finally, the prediction text is obtained by using the text decoder. This mechanism ensures that the diffusion module not only has the ability to model global semantic dependence, but also can dynamically focus on key areas and details in industrial images, so as to realize high-precision cross-modal generation modeling in the semantic latent space.

[0128] As can be seen from the above, the diffusion process is performed in the semantic latent space, and the diffusion model avoids the high complexity problem of direct generation in the discrete word space, and realizes more smooth semantic transition in the continuous space. Not only can the cross-modal semantic consistency from image to text be maintained, but also the global modeling ability and noise robustness of the diffusion model can be utilized to realize high-precision semantic analysis and professional description of complex industrial images, and improve the accuracy and stability of industrial image automatic labeling.

[0129] Therefore, the industrial image automatic labeling method provided in the embodiments of the present application adopts a domain adaptive visual encoder to perform feature extraction on the industrial image, uses a visual encoder optimized through contrast learning to extract corresponding image features, and performs cross-modal alignment with a text latent space. In the text generation stage, in a continuous latent space obtained by fusing image semantic information and text latent representation, a conditional diffusion process of forward noise addition and reverse noise removal is performed, a visual condition is introduced through cross-modal cross attention, a semantic coherent generation modeling process is realized, and the problems of semantic jump and unstable structure that are prone to occur in the word-by-word generation process are effectively solved. Meanwhile, a domain term set is introduced as an industrial term dictionary in the text encoding and decoding process, vocabulary selection constraint and semantic guidance are performed, so as to ensure the accuracy and standardization in aspects of device name, process flow and operation behavior, and significantly improve the professionalism and domain adaptability of the generated text. The generated final image labeling text has semantic consistency and professionalism, and can be directly used in production management, quality detection and risk monitoring scenes. Through the above improvements, the embodiments of the present application can generate description texts that are accurate in semantics, coherent in logic and standard in terms in the industrial image automatic labeling task, greatly reducing the workload and cost of manual labeling, and showing good engineering application value in industrial quality inspection, production monitoring and safety inspection scenes.

[0130] The technical scheme provided in the embodiments of the present application acquires a to-be-labeled image, inputs the to-be-labeled image into a visual encoder for feature encoding to obtain image features, acquires a domain term set related to the industrial image, acquires an initial noise vector, obtains a query vector according to the initial noise vector, obtains a key vector and a value vector according to the image features, inputs the query vector, the key vector and the value vector into a diffusion model, performs iterative feature processing of multiple time steps with the domain term set as a constraint condition, obtains a text latent vector, inputs the text latent vector and the domain term set into a text decoder for decoding, and obtains an image labeling text corresponding to the to-be-labeled image. The embodiments of the present application use the domain term set strongly related to the industrial image as a constraint guide throughout the whole labeling generation process, ensure that the labeling text always revolves around a specific industrial scene, and avoid generalization expressions that deviate from the industrial context. Specifically, the image features of the to-be-labeled image are extracted through the visual encoder, and are converted into the key vector and the value vector. Meanwhile, the query vector is generated based on the initial noise vector. The three are input into the diffusion model together with the time step for iterative feature processing. Under the constraint of the domain term set, the diffusion model will preferentially select expressions containing accurate device names and working condition details, gradually optimize the text latent vector, make it continuously corrected in the direction of conforming to the industrial term standard and fitting the actual image content, and reduce the labeling deviation. In addition, the decoding process of the text decoder is also performed within the range of the domain term set, and the standardized industrial terms are preferentially selected to avoid colloquial and non-standard expressions. Therefore, the embodiments of the present application can significantly improve the accuracy and standardization of industrial image labeling.

[0131] The embodiment of the present application further provides an industrial image automatic labeling device, which can implement the industrial image automatic labeling method. Figure 10 The device comprises: The image encoding module 1010 is configured to acquire a to-be-labeled image, input the to-be-labeled image into a visual encoder for feature encoding, and obtain image features.

[0132] The diffusion generation module 1020 is configured to acquire a domain term set related to the industrial image, acquire an initial noise vector, obtain a query vector according to the initial noise vector, obtain a key vector and a value vector according to the image features, input the query vector, the key vector and the value vector into a diffusion model, perform iterative feature processing for multiple time steps with the domain term set as a constraint condition, and obtain a text latent vector.

[0133] The decoding module 1030 is configured to input the text latent vector and the domain term set into a text decoder for decoding, and obtain an image labeling text corresponding to the to-be-labeled image.

[0134] The specific implementation of the industrial image automatic labeling device of the embodiment is basically the same as the specific implementation of the industrial image automatic labeling method, and will not be repeated here.

[0135] The embodiment of the present application further provides an electronic device, comprising: at least one memory; at least one processor; at least one program; The program is stored in the memory, and the processor executes the at least one program to implement the industrial image automatic labeling method provided by the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), a vehicle-mounted computer, etc.

[0136] Please refer to Figure 11 , Figure 11 The hardware structure of the electronic device of another embodiment is illustrated, which comprises: The processor 1101 can be implemented in the form of a general central processing unit (Central Processing Unit, CPU), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application. The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1102 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1102 and are called and executed by the processor 1101 to implement the industrial image automatic labeling method of the embodiments of the present application. The input / output interface 1103 is configured to realize information input and output. The communication interface 1104 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 1105 is configured to transmit information between various components (for example, the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104) of the device. The processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104 are connected to each other through the bus 1105 to realize the communication connection between them in the device.

[0137] The embodiments of the present application also provide a storage medium. The storage medium is a storage medium, and the storage medium stores a computer program. The computer program is executed by the processor to implement the above-mentioned industrial image automatic labeling method.

[0138] The memory is a non-transitory storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0139] The industrial image automatic labeling method, device, equipment and storage medium provided by the embodiments of the present application are as follows: an image to be labeled is obtained, the image to be labeled is input into a visual encoder for feature encoding to obtain image features, a domain term set related to the industrial image is obtained, an initial noise vector is obtained, a query vector is obtained according to the initial noise vector, a key vector and a value vector are obtained according to the image features, the query vector, the key vector and the value vector are input into a diffusion model, and iterative feature processing of multiple time steps is performed with the domain term set as a constraint condition to obtain a text latent vector. The text latent vector and the domain term set are input into a text decoder for decoding to obtain an image labeling text corresponding to the image to be labeled. In the whole labeling generation process, the embodiments of the present application utilize the domain term set strongly related to the industrial image as a constraint guide to ensure that the labeling text always revolves around a specific industrial scene and avoids generalization expressions that deviate from the industrial context. Specifically, the image features of the image to be labeled are extracted by the visual encoder, and are converted into the key vector and the value vector. Meanwhile, the query vector is generated based on the initial noise vector. The three are input into the diffusion model together with the time step for iterative feature processing. Under the constraint of the domain term set, the diffusion model will preferentially select expressions containing precise equipment names and working condition details, gradually optimize the text latent vector, and make it continuously corrected in the direction of conforming to the industrial term specification and fitting the actual image content, thereby reducing the labeling deviation. In addition, the decoding process of the text decoder is also performed within the range of the domain term set, which preferentially selects standardized industrial terms to avoid colloquial and non-standardized expressions. Therefore, the embodiments of the present application can significantly improve the accuracy and standardization of industrial image labeling.

[0140] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0141] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps or different steps.

[0142] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0143] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented by software, firmware, hardware or a combination thereof.

[0144] The terms "first", "second", "third", "fourth" etc. (if any) in the description of the application and in the claims that follow, if they occur, are used as identifiers for distinguishing between similar objects of the application and are not necessarily used to describe a particular sequential or chronological order. It is to be understood that the use of these terms is interchangeable under appropriate circumstances and embodiments of the application described herein are capable of operating in other sequences than those explicitly described or illustrated herein. Further, the terms "comprise" and "include", and variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises a list of steps or units are not necessarily limited to those steps or units that are expressly listed, but can include additional steps or units not expressly listed or inherent to such process, method, product or apparatus.

[0145] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0146] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not implemented. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0147] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0148] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0149] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0150] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An automatic annotation method for industrial images, characterized in that, include: Obtain the image to be labeled, and input the image to be labeled into a visual encoder for feature encoding to obtain image features; Obtain a set of domain terms related to the industrial image, obtain an initial noise vector, obtain a query vector based on the initial noise vector, obtain a key vector and a value vector based on the image features, input the query vector, the key vector, and the value vector into a diffusion model, and perform iterative feature processing at multiple time steps with the set of domain terms as constraints to obtain a text latent vector; The text latent vector and the domain terminology set are input into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated.

2. The automatic industrial image annotation method according to claim 1, characterized in that, The domain terminology set includes at least domain-specific terms and general terms. The step of inputting the text latent vector and the domain terminology set into a text decoder for decoding to obtain the image annotation text corresponding to the image to be annotated includes: Based on the text latent vector, obtain the hidden state vector corresponding to each position; Calculate the matching score between the hidden state vector and the word vector of each word; The priority coefficient corresponding to the domain word is obtained based on the hidden state vector, and the priority coefficient is multiplied by the corresponding matching score to update the matching score; Calculate the probability value of each word at the position based on all the matching scores, determine the candidate word corresponding to the position based on the probability value, and obtain the image annotation text based on all the candidate words.

3. The automatic industrial image annotation method according to claim 2, characterized in that, The step of obtaining the priority coefficient corresponding to the domain word based on the hidden state vector includes: Obtain the priority weight and priority bias, calculate the product of the priority weight and the hidden state vector, add the priority bias to obtain the priority median value, and calculate the activation value corresponding to the priority median value. Obtain the upper limit and lower limit of the coefficient, calculate the difference between the upper limit and the lower limit, calculate the product of the activation value and the difference, and add the lower limit to obtain the priority coefficient.

4. The automatic industrial image annotation method according to claim 1, characterized in that, The process of inputting the query vector, the key vector, and the value vector into a diffusion model, and performing iterative feature processing at multiple time steps using the domain terminology set as constraints, to obtain text latent vectors includes: Generate a term embedding sequence corresponding to the domain term set, and generate a term key vector and a term value vector based on the term embedding sequence; Obtain the query vector corresponding to the current time step; perform cross-attention calculation on the query vector, the key vector, and the value vector to obtain an image context vector; perform cross-attention calculation on the query vector, the term key vector, and the term value vector to obtain a term context vector; fuse the image context vector and the term context vector to obtain an output vector; calculate the prediction mean and prediction covariance based on the output vector; calculate the query vector corresponding to the next time step based on the prediction mean and prediction covariance; iterate multiple times; and obtain the text latent vector based on the last query vector.

5. The automatic industrial image annotation method according to claim 1, characterized in that, The training process of the diffusion model includes at least the following: Obtain the image sample features and initial latent vector corresponding to the sample image. Add Gaussian noise to the initial latent vector according to the time step to obtain the noise data corresponding to each time step. The sample image contains the corresponding sample annotation text. Starting from the total time step, the noise data, the current time step, and the image sample features are input into the diffusion model to calculate attention and obtain an iterative latent vector. The iterative mean and iterative covariance are calculated based on the iterative latent vector. The noise data corresponding to the next time step is calculated based on the iterative mean and iterative covariance. This process is repeated multiple times. The last iterative latent vector is used as the predicted latent vector. Text is generated based on the domain terminology set to obtain the predicted text. A first loss value is calculated based on the predicted latent vector and the initial latent vector. A second loss value is calculated based on the sample labeled text and the predicted text. A total loss value is calculated based on the first loss value and the second loss value. The model parameters corresponding to the diffusion model are adjusted based on the total loss value until training ends, and the trained diffusion model is obtained.

6. The automatic industrial image annotation method according to claim 5, characterized in that, Obtain the initial latent vector corresponding to the sample image, including: The sample labeled text is input into a text encoder for feature encoding to obtain a sample feature vector, and the sample feature vector is scaled to obtain a scale feature vector; The image sample features are projected to the same dimension as the latent vector of the sample feature vector to obtain the image dimension features. Through at least one repeated expansion, a visual condition matrix with the same text length as the sample labeled text is obtained. The visual condition matrix is ​​multiplied by the gating coefficient and then added to the scale feature vector to obtain the initial latent vector.

7. The automatic industrial image annotation method according to any one of claims 1 to 6, characterized in that, The training process of the visual encoder includes at least the following: Obtain visual training images and corresponding text descriptions, and take each matching pair of visual training images and text descriptions as a positive sample pair, and take other unmatched pairs of visual training images and text descriptions as negative sample pairs; The visual feature vector corresponding to the visual training image is obtained using the visual encoder, and the text feature vector corresponding to the text description is obtained using the text encoder. The similarity between each positive sample pair and the negative sample pair is calculated based on the visual feature vector and the text feature vector, respectively. Based on the similarity, a contrastive learning loss value is calculated, and the model parameters corresponding to the visual encoder are adjusted at least according to the contrastive learning loss value to obtain the trained visual encoder.

8. The automatic industrial image annotation method according to claim 7, characterized in that, The calculation of the contrastive learning loss value based on the similarity includes: For each of the visual training images, obtain the corresponding positive image similarity value and the first similarity value corresponding to each of the text feature vectors, obtain the image similarity value based on the first similarity value, and obtain the first loss value based on the positive image similarity value and the image similarity value; For each of the text descriptions, a second similarity value corresponding to each of the visual feature vectors is obtained, a text similarity value is obtained based on the second similarity value, and a second loss value is obtained based on the image positive similarity value and the text similarity value; The contrastive learning loss value is calculated based on the first loss value and the second loss value.

9. An automatic industrial image annotation device, characterized in that, include: Image encoding module: used to acquire the image to be labeled, input the image to be labeled into the visual encoder for feature encoding, and obtain image features; The diffusion generation module is used to obtain a set of domain terms related to the industrial image, obtain an initial noise vector, obtain a query vector based on the initial noise vector, obtain a key vector and a value vector based on the image features, input the query vector, the key vector, and the value vector into the diffusion model, and perform iterative feature processing at multiple time steps with the set of domain terms as constraints to obtain the text latent vector. Decoding module: Used to input the text latent vector and the domain term set into the text decoder for decoding, so as to obtain the image annotation text corresponding to the image to be annotated.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the industrial image automatic annotation method according to any one of claims 1 to 8.

11. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the industrial image automatic annotation method according to any one of claims 1 to 8.

Images