A language model based defect localization method and system

Abstract

The present application relates to the technical field of workpiece detection, and particularly relates to a defect positioning method and system based on a language model, which comprises the following steps: S100, acquiring an image of a workpiece surface and performing preprocessing; S200, inputting the preprocessed image data into a preliminary positioning model to obtain a preliminary positioning result; S300, generating a semantic description according to image features extracted by CNN; querying historical images according to the preliminary positioning result, and generating a target semantic description according to historical image features extracted by CNN; S400, converting the semantic description into a semantic vector and converting the target semantic description into a target semantic vector; S500, optimizing the semantic vector; S600, determining the position of a workpiece defect through the optimized semantic vector to obtain a current positioning result, comparing the current positioning result with the preliminary positioning result to obtain a final position. The method can improve the efficiency and accuracy of workpiece defect detection.

Description

Technical Field

[0001] This invention relates to the field of workpiece inspection technology, and in particular to a defect localization method and system based on a language model. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing, workpiece defect detection has become increasingly important in ensuring product quality and improving production efficiency. Traditional defect detection methods mainly rely on manual inspection and rule-based image processing technology. These methods typically require operators to possess high levels of professional skills and extensive experience, but even so, it is difficult to avoid missed or false detections due to human eye fatigue or subjective factors. Furthermore, rule-based image processing technology often exhibits insufficient robustness and poor adaptability when processing complex workpiece surfaces, resulting in low detection efficiency and poor accuracy.

[0003] With the development of computer vision and machine learning technologies, deep learning-based defect detection methods have gradually attracted attention. Deep learning models, especially convolutional neural networks (CNNs), have demonstrated excellent performance in image feature extraction and classification tasks. In existing technologies, defect detection uses image data as input and defect location results as output, directly training a preliminary localization model through a CNN. However, most existing deep learning methods require large amounts of labeled data for training, which is often difficult to obtain in practical industrial applications. Furthermore, these models have high hardware resource requirements, and the training and inference processes are time-consuming, making it difficult to meet the real-time and efficiency requirements of industrial production. Therefore, a new workpiece defect localization method is urgently needed to overcome the shortcomings of existing technologies and improve the efficiency and accuracy of defect detection. Summary of the Invention

[0004] The purpose of this invention is to provide a language model-based defect localization method and system, which can improve the efficiency and accuracy of workpiece defect detection.

[0005] The basic solution provided by this invention is a defect localization method and system based on a language model, comprising the following steps:

[0006] S100, Data preprocessing steps: Acquire an image of the workpiece surface and perform preprocessing;

[0007] S200, Preliminary positioning step: Input the preprocessed image data into the preliminary positioning model to obtain the preliminary positioning result;

[0008] S300, Language Model Training Steps: Generate semantic descriptions based on image features extracted by CNN; Query historical images based on preliminary localization results, and generate target semantic descriptions based on historical image features extracted by CNN;

[0009] S400, Semantic Vector Generation Steps: Convert the semantic description into a semantic vector, and convert the target semantic description into a target semantic vector;

[0010] S500, Genetic Algorithm Optimization Steps: Optimize the semantic vector;

[0011] S600, Defect localization steps: Determine the location of the workpiece defect through the optimized semantic vector to obtain the current localization result. Compare the current localization result with the preliminary localization result. When the overlap meets the threshold, the current localization result is taken as the final position. When the overlap does not meet the threshold, the current localization result is added to the training set of the preliminary localization model for retraining, and the preliminary localization result is taken as the final position.

[0012] Furthermore, step S100 includes: acquiring a high-resolution image of the workpiece surface; performing grayscale processing on the image; removing noise using Gaussian filtering; and performing edge detection using Canny edge detection.

[0013] Furthermore, step S300 includes: extracting image features from the preprocessed image using a CNN convolutional neural network; and converting the extracted image features into a semantic description using a Transformer model.

[0014] Furthermore, step S400 includes: using the BERT model to convert the generated semantic description into a semantic vector representation, and converting the target semantic description into a target semantic vector representation.

[0015] Furthermore, step S500 includes: generating an initial semantic vector population; selecting semantic vectors with high fitness for reproduction; generating a new generation of semantic vectors through crossover and mutation operations; defining a fitness function to evaluate the accuracy of each semantic vector; and iteratively performing selection, crossover, and mutation operations for evolution.

[0016] Furthermore, the fitness function is defined as follows:

[0017]

[0018] Among them, v i Semantic vectors It is the target semantic vector.

[0019] The principle and beneficial effects of this invention are as follows:

[0020] 1. High-precision defect detection: By combining convolutional neural networks (CNN) and Transformer models, image features can be extracted efficiently and accurate semantic descriptions can be generated, thereby significantly improving the accuracy of defect detection.

[0021] 2. High degree of automation: In semantic vector optimization, genetic algorithms reduce human intervention through automatic selection, crossover and mutation operations, realize a highly efficient automatic optimization process, and greatly improve the automation level of the detection system.

[0022] Furthermore, it also includes further optimization of semantic vectors using particle swarm optimization: initializing the particle swarm: using the best individual from the genetic algorithm stage as the initial particle swarm for PSO; updating the velocity and position of the particles according to the current position and velocity update formula; calculating the fitness value of each particle, updating the individual optimal value and the global optimal value; iterating until the fitness function converges to the optimal solution; alternating between the genetic algorithm and the particle swarm optimization algorithm to optimize the semantic vectors.

[0023] The velocity and position update formulas in the particle swarm optimization algorithm are as follows:

[0024] v i (t+1)=ωv i (t)+c1r1(p i -x i (t))+c2r2(gx j (t))

[0025] x i (t+1)=t i (x)+v i (t+1)

[0026] Where ω is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers, and p i Let g be the individual optimal position of the particle, and g be the global optimal position.

[0027] The beneficial effects of this scheme are as follows: By introducing Particle Swarm Optimization (PSO) algorithm on top of the genetic algorithm, the semantic vectors are further optimized. The genetic algorithm provides global search capabilities, while PSO further improves optimization accuracy through fast local search. The combination of the two makes the optimization process more precise and improves the accuracy of defect detection. In addition, the alternating iteration of the two algorithms not only avoids getting trapped in local optima but also effectively shortens the optimization time and improves overall efficiency.

[0028] Furthermore, it also includes manual verification of the generated semantic descriptions to improve the accuracy of the semantic vectors.

[0029] The beneficial effects of this solution are: manual verification can identify and correct erroneous or inaccurate semantic descriptions generated by the language model, thereby reducing false positives and false negatives. By reviewing and correcting the semantic descriptions generated by the language model, erroneous descriptions caused by model errors or data biases can be effectively avoided. Attached Figure Description

[0030] Figure 1 This is a flowchart of a language model-based defect localization method. Detailed Implementation

[0031] The following detailed description illustrates the specific implementation method:

[0032] Example 1

[0033] In this embodiment, by providing a language model-based defect localization method and system, the efficiency and accuracy of workpiece defect detection can be improved.

[0034] like Figure 1 The method for defect localization based on a language model includes the following steps: S100, Data preprocessing step: Acquire an image of the workpiece surface and perform preprocessing; S200, Preliminary localization step: Input the preprocessed image data into the preliminary localization model to obtain the preliminary localization result; S300, Language model training step: Generate a semantic description based on the image features extracted by CNN; Query historical images based on the preliminary localization result and generate a target semantic description based on the historical image features extracted by CNN; S400, Semantic vector generation step: Convert the semantic description into a semantic vector and the target semantic description into a target semantic vector; S500, Genetic algorithm optimization step: Optimize the semantic vector; S600, Defect localization step: Determine the location of the workpiece defect through the optimized semantic vector to obtain the current localization result. Compare the current localization result with the preliminary localization result. When the overlap meets the threshold, the current localization result is taken as the final location; when the overlap does not meet the threshold, the current localization result is added to the training set of the preliminary localization model for retraining, and the preliminary localization result is taken as the final location.

[0035] Step S100: Data preprocessing step, using a high-resolution industrial camera to capture images of the workpiece surface.

[0036] In this embodiment, a camera with 5 megapixels or higher is selected to ensure good optical performance and stable exposure control. The shooting environment should be uniformly illuminated to avoid shadows and light spots. Ring light sources or line light sources can be used to provide uniform illumination and reduce surface reflection interference. Under this environment, each workpiece surface is photographed from multiple angles and positions to ensure that the acquired images have comprehensive coverage and a resolution that meets the requirements of subsequent processing, typically above 3000x2000 pixels.

[0037] The acquired color image is converted to grayscale to simplify the image data and reduce computational complexity. Grayscale conversion can be achieved using a weighted average method. Built-in or custom functions can be used in image processing software or programming environments for grayscale conversion. Gaussian filtering is then used to remove noise from the grayscale image. Gaussian filtering uses a Gaussian function to perform a weighted average, smoothing the image and removing random noise. The Canny edge detection algorithm is then used to extract edge information from the image.

[0038] Step S300: Language model training steps.

[0039] Convolutional Neural Networks (CNNs) are used to extract image features from preprocessed images. CNNs extract important features from an image layer by layer through multiple convolutional and pooling operations. Specifically, a CNN consists of multiple convolutional layers, pooling layers, and fully connected layers. Convolutional layers slide convolutional kernels across the image to extract local features, while pooling layers reduce the size of the feature maps through downsampling operations, thus preserving important information while reducing computational cost. The stacking of multiple convolutional and pooling layers allows CNNs to extract image features at different levels, including edge, texture, and shape information. These features will serve as input for subsequent semantic generation steps.

[0040] After feature extraction, the Transformer model is used to convert the extracted image features into semantic descriptions. The Transformer model is a deep learning model based on a self-attention mechanism, which excels at processing sequential data and has powerful semantic understanding capabilities.

[0041] The Transformer model consists of an encoder and a decoder. The encoder encodes the input image features into a latent representation, and the decoder decodes this latent representation into a natural language description. In the encoder, a self-attention mechanism captures global dependencies between image features, and a multi-head attention mechanism enhances the model's expressive power. In the decoder, the latent representation output by the encoder and the contextual information of the current decoding step are combined to generate a natural language description. Training the Transformer model requires a large-scale labeled dataset containing images and their corresponding semantic descriptions. By optimizing the model parameters, the generated semantic descriptions accurately reflect the information in the images. Through these steps, the language model is trained, and the generated semantic descriptions serve as the basis for subsequent semantic vector generation steps.

[0042] Step S400: Semantic Vector Generation Step. The BERT model (Bidirectional Encoder Representations from Transformers) is used to transform these semantic descriptions into semantic vectors, thus transforming the target semantic description into the target semantic vector.

[0043] The BERT model is a pre-trained language model based on Transformer. Through the structure of a bidirectional encoder, it can capture the complex relationships between words in the context, thereby generating a high-dimensional vector representation of semantics.

[0044] The generated semantic descriptions are used as input to the BERT model. The BERT model's input undergoes preprocessing, including word segmentation, adding special tags (such as [CLS] and [SEP] tags), and constructing an input tensor. The preprocessed semantic descriptions are then fed into the BERT model's encoder. The BERT model's encoder consists of multiple Transformer layers, each containing a self-attention mechanism and a feedforward neural network. The self-attention mechanism captures the dependencies between words in the semantic description, while the feedforward neural network further processes these dependencies to generate high-quality feature representations.

[0045] During the encoding process, the BERT model models the context of each word, considering both the words on its left and right sides to generate a bidirectional feature representation that includes contextual information. After multiple layers of encoding, the representation vector of each word output by the BERT model contains rich semantic information. In particular, the vector representation of the [CLS] marker at the first position in the BERT model's output can be used as a global feature vector for the entire semantic description. These vector representations will serve as semantic vectors for subsequent optimization and defect localization steps.

[0046] Step S500: Genetic Algorithm Optimization Step. After generating the semantic vectors, a genetic algorithm is used to optimize these semantic vectors.

[0047] An initial population of semantic vectors is randomly generated, with each vector representing a possible defect feature. This initial population of semantic vectors will serve as the starting point for the optimization process.

[0048] The semantic vectors with high fitness are selected for propagation based on the fitness function. The fitness function is used to evaluate the accuracy of each semantic vector, and it is defined as follows:

[0049]

[0050] Among them, v i Semantic vectors It is the target semantic vector.

[0051] The fitness function calculates the Euclidean distance between each semantic vector and the target vector, and converts the distance into a fitness value by taking its reciprocal. A higher fitness value indicates that the current semantic vector is closer to the target semantic vector. In this way, the quality of each semantic vector can be quantified, and semantic vectors with high fitness can be selected as parents for reproduction.

[0052] After selecting highly fit parent vectors, a new generation of semantic vectors is generated through crossover and mutation operations. Crossover involves swapping partial values ​​between two parent vectors to generate new offspring vectors. This can be achieved by randomly selecting the crossover point, for example, by swapping a portion of two vectors, thus producing diverse offspring. Mutation involves randomly altering certain vector values ​​to introduce new features and diversity. Mutation can be achieved by randomly selecting one or more vector elements and randomly adjusting them, thereby generating new mutated individuals.

[0053] Subsequently, a fitness function is defined to evaluate each newly generated semantic vector, and the optimal individual is selected for reproduction based on the fitness value. Through multiple generations of selection, crossover, and mutation operations, the semantic vector is gradually optimized. In each generation, individuals with high fitness are selected for crossover and mutation, and this process is iterated until the semantic vector gradually approaches the target semantic vector.

[0054] Step S600: Defect localization step. The location of the workpiece defect is determined by the optimized semantic vector to obtain the current localization result. The current localization result is compared with the preliminary localization result. When the overlap meets the threshold, the current localization result is taken as the final position; when the overlap does not meet the threshold, the current localization result is added to the training set of the preliminary localization model for retraining, and the preliminary localization result is taken as the final position.

[0055] Specifically, the optimized semantic vectors contain rich semantic information and can effectively represent various defect features. These semantic vectors are compared pixel-by-pixel with features in the workpiece image, and the location of potential defects on the workpiece surface is identified using a similarity matching method as the current localization result.

[0056] In this embodiment, cosine similarity is used to calculate the similarity between semantic vectors and image features.

[0057] A sliding window operation is performed on the workpiece surface image to extract small regions from the image, with each small region serving as an image feature. Each image feature is also converted into a vector representation.

[0058] For each image feature vector x, calculate its cosine similarity with the optimized semantic vector v:

[0059]

[0060] Where x·v represents the dot product of vectors x and v, and ||x|| and ||v|| represent the norms of vectors x and v, respectively. The cosine similarity value is between -1 and 1, with values ​​closer to 1 indicating greater similarity between the two vectors. All feature vectors of the workpiece image are traversed, and the cosine similarity between each feature vector and the optimized semantic vector is calculated. The region with the highest similarity is selected as the defect location.

[0061] After defect location is completed, an inspection report is generated. The inspection report details the specific location and type of the defect, providing users with further analysis and processing information. The inspection report typically includes the following: workpiece identification information, defect location coordinates, defect type (e.g., cracks, dents, scratches), defect severity, and other relevant information. This information is presented through charts, text descriptions, and diagrams to help users intuitively understand the defect. The inspection report may also include an overall quality assessment of the workpiece, providing guidance for subsequent workpiece repair and quality control.

[0062] By generating detailed inspection reports, users can quickly locate and repair defects on workpieces, thereby improving production efficiency and product quality. The data in the reports can also be used for process improvement and quality management, helping companies to continuously optimize their production processes.

[0063] Example 2

[0064] In this embodiment, the semantic vector is further optimized using a particle swarm optimization algorithm: Initializing the particle swarm: the best individual from the genetic algorithm stage is used as the initial particle swarm of PSO; updating the velocity and position of the particles according to the current position and velocity update formula; calculating the fitness value of each particle, updating the individual optimal value and the global optimal value; iterating until the fitness function converges to the optimal solution; alternating between the genetic algorithm and the particle swarm optimization algorithm to optimize the semantic vector.

[0065] The velocity and position update formulas in the particle swarm optimization algorithm are as follows:

[0066] v i (t+1)=ωv i (t)+c1r1(p i -x i (t))+c2r2(gx i (t))

[0067] x i (t+1)=x i (t)+v i (t+1)

[0068] Where ω is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers, and pi Let g be the individual optimal position of the particle, and g be the global optimal position.

[0069] Through the velocity update formula, the position and velocity of particles are adjusted based on their own experience and the collective experience of the swarm, thus enabling efficient searching within the solution space. A particle's new position is determined by its current position and the updated velocity. In this way, each particle moves within the solution space according to its velocity changes, gradually approaching the optimal solution. This allows the particle swarm to balance between global and local optima, maintaining search diversity while accelerating convergence.

[0070] After each update, the fitness value of each particle is calculated, and the individual optimal value and the global optimal value are updated. The fitness function is also used to evaluate the accuracy of each semantic vector. The individual optimal position is updated by comparing the current particle's fitness value with its historical best fitness value; the global optimal position is updated by comparing the fitness values ​​of all particles. This process is repeated iteratively until the fitness function converges to the optimal solution, that is, the fitness value no longer increases significantly after multiple iterations.

[0071] Throughout the process, genetic algorithms and particle swarm optimization algorithms can be used alternately. Genetic algorithms provide global search capabilities, while particle swarm optimization algorithms provide fast local optimization capabilities. The combination of the two makes semantic vector optimization more efficient and accurate.

[0072] Through further optimization of the particle swarm optimization algorithm, the accuracy and reliability of the semantic vectors have been improved, providing more accurate input for subsequent defect localization steps.

[0073] Example 3

[0074] In this embodiment, the method further includes combining the workpiece's geometric and process information to achieve precise defect location.

[0075] The geometric information of a workpiece refers to its shape, dimensions, surface features, and other geometric attributes. This geometric information can be obtained through CAD models, design drawings, or measured data. The geometric information includes: the workpiece's length, width, height, surfaces, and edges; the curvature, bumps, holes, and other geometric details of the workpiece surface.

[0076] The process information refers to the process parameters and process routes used in the manufacturing process of the workpiece. Process information includes: the workpiece machining method, such as milling, drilling, stamping, and welding; specific machining parameters, such as feed rate, tool type, and depth of cut; and the process route: the machining sequence of the workpiece and the specific content of each process.

[0077] By combining the identification results, geometric information, and process information, defects on the workpiece surface can be located more accurately. The specific implementation steps are as follows:

[0078] Matching recognition results with geometric information: The recognition results provide the approximate location and features of the defect (such as cracks, pits, etc.);

[0079] By combining the workpiece's geometric information, the specific location of the defect on the workpiece surface can be determined. For example, a crack may be located on a particular curved surface or edge.

[0080] Analyzing defect causes by incorporating process information: Understanding process information can help analyze the causes of defects. For example, a crack may be caused by thermal stress generated during the welding process.

[0081] By combining process parameters and process routes, the accuracy of defect location can be further verified, and other potential defect areas can be analyzed.

[0082] By combining geometric and process information, the specific location and characteristics of defects are marked on the CAD model or design drawings of the workpiece. A detailed defect location report is generated, including the precise coordinates, type, size, and cause analysis of the defects.

[0083] By combining the above process with the identification results, geometric information, and process information, the defects can be accurately located, improving the accuracy and efficiency of defect detection and quality control.

[0084] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A defect localization method based on a language model, characterized in that, Includes the following steps: S100, Data preprocessing steps: Acquire an image of the workpiece surface and perform preprocessing; S200, Preliminary positioning step: Input the preprocessed image data into the preliminary positioning model to obtain the preliminary positioning result; S300, Language Model Training Steps: Generate semantic descriptions based on image features extracted by CNN; Query historical images based on preliminary localization results, and generate target semantic descriptions based on historical image features extracted by CNN; S400, Semantic Vector Generation Steps: Convert the semantic description into a semantic vector, and convert the target semantic description into a target semantic vector; S500, Genetic Algorithm Optimization Steps: Optimize the semantic vector; S600, Defect Location Steps: Determine the location of the workpiece defect through the optimized semantic vector to obtain the current location result. Compare the current location result with the preliminary location result. When the overlap meets the threshold, the current location result is taken as the final location. When the overlap does not meet the threshold, the current localization result is added to the training set of the preliminary localization model for retraining, and the preliminary localization result is used as the final location.

2. The defect localization method based on a language model according to claim 1, characterized in that, The S100 step includes: acquiring a high-resolution image of the workpiece surface; performing grayscale processing on the image; removing noise using Gaussian filtering; and performing edge detection using Canny edge detection.

3. The defect localization method based on a language model according to claim 2, characterized in that, The S300 step includes: using a CNN convolutional neural network to extract image features from the preprocessed image; and using a Transformer model to convert the extracted image features into a semantic description.

4. The defect localization method based on a language model according to claim 3, characterized in that, The S400 step includes: using the BERT model to convert the generated semantic description into a semantic vector representation, and converting the target semantic description into a target semantic vector representation.

5. The language model-based defect localization method according to claim 4, characterized in that, The S500 step includes: generating an initial semantic vector population; selecting semantic vectors with high fitness for reproduction; generating a new generation of semantic vectors through crossover and mutation operations; defining a fitness function to evaluate the accuracy of each semantic vector; and iteratively selecting, crossing over, and mutating to evolve the vectors.

6. The language model-based defect localization method according to claim 5, characterized in that, The fitness function is defined as follows: Among them, v i Semantic vectors It is the target semantic vector.

7. The language model-based defect localization method according to claim 1, characterized in that, It also includes further optimization of semantic vectors using particle swarm optimization: The best individual from the genetic algorithm stage is used as the initial particle swarm for the particle swarm optimization algorithm; the velocity and position of the particles are updated according to the current position and velocity update formula; the fitness value of each particle is calculated, and the individual optimal value and the global optimal value are updated; the iteration continues until the fitness function converges to the optimal solution; the semantic vector is optimized by alternating between the genetic algorithm and the particle swarm optimization algorithm.

8. The defect localization method based on a language model according to claim 7, characterized in that, The velocity and position update formulas in the particle swarm optimization algorithm are as follows: v i (t+1)=ωv i (t)+c1r1(p i -x i (t))+c2r2(g-x i (t)) x i (t+1)=x i (t)+v i (t+1) Where ω is the inertia weight, c1 and c2 are learning factors, r1 and r2 are random numbers, and p i Let g be the individual optimal position of the particle, and g be the global optimal position.

9. The defect localization method based on a language model according to claim 1, characterized in that, It also includes manual verification of the generated semantic descriptions to improve the accuracy of the semantic vectors.

10. A defect localization system based on a language model, characterized in that, The system uses the language model-based defect localization method as described in any one of claims 1-9.

Images