Defect area positioning method and device, storage medium and terminal

Abstract

The invention discloses a defect region positioning method and device, a storage medium and a terminal, and the method comprises the following steps: extracting region keyword clauses and defect keyword clauses from prompt words, and segmenting a to-be-processed image into a plurality of local region images; generating each local area image, and performing the following operations: generating a first similarity between the local area image and an area keyword clause, and screening out an area of which the first similarity is greater than a first similarity threshold as a subsequent processing area, and then generating a second similarity between the local area image and the defect keyword clause in a subsequent processing area, constructing a bias coefficient by using the first similarity and the second similarity, carrying out weighted combination on the first similarity and the second similarity to form a comprehensive similarity, and judging whether the area is a defect area by judging whether the comprehensive similarity is greater than a set threshold value or not. According to the positioning method, the meaning of the prompt word can be analyzed in a structured manner, the to-be-detected area is accurately positioned, and the corresponding defect area is positioned.

Description

Technical Field

[0001] This invention relates to the field of industrial processing technology, and in particular to a method, apparatus, storage medium, and terminal for locating defective areas. Background Technology

[0002] In the process of intelligent industrial production, visual inspection technology, as a core means of quality control, has been widely applied in fields such as electronics manufacturing, automobile assembly, and precision instrument processing. Accurate defect location is a crucial step in ensuring product quality. With the rapid development of cross-modal technologies based on Vision-Language Models (VLMs), solutions using natural language prompts to guide defect detection have gradually become a research hotspot. However, defects in industrial scenarios often exhibit locality and diversity, and detection requirements frequently include detailed spatial semantic descriptions (such as "tiny pores on weld edges" or "scratches on the inner ring of a bearing"). Existing technologies still face significant technical bottlenecks in handling the fusion of such complex semantics and spatial constraints.

[0003] Existing VLM-based detection methods mostly employ a holistic semantic matching approach, failing to structurally parse natural language prompts into two independent features: "regional semantics" (e.g., "rolled edge region") and "defect semantics" (e.g., "deep crack"). This results in the model's inability to accurately understand commands related to "specific defects in specific regions." For example, when inputting "detect fatigue cracks at the root of gear teeth," the model might classify the entire gear region as the area of ​​interest or mistakenly include similar defects in other regions in the detection results, significantly reducing localization accuracy.

[0004] Existing technologies also have shortcomings in terms of semantic alignment mechanisms between regions and defects. Current mainstream methods have not yet established a dual alignment mechanism for region and defect semantics, relying solely on cross-modal matching using a single semantic vector. This leads to a "semantic disconnect" between region features and defect features in complex industrial scenarios. For example, when detecting "bubbles on the display screen border," the model may misclassify a border region without bubbles as a defect region because the border region features and bubble defect features have a high matching degree individually, resulting in a large number of false detections.

[0005] Therefore, there is an urgent need for a method to locate defect regions that can structurally parse the semantics of prompt words, establish a dual semantic alignment mechanism between regions and defects, dynamically filter image regions, and introduce spatial constraints, in order to solve the bottleneck problem of existing technologies. Summary of the Invention

[0006] In view of this, the main objective of the present invention is to provide a method, apparatus, storage medium and terminal for locating defective areas.

[0007] To achieve the above object, the technical solution of the present invention is implemented as follows: A method for locating a defective area, comprising the following steps: receiving a prompt word, and extracting a region keyword clause and a defect keyword clause from the prompt word; receiving a to-be-processed image, and dividing the to-be-processed image into multiple local region images; performing the following operations on each local region image: obtaining the coefficient ration corresponding to the local region image, generating a first similarity between the local region image and the region keyword clause, and when the first similarity ≤ the first preset threshold, stop executing this operation, otherwise, generate a second similarity between the local region image and the defect keyword clause, the comprehensive similarity = the first similarity × (1 - ration) + ration × the second similarity, 0 < ration < 1, and when the comprehensive similarity > the second preset threshold, mark the local region image as a defective area.

[0008] As an improvement of an embodiment of the present invention, it further comprises the following steps: obtaining a plurality of region keywords and a plurality of defect keywords, constructing a word region embedding matrix based on the plurality of region keywords, and constructing a defect word embedding matrix based on the plurality of defect keywords; processing the region embedding matrix based on the PCA algorithm to obtain a semantic projection matrix and the semantic projection matrix .

[0009] As an improvement of an embodiment of the present invention, the "extracting a region keyword clause and a defect keyword clause from the prompt word" specifically includes: extracting a region keyword clause and a defect keyword clause from the prompt word, and based on a preset CLIP model and the semantic projection matrix , processing the region keyword clause to obtain a region text embedding vector, and based on a preset CLIP model and the semantic projection matrix , processing the defect keyword clause to obtain a defect text embedding vector; then, performing normalization processing on both the region text embedding vector and the defect text embedding vector.

[0010] As an improvement of an embodiment of the present invention, the "dividing the to-be-processed image into multiple local region images" specifically includes: based on an image cropping module in a preset CLIP model, dividing the to-be-processed image into multiple non-overlapping local region images.

[0011] As an improvement of an embodiment of the present invention, the following steps are further included: extracting the original features of each local region image based on the image encoder in the preset CLIP model, and then mapping the original features to the dimensional space consistent with the regional text embedding vector through the learnable linear matrix in the preset CLIP model. After that, normalization processing is performed to obtain the image feature alignment vector corresponding to each local region image; the "generating the first similarity between the local region image and the regional keyword clause" specifically includes: calculating the first similarity between the local region image and the regional keyword clause by using dot product similarity; the "generating the second similarity between the local region image and the defect keyword clause" specifically includes: calculating the second similarity between the local region image and the defect keyword clause by using dot product similarity.

[0012] As an improvement of an embodiment of the present invention, the "obtaining the coefficient ration corresponding to the local region image" specifically includes: generating the coefficient ration corresponding to the local region image based on the gated attention mechanism.

[0013] As an improvement of an embodiment of the present invention, the following steps are further included: performing the following processing on all local region images marked as defect regions: dividing a display area to be displayed from the local region image, where the length of the display area to be displayed is h and the width is w; constructing a Gaussian kernel G, in which the weight of the central pixel is the highest, and the weights of the pixels closer to the edge are lower; making the brightness value of any pixel in the local region image = the weight corresponding to the pixel × the comprehensive similarity corresponding to the local region image; replacing the corresponding area in the image to be processed with the local region image; displaying the image to be processed.

[0014] An embodiment of the present invention further provides a positioning device for defect regions, including the following modules: a prompt word acquisition module, configured to receive a prompt word and extract a regional keyword clause and a defect keyword clause from the prompt word; an image acquisition module, configured to receive an image to be processed and divide the image to be processed into multiple local region images; a processing module, configured to perform the following operations on each local region image: obtaining the coefficient ration corresponding to the local region image, generating the first similarity between the local region image and the regional keyword clause, and when the first similarity ≤ the first preset threshold, this operation stops execution, otherwise, generating the second similarity between the local region image and the defect keyword clause, and the comprehensive similarity = the first similarity × (1 - ration) + ration × the second similarity, where 0 < ration < 1, and when the comprehensive similarity > the second preset threshold, the local region image is marked as a defect region.

[0015] This invention also provides a terminal, including: one or more processors; and a memory storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described positioning method.

[0016] This invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described positioning method.

[0017] The defect region localization method, apparatus, storage medium, and terminal provided in this invention have the following advantages: This invention discloses a defect region localization method, apparatus, storage medium, and terminal. The localization method includes the following steps: extracting region keyword clauses and defect keyword clauses from prompt words; segmenting the image to be processed into multiple local region images; generating each local region image and performing the following operations: generating a first similarity between the local region image and the region keyword clause; generating a second similarity between the local region image and the defect keyword clause; generating a comprehensive similarity corresponding to the first and second similarities; when both the first similarity and the comprehensive similarity are greater than a threshold, the local region image is marked as a defect region. This localization method can structurally parse the semantics of prompt words and locate the corresponding defect region. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the defect region location method provided in an embodiment of the present invention; Figure 2A , Figure 2B and Figure 2C The experimental results diagram provided in the embodiments of the present invention are shown in the figure. Figure 2A This is a normal rolled edge image. Figure 2B Image of the defective rolled edge. Figure 2C This is a heat map for defect edge detection. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0020] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0021] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings and are used only for the convenience of describing this document and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements, or direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0022] Embodiment 1 of the present invention provides a method for locating defective regions, such as... Figure 1 As shown, it includes the following steps: Step 101: Receive a prompt word, and extract a region keyword clause and a defect keyword clause from the prompt word; The system first receives the prompt word input by the user. The prompt word may be a descriptive text used to specify the image-related information to be processed. Two types of key clauses are extracted from this prompt word, namely the region keyword clause and the defect keyword clause. The region keyword clause may describe information such as the characteristics or names of different regions in the image, such as "engine compartment", "upper left corner region of the circuit board", etc.; The defect keyword clause describes the content related to possible defects, such as "scratches", "cracks", etc.

[0023] Step 102: Receive the image to be processed, and segment the image to be processed into multiple local region images; Receive the preprocessed image. Image preprocessing may include operations such as noise reduction and contrast enhancement, aiming to make subsequent analysis more effective. Segment this preprocessed image into multiple local region images. The reason for doing this is that different regions may have different characteristics, and separate processing can analyze each part more carefully. For example, a large circuit board image can be segmented into individual small circuit module images.

[0024] In practice, install a laser line scan 3D camera on the pipeline side, deploy the 3D camera on the robotic arm, align the lens vertically with the hemming area of the top cover, the robotic arm moves the 3D camera to scan the edge area of the top cover at a speed of 200mm / s, trigger the 3D camera to collect images through IO signals, set the frame rate to 2048fps, and ensure clear imaging of moving objects. The 3D camera outputs a 547×830 pixel luminance map and stores it in the background in real time.

[0025] Step 103: Perform the following operations on each local region image: Obtain the coefficient ration corresponding to the local region image, generate the first similarity between the local region image and the region keyword clause. When the first similarity ≤ the first preset threshold, this operation stops execution. Otherwise, generate the second similarity between the local region image and the defect keyword clause. The comprehensive similarity = the first similarity × (1 - ration) + ration × the second similarity, 0 < ration < 1. When the comprehensive similarity > the second preset threshold, the local region image is marked as a defect region.

[0026] Perform a series of operations on each local region image: Calculate the first similarity between the local region image and the region keyword clause. This measures the matching degree between the content of the local region image and the described region characteristics. The higher the similarity, the more the region conforms to the characteristics described by the region keyword clause.

[0027] Calculate the second similarity between the local region image and the defect keyword clause. This measures the degree of match between the content of the local region image and the described defect features; a higher similarity indicates that the region is more likely to contain a defect.

[0028] The number of local region images is Num. These Num local region images are arranged in a queue, from the head to the tail of the queue, and are designated as the 1st local region image, the 2nd local region image, ..., the Numth local region image. k is a natural number, k = 1, 2, ..., Num. In this application, the subscript k represents the kth local region image.

[0029] The ration coefficient is calculated. By constructing gated attention, the optimal parameters are updated using the gradient descent algorithm. The ration value is between 0 and 1. It is used to balance the proportion of defect similarity and region similarity in the subsequent calculation of comprehensive similarity.

[0030] The overall similarity is calculated using the formula: "Overall Similarity = First Similarity × (1 - ration) + ration × Second Similarity". This formula uses the coefficient ration to comprehensively consider the degree of regional matching and defect matching.

[0031] Finally, a judgment is made. When the overall similarity is greater than the second threshold (a pre-set standard for measuring whether the overall matching degree is high enough), the local area image is marked as a defective area, that is, the area is considered to have a defect.

[0032] This localization method is highly flexible. By extracting keyword clauses from received prompts, it can flexibly adjust the area to be searched and the defect features to be identified according to different application scenarios and needs. For example, when inspecting images of different types of products, only the content of the prompts needs to be changed to adapt to the new inspection task, without requiring extensive modifications to the program.

[0033] The localization method's comprehensive judgment is reasonable. It utilizes a comprehensive similarity calculation method, comprehensively considering the similarity between the local region image and the region keyword clauses, as well as the defect keyword clauses. Furthermore, it uses coefficients... (That is, the ration corresponding to the k-th local region image) can have its weights adjusted in the judgment based on the actual situation, making the judgment result more scientific and reasonable. For example, in some scenarios, more attention may be paid to the matching degree between the region features and the preset, so the weights can be appropriately reduced. Increase the proportion of the first similarity in the overall similarity score.

[0034] This localization method offers detailed local analysis by segmenting the image into multiple local regions for processing. This allows for meticulous analysis of each part of the image, preventing the omission of local defects and improving the accuracy of defect detection. For example, in images of complex mechanical parts, different areas may contain different types of defects; this method can better identify problems in each area.

[0035] In this embodiment, the following steps are also included: obtaining several regional keywords and several defect keywords; constructing a word region embedding matrix based on the several regional keywords; constructing a defect word embedding matrix based on the several defect keywords; and processing the region embedding matrix using the PCA algorithm to obtain a semantic projection matrix. and semantic projection matrix .

[0036] To achieve the separation and effective mapping of "regional keywords" and "defect keywords" in the prompts, this step first constructs two semantic projection matrices. and Then, the built-in CLIP is used for text processing to obtain region text embedding vectors and defect text embedding vectors. After converting the text semantics into a vector space that can be aligned with image features using an internal learnable matrix, a semantic projection matrix is ​​used. and The process involves processing the data and finally normalizing the feature scale to ensure consistency in subsequent similarity calculations. The core of this process is utilizing the semantic projection matrix. and semantic projection matrix The input text is decomposed and processed separately, and then the high-dimensional text is embedded and mapped to a low-dimensional semantic subspace. The specific process is as follows: Obtain several regional keywords, that is, collect common regional descriptive terms in industrial scenarios (such as "rolled edge", "weld", "corner", "edge", "center", "surface", and "hole"), and thus form a regional keyword list, denoted as . ,For example, ="Rolled edge", m is a natural number.

[0037] Obtain a number of defect keywords, that is, collect industrial defect type terms (e.g., "crack", "scratch", "dent", "foreign object", "bubble", and "deformation"), and thus form a defect keyword list, denoted as . ,For example, =“scratches”, where n is a natural number.

[0038] Introducing the CLIP (Contrastive Language-Image Pre-training) model, the CLIP model is a text-image pre-training model that uses contrastive learning. The model outputs a text information encoder. and image encoder ,in, This means that the input is a text word (e.g., scratch or surface), which is encoded and output as a dimensionless string. The vector, This means that the input is an image, which is encoded and output has one dimension. The vector.

[0039] Regional keyword list Constructing a word region embedding matrix Similarly, for the defect keyword list Constructing a defect word embedding matrix .

[0040] region word embedding matrix and defect word embedding matrix Perform PCA (Principal Component Analysis) to calculate its covariance matrix and solve for eigenvalues ​​and eigenvectors. Retain the direction with the largest variance in the data to extract the main semantic direction and construct the semantic projection matrix. and semantic projection matrix : , ,in, The preset principal component dimension is, for example, 64.

[0041] Through the above steps, the target text can be transformed into a vector or matrix form that can be processed by the detection system. By using PCA to discard non-primary semantic directions to some extent, a semantic projection matrix containing the primary semantic directions can be obtained. and This prepares for the subsequent decomposition of natural language semantics.

[0042] In this embodiment, the step of "extracting region keyword clauses and defect keyword clauses from the prompt words" specifically includes: extracting region keyword clauses and defect keyword clauses from the prompt words based on a preset CLIP model and semantic projection matrix. The region keyword clauses are processed to obtain region text embedding vectors, based on a pre-defined CLIP model and semantic projection matrix. The defect keyword clause is processed to obtain the defect text embedding vector; then, both the region text embedding vector and the defect text embedding vector are normalized.

[0043] The core of this step is to parse the input natural language prompts into structured "region keywords" and "defect keywords," laying the foundation for subsequent cross-modal semantic matching. For example, to predict scratch defects in the rolled edge region of an image, you can input the prompt T="detect rolled edge scratches." The region semantic term "rolled edge" and the defect keyword "scratches" can be extracted from the input prompt using the following method. The specific steps are as follows: First, the input prompt T is decomposed into a "region keyword clause" and a "defect keyword clause", denoted as: This is a mathematical formula for direct summation, where T is... and The sign is obtained by performing a straight sum operation. For straight and symbols, where, This represents structured splitting, with specific rules as follows; This represents the keyword vectors describing the region, spatial location, etc., in the region word set. This represents keyword vectors indicating the types and categories of defects described in the defect terminology set.

[0044] Then, the CLIP built-in pre-trained text encoder described above is used. The region text embedding vector is obtained by using the learnable linear matrix W. and defective text embedding vectors Then, using the semantic projection matrix mentioned above ( and Mapping high-dimensional text embeddings to a low-dimensional semantic subspace: , Finally, normalization is performed to unify the feature scale, ensuring consistency in subsequent similarity calculations. , Here, `normalize()` performs L2 normalization, scaling the vector to make its L2 norm equal to 1. This step, through structured parsing and semantic projection, achieves the separation and quantification of the "region-defect" semantics in the prompt words, generating... and These serve as region matching and defect matching vectors, respectively, providing textual semantic references for subsequent image patch selection and alignment.

[0045] In this embodiment, the step of "segmenting the image to be processed into multiple local region images" specifically includes: segmenting the image to be processed into multiple non-overlapping local region images based on the image cropping module in the preset CLIP model.

[0046] The core of this step is to process the input image. The image is cropped into multiple local region images (Patch) and then processed by an image encoder. Extract the feature vector of each patch, ensuring it lies in a matching feature space with the region text embedding vector and defect text embedding vector generated in step 1.2.1 above, thus preparing for cross-modal semantic similarity calculation. The specific steps are as follows: Use the CLIP model's built-in image cropping to crop the input image into K non-overlapping image patches, and then use its built-in image encoder. Extract each image patch The original features are then processed through a built-in learnable linear matrix. The formula represents Belongs to a dimension The real space, representing the set of real numbers, Indicates by The space consisting of real numbers of dimension , where Representing the dimension of the image feature vector, it maps high-dimensional patch features to a format consistent with the text semantic vector. The image feature alignment vector is obtained by normalizing the 3D space. This step transforms the image from pixel space to semantic feature space, each... This represents the semantic information of the corresponding patch (such as texture, shape, defect features, etc.). This is achieved through the projection matrix. The alignment effect makes With text semantic vectors , Being in the same feature space, they provide the feature basis of the image for subsequent steps of region keyword matching and defect keyword matching.

[0047] In this embodiment, the following steps are also included: extracting the original features of each local region image based on the image encoder in the preset CLIP model, and then mapping the original features to a dimensional space consistent with the region text embedding vector through the learnable linear matrix in the preset CLIP model. After that, normalization processing is performed to obtain the image feature alignment vector corresponding to each local region image. The step of "generating the first similarity between the local region image and the region keyword clause" specifically includes: calculating the first similarity between the local region image and the region keyword clause using dot product similarity. The step of "generating the second similarity between the local region image and the defect keyword clause" specifically includes: calculating the second similarity between the local region image and the defect keyword clause using dot product similarity.

[0048] To optimize detection efficiency and meet the requirements of industrial scenarios, and to avoid irrelevant areas lowering detection efficiency, this step aims to filter out a set of candidate patches that semantically match the prompt word "region description" by calculating the similarity between the features of each patch in the image to be processed and the semantic vector of the region, thereby narrowing the search range for defect localization and achieving fast and accurate detection.

[0049] Dynamic patch filtering first involves calculating the features of each patch in the image to be processed using dot product similarity. With region semantic vector semantic matching degree ,in, Let be the feature vector of the k-th patch of the image to be processed. Then, set a similarity threshold. (For example, =0.6 (can be optimized through validation set), filter the set of patch region indexes with scores higher than the threshold. Finally, a set of patch region index values ​​with scores higher than a threshold is selected. These index values ​​are then used to guide subsequent calculations, preventing unimportant regions from affecting detection efficiency.

[0050] Defect semantic similarity calculation, after obtaining the filtered index set Then, the semantic matching degree of defects in the selected region is calculated again using dot product similarity. Then, the gated attention mechanism is introduced, and the gated coefficients are defined. (Right now The corresponding ration), dynamically weighing the contributions of region semantics and defect semantics; This formula defines The calculation method for (i.e., comprehensive similarity): First, (i.e., first similarity) and (i.e., second similarity) splicing (where, (This indicates a concatenation operation), then combined with the weight matrix. Multiply, plus bias Finally, it is passed through the sigmoid function. Activation, mapping the result to Interval. As adaptive weights, used to fuse region scores and defect scores, they have a structure of "feature concatenation + linear transformation + activation," allowing the weights to... The parameters can be updated using gradient descent based on the actual environmental detection target to achieve a better correlation ratio.

[0051] in, The Sigmoid activation function outputs [0,1]. The gating parameters are learnable (optimized through training using the gated loss function below; for example, when the input word is "small scratches on the car door," the parameters are adjusted to make the system focus more on the car door area to avoid misjudgment). The default is... The range is [0.5, 0.5]. , , It is a concatenated vector of region and defect similarity.

[0052] Finally, through the gated coefficient Similarity fused from two semantic dimensions , among which, among which, when When it tends towards 1, tending towards 0, Tend to That is, defect information; similarly, when When it approaches 0, It tends to focus on location information and can autonomously adjust the proportion of relevant correlation based on the semantic input of the actual detection task. Here, we set... A value greater than 0.7 indicates a defect area.

[0053] This step successfully centralized the detection target area, improving detection efficiency. Natural language processing enhanced the interactivity of the detection, and the similarity of fused semantics was used as the basis for judging defects and generating target maps. This made defect extraction more accurate and faster, while also avoiding the influence of other complex but irrelevant features in the background.

[0054] To enable the model to adapt more autonomously to the detection task and focus, and to obtain learnable gating parameters with better performance, a loss function is proposed. ,in, Indicates human-made annotations and The corresponding region constant; it has only two values, 0 or 1, where 0 represents no defects and 1 represents defects. Within the same screening region, if the detection relevance... If the correlation is close to the manually labeled value (i.e., the detection relevance is close to the true result), then Approaching 0, loss function If it is close to zero (small loss), then Learnable gating parameters ( The more accurate the parameters, the better. Therefore, this step proposes that reliable parameters can be trained using the gradient descent algorithm. Specifically: 1. Input an industrial defect dataset D, where each sample contains an image I, a cue word T, and a true defect annotation. Among them, the labeling of actual defects Common annotation software (such as LabelMe, SegmentationMap, etc.) can be used to enable the model to receive location information and assign values ​​to the labeled regions. When the labeled location contains an entire patch, the full value (e.g., 1, 0) is assigned. If there are partial labels in the segmented patch, the values ​​are assigned in the same proportion between [0, 1] according to the proportion of the labeled area.

[0055] 2. Forward computation: Obtain the fusion similarity of candidate patches through the above steps. The initial training uses default parameters; 3. Reverse optimization, with The loss function is updated using the gradient descent algorithm. Minimize the difference between the predicted value and the actual annotation; 4. Iteration Termination: When the loss function converges and no longer decreases significantly or reaches the preset number of iterations, output and apply the optimized gating parameters. .

[0056] Through the above steps, the gated mechanism and the constructed loss function enable the model to dynamically adjust parameters and different semantic proportions based on actual detection, providing an interpretable basis for subsequent heatmap detection.

[0057] In this embodiment, "obtaining the coefficient ration corresponding to the local region image" specifically includes: generating the coefficient ration corresponding to the local region image based on the gated attention mechanism.

[0058] In this embodiment, the following steps are also included: The local region images marked as defect areas are processed as follows: a displayable region is delineated from the local region image, the displayable region having a length of h and a width of w; a... A Gaussian kernel G is used, in which the center pixel has the highest weight and the pixels closer to the edge have lower weights; the brightness value of any pixel in the local region image is equal to the weight of the pixel multiplied by the comprehensive similarity of the local region image; the local region image is used to replace the corresponding region in the image to be processed; the image to be processed is then displayed.

[0059] The similarity of each candidate patch was obtained. Then, similarity can be used to determine defects in the heatmap, so as to achieve defect location and visualization; First, determine the pixel region covered by each candidate patchk. for; ,in, , ( , The coordinates of the top-left corner of each patch are given, where h and w are the height and width of the patch. Then, a Gaussian kernel G is constructed, with the center pixel having the highest weight and decreasing towards the edges to achieve a smooth transition by giving higher scores to the center pixels and lower scores to the edge pixels. ,in, , : Relative coordinates within the patch; The standard deviation of the Gaussian kernel (controls the smoothing range, such as taking...) .

[0060] Finally, the similarity will be merged. Assign to all pixels covered by the patch: Finally, by judging each selected patch... The value is used to determine the depth of the output heatmap, covering its original pixel, and a smooth transition in the heatmap display is achieved through a Gaussian convolution kernel.

[0061] Implementing this solution will yield the following technical effects and significant technological advancements: Through algorithm optimization, the inference time has been reduced from 200-300ms / image in traditional methods to 50-70ms / image, which can meet the real-time detection requirements of industrial production lines and significantly improve production efficiency.

[0062] Meanwhile, in terms of detection accuracy, compared to the original overall defect detection, the false detection rate is reduced by 5%-10%, the ability to resist background interference is enhanced, and it can detect various subtle defects while maintaining high accuracy, with pixel-level AUROC > 98.0%, effectively improving product quality. Natural language interactive processing: A commonly used word set is constructed, and operators can directly obtain visual results by selecting the desired candidate region and defect type, which greatly improves the interpretability of the detection model, shortens debugging time, and avoids unclear visualization results due to the large number of defect types. Different defects in different regions can be directly selected for detection. Gated mechanism for region-defect semantic fusion: Dynamically adapts the "region-defect" semantic dependency relationship of different defects, can autonomously select the contribution ratio of regions and defects according to the actual task, and can optimize the similarity of the two semantic dimensions according to the loss function to better adapt to the actual task. Pixel-level heatmap generation and visualization avoids redundant detection processes in other areas, only performing similarity calculations on relevant defects in relevant areas, and can associate detection results with natural language semantics on a patch-by-patch basis, enabling operators to understand the basis for the detection system's judgment and increasing operators' trust in the detection results.

[0063] test Test dataset: 5 brightness maps captured by a 3D camera Defect detection: The effect of abnormal defect detection is as follows Figure 2A , Figure 2B and Figure 2C As shown.

[0064] Anomaly effect test analysis: In the heatmap, we can see that the blue area is the rolled edge area. This is because after the patch filters out the selected area, it is marked as a candidate area. The fusion similarity score of the defect-free area is 0 and is set to blue. The fusion similarity score of the defective area is higher and the color is darker. At the same time, since this patent processes the input image into patch blocks, the heatmap pixels cover the patch blocks, and the final output patch block heatmap is used for display.

[0065] Detection speed and memory experiment Dataset: 500 images of the car roof captured by a 3D camera.

[0066] Metrics: pixel-level AUROC, inference time (ms / image), memory usage (MB), proportion of true detection locations, false detection rate, and false negative rate.

[0067] Hardware: NVIDIA GFRTX 3090 GPU, Jetson Nano (edge ​​device).

[0068] Baseline: This patented model (direct detection without screening regions), Patchcore (Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, Peter V. Gehler: Towards Total Recall in Industrial Anomaly Detection. CVPR2022:14298-14308).

[0069] Experiment: The results of the test are shown in the table below:

[0070] To verify the effectiveness of the proposed method for region filtering, the actual location ratio is introduced as a way to verify whether the filtered regions meet expectations. The actual location ratio verification is shown in the table below.

[0071] The true position ratio is an index proposed for the present invention, representing the proportion of the selected area in the area to be actually detected (for example, if the area to be actually detected is 100 pixels and the detected area pixels selected by this patent are 96, so the proportion of the selected area in the area to be actually detected is 96%). The true position ratio is affected by the picture size and the division of patches. Since the other methods detect the whole picture, the true position ratio index is not required.

[0072] Embodiment 2 of the present invention provides a positioning device for defective areas, including the following modules: a prompt word acquisition module, configured to receive a prompt word and extract area keyword clauses and defect keyword clauses from the prompt word; an image acquisition module, configured to receive a to-be-processed image and divide the to-be-processed image into multiple local area images; a processing module, configured to perform the following operations on each local area image: obtain the coefficient ration corresponding to the local area image, generate a first similarity between the local area image and the area keyword clause, and when the first similarity ≤ a first preset threshold, stop executing this operation; otherwise, generate a second similarity between the local area image and the defect keyword clause, and the comprehensive similarity = the first similarity × (1 - ration) + ration × the second similarity, where 0 < ration < 1. When the comprehensive similarity > a second preset threshold, the local area image is marked as a defective area.

[0073] Embodiment 3 of the present invention provides a terminal, including: one or more processors; a memory storing one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the positioning method in Embodiment 1.

[0074] Embodiment 4 of the present invention provides a storage medium storing a computer program thereon, and when the program is executed by a processor, the positioning method in Embodiment 1 is implemented.

[0075] It should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0076] The series of detailed descriptions listed above are only specific descriptions of the feasible embodiments of the present invention, and they are not intended to limit the protection scope of the present invention. Any equivalent embodiments or modifications made without departing from the technical spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for locating a defective region, characterized in that, It includes the following steps: Receive a prompt word, and extract a region keyword clause and a defect keyword clause from the prompt word; Receive an image to be processed, and segment the image to be processed into multiple local region images; Perform the following operations on each local region image: Obtain the coefficient ration corresponding to the local region image, generate the first similarity between the local region image and the region keyword clause. When the first similarity ≤ the first preset threshold, stop executing this operation. Otherwise, generate the second similarity between the local region image and the defect keyword clause. The comprehensive similarity = the first similarity × (1 - ration) + ration × the second similarity, where 0 < ration < 1. When the comprehensive similarity > the second preset threshold, the local region image is marked as a defect region.

2. The positioning method according to claim 1, characterized in that, It further includes the following steps: Obtain a number of region keywords and a number of defect keywords, construct a word region embedding matrix based on the number of region keywords, and construct a defect word embedding matrix based on the number of defect keywords; The semantic projection matrix is ​​obtained by processing the region embedding matrix using the PCA algorithm. and semantic projection matrix .

3. The positioning method according to claim 2, characterized in that, The specific content of "extracting a region keyword clause and a defect keyword clause from the prompt word" includes: Region keyword clauses and defect keyword clauses are extracted from the prompt words, based on a preset CLIP model and semantic projection matrix. The region keyword clauses are processed to obtain region text embedding vectors, based on a pre-defined CLIP model and semantic projection matrix. The defect keyword clause is processed to obtain the defect text embedding vector; After that, perform normalization processing on both the region text embedding vector and the defect text embedding vector.

4. The positioning method according to claim 1, characterized in that, The specific content of "segmenting the image to be processed into multiple local region images" includes: Based on the image cropping module in the preset CLIP model, segment the image to be processed into multiple non-overlapping local region images.

5. The positioning method according to claim 1, wherein It further includes the following steps: Extract the original features of each local region image based on the image encoder in the preset CLIP model, and then map the original features to the same dimensional space as the region text embedding vector through the learnable linear matrix in the preset CLIP model. After that, perform normalization processing to obtain the image feature alignment vector corresponding to each local region image; The specific content of "generating the first similarity between the local region image and the region keyword clause" includes: Calculate the first similarity between the local region image and the region keyword clause using dot product similarity; The specific content of "generating the second similarity between the local region image and the defect keyword clause" includes: Calculate the second similarity between the local region image and the defect keyword clause using dot product similarity.

6. The positioning method according to claim 1, characterized in that, The specific content of "obtaining the coefficient ration corresponding to the local region image" includes: Generate the coefficient ration corresponding to the local region image based on the gated attention mechanism.

7. The positioning method according to claim 1, characterized in that, It further includes the following steps: The following processing is performed on all local area images marked as defect areas: a displayable region is delineated from the local area image, the displayable region having a length of h and a width of w; a... A Gaussian kernel G is used, in which the center pixel has the highest weight and the pixels closer to the edge have lower weights; the brightness value of any pixel in the local region image is equal to the weight of the pixel multiplied by the comprehensive similarity of the local region image; the local region image is used to replace the corresponding region in the image to be processed. Display the image to be processed.

8. A device for locating a defective area, characterized in that, ​ ​ ​ A processing module is configured to perform the following operations on each local region image: obtain the coefficient ration corresponding to the local region image, generate a first similarity between the local region image and the region keyword clause, and when the first similarity ≤ the first preset threshold, stop executing this operation; otherwise, generate a second similarity between the local region image and the defect keyword clause, and the comprehensive similarity = the first similarity × (1 - ration) + ration × the second similarity, where 0 < ration < 1. When the comprehensive similarity > the second preset threshold, the local region image is marked as a defective region.

9. A terminal, characterized in that, Comprising: One or more processors; A memory storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the positioning method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, A computer program is stored thereon, and when the program is executed by a processor, it implements the positioning method according to any one of claims 1 to 7.

Images