A method and system for defect detection

Abstract

The application discloses a method and system for defect detection, which improves accuracy and efficiency, and reduces misjudgment and omissions. The technical scheme is as follows: inputting text and images into a multi-modal fusion encoder to obtain text feature vectors and image block tokens; processing the image block tokens in two parallel paths to extract features and locate abnormal areas, and obtaining a prediction result feature map and a pixel prediction result; fusing features of different scales through residual connection and bilinear interpolation to obtain a rough segmentation result, performing binary processing to generate a mask; identifying frames and points, and inputting the identified frames and points into a mask segmentation network to generate a refined mask and a confidence score; fusing the rough segmentation result and the refined mask to obtain a fine-grained segmentation result; inputting a standard object graph as a template to perform template matching with a to-be-detected picture, performing difference processing on a matching area, performing binary processing, and extracting a suspected defect area; and fusing the suspected defect area and the fine-grained segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial automation and intelligence, and particularly relates to a defect detection method and system. BACKGROUND

[0002] With the rapid development of industrial production, product quality control has become a key link in manufacturing. The level of product quality directly affects the market competitiveness and economic benefits of enterprises. In many production processes, defect detection is an important step to ensure that the product quality meets the standards. Traditional defect detection methods mainly rely on manual visual inspection, which has the disadvantages of low efficiency, susceptibility to human factors, and unstable detection results. With the increasing complexity of products, the accuracy and reliability of manual detection are increasingly difficult to meet the needs of modern industry.

[0003] In today's era of automation and intelligence, it is particularly important to use advanced technical means to improve the efficiency and accuracy of defect detection. Although existing automated detection equipment has improved detection speed to some extent, it still has problems of misjudgment and missed judgment when facing complex and variable product surfaces and defect types. These problems not only increase production costs, but also may lead to unqualified products flowing into the market, causing economic losses and damage to brand reputation for enterprises.

[0004] Although there are some automated detection devices on the market, they still have the following problems in actual application:

[0005] Limited recognition ability for complex product surface features and defect types, difficult to adapt to variable production environments;

[0006] Insufficient generalization ability of detection algorithms, requiring retraining and adjustment for new defect types or changing defect features, increasing maintenance costs;

[0007] Insufficient flexibility and adaptability of the equipment, difficult to quickly adapt to production line adjustments and new product detection needs. SUMMARY

[0008] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0009] The present application aims to solve the above problems and provides a defect detection method and system that can improve the accuracy and efficiency of detection, reduce misjudgment and missed judgment, and improve the efficiency and accuracy of product quality control.

[0010] The technical scheme of the present application is as follows: the present application discloses a defect detection method, which comprises the following steps:

[0011] Step S1: inputting a text and an image into a multi-modal fusion encoder to obtain a text feature vector and an image block token;

[0012] Step S2: adopting two parallel first and second paths to process the image block token to extract features and locate an abnormal area, and obtaining a predicted result feature map and a pixel prediction result;

[0013] Step S3: fusing features of different scales through residual connection and bilinear interpolation to obtain a rough segmentation result, and performing binaryzation processing on the rough segmentation result to generate a mask;

[0014] Step S4: identifying a box and a point in a connected region in the mask, and inputting the identified box and point into a mask segmentation network to generate a refined mask and a confidence score;

[0015] Step S5: fusing the rough segmentation result of the multi-modal fusion encoder and the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result;

[0016] Step S6: inputting a standard object image as a template to perform template matching with a to-be-detected image, performing difference processing on a region obtained through the matching, binaryzation and extraction of a suspected defect region;

[0017] Step S7: fusing the suspected defect region and the fine-grained segmentation result obtained in step S5 to complete defect detection.

[0018] According to an embodiment of the defect detection method of the present application, step S1 further comprises the following steps:

[0019] Step S11: inputting a text and a defective product image into a multi-modal fusion encoder;

[0020] Step S12: processing the text by the multi-modal fusion encoder to output a text feature vector;

[0021] Step S13: when processing the defective product image, the multi-modal fusion encoder divides the defective product image into a plurality of small blocks, and each small block is embedded to obtain an image block token.

[0022] According to an embodiment of the defect detection method of the present application, step S2 further comprises the following steps:

[0023] Step S21: in the processing of the first path, capturing row-level features and column-level features of the image block token to accurately locate an abnormal area;

[0024] Step S22: In the processing of the second path, the global features of the image at different scales are understood to comprehensively find the abnormal area.

[0025] According to an embodiment of the defect detection method of the present application, step S21 further comprises:

[0026] Step S211: extracting row-level features and column-level features from the image features corresponding to the image block token through an average pooling layer;

[0027] Step S212: processing the text feature vector using a convolution layer to obtain language features matching the dimension of the image features;

[0028] Step S213: predicting whether the pixels in the row-level features and the column-level features are abnormal through an attention mechanism, after detecting the abnormal points, restoring the feature map of the prediction result to the original image size using bilinear interpolation, and obtaining the abnormal area.

[0029] According to an embodiment of the defect detection method of the present application, step S22 further comprises:

[0030] Step S221: obtaining multi-scale visual features using an average pooling layer with different kernel sizes;

[0031] Step S222: for each scale, processing the text feature vector using a convolution layer to match the feature dimension;

[0032] Step S223: obtaining the prediction results of the pixels in the row-level features and the column-level features through an attention mechanism, and obtaining the abnormal area.

[0033] According to an embodiment of the defect detection method of the present application, step S3 further comprises:

[0034] Step S31: residual connection refers to adding the output of the shallow layer of the network to the output of one or more layers of the deep network after transformation, so that the deep network extracts more types of features;

[0035] Step S32: bilinear interpolation refers to, in the image pooling process, due to the reduction of the image, part of the features is ablated, bilinear interpolation first performs linear interpolation in the x direction to obtain two intermediate values, and then performs the same operation in the y direction for image resampling;

[0036] Step S33: feature fusion outputs the result to obtain a rough segmentation result, and then sets a threshold to binarize the image to obtain a mask.

[0037] According to an embodiment of the defect detection method of the present application, step S4 further comprises:

[0038] Step S41: Extract the frame and point in the mask connected region, input to the mask segmentation network, and the mask segmentation network encodes the prompt information of the image into an embedding vector in real time to predict the segmentation mask;

[0039] Step S42: The mask segmentation network generates a plurality of possible segmentation masks, which represent the segmentation of different regions in the image, including the whole mask, the partial mask and the sub-partial mask, and the segmentation task is guided by fusing the text feature prompt and the image feature;

[0040] Step S43: A confidence score is calculated for each possible segmentation mask, and a final output is obtained according to the confidence score, and if there are multiple masks, a non-maximum suppression is applied to filter out repeated or redundant masks to obtain a more accurate refined mask, and the segmentation result is ensured to be accurate.

[0041] According to an embodiment of the defect detection method of the application, step S5 further comprises:

[0042] Step S51: Extract the rough segmentation result output by the multi-module fusion encoder separately and map it on the first blank matrix;

[0043] Step S52: Map the accurate segmentation result output by the mask segmentation network on the second blank matrix;

[0044] Step S53: Perform weighted averaging on the first blank matrix and the second blank matrix, filter the areas with low confidence, and obtain the fine-grained segmentation result.

[0045] According to an embodiment of the defect detection method of the application, step S6 further comprises:

[0046] Step S61: Input the standard object template image, perform template matching on the detection image, and find the detection area;

[0047] Step S62: Perform difference processing on the template image and the detection area, perform opening operation after binarization, and obtain the mask after removing the noise points;

[0048] Step S62: Perform connected region analysis on the mask, mark all connected abnormal regions in the image, and separately assign a label to each region;

[0049] Step S63: According to the shape and area of the connected region, further remove the noise points;

[0050] Step S64: Extract the contour of the abnormal region to obtain the rough defect region.

[0051] According to an embodiment of the defect detection method of the application, step S7 further comprises:

[0052] Step S71: mapping the fine-grained segmentation result obtained in step S5 to a third blank matrix;

[0053] Step S72: performing an AND logic operation on the coarse defect region and the matrix defect region to obtain a defect detection result.

[0054] The present application also discloses a defect detection system, which comprises:

[0055] An input module is configured to input text and images into a multi-modal fusion encoder to obtain a text feature vector and an image block token.

[0056] An image block token processing module is configured to process the image block token by using two parallel first and second paths to extract features and locate abnormal regions, and obtain a predicted result feature map and a pixel prediction result.

[0057] A mask generation module is configured to fuse features of different scales by residual connection and bilinear interpolation to obtain a coarse segmentation result, and perform binaryzation processing on the obtained coarse segmentation result to generate a mask.

[0058] A confidence score generation module is configured to identify a box and a point in a connected region in the mask, input the identified box and point into a mask segmentation network, and generate a refined mask and a confidence score.

[0059] A fine-grained segmentation module is configured to fuse the coarse segmentation result of the multi-modal fusion encoder and the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result.

[0060] A suspected defect region extraction module is configured to input a standard object image as a template and perform template matching with a to-be-detected image, perform difference processing on the matched region, binaryzation, and extract a suspected defect region.

[0061] A fusion module is configured to fuse the suspected defect region and the fine-grained segmentation result obtained by the fine-grained segmentation module to complete defect detection.

[0062] The present application also discloses a computer system for defect detection, which comprises a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the defect detection method as described above.

[0063] The present application also discloses a computer-readable storage medium for defect detection, which stores program instructions executable by a processor to implement the steps of the defect detection method as described above.

[0064] The present application also discloses a computer program product comprising a computer program which, when executed by a processor, implements the steps of the defect detection method as described above.

[0065] The present application has the following advantages over the prior art: the present application has better generalization for detection, and solves the problem that traditional algorithms are more sensitive to external conditions such as ambient light. In addition, compared with traditional supervised target detection algorithms or classification algorithms, the algorithm structure of the present method can perform more accurate effect splitting under unsupervised conditions. BRIEF DESCRIPTION OF DRAWINGS

[0066] The above features and advantages of the present application can be better understood by reading the following detailed description of embodiments of the present application in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar or related properties or features can have the same or similar reference numbers.

[0067] Figure 1 A flowchart of an embodiment of the method of defect detection of the present application is shown.

[0068] Figure 2 A schematic diagram of an embodiment of the system of defect detection of the present application is shown. DETAILED DESCRIPTION

[0069] The present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be understood as limiting the scope of protection of the present application in any way.

[0070] Figure 1 A flow of an embodiment of the method of defect detection of the present application is shown. Please see Figure 1 The implementation steps of the method of the present embodiment are described in detail as follows.

[0071] Step S1: input the text and image to the multi-modal fusion encoder to obtain the text feature vector and image block token.

[0072] Step S1 further comprises the following processing process:

[0073] Step S11: input the text and defect image to the multi-modal fusion encoder;

[0074] Step S12: the multi-modal fusion encoder processes the text and outputs the text feature vector;

[0075] Step S13: when processing the defect image, the multi-modal fusion encoder divides the defect image into multiple small blocks (patches), and each small block is embedded to obtain an image block token (token).

[0076] By step S1, the present application simultaneously processes the input text and image data using a multi-modal fusion encoder, which can effectively extract text feature vectors and image block tokens. This multi-modal feature fusion technology improves the accuracy of defect detection, as it can simultaneously consider text descriptions and image content, making the detection results more comprehensive and accurate.

[0077] Step S2: Process the image block tokens using two parallel paths (a first path and a second path running in parallel) to extract features and locate abnormal areas, and obtain predicted result feature maps and pixel prediction results.

[0078] Step S2 further includes the following processing process:

[0079] Step S21: That is, the first path, captures the row-level features and column-level features of the image block Token, to accurately locate the abnormal area;

[0080] Step S22: That is, the second path, understands the global features of the image at different scales to comprehensively find abnormal areas.

[0081] The above step S21 further includes the following processing process:

[0082] Step S211: Extract row-level features and column-level features from image features corresponding to the image block Token through an average pooling layer;

[0083] Step S212: Process the text feature vector using a convolution layer to obtain language features matching the dimensions of the image features;

[0084] Step S213: Predict whether the pixels in the row-level features and column-level features are abnormal through an attention mechanism. After detecting abnormal points, restore the predicted result feature map to the original image size using bilinear interpolation to obtain the abnormal area.

[0085] The above step S22 further includes the following processing process:

[0086] Step S221: Use an average pooling layer with different kernel sizes to obtain multi-scale visual features;

[0087] Step S222: For each scale, process the text feature vector using a convolution layer to match the feature dimensions;

[0088] Step S223: Obtain the prediction results of the pixels in the row-level features and column-level features through an attention mechanism to obtain the abnormal area. The attention mechanism formula is:

[0089]

[0090] wherein Q, K, V are the product of the image and three different trainable parameter matrices, i.e. linear transformation to the input pixel, d k is the dimension of K.

[0091] The two parallel paths in step S2 improve processing efficiency and can quickly locate potential abnormal areas, laying a solid foundation for subsequent defect detection.

[0092] Step S3: The features of different scales are fused by residual connection and bilinear interpolation to obtain a rough segmentation result, and the rough segmentation result is binarized to generate a mask.

[0093] Step S31: Residual connection refers to adding the output of the shallow layer of the network to the output of one or more layers in the deep network after transformation, so that the deep network extracts more types of features;

[0094] Step S32: Bilinear interpolation refers to that in the image pooling process, due to the reduction of the image, part of the features is ablated, bilinear interpolation first performs linear interpolation in the x direction to obtain two intermediate values, and then performs the same operation in the y direction for image resampling;

[0095] Step S33: The feature fusion output result is obtained to obtain a rough segmentation result, and the image is binarized after setting a threshold to obtain a mask.

[0096] In step S3, the effective fusion of features of different scales is realized by residual connection and bilinear interpolation technology, the recognition ability of the model for defects of different sizes and shapes is enhanced, and the mask is generated by binarization processing, which provides a basis for subsequent accurate segmentation.

[0097] Step S4: The frame and the point in the connected region in the mask are identified, and the identified frame and the point are input into the mask segmentation network to generate a refined mask and a confidence score.

[0098] Step S4 further includes the following processing process:

[0099] Step S41: The frame and the point in the connected region in the mask are extracted and input into the mask segmentation network, and the mask segmentation network encodes the prompt information of the image, such as background points, masks, etc. into embedding vectors in real time, which are used to predict the segmentation mask.

[0100] Step S42: The mask segmentation network generates a plurality of possible segmentation masks, which represent the segmentation situation of different regions in the image, including the whole mask, the partial mask and the sub-partial mask, and the feature prompts obtained by inputting the text are fused with the image features to guide the segmentation task, and the fusion formula is F = αF text +(1-α)F imageWherein, F is the fused feature, and a is the weight coefficient.

[0101] Step S43: Calculate a confidence score for each possible segmentation mask, obtain the final output according to the confidence score, and filter out redundant masks by applying non-maximum suppression if there are multiple masks, to obtain a more accurate refined mask, and ensure the accuracy of the segmentation result.

[0102] Step S4 generates a refined mask and a confidence score by identifying the frames and points in the connected regions within the mask and inputting them into the mask segmentation network. This step further improves the accuracy of defect detection and provides a confidence score for each detected defect, which helps subsequent decision-making and processing.

[0103] Step S5: Fuse the coarse segmentation result of the multi-module fusion encoder and the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result.

[0104] Step S5 further includes the following processing process:

[0105] Step S51: Extract the coarse segmentation result output by the multi-module fusion encoder separately and map it on the first blank matrix;

[0106] Step S52: Map the accurate segmentation result output by the mask segmentation network on the second blank matrix;

[0107] Step S53: Perform weighted averaging on the first blank matrix and the second blank matrix, filter out areas with low confidence, and obtain a fine-grained segmentation result.

[0108] In step S5, the coarse segmentation result of the multi-module fusion encoder is fused with the refined mask output by the mask segmentation network, and this fusion process ensures the high accuracy and reliability of the defect detection result.

[0109] Step S6: Input the standard object graph as a template and perform template matching with the picture to be detected, perform difference processing on the matched area, binarize and extract the suspected defect area.

[0110] Step S6 further includes the following processing process:

[0111] Step S61: Input the standard object template graph, perform template matching on the picture to be detected, and find the detection area;

[0112] Step S62: Perform difference processing on the template graph and the detection area, perform opening operation after binarization, and obtain the mask after removing noise points;

[0113] Step S62: Connected component analysis is performed on the mask to label all connected abnormal regions in the image and assign a separate label to each region;

[0114] Step S63: The connected regions are filtered according to their shape and area to further exclude noise points.

[0115] Step S64: Contour extraction is performed on the abnormal regions to obtain coarse defect regions.

[0116] Step S6 performs template matching by inputting a standard object image as a template and the image to be detected, and performs difference processing and binarization on the matched region to extract a suspected defect region. This step uses template matching technology to improve the automation of defect detection and reduce human interference.

[0117] Step S7: The suspected defect region is fused with the fine-grained segmentation result obtained in step S5 to complete defect detection.

[0118] Step S7 further includes the following processing process:

[0119] Step S71: The fine-grained segmentation result obtained in step S5 is mapped to a third blank matrix.

[0120] Step S72: The coarse defect region and the matrix defect region are subjected to AND logic operation to obtain a defect detection result.

[0121] In step S7, the suspected defect region is fused with the segmentation result obtained in step S5 to complete defect detection. This final fusion step ensures the completeness and accuracy of defect detection, making the detection result more reliable.

[0122] Figure 2 The principle of an embodiment of the defect detection system of the present application is shown. Please refer to Figure 2 The system of the embodiment includes an input module, an image block token processing module, a mask generation module, a confidence score generation module, a fine-grained segmentation module, a suspected defect region extraction module, and a fusion module.

[0123] The input module is used to input text and images into a multi-modal fusion encoder to obtain text feature vectors and image block tokens. The specific processing of the module is the same as step S1 of the foregoing method embodiment, and will not be repeated here.

[0124] The image block token processing module is used to process image block tokens using two parallel first and second paths to extract features and locate abnormal regions, and obtain prediction result feature maps and pixel prediction results. The specific processing of the module is the same as step S2 of the foregoing method embodiment, and will not be repeated here.

[0125] The mask generation module is configured to fuse features of different scales through a residual connection and bilinear interpolation to obtain a coarse segmentation result, and perform binaryzation processing on the coarse segmentation result to generate a mask. The specific processing of the module is the same as step S3 of the method embodiment described above, and will not be repeated here.

[0126] The confidence score generation module is configured to identify a box and a point in a connected region in the mask, input the identified box and point into the mask segmentation network, and generate a refined mask and a confidence score. The specific processing of the module is the same as step S4 of the method embodiment described above, and will not be repeated here.

[0127] The fine-grained segmentation module is configured to fuse the coarse segmentation result of the multi-module fusion encoder and the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result. The specific processing of the module is the same as step S5 of the method embodiment described above, and will not be repeated here.

[0128] The suspected defect region extraction module is configured to input a standard object image as a template and perform template matching with a to-be-detected image, perform difference processing on the matched region, binaryzation, and extract a suspected defect region. The specific processing of the module is the same as step S6 of the method embodiment described above, and will not be repeated here.

[0129] The fusion module is configured to fuse the suspected defect region and the fine-grained segmentation result obtained in the fine-grained segmentation module to complete defect detection. The specific processing of the module is the same as step S7 of the method embodiment described above, and will not be repeated here.

[0130] In addition, the present application also discloses a computer system for defect detection, comprising a memory, a processor and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the defect detection method embodiment as described above.

[0131] In addition, the present application also discloses a computer readable storage medium for defect detection, which stores program instructions executable by a processor to implement the steps of the defect detection method embodiment as described above.

[0132] In addition, the present application also discloses a computer program product comprising a computer program, which is executed by a processor to implement the steps of the defect detection method embodiment as described above.

[0133] Although the above methods are illustrated and described as a series of actions for the sake of simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of actions, because according to one or more embodiments, some actions can occur in different orders and / or concurrently with other actions illustrated and described herein or not illustrated and described herein but can be understood by those skilled in the art.

[0134] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0135] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0136] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0137] In one or more exemplary embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0138] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of defect detection, characterized by, The method comprises: Step S1: inputting text and images into a multi-modal fusion encoder to obtain a text feature vector and image block tokens; Step S2: using two parallel first and second paths to process the image block tokens to extract features and locate abnormal areas, and obtaining a predicted result feature map and a pixel prediction result; Step S3: fusing features of different scales through residual connection and bilinear interpolation to obtain a coarse segmentation result, binarizing the coarse segmentation result to generate a mask; Step S4: identifying a box and a point in the connected region in the mask, inputting the identified box and point into a mask segmentation network to generate a refined mask and a confidence score; Step S5: fusing the coarse segmentation result of the multi-modal fusion encoder and the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result; Step S6: inputting a standard object image as a template to perform template matching with a to-be-detected image, performing difference processing on the matched region, binarizing and extracting a suspected defect region; Step S7: fusing the suspected defect region and the fine-grained segmentation result obtained in step S5 to complete defect detection; Step S2 further comprises: Step S21: in the processing of the first path, capturing row-level features and column-level features of the image block tokens to accurately locate abnormal areas; Step S22: in the processing of the second path, understanding global features of the image at different scales to comprehensively find abnormal areas; Step S21 further comprises: Step S211: extracting row-level features and column-level features from image features corresponding to the image block tokens through an average pooling layer; Step S212: processing the text feature vector using a convolution layer to obtain language features matching the dimension of the image features; Step S213: predicting whether the pixels in the row-level features and the column-level features are abnormal through an attention mechanism, and after detecting an abnormal point, restoring the feature map of the prediction result to the original image size using bilinear interpolation to obtain an abnormal area; Step S22 further comprises: Step S221: using average pooling layers with different kernel sizes to obtain multi-scale visual features; Step S222: for each scale, processing the text feature vector using a convolution layer to match the feature dimension; Step S223: obtaining the prediction result of the pixels in the row-level features and the column-level features through an attention mechanism to obtain an abnormal area, wherein the attention mechanism formula is: where Q, K, V are products of the image with three different trainable parameter matrices, i.e. linear transformations on the input pixels, d k is the dimension of K; Step S3 further comprises: Step S31: residual connection refers to adding the output of the shallow layer of the network to the output of one or more layers in the deep part of the network after transformation, so that the deep network extracts more types of features; Step S32: bilinear interpolation refers to, in the image pooling process, due to the reduction of the image, part of the features is eliminated, bilinear interpolation first performs linear interpolation in the x direction to obtain two intermediate values, and then performs the same operation in the y direction for image resampling; Step S33: feature fusion outputs a result to obtain a coarse segmentation result, and after setting a threshold, the image is binarized to obtain a mask.

2. The method of defect detection according to claim 1, wherein, Step S1 further comprises: Step S11: inputting text and defect images into a multi-modal fusion encoder; Step S12: The multi-modal fusion encoder processes the text and outputs a text feature vector; Step S13: When processing the defective product image, the multi-modal fusion encoder divides the defective product image into multiple small blocks, and each small block is embedded to obtain an image block token.

3. The method of defect detection according to claim 1, wherein, Step S4 further comprises: Step S41: Extract the frame and point in the mask connected region and input it to the mask segmentation network. The mask segmentation network encodes the prompt information of the image into an embedding vector in real time to predict the segmentation mask; Step S42: The mask segmentation network generates multiple possible segmentation masks, which represent the segmentation of different regions in the image, including the whole mask, the partial mask and the sub-partial mask. The feature prompts obtained by inputting the text are fused with the image features to guide the segmentation task; Step S43: Calculate a confidence score for each possible segmentation mask. The final output is obtained according to the confidence score. If there are multiple masks, apply non-maximum suppression to filter out redundant masks to obtain more accurate refined masks and ensure the accuracy of the segmentation result.

4. The method of defect detection according to claim 1, wherein, Step S5 further comprises: Step S51: Extract the coarse segmentation result output by the multi-module fusion encoder separately and map it on the first blank matrix; Step S52: Map the accurate segmentation result output by the mask segmentation network on the second blank matrix; Step S53: Perform weighted averaging on the first blank matrix and the second blank matrix, filter the areas with low confidence, and obtain the fine-grained segmentation result.

5. The method of defect detection according to claim 1, wherein, Step S6 further comprises: Step S61: Input the standard object template image and perform template matching on the detection area in the image to be detected; Step S62: Perform difference processing on the template image and the detection area, binarize and perform open operation to remove noise points to obtain a mask; Step S62: Perform connected region analysis on the mask, mark all connected abnormal regions in the image, and assign a label to each region; Step S63: According to the shape and area of the connected region, further exclude noise points; Step S64: Extract the contour of the abnormal region to obtain a coarse defect region.

6. The method of defect detection according to claim 1, wherein, Step S7 further comprises: Step S71: Map the fine-grained segmentation result obtained in step S5 to the third blank matrix; Step S72: Perform AND logic operation on the coarse defect region and the matrix defect region to obtain the defect detection result.

7. A system for defect detection, characterized by The system comprises: An input module for inputting text and images to the multi-modal fusion encoder to obtain text feature vectors and image block tokens; An image block token processing module for processing image block tokens using two parallel first and second paths to extract features and locate abnormal regions and obtain prediction result feature maps and pixel prediction results; A mask generation module for fusing features of different scales through residual connection and bilinear interpolation to obtain a coarse segmentation result, and performing binarization processing on the obtained coarse segmentation result to generate a mask; A confidence score generation module for identifying frames and points in the mask connected region and inputting the identified frames and points into the mask segmentation network to generate a refined mask and a confidence score; A confidence score generation module for identifying frames and points in the mask connected region and inputting the identified frames and points into the mask segmentation network to generate a refined mask and a confidence score; The fine-grained segmentation module is configured to fuse the coarse segmentation result of the multi-module fusion encoder and the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result. The suspected defect area extraction module is configured to input a standard object image as a template to perform template matching with a to-be-detected image, perform difference processing on a region obtained by the matching, binarize the region, and extract a suspected defect area. The fusion module is configured to fuse the suspected defect area and the fine-grained segmentation result obtained by the fine-grained segmentation module to complete defect detection. The image tile processing module further includes the following steps: In the first path, the row-level features and column-level features of the image tile are captured to accurately locate the abnormal area. In the second path, the global features of the image at different scales are understood to comprehensively find the abnormal area. The first step further includes: The row-level features and column-level features are extracted from the image features corresponding to the image tile through an average pooling layer. The language features matching the dimensions of the image features are obtained by processing the text feature vector through a convolution layer. The attention mechanism is used to predict whether the pixels in the row-level features and column-level features are abnormal, and after detecting the abnormal points, the feature map of the prediction result is restored to the original image size using bilinear interpolation to obtain the abnormal area. The second step further includes: The multi-scale visual features are obtained using average pooling layers with different kernel sizes. For each scale, the text feature vector is processed through a convolution layer to match the feature dimensions. The attention mechanism is used to obtain the prediction results of the pixels in the row-level features and column-level features to obtain the abnormal area, and the attention mechanism formula is as follows: where Q, K, V are products of the image with three different trainable parameter matrices, i.e. linear transformations on the input pixels, d k is the dimension of K; The mask generation module further includes the following steps: Residual connection refers to adding the output of the shallow layer of the network to the output of one or more deep layers of the network after transformation, so that more types of features are extracted by the deep network. Bilinear interpolation refers to linear interpolation in the x direction to obtain two intermediate values, and then the same operation is performed in the y direction for image resampling during image pooling. The feature fusion output result is obtained, and the coarse segmentation result is obtained after setting a threshold and binarizing the image to obtain a mask.

8. A computer system for defect detection, characterized by, The memory, the processor, and program instructions stored in the memory and executable by the processor are included, wherein the processor executes the program instructions to implement the steps of the defect detection method in any one of claims 1 to 6.

9. A computer readable storage medium for defect detection, the computer readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising: The memory has program instructions executable by the processor to implement the steps of the defect detection method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the defect detection method in any one of claims 1 to 6.

Images