Defect detection method and system

Abstract

The invention discloses a defect detection method and system, which improve accuracy and efficiency and reduce misjudgment and missed judgment. According to the technical scheme, a text and an image are input into a multi-modal fusion encoder, and a text feature vector and an image block token are obtained; processing the image block token by adopting two parallel paths, extracting features and positioning an abnormal region, and obtaining a prediction result feature map and a pixel prediction result; features of different scales are fused through residual connection and bilinear interpolation, a rough segmentation result is obtained, binarization processing is carried out, and a mask is generated; frames and points are recognized, the recognized frames and points are input into a mask segmentation network, and refined masks and confidence scores are generated; fusing the rough segmentation result with the refined mask to obtain a fine-grained segmentation result; inputting a standard object image as a template to perform template matching with the to-be-detected image, performing difference processing and binaryzation on a matching region, and extracting a suspected defect region; and fusing the suspected defect area with a fine-grained segmentation result.

Description

Technical Field

[0001] The present invention relates to the field of industrial automation and intelligent technology, and in particular to a defect detection method and system. Background Art

[0002] With the rapid development of industrial production, product quality control has become a key link in the manufacturing industry. The quality of products directly affects the market competitiveness and economic benefits of enterprises. In many production processes, defect detection is an important step to ensure that product quality meets standards. Traditional defect detection methods mainly rely on manual visual inspection, which has disadvantages such as low efficiency and susceptibility to human factors, resulting in instability of detection results. With the increase in product complexity, the accuracy and reliability of manual inspection are increasingly difficult to meet the needs of modern industry.

[0003] With the continuous advancement of automation and intelligent technology, it is particularly important to use advanced technical means to improve the efficiency and accuracy of defect detection. Although the existing automated inspection equipment has improved the inspection speed to a certain extent, it still has problems of misjudgment and missed judgment when facing complex and changeable product surfaces and defect types. These problems not only increase production costs, but may also cause unqualified products to enter the market, causing economic losses to enterprises and damage to brand reputation.

[0004] Although there are some automated testing equipment on the market, they still have the following problems in practical applications:

[0005] The ability to identify surface features and defect types of complex products is limited, making it difficult to adapt to changing production environments;

[0006] The generalization ability of the detection algorithm is insufficient. New defect types or changed defect characteristics need to be retrained and adjusted, which increases maintenance costs.

[0007] The equipment lacks flexibility and adaptability, making it difficult to quickly adapt to production line adjustments and new product testing needs. Summary of the invention

[0008] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0009] The purpose of the present invention is to solve the above-mentioned problems and to provide a method and system for defect detection, which can improve the accuracy and efficiency of detection, reduce misjudgments and missed judgments, and improve the efficiency and accuracy of product quality control.

[0010] The technical solution of the present invention is: the present invention discloses a defect detection method, the method comprising:

[0011] Step S1: Input text and image into the multimodal fusion encoder to obtain text feature vectors and image block tokens;

[0012] Step S2: using two parallel first paths and second paths to process the image block tokens to extract features and locate abnormal areas, and obtain a prediction result feature map and a pixel prediction result;

[0013] Step S3: features of different scales are fused through residual connection and bilinear interpolation to obtain a rough segmentation result, and the obtained rough segmentation result is binarized to generate a mask;

[0014] Step S4: Identify boxes and points in the connected area within the mask, input the identified boxes and points into the mask segmentation network, and generate a refined mask and confidence score;

[0015] Step S5: Fusing the coarse segmentation result of the multi-module fusion encoder with the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result;

[0016] Step S6: input the standard object image as a template to perform template matching with the image to be inspected, perform difference processing on the matched area, binarize and extract the suspected defect area;

[0017] Step S7: Fuse the suspected defect area with the fine-grained segmentation result obtained in step S5 to complete defect detection.

[0018] According to an embodiment of the defect detection method of the present invention, step S1 further comprises:

[0019] Step S11: inputting the text and defective product image into a multimodal fusion encoder;

[0020] Step S12: the multimodal fusion encoder processes the text and outputs a text feature vector;

[0021] Step S13: When processing the defective product image, the multimodal fusion encoder divides the defective product image into multiple small blocks, and embeds each small block to obtain an image block token.

[0022] According to an embodiment of the defect detection method of the present invention, step S2 further comprises:

[0023] Step S21: In the processing of the first path, the row-level features and column-level features of the image block tokens are captured to accurately locate the abnormal area;

[0024] Step S22: In the processing of the second path, the global features of the image at different scales are understood to comprehensively find abnormal areas.

[0025] According to an embodiment of the defect detection method of the present invention, step S21 further comprises:

[0026] Step S211: extracting row-level features and column-level features from the image features corresponding to the image block tokens through an average pooling layer;

[0027] Step S212: Processing the text feature vector using a convolutional layer to obtain language features that match the dimensions of the image features;

[0028] Step S213: Use the attention mechanism to predict whether the pixels in the row-level features and column-level features are abnormal. After detecting the abnormal points, use bilinear interpolation to restore the feature map of the predicted results to the original image size to obtain the abnormal area.

[0029] According to an embodiment of the defect detection method of the present invention, step S22 further comprises:

[0030] Step S221: using average pooling layers with different kernel sizes to obtain multi-scale visual features;

[0031] Step S222: For each scale, use a convolutional layer to process the text feature vector to match the feature dimension;

[0032] Step S223: Obtain the prediction results of the pixels in the row-level features and column-level features through the attention mechanism to obtain the abnormal area.

[0033] According to an embodiment of the defect detection method of the present invention, step S3 further comprises:

[0034] Step S31: Residual connection refers to adding the output of the shallow layer of the network directly to the output of one or more layers in the deep layer of the network after transformation, so that the deep network can extract more types of features;

[0035] Step S32: Bilinear interpolation means that in the process of image pooling, due to the reduction of the image, some features are ablated, and bilinear interpolation first performs linear interpolation in the x direction to obtain two intermediate values, and then performs the same operation in the y direction for image resampling;

[0036] Step S33: feature fusion output results are obtained to obtain a rough segmentation result, and then the image is binarized after setting a threshold to obtain a mask.

[0037] According to an embodiment of the defect detection method of the present invention, step S4 further comprises:

[0038] Step S41: extracting boxes and points in the mask connected area and inputting them into the mask segmentation network. The mask segmentation network encodes the prompt information of the image into an embedded vector in real time to predict the segmentation mask;

[0039] Step S42: The mask segmentation network generates multiple possible segmentation masks, which represent the segmentation conditions of different regions in the image, including overall masks, partial masks, and sub-partial masks. Feature hints obtained by inputting text are fused with image features to guide the segmentation task.

[0040] Step S43: Calculate a confidence score for each possible segmentation mask, and obtain the final output based on the confidence score. If there are multiple masks, apply non-maximum suppression to filter out repeated or redundant masks to obtain a more accurate refined mask to ensure the accuracy of the segmentation result.

[0041] According to an embodiment of the defect detection method of the present invention, step S5 further comprises:

[0042] Step S51: extracting the rough segmentation result output by the multi-module fusion encoder separately and mapping it on the first blank matrix;

[0043] Step S52: Mapping the precise segmentation result output by the mask segmentation network onto the second blank matrix;

[0044] Step S53: taking a weighted average of the first blank matrix and the second blank matrix, filtering the areas with low confidence, and obtaining a fine-grained segmentation result.

[0045] According to an embodiment of the defect detection method of the present invention, step S6 further comprises:

[0046] Step S61: input a standard object template image, perform template matching on the image to be detected, and search for the detection area;

[0047] Step S62: performing difference processing on the template image and the detection area, performing an opening operation after binarization, and obtaining a mask after removing noise points;

[0048] Step S62: Performing connected region analysis on the mask, marking all connected abnormal regions in the image, and assigning a label to each region separately;

[0049] Step S63: Screening based on the shape and area of ​​the connected regions to further eliminate noise points;

[0050] Step S64: extract the contour of the abnormal area to obtain a rough defect area.

[0051] According to an embodiment of the defect detection method of the present invention, step S7 further comprises:

[0052] Step S71: mapping the fine-grained segmentation result obtained in step S5 onto a third blank matrix;

[0053] Step S72: Perform a logical AND operation on the rough defect area and the matrix defect area to obtain a defect detection result.

[0054] The present invention also discloses a defect detection system, the system comprising:

[0055] Input module, used to input text and images into the multimodal fusion encoder to obtain text feature vectors and image block tokens;

[0056] An image block token processing module, used to process the image block tokens using two parallel first paths and second paths to extract features and locate abnormal areas, and obtain a prediction result feature map and a pixel prediction result;

[0057] The mask generation module is used to fuse features of different scales through residual connection and bilinear interpolation to obtain a rough segmentation result, and to perform binarization processing on the obtained rough segmentation result to generate a mask;

[0058] A confidence score generation module is used to identify boxes and points in the connected area within the mask, input the identified boxes and points into the mask segmentation network, and generate a refined mask and confidence score;

[0059] The fine-grained segmentation module is used to fuse the coarse segmentation result of the multi-module fusion encoder with the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result;

[0060] The suspected defect area extraction module is used to input the standard object image as a template to perform template matching with the image to be tested, perform difference processing on the matched area, binarize and extract the suspected defect area;

[0061] The fusion module is used to fuse the suspected defect area with the fine-grained segmentation result obtained in the fine-grained segmentation module to complete defect detection.

[0062] The present invention also discloses a computer system for defect detection, comprising a memory, a processor and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the defect detection method described above.

[0063] The present invention also discloses a computer-readable storage medium for defect detection, which stores program instructions executable by a processor to implement the steps of the defect detection method as described above.

[0064] The present invention also discloses a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the defect detection method described above are implemented.

[0065] Compared with the prior art, the present invention has the following beneficial effects: the present invention has better generalization for detection, and solves the problem that the traditional algorithm is more sensitive to external conditions such as ambient light. In addition, compared with the traditional supervised target detection algorithm or classification algorithm, the algorithm structure of the present method can perform more accurate effect splitting in an unsupervised situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0067] Figure 1 A flow chart showing an embodiment of a defect detection method of the present invention.

[0068] Figure 2 A schematic diagram showing an embodiment of a defect detection system of the present invention. DETAILED DESCRIPTION

[0069] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are only exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0070] Figure 1 The flow chart of an embodiment of the defect detection method of the present invention is shown. Figure 1 , the implementation steps of the method of this embodiment are described in detail as follows.

[0071] Step S1: Input text and image into the multimodal fusion encoder to obtain text feature vector and image block token.

[0072] Step S1 further includes the following processing steps:

[0073] Step S11: inputting the text and defective product image into a multimodal fusion encoder;

[0074] Step S12: the multimodal fusion encoder processes the text and outputs a text feature vector;

[0075] Step S13: When processing the defective product image, the multimodal fusion encoder divides the defective product image into multiple small patches (patches), and embeds each small patch to obtain an image block token.

[0076] Through step S1, the present invention uses a multimodal fusion encoder to simultaneously process the input text and image data, and can effectively extract text feature vectors and image block tokens. This multimodal feature fusion technology improves the accuracy of defect detection because it can simultaneously consider text descriptions and image content, making the detection results more comprehensive and accurate.

[0077] Step S2: Use two parallel paths (a first path and a second path running in parallel) to process the image block token to extract features and locate abnormal areas, and obtain a prediction result feature map and a pixel prediction result.

[0078] Step S2 further includes the following processing steps:

[0079] Step S21: i.e. the first path, capturing the row-level features and column-level features of the image block Token to accurately locate the abnormal area;

[0080] Step S22: i.e. the second path, understanding the global features of images at different scales to comprehensively find abnormal areas.

[0081] The above step S21 further includes the following processing steps:

[0082] Step S211: extracting row-level features and column-level features from the image features corresponding to the image block Token through an average pooling layer;

[0083] Step S212: Processing the text feature vector using a convolutional layer to obtain language features that match the dimensions of the image features;

[0084] Step S213: Use the attention mechanism to predict whether the pixels in the row-level features and column-level features are abnormal. After detecting the abnormal points, use bilinear interpolation to restore the feature map of the predicted results to the original image size to obtain the abnormal area.

[0085] The above step S22 further includes the following processing steps:

[0086] Step S221: using average pooling layers with different kernel sizes to obtain multi-scale visual features;

[0087] Step S222: For each scale, use a convolutional layer to process the text feature vector to match the feature dimension;

[0088] Step S223: Obtain the prediction results of the pixels in the row-level features and column-level features through the attention mechanism to obtain the abnormal area. The attention mechanism formula is:

[0089]

[0090] Among them, Q, K, and V are the products of the image and three different trainable parameter matrices, that is, linear transformation of the input pixels, d k is the dimension of K.

[0091] The two parallel path designs in step S2 improve processing efficiency and can quickly locate potential abnormal areas, laying a solid foundation for subsequent defect detection.

[0092] Step S3: Features of different scales are fused through residual connection and bilinear interpolation to obtain a rough segmentation result, and the obtained rough segmentation result is binarized to generate a mask.

[0093] Step S31: Residual connection refers to adding the output of the shallow layer of the network directly to the output of one or more layers in the deep layer of the network after transformation, so that the deep network can extract more types of features;

[0094] Step S32: Bilinear interpolation means that in the process of image pooling, due to the reduction of the image, some features are ablated, and bilinear interpolation first performs linear interpolation in the x direction to obtain two intermediate values, and then performs the same operation in the y direction for image resampling;

[0095] Step S33: feature fusion output results are obtained to obtain a rough segmentation result, and then the image is binarized after setting a threshold to obtain a mask.

[0096] In step S3, the residual connection and bilinear interpolation technology are used to effectively integrate features of different scales, enhance the model's ability to recognize defects of different sizes and shapes, and generate masks through binarization processing, which provides a basis for subsequent precise segmentation.

[0097] Step S4: Identify boxes and points in the connected area within the mask, input the identified boxes and points into the mask segmentation network, and generate a refined mask and confidence score.

[0098] Step S4 further includes the following processing steps:

[0099] Step S41: extract the boxes and points in the mask connected area and input them into the mask segmentation network. The mask segmentation network encodes the image prompt information, such as background points, masks, etc., into an embedded vector in real time to predict the segmentation mask.

[0100] Step S42: The mask segmentation network generates multiple possible segmentation masks, which represent the segmentation of different regions in the image, including overall masks, partial masks, and sub-partial masks. Feature hints are obtained by inputting text and fused with image features to guide the segmentation task. The fusion formula is F = αF text +(1-α)F image, where F is the fused feature and α is the weight coefficient.

[0101] Step S43: Calculate a confidence score for each possible segmentation mask, and obtain the final output based on the confidence score. If there are multiple masks, apply non-maximum suppression to filter out repeated or redundant masks to obtain a more accurate refined mask to ensure the accuracy of the segmentation result.

[0102] Step S4 generates a refined mask and confidence score by identifying boxes and points in the connected regions within the mask and inputting them into the mask segmentation network. This step further improves the accuracy of defect detection and provides a confidence score for each detected defect, which helps with subsequent decision-making and processing.

[0103] Step S5: Fuse the coarse segmentation result of the multi-module fusion encoder with the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result.

[0104] Step S5 further includes the following processing steps:

[0105] Step S51: extracting the rough segmentation result output by the multi-module fusion encoder separately and mapping it on the first blank matrix;

[0106] Step S52: Mapping the precise segmentation result output by the mask segmentation network onto the second blank matrix;

[0107] Step S53: taking a weighted average of the first blank matrix and the second blank matrix, filtering the areas with low confidence, and obtaining a fine-grained segmentation result.

[0108] In step S5, the coarse segmentation result of the multi-module fusion encoder is fused with the refined mask output by the mask segmentation network. This fusion process ensures the high accuracy and reliability of the defect detection results.

[0109] Step S6: input the standard object image as a template to perform template matching with the image to be inspected, perform difference processing on the matched area, binarize it and extract the suspected defect area.

[0110] Step S6 further includes the following processing steps:

[0111] Step S61: input a standard object template image, perform template matching on the image to be detected, and search for the detection area;

[0112] Step S62: performing difference processing on the template image and the detection area, performing an opening operation after binarization, and obtaining a mask after removing noise points;

[0113] Step S62: Performing connected region analysis on the mask, marking all connected abnormal regions in the image, and assigning a label to each region separately;

[0114] Step S63: Screening based on the shape and area of ​​the connected regions to further eliminate noise points;

[0115] Step S64: extract the contour of the abnormal area to obtain a rough defect area.

[0116] Step S6 performs template matching with the image to be inspected by inputting the standard object image as a template, and performs difference processing and binarization on the matched area to extract the suspected defect area. This step uses template matching technology to improve the automation of defect detection and reduce the interference of human factors.

[0117] Step S7: Fuse the suspected defect area with the fine-grained segmentation result obtained in step S5 to complete defect detection.

[0118] Step S7 further includes the following processing steps:

[0119] Step S71: mapping the fine-grained segmentation result obtained in step S5 onto a third blank matrix;

[0120] Step S72: Perform a logical AND operation on the rough defect area and the matrix defect area to obtain a defect detection result.

[0121] In step S7, the suspected defect area is fused with the segmentation result obtained in step S5 to complete the defect detection. This final fusion step ensures the integrity and accuracy of the defect detection, making the detection result more reliable.

[0122] Figure 2 The principle of an embodiment of the defect detection system of the present invention is shown. Figure 2 ,The system of this embodiment includes: an input module, an image block token processing module, a mask generation module, a confidence score generation module, a fine-grained segmentation module, a suspected defect area extraction module, and a fusion module.

[0123] The input module is used to input text and images into the multimodal fusion encoder to obtain text feature vectors and image block tokens. The specific processing of the module is the same as step S1 of the aforementioned method embodiment, which will not be repeated here.

[0124] The image block token processing module is used to process the image block tokens using two parallel first paths and second paths to extract features and locate abnormal areas, and obtain a prediction result feature map and a pixel prediction result. The specific processing of the module is the same as step S2 of the aforementioned method embodiment, and will not be repeated here.

[0125] The mask generation module is used to fuse features of different scales through residual connection and bilinear interpolation to obtain a rough segmentation result, binarize the obtained rough segmentation result, and generate a mask. The specific processing of the module is the same as step S3 of the aforementioned method embodiment, and will not be repeated here.

[0126] The confidence score generation module is used to identify boxes and points in the connected area in the mask, input the identified boxes and points into the mask segmentation network, and generate a refined mask and a confidence score. The specific processing of the module is the same as step S4 of the aforementioned method embodiment, and will not be repeated here.

[0127] The fine-grained segmentation module is used to fuse the coarse segmentation result of the multi-module fusion encoder with the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result. The specific processing of the module is the same as step S5 of the aforementioned method embodiment, which will not be repeated here.

[0128] The suspected defect area extraction module is used to input the standard object image as a template to perform template matching with the image to be detected, perform difference processing on the matched area, binarize and extract the suspected defect area. The specific processing of the module is the same as step S6 of the aforementioned method embodiment, and will not be repeated here.

[0129] The fusion module is used to fuse the suspected defect area with the fine-grained segmentation result obtained in the fine-grained segmentation module to complete defect detection. The specific processing of the module is the same as step S7 of the aforementioned method embodiment, and will not be repeated here.

[0130] In addition, the present invention also discloses a computer system for defect detection, including a memory, a processor, and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the defect detection method embodiment described above.

[0131] In addition, the present invention also discloses a computer-readable storage medium for defect detection, which stores program instructions executable by a processor to implement the steps of the defect detection method embodiment described above.

[0132] In addition, the present invention also discloses a computer program product, including a computer program, which implements the steps of the defect detection method embodiment described above when the computer program is executed by a processor.

[0133] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0134] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0135] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0136] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0137] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0138] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the universal principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A defect detection method, characterized in that: Methods include: Step S1: Input text and image into the multimodal fusion encoder to obtain text feature vectors and image block tokens; Step S2: using two parallel first paths and second paths to process the image block tokens to extract features and locate abnormal areas, and obtain a prediction result feature map and a pixel prediction result; Step S3: features of different scales are fused through residual connection and bilinear interpolation to obtain a rough segmentation result, and the obtained rough segmentation result is binarized to generate a mask; Step S4: Identify boxes and points in the connected area within the mask, input the identified boxes and points into the mask segmentation network, and generate a refined mask and confidence score; Step S5: Fusing the coarse segmentation result of the multi-module fusion encoder with the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result; Step S6: input the standard object image as a template to perform template matching with the image to be inspected, perform difference processing on the matched area, binarize and extract the suspected defect area; Step S7: Fuse the suspected defect area with the fine-grained segmentation result obtained in step S5 to complete defect detection.

2. The defect detection method according to claim 1, characterized in that: Step S1 further comprises: Step S11: inputting the text and defective product image into a multimodal fusion encoder; Step S12: the multimodal fusion encoder processes the text and outputs a text feature vector; Step S13: When processing the defective product image, the multimodal fusion encoder divides the defective product image into multiple small blocks, and embeds each small block to obtain an image block token.

3. The defect detection method according to claim 1, characterized in that: Step S2 further comprises: Step S21: In the processing of the first path, the row-level features and column-level features of the image block tokens are captured to accurately locate the abnormal area; Step S22: In the processing of the second path, the global features of the image at different scales are understood to comprehensively find abnormal areas.

4. The defect detection method according to claim 3, characterized in that: Step S21 further includes: Step S211: extracting row-level features and column-level features from the image features corresponding to the image block tokens through an average pooling layer; Step S212: Processing the text feature vector using a convolutional layer to obtain language features that match the dimensions of the image features; Step S213: Use the attention mechanism to predict whether the pixels in the row-level features and column-level features are abnormal. After detecting the abnormal points, use bilinear interpolation to restore the feature map of the predicted results to the original image size to obtain the abnormal area.

5. The defect detection method according to claim 4, characterized in that: Step S22 further includes: Step S221: using average pooling layers with different kernel sizes to obtain multi-scale visual features; Step S222: For each scale, use a convolutional layer to process the text feature vector to match the feature dimension; Step S223: Obtain the prediction results of the pixels in the row-level features and column-level features through the attention mechanism to obtain the abnormal area.

6. The defect detection method according to claim 1, characterized in that: Step S3 further comprises: Step S31: Residual connection refers to adding the output of the shallow layer of the network directly to the output of one or more layers in the deep layer of the network after transformation, so that the deep network can extract more types of features; Step S32: Bilinear interpolation means that in the process of image pooling, due to the reduction of the image, some features are ablated, and bilinear interpolation first performs linear interpolation in the x direction to obtain two intermediate values, and then performs the same operation in the y direction for image resampling; Step S33: feature fusion output results are obtained to obtain a rough segmentation result, and then the image is binarized after setting a threshold to obtain a mask.

7. The defect detection method according to claim 1, characterized in that: Step S4 further comprises: Step S41: extracting boxes and points in the mask connected area and inputting them into the mask segmentation network. The mask segmentation network encodes the prompt information of the image into an embedded vector in real time to predict the segmentation mask; Step S42: The mask segmentation network generates multiple possible segmentation masks, which represent the segmentation conditions of different regions in the image, including overall masks, partial masks, and sub-partial masks. Feature hints obtained by inputting text are fused with image features to guide the segmentation task. Step S43: Calculate a confidence score for each possible segmentation mask, and obtain the final output based on the confidence score. If there are multiple masks, apply non-maximum suppression to filter out repeated or redundant masks to obtain a more accurate refined mask to ensure the accuracy of the segmentation result.

8. The defect detection method according to claim 1, characterized in that: Step S5 further comprises: Step S51: extracting the rough segmentation result output by the multi-module fusion encoder separately and mapping it on the first blank matrix; Step S52: Mapping the precise segmentation result output by the mask segmentation network onto the second blank matrix; Step S53: taking a weighted average of the first blank matrix and the second blank matrix, filtering the areas with low confidence, and obtaining a fine-grained segmentation result.

9. The defect detection method according to claim 1, characterized in that: Step S6 further comprises: Step S61: input a standard object template image, perform template matching on the image to be detected, and search for the detection area; Step S62: performing difference processing on the template image and the detection area, performing an opening operation after binarization, and obtaining a mask after removing noise points; Step S62: Performing connected region analysis on the mask, marking all connected abnormal regions in the image, and assigning a label to each region separately; Step S63: Screening based on the shape and area of ​​the connected regions to further eliminate noise points; Step S64: extract the contour of the abnormal area to obtain a rough defect area.

10. The defect detection method according to claim 1, characterized in that: Step S7 further comprises: Step S71: mapping the fine-grained segmentation result obtained in step S5 onto a third blank matrix; Step S72: Perform a logical AND operation on the rough defect area and the matrix defect area to obtain a defect detection result.

11. A defect detection system, characterized in that: The system includes: Input module, used to input text and images into the multimodal fusion encoder to obtain text feature vectors and image block tokens; An image block token processing module, used to process the image block tokens using two parallel first paths and second paths to extract features and locate abnormal areas, and obtain a prediction result feature map and a pixel prediction result; The mask generation module is used to fuse features of different scales through residual connection and bilinear interpolation to obtain a rough segmentation result, and to perform binarization processing on the obtained rough segmentation result to generate a mask; A confidence score generation module is used to identify boxes and points in the connected area within the mask, input the identified boxes and points into the mask segmentation network, and generate a refined mask and confidence score; The fine-grained segmentation module is used to fuse the coarse segmentation result of the multi-module fusion encoder with the refined mask output by the mask segmentation network to obtain a further fine-grained segmentation result; The suspected defect area extraction module is used to input the standard object image as a template to perform template matching with the image to be tested, perform difference processing on the matched area, binarize and extract the suspected defect area; The fusion module is used to fuse the suspected defect area with the fine-grained segmentation result obtained in the fine-grained segmentation module to complete defect detection.

12. A computer system for defect detection, characterized in that: The method comprises a memory, a processor and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the defect detection method according to any one of claims 1 to 10.

13. A computer-readable storage medium for defect detection, characterized in that: It stores program instructions executable by a processor to implement the steps of the defect detection method as claimed in any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the defect detection method according to any one of claims 1 to 10 are implemented.

Images