Defect detection method and apparatus, computer device, and storage medium

Abstract

Embodiments of the present application relate to the technical field of defect detection, and provide a defect detection method and apparatus, a computer device, and a storage medium. The method comprises: acquiring training sample data; performing extraction on a sample defect image on the basis of a defect region extraction model to obtain a sample defect positioning image; performing detection on the sample defect positioning image on the basis of an image defect detection model to obtain a sample detection result; calculating a first loss value on the basis of sample defect mask data and predicted defect mask data, and calculating a second loss value on the basis of a sample defect category, a predicted defect category, a sample bounding box and a predicted bounding box, so as to determine a model loss value; updating model parameters of the defect region extraction model and the image defect detection model on the basis of the model loss value, so as to construct a target defect detection model; and performing detection on a target detection image on the basis of the target defect detection model to obtain a target detection result. The embodiments of the present application can improve the defect detection efficiency and accuracy of objects to be detected.

Description

Defect detection methods and devices, computer equipment, storage media Technical Field

[0001] This application relates to the field of defect detection technology, and in particular to a defect detection method and apparatus, computer equipment, and storage medium. Background Technology

[0002] Defect detection refers to the technology of detecting defects on the surface of an object under test. By acquiring images of the object under test and performing defect detection on these images, defects can be located and repaired in a timely manner, ensuring the object's usability in practical applications. For example, in a nuclear reactor, the control rod guide pin is a critical component, primarily used to control the reactor's reaction rate and maintain its safety. The control rod guide pin needs to be inserted into its corresponding pin hole, and secured by bolts or wheels on its forked portion to ensure the accuracy and stability of the mechanical components. Therefore, defect detection on images acquired of the pin holes corresponding to the control rod guide pins can ensure the quality of the control rod insertion into the guide tube during operation, thereby improving the reliability and safety of the nuclear power plant.

[0003] Currently, the relevant technologies for detecting defects in the pin holes corresponding to the cotter pins of the control rod guide cylinder usually rely on manual operation. This involves manually judging the images of the pin holes corresponding to the cotter pins of the control rod guide cylinder. This defect detection method is inefficient and has low accuracy. Summary of the Invention

[0004] The main objective of this application is to provide a defect detection method, apparatus, computer equipment, and storage medium that can improve the efficiency and accuracy of defect detection of the object under test.

[0005] To achieve the above objectives, a first aspect of this application provides a defect detection method, the method comprising:

[0006] Acquire training sample data, which includes a sample defect image of the sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; wherein, the sample bounding box is used to indicate the region information of the defect location of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect location of the sample object to be detected in the sample bounding box.

[0007] Based on the defect region extraction model, the defect region of the sample defect image is extracted to obtain the sample defect localization image;

[0008] The sample defect location image is subjected to defect detection based on the image defect detection model to obtain sample detection results; wherein, the sample detection results include predicted bounding boxes, predicted defect categories and predicted defect mask data;

[0009] A first loss value is calculated based on the sample defect mask data and the predicted defect mask data. A second loss value is calculated based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box. A model loss value is determined based on the first loss value and the second loss value.

[0010] The model parameters of the defect region extraction model and the image defect detection model are updated based on the model loss value, and a target defect detection model is constructed based on the updated defect region extraction model and the image defect detection model.

[0011] A target detection image is acquired, and defect detection is performed on the target detection image based on the target defect detection model to obtain a target detection result; wherein, the target detection result includes target bounding box, target defect category, and target defect mask data.

[0012] In some embodiments, the defect region extraction of the sample defect image based on the defect region extraction model to obtain a sample defect localization image includes:

[0013] Based on the defect region extraction model, the defect region of the sample defect image is extracted to obtain the first sample localization bounding box and the first sample localization confidence.

[0014] Based on the first sample localization bounding box and the first sample localization confidence, the sample defect image is extracted to obtain the sample defect localization image.

[0015] In some embodiments, the step of extracting the sample defect image based on the first sample localization bounding box and the first sample localization confidence to obtain the sample defect localization image includes:

[0016] If the location confidence of the first sample is less than a preset confidence threshold, multi-scale feature fusion is performed on the defective image of the sample based on a pre-trained multi-scale feature fusion model to obtain a sample fused feature image.

[0017] Based on the defect region extraction model, defect regions are extracted from the sample fusion feature image to obtain the second sample localization bounding box and the second sample localization confidence.

[0018] Based on the second sample localization bounding box and the second sample localization confidence, the sample defect image is extracted to obtain the sample defect localization image.

[0019] In some embodiments, determining the model loss value based on the first loss value and the second loss value includes:

[0020] Obtain the instance segmentation task and defect detection accuracy value of the sample object to be detected;

[0021] The sample weights are determined based on the instance segmentation task and the defect detection accuracy value, and the sample weights include a first weight corresponding to the first loss value and a second weight corresponding to the second loss value.

[0022] The model loss value is obtained by weighting the first weight, the first loss value, the second weight, and the second loss value.

[0023] In some embodiments, the image defect detection model includes an image feature extraction layer, a multi-scale feature extraction layer, a region detection layer, a spatial feature extraction layer, and a classification layer. The step of performing defect detection on the sample defect localization image based on the image defect detection model to obtain sample detection results includes:

[0024] Based on the image feature extraction layer, feature extraction is performed on the sample defect location image to obtain the initial features of the sample;

[0025] Based on the multi-scale feature extraction layer, multi-scale feature extraction is performed on the initial features of the sample to obtain multiple candidate features of the sample in different dimensions;

[0026] Based on the region detection layer, the initial features of the sample are used to detect regional targets, resulting in multiple candidate region bounding boxes for the sample.

[0027] Based on the spatial feature extraction layer, spatial features are extracted from the sample candidate features of multiple dimensions and the bounding boxes of multiple sample candidate regions to obtain sample spatial features;

[0028] Based on the classification layer, defect detection is performed on the spatial features of the sample to obtain the sample detection results.

[0029] In some embodiments, the defect detection based on the classification layer of the sample space features to obtain sample detection results includes:

[0030] The mask prediction branch of the classification layer performs pixel-level prediction on the sample space features to obtain the predicted defect mask data; the mask prediction branch is a structure built based on lightweight depthwise separable convolution optimization.

[0031] Based on the category prediction branch of the classification layer, the category of the sample space features is predicted to obtain the predicted defect category;

[0032] The bounding box prediction branch of the classification layer predicts bounding boxes based on the sample space features to obtain the predicted bounding boxes.

[0033] In some embodiments, the sample detection result further includes a confidence prediction value, wherein the calculation of a second loss value based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box includes:

[0034] Based on the initial features of the sample, prior boxes are extracted to obtain multiple candidate prior boxes;

[0035] The confidence loss is calculated based on the preset defect category, multiple candidate prior boxes, and the confidence prediction value to obtain the first sub-loss value;

[0036] Based on the predicted bounding box, the sample bounding box, and multiple candidate prior boxes, the location loss is calculated to obtain the second sub-loss value;

[0037] Based on the predicted defect category and the sample defect category, a category loss is calculated to obtain a third sub-loss value;

[0038] The second loss value is obtained by weighting the first sub-loss value, the second sub-loss value, and the third sub-loss value.

[0039] To achieve the above objectives, a second aspect of this application provides a defect detection device, the device comprising:

[0040] The acquisition module is used to acquire training sample data, which includes a sample defect image of the sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; the sample bounding box is used to indicate the region information of the defect location of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect location of the sample object to be detected in the sample bounding box.

[0041] The region extraction module is used to extract the defect region from the sample defect image based on the defect region extraction model to obtain the sample defect localization image.

[0042] The first detection module is used to perform defect detection on the sample defect localization image based on the image defect detection model to obtain sample detection results, which include predicted bounding boxes, predicted defect categories and predicted defect mask data.

[0043] The loss calculation module is used to calculate a first loss value based on the sample defect mask data and the predicted defect mask data, calculate a second loss value based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and determine the model loss value based on the first loss value and the second loss value.

[0044] The update module is used to update the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and to construct a target defect detection model based on the updated defect region extraction model and the image defect detection model.

[0045] The second detection module is used to acquire the target detection image and perform defect detection on the target detection image based on the target defect detection model to obtain the target detection result, which includes the target bounding box, the target defect category and the target defect mask data.

[0046] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in any one of the embodiments of the first aspect above.

[0047] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the embodiments of the first aspect.

[0048] The defect detection method, apparatus, computer device, and storage medium proposed in this application can construct a target defect detection model with significantly improved detection efficiency and accuracy through joint training of a defect region extraction model and an image defect detection model. Specifically, training sample data is first acquired, which includes a sample defect image of the sample object to be detected, the sample defect category, the sample bounding box, and the sample defect mask data. The sample bounding box indicates the region information of the defect location of the sample object in the sample defect image, and the sample defect mask data indicates the pixel boundary of the defect location of the sample object within the sample bounding box. Then, defect region extraction is performed on the sample defect image based on the defect region extraction model to obtain a sample defect localization image; defect detection is then performed on the sample defect localization image based on the image defect detection model to obtain a sample detection result. This sample detection result includes a predicted bounding box, a predicted defect category, and predicted defect mask data. Furthermore, a first loss value is calculated based on sample defect mask data and predicted defect mask data, and a second loss value is calculated based on sample defect category, predicted defect category, sample bounding box, and predicted bounding box. The model loss value is then determined based on the first and second loss values. Finally, the model parameters of the defect region extraction model and image defect detection model can be updated based on the model loss value, and a target defect detection model is constructed based on the updated defect region extraction model and image defect detection model. In practical applications, the object to be tested in this application can be the interior of the pin hole corresponding to the cotter pin of the control rod guide cylinder; that is, the acquired target detection image is the image of the inner wall of the pin hole. Defect detection is performed on the target detection image based on the target defect detection model to obtain the target detection result, which includes the target bounding box, target defect category, and target defect mask data, to accurately and efficiently detect scratch defects on the inner wall of the pin hole. Thus, this application can effectively improve the efficiency and accuracy of defect detection for the object to be tested. Attached Figure Description

[0049] Figure 1 is a flowchart of a defect detection method provided in an embodiment of this application;

[0050] Figure 2 is a schematic diagram of a control system for defect detection of the inner wall of the pin hole in an embodiment of this application.

[0051] Figure 3 is a schematic diagram of a host computer software scheme for defect detection of the inner wall of the pin hole according to an embodiment of this application.

[0052] Figure 4 is a flowchart of step S120 in Figure 1;

[0053] Figure 5 is a flowchart of step S420 in Figure 4;

[0054] Figure 6 is a flowchart of a method for defect detection of the inner wall of a pin hole provided in an embodiment of this application;

[0055] Figure 7 is a flowchart of step S130 in Figure 1;

[0056] Figure 8 is a schematic diagram of a model structure of the image defect detection model provided in an embodiment of this application;

[0057] Figure 9A is a schematic diagram of a standard convolution structure;

[0058] Figure 9B is a schematic diagram of a convolutional structure for depthwise convolution;

[0059] Figure 9C is a schematic diagram of a convolutional structure with depthwise separable convolution;

[0060] Figure 10A is a schematic diagram of a model structure for a standard convolution process;

[0061] Figure 10B is a schematic diagram of a model structure for a depthwise separable convolution process;

[0062] Figure 11 is a schematic diagram of an improved multi-scale feature extraction layer provided in an embodiment of this application;

[0063] Figure 12 is a flowchart of step S750 in Figure 7;

[0064] Figure 13 is a specific flowchart of calculating the second loss value provided in an embodiment of this application;

[0065] Figure 14 is a specific flowchart of determining the model loss value provided in an embodiment of this application;

[0066] Figure 15 is a structural schematic diagram of a defect detection device provided in an embodiment of this application;

[0067] Figure 16 is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., used in the specification, claims, and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0070] First, let's analyze some of the terms used in this application:

[0071] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. AI is a branch of computer science that attempts to understand the nature of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0072] A mask is a technique used to label or filter data. In neural networks, masks can be used to indicate which input data should be considered and which should be ignored. This can be used for tasks such as processing sequential data and padding data.

[0073] A binary mask is a precise mask composed of binary bits, or a pixel-level mask used in image annotation to identify specific objects or regions in an image. These masks can be used to train models to detect and segment specific target instances, such as identifying and locating an object in an image.

[0074] An instance refers to a single object instance in an image or video, including its precise pixel boundaries. The goal of instance segmentation is to predict the exact pixel boundaries of each individual object in an image, thereby enabling pixel-level classification and recognition of each object.

[0075] Prior bounding boxes are a set of pre-defined boxes on the feature map used for object detection. They represent the possible locations and sizes of objects in the image. Therefore, by pre-defining a set of boxes of different sizes and proportions on the feature map, the accuracy of the model in detecting objects can be improved.

[0076] Defect detection refers to the technology of detecting defects on the surface of an object under test. By acquiring images of the object under test and performing defect detection on these images, defects can be repaired promptly, ensuring the object's usability in practical applications. For example, in a nuclear reactor, the cotter pin of the control rod guide tube is a critical component, primarily used to control the reactor's reaction rate and maintain its safety. When the control rod is inserted into the guide tube, the cotter pin is released and slides into the tube. This process opens the bottom of the guide tube, allowing the control rod to be inserted into the reactor core and its position adjusted as needed. When it is necessary to slow down the reaction rate in the reactor, the control rod is inserted deeper to absorb neutrons and slow the reaction. Conversely, when it is necessary to speed up the reaction rate, the control rod is raised to reduce the number of neutrons absorbed. The cotter pin's function is to ensure that the control rod is correctly inserted into the guide tube, and the guide tube can only be opened when the control rod is fully inserted. This prevents the control rod from accidentally slipping out of the guide tube, which could lead to reactor runaway. The cotter pins of the control rod guide tubes need to be inserted into the corresponding pin holes, and secured by bolts or wheels on the forked portion to ensure the accuracy and stability of the mechanical components. Therefore, in the field of nuclear power equipment maintenance, in-service inspection of the control rod guide tube cotter pins—that is, defect detection of images captured of the pin holes corresponding to the control rod guide tube cotter pins—allows for real-time monitoring of their quality status during operation. This ensures the quality of the control rods during insertion into the guide tubes, thereby improving the reliability and safety of the nuclear power plant.

[0077] The target pin hole is located inside the upper core plate, which is part of the upper in-core component assembly and is stored on a dedicated stainless steel storage rack in the component pool during overhauls. The shape and dimensions of the component pool and storage rack are known. The bottom and inner walls of the component pool are coated with stainless steel, for example, with a surface roughness of approximately Ra6.3 (meaning a roughness of 6.3 micrometers). The storage rack is supported by four 210 mm diameter columns, and there are a total of 122 pin holes. Therefore, the target pin hole needs to be located. The upper core plate is 76 mm thick, and the pin hole diameter is 21 mm. There should be no obstructions inside the pin hole during inspection, requiring inspection to cover the entire depth, with a focus on inspecting the upper 41 mm area for wear, scratches, burrs, etc. Currently, the defect detection method for the pin holes corresponding to the cotter pins of the control rod guide tubes typically uses manual operation, i.e., manually judging the images acquired of the pin holes corresponding to the cotter pins of the control rod guide tubes. This defect detection method is inefficient and has low accuracy.

[0078] Based on this, embodiments of this application provide a defect detection method and apparatus, computer equipment, and storage medium, which can improve the efficiency and accuracy of defect detection of the object under test, and further improve the efficiency and accuracy of visual detection of defects on the inner wall surface of the cotter pin hole of the control rod guide cylinder.

[0079] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0080] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0081] The defect detection method provided in this application relates to the field of artificial intelligence technology. The defect detection method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing the defect detection method, but is not limited to the above forms.

[0082] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0083] Please refer to Figure 1, which is an optional flowchart of a defect detection method provided in an embodiment of this application. In some embodiments of this application, the method in Figure 1 may specifically include, but is not limited to, steps S110 to S160. These six steps will be described in detail below with reference to Figure 1.

[0084] Step S110: Obtain training sample data;

[0085] Step S120: Extract the defect region from the sample defect image based on the defect region extraction model to obtain the sample defect localization image;

[0086] Step S130: Perform defect detection on the sample defect localization image based on the image defect detection model to obtain the sample detection result;

[0087] Step S140: Calculate a first loss value based on sample defect mask data and predicted defect mask data; calculate a second loss value based on sample defect category, predicted defect category, sample bounding box, and predicted bounding box; and determine the model loss value based on the first loss value and the second loss value.

[0088] Step S150: Update the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and construct the target defect detection model based on the updated defect region extraction model and the image defect detection model.

[0089] Step S160: Obtain the target detection image and perform defect detection on the target detection image based on the target defect detection model to obtain the target detection result.

[0090] In step S110 of some embodiments, when constructing the target defect detection model, a sample dataset can be obtained first. This sample dataset includes multiple training sample data. The training sample data includes a sample defect image of the object to be detected, the sample defect category, the sample bounding box, and sample defect mask data. The sample bounding box indicates the region information of the defect location of the object to be detected in the sample defect image, and the sample defect mask data indicates the pixel boundary of the defect location of the object to be detected within the sample bounding box.

[0091] It should be noted that the defect images of the sample object to be inspected can be acquired using image acquisition equipment. For example, if the sample object to be inspected is a pin hole, when acquiring images of the inner wall of the pin hole, the visual guidance of the video inspection probe can be used. By adjusting the position of the two-dimensional positioning module that extends into the pin hole, the axis of the video inspection probe is aligned with the axis of the pin hole. Then, the probe is stepped into the pin hole to obtain an image of the inner wall of the pin hole, thus obtaining the initial defect image of the sample.

[0092] Furthermore, after obtaining multiple initial defect images of samples, this application can preprocess these images to obtain a final sample defect image. Specifically, this application can perform image data enhancement on the multiple initial defect images of samples, that is, using the axis of the video detection probe as a central reference, identifying the outer edge of the pin hole to be detected and creating a calibration circle, and selecting a second calibration circle coaxially within the camera's field of view. Further, the image within the calibration circle can be expanded, and the area with less distortion can be selected for defect detection; that is, the selected image can be used as the sample defect image to eliminate the distortion effect of the underwater environment on the wide-angle camera's captured image. This application utilizes a probe stepping insertion into the pin hole to achieve a full-range inspection of the pin hole's inner wall, and the inspection process can be recorded to ensure the integrity and accuracy of the detection image.

[0093] It should be noted that this application can select a range with less distortion based on the camera's field of view, reducing the selection of distorted images, thereby constructing a sample dataset based on valid sample defect images. Furthermore, considering the dim environment inside the pin hole, a ring-shaped illumination lamp can be used at the probe's front end to provide uniform illumination, ensuring that a single image acquisition can achieve 360° full-range imaging within a certain axial range of the inner wall, and the inspection process can be recorded. Moreover, the probe can be connected to a ground computer via a signal cable, ensuring that the transmitted images can be further processed.

[0094] It should be noted that the sample defect category is used to characterize the types of defects and non-defect categories that may appear in the sample object to be tested. For example, if the sample object to be tested is the inner wall of a pin hole, the corresponding sample defect categories include wear, scratches, burrs, etc.

[0095] It should be noted that the defect mask data in this application is a binary mask data, which can accurately determine the location of the defect target in the sample defect image. The mask is typically a binary image with the same size as the bounding box, and each pixel value in the mask data indicates whether the pixel belongs to the target object, i.e., the specific location information. The mask data in this application is typically a binary string, which plays a positioning role in the algorithm, converting the location information into binary numbers and providing it to the host computer, where it is not displayed. The sample defect mask data is used to indicate the specific location information of the defect target in the sample defect image of the sample object to be detected. The area with a value of 1 in the defect mask data represents the location of the defect target; by performing a simple coordinate transformation on the coordinates with a value of 1, the specific location and shape of the defect target can be obtained.

[0096] It should be noted that the sample bounding box is used to indicate the approximate location information of the defect target in the sample defect image of the sample object to be detected. It can provide the approximate location and size information of the defect target. By determining the bounding box first, the image area to be processed can be narrowed down, thereby reducing the amount of computation, improving processing efficiency, and helping to accelerate subsequent processing steps. Both the mask data and the bounding box in this application have their own important roles, providing key information for instance segmentation and object detection, respectively. That is, although the binary mask data can determine the location of the defect target, the output bounding box is still necessary. Because there may be multiple targets in the image, the bounding box can help distinguish different defect targets. Although the binary mask data can segment defect targets to obtain location information, the bounding box provides a simple and effective way to manage and locate multiple targets. This synergistic effect makes the model more accurate and efficient in instance segmentation.

[0097] It should be noted that if the sample to be inspected is the inner wall of a pin hole, refer to Figure 2, which shows a schematic diagram of the control system for defect detection of the inner wall of a pin hole in this embodiment of the present application. In the video detection module, the field of view of the panoramic gimbal's camera can be used to acquire images of the sample to be inspected. The pin hole detection module can execute the defect detection method of this application to achieve defect detection of the inner wall of the pin hole.

[0098] It should be noted that, referring to Figure 3, Figure 3 shows the schematic diagram of the host computer software scheme corresponding to the defect detection of the inner wall of the pin hole in this embodiment of the application. In practical applications, the pin hole probe module in the schematic diagram of this host computer software scheme can execute the defect detection method of this application. This application adopts a dual-mode technical solution of industrial wireless communication control and CAN bus wired communication control, which can effectively improve the reliability of the system. Furthermore, before inspecting the current pin hole, the next pin hole can be photographed, and the positions of other pin holes can be marked. Through coordinate calculation, the position information of the robot and other pin holes is obtained, and the positions of the pin holes that have been worked, are being worked, and are waiting to be worked are marked in the host computer interface. Each pin hole is traversed with the shortest path, thereby improving the efficiency of the detection operation.

[0099] It should be noted that after obtaining multiple training sample data, this application can uniformly expand the defective images of the samples and adjust the pixels of the expanded images. Furthermore, the multiple training sample data can be divided into training set, test set, and validation set. The training set is the original sample dataset used to train the model. In this set, the model attempts to learn the inherent patterns and regularities of the data, continuously adjusting the model parameters to achieve optimal performance on the training set. The validation set is a subset of data separated from the sample dataset, used to evaluate the model's performance and adjust model parameters during training to avoid overfitting. The test set is the data set used to finally evaluate the model's performance. Its main purpose is to evaluate the model's generalization ability, i.e., its performance on unseen data. The test set should have a sufficient sample size to ensure the reliability of the evaluation results.

[0100] In step S120 of some embodiments, the defect region extraction model refers to a model capable of extracting defect regions from a defective image. Before inputting a sample defective image into the defect region extraction model, the sample defective image can be preprocessed. This preprocessing can include scaling, cropping, denoising, etc., to adapt to the model's input requirements. The defect region extraction model in this application can be a YOLO model, a YOLOv9 model, a Faster Region-based Convolutional Network (Faster R-CNN), etc., and is not specifically limited here. This application extracts defect regions from sample defective images using the defect region extraction model, enabling rapid and accurate identification of defective regions in the image, and further extracting features from the identified defective regions in subsequent image defect detection models.

[0101] Please refer to Figure 4, which is a flowchart illustrating step S120 provided in an embodiment of this application. In some embodiments of this application, step S120 may include, but is not limited to, steps S410 to S420. These two steps will be described in detail below with reference to Figure 4.

[0102] Step S410: Based on the defect region extraction model, the defect region of the sample defect image is extracted to obtain the first sample localization bounding box and the first sample localization confidence.

[0103] Step S420: Based on the first sample localization bounding box and the first sample localization confidence, the sample defect image is extracted to obtain the sample defect localization image.

[0104] In steps S410 and S420 of some embodiments, the defect region extraction model can detect and mark defect regions in the image, and generate a first sample localization bounding box to accurately locate the position of the defect target in the sample defect image. It also generates a first sample localization confidence score, which represents the degree of confidence the defect region extraction model has in the detection accuracy. Further, image extraction can be performed on the sample defect image based on the first sample localization bounding box and the first sample localization confidence score. Specifically, the defect-related portion of the image can be extracted from the sample defect image based on the first sample localization bounding box, and the accuracy of the extracted image can be determined based on the first sample localization confidence score, so that the obtained sample defect localization image can more accurately reflect the location and features of the defect.

[0105] It's important to note that the bounding box is a rectangular box used to mark the location of the defect in the image. This box is typically represented by four coordinates: the center point (x, y), the width (w), and the height (h). These coordinates can be normalized values ​​relative to the image size or absolute pixel coordinates. Location confidence is a numerical value representing the model's degree of confidence that the predicted bounding box contains the defect. Confidence is typically a floating-point number between 0 and 1, where 1 indicates complete confidence in the target within the bounding box, and 0 indicates no confidence. Confidence considers not only whether the bounding box contains the target object but also its accuracy, i.e., the degree to which the bounding box coincides with the actual target boundary.

[0106] In the above embodiments, this application can first utilize a defect region extraction model (such as YOLOv9 model, Faster R-CNN, etc.) to perform global detection on the sample defect image (such as the image of the region to be detected in the image of the inside of a pin hole). A first sample localization bounding box and a first sample localization confidence (i.e., class confidence) are predicted in a single neural network to achieve defect target detection and preliminary localization of the sample object to be detected (such as the inner wall of a pin hole). Furthermore, the defect region extraction model can improve the accuracy of defect region extraction through improved network structure, training strategies, or data augmentation techniques, enabling more accurate localization of defect targets under the same conditions, reducing the false negative rate, and thus improving the accuracy of defect detection. Further, the defect region extraction model can also achieve faster inference speed at the same accuracy by optimizing the network structure.

[0107] Please refer to Figure 5, which is a flowchart illustrating step S420 in an embodiment of this application. In some embodiments of this application, step S420 may include, but is not limited to, steps S510 to S530. These three steps will be described in detail below with reference to Figure 5.

[0108] Step S510: If the location confidence of the first sample is less than the preset confidence threshold, perform multi-scale feature fusion on the sample defect image based on the pre-trained multi-scale feature fusion model to obtain the sample fused feature image.

[0109] Step S520: Based on the defect region extraction model, the defect region is extracted from the sample fusion feature image to obtain the second sample localization bounding box and the second sample localization confidence.

[0110] Step S530: Based on the second sample location bounding box and the second sample location confidence, the sample defect image is extracted to obtain the sample defect location image.

[0111] In step S510 of some embodiments, the preset confidence threshold refers to the threshold that the confidence level of the extracted sample defect localization image must meet. A higher preset confidence threshold results in a more reliable sample defect localization image, which more accurately contains the defect target and is closer to the true bounding box. If the confidence level of the first sample localization is less than the preset confidence threshold, it indicates that the defect region extraction model is not confident or accurate enough in locating and extracting the defect region in the sample defect image. In this case, a multi-scale feature fusion model can be used to perform multi-scale feature fusion on the sample defect image to obtain a sample fused feature image. This multi-scale feature fusion model can be a deep neural network designed to extract feature information of different scales and levels from the sample defect image. Then, by using multi-scale feature fusion technology, image features can be fused and integrated at different resolutions to obtain a sample fused feature image that provides a more comprehensive understanding of the image content and structure, thus improving the system's understanding and localization accuracy of the defect region. Therefore, the sample fused feature image obtained at this time can contain feature information at multiple scales, providing a more comprehensive and richer feature representation for subsequent steps, thereby improving the accuracy and reliability of defect detection and localization.

[0112] It should be noted that the preset confidence threshold in this application may also include a preset low confidence threshold. If the localization confidence of the first sample is less than the preset low confidence threshold, it indicates that the model has a low degree of confidence in the detected target. In this case, it can be said that the sample defect image has a big problem, and the image can be re-acquired to avoid wasting computer resources on subsequent invalid operations.

[0113] In steps S520 to S530 of some embodiments, after obtaining the sample fusion feature image, this application can extract the defect region from the sample fusion feature image based on the defect region extraction model to obtain the second sample localization bounding box and the second sample localization confidence. The forms of the second sample localization bounding box and the second sample localization confidence are the same as those of the first sample localization bounding box and the first sample localization confidence, only the model input is different, which will not be elaborated here. Further, image extraction can be performed on the sample defect image based on the second sample localization bounding box and the second sample localization confidence to obtain the sample defect localization image, and the second sample localization confidence is greater than or equal to a preset confidence threshold.

[0114] In step S130 of some embodiments, the sample detection result includes predicted bounding box, predicted defect category, and predicted defect mask data. Specific explanations of the bounding box, defect category, and defect mask data can be found in the descriptions of the above embodiments, and will not be repeated here. Please refer to Figure 6, which shows a flowchart of a method for defect detection of the inner wall of a pin hole provided by an embodiment of this application. In this application, an image of the inner wall of the pin hole can be acquired and preprocessed to obtain a sample defect image. Then, the sample defect image is labeled with categories to construct a sample dataset. A training set, a test set, and a validation set are constructed based on the sample dataset. The training set includes multiple training sample data. Further, a defect region extraction model is used to extract the defect region from the sample defect image to obtain a sample defect location image. This sample defect location image is then input into an image defect detection model for defect detection to obtain a sample detection result. After training, a target defect detection model 610 can be constructed based on the parameter-updated defect region extraction model and image defect detection model. The target defect detection model 610 is then evaluated using the test set and validation set.

[0115] Please refer to Figure 7, which is a flowchart illustrating step S130 in an embodiment of this application. In some embodiments of this application, step S130 may include, but is not limited to, steps S710 to S750. These five steps will be described in detail below with reference to Figure 7.

[0116] Step S710: Extract features from the sample defect location image based on the image feature extraction layer to obtain the initial features of the sample;

[0117] Step S720: Based on the multi-scale feature extraction layer, multi-scale feature extraction is performed on the initial features of the sample to obtain multiple candidate features of the sample with different dimensions;

[0118] Step S730: Based on the region detection layer, perform region target detection on the initial features of the samples to obtain multiple candidate region bounding boxes;

[0119] Step S740: Based on the spatial feature extraction layer, spatial features are extracted from multiple candidate features of samples in different dimensions and bounding boxes of multiple candidate regions of samples to obtain sample spatial features;

[0120] Step S750: Defect detection is performed on the spatial features of the sample based on the classification layer to obtain the sample detection results.

[0121] In steps S710 to S750 of some embodiments, referring to Figure 8, Figure 8 shows a schematic diagram of a model structure of the image defect detection model provided in this application embodiment. This image defect detection model includes an image feature extraction layer, a multi-scale feature extraction layer, a region detection layer, a spatial feature extraction layer, and a classification layer. The image feature extraction layer of this application can be a structure built based on an improved Mobilenet attention mechanism. This improved Mobilenet attention mechanism is an improved structure based on the Single Shot Multibox Detector (SSD), which can integrate the idea of ​​depthwise decomposable convolution in the network and replace the base network in the traditional SSD for feature extraction. Therefore, the improved image feature extraction layer is a lightweight object detection algorithm with a small model size and efficient computation, enabling faster inference speed. Combining it with other structures of the image defect detection model can improve the computational efficiency of the defect detection device while maintaining high detection accuracy, making it better suited for scenarios requiring real-time object detection and instance segmentation. Furthermore, the improved image feature extraction layer performs well in resource-constrained environments, such as mobile devices and embedded systems. By combining it with other structures of image defect detection models, more comprehensive object detection and segmentation tasks can be achieved in these environments, providing more possibilities and better adaptability for mobile applications.

[0122] It should be noted that, referring to Figures 9A-9C, some convolutional structure diagrams provided in the embodiments of this application are shown, wherein it is assumed that the input feature map is (D F ×D F ×M), the kernel size is (D ×M), K ×D K ×N), where M and N are the number of input and output channels, respectively, and D F and D K This corresponds to the number of rows and columns. Based on this, the specific computational complexity is: the standard convolution is D. F ×D F ×M×D K ×D K ×N, depthwise convolution is D F ×D F ×M×D K ×D K A 1×1 point convolution is D F ×D F ×M×N, depthwise separable convolution is (D F ×D F ×M×D K ×D K )+(D F ×D F(×M×N). Referring to Figures 10A and 10B, Figure 10A shows a schematic diagram of the standard convolution process, and Figure 10B shows a schematic diagram of the depthwise separable convolution process. This embodiment combines an improved Mobilenet attention mechanism (i.e., SSD_Mobilenet) obtained through a depthwise separable convolution process, and uses the convolutional and pooling layers of this improved Mobilenet attention mechanism as the structure of the image feature extraction layer. In this process, this application can appropriately adjust the size and number of channels of the output feature map in the image feature extraction layer to adapt to the input requirements of the image defect detection model, thereby better achieving fast and efficient real-time target detection and instance segmentation tasks. The improved Mobilenet of this application can integrate the idea of ​​depthwise decomposition convolution in the network, using Mobilenet to replace the base network VGG16 in the traditional SSD for feature extraction, and removing the final fully connected layer and Softmax in the network to better achieve fast and efficient real-time target detection and instance segmentation tasks.

[0123] In step S720 of some embodiments, the multi-scale feature extraction layer can be the model structure obtained after optimizing the Feature Pyramid Network (FPN) based on the Squeeze-and-Excitation (SE) module. The SE module is an attention mechanism used to enhance deep neural network models. It aims to effectively adjust the importance of feature maps by learning the channel correlations from the intermediate feature maps of the network, thereby increasing the model's attention to important features. Referring to Figure 11, since the FPN consists of three parts: a bottom-up convolutional neural network process, a top-down convolutional neural network process, and a side connection process between features, this application can add an SE module between the corresponding layers of the bottom-up and top-down convolutional neural network processes of the FPN to perform feature fusion between feature layers of different scales. The SE module can learn the importance of each channel, thereby adjusting and optimizing the correlation between input feature map channels. During training, the SE module can adaptively learn the weights of each channel, thereby helping the deep neural network model better capture and utilize feature information, improving the model's performance and generalization ability. Among them, the multiple sample candidate features obtained based on the multi-scale feature extraction layer contain semantic and local feature information of the image, and have different resolutions and semantic levels.

[0124] In step S730 of some embodiments, after obtaining the initial features of the sample, this application can perform regional target detection on the initial features of the sample based on the region detection layer to obtain multiple candidate region bounding boxes. The region detection layer can be constructed based on a Region Proposal Network (RPN) to generate multiple candidate region bounding boxes that may contain defective targets.

[0125] In steps S740 and S750 of some embodiments, the spatial feature extraction layer can extract spatial features from multiple candidate features of samples of different dimensions and multiple candidate region bounding boxes of samples to obtain sample spatial features. This spatial feature extraction layer can be constructed based on the Region of Interest Align (RoI Align) method. RoI Align can be used to extract features on RoIs of different sizes, instead of using the original RoI pooling method, to more accurately capture the spatial information of the defect target. Further, defect detection is performed on the sample spatial features based on the classification layer to obtain the sample detection result.

[0126] Please refer to Figure 12, which is a flowchart of step S750 provided in an embodiment of this application. In some embodiments of this application, step S750 may include, but is not limited to, steps S1210 to S1230. These three steps will be described in detail below with reference to Figure 12.

[0127] Step S1210: Pixel-level prediction of sample space features is performed based on the mask prediction branch of the classification layer to obtain predicted defect mask data.

[0128] Step S1220: Based on the category prediction branch of the classification layer, perform category prediction on the sample space features to obtain the predicted defect category;

[0129] Step S1230: Based on the bounding box prediction branch of the classification layer, bounding box prediction is performed on the sample space features to obtain the predicted bounding box.

[0130] In step S1210 of some embodiments, as shown in FIG8, the mask prediction branch of this application refers to a structure including a mask head and a structure optimized based on lightweight depthwise separable convolution. The mask head is used to generate an accurate mask for the target instance. The lightweight depthwise separable convolution can employ the improved MobileNet model described in the above embodiments or other model structures capable of generating accurate masks, which will not be elaborated further here. This approach yields more accurate mask data, enabling precise pixel-level segmentation of defective targets and thus improving the instance segmentation accuracy of defect detection.

[0131] It should be noted that the overall architecture of the image defect detection model in this application can be built based on Region-Convolutional Neural Networks (R-CNN).

[0132] In steps S1220 and S1230 of some embodiments, the category prediction branch of the classification layer is used to predict the category of the sample space features to obtain the predicted defect category, so as to accurately identify the defect category of the defect target in the sample defect image. At the same time, the bounding box prediction branch of the classification layer is used to predict the bounding box of the sample space features to obtain the predicted bounding box, so as to accurately delineate the location of the defect target.

[0133] In step S140 of some embodiments, after obtaining the sample detection result, this application can calculate a first loss value based on the sample defect mask data and the predicted defect mask data. A second loss value is calculated based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box, and the model loss value is determined based on the first loss value and the second loss value.

[0134] It should be noted that the process of calculating the first loss value based on the sample defect mask data and the predicted defect mask data can be seen in the following formula 1:

[0135] Where N1 is the number of pixels corresponding to the sample defect localization image, n represents the number of data contained in the sample defect mask data, and y i p is the i-th mask value corresponding to the sample defect mask data. i To predict the i-th mask value corresponding to the defect mask data, Loss1 is the loss calculated only for the pixel position of the defect target instance in the sample curve localization image. In this way, background pixels can be ignored, reducing the amount of computation during training.

[0136] This application determines the model loss value for model parameter adjustment by using a first loss value determined by a joint instance segmentation task and a second loss value determined by a real-time target detection task, which can improve the detection accuracy of the model in practical applications.

[0137] Please refer to Figure 13, which is a specific flowchart of calculating the second loss value provided in an embodiment of this application. In some embodiments of this application, the process of calculating the second loss value may include, but is not limited to, steps S1310 to S1350. These five steps will be described in detail below with reference to Figure 13.

[0138] Step S1310: Extract prior boxes based on the initial features of the samples to obtain multiple candidate prior boxes;

[0139] Step S1320: Calculate the confidence loss based on the preset defect category, multiple candidate prior boxes, and confidence prediction values ​​to obtain the first sub-loss value;

[0140] Step S1330: Calculate the location loss based on the predicted bounding box, the sample bounding box, and multiple candidate prior boxes to obtain the second sub-loss value;

[0141] Step S1340: Calculate the category loss based on the predicted defect category and the sample defect category to obtain the third sub-loss value;

[0142] Step S1350: The first sub-loss value, the second sub-loss value, and the third sub-loss value are weighted and calculated to obtain the second loss value.

[0143] In steps S1310 to S1350 of some embodiments, the multiple candidate prior boxes generated by this application include multiple positive sample prior boxes. This application can calculate the confidence loss based on a preset defect category, multiple positive sample prior boxes, and a confidence prediction value to obtain a first sub-loss value. The calculation process of the first sub-loss value is shown in Formula 2 below:

[0144] Where Loss2 represents the second loss value, N2 represents the number of positive prior boxes, and L conf L represents the confidence loss value, i.e., the first sub-loss value. loc L represents the regression loss value, i.e., the second sub-loss value. p α1 represents the category loss, i.e., the third sub-loss value; c represents the confidence prediction value; l represents the location information of the predicted bounding box; g is the location parameter of the true target under the correct category label, i.e., the location information of the sample bounding box; α1, α2, and α3 are the weight coefficients corresponding to each loss value, and the sum of the weight coefficients can be 1.

[0145] It should be noted that positive sample prior boxes refer to candidate prior boxes in defect detection tasks whose intersection over union (IOU) with the true bounding box (i.e., sample bounding box) is greater than a set threshold.

[0146] It should be noted that, for x, Assuming N² matches are selected, then In this context, i represents the i-th default value, j represents the j-th sample bounding box, and p represents the p-th defect category.

[0147] It should be noted that L conf It consists of two parts: positive and negative sample errors. The process of calculating the first sub-loss value is shown in Formula 3 below:

[0148] Wherein, the subscript Pos represents a positive sample, the subscript Neg represents a negative sample, i.e., the background, and ecp represents a cumulative distribution function.

[0149] It should be noted that the calculation process for the second sub-loss value is shown in Formulas 4 and 5 below:

[0150] Where k represents the k-th defect category.

[0151] It should be noted that the third sub-loss value is the class loss, which is used in classification tasks to measure the difference between the model's prediction and the true label, and is one of the optimization objectives during model training. Common class loss functions include cross-entropy loss and softmax loss. The values ​​of α1, α2, and α3 in this application can be flexibly set according to actual needs and are not limited.

[0152] Please refer to Figure 14, which is a flowchart illustrating the determination of model loss value according to an embodiment of this application. In some embodiments of this application, the process of determining model loss value may include, but is not limited to, steps S1410 to S1430. These three steps will be described in detail below with reference to Figure 14.

[0153] Step S1410: Obtain the instance segmentation task and defect detection accuracy value of the sample object to be detected;

[0154] Step S1420: Determine sample weights based on instance segmentation task and defect detection accuracy value;

[0155] Step S1430: Perform a weighted calculation based on the first weight, the first loss value, the second weight, and the second loss value to obtain the model loss value.

[0156] In step S1410 of some embodiments, since the sample objects to be detected may come from different products or production batches, each sample object to be detected needs to undergo an instance segmentation task (i.e., separating different defect target instances from the image) and measure the defect detection accuracy value (i.e., the error between the predicted result and the actual annotation). Thus, the instance segmentation task and the defect detection accuracy value may be the same or different in different sample objects to be detected.

[0157] In step S1420 of some embodiments, the process of determining sample weights based on the instance segmentation task and the defect detection accuracy value can be done using a model. That is, the instance segmentation task and the defect detection accuracy value are input into a pre-trained weight prediction model (a neural network model, not limited) to predict the weights and obtain the sample weights. The sample weights include the first weights corresponding to the first loss value and the second weights corresponding to the second loss value.

[0158] In other embodiments, the process of determining sample weights based on instance segmentation tasks and defect detection accuracy values ​​can be achieved using a lookup table. That is, corresponding sample weights are matched to a pre-defined weight matching table based on the instance segmentation task and defect detection accuracy values. In designing the overall model loss value, this application considers the performance of instance segmentation tasks and defect detection accuracy values ​​in defect detection tasks. By adjusting the corresponding weight coefficients, the influence between the two can be flexibly balanced. For example, if more emphasis is placed on the accuracy of instance segmentation tasks, the first weight can be increased; if more emphasis is placed on inference speed and model size, the second weight can be increased.

[0159] In step S1430 of some embodiments, the process of calculating the model loss value based on the first weight, the first loss value, the second weight, and the second loss value is shown in Formula 6 below: Loss total =α0Loss1+β0Loss2 (Formula 6)

[0160] Among them, Loss total Let α0 represent the model loss value, β0 represent the first weight, and β0 represent the second weight.

[0161] In step S150 of some embodiments, after obtaining the model loss value, this application can update the model parameters of the defect region extraction model and the image defect detection model based on the model loss value until a preset termination condition is met, and select the defect region extraction model and the image defect detection model corresponding to the minimum model loss value during training to construct the target defect detection model. Further, the test set and validation set can be input into the trained target defect detection model for defect detection to obtain defect detection results.

[0162] It should be noted that the preset termination condition can be either the model training ends after all training sample data has been trained, or the model training ends when the model loss value is less than the preset loss value threshold. No specific limitation is made here.

[0163] In the above embodiments, when training the defect region extraction model and the image defect detection model, this application considers the performance of instance segmentation and defect detection accuracy in the defect detection task. By adjusting the corresponding weight coefficients, the influence between the two can be flexibly balanced. Furthermore, by jointly determining the model loss value of this application from the first loss value, the confidence loss, regression loss, and category loss in the second loss value, the detection accuracy of the target defect detection model can be effectively improved. In addition, invalid data can be eliminated or preprocessed in the early processing stage of the defect region extraction model, which can effectively improve the defect detection efficiency of the target object.

[0164] In step S160 of some embodiments, the target detection result includes the target bounding box, the target defect category, and the target defect mask data. Specifically, the defect targets in the target detection image can be segmented based on the target bounding box and the target defect mask data.

[0165] This application provides a defect detection method, apparatus, computer device, and storage medium. This application utilizes a series of candidate prior boxes of different scales generated by an image defect detection model to classify detected defect targets and regress their location coordinates, significantly reducing the computational load of the model and improving defect detection speed, making it more suitable for real-time detection needs in industry. Furthermore, by combining the constructed model loss function, the predicted bounding boxes can become increasingly closer to the actual sample bounding boxes during training, effectively improving the accuracy and efficiency of pin hole inner wall scratch defect detection. The improved MobileNet has a smaller model size and fewer parameters. This application further reduces the overall model size by combining the improved MobileNet with other structures of image defect detection models (such as the improved Mask R-CNN), reducing resource consumption for model deployment and operation, and achieving model lightweighting. This makes it better suited for resource-constrained environments, enabling more comprehensive target detection and instance segmentation tasks, and giving the model wider application capabilities and applicability. Compared to the Fast R-CNN used in related technologies, the improved target defect detection model in this application can be used not only for target detection but also for instance segmentation. This not only enables more accurate detection of defective targets but also provides detailed segmentation information for defective targets, making it more suitable for application scenarios that require fine segmentation.

[0166] Please refer to Figure 15. Figure 15 is a schematic diagram of the module structure of the defect detection device provided in the embodiment of this application. The device can implement the defect detection method of the above embodiment. The device includes an acquisition module 1510, a region extraction module 1520, a first detection module 1530, a loss calculation module 1540, an update module 1550, and a second detection module 1560.

[0167] The acquisition module 1510 is used to acquire training sample data, which includes sample defect images of the sample object to be detected, sample defect categories, sample bounding boxes, and sample defect mask data. The sample bounding boxes are used to indicate the region information of the defect parts of the sample object to be detected in the sample defect image, and the sample defect mask data are used to indicate the pixel boundaries of the defect parts of the sample object to be detected within the sample bounding boxes.

[0168] The region extraction module 1520 is used to extract the defect region from the sample defect image based on the defect region extraction model to obtain the sample defect localization image.

[0169] The first detection module 1530 is used to perform defect detection on the sample defect localization image based on the image defect detection model, and obtain the sample detection result, which includes the predicted bounding box, the predicted defect category and the predicted defect mask data.

[0170] The loss calculation module 1540 is used to calculate a first loss value based on sample defect mask data and predicted defect mask data, calculate a second loss value based on sample defect category, predicted defect category, sample bounding box and predicted bounding box, and determine the model loss value based on the first loss value and the second loss value.

[0171] The update module 1550 is used to update the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and to construct the target defect detection model based on the updated defect region extraction model and the image defect detection model.

[0172] The second detection module 1560 is used to acquire the target detection image and perform defect detection on the target detection image based on the target defect detection model to obtain the target detection result, which includes the target bounding box, the target defect category and the target defect mask data.

[0173] It should be noted that the defect detection device in this application embodiment is used to implement the defect detection method in the above embodiment. The defect detection device in this application embodiment corresponds to the aforementioned defect detection method. For the specific processing procedure, please refer to the aforementioned defect detection method, which will not be repeated here.

[0174] This application also provides a computer device comprising: at least one memory, at least one processor, and at least one computer program. The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement any of the defect detection methods described in the above embodiments. The computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0175] Please refer to Figure 16, which illustrates the hardware structure of a computer device according to another embodiment. The computer device includes:

[0176] The processor 1610 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0177] The memory 1620 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1620 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1620 and is called and executed by the processor 1610 using the defect detection method of the embodiments of this application.

[0178] The 1630 input / output interface is used to implement information input and output.

[0179] The communication interface 1640 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0180] Bus 1650 transmits information between various components of the device (e.g., processor 1610, memory 1620, input / output interface 1630, and communication interface 1640);

[0181] The processor 1610, memory 1620, input / output interface 1630 and communication interface 1640 are connected to each other within the device via bus 1650.

[0182] This application also provides a storage medium, which is a computer-readable storage medium, storing a computer program for causing a computer to execute the defect detection method described in the above embodiments.

[0183] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0184] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0185] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0186] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0188] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0189] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0191] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A defect detection method, characterized in that, The method includes: Acquire training sample data, which includes a sample defect image of the sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; wherein, the sample bounding box is used to indicate the region information of the defect location of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect location of the sample object to be detected in the sample bounding box. Based on the defect region extraction model, the defect region of the sample defect image is extracted to obtain the sample defect localization image; The sample defect location image is subjected to defect detection based on the image defect detection model to obtain sample detection results; wherein, the sample detection results include predicted bounding boxes, predicted defect categories and predicted defect mask data; A first loss value is calculated based on the sample defect mask data and the predicted defect mask data. A second loss value is calculated based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box. A model loss value is determined based on the first loss value and the second loss value. The model parameters of the defect region extraction model and the image defect detection model are updated based on the model loss value, and a target defect detection model is constructed based on the updated defect region extraction model and the image defect detection model. A target detection image is acquired, and defect detection is performed on the target detection image based on the target defect detection model to obtain a target detection result; wherein, the target detection result includes target bounding box, target defect category, and target defect mask data.

2. The method according to claim 1, characterized in that, The defect region extraction based on the defect region extraction model is used to extract the defect region from the sample defect image to obtain a sample defect localization image, including: Based on the defect region extraction model, the defect region of the sample defect image is extracted to obtain the first sample localization bounding box and the first sample localization confidence. Based on the first sample localization bounding box and the first sample localization confidence, the sample defect image is extracted to obtain the sample defect localization image.

3. The method according to claim 2, characterized in that, The step of extracting the sample defect image based on the first sample localization bounding box and the first sample localization confidence to obtain the sample defect localization image includes: If the location confidence of the first sample is less than a preset confidence threshold, multi-scale feature fusion is performed on the defective image of the sample based on a pre-trained multi-scale feature fusion model to obtain a sample fused feature image. Based on the defect region extraction model, defect regions are extracted from the sample fusion feature image to obtain the second sample localization bounding box and the second sample localization confidence. Based on the second sample localization bounding box and the second sample localization confidence, the sample defect image is extracted to obtain the sample defect localization image.

4. The method according to claim 3, characterized in that, The second sample location confidence level is greater than or equal to the preset confidence threshold.

5. The method according to claim 2, characterized in that, The step of extracting the sample defect image based on the first sample localization bounding box and the first sample localization confidence level to obtain the sample defect localization image further includes: Based on the first sample localization bounding box, the image related to the defect is extracted from the sample defect image, and the accuracy of the extracted image is determined based on the first sample localization reliability. The sample defect localization image can reflect the location and characteristics of the defect.

6. The method according to any one of claims 1 to 3, characterized in that, The step of determining the model loss value based on the first loss value and the second loss value includes: Obtain the instance segmentation task and defect detection accuracy value of the sample object to be detected; The sample weights are determined based on the instance segmentation task and the defect detection accuracy value, and the sample weights include a first weight corresponding to the first loss value and a second weight corresponding to the second loss value. The model loss value is obtained by weighting the first weight, the first loss value, the second weight, and the second loss value.

7. The method according to claim 6, characterized in that, The step of determining sample weights based on the instance segmentation task and the defect detection accuracy value includes: The corresponding sample weights are matched based on the instance segmentation task and the defect detection accuracy value using a pre-defined weight matching table.

8. The method according to any one of claims 1 to 3, characterized in that, The image defect detection model includes an image feature extraction layer, a multi-scale feature extraction layer, a region detection layer, a spatial feature extraction layer, and a classification layer. The defect detection is performed on the sample defect location image based on the image defect detection model to obtain the sample detection result, including: Based on the image feature extraction layer, feature extraction is performed on the sample defect location image to obtain the initial features of the sample; Based on the multi-scale feature extraction layer, multi-scale feature extraction is performed on the initial features of the sample to obtain multiple candidate features of the sample in different dimensions; Based on the region detection layer, the initial features of the sample are used to detect regional targets, resulting in multiple candidate region bounding boxes for the sample. Based on the spatial feature extraction layer, spatial features are extracted from the sample candidate features of multiple dimensions and the bounding boxes of multiple sample candidate regions to obtain sample spatial features; Based on the classification layer, defect detection is performed on the spatial features of the sample to obtain the sample detection results.

9. The method according to claim 8, characterized in that, The sample candidate features include semantic and local feature information of the image, and the multi-scale feature extraction layer has different resolutions and semantic levels.

10. The method according to claim 8, characterized in that, The defect detection based on the classification layer of the sample space features to obtain sample detection results includes: The mask prediction branch of the classification layer performs pixel-level prediction on the sample space features to obtain the predicted defect mask data; the mask prediction branch is a structure built based on lightweight depthwise separable convolution optimization. Based on the category prediction branch of the classification layer, the category of the sample space features is predicted to obtain the predicted defect category; The bounding box prediction branch of the classification layer predicts bounding boxes based on the sample space features to obtain the predicted bounding boxes.

11. The method according to claim 8, characterized in that, The sample detection result also includes a confidence prediction value. The calculation of the second loss value based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box includes: Based on the initial features of the sample, prior boxes are extracted to obtain multiple candidate prior boxes; The confidence loss is calculated based on the preset defect category, multiple candidate prior boxes, and the confidence prediction value to obtain the first sub-loss value; Based on the predicted bounding box, the sample bounding box, and multiple candidate prior boxes, the location loss is calculated to obtain the second sub-loss value; Based on the predicted defect category and the sample defect category, a category loss is calculated to obtain a third sub-loss value; The second loss value is obtained by weighting the first sub-loss value, the second sub-loss value, and the third sub-loss value.

12. The method according to claim 11, characterized in that, The multiple candidate prior boxes include multiple positive sample prior boxes.

13. A defect detection device, characterized in that, The device includes: The acquisition module is used to acquire training sample data, which includes a sample defect image of the sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; the sample bounding box is used to indicate the region information of the defect location of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect location of the sample object to be detected in the sample bounding box. The region extraction module is used to extract the defect region from the sample defect image based on the defect region extraction model to obtain the sample defect localization image. The first detection module is used to perform defect detection on the sample defect localization image based on the image defect detection model to obtain sample detection results, which include predicted bounding boxes, predicted defect categories and predicted defect mask data. The loss calculation module is used to calculate a first loss value based on the sample defect mask data and the predicted defect mask data, calculate a second loss value based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and determine the model loss value based on the first loss value and the second loss value. The update module is used to update the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and to construct a target defect detection model based on the updated defect region extraction model and the image defect detection model. The second detection module is used to acquire the target detection image and perform defect detection on the target detection image based on the target defect detection model to obtain the target detection result, which includes the target bounding box, the target defect category and the target defect mask data.

14. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 13.

Images