Defect detection method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a defect detection method and device, computer equipment and a storage medium, and belongs to the technical field of defect detection. The method comprises the following steps: acquiring training sample data, and extracting a sample defect image based on a defect region extraction model to obtain a sample defect positioning image; detecting the sample defect positioning image based on the image defect detection model to obtain a sample detection result; calculating a first loss value based on the sample defect mask data and the predicted defect mask data, and calculating a second loss value based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box to determine a model loss value; updating model parameters of the defect area extraction model and the image defect detection model based on the model loss value so as to construct a target defect detection model; and detecting the target detection image based on the target defect detection model to obtain a target detection result. According to the embodiment of the invention, the defect detection efficiency and accuracy of the to-be-detected object can be improved.

Description

Technical Field

[0001] The present application relates to the field of defect detection technology, and in particular to a defect detection method and device, computer equipment, and storage medium. Background Art

[0002] Defect detection refers to the technology of detecting defects on the surface of the object to be tested. By acquiring images of the object to be tested and performing defect detection on the acquired images, the defect position of the object to be tested can be repaired in time to ensure the use of the object to be tested in practical applications. For example, in a nuclear reactor, the cotter pin of the control rod guide tube is a key component, which is mainly used to control the reaction rate of the nuclear reactor and maintain the safety of the nuclear reactor. The cotter pin of the control rod guide tube needs to be inserted into the pin hole corresponding to the cotter pin of the control rod guide tube, and the wheel on the bolt or shaft is fixed on the forked part to ensure the accuracy and stability between the mechanical parts. Therefore, defect detection of the image acquired from the pin hole corresponding to the cotter pin of the control rod guide tube can ensure the quality status of the control rod during the operation of inserting the guide tube, thereby improving the reliability and safety of the nuclear power plant.

[0003] At present, the related technology usually adopts a manual operation method for the defect detection method of the pin hole corresponding to the cotter pin of the control rod guide tube, that is, manually judging the image collected from the pin hole corresponding to the cotter pin of the control rod guide tube. This defect detection method is low in efficiency and low in detection accuracy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a defect detection method and apparatus, computer equipment, and storage medium, which can improve the efficiency and accuracy of defect detection of the object to be detected.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a defect detection method, the method comprising:

[0006] Acquire training sample data, the training sample data including a sample defect image of a sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; wherein the sample bounding box is used to indicate the area information of the defect part of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect part of the sample object to be detected in the sample bounding box;

[0007] Extracting defect regions from the sample defect image based on a defect region extraction model to obtain a sample defect location image;

[0008] Performing defect detection on the sample defect location image based on the image defect detection model to obtain a sample detection result; wherein the sample detection result includes a predicted bounding box, a predicted defect category, and predicted defect mask data;

[0009] Calculating a first loss value based on the sample defect mask data and the predicted defect mask data, calculating a second loss value based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box, and determining a model loss value based on the first loss value and the second loss value;

[0010] updating the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and constructing a target defect detection model based on the updated defect region extraction model and the image defect detection model;

[0011] A target detection image is acquired, and defect detection is performed on the target detection image based on the target defect detection model to obtain a target detection result; wherein the target detection result includes a target bounding box, a target defect category, and a target defect mask data.

[0012] In some embodiments, the defect region extraction from the sample defect image based on the defect region extraction model to obtain the sample defect location image includes:

[0013] Extracting defect regions from the sample defect image based on a defect region extraction model to obtain a first sample positioning bounding box and a first sample positioning confidence;

[0014] Image extraction is performed on the sample defect image based on the first sample positioning boundary box and the first sample positioning reliability to obtain a sample defect positioning image.

[0015] In some embodiments, performing image extraction on the sample defect image based on the first sample positioning bounding box and the first sample positioning confidence to obtain the sample defect positioning image includes:

[0016] If the first sample positioning reliability is less than a preset reliability threshold, multi-scale feature fusion is performed on the sample defect image based on a pre-trained multi-scale feature fusion model to obtain a sample fusion feature image;

[0017] Extracting defect regions from the sample fusion feature image based on the defect region extraction model to obtain a second sample positioning bounding box and a second sample positioning reliability;

[0018] Image extraction is performed on the sample defect image based on the second sample positioning boundary box and the second sample positioning reliability to obtain a sample defect positioning image.

[0019] In some embodiments, determining the model loss value based on the first loss value and the second loss value includes:

[0020] Obtaining the instance segmentation task and defect detection accuracy value of the sample object to be detected;

[0021] Determine a sample weight based on the instance segmentation task and the defect detection accuracy value, the sample weight comprising a first weight corresponding to the first loss value and a second weight corresponding to the second loss value;

[0022] A weighted calculation is performed according to the first weight, the first loss value, the second weight and the second loss value to obtain a model loss value.

[0023] In some embodiments, the image defect detection model includes an image feature extraction layer, a multi-scale feature extraction layer, a region detection layer, a spatial feature extraction layer, and a classification layer. The image defect detection model is used to perform defect detection on the sample defect location image to obtain a sample detection result, including:

[0024] Extracting features from the sample defect location image based on the image feature extraction layer to obtain sample initial features;

[0025] Based on the multi-scale feature extraction layer, multi-scale feature extraction is performed on the initial features of the sample to obtain a plurality of sample candidate features of different dimensions;

[0026] Performing regional target detection on the sample initial features based on the region detection layer to obtain a plurality of sample candidate region bounding boxes;

[0027] Based on the spatial feature extraction layer, spatial feature extraction is performed on the sample candidate features of multiple different dimensions and the multiple sample candidate region bounding boxes to obtain sample spatial features;

[0028] Defect detection is performed on the sample space features based on the classification layer to obtain a sample detection result.

[0029] In some embodiments, performing defect detection on the sample space features based on the classification layer to obtain a sample detection result includes:

[0030] The mask prediction branch of the classification layer performs pixel-level prediction on the sample space features to obtain the predicted defect mask data; the mask prediction branch is a structure constructed based on lightweight depth-separable convolution optimization;

[0031] Performing category prediction on the sample space feature based on the category prediction branch of the classification layer to obtain the predicted defect category;

[0032] Based on the bounding box prediction branch of the classification layer, a bounding box prediction is performed on the sample space feature to obtain the predicted bounding box.

[0033] In some embodiments, the sample detection result further includes a confidence prediction value, and the second loss value is calculated based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box, including:

[0034] Extracting a priori frames based on the initial features of the sample to obtain multiple candidate priori frames;

[0035] Calculate the confidence loss based on the preset defect category, the plurality of candidate prior boxes and the confidence prediction value to obtain a first sub-loss value;

[0036] Performing position loss calculation based on the predicted bounding box, the sample bounding box, and the plurality of candidate prior boxes to obtain a second sub-loss value;

[0037] Performing a category loss calculation based on the predicted defect category and the sample defect category to obtain a third sub-loss value;

[0038] The first sub-loss value, the second sub-loss value and the third sub-loss value are weightedly calculated to obtain the second loss value.

[0039] To achieve the above object, a second aspect of an embodiment of the present application provides a defect detection method, the device comprising:

[0040] an acquisition module, configured to acquire training sample data, wherein the training sample data includes a sample defect image of a sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; the sample bounding box is used to indicate the area information of the defect part of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect part of the sample object to be detected in the sample bounding box;

[0041] A region extraction module, used to extract defect regions from the sample defect image based on a defect region extraction model to obtain a sample defect positioning image;

[0042] A first detection module, configured to perform defect detection on the sample defect location image based on an image defect detection model to obtain a sample detection result, wherein the sample detection result includes a predicted bounding box, a predicted defect category, and predicted defect mask data;

[0043] a loss calculation module, configured to calculate a first loss value based on the sample defect mask data and the predicted defect mask data, calculate a second loss value based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and determine a model loss value based on the first loss value and the second loss value;

[0044] An updating module, used for updating the model parameters of the defect area extraction model and the image defect detection model based on the model loss value, and constructing a target defect detection model based on the updated defect area extraction model and the image defect detection model;

[0045] The second detection module is used to acquire a target detection image, and perform defect detection on the target detection image based on the target defect detection model to obtain a target detection result, wherein the target detection result includes a target bounding box, a target defect category, and a target defect mask data.

[0046] To achieve the above-mentioned purpose, the third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in any one of the embodiments of the first aspect when executing the computer program.

[0047] To achieve the above objectives, the fourth aspect of the embodiments of the present application further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the embodiments of the first aspect is implemented.

[0048] The defect detection method and apparatus, computer equipment, and storage medium proposed in the embodiments of the present application can construct a target defect detection model with significantly improved detection efficiency and accuracy through the joint training of a defect region extraction model and an image defect detection model. Specifically, first obtain training sample data, which includes a sample defect image of a sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; wherein the sample bounding box is used to indicate the regional information of the defective part of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defective part of the sample object to be detected in the sample bounding box. Then, based on the defect region extraction model, the sample defect image is subjected to defect region extraction to obtain a sample defect location image; based on the image defect detection model, defect detection is performed on the sample defect location image to obtain a sample detection result; the sample detection result includes a predicted bounding box, a predicted defect category, and a predicted defect mask data. Furthermore, a first loss value is calculated based on the sample defect mask data and the predicted defect mask data, a second loss value is calculated based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and a model loss value is determined based on the first loss value and the second loss value; finally, the model parameters of the defect area extraction model and the image defect detection model can be updated based on the model loss value, and a target defect detection model is constructed based on the updated defect area extraction model and the image defect detection model. In practical applications, the object to be tested of the present application can be the inside of the pin hole corresponding to the cotter pin of the control rod guide cylinder, that is, the target detection image obtained is the image of the inner wall of the pin hole, and defect detection is performed on the target detection image based on the target defect detection model to obtain a target detection result, which includes a target bounding box, a target defect category and a target defect mask data, so as to accurately and efficiently detect scratch defects on the inner wall of the pin hole. In this way, the present application can effectively improve the efficiency and accuracy of defect detection of the object to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a defect detection method provided by an embodiment of the present application;

[0050] Figure 2 This is a schematic diagram of a control system for detecting defects on the inner wall of a pin hole in an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of a host computer software solution corresponding to the defect detection of the inner wall of the pin hole in the embodiment of the present application;

[0052] Figure 4 yes Figure 1 A flowchart of step S120 in FIG.

[0053] Figure 5 yes Figure 4A flowchart of step S420 in FIG.

[0054] Figure 6 It is a flow chart of a method for detecting defects on the inner wall of a pin hole provided in an embodiment of the present application;

[0055] Figure 7 yes Figure 1 A flowchart of step S130 in FIG.

[0056] Figure 8 is a schematic diagram of a model structure of an image defect detection model provided in an embodiment of the present application;

[0057] Fig.9A This is a convolution structure diagram of standard convolution;

[0058] Fig. 9B It is a schematic diagram of a convolution structure of deep convolution;

[0059] Fig. 9C It is a schematic diagram of a convolution structure of depth-separable convolution;

[0060] Fig. 10A It is a schematic diagram of the model structure of the standard convolution process;

[0061] Fig. 10B It is a schematic diagram of the model structure of the depth-separable convolution process;

[0062] Fig.11 is a structural schematic diagram of an improved multi-scale feature extraction layer provided in an embodiment of the present application;

[0063] Fig.12 Figure 7 A flowchart of step S750 in FIG.

[0064] Fig.13 is a specific flow chart for calculating the second loss value provided in an embodiment of the present application;

[0065] Fig.14 is a specific flow chart for determining the model loss value provided in an embodiment of the present application;

[0066] Fig.15 is a structural schematic diagram of a defect detection device provided in an embodiment of the present application;

[0067] Fig.16 It is a hardware structure diagram of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of the present application. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0070] First, some nouns involved in this application are analyzed:

[0071] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems.

[0072] Mask: refers to a technique for marking or filtering data. In neural networks, masks can be used to indicate which input data should be considered and which should be ignored. This can be used for tasks such as processing sequence data and filling data.

[0073] Binary mask: refers to an exact mask consisting of binary bits, or a pixel-level mask in image annotation, used to identify specific objects or regions in an image. These masks can be used to train models to detect and segment specific target instances, for example, to identify and locate an object in an image.

[0074] Instance: refers to a single object instance in an image or video, including its exact pixel boundaries. The goal of instance segmentation is to predict the exact pixel boundaries of each individual object in the image, thereby enabling pixel-level classification and recognition of each object.

[0075] Prior boxes: A set of boxes preset on the feature map for target detection, which can indicate the possible location and size of the target in the image. Therefore, by presetting a set of boxes of different sizes and proportions on the feature map, the accuracy of the model in detecting targets can be improved.

[0076] Defect detection refers to the technology of detecting defects on the surface of the object to be tested. By capturing images of the object to be tested and performing defect detection on the captured images, the defect position of the object to be tested can be repaired in time to ensure the use of the object to be tested in practical applications. For example, in a nuclear reactor, the control rod guide tube cotter pin is a key component that is mainly used to control the reaction rate of the nuclear reactor and maintain the safety of the nuclear reactor. When the control rod is inserted into the guide tube, the cotter pin is released and slides into the guide tube. This process opens the bottom of the guide tube, allowing the control rod to be inserted into the reactor core and adjust its position when necessary. When the reaction rate in the reactor needs to be slowed down, the control rod is inserted deeper to absorb neutrons and slow down the reaction. Conversely, when the reaction rate needs to be accelerated, the control rod is raised and the number of neutrons absorbed is reduced. The role of the cotter pin is to ensure that the control rod is correctly inserted into the guide tube, and the guide tube can only be opened when the control rod is inserted in place. This prevents the control rod from accidentally slipping out of the guide tube, causing the reactor to lose control. The cotter pin of the control rod guide tube needs to be inserted into the pin hole corresponding to the cotter pin of the control rod guide tube, and the wheel on the bolt or shaft is fixed on the forked part to ensure the accuracy and stability between the mechanical parts. Therefore, in the field of nuclear power equipment maintenance, the cotter pin of the control rod guide tube is inspected in service, that is, the image collected from the pin hole corresponding to the cotter pin of the control rod guide tube is inspected for defects to detect the quality status during its operation in real time, which can ensure the quality status of the control rod inserted into the guide tube during operation, thereby improving the reliability and safety of nuclear power plants.

[0077] The target pin hole is located in the upper core plate, which is part of the upper in-core component assembly and is stored on a special stainless steel storage rack in the component pool during overhaul. The shape and size of the component pool and the storage rack are known. The bottom and inner wall of the component pool are coated with stainless steel. For example, the surface roughness of the stainless steel coating is about Ra6.3 (referring to a roughness of 6.3 microns). The storage rack is supported by 4 columns with a diameter of 210 mm. There are 122 pin holes in total, so the target pin holes need to be positioned. The thickness of the upper core plate is 76 mm, and the diameter of the pin hole is 21 mm. There are no obstacles in the pin hole during detection. The full depth needs to be detected, focusing on the wear, scratches, burrs, etc. of the inner wall in the upper 41 mm area. At present, the defect detection method of the pin hole corresponding to the cotter pin of the control rod guide tube in the relevant technology usually adopts a manual operation method, that is, manually judging the image collected from the pin hole corresponding to the cotter pin of the control rod guide tube. This defect detection method is low in efficiency and low in detection accuracy.

[0078] Based on this, the embodiments of the present application provide a defect detection method and device, computer equipment, and storage medium, which can improve the efficiency and accuracy of defect detection of the object to be detected, and further improve the efficiency and accuracy of visual detection of surface defects on the inner wall of the cotter pin hole of the control rod guide tube.

[0079] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0080] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0081] The defect detection method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The defect detection method provided in the embodiment of the present application can be applied to the terminal, can be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms and other basic cloud computing services; the software can be an application that implements the defect detection method, etc., but is not limited to the above forms.

[0082] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (Personal Computer, PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0083] See also Figure 1 , Figure 1 is an optional flow chart of the defect detection method provided in the embodiments of the present application. In some embodiments of the present application, Figure 1 The method may specifically include but is not limited to steps S110 to S160. Figure 1 These six steps are introduced in detail.

[0084] Step S110, obtaining training sample data;

[0085] Step S120, extracting defect regions from the sample defect image based on the defect region extraction model to obtain a sample defect location image;

[0086] Step S130, performing defect detection on the sample defect location image based on the image defect detection model to obtain a sample detection result;

[0087] Step S140, calculating a first loss value based on the sample defect mask data and the predicted defect mask data, calculating a second loss value based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and determining a model loss value based on the first loss value and the second loss value;

[0088] Step S150, updating the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and constructing a target defect detection model based on the updated defect region extraction model and the image defect detection model;

[0089] Step S160, acquiring a target detection image, and performing defect detection on the target detection image based on a target defect detection model to obtain a target detection result.

[0090] In step S110 of some embodiments, when constructing a target defect detection model, a sample data set may be first obtained, and the sample data set includes a plurality of training sample data. The training sample data includes a sample defect image of a sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data. The sample bounding box is used to indicate the area information of the defective part of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defective part of the sample object to be detected in the sample bounding box.

[0091] It should be noted that the sample defect image of the sample object to be detected can be collected by using an image acquisition device. For example, if the sample object to be detected is a pin hole, when collecting the image of the inner wall of the pin hole, the visual guidance of the video detection probe can be used. By adjusting the position of the two-dimensional positioning module extended into the pin hole, the axis of the video detection probe is aligned with the axis of the pin hole, and then the probe is stepped into the pin hole to obtain the image of the inner wall of the pin hole, that is, to obtain the initial defect image of the sample.

[0092] Furthermore, after obtaining multiple sample initial defect images, the present application can pre-process the multiple sample initial defect images to obtain sample defect images. Specifically, the present application can perform image data enhancement on multiple sample initial defect images, that is, use the axis of the video detection probe as the center reference, identify the outer edge of the detected pin hole and make a calibration circle, and select a second calibration circle coaxially within the camera field of view. Furthermore, the image within the calibration circle range can be expanded, and a range with less distortion can be selected for defect detection, that is, the selected image is used as a sample defect image to eliminate the distortion effect of the underwater environment on the image taken by the wide-angle camera. The present application uses the probe to step into the pin hole, which can realize a full-range inspection of the inner wall of the pin hole, and the inspection process can be recorded to ensure the integrity and accuracy of the detection image.

[0093] It should be noted that the present application can select a range with smaller distortion according to the camera field of view, reduce the selection of distorted images, and thus construct a sample data set based on valid sample defect images. In addition, considering the dim environment inside the pin hole, the front end of the probe used can use a ring-shaped lighting lamp to provide uniform lighting, so as to ensure that a single image acquisition can image the entire 360° range within a certain axial interval of the inner wall, and the inspection process can be recorded. In addition, the probe can be connected to the ground computer through a signal line, so as to ensure that the transmitted images can be developed secondary.

[0094] It should be noted that the sample defect category is used to characterize the possible defect categories and non-defect categories of the sample object to be tested. For example, if the sample object to be tested is the inner wall of a pin hole, the corresponding sample defect categories include wear, scratches, burrs, etc.

[0095] It should be noted that the defect mask data of the present application is a binary mask data, which can accurately determine the position of the defect target in the sample defect image. The mask is usually a binary image with the same size as the bounding box, and each pixel value in the mask data indicates whether the pixel belongs to the target object, that is, the specific location information. The mask data of the present application is usually a binary string, which plays a positioning role in the algorithm, converts the location information into a binary number, and gives it to the host computer, which is not displayed in the host computer. The sample defect mask data is used to indicate the specific location information of the defect target in the sample defect image of the sample object to be detected. The area with a value of 1 in the defect mask data is the position of the defect target. The specific position and shape of the defect target can be obtained by a simple coordinate transformation of the coordinate with a value of 1.

[0096] It should be noted that the sample bounding box is used to indicate the rough position information of the defect target in the sample defect image of the sample object to be detected, and can provide the approximate position and size information of the defect target. By first determining the bounding box, the image area to be processed can be reduced, thereby reducing the amount of calculation, improving processing efficiency, and helping to accelerate subsequent processing steps. The mask data and bounding box of this application both have their own important roles, and can provide key information for instance segmentation and target detection, respectively. That is to say, although the binary mask data can determine the position of the defect target, it is still necessary to output the bounding box. Because there may be multiple targets in the image, the bounding box can help distinguish different defect targets. Although the binary mask data can segment the defect target to obtain location information, the bounding box provides a simple and effective way to manage and locate multiple targets. This synergy makes the model more accurate and efficient when performing instance segmentation.

[0097] It should be noted that if the sample object to be tested is the inner wall of a pin hole, refer to Figure 2 , Figure 2 The schematic diagram of the control system for defect detection on the inner wall of the pin hole in the embodiment of the present application is shown. In the video detection module, the camera field of view of the panoramic gimbal can be used to collect images of the sample object to be detected, and the defect detection method of the present application can be executed in the pin hole detection module to realize defect detection on the inner wall of the pin hole.

[0098] It should be noted that, refer to Figure 3 , Figure 3The schematic diagram of the upper computer software solution corresponding to the defect detection of the inner wall of the pin hole in the embodiment of the present application is shown. In practical applications, the pin hole probe module in the schematic diagram of the upper computer software solution can execute the defect detection method of the present application. The present application adopts the technical solution of industrial wireless communication control and CAN bus wired communication control dual standards, which can effectively improve the reliability of the system. In addition, before checking the current pin hole, the next pin hole can be photographed, and the positions of other pin holes can be marked. Through coordinate calculation, the position information of the robot and other pin holes can be obtained, and the positions of the pin holes that have been operated, are being operated, and are to be operated are marked in the upper computer interface, and each pin hole is traversed in the shortest path, thereby improving the efficiency of the detection operation.

[0099] It should be noted that after obtaining multiple training sample data, the present application can uniformly expand the sample defect images therein and adjust the pixels of the expanded images. Furthermore, the multiple training sample data can be divided into training set, test set, and validation set. Among them, the training set is the original sample data set used to train the model. In this set, the model will try to learn the inherent laws and patterns of the data, and optimize the performance of the model on the training set by continuously adjusting the model parameters. The validation set is a part of the data separated from the sample data set, which is used to evaluate the performance of the model and adjust the model parameters during the model training process to avoid overfitting. The test set is a data set used to finally evaluate the performance of the model. The main purpose is to evaluate the generalization ability of the model, that is, the performance of the model on unseen data. The test set should have a sufficient sample size to ensure the reliability of the evaluation results.

[0100] In step S120 of some embodiments, the defect area extraction model refers to a model that can extract defect areas in a defect image. Before inputting the sample defect image into the defect area extraction model, the sample defect image can be preprocessed first. The preprocessing at this time can be scaling, cropping, denoising, etc. to meet the input requirements of the model. The defect area extraction model of the present application can be a YOLO model, a YOLOv9 model, a fast regional convolutional neural network (Faster Region-based Convolutional Network, Faster R-CNN), etc., which are not specifically limited here. The present application extracts defect areas from sample defect images through a defect area extraction model, which can quickly and accurately identify defect areas in the image, and further extract features of the identified defect areas in the subsequent image defect detection model.

[0101] Please refer to Figure 4 , Figure 4 is a specific flow chart of step S120 provided in an embodiment of the present application. In some embodiments of the present application, step S120 may specifically include but is not limited to steps S410 to S420. Figure 4 These two steps are introduced in detail.

[0102] Step S410, extracting defect regions from the sample defect image based on the defect region extraction model to obtain a first sample positioning bounding box and a first sample positioning reliability;

[0103] Step S420: performing image extraction on the sample defect image based on the first sample positioning bounding box and the first sample positioning reliability to obtain a sample defect positioning image.

[0104] In step S410 and step S420 of some embodiments, the defect area extraction model can detect and mark the defect area in the image, and generate a first sample positioning bounding box to accurately locate the position of the defect target in the sample defect image, and also generate a first sample positioning confidence score, which is used to indicate the degree of confidence of the defect area extraction model in the detection accuracy. Further, the sample defect image can be extracted based on the first sample positioning bounding box and the first sample positioning confidence score. Specifically, a partial image related to the defect can be extracted from the sample defect image according to the first sample positioning bounding box, and the accuracy of the extracted image can be determined according to the first sample positioning confidence score, so that the obtained sample defect positioning image can more accurately reflect the location and characteristics of the defect.

[0105] It should be noted that the positioning bounding box (Bounding Box) is a rectangular box used to mark the position of the defective target in the image. This rectangular box is usually represented by four coordinates: the coordinates of the center point of the bounding box (x, y), and the width (w) and height (h) of the bounding box. These coordinates can be normalized values ​​relative to the image size or absolute pixel coordinates. Positioning confidence (Confidence): is a numerical value that can indicate the degree of confidence of the model that the predicted bounding box contains defective targets. Confidence is usually a floating point number between 0 and 1, where 1 means that the model has complete confidence in the target in the bounding box and 0 means no confidence at all. Confidence not only takes into account whether the bounding box contains the target object, but also the accuracy of the bounding box, that is, the degree of overlap between the bounding box and the actual target boundary.

[0106] In the above embodiment, the present application can first use a defect area extraction model (such as a YOLOv9 model, FasterR-CNN, etc.) to perform a full-domain detection on a sample defect image (such as an image of the area to be detected of the internal image of a pin hole), and predict a first sample positioning bounding box and a first sample positioning confidence (i.e., category confidence) in a single neural network to achieve defect target detection and preliminary positioning of defects in the sample object to be detected (such as the inner wall of the pin hole). In addition, the defect area extraction model can improve the accuracy of defect area extraction through improved network structure, training strategy, or data enhancement technology, so that under the same conditions, the defect target can be more accurately located, the missed detection rate can be reduced, and the accuracy of defect detection can be improved. Furthermore, the defect area extraction model can also achieve a faster inference speed at the same accuracy rate by optimizing the network structure.

[0107] Please refer to Figure 5 , Figure 5 is a specific flow chart of step S420 provided in an embodiment of the present application. In some embodiments of the present application, step S420 may specifically include but is not limited to steps S510 to S530. Figure 5 These three steps are introduced in detail.

[0108] Step S510, if the first sample positioning reliability is less than a preset reliability threshold, multi-scale feature fusion is performed on the sample defect image based on a pre-trained multi-scale feature fusion model to obtain a sample fusion feature image;

[0109] Step S520, extracting the defect area from the sample fusion feature image based on the defect area extraction model to obtain a second sample positioning bounding box and a second sample positioning reliability;

[0110] Step S530 , performing image extraction on the sample defect image based on the second sample positioning bounding box and the second sample positioning reliability to obtain a sample defect positioning image.

[0111] In step S510 of some embodiments, the preset confidence threshold refers to the threshold that the confidence corresponding to the extracted sample defect location image needs to meet. The higher the preset confidence threshold, the higher the reliability of the obtained sample defect location image. At this time, the obtained sample defect location image can not only contain the defect target more accurately, but also be closer to the real bounding box. If the first sample location confidence is less than the preset confidence threshold, it means that the defect area extraction model is not confident or accurate enough in the location extraction of the defect area in the sample defect image. At this time, the sample defect image can be multi-scale feature fusion based on the pre-trained multi-scale feature fusion model to obtain a sample fusion feature image. The multi-scale feature fusion model can be a deep neural network, which is designed to extract feature information of different scales and levels from the sample defect image. Then, by using the multi-scale feature fusion technology, the features of the image can be fused and integrated at different resolutions to obtain a sample fusion feature image that can more comprehensively understand the image content and structure, which helps to improve the system's understanding of the defect area and the accuracy of positioning. Therefore, the sample fusion feature image obtained at this time can contain feature information of multiple scales, providing a more comprehensive and rich feature representation for subsequent steps, thereby improving the accuracy and reliability of defect detection and positioning.

[0112] It should be noted that the preset confidence threshold of the present application may also include a preset low confidence threshold. If the positioning confidence of the first sample is less than the preset low confidence threshold, it means that the model has a low degree of confidence in the detected target. At this time, it can be explained that there are major problems with the sample defect image, and the image can be re-captured to avoid subsequent invalid operations that waste computer resources.

[0113] In steps S520 to S530 of some embodiments, after obtaining the sample fusion feature image, the present application can perform defect area extraction on the sample fusion feature image based on the defect area extraction model to obtain a second sample positioning bounding box and a second sample positioning reliability. At this time, the forms of the second sample positioning bounding box and the second sample positioning reliability are respectively the same as those of the first sample positioning bounding box and the first sample positioning reliability, except that the input of the model is different, which will not be repeated here. Furthermore, image extraction can be performed on the sample defect image based on the second sample positioning bounding box and the second sample positioning reliability to obtain a sample defect positioning image, and the second sample positioning reliability at this time is greater than or equal to a preset reliability threshold.

[0114] In step S130 of some embodiments, the sample detection result includes a predicted bounding box, a predicted defect category, and predicted defect mask data. The specific explanation of the bounding box, defect category, and defect mask data can be found in the description of the above embodiment, and will not be repeated here. Figure 6 , Figure 6A flow chart of a method for defect detection on the inner wall of a pin hole provided by an embodiment of the present application is shown. Among them, the present application can first obtain an image of the inner wall of the pin hole and perform preprocessing to obtain a sample defect image, and then perform category labeling on the sample defect image to construct a sample data set, and construct a training set, a test set and a validation set based on the sample data set, and the training set includes multiple training sample data. Furthermore, the defect area extraction model is used to extract the defect area of ​​the sample defect image to obtain a sample defect location image, and the sample defect location image is input into the image defect detection model for defect detection to obtain a sample detection result. After the training is completed, the target defect detection model 610 can be constructed based on the defect area extraction model and the image defect detection model after parameter update, and the target defect detection model 610 can be evaluated using the test set and the validation set.

[0115] Please refer to Figure 7 , Figure 7 is a specific flow chart of step S130 provided in an embodiment of the present application. In some embodiments of the present application, step S130 may specifically include but is not limited to steps S710 to S750. Figure 7 These five steps are introduced in detail.

[0116] Step S710, extracting features from the sample defect location image based on the image feature extraction layer to obtain the sample initial features;

[0117] Step S720, performing multi-scale feature extraction on the sample initial features based on the multi-scale feature extraction layer to obtain a plurality of sample candidate features of different dimensions;

[0118] Step S730, performing regional target detection on the sample initial features based on the region detection layer to obtain a plurality of sample candidate region bounding boxes;

[0119] Step S740, performing spatial feature extraction on a plurality of sample candidate features of different dimensions and a plurality of sample candidate region bounding boxes based on the spatial feature extraction layer to obtain sample spatial features;

[0120] Step S750, performing defect detection on the sample space features based on the classification layer to obtain a sample detection result.

[0121] In steps S710 to S750 of some embodiments, reference is made to Figure 8 , Figure 8A schematic diagram of a model structure of an image defect detection model provided by an embodiment of the present application is shown, and the image defect detection model includes an image feature extraction layer, a multi-scale feature extraction layer, a regional detection layer, a spatial feature extraction layer and a classification layer. Among them, the image feature extraction layer of the present application can be a structure constructed based on the improved Mobilenet attention mechanism, and the improved Mobilenet attention mechanism refers to an improved structure based on the single-stage target detector (Single Shot Multibox Detector, SSD), which can integrate the idea of ​​deep decomposable convolution in the network and replace the basic network in the traditional SSD for feature extraction. Therefore, the improved image feature extraction layer is a lightweight target detection algorithm with a small model size and an efficient calculation method, which can achieve a faster reasoning speed. When used in combination with other structures of the image defect detection model, the computational efficiency of the defect detection device can be improved while maintaining a high detection accuracy, and it is better suitable for scenes requiring real-time target detection and instance segmentation. At the same time, the improved image feature extraction layer performs well in resource-constrained environments, such as mobile devices, embedded systems, etc. Combining it with other structures of the image defect detection model can achieve more comprehensive object detection and segmentation tasks in these environments, providing more possibilities and better adaptability for mobile applications.

[0122] It should be noted that, refer to Figure 9A-9C , shows some convolution structure schematic diagrams provided in the embodiments of the present application, where it is assumed that the input feature map is (D F ×D F ×M), the convolution kernel size is (D K ×D K ×N), where M and N are the number of input and output channels respectively, D F and D K is the corresponding number of rows and columns. Based on this, the corresponding amount of calculation is: the standard convolution is D F ×D F ×M×D K ×D K ×N, the depth of the convolution is D F ×D F ×M×D K ×D K , 1×1 point convolution is D F ×D F ×M×N, the depth-wise separable convolution is (D F ×D F ×M×D K ×D K )+(D F ×D F ×M×N). Reference Fig. 10A and Fig. 10B , Fig. 10A shows a schematic diagram of the structure of the standard convolution process, Fig. 10B A structural schematic diagram of the deep separable convolution process is shown. The embodiment of the present application combines the improved Mobilenet attention mechanism (i.e., SSD_Mobilenet) obtained by the deep separable convolution process, and uses the convolution layer and pooling layer of the improved Mobilenet attention mechanism as the structure of the image feature extraction layer. In this process, the present application can appropriately adjust the size and number of channels of the output feature map in the image feature extraction layer to adapt to the input requirements of the image defect detection model, so as to better achieve fast and efficient real-time target detection and instance segmentation tasks. The improved Mobilenet of the present application can integrate the idea of ​​deep decomposition convolution in the network, use Mobilenet to replace the basic network VGG16 in the traditional SSD for feature extraction, and remove the last fully connected layer and Softmax in the network, so as to better achieve fast and efficient real-time target detection and instance segmentation tasks.

[0123] In step S720 of some embodiments, the multi-scale feature extraction layer may be a model structure obtained by optimizing a feature pyramid network (FPN) based on a compression and excitation module (Squeeze-and-Excitation, SE). The SE module is an attention mechanism for enhancing a deep neural network model, which aims to effectively adjust the importance of feature maps by learning channel correlations from feature maps in the middle layer of the network, so as to increase the model's attention to important features. Fig.11 , because FPN consists of three parts: a bottom-up convolutional neural network process, a top-down convolutional neural network process, and a side connection process between features, the present application can add a SE module between the corresponding layers of the FPN bottom-up convolutional neural network process and the top-down convolutional neural network process, and perform feature fusion between feature layers of different scales. The SE module can learn the importance of each channel, thereby adjusting and optimizing the correlation between channels of the input feature map. During the training process, the SE module can adaptively learn the weight of each channel, thereby helping the deep neural network model to better capture and utilize feature information, and improve the performance and generalization ability of the model. Among them, the sample candidate features of multiple different dimensions obtained based on the multi-scale feature extraction layer contain the semantic and local feature information of the image, with different resolutions and semantic levels.

[0124] In step S730 of some embodiments, after obtaining the sample initial features, the present application may perform regional target detection on the sample initial features based on the region detection layer to obtain multiple sample candidate region bounding boxes. The region detection layer may be constructed based on a region proposal network (RPN) to generate multiple sample candidate region bounding boxes that may contain defect targets.

[0125] In step S740 and step S750 of some embodiments, the spatial feature extraction layer can extract spatial features from multiple sample candidate features of different dimensions and multiple sample candidate region bounding boxes to obtain sample spatial features. The spatial feature extraction layer can be constructed based on the region of interest alignment (RoI Align) method. RoI Align can be used to extract features on RoIs of different sizes instead of using the original RoI pooling method to more accurately capture the spatial information of defect targets. Further, defect detection is performed on the sample spatial features based on the classification layer to obtain sample detection results.

[0126] Please refer to Fig.12 , Fig.12 is a specific flow chart of step S750 provided in an embodiment of the present application. In some embodiments of the present application, step S750 may specifically include but is not limited to steps S1210 to S1230. Fig.12 These three steps are introduced in detail.

[0127] Step S1210, performing pixel-level prediction on sample space features based on the mask prediction branch of the classification layer to obtain predicted defect mask data;

[0128] Step S1220, performing category prediction on the sample space features based on the category prediction branch of the classification layer to obtain a predicted defect category;

[0129] Step S1230 , performing bounding box prediction on the sample space features based on the bounding box prediction branch of the classification layer to obtain a predicted bounding box.

[0130] In step S1210 of some embodiments, Figure 8 As shown, the mask prediction branch of the present application refers to a structure including a mask head and an optimized structure based on a lightweight depth-separable convolution. Among them, the mask head is used to generate an accurate mask of the target instance. The lightweight depth-separable convolution can adopt the improved Mobilenet or other model structures that can generate accurate masks in the above embodiments, which will not be repeated here. In this way, more accurate mask data can be obtained, and accurate pixel-level segmentation of defective targets can be achieved, thereby improving the instance segmentation accuracy of defect detection.

[0131] It should be noted that the overall architecture of the image defect detection model of the present application can be constructed based on Region-Convolutional Neural Networks (R-CNN).

[0132] In step S1220 and step S1230 of some embodiments, the category prediction branch of the classification layer performs category prediction on the sample space features to obtain a predicted defect category to accurately calibrate the defect category of the defect target in the sample defect image. At the same time, the bounding box prediction branch of the classification layer performs bounding box prediction on the sample space features to obtain a predicted bounding box to accurately circle the position of the defect target.

[0133] In step S140 of some embodiments, after obtaining the sample detection result, the present application can calculate a first loss value based on the sample defect mask data and the predicted defect mask data, calculate a second loss value based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and determine the model loss value based on the first loss value and the second loss value.

[0134] It should be noted that the process of calculating the first loss value based on the sample defect mask data and the predicted defect mask data can be shown in the following formula 1:

[0135]

[0136] Among them, N1 is the number of pixels corresponding to the sample defect positioning image, n is the number of data contained in the sample defect mask data, and y i is the i-th mask value corresponding to the sample defect mask data, p i To predict the i-th mask value corresponding to the defect mask data, Loss1 is the loss calculated only for the pixel position of the defect target instance in the sample curve positioning image. In this way, background pixels can be ignored and the amount of training calculation can be reduced.

[0137] The present application determines the model loss value used for model parameter adjustment by combining the first loss value determined by the instance segmentation task and the second loss value of the target real-time detection task, which can improve the detection accuracy of the model in practical applications.

[0138] Please refer to Fig.13 , Fig.13 is a specific flow chart of calculating the second loss value provided by the embodiment of the present application. In some embodiments of the present application, the process of calculating the second loss value may specifically include but is not limited to steps S1310 to S1350. Fig.13 These five steps are introduced in detail.

[0139] Step S1310, extracting a priori boxes based on the initial features of the sample to obtain multiple candidate priori boxes;

[0140] Step S1320, performing confidence loss calculation based on a preset defect category, multiple candidate prior boxes and confidence prediction values ​​to obtain a first sub-loss value;

[0141] Step S1330, performing position loss calculation based on the predicted bounding box, the sample bounding box, and the plurality of candidate prior boxes to obtain a second sub-loss value;

[0142] Step S1340, performing category loss calculation based on the predicted defect category and the sample defect category to obtain a third sub-loss value;

[0143] Step S1350, performing weighted calculation on the first sub-loss value, the second sub-loss value and the third sub-loss value to obtain a second loss value.

[0144] In steps S1310 to S1350 of some embodiments, the multiple candidate prior frames generated by the present application include multiple positive sample prior frames. The present application can perform confidence loss calculation based on the preset defect category, multiple positive sample prior frames and confidence prediction values ​​to obtain a first sub-loss value, and the calculation process of the first sub-loss value is shown in the following formula 2:

[0145]

[0146] Among them, Loss2 represents the second loss value, N2 represents the number of positive sample prior frames, and L conf Represents the confidence loss value, that is, the first sub-loss value, L loc Represents the regression loss value, that is, the second sub-loss value, L p represents the category loss, that is, the third sub-loss value, c represents the confidence prediction value; l represents the location information of the predicted bounding box, g is the location parameter of the true target under the correct category annotation, that is, the location information of the sample bounding box, α1, α2, α3 are the weight coefficients corresponding to each loss value, and the sum of the weight coefficients can be 1.

[0147] It should be noted that the positive sample prior box refers to a candidate prior box whose intersection over union (IOU) with the true bounding box (i.e., the sample bounding box) is greater than a set threshold in the defect detection task.

[0148] It should be noted that for x, Assume that N2 matches are selected, then The i in represents the i-th default value, j represents the j-th sample bounding box, and p represents the p-th defect category.

[0149] It should be noted that Lconf It includes two parts: positive and negative sample errors. The process of calculating the first sub-loss value is shown in the following formula 3:

[0150]

[0151] Among them, the subscript Pos represents a positive sample, the subscript Neg represents a negative sample, that is, the background, and ecp represents a cumulative distribution function.

[0152] It should be noted that the calculation process of the second sub-loss value is shown in the following formula 4 and formula 5:

[0153]

[0154] Where k represents the kth defect category.

[0155] It should be noted that the third sub-loss value is the category loss, which is used to measure the difference between the model prediction result and the true label in the classification task, and is one of the optimization goals during model training. Common category loss functions include cross-entropy loss function (Cross-Entropy Loss), Softmax Loss, etc. The values ​​corresponding to α1, α2, and α3 in this application can be flexibly set according to actual needs without limitation.

[0156] Please refer to Fig.14 , Fig.14 is a specific flow chart for determining the model loss value provided by the embodiment of the present application. In some embodiments of the present application, the process of determining the model loss value may specifically include but is not limited to steps S1410 to S1430. Fig.14 These three steps are introduced in detail.

[0157] Step S1410, obtaining the instance segmentation task and defect detection accuracy value of the sample object to be detected;

[0158] Step S1420, determining a sample weight based on the instance segmentation task and the defect detection accuracy value;

[0159] Step S1430, performing weighted calculation according to the first weight, the first loss value, the second weight and the second loss value to obtain the model loss value.

[0160] In step S1410 of some embodiments, since the sample objects to be detected may come from different products or production batches, each sample object to be detected needs to perform an instance segmentation task (i.e., separate different defect target instances from the image) and measure the defect detection accuracy value (i.e., the error between the predicted result and the actual annotation). In this way, the instance segmentation task and the defect detection accuracy value may be the same or different in different sample objects to be detected.

[0161] In step S1420 of some embodiments, the process of determining sample weights based on instance segmentation tasks and defect detection accuracy values ​​can adopt a model approach, that is, the instance segmentation task and defect detection accuracy value are input into a pre-trained weight prediction model (a neural network model, not limited) for weight prediction to obtain sample weights, which include a first weight corresponding to the first loss value and a second weight corresponding to the second loss value.

[0162] In other embodiments, the process of determining sample weights based on instance segmentation tasks and defect detection accuracy values ​​can be done by table lookup, that is, by matching the corresponding sample weights according to the instance segmentation tasks and defect detection accuracy values ​​in a pre-set weight matching table. When designing the overall model loss value, the present application takes into account the performance of instance segmentation tasks and defect detection accuracy values ​​in defect detection tasks, and by adjusting the corresponding weight coefficients, the influence between the two can be flexibly balanced. For example, if you pay more attention to the accuracy of the instance segmentation task, you can increase the first weight; if you pay more attention to the inference speed and model size, you can increase the second weight.

[0163] In step S1430 of some embodiments, the process of calculating the model loss value according to the first weight, the first loss value, the second weight and the second loss value is shown in the following formula 6:

[0164] Loss total =α0Loss1+β0Loss2 (Formula 6)

[0165] Among them, Loss total Represents the model loss value, α0 represents the first weight, and β0 represents the second weight.

[0166] In step S150 of some embodiments, after obtaining the model loss value, the present application can update the model parameters of the defect region extraction model and the image defect detection model based on the model loss value until the preset end condition is met, and select the defect region extraction model and the image defect detection model corresponding to the minimum model loss value during the training process to construct the target defect detection model. Furthermore, the test set and the validation set can be input into the trained target defect detection model for defect detection to obtain the defect detection result.

[0167] It should be noted that the preset end condition can be that the model training ends after all training sample data are trained, or that the model training ends when the model loss value is less than a preset loss value threshold, and no specific limitation is made here.

[0168] In the above embodiments, when training the defect region extraction model and the image defect detection model, the present application takes into account the performance of the instance segmentation task and the defect detection accuracy value in the defect detection task, and by adjusting the corresponding weight coefficients, the influence between the two can be flexibly balanced. Moreover, by jointly determining the model loss value of the present application by combining the confidence loss, regression loss and category loss in the first loss value and the second loss value, the detection accuracy of the target defect detection model on the object to be tested can be effectively improved, and invalid data can be eliminated or preprocessed in the early processing stage of the defect region extraction model, which can effectively improve the defect detection efficiency of the object to be tested.

[0169] In step S160 of some embodiments, the target detection result includes a target bounding box, a target defect category, and target defect mask data, wherein instance segmentation of defect targets in the target detection image can be performed based on the target bounding box and the target defect mask data.

[0170] The defect detection method and device, computer equipment, and storage medium provided in the embodiments of the present application can utilize a series of candidate prior frames of different scales generated by the image defect detection model to classify the detected defective targets and regress the position coordinates, which greatly reduces the computational complexity of the model, improves the defect detection speed, and is more suitable for the needs of real-time detection in industry. Moreover, by combining the constructed model loss function, the result obtained by the predicted bounding box can be made closer and closer to the real sample bounding box during the training process, which can effectively improve the accuracy and efficiency of the detection of scratch defects on the inner wall of the pin hole. Among them, the improved Mobilenet has a smaller model size and parameter amount. The present application combines the improved Mobilenet with other structures of the image defect detection model (such as the improved Mask R-CNN) to further reduce the size of the overall model, reduce the resource consumption of model deployment and operation, and realize model lightweighting. In this way, it can be better applied to resource-constrained environments, realize more comprehensive target detection and instance segmentation tasks, and make the model have a wider range of application capabilities and applicability. Compared with the Fast R-CNN used in the related art, the improved target defect detection model of the present application can be used not only for target detection, but also for instance segmentation, so that it can not only detect defective targets more accurately, but also provide detailed segmentation information of defective targets, which is more suitable for application scenarios that require fine segmentation.

[0171] See also Fig.15 , Fig.15It is a schematic diagram of the module structure of the defect detection device provided in an embodiment of the present application. The device can implement the defect detection method of the above embodiment. The device includes an acquisition module 1510, a region extraction module 1520, a first detection module 1530, a loss calculation module 1540, an update module 1550 and a second detection module 1560.

[0172] The acquisition module 1510 is used to acquire training sample data, where the training sample data includes a sample defect image of a sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; the sample bounding box is used to indicate the area information of the defect part of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect part of the sample object to be detected in the sample bounding box;

[0173] The region extraction module 1520 is used to extract the defect region of the sample defect image based on the defect region extraction model to obtain a sample defect positioning image;

[0174] A first detection module 1530 is used to perform defect detection on the sample defect location image based on the image defect detection model to obtain a sample detection result, where the sample detection result includes a predicted bounding box, a predicted defect category, and predicted defect mask data;

[0175] A loss calculation module 1540 is used to calculate a first loss value based on the sample defect mask data and the predicted defect mask data, calculate a second loss value based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and determine a model loss value based on the first loss value and the second loss value;

[0176] An updating module 1550 is used to update the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and to construct a target defect detection model based on the updated defect region extraction model and the image defect detection model;

[0177] The second detection module 1560 is used to obtain a target detection image, and perform defect detection on the target detection image based on a target defect detection model to obtain a target detection result, which includes a target bounding box, a target defect category, and a target defect mask data.

[0178] It should be noted that the defect detection device of the embodiment of the present application is used to implement the defect detection method of the above embodiment. The defect detection device of the embodiment of the present application corresponds to the above defect detection method. Please refer to the above defect detection method for the specific processing process, which will not be repeated here.

[0179] The embodiment of the present application further provides a computer device, the computer device comprising: at least one memory, at least one processor, at least one computer program, the at least one computer program being stored in the at least one memory, and the at least one processor executing the at least one computer program to implement any one of the defect detection methods in the above embodiments. The computer device may be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0180] See also Fig.16 , Fig.16 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0181] The processor 1610 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0182] The memory 1620 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1620 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1620, and the processor 1610 calls and executes the defect detection method of the embodiment of the present application;

[0183] Input / output interface 1630, used to implement information input and output;

[0184] Communication interface 1640, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0185] Bus 1650 , which transmits information between various components of the device (e.g., processor 1610 , memory 1620 , input / output interface 1630 , and communication interface 1640 );

[0186] The processor 1610 , the memory 1620 , the input / output interface 1630 , and the communication interface 1640 are connected to each other in communication within the device via the bus 1650 .

[0187] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the defect detection method in the above embodiment.

[0188] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0189] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0190] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0191] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0193] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0194] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0195] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0196] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0199] The preferred embodiments of the present invention are described with reference to the accompanying drawings, but the scope of the present invention is not limited by the above. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A defect detection method, characterized in that: The method comprises: Acquire training sample data, the training sample data including a sample defect image of a sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; wherein the sample bounding box is used to indicate the area information of the defect part of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect part of the sample object to be detected in the sample bounding box; Extracting defect regions from the sample defect image based on a defect region extraction model to obtain a sample defect location image; Performing defect detection on the sample defect location image based on the image defect detection model to obtain a sample detection result; wherein the sample detection result includes a predicted bounding box, a predicted defect category, and predicted defect mask data; Calculating a first loss value based on the sample defect mask data and the predicted defect mask data, calculating a second loss value based on the sample defect category, the predicted defect category, the sample bounding box, and the predicted bounding box, and determining a model loss value based on the first loss value and the second loss value; updating the model parameters of the defect region extraction model and the image defect detection model based on the model loss value, and constructing a target defect detection model based on the updated defect region extraction model and the image defect detection model; A target detection image is acquired, and defect detection is performed on the target detection image based on the target defect detection model to obtain a target detection result; wherein the target detection result includes a target bounding box, a target defect category, and a target defect mask data.

2. The method according to claim 1, characterized in that The defect region extraction model is used to extract the defect region of the sample defect image to obtain a sample defect location image, including: Extracting defect regions from the sample defect image based on a defect region extraction model to obtain a first sample positioning bounding box and a first sample positioning confidence; Image extraction is performed on the sample defect image based on the first sample positioning boundary box and the first sample positioning reliability to obtain a sample defect positioning image.

3. The method according to claim 2, characterized in that The extracting the sample defect image based on the first sample positioning bounding box and the first sample positioning reliability to obtain the sample defect positioning image includes: If the first sample positioning reliability is less than a preset reliability threshold, multi-scale feature fusion is performed on the sample defect image based on a pre-trained multi-scale feature fusion model to obtain a sample fusion feature image; Extracting defect regions from the sample fusion feature image based on the defect region extraction model to obtain a second sample positioning bounding box and a second sample positioning reliability; Image extraction is performed on the sample defect image based on the second sample positioning boundary box and the second sample positioning reliability to obtain a sample defect positioning image.

4. The method according to any one of claims 1 to 3, characterized in that: The determining the model loss value based on the first loss value and the second loss value includes: Obtaining the instance segmentation task and defect detection accuracy value of the sample object to be detected; Determine a sample weight based on the instance segmentation task and the defect detection accuracy value, the sample weight comprising a first weight corresponding to the first loss value and a second weight corresponding to the second loss value; A weighted calculation is performed according to the first weight, the first loss value, the second weight and the second loss value to obtain a model loss value.

5. The method according to any one of claims 1 to 3, characterized in that: The image defect detection model includes an image feature extraction layer, a multi-scale feature extraction layer, a region detection layer, a spatial feature extraction layer, and a classification layer. The image defect detection model is used to perform defect detection on the sample defect location image to obtain a sample detection result, including: Extracting features from the sample defect location image based on the image feature extraction layer to obtain sample initial features; Based on the multi-scale feature extraction layer, multi-scale feature extraction is performed on the initial features of the sample to obtain a plurality of sample candidate features of different dimensions; Performing regional target detection on the sample initial features based on the region detection layer to obtain a plurality of sample candidate region bounding boxes; Based on the spatial feature extraction layer, spatial feature extraction is performed on the sample candidate features of multiple different dimensions and the multiple sample candidate region bounding boxes to obtain sample spatial features; Defect detection is performed on the sample space features based on the classification layer to obtain a sample detection result.

6. The method according to claim 5, characterized in that The performing defect detection on the sample space feature based on the classification layer to obtain a sample detection result includes: The mask prediction branch of the classification layer performs pixel-level prediction on the sample space features to obtain the predicted defect mask data; the mask prediction branch is a structure constructed based on lightweight depth-separable convolution optimization; Performing category prediction on the sample space feature based on the category prediction branch of the classification layer to obtain the predicted defect category; Based on the bounding box prediction branch of the classification layer, a bounding box prediction is performed on the sample space feature to obtain the predicted bounding box.

7. The method according to claim 5, characterized in that The sample detection result also includes a confidence prediction value, and the second loss value is calculated based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, including: Extracting a priori frames based on the initial features of the sample to obtain multiple candidate priori frames; Calculate the confidence loss based on the preset defect category, the plurality of candidate prior boxes and the confidence prediction value to obtain a first sub-loss value; Performing position loss calculation based on the predicted bounding box, the sample bounding box, and the plurality of candidate prior boxes to obtain a second sub-loss value; Performing a category loss calculation based on the predicted defect category and the sample defect category to obtain a third sub-loss value; The first sub-loss value, the second sub-loss value and the third sub-loss value are weightedly calculated to obtain the second loss value.

8. A defect detection device, characterized in that: The device comprises: an acquisition module, configured to acquire training sample data, wherein the training sample data includes a sample defect image of a sample object to be detected, a sample defect category, a sample bounding box, and sample defect mask data; the sample bounding box is used to indicate the area information of the defect part of the sample object to be detected in the sample defect image, and the sample defect mask data is used to indicate the pixel boundary of the defect part of the sample object to be detected in the sample bounding box; A region extraction module, used to extract defect regions from the sample defect image based on a defect region extraction model to obtain a sample defect positioning image; A first detection module, configured to perform defect detection on the sample defect location image based on an image defect detection model to obtain a sample detection result, wherein the sample detection result includes a predicted bounding box, a predicted defect category, and predicted defect mask data; a loss calculation module, configured to calculate a first loss value based on the sample defect mask data and the predicted defect mask data, calculate a second loss value based on the sample defect category, the predicted defect category, the sample bounding box and the predicted bounding box, and determine a model loss value based on the first loss value and the second loss value; An updating module, used for updating the model parameters of the defect area extraction model and the image defect detection model based on the model loss value, and constructing a target defect detection model based on the updated defect area extraction model and the image defect detection model; The second detection module is used to acquire a target detection image, and perform defect detection on the target detection image based on the target defect detection model to obtain a target detection result, wherein the target detection result includes a target bounding box, a target defect category, and a target defect mask data.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images