Defect recognition method and apparatus that integrates target detection model and image segmentation model

Abstract

This invention provides a defect identification method and apparatus that integrates a target detection model and an image segmentation model. The method includes: acquiring a surface image of a device to be inspected; inputting the surface image of the device to be inspected into a target detection model to obtain a defect detection result; inputting the surface image of the device to be inspected into an image segmentation model to obtain a defect segmentation result; and obtaining a defect identification result of the device to be inspected based on the defect detection result and the defect segmentation result. The target detection model is trained based on the surface image of a sample device and the defect detection label of the sample device, and the image segmentation model is trained based on the surface image of the sample device and the defect segmentation label of the sample device. This invention enables efficient and accurate device defect detection using computer vision, reduces errors in device defect detection, improves detection accuracy, and saves human resources.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a defect recognition method and apparatus that integrates target detection model and image segmentation model. Background Technology

[0002] Gears are a typical power transmission device in the automotive industry, playing a crucial role in every vehicle. Therefore, defects in gears will directly affect the performance and lifespan of the car.

[0003] Currently, there are many professional instruments for measuring gear parameters on the market. However, considering factors such as price and practicality, most manufacturers still use manual inspection to detect gear defects. This method often consumes a lot of human resources, is slow, and is prone to false detection and missed detection due to human eye fatigue, making it difficult to meet the needs of modern industrial mass production. Summary of the Invention

[0004] This invention provides a defect recognition method and apparatus that integrates target detection model and image segmentation model to solve the problems of low detection efficiency and low detection accuracy caused by manual defect detection in the prior art, and to achieve automatic and accurate defect detection.

[0005] This invention provides a defect recognition method that integrates target detection model and image segmentation model, comprising:

[0006] Acquire a surface image of the device to be inspected;

[0007] The surface image of the device to be inspected is input into the target detection model to obtain the defect detection result of the device to be inspected;

[0008] The surface image of the device to be inspected is input into the image segmentation model to obtain the defect segmentation result of the device to be inspected;

[0009] Based on the defect detection results and the defect segmentation results, the defect identification results of the device under test are obtained;

[0010] The target detection model is trained based on the surface image of the sample device and the defect detection label of the sample device, and the image segmentation model is trained based on the surface image of the sample device and the defect segmentation label of the sample device.

[0011] According to the defect recognition method that integrates an object detection model and an image segmentation model provided by the present invention, the training steps of the object detection model include:

[0012] Obtain a surface image of the sample device;

[0013] Based on multiple preset defect types, the surface image of the sample device is labeled with corresponding defect segmentation tags;

[0014] Based on the defect segmentation label, mark the surface image of the sample device with the corresponding defect detection label;

[0015] Based on the surface image of the sample device and the defect detection label of the sample device, a defect detection dataset is constructed;

[0016] Based on the defect detection dataset, the parameters of the original target detection model are pre-trained to obtain pre-trained parameters;

[0017] Based on the pre-trained parameters, the parameters of the improved target detection model are iteratively trained to obtain the target detection model;

[0018] The improved target detection model is obtained by improving the original backbone network of the original target detection model based on the original target detection model. The improvement of the original backbone network includes replacing at least one convolutional network layer in the original backbone network of the original target detection model with a deformable convolutional network layer, and adding a symmetric network structure composed of an encoder and a decoder to the original backbone network.

[0019] According to the defect recognition method integrating an object detection model and an image segmentation model provided by the present invention, the step of iteratively training the parameters of the improved object detection model based on the pre-trained parameters to obtain the object detection model includes:

[0020] The parameters of the improved target detection model are initialized based on the pre-trained parameters.

[0021] The parameters of other networks in the improved target detection model after fixed initialization, except for the improved backbone network, are used to train the parameters of the improved backbone network according to the defect detection dataset until the first preset termination condition is met, so as to obtain the training parameters of the improved backbone network.

[0022] Based on the training parameters of the improved backbone network, the parameters of the initialized improved target detection model are trained until the second preset termination condition is met, thus obtaining the target detection model.

[0023] According to the defect identification method that integrates target detection model and image segmentation model provided by the present invention, the step of acquiring the surface image of the sample device includes:

[0024] Obtain the original surface image of the sample device;

[0025] The original surface image is preprocessed and augmented to obtain the surface image of the sample device;

[0026] The preprocessing includes one or more combinations of normalization, histogram equalization, and noise reduction; the image augmentation includes one or more combinations of overlaying a randomly generated background image, random cropping, and random flipping.

[0027] According to the defect identification method that integrates target detection model and image segmentation model provided by the present invention, the loss function of the target detection model is generated based on regression loss function and classification loss function;

[0028] The regression loss function is generated based on the defect detection results of the sample equipment and the defect detection labels of the sample equipment, including angle loss, distance loss, shape loss, and cross-union ratio loss.

[0029] The classification loss function is constructed based on the zoom loss determined by the defect detection results and defect detection labels of the sample device.

[0030] According to the defect recognition method that integrates target detection model and image segmentation model provided by the present invention, for the current training iteration, the current learning rate of the target detection model is calculated based on a first learning rate and / or a second learning rate;

[0031] The first learning rate is calculated based on the current iteration number, the maximum iteration number, and the maximum learning rate;

[0032] The second learning rate is calculated based on the maximum learning rate, the minimum learning rate, the current iteration number, and the maximum iteration number.

[0033] According to a defect identification method that integrates a target detection model and an image segmentation model provided by the present invention, the step of obtaining the defect identification result of the device to be detected based on the defect detection result and the defect segmentation result includes:

[0034] The defect detection results are subjected to non-maximum suppression processing, and defect detection boxes with a cross-union ratio greater than a preset threshold are obtained from the processed defect detection results; the defect detection box is the bounding box of the detected defect region;

[0035] In the defect segmentation results, obtain the defect segmentation boxes with an intersection-union ratio greater than the preset threshold; the defect segmentation box is the bounding box of the segmented defect region.

[0036] The defect identification result is obtained based on the confidence level corresponding to the defect detection box and the confidence level corresponding to the defect segmentation box.

[0037] The present invention also provides a defect recognition device that integrates a target detection model and an image segmentation model, comprising:

[0038] The acquisition module is used to acquire surface images of the device to be inspected.

[0039] The detection module is used to input the surface image of the device to be inspected into the target detection model to obtain the defect detection result of the device to be inspected;

[0040] The segmentation module is used to input the surface image of the device to be inspected into the image segmentation model to obtain the defect segmentation result of the device to be inspected.

[0041] The identification module is used to obtain the defect identification result of the device under test based on the defect detection result and the defect segmentation result;

[0042] The target detection model is trained based on the surface image of the sample device and the defect detection label of the sample device, and the image segmentation model is trained based on the surface image of the sample device and the defect segmentation label of the sample device.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the defect identification method of fusing target detection model and image segmentation model as described above.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the defect identification method that fuses the target detection model and the image segmentation model as described above.

[0045] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the defect identification method that fuses the target detection model and the image segmentation model as described above.

[0046] The defect identification method and apparatus that integrates target detection model and image segmentation model provided by this invention inputs the surface image of the device to be inspected into the target detection model and the image segmentation model respectively to obtain the corresponding defect detection result and defect segmentation result of the device to be inspected; then, based on the obtained defect detection result and defect segmentation result, the defect identification result of the device to be inspected is obtained, thereby realizing the efficient and accurate detection of equipment defects using computer vision, reducing the error of equipment defect detection, improving detection accuracy, and saving human resources. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is one of the flowcharts illustrating the defect recognition method that integrates target detection model and image segmentation model provided by the present invention;

[0049] Figure 2 This is the second flowchart of the defect recognition method that integrates target detection model and image segmentation model provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the original image segmentation model in this invention;

[0051] Figure 4 This is a curve comparison between the predictions made by the fusion model provided by this invention and those made using a single method;

[0052] Figure 5 This is a flowchart illustrating the training method for the target detection model provided by the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of the improved target detection model provided by the present invention;

[0054] Figure 7 This is a comparison chart of the PR curves of the improved target detection model with added deformable convolution provided by this invention and the original target detection model;

[0055] Figure 8 This is a comparison chart of the PR curves of the improved target detection model provided by this invention and the original target detection model that only adds deformable convolution;

[0056] Figure 9 This is a flowchart illustrating the training method for the image segmentation model provided by the present invention;

[0057] Figure 10 This is a flowchart illustrating the training method for the target detection model provided by the present invention;

[0058] Figure 11 This is a schematic diagram of the defect recognition device that integrates target detection model and image segmentation model provided by the present invention;

[0059] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0061] In recent years, with the continuous advancement of intelligent manufacturing, AI+ (Artificial Intelligence Plus) industrial quality inspection has become an important application scenario in the field of industrial intelligence. The rapid development of machine vision technology has brought new possibilities to intelligent manufacturing, and its high efficiency and accuracy have led to its increasing application in industrial quality inspection. Considering how to combine machine vision with equipment defect detection to improve production efficiency, reduce scrap rates, and increase overall efficiency is key to promoting the development of the entire equipment defect detection industry.

[0062] Currently, the machine vision industry is still dominated by a few leading international companies. Some of these companies almost monopolize more than 50% of the global vision inspection market, providing solutions based on core components and technologies (operating systems, sensors, etc.). Although individual companies have made some technological progress in machine vision inspection solutions in recent years, there is still a significant gap compared to these leading international companies. Therefore, gear defect detection is of great significance for improving industrial quality inspection efficiency and ensuring product quality.

[0063] This invention aims to overcome the shortcomings of manual defect detection, which results in low detection efficiency and low detection accuracy. It provides a defect recognition method that integrates target detection model and image segmentation model. This method establishes a network framework for equipment defect detection through data preprocessing, data augmentation, model improvement and model fusion, so as to realize efficient and accurate equipment defect detection by using computer vision instead of human eyes.

[0064] The following is combined with Figure 1 This invention describes a defect identification method that integrates a target detection model and an image segmentation model.

[0065] Figure 1 This is one of the flowcharts illustrating the defect identification method provided in this embodiment, such as... Figure 1 As shown, the method includes:

[0066] Step 101: Obtain a surface image of the device to be inspected;

[0067] The device to be inspected can be any device that requires defect detection. For example, the device to be inspected can be a gear, bearing, etc., but this embodiment does not specifically limit it.

[0068] like Figure 2 As shown, after acquiring the surface image of the device to be inspected, a detection model is selected for the acquired data, and the data is processed by a fusion method of object detection and semantic segmentation. Specifically, an object detection model is selected to perform object detection on the surface image of the device to be inspected, and an image segmentation model is selected to perform semantic segmentation on the surface image of the device to be inspected. After obtaining the training results, the object detection model and the image segmentation model are fused to obtain the final result.

[0069] Step 102: Input the surface image of the device to be inspected into the target detection model to obtain the defect detection result of the device to be inspected;

[0070] Step 103: Input the surface image of the device to be inspected into the image segmentation model to obtain the defect segmentation result of the device to be inspected;

[0071] The target detection model is trained based on the surface image of the sample device and the defect detection label of the sample device, and the image segmentation model is trained based on the surface image of the sample device and the defect segmentation label of the sample device.

[0072] Before executing step 102, the initial target detection model can be trained to obtain the target detection model. Specifically, the initial target detection model can be constructed first, and then the surface image of the sample device and the defect detection labels marked on the surface image of the sample device can be collected. The initial target detection model can then be trained to obtain the target detection model.

[0073] The initial object detection model includes the original object detection model and / or an improved object detection model of the original object detection model; the original object detection model can be constructed based on PP-YOLOE (an improved single-stage anchor-free object detection model based on the YOLO model).

[0074] The PP-YOLOE detection model is a high-performance and easy-to-deploy industrial-grade advanced target model. It includes modules such as the backbone network CSPREpResNet (Cross Stage Partial-Regional Memory Network), the feature fusion network CSPPAN (Cross Stage Partial Path Aggregation network), the lightweight ET-Head (Efficient Task-aligned Head), and the improved dynamic matching algorithm TAL (Task Alignment Learning). Furthermore, a series of models have been designed for different application scenarios. The entire PP-YOLOE model series achieves industry-leading accuracy, speed, and cost-effectiveness.

[0075] After training the target detection model, the surface image of the device to be detected is input into the target detection model. The target detection model then performs defect detection on the device to be detected, and the defect detection result of the device to be detected is obtained.

[0076] Optionally, when inputting the surface image of the device to be inspected into the target detection model, the surface image can be directly input into the target detection model; alternatively, the surface image of the device to be inspected can be preprocessed first to obtain a surface image with higher quality, thereby improving the defect detection accuracy. It should be noted that whether or not to preprocess the surface image of the device to be inspected can be set according to the actual scenario, and this embodiment does not specifically limit it in this regard.

[0077] Similarly, a tactile image segmentation model can be obtained by first training an initial image segmentation model. Specifically, an initial image segmentation model can be constructed first, and then the surface image of the sample device and the corresponding defect segmentation labels marked on the surface image of the sample device can be collected. The initial image segmentation model can then be trained to obtain the image segmentation model.

[0078] like Figure 3 The diagram shows the structure of the initial image segmentation model. This model is built upon U-Net++ (a novel segmentation architecture based on nested and dense skip connections); specifically, the first convolutional network layer of U-Net++ is replaced with a deformable convolutional structure, resulting in the improved U-Net++ model, which is the initial image segmentation model.

[0079] U-Net++ is an improvement and innovation based on the U-Net network (image semantic segmentation network). It includes downsampling, upsampling, skip connections, and convolutional networks. Building upon the U-shaped network structure of the Encoder and Decoder in U-Net, it places convolutional layers in the skip paths, bridging the semantic gap between the Encoder and Decoder networks. It also designs a series of nested, dense skip pathways. This flexible network structure improves gradient flow and, combined with deep supervision, allows for model pruning.

[0080] After training the image segmentation model, the surface image of the device to be inspected is input into the image segmentation model. The image segmentation model then performs defect segmentation on the device to be inspected, and the defect segmentation result of the device to be inspected is obtained.

[0081] Step 104: Based on the defect detection results and the defect segmentation results, obtain the defect identification results of the device under test;

[0082] Optionally, after obtaining the defect detection results and defect segmentation results of the device under test, the defect detection results and defect segmentation results can be comprehensively considered, and the defect identification results of the device under test can be obtained based on the comprehensive consideration results.

[0083] Optionally, the comprehensive consideration of defect detection results and defect segmentation results can be to select the result with the highest confidence from the confidence of the defect detection box in the defect detection result and the confidence of the defect segmentation box in the defect segmentation result as the final defect identification result; or it can be to fuse the defect detection results and defect segmentation results and use the fused result as the final defect identification result. The specific settings are based on the actual scenario, and this embodiment does not impose specific limitations on this.

[0084] To further verify the effectiveness of the defect detection model provided in this embodiment, the defect detection method combining the target detection model and the semantic segmentation model in this embodiment is compared with the defect detection method based on a single defect detection model.

[0085] like Figure 4The figure shows a comparison of the PR (Precision-Recall) curves between the prediction results using the fusion model and the prediction results using only a single method. It can be seen that the defect detection method combined with object detection and semantic segmentation in this embodiment has higher detection accuracy and solves the shortcomings of the existing single model structure.

[0086] This embodiment inputs the surface image of the device to be inspected into the target detection model and the image segmentation model respectively to obtain the corresponding defect detection results and defect segmentation results of the device to be inspected. Then, based on the obtained defect detection results and defect segmentation results, the defect identification results of the device to be inspected are obtained. This establishes a framework for efficient and rapid gear defect detection in industrial production, realizes efficient and accurate equipment defect detection using computer vision, reduces equipment defect detection errors, improves detection accuracy, and saves human resources.

[0087] Based on the above embodiments, Figure 5 This is a flowchart illustrating the training method of the target detection model provided in this embodiment.

[0088] like Figure 5 As shown, the training steps for the object detection model include:

[0089] Step 501: Obtain a surface image of the sample device;

[0090] Optionally, after acquiring the surface image of the sample device, the surface image of the sample device can be directly input into the original target detection model; or the surface image of the sample device can be preprocessed and / or image augmented before being input into the original target detection model. It should be noted that whether or not the surface image of the sample device is preprocessed and / or image augmented can be set according to the actual scenario, and this embodiment does not specifically limit it.

[0091] The surface images of the sample equipment include images of defective surfaces as well as some images of normal surfaces.

[0092] Step 502: According to multiple preset defect types, mark the surface image of the sample device with corresponding defect segmentation labels;

[0093] The defect types in the defect surface images of the sample equipment are labeled and a label file is generated. During labeling, the labeling is performed according to the requirements of semantic segmentation, with each defect corresponding to multiple point coordinates, resulting in a segmentation (segmented region).

[0094] Semantic segmentation involves assigning a label to each point in an image based on its semantic meaning, thus distinguishing different types of objects in the image. It can be understood as a pixel-level classification task.

[0095] Semantic segmentation is a typical computer vision problem that involves taking some raw data (e.g., planar images) as input and transforming them into masks with highlighted regions of interest.

[0096] At this point, according to the requirements of semantic segmentation, the defect surface image of the sample device can be divided into a foreground region and a segmentation region.

[0097] The foreground region (which is equivalent to the background) is the part of the area without defects; the defective region will be segmented out as the segmentation region.

[0098] Obtaining defect segmentation labels can be done by labeling each pixel, specifying whether each pixel is labeled as 0 or 1. Here, 0 represents that the pixel is located in the foreground region; 1 represents that the pixel is located in the segmentation region.

[0099] This defect segmentation method yields the defect segmentation labels corresponding to the surface image markers of the sample device.

[0100] Step 503: Mark the surface image of the sample device with the corresponding defect detection label according to the defect segmentation label;

[0101] For the defect segmentation label corresponding to the surface image mark of the sample device, take the maximum and minimum values ​​of the X and Y coordinates corresponding to each defect region to generate a bounding box (bbox), which is the defect detection box of the target detection model.

[0102] Among them, the rectangular label box (bbox) is a rectangle that marks the target on the image. Common labeling formats include PascalVOC (top left and bottom right), COCO (top left width and height), and YOLO (center coordinates - width and height).

[0103] Ultimately, two types of labels can be obtained for the same defect area: defect segmentation labels and defect detection labels.

[0104] Step 504: Construct a defect detection dataset based on the surface image of the sample device and the defect detection label of the sample device;

[0105] Then, a defect detection dataset for the sample device is constructed using the defect surface images of the sample device and their corresponding defect detection labels, which is used for training the subsequent target detection model.

[0106] Step 505: Based on the defect detection dataset, pre-train the parameters of the original target detection model to obtain pre-trained parameters;

[0107] When training the original target detection model using the obtained defect detection dataset of the equipment, the loss value can be calculated through the loss function to obtain the pre-training parameters of the original target detection model.

[0108] Optionally, the loss function used to train the original object detection model can be jointly constructed based on the classification loss function and the regression loss function. The specific function structures of the classification loss function and the regression loss function can be determined according to the actual scenario. This embodiment does not specifically limit the selection of the loss function. For example, a modified loss function can be used for classification loss, and a bounding box regression function can be used for regression loss. Step 506: According to the pre-trained parameters, the parameters of the improved object detection model are iteratively trained to obtain the object detection model;

[0109] Optionally, after obtaining the pre-trained parameters, the parameters of the improved object detection model can be iteratively trained by backpropagation based on the loss function, and the object detection model can be obtained when the training reaches a certain convergence condition.

[0110] The improved target detection model is obtained by improving the original backbone network of the original target detection model based on the original target detection model. The improvement of the original backbone network includes replacing at least one convolutional network layer in the original backbone network of the original target detection model with a deformable convolutional network layer, and adding a symmetric network structure composed of an encoder and a decoder to the original backbone network.

[0111] like Figure 6 The diagram shows the structure of the improved target detection model. Specifically, it uses deformable convolution and multi-scale feature fusion techniques. In the feature extraction module of PP-YOLOE, the convolutional neural network (CNN) in the first layer ResBlock (Residual Block) of CSPREpResNet is replaced with a deformable convolutional network (DCN). An adjustment vector for the direction of the convolution kernel is added to the traditional neural network, making the convolution result more closely match the defect region.

[0112] like Figure 7As shown in the figure, the PR (Mean Average Precision) curves of the original object detection model with added deformable convolution and the original object detection model are compared. It can be clearly seen that the MAP (Mean Average Precision) of the object detection model with added deformable convolution (hereinafter referred to as DCN+PP-YOLOE) is higher than that of the original object detection model (hereinafter referred to as PP-YOLOE), indicating that the object detection model has a better prediction effect after using DCN.

[0113] Furthermore, in order to fully fuse features, a symmetric network structure (also known as U-Net) consisting of an encoder and a decoder is added to the original backbone network. This allows for the first multi-scale feature fusion of the feature maps after passing through CSPRepResNet using the U-Net concept, thus fully realizing information exchange between different receptive fields.

[0114] like Figure 8 As shown in the figure, the PR comparison curves of the improved object detection model (hereinafter also referred to as UNet+DCN+PP-YOLOE) and the original object detection model with added deformable convolution are shown. It can be clearly seen that the MAP of the improved object detection model after feature fusion by U-Net for device defect detection is significantly higher, indicating that the prediction effect of the object detection model is further improved after using U-Net feature fusion.

[0115] Figure 9 This is a flowchart illustrating the training method of the image segmentation model provided in this embodiment.

[0116] like Figure 9 As shown, after obtaining the defect segmentation labels, the training steps of the image segmentation model include:

[0117] Step 901: Construct a defect segmentation dataset based on the surface image of the sample device and the defect segmentation label of the sample device;

[0118] Then, a defect segmentation dataset for the sample device is constructed using the defect surface images of the sample device and their corresponding defect segmentation labels, which is used for training the subsequent image segmentation model.

[0119] Step 902: Based on the defect segmentation dataset, pre-train the parameters of the original image segmentation model to obtain pre-trained parameters;

[0120] When training the original image segmentation model using the obtained defect segmentation dataset of the device, the loss value can be calculated through a loss function to obtain the pre-training parameters of the deep learning model.

[0121] Optionally, the choice of the loss function used to train the original image segmentation model can be determined according to the actual scenario. For example, the cross-entropy loss function can be chosen to construct the loss function of the original image segmentation model, so as to calculate the loss value when performing multi-class classification. The cross-entropy loss function is as follows:

[0122]

[0123] Where p = [p0, ... p C-1 [Refers to the probability distribution, where each p] i The probability that a sample belongs to the i-th class is represented by y = [y0, ..., y]. C-1 ] refers to the one-hot encoded representation of the sample label, y i =1 indicates that the sample belongs to the i-th class; C represents the sample label.

[0124] Step 903: Based on the pre-trained parameters, iteratively train the parameters of the improved image segmentation model to obtain the image segmentation model;

[0125] After obtaining the pre-trained parameters, the parameters of the improved image segmentation model can be iteratively trained by backpropagation based on the loss function. Once the training reaches a certain convergence condition, the image segmentation model will be obtained.

[0126] The improved image segmentation model is obtained by improving the original backbone network of the original image segmentation model based on the original image segmentation model. The improvement of the original backbone network includes replacing at least one convolutional network layer in the original backbone network of the original image segmentation model with a deformable convolutional network layer. For example, replacing the first convolutional network structure of U-Net++ with a deformable convolutional structure, the improved U-Net++ model is obtained, which is the improved image segmentation model.

[0127] In this embodiment, defect segmentation labels corresponding to the surface image markings of the sample equipment are used to obtain defect detection labels, thus constructing a defect detection dataset for the sample equipment. Then, the parameters of the original target detection model are pre-trained using this defect detection dataset to obtain pre-trained parameters. Finally, based on the pre-trained parameters, the parameters of the improved target detection model are iteratively trained to obtain the target detection model. Simultaneously, a defect segmentation dataset for the sample equipment is constructed using the defect segmentation labels corresponding to the surface image markings. Then, the parameters of the original image segmentation model are pre-trained using this defect segmentation dataset to obtain pre-trained parameters. Finally, based on the pre-trained parameters, the parameters of the improved image segmentation model are iteratively trained to obtain the image segmentation model. This completes the training of both the target detection model and the image segmentation model, improving their training accuracy and efficiency, and further enabling efficient and accurate equipment defect detection using computer vision.

[0128] Based on the above embodiments, in this embodiment, the parameters of the improved target detection model are iteratively trained according to the pre-trained parameters to obtain the target detection model. Specifically, the obtained pre-trained parameters are transferred to the improved target detection model, and then the improved model network is trained using a defect detection dataset to obtain a parameter file suitable for equipment defect detection, thus completing the construction of the target detection model.

[0129] like Figure 10 The diagram shown is a flowchart illustrating the iterative training method for the target detection model provided in this embodiment.

[0130] like Figure 10 As shown, the iterative training steps of the object detection model include:

[0131] Step 1001: Initialize the parameters of the improved target detection model according to the pre-trained parameters;

[0132] Step 1002: Fix the parameters of other networks in the improved target detection model after initialization, except for the improved backbone network. Based on the defect detection dataset, train the parameters of the improved backbone network until the first preset termination condition is met, and obtain the training parameters of the improved backbone network.

[0133] Because the model structure has been replaced and added, the improved target detection model is trained by fine-tuning. Specifically, this includes: freezing the common feature extraction layer, that is, using the pre-trained parameters directly as the initialization parameters of the improved target detection model; and fine-tuning the parameters of the improved backbone network using the defect detection dataset, while keeping the parameters of other parts fixed until the first preset termination condition is met.

[0134] The training strategy can be a multi-GPU training strategy that combines single-precision floating-point (Float) and half-precision floating-point (Float16) and data parallelism to reduce the GPU memory occupied by the model and improve the computing speed.

[0135] Step 1003: Based on the training parameters of the improved backbone network, train the parameters of the initialized improved target detection model until the second preset termination condition is met, and obtain the target detection model.

[0136] After fine-tuning to meet the first preset termination condition, all layers are unfrozen, and the object detection model is fine-tuned and trained as a whole with a smaller learning rate until the second preset termination condition is met, finally obtaining the object detection model.

[0137] Similarly, following the training method described above, the parameters of the improved image segmentation model can be iteratively trained based on the pre-trained parameters to obtain the image segmentation model.

[0138] This embodiment first initializes the parameters of the improved object detection model based on pre-trained parameters. Then, it fixes the parameters of all networks in the initialized improved object detection model except for the improved backbone network, and trains the parameters of the improved backbone network using a defect detection dataset until a first preset termination condition is met, obtaining the training parameters of the improved backbone network. Finally, based on the training parameters of the improved backbone network, it trains the initialized parameters of the improved object detection model until a second preset termination condition is met, obtaining the object detection model. Then, iterative training is performed using the same method to obtain an image segmentation model, thereby improving the training accuracy and efficiency of both the object detection model and the image segmentation model, and further enabling efficient and accurate equipment defect detection using computer vision.

[0139] Based on the above embodiments, the acquisition of the surface image of the sample device in this embodiment includes: acquiring the original surface image of the sample device; preprocessing and image augmentation of the original surface image to obtain the surface image of the sample device; wherein, the preprocessing includes one or more combinations of normalization processing, histogram equalization processing and noise reduction processing; the image augmentation includes one or more combinations of overlaying a randomly generated background image, random cropping and random flipping.

[0140] The purpose of normalization is to change the range of image pixel values ​​from [0-255] to [0-1], thereby accelerating network training.

[0141] Optionally, the specific choice of image normalization processing can be set according to the actual scenario. For example, in this embodiment, the maximum and minimum value normalization method can be selected for image normalization processing.

[0142] The normalization method for maximum and minimum values ​​is to take the difference between the data and the minimum value of the column, and then divide by the range. The formula is as follows:

[0143]

[0144] Where norm is the normalization result, x i The values ​​represent the pixel values ​​of the image, with max(x) and min(x) representing the maximum and minimum values ​​of the image pixels, respectively.

[0145] In this process, due to uneven lighting, histogram equalization is performed on the image to ensure that the probability distribution of each gray value in the transformed image is the same, thereby enhancing the image contrast.

[0146] Histogram equalization alters the grayscale values ​​of pixels in an image by changing its histogram, primarily used to enhance the contrast of images with a narrow dynamic range. The original image may lack clarity because its grayscale distribution is concentrated in a narrow range. For example, an overexposed image will have its grayscale levels concentrated in the high-brightness range, while an underexposed image will have its grayscale levels concentrated in the low-brightness range. Histogram equalization transforms the original image's histogram into a more uniform (equalized) distribution, thus increasing the dynamic range of grayscale value differences between pixels and enhancing the overall image contrast. In other words, the basic principle of histogram equalization is to broaden the grayscale values ​​that have a large number of pixels (i.e., those that play a major role in the image) and merge the grayscale values ​​that have a small number of pixels (i.e., those that do not play a major role), thereby increasing contrast, making the image clearer, and achieving the desired enhancement.

[0147] Among them, noise reduction processing can be achieved by selecting mean filtering, using a low-pass filter to remove high-frequency signals in the training image, thereby helping to eliminate sharp noise in the image and achieving image noise reduction function.

[0148] Furthermore, various data augmentation strategies can be employed when training the object detection model to improve its performance through image augmentation. Image augmentation can include one or more combinations of techniques such as overlaying randomly generated background images, random cropping, and random flipping.

[0149] Among them, the superimposed randomly generated background image is based on the background characteristics and uses a random expansion method to generate a background image. The training samples are then placed in it at a random ratio to enhance the diversity of the data.

[0150] Optionally, the background image can be a randomly generated image. For example, in this embodiment, a black background can be selected as the generated background image; however, this embodiment does not impose any specific limitations on this.

[0151] Among them, random cropping is based on background characteristics and randomly cropped from the randomly expanded samples at a random ratio to generate a new round of training samples, further enhancing the diversity of the data.

[0152] Random flipping is based on the characteristic of vertical symmetry in the data, and a certain proportion of the training samples are flipped vertically.

[0153] In addition, random pixel content transformation can be performed on the training samples in this embodiment to further increase the diversity of the data.

[0154] In this embodiment, the original surface image of the sample device is preprocessed by one or more combinations of normalization, histogram equalization and noise reduction, and the original surface image of the sample device is augmented by one or more combinations of superimposing randomly generated background images, random cropping and random flipping, to obtain a richer surface image of the sample device. This improves the training accuracy and efficiency of the target detection model and the image segmentation model, and thus improves the accuracy of device defect detection.

[0155] Based on the above embodiments, the loss function of the target detection model in this embodiment is constructed based on a regression loss function and a classification loss function; wherein, the regression loss function is constructed based on the angle loss, distance loss, shape loss and intersection-union ratio loss determined by the defect detection results and defect detection labels of the sample device; the classification loss function is constructed based on the zoom loss determined by the defect detection results and defect detection labels of the sample device.

[0156] During the training of the object detection model, the loss function includes a regression loss function for predicting bounding boxes and a classification loss function for classifying defect types.

[0157] The regression function can be the SIoU Loss (Scylla-IoU Loss, an intersection-over-union loss function) of the corresponding proprietary Scylla-Net (a convolutional neural network that uses a genetic algorithm to define its architecture for a specific dataset with a given predefined layer type). This loss consists of four cost functions: angle cost, distance cost, shape cost, and intersection-over-union (IoU) cost. The regression loss function L is defined as follows: box for:

[0158]

[0159] Where Δ is the distance cost defined based on angle cost, Ω is the shape cost, and IoU is the intersection-union ratio cost.

[0160] The classification loss function can be the variable loss (VFL), and the formula for calculating the classification loss function VFL(·) is as follows:

[0161]

[0162] Where α is the loss weight value for foreground and background; p γ q represents the weights of different samples, with the loss weights for difficult samples increasing; p is the predicted IACS (IoU-aware classification score); q is the target IoU score.

[0163] It should be noted that when training object detection or image segmentation models, to address various imbalances and the presence of a small number of negative samples, an online hard example mining (OHEM) strategy can be employed for model training. This involves using OHEM to calculate the loss value and backpropagating it. The core of OHEM is selecting difficult samples as training data; expanding the dataset significantly improves the performance of OHEM.

[0164] Furthermore, when training object detection and image segmentation models using loss functions, directionality can be introduced into the cost of the loss function to achieve faster convergence during the training phase and better performance in inference.

[0165] This embodiment uses a regression loss function (angle loss, distance loss, shape loss, and cross-union ratio loss determined by the defect detection results and defect detection labels of the sample equipment) and a classification loss function (zoom loss determined by the defect detection results and defect detection labels of the sample equipment) to jointly construct a loss function for training the target detection model. Furthermore, it uses a cross-entropy loss function to train the image segmentation model, calculating the loss values ​​of both the target detection model and the image segmentation model. This allows for the rapid and accurate acquisition of the target detection model and the image segmentation model, thereby enabling efficient and accurate equipment defect detection using computer vision.

[0166] Based on the above embodiments, in this embodiment, for the current iteration of training, the current learning rate of the target detection model is calculated based on a first learning rate and / or a second learning rate; the first learning rate is calculated based on the current iteration number, the maximum iteration number, and the maximum learning rate; the second learning rate is calculated based on the maximum learning rate, the minimum learning rate, the current iteration number, and the maximum iteration number.

[0167] The learning rate is a tuning parameter in optimization algorithms used in machine learning and statistics. This parameter determines the step size in each iteration of model training, so that the loss function converges to its minimum value.

[0168] During the training of object detection models and image segmentation models, the learning rate decay strategy can be either the first learning rate or the second learning rate, or a strategy of superimposing (weighted summation) the two learning rates.

[0169] The first learning rate, lr1, can be obtained using Warmup, specifically calculated based on the current iteration number, the maximum iteration number, and the maximum learning rate. Its function is:

[0170]

[0171] Among them, T warmup Let t be the maximum number of iterations, t be the current iteration number, and lr be the maximum number of iterations. max This is the maximum learning rate. This strategy helps to mitigate the premature overfitting of the model to the mini-batch in the initial stage, maintaining a stable distribution.

[0172] The second learning rate, lr2, can be obtained using Cosine Annealing, specifically calculated based on the maximum learning rate, minimum learning rate, current iteration number, and maximum iteration number. Its function is:

[0173]

[0174] Among them, lr max To minimize the learning rate, lr max For the maximum learning rate, T cur T represents the current epoch (iteration number). max This represents the maximum epoch. This strategy aims to "escape" local minima and find a path to the global minimum.

[0175] Similarly, image segmentation models can also be trained using a first learning rate and / or a second learning rate.

[0176] In this embodiment, the current learning rate of the target detection model is obtained by superimposing (weighted summation) one or two learning rates, which are calculated from the first learning rate based on the current iteration number, the maximum iteration number, and the maximum learning rate, and the second learning rate based on the maximum learning rate, the minimum learning rate, the current iteration number, and the maximum iteration number. This enables the rapid training of the target detection model and the efficient and accurate detection of equipment defects using computer vision.

[0177] Based on the above embodiments, the step of obtaining the defect identification result of the device under test according to the defect detection result and the defect segmentation result in this embodiment includes: performing non-maximum suppression processing on the defect detection result, and obtaining defect detection boxes with an intersection-union ratio greater than a preset threshold from the processed defect detection result; the defect detection box is the bounding box where the detected defect region is located; obtaining defect segmentation boxes with an intersection-union ratio greater than the preset threshold from the defect segmentation result; the defect segmentation box is the bounding box where the segmented defect region is located; and obtaining the defect identification result according to the confidence level corresponding to the defect detection box and the confidence level corresponding to the defect segmentation box.

[0178] Non-Maximum Suppression (NMS) suppresses elements that are not maxima, essentially a local maximum search. Here, "local" refers to a neighborhood, which has two variable parameters: its dimension and its size.

[0179] Nonmaximal suppression is an edge refinement technique that helps suppress all gradient values ​​except for local maxima (by setting them to 0), indicating the location with the strongest intensity value change.

[0180] Specifically, Multi-Class NMS (Multi-Class Maximum Suppression) can be selected to perform NMS within each category of the defect detection result for post-processing, and then fused with the segmentation result output by the image segmentation model. By fusing the object detection model and the semantic segmentation model, an accurate prediction result can be obtained in the end.

[0181] After performing nonmaximum suppression processing on the defect detection results, defect detection boxes with a cross-union ratio greater than a preset threshold are obtained from the defect detection results, and these defect detection boxes are used as the bounding boxes of the detected defect regions.

[0182] Among them, the defect detection box with a larger intersection-union ratio (IUR) can detect defect points more accurately; that is, when the IUR is greater than a preset threshold, the area bounded by the defect detection box can be determined as the defect area.

[0183] Optionally, the preset threshold can be set according to the actual scenario, and this embodiment does not specifically limit it.

[0184] Similarly, in the defect segmentation results, defect segmentation boxes with an intersection-union ratio greater than a preset threshold are obtained, and these defect segmentation boxes are used as the bounding boxes of the segmented defect regions.

[0185] Finally, the detection result of the box with the higher confidence level between the defect detection box and the defect segmentation box is selected as the final defect identification result.

[0186] Specifically, if the confidence level of the defect detection box obtained by the defect detection model is relatively high, the defect detection result corresponding to the defect detection model is used as the final defect identification result; if the confidence level of the defect segmentation box obtained by the defect segmentation model is the highest, the defect segmentation result obtained by the defect segmentation model is used as the final identification result.

[0187] This embodiment first performs non-maximum suppression processing on the defect detection results; then, it obtains defect detection boxes with an intersection-union ratio (IU) greater than a preset threshold from the defect detection results, and uses these defect detection boxes as the bounding boxes of the detected defect regions; next, it obtains defect segmentation boxes with an IU greater than a preset threshold from the defect segmentation results, and uses these defect segmentation boxes as the bounding boxes of the segmented defect regions; finally, it selects the detection result of the box with the higher confidence level between the defect detection box and the defect segmentation box as the final defect identification result, thereby completing the defect identification result of the device under test based on the defect detection results and defect segmentation results, and realizing the efficient and accurate detection of equipment defects using computer vision.

[0188] The defect identification device for the fusion of target detection model and image segmentation model provided by the present invention will be described below. The defect identification device for the fusion of target detection model and image segmentation model described below can be referred to in correspondence with the defect identification method for the fusion of target detection model and image segmentation model described above.

[0189] like Figure 11 The diagram shown is a structural schematic of the defect recognition device that integrates a target detection model and an image segmentation model provided by the present invention. The device includes:

[0190] The acquisition module 1101 is used to acquire a surface image of the device to be inspected;

[0191] The detection module 1102 is used to input the surface image of the device to be inspected into the target detection model to obtain the defect detection result of the device to be inspected;

[0192] The segmentation module 1103 is used to input the surface image of the device to be inspected into the image segmentation model to obtain the defect segmentation result of the device to be inspected;

[0193] The identification module 1104 is used to obtain the defect identification result of the device under test based on the defect detection result and the defect segmentation result;

[0194] The target detection model is trained based on the surface image of the sample device and the defect detection label of the sample device, and the image segmentation model is trained based on the surface image of the sample device and the defect segmentation label of the sample device.

[0195] The defect recognition device that integrates target detection model and image segmentation model provided in this embodiment obtains the corresponding defect detection results and defect segmentation results of the device by inputting the surface image of the device to be inspected into the target detection model and the image segmentation model respectively. Then, based on the obtained defect detection results and defect segmentation results, the defect recognition results of the device to be inspected are obtained. This establishes a framework for efficient and rapid gear defect detection in industrial production, realizes efficient and accurate equipment defect detection using computer vision, reduces equipment defect detection errors, improves detection accuracy, and saves human resources.

[0196] Based on the above embodiments, the apparatus in this embodiment further includes a training module, specifically used for: acquiring a surface image of the sample device; labeling the surface image of the sample device with corresponding defect segmentation tags according to multiple preset defect types; labeling the surface image of the sample device with corresponding defect detection tags according to the defect segmentation tags; constructing a defect detection dataset based on the surface image of the sample device and the defect detection tags of the sample device; pre-training the parameters of the original target detection model based on the defect detection dataset to obtain pre-training parameters; iteratively training the parameters of the improved target detection model based on the pre-training parameters to obtain the target detection model; wherein, the improved target detection model is obtained by improving the original backbone network of the original target detection model based on the original target detection model; the improvement of the original backbone network includes replacing at least one convolutional network layer in the original backbone network of the original target detection model with a deformable convolutional network layer, and adding a symmetric network structure composed of an encoder and a decoder to the original backbone network.

[0197] Based on the above embodiments, the training module in this embodiment further includes an iterative training module, specifically used for: initializing the parameters of the improved target detection model according to the pre-training parameters; fixing the parameters of other networks in the initialized improved target detection model except for the improved backbone network, and training the parameters of the improved backbone network according to the defect detection dataset until a first preset termination condition is met to obtain the training parameters of the improved backbone network; and training the parameters of the initialized improved target detection model according to the training parameters of the improved backbone network until a second preset termination condition is met to obtain the target detection model.

[0198] Based on the above embodiments, the training module in this embodiment further includes an image processing module, specifically used for: acquiring the original surface image of the sample device; preprocessing and image augmentation of the original surface image to obtain the surface image of the sample device; wherein, the preprocessing includes one or more combinations of normalization processing, histogram equalization processing, and noise reduction processing; the image augmentation includes one or more combinations of overlaying a randomly generated background image, random cropping, and random flipping.

[0199] Based on the above embodiments, in the training module of this embodiment, the loss function of the target detection model is constructed and generated based on the regression loss function and the classification loss function; wherein, the regression loss function is constructed and generated based on the angle loss, distance loss, shape loss and intersection-union ratio loss determined by the defect detection results and defect detection labels of the sample device; the classification loss function is constructed and generated based on the zoom loss determined by the defect detection results and defect detection labels of the sample device.

[0200] Based on the above embodiments, in the training module of this embodiment, for the current iteration of training, the current learning rate of the target detection model is calculated based on a first learning rate and / or a second learning rate; the first learning rate is calculated based on the current iteration number, the maximum iteration number, and the maximum learning rate; the second learning rate is calculated based on the maximum learning rate, the minimum learning rate, the current iteration number, and the maximum iteration number.

[0201] Based on the above embodiments, the identification module 1104 in this embodiment is further configured to: perform non-maximum suppression processing on the defect detection results, and obtain defect detection boxes with an intersection-union ratio greater than a preset threshold from the processed defect detection results; the defect detection box is the bounding box where the detected defect region is located; obtain defect segmentation boxes with an intersection-union ratio greater than the preset threshold from the defect segmentation results; the defect segmentation box is the bounding box where the segmented defect region is located; and obtain the defect identification result based on the confidence level corresponding to the defect detection box and the confidence level corresponding to the defect segmentation box.

[0202] Figure 12 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 12 As shown, the electronic device may include a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. The processor 1210 can call logical instructions in the memory 1230 to execute a defect identification method that fuses a target detection model and an image segmentation model. This method includes: acquiring a surface image of a device to be detected; inputting the surface image of the device to be detected into a target detection model to obtain a defect detection result; inputting the surface image of the device to be detected into an image segmentation model to obtain a defect segmentation result; and obtaining a defect identification result of the device to be detected based on the defect detection result and the defect segmentation result. The target detection model is trained based on a surface image of a sample device and the defect detection label of the sample device, and the image segmentation model is trained based on the surface image of the sample device and the defect segmentation label of the sample device.

[0203] Furthermore, the logical instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the defect identification method that integrates the target detection model and the image segmentation model provided by the above methods. The method includes: acquiring a surface image of a device to be inspected; inputting the surface image of the device to be inspected into a target detection model to obtain a defect detection result of the device to be inspected; inputting the surface image of the device to be inspected into an image segmentation model to obtain a defect segmentation result of the device to be inspected; and acquiring a defect identification result of the device to be inspected based on the defect detection result and the defect segmentation result. The target detection model is trained based on a surface image of a sample device and a defect detection label of the sample device, and the image segmentation model is trained based on a surface image of the sample device and a defect segmentation label of the sample device.

[0205] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a defect identification method that integrates a target detection model and an image segmentation model provided by the methods described above. The method includes: acquiring a surface image of a device to be inspected; inputting the surface image of the device to be inspected into a target detection model to obtain a defect detection result of the device to be inspected; inputting the surface image of the device to be inspected into an image segmentation model to obtain a defect segmentation result of the device to be inspected; and acquiring a defect identification result of the device to be inspected based on the defect detection result and the defect segmentation result. The target detection model is trained based on a surface image of a sample device and a defect detection label of the sample device, and the image segmentation model is trained based on a surface image of the sample device and a defect segmentation label of the sample device.

[0206] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0207] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A defect recognition method integrating target detection model and image segmentation model, characterized in that, include: Acquire a surface image of the device to be inspected; The surface image of the device to be inspected is input into the target detection model to obtain the defect detection result of the device to be inspected; The surface image of the device to be inspected is input into the image segmentation model to obtain the defect segmentation result of the device to be inspected; Based on the defect detection results and the defect segmentation results, the defect identification results of the device under test are obtained, including: The defect detection results are subjected to non-maximum suppression processing, and defect detection boxes with a cross-union ratio greater than a preset threshold are obtained from the processed defect detection results; the defect detection box is the bounding box of the detected defect region; In the defect segmentation results, obtain the defect segmentation boxes with an intersection-union ratio greater than the preset threshold; the defect segmentation box is the bounding box of the segmented defect region. The defect identification result is obtained based on the confidence scores corresponding to the defect detection box and the defect segmentation box. The target detection model is obtained by training an initial target detection model based on the surface image of the sample device and the defect detection label of the sample device; the image segmentation model is obtained by training an initial image segmentation model based on the surface image of the sample device and the defect segmentation label of the sample device. The initial target detection model is constructed based on the PP-YOLOE detection model, and the initial image segmentation model is constructed based on the improved U-Net++ model. The improved U-Net++ model is obtained by replacing the first layer convolutional network structure of the U-Net++ model with a deformable convolutional structure.

2. The defect recognition method based on the fusion of target detection model and image segmentation model according to claim 1, characterized in that, The training steps of the object detection model include: Obtain a surface image of the sample device; Based on multiple preset defect types, the surface image of the sample device is labeled with corresponding defect segmentation tags; Based on the defect segmentation label, mark the surface image of the sample device with the corresponding defect detection label; Based on the surface image of the sample device and the defect detection label of the sample device, a defect detection dataset is constructed; Based on the defect detection dataset, the parameters of the original target detection model are pre-trained to obtain pre-trained parameters; Based on the pre-trained parameters, the parameters of the improved target detection model are iteratively trained to obtain the target detection model; The improved target detection model is obtained by improving the original backbone network of the original target detection model based on the original target detection model. The improvement of the original backbone network includes replacing at least one convolutional network layer in the original backbone network of the original target detection model with a deformable convolutional network layer, and adding a symmetric network structure composed of an encoder and a decoder to the original backbone network.

3. The defect recognition method based on the fusion of target detection model and image segmentation model according to claim 2, characterized in that, The step of iteratively training the parameters of the improved object detection model based on the pre-trained parameters to obtain the object detection model includes: The parameters of the improved target detection model are initialized based on the pre-trained parameters. The parameters of other networks in the improved target detection model after fixed initialization, except for the improved backbone network, are used to train the parameters of the improved backbone network according to the defect detection dataset until the first preset termination condition is met, so as to obtain the training parameters of the improved backbone network. Based on the training parameters of the improved backbone network, the parameters of the initialized improved target detection model are trained until the second preset termination condition is met, thus obtaining the target detection model.

4. The defect recognition method based on the fusion of target detection model and image segmentation model according to claim 2, characterized in that, The step of acquiring the surface image of the sample device includes: Obtain the original surface image of the sample device; The original surface image is preprocessed and augmented to obtain the surface image of the sample device; The preprocessing includes one or more combinations of normalization, histogram equalization, and noise reduction; the image augmentation includes one or more combinations of overlaying a randomly generated background image, random cropping, and random flipping.

5. The defect identification method based on the fusion of target detection model and image segmentation model according to any one of claims 1-4, characterized in that, The loss function of the target detection model is generated based on the regression loss function and the classification loss function; The regression loss function is generated based on the defect detection results of the sample equipment and the defect detection labels of the sample equipment, including angle loss, distance loss, shape loss, and cross-union ratio loss. The classification loss function is constructed based on the zoom loss determined by the defect detection results and defect detection labels of the sample device.

6. The defect recognition method based on the fusion of target detection model and image segmentation model according to any one of claims 1-4, characterized in that, For the current training iteration, the current learning rate of the object detection model is calculated based on the first learning rate and / or the second learning rate; The first learning rate is calculated based on the current iteration number, the maximum iteration number, and the maximum learning rate; The second learning rate is calculated based on the maximum learning rate, the minimum learning rate, the current iteration number, and the maximum iteration number.

7. A defect recognition device that integrates a target detection model and an image segmentation model, characterized in that, include: The acquisition module is used to acquire surface images of the device to be inspected; The detection module is used to input the surface image of the device to be inspected into the target detection model to obtain the defect detection result of the device to be inspected; The segmentation module is used to input the surface image of the device to be inspected into the image segmentation model to obtain the defect segmentation result of the device to be inspected. The identification module is used to obtain the defect identification result of the device under test based on the defect detection result and the defect segmentation result, including: The defect detection results are subjected to non-maximum suppression processing, and defect detection boxes with a cross-union ratio greater than a preset threshold are obtained from the processed defect detection results; the defect detection box is the bounding box of the detected defect region; In the defect segmentation results, obtain the defect segmentation boxes with an intersection-union ratio greater than the preset threshold; the defect segmentation box is the bounding box of the segmented defect region. The defect identification result is obtained based on the confidence scores corresponding to the defect detection box and the defect segmentation box. The target detection model is obtained by training an initial target detection model based on the surface image of the sample device and the defect detection label of the sample device; the image segmentation model is obtained by training an initial image segmentation model based on the surface image of the sample device and the defect segmentation label of the sample device. The initial target detection model is constructed based on the PP-YOLOE detection model, and the initial image segmentation model is constructed based on the improved U-Net++ model. The improved U-Net++ model is obtained by replacing the first layer convolutional network structure of the U-Net++ model with a deformable convolutional structure.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the defect identification method that fuses the target detection model and the image segmentation model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the defect identification method that integrates the target detection model and the image segmentation model as described in any one of claims 1 to 6.

Images