Training of defect detection model, defect detection method and device for industrial products

Abstract

The present disclosure relates to a training method and device of a defect detection model, and a defect detection method of an industrial product, and relates to the technical field of industrial product defect detection. The method comprises: performing defect detection and image reconstruction on historical defect detection images based on a network model to be trained, to obtain defect detection results and image reconstruction results; determining a first loss value according to the defect detection results and defect annotation results of the historical defect detection images; determining a second loss value according to the historical defect detection images and the image reconstruction results, and determining a total loss value according to the first loss value and the second loss value; training the network model to be trained based on the total loss value, to obtain a trained network model, and determining a product defect detection model based on the trained network model. The present disclosure improves the accuracy of the product defect detection model.

Description

Technical Field

[0001] This disclosure relates to the field of industrial product defect detection technology, specifically to a defect detection model training method, a defect detection model training device, and an industrial product defect detection method. Background Technology

[0002] Existing defect detection models can only detect complete structural defects in industrial products, but cannot detect incomplete structural defects, which results in low accuracy of the models in the detection of incomplete structural defects.

[0003] It should be noted that the information in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide a training method for a defect detection model, a training device for a defect detection model, and a defect detection method for industrial products, thereby overcoming, to at least a certain extent, the problem of low model accuracy caused by limitations and defects in related technologies.

[0005] According to one aspect of this disclosure, a method for training a defect detection model is provided, comprising:

[0006] Based on the network model to be trained, defect detection and image reconstruction are performed on historical defect detection images to obtain defect detection results and image reconstruction results; wherein, the image reconstruction results are used to enhance the network model to be trained to perceive the product structure of the industrial products corresponding to the historical defect detection images.

[0007] The first loss value is determined based on the defect detection results and the defect annotation results of the historical defect detection images;

[0008] A second loss value is determined based on the historical defect detection images and image reconstruction results, and a total loss value is determined based on the first loss value and the second loss value.

[0009] The network model to be trained is trained based on the total loss value to obtain the trained network model, and the product defect detection model is determined based on the trained network model.

[0010] In one exemplary embodiment of this disclosure, the network model to be trained includes a backbone feature extraction network, a feature fusion network, a defect prediction network, and a structure-aware network;

[0011] Specifically, based on the network model to be trained, defect detection and image reconstruction are performed on historical defect detection images to obtain defect detection results and image reconstruction results, including:

[0012] The historical defect detection image is downsampled based on the backbone feature extraction network to obtain multiple local image features at different scales.

[0013] The feature fusion network is used to fuse the local image features of multiple different scales to obtain global image features, and the defect prediction network is used to detect defects in the global image features to obtain the defect detection result.

[0014] The historical defect detection image is masked based on the structure-aware network to obtain a masked image region, and the masked image region is decoded based on the global image features to obtain the image reconstruction result.

[0015] In one exemplary embodiment of this disclosure, the backbone feature extraction network includes a convolutional backbone layer, an original feature sampling layer, and N feature downsampling layers, where N is a positive integer;

[0016] Specifically, the historical defect detection image is downsampled based on the backbone feature extraction network to obtain multiple local image features at different scales, including:

[0017] Based on the convolutional dry layer, basic features are extracted from the historical defect detection image to obtain basic feature extraction results. Then, based on the original feature sampling layer, initial sampling is performed on the basic feature extraction results to obtain initial feature sampling results.

[0018] The initial feature sampling result is downsampled for the first time based on the first feature downsampling layer among the N feature downsampling layers to obtain the first local image feature;

[0019] The first local image features are downsampled a second time based on the first feature downsampling layer among the N feature downsampling layers to obtain the second local image features;

[0020] The generation process of the second local image feature is repeated sequentially to obtain the third local image feature, ..., the Nth local image feature.

[0021] In one exemplary embodiment of this disclosure, the feature fusion network includes an original feature fusion network and N sub-feature fusion networks;

[0022] Specifically, the feature fusion network fuses the local image features at multiple different scales to obtain global image features, including:

[0023] Based on the original feature fusion network, the Nth local image feature among the multiple local image features of different scales is fused to obtain the original global feature;

[0024] The first sub-feature fusion network in the N sub-feature fusion network is used to fuse the original global features and N-1 local image features of multiple different scales to obtain the first global feature;

[0025] The second feature fusion network in the N sub-feature fusion network fuses the first global feature and the N-2th local feature in multiple local image features of different scales to obtain the second global feature.

[0026] The generation process of the second global feature is repeated sequentially to obtain the third global feature, ..., the Nth global feature.

[0027] In one exemplary embodiment of this disclosure, the defect region prediction network includes a first convolutional layer, a defect category detection layer, and a defect region detection layer;

[0028] Specifically, the defect detection is performed on the global image features based on the defect prediction network to obtain the defect detection result, including:

[0029] The global image features are convolved based on the first convolutional layer to obtain the first convolutional processing result;

[0030] Based on the defect category detection, the first convolution processing result is classified to obtain the defect category detection result of the target object included in the global image features; wherein, the target object is the defect corresponding to the industrial product;

[0031] Based on the defect region detection layer, the first convolution processing result is subjected to border detection processing to obtain the defect location detection result of the target object included in the global image features.

[0032] In one exemplary embodiment of this disclosure, the structure-aware network includes a masking layer, an embedding layer, a decoding layer, and an image reconstruction layer;

[0033] Specifically, the historical defect detection image is masked based on the structure-aware network to obtain a masked image region, and the masked image region is decoded based on the global image features to obtain the image reconstruction result, including:

[0034] The historical defect detection image is masked based on the masking layer to obtain a masked image region and a local masked image corresponding to the masked image region.

[0035] Based on the embedding layer, the local mask image is embedded into the mask image region corresponding to the historical defect detection image to obtain a global mask image;

[0036] Based on the decoding layer, the global image features are decoded to obtain the local image decoding result corresponding to the masked image region;

[0037] Based on the image reconstruction layer, the local image decoding result and the global mask image are used to reconstruct the image, and the image reconstruction result is obtained.

[0038] In one exemplary embodiment of this disclosure, the historical defect detection image is masked to obtain a masked image region and a local masked image corresponding to the masked image region, including:

[0039] The original global features, the first global features, ..., the Nth global features in the global image features are scaled, and the scaled global features are added together to obtain the global summed features;

[0040] Based on the global feature width and global feature length of the global summation feature, the mask image region is determined, and the image region in the historical defect detection image corresponding to the position of the mask image region is masked to obtain a local mask image corresponding to the mask image region.

[0041] In one exemplary embodiment of this disclosure, the defect annotation result of the historical defect detection image includes the original actual border and the actual defect category corresponding to the original actual border;

[0042] The defect detection results include defect location detection results and defect category detection results corresponding to the defect location detection results;

[0043] The determination of the first loss value based on the defect detection results and the defect annotation results of the historical defect detection images includes:

[0044] The current predicted bounding box is determined based on the defect location detection results;

[0045] The current predicted bounding box is classified to obtain the foreground predicted bounding box and the background predicted bounding box;

[0046] The position loss value is determined based on the predicted background bounding box and the original actual bounding box;

[0047] The category loss value is determined based on the defect category detection results and the actual defect category, and the first loss value is determined based on the location loss value and the category loss value.

[0048] In one exemplary embodiment of this disclosure, determining the predicted bounding box based on the defect location detection result includes:

[0049] Determine the location anchor points and stride information of global image features;

[0050] The defect location detection result is decoded based on the location anchor points and step size information to obtain a predicted bounding box; wherein the number of the first bounding box of the predicted bounding box is the same as the number of anchor points of the location anchor points.

[0051] In one exemplary embodiment of this disclosure, the current predicted bounding box is classified to obtain a foreground predicted bounding box and a background predicted bounding box, including:

[0052] Calculate the border matching degree between the current predicted border and the original actual border, and sort the current predicted borders according to the border matching degree to obtain the border sorting result;

[0053] Calculate the number of second borders that the original actual borders can match the currently predicted borders;

[0054] Based on the border sorting result, extract the current predicted border corresponding to the second border quantity from the current predicted borders as the foreground predicted border corresponding to the target actual border, and take at least some of the other borders in the current predicted borders other than the foreground predicted borders as the background predicted borders corresponding to the target actual border.

[0055] In one exemplary embodiment of this disclosure, calculating the border matching degree between the currently predicted border and the original actual border includes:

[0056] Determine whether the predicted center point of the current predicted border is included within the range of the first border corresponding to the original actual border, or whether the predicted center point is included within the first preset area corresponding to the actual center point of the original actual border.

[0057] If the predicted center point is not included within the first border or is not included within the first preset area, then the border matching degree between the current predicted border and the original actual border is determined to be infinitesimal.

[0058] If the predicted center point is included within the first border or within the first preset area, the border matching degree is determined based on the first extended intersection-union ratio between the current predicted border and the original actual border.

[0059] In one exemplary embodiment of this disclosure, determining the border matching degree based on a first extended intersection-union ratio (IUU) between the currently predicted border and the original actual border includes:

[0060] Determine the border position offset and border size offset between the current predicted border and the original actual border, and adjust the original actual border according to the border position offset and border size offset to obtain the target actual border;

[0061] Determine the generalized intersection-union ratio (GUOR) between the target actual bounding box and the current predicted bounding box, and based on the GUOR, determine the first extended intersection-union ratio (AUR) between the current predicted bounding box and the original actual bounding box;

[0062] Determine the category difference between the defect category detection result corresponding to the current predicted bounding box and the actual defect category corresponding to the original actual bounding box, and determine the bounding box matching degree based on the category difference and the first extended intersection-union ratio.

[0063] In one exemplary embodiment of this disclosure, calculating the number of second borders that the original actual border can match the currently predicted border includes:

[0064] Determine the regression prediction bounding box corresponding to the current prediction bounding box, and determine the second extended intersection-union ratio between the original actual bounding box and the regression prediction bounding box;

[0065] The number of second borders that the original actual borders can match based on the second extended intersection-union ratio is determined.

[0066] In one exemplary embodiment of this disclosure, determining a position loss value based on the foreground predicted bounding box and the original actual bounding box in the defect annotation results of the historical defect detection image includes:

[0067] Calculate the third extended cross-union ratio (CUIR) between the predicted foreground bounding box and the original actual bounding box in the defect annotation result of the historical defect detection image, and determine the position loss value based on the third extended CUIR.

[0068] In one exemplary embodiment of this disclosure, determining a second loss value based on the historical defect detection image and the image reconstruction result includes:

[0069] The historical defect detection image and the image reconstruction result are subjected to grayscale processing to obtain a first grayscale image and a second grayscale image;

[0070] Extract the first grayscale value of the first grayscale image and the second grayscale value of the second grayscale image, and determine the second loss value based on the difference between the first grayscale value and the second grayscale value.

[0071] In one exemplary embodiment of this disclosure, determining a product defect detection model based on a trained network model includes:

[0072] The structure-aware network in the trained network model is pruned, and the product defect detection model is generated based on the pruned network model.

[0073] According to one aspect of this disclosure, a method for detecting defects in an industrial product is provided, comprising:

[0074] Acquire images of the industrial products to be inspected;

[0075] The product image to be detected is input into the product defect detection model to obtain the defect detection result; wherein, the product defect detection model is trained by any of the defect detection model training methods described above;

[0076] The defect location and the corresponding defect category of the industrial product to be inspected are determined based on the defect detection results.

[0077] According to one aspect of this disclosure, a training apparatus for a defect detection model is provided, comprising:

[0078] The defect detection module is used to perform defect detection and image reconstruction on historical defect detection images based on the network model to be trained, and obtain defect detection results and image reconstruction results; wherein, the image reconstruction results are used to enhance the network model to be trained to perceive the product structure of the industrial product corresponding to the historical defect detection images.

[0079] The first loss value determination module is used to determine a first loss value based on the defect detection results and the defect annotation results of the historical defect detection images;

[0080] The total loss value determination module is used to determine a second loss value based on the historical defect detection image and the image reconstruction result, and to determine a total loss value based on the first loss value and the second loss value;

[0081] The product defect detection model determination module is used to train the network model to be trained based on the total loss value, obtain the trained network model, and determine the product defect detection model based on the trained network model.

[0082] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a training method for a defect detection model as described in any one of the preceding claims, and a defect detection method for an industrial product as described in any one of the preceding claims.

[0083] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0084] Processor; and

[0085] Memory for storing the executable instructions of the processor;

[0086] The processor is configured to execute the training method of the defect detection model described in any one of the above-mentioned methods, and the defect detection method of the industrial product described in any one of the above-mentioned methods, by executing the executable instructions.

[0087] This disclosure provides a training method for a defect detection model. The method involves performing defect detection and image reconstruction on historical defect detection images based on a network model to be trained, obtaining defect detection results and image reconstruction results. Then, a first loss value is determined based on the defect detection results and the defect annotation results of the historical defect detection images. A second loss value is then determined based on the historical defect detection images and the image reconstruction results, and a total loss value is determined based on the first and second loss values. Finally, the network model to be trained is trained based on the total loss value to obtain a trained network model. Based on the trained network model, a product defect detection model is determined. Because the model's training process considers not only its defect detection capability but also its perception and understanding of the product structure of industrial products corresponding to historical defect detection images, the model can perceive both complete and incomplete structural defects during defect detection. This solves the problem of low accuracy in detecting incomplete structural defects and improves the accuracy of the obtained product defect detection model.

[0088] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0089] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0090] Figure 1 The flowchart schematically illustrates a method for training a defect detection model according to an exemplary embodiment of the present disclosure.

[0091] Figure 2 The diagram schematically illustrates an example structure of a network model to be trained according to an exemplary embodiment of the present disclosure.

[0092] Figure 3The diagram schematically illustrates an example structure of a backbone feature extraction network in a network model to be trained according to an exemplary embodiment of the present disclosure.

[0093] Figure 4 The diagram schematically illustrates an example structure of a feature fusion network in a network model to be trained according to an exemplary embodiment of the present disclosure.

[0094] Figure 5 The diagram schematically illustrates an example structure of a defect region prediction network in a network model to be trained according to an exemplary embodiment of the present disclosure.

[0095] Figure 6 The diagram schematically illustrates an example structure of a structure-aware network in a network model to be trained according to an exemplary embodiment of the present disclosure.

[0096] Figure 7 The illustration shows an example diagram illustrating a specific process for determining a defect detection result and an image reconstruction result according to an example embodiment of the present disclosure.

[0097] Figure 8 This diagram schematically illustrates a scene example of a historical defect detection image corresponding to a display screen according to an exemplary embodiment of this disclosure.

[0098] Figure 9 The illustration shows a scene example diagram of a feature pyramid obtained according to an exemplary embodiment of the present disclosure.

[0099] Figure 10 The illustration schematically depicts a scenario example of feature fusion of local image features according to an exemplary embodiment of the present disclosure.

[0100] Figure 11 The illustration shows an example scenario of convolutional processing of global image features according to an example embodiment of the present disclosure.

[0101] Figure 12 The illustration schematically shows a scene example of a partial mask image obtained according to an exemplary embodiment of the present disclosure.

[0102] Figure 13 The diagram illustrates a method flowchart for determining a specific process of a first loss function according to an exemplary embodiment of the present disclosure.

[0103] Figure 14 An example diagram illustrating the location anchor point of an image feature according to an exemplary embodiment of the present disclosure is shown.

[0104] Figure 15 This illustration schematically shows a scene example of the current actual border of a labeled historical defect detection image according to an exemplary embodiment of the present disclosure.

[0105] Figure 16 The illustration shows a scenario example diagram of a product defect detection model according to an exemplary embodiment of the present disclosure.

[0106] Figure 17 The flowchart illustrates a defect detection method for an industrial product according to an exemplary embodiment of the present disclosure.

[0107] Figure 18 The diagram schematically illustrates a structural example of a training apparatus for a defect detection model according to an exemplary embodiment of the present disclosure.

[0108] Figure 19 This diagram schematically illustrates a structural example of a defect detection device for an industrial product according to an exemplary embodiment of the present disclosure.

[0109] Figure 20 An electronic device is illustrated schematically according to an example embodiment of the present disclosure for implementing a training method for a defect detection device and a defect detection method for industrial products. Detailed Implementation

[0110] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0111] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0112] In industrial defect detection, there exists an incomplete structural defect that cannot be detected using conventional detection models. This incomplete structural defect, as described here, can characterize a missing one or more structural categories in an industrial product. For example, a structure that should be present at a certain location may not actually be present, thus constituting a defect. Existing defect detection algorithms can only detect defects present in existing structures, not those of missing structural categories. For instance, when the industrial product is a display screen, the manufacturing process involves spacers (e.g., columnar spacers) used to separate the glass substrate. These spacers exist before the pixels and are numerous. Furthermore, due to imperfections in the manufacturing process, some locations may lack these spacers. Therefore, detecting the missing spacers and their specific locations during defect detection of this industrial product becomes a crucial problem to solve.

[0113] Based on this, this exemplary embodiment first provides a method for training a defect detection model, which can run on terminal devices, servers, server clusters, or cloud servers, etc. Of course, those skilled in the art can also run the method disclosed herein on other platforms as needed, and this exemplary embodiment does not impose any special limitations on this. Specifically, refer to... Figure 1 As shown, the training method for this defect detection model may include the following steps:

[0114] Step S110. Based on the network model to be trained, perform defect detection and image reconstruction on the historical defect detection images to obtain defect detection results and image reconstruction results; wherein, the image reconstruction results are used to enhance the network model to be trained's ability to perceive the product structure of the industrial products corresponding to the historical defect detection images.

[0115] Step S120. Determine the first loss function based on the defect detection results and the defect annotation results of the historical defect detection images;

[0116] Step S130. Determine the second loss function based on the historical defect detection image and the image reconstruction result, and determine the target loss function based on the first loss function and the second loss function;

[0117] Step S140. Train the network model to be trained based on the target loss function to obtain the trained network model, and determine the product defect detection model based on the trained network model.

[0118] In the training method of the defect detection model described above, defect detection and image reconstruction can be performed on historical defect detection images based on the network model to be trained, resulting in defect detection results and image reconstruction results. Then, a first loss value is determined based on the defect detection results and the defect annotation results of the historical defect detection images. Subsequently, a second loss value is determined based on the historical defect detection images and image reconstruction results, and a total loss value is determined based on the first and second loss values. Finally, the network model to be trained is trained based on the total loss value to obtain a trained network model, and a product defect detection model is determined based on the trained network model. Since the model considers not only the defect detection capability of the model during training, but also the model's perception and understanding of the product structure of industrial products corresponding to historical defect detection images, the model can perceive not only complete structural defects but also incomplete structural defects during defect detection. This solves the problem of low accuracy of the model in the detection scenario of incomplete structural defects, thus improving the accuracy of the obtained product defect detection model.

[0119] The training method of the defect detection model described in the exemplary embodiments of this disclosure will be explained and described in detail below with reference to the accompanying drawings.

[0120] First, the application scenarios of the exemplary embodiments of this disclosure will be explained and described. Specifically, the product defect detection model obtained based on the training method of the defect detection model described in the exemplary embodiments of this disclosure can not only be used to detect the types of structural defects (i.e., complete structural defects) and the locations of structural defects in existing product structures in industrial products, but also to detect the missing categories (i.e., incomplete structural defects) and the locations of missing product structures in industrial products; for example, taking an industrial product as a display screen, it can not only detect scratches present in the display screen and the locations of scratches in the display screen, but also detect missing spacers in the display screen and the locations of missing spacers.

[0121] Secondly, the network model to be trained involved in the exemplary embodiments of this disclosure will be explained and described. Specifically, refer to... Figure 2 As shown, the network model to be trained may include a first input layer 201, a backbone network 202, a feature pyramid network 203, a defect prediction network 204, a structure-aware network 205, a first output layer 206, and a second output layer 207. The role of each model layer in the defect prediction process will be detailed later and will not be elaborated further here.

[0122] In one example embodiment, reference is made to... Figure 3 As shown, the backbone feature extraction network may include a convolutional stem layer (Stem) 301, an original feature sampling layer (Layer0) 302, and N feature downsampling layers, where N is a positive integer. Specifically, taking N as 4 as an example, the backbone feature extraction network may include a first feature downsampling layer (Layer1) 303, a second feature downsampling layer (Layer2) 304, a third feature downsampling layer (Layer3) 305, and a fourth feature downsampling layer (Layer4) 306. The role of each model layer in the defect prediction process will be detailed later and will not be elaborated further here.

[0123] In one example embodiment, reference is made to... Figure 4 As shown, the feature fusion network consists of an original feature fusion network 401 and N sub-feature fusion networks, where N is a positive integer greater than or equal to 3. The values ​​of N recorded here correspond one-to-one with the values ​​of N in the feature downsampling layer. In practical applications, the specific value of N can be set according to actual needs; this example does not impose any special restrictions on this. Specifically, taking a value of 4 for N as an example, the feature fusion network can include a first sub-feature fusion network 402, a second sub-feature fusion network 403, a third sub-feature fusion network 403, and a fourth sub-feature fusion network 404. The roles of each model layer in the defect prediction process will be detailed later and will not be elaborated further here.

[0124] In one example embodiment, reference is made to... Figure 5 As shown, the defect region prediction network may include a first convolutional layer 501, a defect category detection layer 502, and a defect region detection layer 503; the role of each model layer in the defect process will be detailed later, and will not be elaborated further here.

[0125] In one example embodiment, reference is made to... Figure 6 As shown, the structure-aware network includes a masking layer 601, an embedding layer 602, a decoding layer 603, and an image reconstruction layer 604; the roles of each model layer in the defect process will be detailed later, and will not be elaborated further here.

[0126] The following will combine Figures 2-6 right Figure 1 The training method of the defect detection model shown will be further explained and illustrated. Specifically:

[0127] In step S110, the historical defect detection images are used for defect detection and image reconstruction based on the network model to be trained, so as to obtain the defect detection results and image reconstruction results.

[0128] Specifically, the image reconstruction results described herein can be used to enhance the network model under training's ability to perceive the product structure of industrial products corresponding to the historical defect detection images, and also enhance the model's understanding of the product structure. Based on this, if the model can accurately perceive and understand the product structure, it can accurately predict the imperfections included in the product. Further, refer to... Figure 7 As shown, the process of performing defect detection and image reconstruction on historical defect detection images based on the network model to be trained, to obtain defect detection results and image reconstruction results, can include the following steps:

[0129] Step S710: Based on the backbone feature extraction network, the historical defect detection image is downsampled to obtain multiple local image features at different scales.

[0130] Specifically, the determination process of multiple local image features at different scales can be implemented as follows: Basic features are extracted from the historical defect detection image based on the convolutional layer to obtain basic feature extraction results; initial sampling is performed on the basic feature extraction results based on the original feature sampling layer to obtain initial feature sampling results; the initial feature sampling results are downsampled for the first time based on the first feature downsampling layer among the N feature downsampling layers to obtain first local image features; the first local image features are downsampled for the second time based on the first feature downsampling layer among the N feature downsampling layers to obtain second local image features; the generation process of the second local image features is repeated sequentially to obtain third local image features, ..., the Nth local image features. Furthermore, taking an N value of 4 as an example, the specific process for determining local image features at different scales is as follows: Based on the first feature downsampling layer, the initial feature sampling result is downsampled for the first time to obtain the first local image feature; then, based on the second feature downsampling layer, the first local image feature is downsampled for the second time to obtain the second local image feature; based on the third feature downsampling layer, the second local image feature is downsampled for the third time to obtain the third local image feature; and finally, based on the fourth feature downsampling layer, the third local image feature is downsampled for the fourth time to obtain the fourth local image feature. Specifically, taking an industrial product as an example, the historical defect detection images recorded here can be referenced... Figure 8 As shown, the historical defect detection image can be denoted as I, the image height of the historical defect detection image is H, the image width is W, the number of pixel channels of the image is C, and the number of historical defect detection images in the same batch is N; furthermore, the feature sampling layer described here can be a residual network ResNet, or a deep learning model based on the Transformer architecture, and this example does not impose any special restrictions on it.

[0131] In practical applications, the historical defect detection image I is first input into the convolutional stem layer (Stem) of the backbone feature extraction network; secondly, the obtained basic feature extraction results are input into Layer 0 to obtain the initial feature sampling results; then, the obtained initial feature sampling results are input into Layer 1 to obtain the first local feature; further, the obtained first local feature is input into Layer 2 to obtain the second local feature; even further, the obtained second local feature is input into Layer 3 to obtain the third local feature; finally, the obtained third local feature is input into Layer 4 to obtain the fourth local feature; wherein, the obtained basic feature extraction result I0 can be represented as (H0, W0, C0), the obtained first local feature I1 can be represented as (H1, W1, C1), the obtained second local feature I2 can be represented as (H2, W2, C2), the obtained third local feature I3 can be represented as (H3, W3, C3), and the obtained fourth local feature I4 can be represented as (H4, W4, C4); further, H i =H2 i W i =W2 i C i =C2 i The value of i ranges from [0, 5].

[0132] Step S720: Based on the feature fusion network, the local image features of multiple different scales are fused to obtain global image features, and the defect detection is performed on the global image features based on the defect prediction network to obtain the defect detection result.

[0133] In this example embodiment, firstly, multiple local image features at different scales are fused based on a feature fusion network to obtain global image features. Specifically, this can be achieved as follows: the Nth local image feature among the multiple local image features at different scales is fused based on the original feature fusion network to obtain the original global feature; the original global feature and the N-1th local image features among the multiple local image features at different scales are fused based on the first sub-feature fusion network among the N sub-feature fusion networks to obtain the first global feature; the first global feature and the N-2th local feature among the multiple local image features at different scales are fused based on the second feature fusion network among the N sub-feature fusion networks to obtain the second global feature; the generation process of the second global feature is repeated sequentially to obtain the third global feature, ..., the Nth global feature. Furthermore, taking N as a value of 4 as an example, the specific implementation process for fusing local image features at different scales is as follows: Based on the original feature fusion network, the fourth local image features are fused to obtain the original global features; based on the first feature fusion network, the original global features and the third local image features are fused to obtain the first global features; based on the second feature fusion network, the first global features and the second local features are fused to obtain the second global features; based on the third feature fusion network, the second global features and the first local features are fused to obtain the third global features; based on the fourth feature fusion network, the third global features and the initial feature sampling results are fused to obtain the fourth global features. Specifically, in the feature fusion process, a feature pyramid approach can be used to achieve a deep-to-shallow approach; see reference... Figure 9 As shown, the feature pyramid described here can be represented in an inverted triangle shape. The bottom layer 901 includes the original global feature F4 obtained by fusing the fourth local image features; the middle layer 902 includes the first global feature F3 obtained by fusing the original global feature and the third local image features; the middle layer 903 includes the second global feature F2 obtained by fusing the first global feature and the second local image features; the middle layer 904 includes the third global feature F1 obtained by fusing the second global feature and the first local image features; and the top layer 905 includes the fourth global feature F0 obtained by fusing the third global feature and the initial feature sampling result. Further, for a specific example diagram of the feature fusion process, please refer to... Figure 10 As shown.

[0134] Secondly, defect detection is performed on the global image features based on the defect prediction network to obtain the defect detection result. Specifically, this can be achieved as follows: the global image features are convolved based on the first convolutional layer to obtain the first convolutional processing result, and the first convolutional processing result is classified based on the defect category detection to obtain the defect category detection result of the target object included in the global image features; wherein, the target object mentioned here refers to the defect corresponding to an industrial product; and the first convolutional processing result is bounding box detection based on the defect region detection layer to obtain the defect location detection result of the target object included in the global image features. Specifically, the first convolutional layer mentioned here can be multiple convolutional layers, and the specific number of convolutional layers can be determined according to actual needs. This example does not impose any special restrictions on this. At the same time, the number of first convolutional layers can correspond one-to-one with the number of layers of the global features mentioned above, and the weights can be shared among the first convolutional layers. For a specific scenario example diagram of the convolutional processing process, please refer to... Figure 11 As shown; after convolution processing, two different convolutional networks can be used as prediction heads to separately generate bounding boxes (i.e., defect location detection results) and class prediction parameters (i.e., defect class detection results); furthermore, the parameter shape of the predicted defect location detection results is [N, 4, H]. i W i The predicted defect category detection result has the parameter shape [N,C,H]. i W i Let r and c be the parameters of the defect location detection results, respectively; and let the parameter shape be [N, 4, H]. i W i The 4 in the figure represents the four coordinate points of the rectangle; the parameter shape of the defect category detection result [N,C,H] i W i In the diagram, C represents the numerical value of the specific defect category. It should be further clarified that the defect categories described here can be broadly divided into complete structural defects and incomplete structural defects. For complete structural defects, taking a display screen as an example, this may include, but is not limited to, scratches, breaks, chipping, cracks, and burrs on the display screen. Furthermore, for incomplete structural defects, again using a display screen as an example, this may include, but is not limited to, the absence of a signal interface, the absence of one or more spacers, or the absence of a panel in the overall structure of the display screen. In practical applications, this model can be used to detect all defects in an industrial product at once, along with the specific category and location of each defect, thereby improving both detection efficiency and the accuracy of the detection results.

[0135] Step S730: The historical defect detection image is masked based on the structure-aware network to obtain a masked image region, and the masked image region is decoded based on the global image features to obtain the image reconstruction result.

[0136] Specifically, the process of determining the image reconstruction result can be implemented as follows: The historical defect detection image is masked based on the masking layer to obtain a masked image region and a local masked image corresponding to the masked image region; the local masked image is embedded into the masked image region corresponding to the historical defect detection image based on the embedding layer to obtain a global masked image; the global image features are decoded based on the decoding layer to obtain a local image decoding result corresponding to the masked image region; and the image reconstruction layer performs image reconstruction on the local image decoding result and the global masked image to obtain the image reconstruction result.

[0137] In an exemplary embodiment, taking a value of 4 for N as an example, the historical defect detection image is masked to obtain a masked image region and a local masked image corresponding to the masked image region. This can be achieved as follows: the original global features, the first global features, ..., the Nth global features in the global image features are scaled, and the scaled global features are added together to obtain a global summed feature; the masked image region is determined based on the global feature width and global feature length of the global summed feature, and the image region in the historical defect detection image corresponding to the position of the masked image region is masked to obtain a local masked image corresponding to the masked image region.

[0138] The following section will further explain and illustrate the specific process of determining the image reconstruction result. Specifically, in the image reconstruction process, firstly, the global features F from different layers of the feature pyramid are... i Scaling to the same size and adding them together yields the global summative feature, denoted as feature F. d Among them, F d The width and height are denoted as W respectively. d and H d Secondly, randomly select from Fd[:,:,:,W d / / 3:2W d / / 3] and Fd[:,:,H d / / 3:2H d / / 3,:] One of the two regions is selected as the mask region; where Fd[:,:,:,W d / / 3:2W d / / 3] can be used to characterize the lateral direction. The middle region, Fd[:,:,Hd / / 3:2H d / / 3,:] can be used to represent the longitudinal direction. The middle region; in practical applications, taking industrial products as an example, since the screen structure of a display screen follows simple mechanical rules, the structure of the middle region can be easily deduced based on the front and back parts, given an understanding of the screen structure. Furthermore, the obtained local mask image can be referenced... Figure 12 As shown; then, set the mask region to Tensor(T) e ); among them, the horizontal Taking the middle region as an example, the specific process of setting the mask region can be shown by the following formula (1):

[0139]

[0140] Furthermore, using multi-layer convolutions on F d Decoding is performed; the specific decoding process can be shown by the following formula (2):

[0141] F d =Decode(F d ); Formula (2)

[0142] Finally, a convolutional layer is used as the prediction head to generate the reconstructed image I. g The specific image reconstruction process can be represented by the following formula (3):

[0143] I g =conv(F d ); Formula (3)

[0144] In step S120, a first loss value is determined based on the defect detection result and the defect annotation result of the historical defect detection image.

[0145] Specifically, the defect annotation results of the historical defect detection images recorded here include the original actual border and the actual defect category corresponding to the original actual border; the defect detection results recorded here include the defect location detection results and the defect category detection results corresponding to the defect location detection results; that is, in practical applications, defect locations and defect categories appear in pairs; for example, a defect exists at location 1, and the corresponding defect category is scratch; or, for example, a defect exists at location 2, and the corresponding defect category is missing a spacer, etc. Further, refer to... Figure 13 As shown, determining the first loss value based on the defect detection results and the defect annotation results of historical defect detection images may include the following steps:

[0146] Step S1310: Determine the current predicted bounding box based on the defect location detection result, classify the current predicted bounding box to obtain the foreground predicted bounding box and the background predicted bounding box.

[0147] In this example embodiment, firstly, a predicted bounding box is determined based on the defect location detection result; specifically, this can be achieved as follows: firstly, the location anchor points and stride information of the global image features are determined; secondly, the defect location detection result in the defect detection result is decoded based on the location anchor points and stride information to obtain the predicted bounding box; wherein, the number of the first bounding box of the predicted bounding box is the same as the number of anchor points of the location anchor points.

[0148] Secondly, the current predicted bounding boxes are classified to obtain foreground predicted bounding boxes and background predicted bounding boxes. Specifically, this can be achieved as follows: calculate the bounding box matching degree between the current predicted bounding box and the original actual bounding box, and sort the current predicted bounding boxes according to the bounding box matching degree to obtain a bounding box sorting result; calculate the number of second bounding boxes that the original actual bounding box can match; extract the current predicted bounding boxes corresponding to the number of second bounding boxes from the current predicted bounding boxes according to the bounding box sorting result as the foreground predicted bounding boxes corresponding to the target actual bounding box, and take at least some of the other bounding boxes in the current predicted bounding boxes other than the foreground predicted bounding boxes as the background predicted bounding boxes corresponding to the target actual bounding box.

[0149] In one exemplary embodiment, calculating the border matching degree between the current predicted border and the original actual border can be achieved as follows: determining whether the predicted center point of the current predicted border is included within the first border range corresponding to the original actual border, or whether the predicted center point is included within the first preset region corresponding to the actual center point of the original actual border; if the predicted center point is not included within the first border range or is not included within the first preset region, then the border matching degree between the current predicted border and the original actual border is determined to be infinitesimal; if the predicted center point is included within the first border range or is included within the first preset region, then the border matching degree is determined based on the first extended intersection-union ratio between the current predicted border and the original actual border.

[0150] In one exemplary embodiment, determining the border matching degree based on the first extended intersection-union ratio (ECU) between the currently predicted border and the original actual border can be achieved as follows: determining the border position offset and border size offset between the currently predicted border and the original actual border, and adjusting the original actual border according to the border position offset and border size offset to obtain a target actual border; determining the generalized intersection-union ratio (GURR) between the target actual border and the currently predicted border, and determining the first extended intersection-union ratio (ECU) between the currently predicted border and the original actual border based on the GURR; determining the category difference between the defect category detection result corresponding to the currently predicted border and the actual defect category corresponding to the original actual border, and determining the border matching degree based on the category difference and the first extended intersection-union ratio (ECU).

[0151] In one exemplary embodiment, the number of second borders that the original actual border can match the current predicted border can be calculated as follows: determine the regression predicted border corresponding to the current predicted border, and determine the second extended cross-union ratio between the original actual border and the regression predicted border; determine the number of second borders that the original actual border can match the current predicted border based on the second extended cross-union ratio.

[0152] Step S1320: Determine the position loss value based on the predicted background bounding box and the original actual bounding box.

[0153] Specifically, the determination of the position loss value can be achieved as follows: calculate the third extended cross-union ratio between the predicted foreground bounding box and the original actual bounding box in the defect annotation result of the historical defect detection image, and determine the position loss value based on the third extended cross-union ratio.

[0154] Step S1330: Determine the category loss value based on the defect category detection result and the actual defect category, and determine the first loss value based on the location loss value and the category loss value.

[0155] The following will further explain and illustrate the specific process of determining the first loss value. Specifically, in the process of constructing the first loss value, firstly, position anchors are generated for each layer of global features in the feature pyramid; the specific process of generating the position anchors can be shown in the following formula (4):

[0156] A i = meshgrid(H i W i ); Formula (4)

[0157] Among them, A iH represents the location anchor point of the global feature in the i-th layer, and meshgrid(·) is the location anchor point generation function; i and W i Let be the feature height and feature width of the global feature at layer i; further, taking a global feature height and feature width of 5×5 as an example, the schematic diagram of the obtained position anchor points can be referenced. Figure 14 As shown; in Figure 14 The anchor points shown may include, but are not limited to, a series of coordinate values ​​such as (0,0), (1,0), (2,0), ..., (4,4); and in practical applications, the A of the anchor points... i The shape can be represented as [2,H] i W i It can also be broadcast as a shape [N,2,H] i W i ]; where 2 represents the number of channels (including both the horizontal and vertical axes), H i and W i These represent the vertical and horizontal coordinates of the anchor point in the feature map, respectively; N represents the number of images in the same batch.

[0158] Secondly, step size information is generated for each layer of global features in the feature pyramid; the specific generation process of the step size information can be shown in the following formula (5):

[0159] S i =2 i ; Formula (5)

[0160] Among them, S i This represents the compensation information for the global features of the i-th layer. In practical applications, the step size can be used to represent the downsampling factor of the current feature map relative to the original image. For example, the 0th layer represents the original image with a downsampling factor of 1; the 3rd layer has a downsampling factor of 2 cubed, which is 8 times. The step size is broadcast as a shape [N,1,H]. i W i ]; where 1 represents the number of channels.

[0161] Then, the defect location detection results are decoded using preset position anchor points and step size information to obtain the predicted bounding boxes (bboxes); the decoding process of the predicted bounding boxes (bboxes) can be shown in the following formulas (6) and (7):

[0162] b0[:,:,2]=(r[:,:2]+A)*S; Formula (6)

[0163] b0[:,2:]=e r[:,2:] *S; Formula (7)

[0164] Where b0[:,:,2] represents the predicted bounding box obtained with the center point in the defect location detection result as the parameter, b0[:,2:] represents the predicted bounding box obtained with the width and height in the defect location detection result as the parameter; r[:,:2] represents the center point in the defect location detection result, r[:,2:] represents the width and height in the defect location detection result; A represents the location anchor point, and S represents the step size information.

[0165] Furthermore, after obtaining the predicted bounding boxes, the following algorithm can be used for label assignment. Here, label assignment refers to dividing the predicted bounding boxes into foreground and background bounding boxes. Therefore, when determining the positional loss function in the first loss function, the focus can be placed on the foreground bounding boxes. Based on this, not only can the accuracy of the obtained model be improved, but the training efficiency can also be increased. It should be further explained here that since incomplete structures (i.e., missing structures) do not have corresponding features at the target location, the defect locations labeled by the annotator will not be as precise as those labeled for a complete target (i.e., an existing structural defect). Therefore, to address this inaccuracy in label location, the label assignment algorithm described here is needed to construct the loss function. Specifically, the implementation process of the label assignment algorithm may include the following steps:

[0166] a) Calculate the border matching degree between the predicted bounding boxes (bboxes) and the original actual bounding boxes (gt bboxes); specifically, in the process of calculating the border matching degree, if the predicted center point of a predicted bounding box (bboxes) is not within the first bounding box range corresponding to the original actual bounding box (gt bbox), or is not within the first preset area corresponding to the actual center point of the original actual bounding box (gt bbox) (the size of the first preset area is usually defined as 2.5 times the downsampling ratio of the current tensor relative to the original image), its matching degree is set to negative infinity (that is, the border matching degree between the current predicted bounding box and the original actual bounding box can be set to infinitesimal); otherwise, the border matching degree is determined according to the generalized intersection over union (GIOU) between the current predicted bounding box and the original actual bounding box; specifically, the specific calculation process of the generalized intersection over union between the current predicted bounding box and the original actual bounding box can be shown in the following formulas (8)-(16):

[0167]

[0168] offset=max(offset|-σ,0)sign(offset); Formula (9)

[0169]

[0170]

[0171] b' g [:,:2]=b gc -b gs ; Formula (14)

[0172] b g '[:,2:]=b gc +b gs ; Formula (15)

[0173] Generalized intersection-union ratio = GIOU(b) g ',b o ); Formula (16)

[0174] Among them, b o b represents the predicted bounding box. g ' represents the updated predicted bounding box, offset is the bounding box position offset, diff is the bounding box size offset, and b g [:,:2] represents the original actual bounding box obtained with the actual center point as the parameter, b g [:,2:] represents the original actual border obtained using the actual width and height as parameters, b o [:,:2] represents the predicted bounding box obtained with the predicted center point as a parameter, b o [:,2:] represents the predicted border obtained from the predicted width and height; σ represents the tolerance for the annotation position. This indicates the tolerance for the size of the annotation border; b gc b is the first intermediate variable. gs This represents the second intermediate variable.

[0175] After obtaining the generalized intersection-over-union ratio (GOU), the first extended intersection-over-union ratio (EIOU) can be determined based on the GOU. Then, the category difference between the defect category detection result corresponding to the current predicted bounding box and the actual defect category corresponding to the original actual bounding box is determined. Finally, the bounding box matching degree is set to: -(category difference + first EIOU). After obtaining the bounding box matching degree, the current predicted bounding boxes can be sorted according to the bounding box matching degree to obtain the bounding box sorting result.

[0176] b) Calculate the second EIOU between the current actual bounding box (gt bbox) and the regression predicted bounding box (bboxes) corresponding to the predicted bounding box (bboxes); where, taking an industrial product as an example, the specific current actual bounding box can be referenced... Figure 15As shown in 1501; then, the sum of the second EIOUs is calculated to obtain the summation result; at the same time, the obtained summation result is the number of predicted bounding boxes (bboxes) that can be matched by the corresponding current actual bounding box gt bbox (i.e., the number of second bounding boxes); the minimum number that can be matched is 1; it should also be noted here that if there are multiple regression predicted bounding boxes corresponding to the current actual bounding box, the second EIOUs corresponding to each regression predicted bounding box are summed to obtain the number of second bounding boxes; if there is only one regression predicted bounding box corresponding to the current actual bounding box, the second EIOU is directly used as the number of second bounding boxes; at the same time, if the second EIOU is less than 1, the number of second bounding boxes is set to 1.

[0177] c) Based on the border sorting results and the number of second borders, select the corresponding number of bboxes with the highest matching degree for each current actual border gt bbox; that is, the predicted border with the highest matching degree can be selected from the border sorting results based on the number of second borders as the foreground predicted border, and then the remaining predicted borders are used as the background border; based on this, the label assignment of the predicted borders can be realized.

[0178] 7) Based on the assignment results of the above label assignment algorithm, and using EIOU loss and cross-entropy loss as the localization and classification losses respectively, the first loss value is obtained; wherein, the obtained first loss function can be shown in the following formula (17);

[0179]

[0180] Where L0 is the first loss function, For the category loss function, λ0 is the location loss function, and λ0 is the loss balance parameter, which can be used to balance the location and classification losses. In practical applications, the specific value of λ0 can be set according to expert experience or determined based on the parameter prediction model. This example does not impose any special restrictions on this. Under the premise that the first loss function exists, the first loss value can be obtained by substituting the specific value into formula (17). Furthermore, the category loss function includes the loss of the foreground predicted bounding box and the background predicted bounding box, while the location loss function only includes the loss of the foreground predicted bounding box. Therefore, it is necessary to set the corresponding balance parameter to balance the location and classification losses.

[0181] In step S130, a second loss value is determined based on the historical defect detection image and the image reconstruction result, and a total loss value is determined based on the first loss value and the second loss value.

[0182] In this example embodiment, firstly, a second loss value is determined based on the historical defect detection image and the image reconstruction result. Specifically, this can be achieved as follows: the historical defect detection image and the image reconstruction result are processed in grayscale to obtain a first grayscale image and a second grayscale image; the first grayscale value of the first grayscale image and the second grayscale value of the second grayscale image are extracted, and the second loss value is determined based on the difference between the first grayscale value and the second grayscale value. The obtained second loss function can be referenced as shown in the following formula (18):

[0183] L rec =|I g -I|; Formula (18)

[0184] Among them, L rec For the second loss function, I g The obtained image reconstruction result is I, which is the historical defect detection image. Under the premise that the second loss function exists, the first gray value and the second gray value are substituted into formula (18) to obtain the second loss value.

[0185] Secondly, the total loss value is determined based on the first loss value and the second loss value; wherein, the obtained target loss function can be expressed by the following formula (19):

[0186] L = L0 + L rec ; Formula (19)

[0187] Meanwhile, in practical applications, the first loss function and the second loss function can be directly added together to obtain the target loss function, or the first loss function and the second loss function can be assigned corresponding weight values ​​and weighted summed to obtain the target loss function. This example does not impose any special restrictions on this. Furthermore, under the premise that the target loss function exists, the first loss value and the second loss value can be substituted into formula (19) to obtain the total loss value.

[0188] In step S140, the network model to be trained is trained based on the total loss value to obtain the trained network model, and the product defect detection model is determined based on the trained network model.

[0189] In this example embodiment, after obtaining the total loss value, all model parameters included in the network model to be trained can be adjusted to obtain the trained network model. Then, based on the trained network model, a product defect detection model is determined. Specifically, this can be achieved by pruning the structure-aware network in the trained network model and generating the product defect detection model based on the pruned network model. That is, compared to the network model to be trained, the obtained product defect detection model may include a backbone feature extraction network, a feature fusion network, and a defect prediction network. The specific model structure can be found in [reference needed]. Figure 16 As shown.

[0190] Thus, the training method for the defect detection model described in the exemplary embodiments of this disclosure has been fully implemented. Furthermore, the exemplary embodiments of this disclosure also provide a defect detection method for industrial products; specifically, refer to... Figure 17 As shown, the defect detection method for this industrial product may include the following steps:

[0191] Step S1710: Obtain the image of the industrial product to be inspected;

[0192] Step S1720: Input the image of the product to be detected into the product defect detection model to obtain the defect detection result; wherein, the product defect detection model is trained by the training method of the defect detection model described in any one of the above-mentioned methods;

[0193] Step S1730: Determine the defect location of the industrial product to be inspected and the defect category corresponding to the defect location based on the defect detection results.

[0194] In the aforementioned defect detection method for industrial products, the model, during training, considers not only its defect detection capability but also its ability to perceive and understand the product structure of industrial products corresponding to historical defect detection images. Based on this, the model can perceive both complete and incomplete structural defects during defect detection, thus solving the problem of low accuracy in detecting incomplete structural defects and improving the accuracy of the obtained defect detection results. Furthermore, the obtained defect detection results can detect not only the defect location and type of known structures but also the defect location and type of missing structures.

[0195] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0196] This disclosure also provides an example embodiment of a training apparatus for a defect detection model. Specifically, refer to... Figure 18 As shown, the training device for the defect detection model may include a defect detection module 1810, a first loss value determination module 1820, a total loss value determination module 1830, and a product defect detection model determination module 1840. Wherein:

[0197] The defect detection module 1810 can be used to perform defect detection and image reconstruction on historical defect detection images based on the network model to be trained, and obtain defect detection results and image reconstruction results; wherein, the image reconstruction results are used to enhance the network model to be trained to perceive the product structure of the industrial product corresponding to the historical defect detection image.

[0198] The first loss value determination module 1820 can be used to determine a first loss value based on the defect detection result and the defect annotation result of the historical defect detection image;

[0199] The total loss value determination module 1830 can be used to determine a second loss value based on the historical defect detection image and the image reconstruction result, and to determine a target loss value based on the first loss value and the second loss value.

[0200] The product defect detection model determination module 1840 can be used to train the network model to be trained based on the target loss value, obtain the trained network model, and determine the product defect detection model based on the trained network model.

[0201] In an exemplary embodiment of this disclosure, the network model to be trained includes a backbone feature extraction network, a feature fusion network, a defect prediction network, and a structure-aware network. The process of performing defect detection and image reconstruction on historical defect detection images based on the network model to be trained, to obtain defect detection results and image reconstruction results, includes: downsampling the historical defect detection images based on the backbone feature extraction network to obtain multiple local image features at different scales; fusing the multiple local image features at different scales based on the feature fusion network to obtain global image features, and performing defect detection on the global image features based on the defect prediction network to obtain the defect detection results; and masking the historical defect detection images based on the structure-aware network to obtain masked image regions, and decoding the masked image regions based on the global image features to obtain the image reconstruction results.

[0202] In an exemplary embodiment of this disclosure, the backbone feature extraction network includes a convolutional backbone layer, an original feature sampling layer, and N feature downsampling layers, where N is a positive integer. The process of downsampling the historical defect detection image based on the backbone feature extraction network to obtain multiple local image features at different scales includes: extracting basic features from the historical defect detection image based on the convolutional backbone layer to obtain basic feature extraction results; performing initial sampling on the basic feature extraction results based on the original feature sampling layer to obtain initial feature sampling results; performing a first downsampling on the initial feature sampling results based on a first feature downsampling layer among the N feature downsampling layers to obtain a first local image feature; performing a second downsampling on the first local image feature based on the first feature downsampling layer among the N feature downsampling layers to obtain a second local image feature; and repeating the generation process of the second local image feature sequentially to obtain a third local image feature, ..., an Nth local image feature.

[0203] In an exemplary embodiment of this disclosure, the feature fusion network includes an original feature fusion network and N sub-feature fusion networks. The process of fusing multiple local image features at different scales based on the feature fusion network to obtain global image features includes: fusing the Nth local image feature among the multiple local image features at different scales based on the original feature fusion network to obtain an original global feature; fusing the original global feature and N-1 local image features among the multiple local image features at different scales based on a first sub-feature fusion network among the N sub-feature fusion networks to obtain a first global feature; fusing the first global feature and the N-2th local feature among the multiple local image features at different scales based on a second feature fusion network among the N sub-feature fusion networks to obtain a second global feature; and repeating the generation process of the second global feature sequentially to obtain a third global feature, ..., an Nth global feature.

[0204] In an exemplary embodiment of this disclosure, the defect region prediction network includes a first convolutional layer, a defect category detection layer, and a defect region detection layer; wherein, performing defect detection on the global image features based on the defect prediction network to obtain the defect detection result includes: performing convolution processing on the global image features based on the first convolutional layer to obtain a first convolution processing result; performing classification processing on the first convolution processing result based on the defect category detection to obtain a defect category detection result for the target object included in the global image features; wherein the target object is a defect corresponding to an industrial product; and performing bounding box detection processing on the first convolution processing result based on the defect region detection layer to obtain a defect location detection result for the target object included in the global image features.

[0205] In an exemplary embodiment of this disclosure, the structure-aware network includes a masking layer, an embedding layer, a decoding layer, and an image reconstruction layer. The process involves masking the historical defect detection image using the structure-aware network to obtain a masked image region, and then decoding the masked image region based on the global image features to obtain the image reconstruction result. This includes: masking the historical defect detection image using the masking layer to obtain a masked image region and a local masked image corresponding to the masked image region; embedding the local masked image into the masked image region corresponding to the historical defect detection image using the embedding layer to obtain a global masked image; decoding the global image features using the decoding layer to obtain a local image decoding result corresponding to the masked image region; and reconstructing the image using the local image decoding result and the global masked image using the image reconstruction layer to obtain the image reconstruction result.

[0206] In one exemplary embodiment of this disclosure, masking processing is performed on the historical defect detection image to obtain a masked image region and a local masked image corresponding to the masked image region. This includes: scaling the original global features, the first global feature, ..., the Nth global feature in the global image features, and adding the scaled global features to obtain a global summed feature; determining the masked image region based on the global feature width and the global feature length of the global summed feature; and performing masking processing on the image region in the historical defect detection image corresponding to the position of the masked image region to obtain a local masked image corresponding to the masked image region.

[0207] In an exemplary embodiment of this disclosure, the defect annotation result of the historical defect detection image includes the original actual bounding box and the actual defect category corresponding to the original actual bounding box; the defect detection result includes the defect location detection result and the defect category detection result corresponding to the defect location detection result; wherein, determining the first loss value based on the defect detection result and the defect annotation result of the historical defect detection image includes: determining the current predicted bounding box based on the defect location detection result; classifying the current predicted bounding box to obtain a foreground predicted bounding box and a background predicted bounding box; determining a location loss value based on the background predicted bounding box and the original actual bounding box; determining a category loss value based on the defect category detection result and the actual defect category, and determining the first loss value based on the location loss value and the category loss value.

[0208] In an exemplary embodiment of this disclosure, determining a predicted bounding box based on the defect location detection result includes: determining the location anchor points and step size information of global image features; decoding the defect location detection result based on the location anchor points and step size information to obtain a predicted bounding box; wherein the number of first bounding boxes of the predicted bounding box is the same as the number of anchor points of the location anchor points.

[0209] In one exemplary embodiment of this disclosure, classifying the current predicted bounding box to obtain foreground predicted bounding boxes and background predicted bounding boxes includes: calculating the bounding box matching degree between the current predicted bounding box and the original actual bounding box, and sorting the current predicted bounding boxes according to the bounding box matching degree to obtain a bounding box sorting result; calculating the second number of current predicted bounding boxes that can be matched by the original actual bounding box; extracting current predicted bounding boxes corresponding to the second number of bounding boxes from the current predicted bounding boxes according to the bounding box sorting result as foreground predicted bounding boxes corresponding to the target actual bounding box, and taking at least a portion of the other bounding boxes in the current predicted bounding boxes other than the foreground predicted bounding boxes as background predicted bounding boxes corresponding to the target actual bounding box.

[0210] In an exemplary embodiment of this disclosure, calculating the border matching degree between the current predicted border and the original actual border includes: determining whether the predicted center point of the current predicted border is included within a first border range corresponding to the original actual border, or whether the predicted center point is included within a first preset region corresponding to the actual center point of the original actual border; if the predicted center point is not included within the first border range or is not included within the first preset region, then the border matching degree between the current predicted border and the original actual border is determined to be infinitesimal; if the predicted center point is included within the first border range or is included within the first preset region, then the border matching degree is determined based on a first extended intersection-union ratio between the current predicted border and the original actual border.

[0211] In one exemplary embodiment of this disclosure, determining the border matching degree based on a first extended intersection-union ratio (IUU) between the currently predicted border and the original actual border includes: determining a border position offset and a border size offset between the currently predicted border and the original actual border, and adjusting the original actual border according to the border position offset and the border size offset to obtain a target actual border; determining a generalized intersection-union ratio (GUU) between the target actual border and the currently predicted border, and determining a first extended intersection-union ratio (GUU) between the currently predicted border and the original actual border based on the GUU; determining a category difference between a defect category detection result corresponding to the currently predicted border and an actual defect category corresponding to the original actual border, and determining the border matching degree based on the category difference and the first extended intersection-union ratio (GUU).

[0212] In one exemplary embodiment of this disclosure, calculating the number of second borders that the original actual border can match the current predicted border includes: determining a regression predicted border corresponding to the current predicted border, and determining a second extended intersection-union ratio (ECU) between the original actual border and the regression predicted border; and determining the number of second borders that the original actual border can match the current predicted border based on the second extended ECU.

[0213] In one exemplary embodiment of this disclosure, determining a position loss value based on the foreground predicted bounding box and the original actual bounding box in the defect annotation result of the historical defect detection image includes: calculating a third extended cross-union ratio (CUI) between the foreground predicted bounding box and the original actual bounding box in the defect annotation result of the historical defect detection image, and determining the position loss value based on the third extended CUI.

[0214] In one exemplary embodiment of this disclosure, determining a second loss value based on the historical defect detection image and the image reconstruction result includes: performing grayscale processing on the historical defect detection image and the image reconstruction result to obtain a first grayscale image and a second grayscale image; extracting a first grayscale value from the first grayscale image and a second grayscale value from the second grayscale image; and determining the second loss value based on the difference between the first grayscale value and the second grayscale value.

[0215] In one exemplary embodiment of this disclosure, determining a product defect detection model based on a trained network model includes: pruning the structure-aware network in the trained network model, and generating the product defect detection model based on the pruned network model.

[0216] This disclosure also provides an example embodiment of a defect detection device for industrial products. Specifically, refer to... Figure 19 As shown, the defect detection device for industrial products may include a product image acquisition module 1910, a defect detection result determination module 1920, and a defect location and category determination module 1930. Wherein:

[0217] The product image acquisition module can be used to acquire product images of industrial products to be inspected.

[0218] The defect detection result determination module 1920 can be used to input the image of the product to be detected into the product defect detection model to obtain the defect detection result; wherein, the product defect detection model is trained by the defect detection model training method described in any one of the above.

[0219] The defect location and category determination module 1930 can be used to determine the defect location of the industrial product to be inspected and the defect category corresponding to the defect location based on the defect detection results.

[0220] The specific details of the training device for the aforementioned defect detection model and the defect detection device for industrial products have been described in detail in the corresponding training methods for defect detection models and defect detection methods for industrial products, so they will not be repeated here.

[0221] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0222] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0223] In exemplary embodiments of this disclosure, an electronic device capable of implementing the above-described methods is also provided. Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as circuits, modules, or systems.

[0224] The following reference Figure 20 To describe an electronic device 2000 according to such an embodiment of the present disclosure. Figure 20 The electronic device 2000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0225] like Figure 20 As shown, the electronic device 2000 is manifested in the form of a general-purpose computing device. The components of the electronic device 2000 may include, but are not limited to: at least one processing unit 2010, at least one storage unit 2020, a bus 2030 connecting different system components (including storage unit 2020 and processing unit 2010), and a display unit 2040.

[0226] The storage unit stores program code that can be executed by the processing unit 2010, causing the processing unit 2010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 2010 can perform actions such as... Figure 1 Step S110: Based on the network model to be trained, perform defect detection and image reconstruction on the historical defect detection image to obtain defect detection results and image reconstruction results; wherein, the image reconstruction results are used to enhance the network model to be trained's perception ability of the product structure of the industrial product corresponding to the historical defect detection image; Step S120: Determine a first loss value based on the defect detection results and the defect annotation results of the historical defect detection image; Step S130: Determine a second loss value based on the historical defect detection image and the image reconstruction results, and determine a total loss value based on the first loss value and the second loss value; Step S140: Train the network model to be trained based on the total loss value to obtain a trained network model, and determine a product defect detection model based on the trained network model.

[0227] For example, the processing unit 2010 can perform actions such as Figure 17 Step S1710: Obtain the image of the industrial product to be inspected; Step S1720: Input the image of the industrial product to be inspected into the product defect detection model to obtain the defect detection result; wherein, the product defect detection model is trained by any of the above-described defect detection model training methods; Step S1730: Determine the defect location of the industrial product to be inspected and the defect category corresponding to the defect location based on the defect detection result.

[0228] Storage unit 2020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 20201 and / or cache memory 20202, and may further include read-only memory (ROM) 20203.

[0229] The storage unit 2020 may also include a program / utility 20204 having a set (at least one) program module 20205, such program module 20205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0230] Bus 2030 can represent one or more of several types of bus structures, including memory cell bus or memory cell controller, peripheral bus, graphics acceleration port, processing unit, or local bus using any of the multiple bus structures.

[0231] Electronic device 2000 can also communicate with one or more external devices 2100 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 2000, and / or with any device that enables electronic device 2000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 2050. Furthermore, electronic device 2000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 2060. As shown, network adapter 2060 communicates with other modules of electronic device 2000 via bus 2030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 2000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0232] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods described in the embodiments of this disclosure.

[0233] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Exemplary Methods" section above.

[0234] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0235] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0236] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0237] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0238] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0239] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0240] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not invented by this disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A training method for a defect detection model, characterized in that, include: Based on the network model to be trained, defect detection and image reconstruction are performed on historical defect detection images to obtain defect detection results and image reconstruction results; wherein, the image reconstruction results are used to enhance the network model to be trained to perceive the product structure of the industrial products corresponding to the historical defect detection images. The first loss value is determined based on the defect detection results and the defect annotation results of the historical defect detection images; A second loss value is determined based on the historical defect detection images and image reconstruction results, and a total loss value is determined based on the first loss value and the second loss value. The network model to be trained is trained based on the total loss value to obtain the trained network model, and the product defect detection model is determined based on the trained network model.

2. The training method for the defect detection model according to claim 1, characterized in that, The network model to be trained includes a backbone feature extraction network, a feature fusion network, a defect prediction network, and a structure-aware network. Specifically, based on the network model to be trained, defect detection and image reconstruction are performed on historical defect detection images to obtain defect detection results and image reconstruction results, including: The historical defect detection image is downsampled based on the backbone feature extraction network to obtain multiple local image features at different scales. The feature fusion network is used to fuse the local image features of multiple different scales to obtain global image features, and the defect prediction network is used to detect defects in the global image features to obtain the defect detection result. The historical defect detection image is masked based on the structure-aware network to obtain a masked image region, and the masked image region is decoded based on the global image features to obtain the image reconstruction result.

3. The training method for the defect detection model according to claim 2, characterized in that, The backbone feature extraction network includes a convolutional backbone layer, an original feature sampling layer, and N feature downsampling layers, where N is a positive integer. Specifically, the historical defect detection image is downsampled based on the backbone feature extraction network to obtain multiple local image features at different scales, including: Based on the convolutional dry layer, basic features are extracted from the historical defect detection image to obtain basic feature extraction results. Then, based on the original feature sampling layer, initial sampling is performed on the basic feature extraction results to obtain initial feature sampling results. The initial feature sampling result is downsampled for the first time based on the first feature downsampling layer among the N feature downsampling layers to obtain the first local image feature; The first local image features are downsampled a second time based on the first feature downsampling layer among the N feature downsampling layers to obtain the second local image features; The generation process of the second local image feature is repeated sequentially to obtain the third local image feature, ..., the Nth local image feature.

4. The training method for the defect detection model according to claim 2, characterized in that, The feature fusion network includes an original feature fusion network and N sub-feature fusion networks; Specifically, the feature fusion network fuses the local image features at multiple different scales to obtain global image features, including: Based on the original feature fusion network, the Nth local image feature among the multiple local image features of different scales is fused to obtain the original global feature; The first sub-feature fusion network in the N sub-feature fusion network is used to fuse the original global features and N-1 local image features of multiple different scales to obtain the first global feature; The second feature fusion network in the N sub-feature fusion network fuses the first global feature and the N-2th local feature in multiple local image features of different scales to obtain the second global feature. The generation process of the second global feature is repeated sequentially to obtain the third global feature, ..., the Nth global feature.

5. The training method for the defect detection model according to claim 2, characterized in that, The defect region prediction network includes a first convolutional layer, a defect category detection layer, and a defect region detection layer. Specifically, the defect detection is performed on the global image features based on the defect prediction network to obtain the defect detection result, including: The global image features are convolved based on the first convolutional layer to obtain the first convolutional processing result; Based on the defect category detection, the first convolution processing result is classified to obtain the defect category detection result of the target object included in the global image features; wherein, the target object is the defect corresponding to the industrial product; Based on the defect region detection layer, the first convolution processing result is subjected to border detection processing to obtain the defect location detection result of the target object included in the global image features.

6. The training method for the defect detection model according to claim 2, characterized in that, The structure-aware network includes a masking layer, an embedding layer, a decoding layer, and an image reconstruction layer; Specifically, the historical defect detection image is masked based on the structure-aware network to obtain a masked image region, and the masked image region is decoded based on the global image features to obtain the image reconstruction result, including: The historical defect detection image is masked based on the masking layer to obtain a masked image region and a local masked image corresponding to the masked image region. Based on the embedding layer, the local mask image is embedded into the mask image region corresponding to the historical defect detection image to obtain a global mask image; Based on the decoding layer, the global image features are decoded to obtain the local image decoding result corresponding to the masked image region; Based on the image reconstruction layer, the local image decoding result and the global mask image are used to reconstruct the image, and the image reconstruction result is obtained.

7. The training method for the defect detection model according to claim 6, characterized in that, The historical defect detection image is masked to obtain a masked image region and a local masked image corresponding to the masked image region, including: The original global features, the first global features, ..., the Nth global features in the global image features are scaled, and the scaled global features are added together to obtain the global summed features; Based on the global feature width and global feature length of the global summation feature, the mask image region is determined, and the image region in the historical defect detection image corresponding to the position of the mask image region is masked to obtain a local mask image corresponding to the mask image region.

8. The training method for the defect detection model according to claim 1, characterized in that, The defect annotation results of the historical defect detection images include the original actual border and the actual defect category corresponding to the original actual border; The defect detection results include defect location detection results and defect category detection results corresponding to the defect location detection results; The determination of the first loss value based on the defect detection results and the defect annotation results of the historical defect detection images includes: The current predicted bounding box is determined based on the defect location detection result. The current predicted bounding box is then classified to obtain the foreground predicted bounding box and the background predicted bounding box. The position loss value is determined based on the predicted background bounding box and the original actual bounding box; The category loss value is determined based on the defect category detection results and the actual defect category, and the first loss value is determined based on the location loss value and the category loss value.

9. The training method for the defect detection model according to claim 8, characterized in that, Determining the predicted bounding box based on the defect location detection results includes: Determine the location anchor points and stride information of global image features; The defect location detection result is decoded based on the location anchor points and step size information to obtain a predicted bounding box; wherein the number of the first bounding box of the predicted bounding box is the same as the number of anchor points of the location anchor points.

10. The training method for the defect detection model according to claim 8, characterized in that, The currently predicted bounding boxes are classified to obtain foreground predicted bounding boxes and background predicted bounding boxes, including: Calculate the border matching degree between the current predicted border and the original actual border, and sort the current predicted borders according to the border matching degree to obtain the border sorting result; Calculate the number of second borders that the original actual borders can match the currently predicted borders; Based on the border sorting result, extract the current predicted border corresponding to the second border quantity from the current predicted borders as the foreground predicted border corresponding to the target actual border, and take at least some of the other borders in the current predicted borders other than the foreground predicted borders as the background predicted borders corresponding to the target actual border.

11. The training method for the defect detection model according to claim 10, characterized in that, Calculating the border matching degree between the current predicted border and the original actual border includes: Determine whether the predicted center point of the current predicted border is included within the range of the first border corresponding to the original actual border, or whether the predicted center point is included within the first preset area corresponding to the actual center point of the original actual border. If the predicted center point is not included within the first border or is not included within the first preset area, then the border matching degree between the current predicted border and the original actual border is determined to be infinitesimal. If the predicted center point is included within the first border or within the first preset area, the border matching degree is determined based on the first extended intersection-union ratio between the current predicted border and the original actual border.

12. The training method for the defect detection model according to claim 11, characterized in that, Determining the border matching degree based on the first extended intersection-union ratio (IUU) between the currently predicted border and the original actual border includes: Determine the border position offset and border size offset between the current predicted border and the original actual border, and adjust the original actual border according to the border position offset and border size offset to obtain the target actual border; Determine the generalized intersection-union ratio (GUOR) between the target actual bounding box and the current predicted bounding box, and based on the GUOR, determine the first extended intersection-union ratio (AUR) between the current predicted bounding box and the original actual bounding box; Determine the category difference between the defect category detection result corresponding to the current predicted bounding box and the actual defect category corresponding to the original actual bounding box, and determine the bounding box matching degree based on the category difference and the first extended intersection-union ratio.

13. The training method for the defect detection model according to claim 10, characterized in that, Calculating the number of second borders that the original actual border can match the currently predicted border includes: Determine the regression prediction bounding box corresponding to the current prediction bounding box, and determine the second extended intersection-union ratio between the original actual bounding box and the regression prediction bounding box; The number of second borders that the original actual borders can match based on the second extended intersection-union ratio is determined.

14. The training method for the defect detection model according to claim 8, characterized in that, Based on the predicted foreground bounding box and the original actual bounding box in the defect annotation results of the historical defect detection image, the position loss value is determined, including: Calculate the third extended cross-union ratio (CUIR) between the predicted foreground bounding box and the original actual bounding box in the defect annotation result of the historical defect detection image, and determine the position loss value based on the third extended CUIR.

15. The training method for the defect detection model according to claim 1, characterized in that, The second loss value is determined based on the historical defect detection images and image reconstruction results, including: The historical defect detection image and the image reconstruction result are subjected to grayscale processing to obtain a first grayscale image and a second grayscale image; Extract the first grayscale value of the first grayscale image and the second grayscale value of the second grayscale image, and determine the second loss value based on the difference between the first grayscale value and the second grayscale value.

16. The training method for the defect detection model according to claim 1, characterized in that, Based on the trained network model, a product defect detection model is determined, including: The structure-aware network in the trained network model is pruned, and the product defect detection model is generated based on the pruned network model.

17. A method for detecting defects in industrial products, characterized in that, include: Acquire images of the industrial products to be inspected; The product image to be detected is input into the product defect detection model to obtain the defect detection result; wherein the product defect detection model is trained by the training method of the defect detection model according to any one of claims 1-16; The defect location and the corresponding defect category of the industrial product to be inspected are determined based on the defect detection results.

18. A training device for a defect detection model, characterized in that, include: The defect detection module is used to perform defect detection and image reconstruction on historical defect detection images based on the network model to be trained, and obtain defect detection results and image reconstruction results; wherein, the image reconstruction results are used to enhance the network model to be trained to perceive the product structure of the industrial product corresponding to the historical defect detection images. The first loss value determination module is used to determine a first loss value based on the defect detection results and the defect annotation results of the historical defect detection images; The total loss value determination module is used to determine a second loss value based on the historical defect detection image and the image reconstruction result, and to determine a total loss value based on the first loss value and the second loss value; The product defect detection model determination module is used to train the network model to be trained based on the total loss value, obtain the trained network model, and determine the product defect detection model based on the trained network model.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the defect detection model according to any one of claims 1-16, and the defect detection method of the industrial product according to claim 17.

20. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the training method of the defect detection model according to any one of claims 1-16 and the defect detection method of the industrial product according to claim 17 by executing the executable instructions.

Images