An appearance defect detection method and device, a storage medium and an electronic device

Abstract

The application provides an appearance defect detection method and device, a storage medium and electronic equipment. A feature extraction branch extracts features of defect pixel points in a target image according to a preset scale to obtain image features in N dimensions. A feature fusion branch fuses image features in adjacent dimensions to obtain N-1 fusion features. A decoding branch decodes the N-1 fusion features to obtain a decoded image result, wherein the decoded image result includes first-type feature pixel points and second-type feature pixel points. The first-type feature pixel points represent defect pixel points, and the second-type feature pixel points represent non-defect pixel points. The method can accurately detect defects in a low-contrast environment, i.e., when the difference between the defects and the background is not obvious. The method can eliminate the interference of factors such as stains, shadows and uneven illumination on defect detection, and can adapt to defect detection of various shapes and sizes.

Description

Technical Field

[0001] This application relates to the field of images, and more specifically, to a method, apparatus, storage medium, and electronic device for detecting appearance defects. Background Technology

[0002] With the development of science and technology and the improvement of people's living standards, the requirements for the quality of industrial products are becoming increasingly higher. Surface quality inspection is a crucial step before products leave the factory. Traditional inspection methods involve experienced workers visually inspecting products and recording defective product numbers for further processing. This method has many problems, such as worker fatigue from prolonged visual inspection leading to missed or false detections, the significant subjective influence of manual inspection, and low inspection efficiency. Therefore, there is an urgent need in industrial production to apply advanced, automated, accurate, and efficient surface defect detection systems to effectively free up manual labor and improve industrial production efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, storage medium, and electronic device for detecting appearance defects, so as to at least partially improve the above-mentioned problems.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0005] In a first aspect, embodiments of this application provide a method for detecting appearance defects, applied to an electronic device, wherein the electronic device is equipped with a pre-trained network model, the network model including a feature extraction branch, a feature fusion branch, and a decoding branch, and the method includes:

[0006] The feature extraction branch extracts features from defective pixels in the target image according to a preset scale to obtain image features in N dimensions, wherein the target image is the acquired image of the detection object;

[0007] The feature fusion branch fuses image features from adjacent dimensions to obtain N-1 fused features;

[0008] The decoding branch performs decoding based on the N-1 fusion features to obtain a decoded image result, wherein the decoded image result includes a first type of feature pixels and a second type of feature pixels, the first type of feature pixels representing defective pixels and the second type of feature pixels representing non-defective pixels.

[0009] Secondly, embodiments of this application provide an appearance defect detection device, applied to electronic devices, the device comprising:

[0010] The feature extraction unit is used to extract features from defective pixels in the target image according to a preset scale to obtain image features in N dimensions, wherein the target image is the acquired image of the detection object;

[0011] The feature fusion unit is used to fuse image features in adjacent dimensions to obtain N-1 fused features;

[0012] The decoding unit is used to decode based on the N-1 fused features to obtain a decoded image result, wherein the decoded image result includes a first type of feature pixels and a second type of feature pixels, the first type of feature pixels representing defective pixels and the second type of feature pixels representing non-defective pixels.

[0013] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method.

[0014] Fourthly, embodiments of this application provide an electronic device, the electronic device comprising: a processor and a memory, the memory being used to store one or more programs; when the one or more programs are executed by the processor, the above-described method is implemented.

[0015] Compared to existing technologies, the appearance defect detection method, apparatus, storage medium, and electronic device provided in this application have the following features: a feature extraction branch extracts features from defect pixels in a target image according to a preset scale to obtain N-dimensional image features; a feature fusion branch, where the target image is the object of detection, fuses image features in adjacent dimensions to obtain N-1 fused features; and a decoding branch decodes based on the N-1 fused features to obtain a decoded image result, which includes a first type of feature pixels and a second type of feature pixels, where the first type of feature pixels represents defect pixels and the second type of feature pixels represents non-defect pixels. This method can accurately detect defects even in low-contrast conditions, i.e., when the difference between the defect and the background is not significant. It can eliminate interference from stains, shadows, uneven lighting, and other factors, and can adapt to the detection of defects of various shapes and sizes, thus overcoming the problems existing in existing technologies.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0019] Figure 2 This is a schematic diagram of the network model provided in the embodiments of this application;

[0020] Figure 3 A schematic flowchart illustrating the appearance defect detection method provided in this application embodiment;

[0021] Figure 4 A schematic diagram of the sub-steps of S102 provided in the embodiments of this application;

[0022] Figure 5 This is a schematic diagram of the structure of the SCA provided in an embodiment of this application;

[0023] Figure 6 A schematic diagram of the sub-steps of S103 provided in the embodiments of this application;

[0024] Figure 7 A schematic diagram of the structure of CEF provided in the embodiments of this application;

[0025] Figure 8 A test result diagram of the MT dataset provided in the embodiments of this application;

[0026] Figure 9 The test results of the NEU-Seg dataset provided in the embodiments of this application are shown in the figure.

[0027] Figure 10 The test result diagram of the road defect dataset provided in the embodiments of this application;

[0028] Figure 11 A graph showing the test results of the RSDD dataset provided in the embodiments of this application;

[0029] Figure 12 This is a schematic diagram of a unit of the appearance defect detection device provided in an embodiment of this application.

[0030] In the diagram: 10-Processor; 11-Memory; 12-Bus; 13-Communication interface; 201-Feature extraction unit; 202-Feature fusion unit; 203-Decoding unit. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0034] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0035] In the description of this application, it should be noted that the terms "upper", "lower", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0036] In the description of this application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0037] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0038] In recent years, machine vision and image processing technologies have developed rapidly and have been applied to surface defect detection systems. To effectively detect defects, researchers have proposed methods based on defect feature extraction, which can be mainly divided into three categories: statistical methods, spectral methods, and model-based methods. However, the choice of which and how many features to use is highly subjective, and the selected features directly affect the defect detection performance. Furthermore, the detection results of these methods are also highly susceptible to environmental factors. Changes in lighting, background, and camera angle have a significant impact on detection performance. In other words, these traditional machine vision methods have poor generalization performance and struggle to accurately detect complex industrial surface defects. The development of deep learning and computer vision has provided ideas and theoretical support for the detection of complex defects. Deep learning methods can automatically extract effective feature information of objects, overcoming the shortcomings of manual feature selection. In particular, the use of multi-scale feature fusion and attention mechanisms can obtain richer and more comprehensive defect feature information.

[0039] However, using computer vision for surface defect detection still faces three major challenges: 1) Low contrast: The difference between defects and the background is not obvious, and some defects are very small, making them difficult to distinguish and detect; 2) Interference factors such as stains, shadows, and uneven lighting in the image increase the complexity of defect detection; 3) Defects vary greatly in shape, size, and other dimensions. The generation of defects is random due to factors such as different production lines and image acquisition equipment.

[0040] To address the three major challenges of existing surface defect detection methods, this application proposes a semantically aligned cross-scale feature enhancement fusion network (SCFNet), hereinafter referred to as the network model. This network model mainly consists of an encoder and a decoder, used for pixel binary classification segmentation. In the encoder, a novel neighboring feature fusion method is proposed. Multi-level features from the Transformer backbone are fused using a proposed semantically aligned attention mechanism for every two adjacent levels, and channel and spatial attention are used to enhance the highest-level features. In the decoder, a foreground enhancement module is proposed to enhance the defect features as the foreground. This application provides a multi-scenario applicable method for detecting surface defects in industrial products, which can comprehensively detect surface defects and thus overcome the aforementioned problems.

[0041] This application provides an electronic device, which may be a server device, a computer device, or other terminal devices with signal processing capabilities, such as a mobile phone. The electronic device is equipped with a pre-trained network model, which includes a feature extraction branch, a feature fusion branch, and a decoding branch.

[0042] Please refer to Figure 1 This is a schematic diagram of the structure of an electronic device. The electronic device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12. The processor 10 is used to execute executable modules, such as computer programs, stored in the memory 11.

[0043] Processor 10 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the appearance defect detection method can be completed through integrated logic circuits in the hardware or software instructions within processor 10. The processor 10 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0044] The memory 11 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0045] Bus 12 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. Figure 1 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus 12 or one type of bus 12.

[0046] The memory 11 is used to store programs, such as programs corresponding to an appearance defect detection device. The appearance defect detection device includes at least one software functional module that can be stored in the memory 11 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device. Upon receiving an execution instruction, the processor 10 executes the program to implement the appearance defect detection method.

[0047] Possibly, the electronic device provided in this application embodiment also includes a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus. The communication interface 13 is used to establish communication connections with other terminals, for example, to acquire target images transmitted by other terminals based on the communication interface 13.

[0048] To address the challenges of existing deep learning-based surface defect detection methods, this application proposes a semantically aligned cross-scale fusion network, called SCFNet (Semantic-aligned Cross-scale enhanced Fusion Network). It aligns multi-scale features through cross-attention and enhances the fusion of features from the encoder across scales in the decoder, thereby improving detection accuracy and robustness. Please refer to [reference needed]. Figure 2 , Figure 2 This is a schematic diagram of the network model provided in an embodiment of this application. Figure 2 As shown, the network model includes a feature extraction branch, a feature fusion branch, and a decoding branch. The feature extraction branch includes one Embedding Block and N-1 Merging Blocks, the feature fusion branch includes N-1 SCAs, and the decoding branch includes N-1 CEFs. Optionally, the network model also includes CBAM.

[0049] Figure 2 The leftmost blocks are the backbone Swing Transformer structure, SCA is the semantically aligned attention fusion structure, CEF is the cross-scale enhanced fusion structure, and CBAM is the channel and spatial attention structure.

[0050] It should be understood that, due to the concise and efficient structure of the Swin Transformer backbone network, it can be trained directly end-to-end without the need for cumbersome multiple or sub-structural training. Therefore, in this application, the pre-trained parameters of the Swin Transformer backbone network are first loaded as parameters for the feature extraction branch. These parameters are obtained through classification training on ImageNet1k, which is unrelated to defect detection, using a transfer learning method widely used in computer vision. Other parameters in the network are obtained using the default Kaiming initialization method in the PyTorch framework. This operation is primarily to accelerate network convergence, especially for the transformer backbone; loading pre-trained parameters can significantly speed up network training and may also improve accuracy to some extent.

[0051] It should be noted that, Figure 2 The example uses N=4, but this is not a limitation. Figure 2 The arrow symbol in the diagram indicates that the output at the end of the arrow is used as the input to the unit the arrow is pointing to.

[0052] It should be understood that, Figure 1 The structure shown is only a partial schematic diagram of the electronic device; the electronic device may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0053] The appearance defect detection method provided in this application embodiment can be applied to, but is not limited to, [various applications]. Figure 1 For the specific process of the electronic devices shown, please refer to [link / reference]. Figure 3 The methods for detecting appearance defects include S101, S102 and S103, which are described in detail below.

[0054] S101, the feature extraction branch extracts features from defective pixels in the target image according to a preset scale to obtain N-dimensional image features.

[0055] The target image is the captured image of the object to be detected.

[0056] Optionally, the feature extraction branch uses a pre-trained Swin Transformer as the backbone network to extract multi-scale features, and the extracted N stage features are denoted as {T}. i}, 1≤i≤N, T i T represents the image features of the i-th dimension. i The dimension is T i+1 T is twice the dimension of T.i The dimension size is Where H is the height of the target image, W is the width of the target image, and C is the number of channels of the target image.

[0057] Optionally, the object of inspection can be a road or workpiece product, such as hot-rolled strip steel, magnetic tiles, and rails, etc.

[0058] Through extensive practice and summarization, the inventors discovered that the network model achieves optimal detection efficiency and accuracy when N is 4.

[0059] S102, the feature fusion branch fuses image features in adjacent dimensions to obtain N-1 fused features.

[0060] Optionally, the feature fusion branch is the encoder in the network model. In the encoder stage, the proposed Semantic Alignment Attention (SCA) is used to fuse neighboring features from the backbone, adaptively learning different features from high and low layers to avoid semantic information misalignment in feature fusion. The specific process is as follows: Figure 2 As shown in the SCA, the cross-attention mechanism is used to fuse high and low resolution features from adjacent dimensions, where Q represents high-resolution features from low dimensions, K represents low-resolution features from high dimensions, and V is the sum of pooled high and low resolution features.

[0061] Then, matrix multiplication is used to calculate the similarity between high-level and low-level features. After normalization, a softmax activation function is used to generate a weight matrix to weight the features, and then addition is used to fuse them to obtain N-1 fused features.

[0062] S103, the decoding branch decodes based on N-1 fused features to obtain the decoded image result.

[0063] The decoded image results include a first type of feature pixels and a second type of feature pixels. The first type of feature pixels represents defective pixels, and the second type of feature pixels represents non-defective pixels.

[0064] It should be understood that appearance defects can be accurately located using the first type of feature pixels. Defects can be accurately detected even in low-contrast conditions, where the difference between the defect and the background is not significant. This eliminates interference from factors such as stains, shadows, and uneven lighting, and is adaptable to defects of various shapes and sizes, thus overcoming the problems of existing technologies.

[0065] In summary, the appearance defect detection method provided in this application includes: a feature extraction branch extracting features from defect pixels in a target image according to a preset scale to obtain N-dimensional image features; a feature fusion branch fusing adjacent-dimensional image features to obtain N-1 fused features; and a decoding branch decoding based on the N-1 fused features to obtain a decoded image result, wherein the decoded image result includes a first type of feature pixels and a second type of feature pixels, where the first type of feature pixels represents defect pixels and the second type of feature pixels represents non-defect pixels. This method can accurately detect defects even in low-contrast conditions, i.e., when the difference between the defect and the background is not significant. It can eliminate interference from stains, shadows, uneven lighting, and other factors affecting defect detection, and can adapt to defect detection of various shapes and sizes, thereby overcoming the problems existing in the prior art.

[0066] exist Figure 3 Based on this, for the content in S102, this application embodiment also provides a possible implementation method, please refer to... Figure 4 S102 includes: S102-1, S102-2 and S102-3, which are described in detail below.

[0067] S102-1, Feature fusion branch sums the features of image features in adjacent dimensions.

[0068] Alternatively, the formula for the sum of characteristics is:

[0069] V h +V l =Dc[Maxp(X h )+Dc(X l )];

[0070] Among them, V h +V l The features are represented by Dc, the depthwise separable convolution is represented by Maxp, and the pooling downsampling is represented by X. h X represents high-dimensional image features in adjacent dimensions. l Characterize low-dimensional image features in adjacent dimensions.

[0071] The specific process is as follows: Figure 2 As shown in the SCA, the cross-attention mechanism is used to fuse adjacent high and low resolution features, where Q comes from the high resolution features of the lower layer, K comes from the low resolution features of the higher layer, and V comes from the pooled high and low resolution features.

[0072] S102-2, the feature fusion branch determines the weight matrix based on the similarity of image features in adjacent dimensions.

[0073] S102-3, the feature fusion branch performs weighted fusion of features based on the weight matrix to obtain fused features.

[0074] Optionally, the formula for fusing features is:

[0075]

[0076] Q = Dc(X) h );

[0077] K = Dc(X) l );

[0078] Where F represents the fusion feature, softmax is the activation function, and d k The channel dimension that represents K.

[0079] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the SCA structure provided in an embodiment of this application. Figure 5 HR in this context is equivalent to X h Characterize high-dimensional image features in adjacent dimensions. Figure 5 In this context, LR is equivalent to X. l Characterize low-dimensional image features in adjacent dimensions. Through... Figure 5 The SCA shown can perform the above steps S102-1 to S102-3.

[0080] Optionally, the network model uses the CBAM module to spatially and channel-wise enhance the high-level semantic information (i.e., the image in the Nth dimension), further enriching the defect semantic information.

[0081] It should be understood that the multi-scale information (i.e., N-1 fused features) from the encoder first passes through a cross-scale enhancement module (CEF), which enhances each other by utilizing the similarity between features at adjacent scales. The two features are upsampled to another scale, and then the two features are multiplied at different resolutions. The defect foreground information is greatly enhanced. To reduce information loss, the original features are added back to the enhanced features. Finally, the two features are concatenated and further enhanced by channel attention.

[0082] exist Figure 3 Based on this, for the content in S103, this application embodiment also provides a possible implementation method, please refer to... Figure 6 S103 includes S103-1 and S103-2, which are described in detail below.

[0083] S103-1, the decoding branch obtains the decoding result corresponding to the N-1th fusion feature based on the N-1th fusion feature and the decoding reference feature.

[0084] Among them, the decoding reference feature is the reference feature after spatial and channel enhancement of the image features in the Nth dimension.

[0085] Optionally, the formula for decoding the reference feature is:

[0086] F′=M c (T N )⊙T N ;

[0087] F″=M s (F′)⊙F′;

[0088] Among them, T N The Nth dimension of the image features is represented by F″, which represents the decoding reference features, and M... C and M S represents calculating the channel and spatial attention matrices, respectively, and ⊙ represents element-wise multiplication.

[0089] S103-2, the decoding branch obtains the decoding result corresponding to the i-th fusion feature based on the decoding results corresponding to the i-th fusion feature and the (i+1)-th fusion feature.

[0090] Where 1≤i≤N-2, the decoding result corresponding to the first fusion feature is the decoded image result.

[0091] Optionally, the formula for the decoding result corresponding to the i-th fused feature is:

[0092]

[0093]

[0094] Among them, F h ′ and F l The concatenation result of F represents the decoding result. h F represents the i-th fusion feature. l Represents the decoding result corresponding to the (i+1)th fused feature, Up represents upsampling, Maxp represents pooling downsampling, and Conv (1×1) The following character represents a batch-normalized 1×1 convolution, δ represents the activation function, and ⊙ represents successive multiplication. It represents the addition of elements one by one.

[0095] Alternatively, channel attention can be used to target F. h ′ and F l The splicing result of ′ is further enhanced, and the enhanced result is used as the decoding result.

[0096] Optionally, F l After upsampling, and F hThe features are directly concatenated to obtain the decoded and fused features. To enhance feature representation and reduce information loss, channel attention is added after concatenation. Specifically, global average pooling is first used to integrate the global information of the feature map, and a weight vector is obtained through 1×1 convolution and softmax activation. The weight vector is then multiplied by the original features to obtain the weighted features. This process is described as follows:

[0097] F f =Cat(F′) h ,Up(F′ l ));

[0098] F′ f =ε(Conv 1×1 (Gap(F f )))×n×F f ;

[0099] Where Cat represents the join operation, Gap represents the global average pooling operation, and n represents F. f The number of channels, where ε represents the softmax activation function.

[0100] It should be understood that the decoding result corresponding to the (N-1)th fused feature also applies to the above formula. When the (N-1)th fused feature is used, F h Characterizing the (N-1)th fusion feature, F l It equals F″.

[0101] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of CEF provided in an embodiment of this application. Figure 7 In this context, HR is equivalent to F. h Characterize the i-th fusion feature, Figure 7 In this context, LR is equivalent to F. l This represents the decoding result corresponding to the (i+1)th fused feature. (Through...) Figure 7 The CEF shown can complete the above steps S103-1 to S103-2.

[0102] Optionally, in the scheme of this application, the decoding branch includes N-1 decoders, for example... Figure 2 In the CEF, the i-th decoder is used to obtain the decoding result corresponding to the i-th fused feature. Regarding the training of the network model, this application embodiment also provides a possible implementation method. It should be noted that the training process can be completed in an electronic device or in other terminals, and is not limited thereto.

[0103] Specifically, the network is trained using real defect samples and corresponding pixel-level annotations provided in the training set. The pixel-level image annotations are grayscale images. During training, the annotation images are normalized, normalizing the pixel value range to 0 to 1. That is, the closer the location in the grayscale image is to the defect, the closer the value is to 1, and the closer the location is to the background, the closer the value is to 0. The network model output is a single-channel grayscale image the size of the input image. The goal is to make the output grayscale image as consistent as possible with the annotation image, thereby accurately predicting the pixel location of the defect. Therefore, training requires calculating the loss (error) on both images, and then using the loss to backpropagate and correct the network parameters to reduce the loss. This process is repeated multiple times until the network performance is sufficiently good.

[0104] In this application, binary cross-entropy loss combined with IoU loss is used to calculate the loss between the predicted result and the actual segmentation position, thereby adjusting the network parameters according to the magnitude of the loss. Optionally, in the output stage of the SCFNet network, when N is 4, the three-level output of the decoder is upsampled by 16, 8, and 4 times to the input size and the labeled image, respectively, to calculate the loss, perform deep supervised training, and use binary cross-entropy loss (bce) and IoU loss for joint training to accelerate the training speed and improve the detection accuracy.

[0105] Optionally, the joint loss function of the network model is:

[0106]

[0107] Among them, L total Characterizing the joint loss function, Characterizing the binary cross-entropy loss of the i-th decoder, The Intersection over Union (IoU) loss of the i-th decoder is represented.

[0108] Binary cross-entropy loss is applied to binary classification tasks and is defined as follows:

[0109] I bce =-∑ (x,y) [G(x,y)log(S(x,y))+(1-G(x,y))log(1-S(x,y))]

[0110] The Intersection over Union (IoU) loss is used to evaluate the similarity between G and S, and is defined as:

[0111]

[0112] Where G is the pixel label (GT), S is the predicted segmentation map, i.e. the output of the network model, and (x,y) represents the pixel coordinates.

[0113] The SCFNet network structure provided in this application embodiment has been experimentally verified on four widely used magnetic tile defect datasets (MT), hot-rolled strip steel surface defect dataset (NEU-Seg), road defect dataset, and track surface discrete defect dataset (RSDD). The experiments show that the detection performance of the SCFNet network structure has reached the most advanced level at home and abroad.

[0114] The experimental verification results were evaluated using precision, recall, F-measure, and intersection-over-union (IoU), all commonly used in industrial product surface defect detection. Detailed definitions are as follows:

[0115]

[0116]

[0117]

[0118]

[0119] In this model, TP, FP, and FN represent the number of correctly detected defective pixels, incorrectly detected defective pixels, and undetected defective pixels, respectively. PR and GT are the output predicted value and the labeled ground truth value of the defective image, respectively. Since precision and recall are inversely proportional, the F-measure is used to better evaluate the performance of this method. In the experiments, these four metrics are the experimental averages on the test set. The experiments show that the actual performance of the SCFNet network structure has reached the state-of-the-art level both domestically and internationally.

[0120] For details, please refer to Figure 8 , Figure 9 , Figure 10 as well as Figure 11 , Figure 8 This is a graph showing the test results of the MT dataset provided in the embodiments of this application. Figure 9 This is a graph showing the test results of the NEU-Seg dataset provided in the embodiments of this application. Figure 10 This is a test result image of the road defect dataset provided in the embodiments of this application. Figure 11 The image shows the test results of the RSDD dataset provided in this application embodiment. For each dataset, from left to right, the first column is the original input image, the second column is the labeled real defect location, the third column is the network's predicted defect location, and each row from top to bottom represents different types of defects.

[0121] Please see Figure 12 , Figure 12 An appearance defect detection device is provided as an embodiment of this application. Optionally, the appearance defect detection device is applied to the electronic device described above.

[0122] The appearance defect detection device includes a feature extraction unit 201, a feature fusion unit 202, and a decoding unit 203, which are respectively equivalent to the feature extraction branch, feature fusion branch, and decoding branch in the network model mentioned above.

[0123] The feature extraction unit 201 is used to extract features from defect pixels in the target image according to a preset scale to obtain image features in N dimensions, wherein the target image is the acquired image of the detection object;

[0124] The feature fusion unit 202 is used to fuse image features in adjacent dimensions to obtain N-1 fused features;

[0125] The decoding unit 203 is used to decode based on N-1 fused features to obtain a decoded image result, wherein the decoded image result includes a first type of feature pixels and a second type of feature pixels, the first type of feature pixels representing defective pixels and the second type of feature pixels representing non-defective pixels.

[0126] It should be noted that the appearance defect detection device provided in this embodiment can execute the method flow shown in the above-described method flow embodiment to achieve the corresponding technical effects. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above-described embodiments.

[0127] This application also provides a storage medium storing computer instructions and programs, which, when read and run, execute the appearance defect detection method described in the above embodiments. The storage medium may include memory, flash memory, registers, or a combination thereof.

[0128] The following provides an electronic device, which can be a server device, a computer device, or other terminal devices with signal processing capabilities, such as a mobile phone. This electronic device... Figure 1 As shown, the above-described appearance defect detection method can be implemented. Specifically, the electronic device includes: a processor 10, a memory 11, and a bus 12. The processor 10 may be a CPU. The memory 11 is used to store one or more programs, which, when executed by the processor 10, perform the appearance defect detection method of the above embodiment.

[0129] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0130] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0131] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0133] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for detecting appearance defects, characterized in that, Applied to electronic devices, the electronic devices being deployed with a pre-trained network model, the network model including a feature extraction branch, a feature fusion branch, and a decoding branch, the method includes: The feature extraction branch extracts features from defective pixels in the target image according to a preset scale to obtain image features in N dimensions, wherein the target image is the acquired image of the detection object; The feature fusion branch fuses image features from adjacent dimensions to obtain N-1 fused features; The decoding branch decodes based on the N-1 fusion features to obtain a decoded image result, wherein the decoded image result includes a first type of feature pixels and a second type of feature pixels, the first type of feature pixels representing defective pixels and the second type of feature pixels representing non-defective pixels; The decoding branch performs decoding based on the N-1 fusion features to obtain a decoded image result, including: the decoding branch obtains the decoding result corresponding to the N-1th fusion feature based on the N-1th fusion feature and the decoding reference feature; wherein, the decoding reference feature is a reference feature after spatial and channel enhancement of the image feature of the Nth dimension; the decoding branch obtains the decoding result corresponding to the i-th fusion feature based on the decoding results corresponding to the i-th fusion feature and the i+1th fusion feature; wherein, 1≤i≤N-2, and the decoding result corresponding to the 1st fusion feature is the decoded image result; The formula for the decoding result corresponding to the i-th fused feature is: in, and The concatenation result represents the decoding result, F h F represents the i-th fusion feature. l Represents the decoding result corresponding to the (i+1)th fused feature, Up represents upsampling, Maxp represents pooling downsampling, and Conv (1×1) The following character represents a batch-normalized 1×1 convolution, δ represents the activation function, and ⊙ represents successive multiplication. It represents the addition of elements one by one.

2. The appearance defect detection method as described in claim 1, characterized in that, The feature fusion branch fuses image features in adjacent dimensions to obtain N-1 fused features, including: The feature fusion branch sums the features of image features in adjacent dimensions; The feature fusion branch determines the weight matrix based on the similarity of image features in adjacent dimensions; The feature fusion branch performs weighted fusion of the features based on the weight matrix to obtain the fused features.

3. The appearance defect detection method as described in claim 2, characterized in that, The formula for the sum of the features is: ； Among them, V h +V l The features are represented by Dc, which represents depthwise separable convolution, and Maxp represents pooling downsampling. h X represents high-dimensional image features in adjacent dimensions. l Characterize low-dimensional image features in adjacent dimensions.

4. The appearance defect detection method as described in claim 3, characterized in that, The formula for the fusion feature is: Where F represents the fusion feature, softmax is the activation function, and d k The channel dimension that represents K.

5. The appearance defect detection method as described in claim 1, characterized in that, The decoding branch includes N-1 decoders, where the i-th decoder is used to obtain the decoding result corresponding to the i-th fused feature. The joint loss function of the network model is: ； in, L total Characterizing the joint loss function, Characterizing the binary cross-entropy loss of the i-th decoder, The Intersection over Union (IoU) loss of the i-th decoder is represented.

6. A device for detecting appearance defects, characterized in that, Applied to electronic devices, the device includes: The feature extraction unit is used to extract features from defective pixels in the target image according to a preset scale to obtain image features in N dimensions, wherein the target image is the acquired image of the detection object; The feature fusion unit is used to fuse image features in adjacent dimensions to obtain N-1 fused features; The decoding unit is used to decode based on the N-1 fused features to obtain a decoded image result, wherein the decoded image result includes a first type of feature pixels and a second type of feature pixels, the first type of feature pixels representing defective pixels and the second type of feature pixels representing non-defective pixels; The decoding based on the N-1 fusion features to obtain the decoded image result includes: the decoding unit obtaining the decoding result corresponding to the N-1th fusion feature based on the N-1th fusion feature and the decoding reference feature; wherein, the decoding reference feature is a reference feature after spatial and channel enhancement of the image feature of the Nth dimension; the decoding unit obtaining the decoding result corresponding to the i-th fusion feature based on the decoding result corresponding to the i-th fusion feature and the i+1th fusion feature; wherein, 1≤i≤N-2, and the decoding result corresponding to the 1st fusion feature is the decoded image result; The formula for the decoding result corresponding to the i-th fused feature is: in, and The concatenation result represents the decoding result, F h F represents the i-th fusion feature. l Represents the decoding result corresponding to the (i+1)th fused feature, Up represents upsampling, Maxp represents pooling downsampling, and Conv (1×1) The following character represents a batch-normalized 1×1 convolution, δ represents the activation function, and ⊙ represents successive multiplication. It represents the addition of elements one by one.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, include: Processor and memory, the memory being used to store one or more programs; When the one or more programs are executed by the processor, the method as described in any one of claims 1-5 is implemented.

Images