A wood defect detection method based on an improved detr model

Abstract

The application discloses a wood defect detection method based on an improved DETR model, and belongs to the technical field of intelligently identifying wood.The wood defect detection is carried out by using the DETR model, picture features are interacted based on a nonparametric discrete Fourier transform, and a self-attention layer in the original DETR is replaced, so that the parameter quantity is greatly reduced; and the way of generating a target query vector in the original DETR is changed, the convergence speed is accelerated, and the calculation complexity is simplified.The wood defect detection method has good wood defect positioning and classification capabilities, greatly reduces production cost, realizes high automation of wood grade classification, and improves product quality.

Description

TECHNICAL FIELD

[0001] The application discloses a wood defect detection method based on an improved DETR model and belongs to the technical field of intelligent wood recognition. BACKGROUND

[0002] Automatic detection of wood defects is a prerequisite for high-quality wood processing. Wood defects are generally classified as knots, live knots, bark, cracks, notches, and dry scars. These defects seriously affect the yield of wood-based panels, the quality of finished products, and the aesthetic level of the appearance; they are also one of the main factors affecting paint quality, wood classification, and rational sawing. However, most of the production in many wood processing plants is still in a semi-mechanical or even primitive manual production state. The selection of panels requires experienced workers to classify them based on color, texture, defect number and size, and structural characteristics. This method not only consumes a lot of manpower and financial resources, but also increases costs.

[0003] Currently, although some wood detection methods based on deep learning have emerged, these methods are limited by pre-generated anchor boxes and non-maximum suppression, and have not achieved end-to-end detection. In addition, the quality of pre-generated anchor boxes and non-maximum suppression depends on prior information, increasing the difficulty of training.

[0004] Chinese patent document CN114529536A discloses a solid wood quality detection method. This document uses a combination of 2D planar image deep learning method and 3D point cloud image machine vision method to automatically detect solid wood panels. It can accurately and efficiently indicate the type, location, and extent of wood panel defects, identify wood panel color classification, and measure panel length, width, and thickness information.

[0005] Chinese patent document CN113066079A discloses a wood defect automatic detection method, device, system, and storage medium. The method includes: obtaining an original scan image of a wood panel; performing background difference processing on the original scan image to remove the background area from the original scan image and obtain a wood panel area image; adjusting the image size of the wood panel area image according to a predetermined size to obtain a detection image; after performing brightness adaptive enhancement on the detection image, inputting the detection image into a detection analysis deep learning model to obtain detected defect information; locating the position coordinates of each defect in the wood panel area image according to the defect information, and outputting the integrated defect information.

[0006] Chinese patent document CN112258495A discloses a building wood crack identification method based on a convolutional neural network, which includes: collecting building wood crack images and selecting pictures containing cracks; classifying and labeling building wood crack data sets, establishing large and small data sets, and dividing them into training sets, validation sets, and test sets in proportion; loading a convolutional neural network model, initializing the model, and training the model based on large and small data sets respectively; setting different hyperparameters, introducing validation set pictures for testing, and finding the optimal hyperparameter value; testing the model based on the obtained optimal model, outputting various detection numerical indicators, and determining whether the expected value is reached; and comparing and analyzing existing image processing methods.

[0007] Chinese patent document CN111951223A discloses a wood knot flaw image automatic identification method and system. After reconstruction based on the low-frequency image of the image and the high-frequency image of the updated image through a wood image enhancement module, the contrast of the image can be enhanced, the speckle artifacts in the reconstructed image can be removed, and the quality of the image is improved. Through a wood image defect identification module, a deep learning algorithm is trained according to different types of defect image blocks under different textures of wood, and the trained deep learning algorithm is used to detect and identify the defects of the wood. The method has high recognition accuracy, strong robustness, and fast speed.

[0008] Chinese patent document CN111862028A discloses a wood defect detection and sorting device and method based on a depth camera and deep learning. The RGB image and depth information of the wood surface are collected by a depth camera, and the RGBD color depth information is reconstructed by a GAN network combined with wavelet transform. In the wavelet transform, the wood cracks in the training data are manually labeled to form an adaptive crack wavelet basis function. On this basis, wavelet reconstruction is performed to improve the efficiency of subsequent algorithms, and a deep learning algorithm is used to analyze the defect type. The algorithm can sort wood of different defect types, improving the efficiency of defect discrimination and greatly improving the efficiency of sorting.

[0009] In summary, the existing patent documents have the following deficiencies:

[0010] 1. The detection of defect types is limited. The documents only detect three types of wood defects: gnawing head, insect hole, and cracking, which cannot meet the detection of multiple wood defects in the wood processing scene.

[0011] 2. The existing patent documents are sensitive to ambient noise, so noise problems caused by light brightness and lens position will affect the quality of the collected wood pictures.

[0012] 3. The model structure of the convolutional neural network in the existing patent documents is relatively simple, and the feature extraction capability for wood defects is relatively insufficient, making it difficult to identify some relatively complex wood defects.

[0013] 4. Existing patent literature utilizes cropped images of the wood surface for knot defect identification. However, in practice, the knot region occupies a small proportion of the wood surface image, and its features are easily obscured when extracting features from the entire image. Therefore, it cannot effectively extract knot features from the entire image. This fails to meet the requirements for detecting multiple wood defects in the entire image during wood processing.

[0014] 5. Existing patent literature uses GAN networks combined with wavelet transform to reconstruct RGBD color depth information. However, wavelet transform requires manual selection of wavelet basis, which makes wood defect detection more susceptible to human factors. It requires a lot of time to select a suitable wavelet basis, so it cannot achieve end-to-end detection and can only detect cracks in wood, which is difficult to meet the end-to-end detection of multiple wood defects in wood processing scenarios.

[0015] Therefore, establishing an end-to-end timber testing system has significant theoretical implications and practical application value. Summary of the Invention

[0016] To address the shortcomings of existing technologies, this invention discloses a wood defect detection method based on an improved DETR model. Invention Overview:

[0018] This invention employs the DETR model for wood defect detection. DETR, short for DeterminationTransformer, is a Transformer-based detector. DETR was the first successful end-to-end detector to utilize Transformer for object detection, eliminating the need for non-maximum suppression post-processing and anchor box generation. However, simply transplanting DETR to wood detection suffers from slow convergence due to its large parameter count and long training time.

[0019] This invention uses parameterless discrete Fourier transform to perform image feature interaction, replacing the self-attention layer in the original DETR, which greatly reduces the number of parameters; and it changes the way the original DETR generates the target query vector, which speeds up the convergence speed and simplifies the computational complexity.

[0020] Therefore, the defect detection method proposed in this invention has good ability to locate and classify wood defects, greatly reduces production costs, achieves a high degree of automation in wood grade classification, and improves product quality.

[0021] The detailed technical solution of this invention is as follows:

[0022] A method for detecting wood defects based on an improved DETR model, characterized by comprising:

[0023] (1) Obtain images containing wood defects including but not limited to knots, wane, bark, cracks, notches, and sapwood as a dataset;

[0024] (2) Data augmentation is performed on the images, and data augmentation is performed on the samples by means of image flipping, rotation, cropping, and noise addition to improve the quality of the samples and further improve the robustness and generalization ability of the model;

[0025] (3) The data is labeled to form the types and positions of wood defects, and the training set, the validation set, and the test set are divided in a ratio of 7:2:1;

[0026] (4) The improved DETR model is trained using a graphics card, in the embodiment, one graphics card, two graphics cards, four graphics cards, and eight graphics cards are used to train the improved DETR model and save the model parameters with the highest recognition accuracy, i.e., the optimal model parameters, when the model training is completed;

[0027] (5) The optimal model parameters are loaded into the improved DETR model, and then the wood defects are detected;

[0028] The improved DETR model comprises a picture feature extraction part and a Transformer structure part.

[0029] The picture feature extraction part comprises a convolutional neural network and position encoding, and the Transformer structure part comprises a Transformer encoder module, a Transformer decoder module, a feedforward neural network, a bipartite graph matching module, and a loss calculation module.

[0030] The original attention sublayer in the Transformer encoder module is replaced with a parameter-free discrete Fourier sublayer, and the originally randomly generated target query vector is improved into a target query vector with prior information. After being processed by an alpha layer encoder and an alpha layer decoder, the result of target detection is output using a feedforward neural network twice, where alpha is 6, and alpha' is determined according to actual scene experience and does not belong to the content protected by the present application; as shown in Figure 4 、 5 During training, bipartite matching assigns a unique prediction to the ground truth, and a mismatched prediction produces a "no class" prediction.

[0031] According to the present application, the backbone network of the improved DETR model is a pre-trained residual convolutional network resnet50, and the Figure 1Generally, the convolutional neural network is used for feature extraction and size reduction, the feature map output by the convolutional neural network contains 2048 channels, and the original image is reduced by 32 times in length and width, and each channel is used to extract different features of the image.

[0032] According to the application, the Transformer encoder module comprises 6 layers of encoders, each layer of encoder comprises a discrete Fourier transform process and a feedforward neural network process in sequence;

[0033] The discrete Fourier transform process comprises the following in parallel: a feature addition and normalization processing step directly performed on the image to which the position encoding is added after the convolutional neural network processing, and a feature addition and normalization processing step performed after the discrete Fourier transform;

[0034] The feedforward neural network process comprises the following in parallel: a feature addition and normalization processing step directly performed on the input of the feedforward neural network, and a feature addition and normalization processing step performed after the feedforward neural network on the input of the feedforward neural network, forming the input of the Transformer decoder.

[0035] The application can reduce the complexity and memory occupation of the model architecture by replacing the attention layer with a linear conversion, and significantly improve the speed of the encoder in DETR at a small cost of accuracy:

[0036] The 6-layer encoder extracts important information from a large amount of interaction information through the self-attention mechanism, but the self-attention mechanism has a large number of parameters, and it needs to map picture information of different regions to the same feature space when extracting features, and often the deep attention extraction feature is better than the shallow attention, which further increases the number of parameters, seriously limiting the running and training speed of the DETR model, therefore, it is particularly important to use a few-parameter or no-parameter feature interaction method to optimize the DETR model. In order to solve the above problems, the non-parametric discrete Fourier transform is used to replace the attention layer in the technical solution, so that the DETR model can achieve a good balance between speed, memory occupation and accuracy, and the encoder structure diagram after using the discrete Fourier transform to replace the attention sublayer is as follows Figure 4

[0037] According to the application, the method of adding position encoding to the image processed by the convolutional neural network comprises the following steps of:

[0038] ​a. Dimension compression: the C*H*W dimensional feature map output by the backbone network is first processed by a 1*1 convolution kernel to reduce the dimension, and the number of channels is compressed from C to d, that is, a new d*H*W dimensional feature map is obtained; wherein C is the number of convolution channels; H is the height of the feature map; W is the width of the feature map; d is the number of compressed channels;

[0039] b. Conversion to serialized data: since the encoder module requires input as a sequence, the spatial dimension (height and width) is compressed into one dimension, that is, the d*H*W dimensional new feature map obtained in the previous step is deformed into a d*HW dimensional feature vector, that is, a d*E dimensional feature vector, wherein E=HW;

[0040] c. Adding position encoding: since the Transformer encoder model is order-independent, and the E dimension of the d*E dimensional feature vector is related to the position of the original image, the d*HW dimensional feature vector is added with position encoding to reflect the position information.

[0041] According to the present application, the target query vector generation method with prior information is as follows:

[0042] The input of the Transformer encoder module is X e , wherein , wherein is a real set; it is mapped into a latent target query vector with prior information through a linear layer formula, and the linear layer formula is as follows:

[0043] X o =tanh(X e W m ) (I)

[0044] In formula (I), tanh is an activation function; is a weight matrix for extracting prior information; q represents the dimension of a real set, which is a latent target query vector; m is an index without actual meaning, and is used to distinguish different weight matrices;

[0045] Calculate the score of each latent target query vector Select the top N latent target query vectors with high scores as the final N target query vectors, and the score function is as follows:

[0046] x s =sigmoid(X o W s ) (II)

[0047] In formula (II), sigmoid is an activation function; q and 1 represent the real set dimension, the weight matrix of the scoring function respectively.

[0048] The target query vector in the DETR is randomly generated, which increases the learning cost and learning time of the target query vector, and causes the model to converge slowly, in order to solve the above problems, the technical scheme improves the generation mode of the target query vector, and generates the target query vector by introducing prior information, the generation mode based on the prior information contains the feature information of the target, and the convergence speed of the model is accelerated.

[0049] According to the application, the Transformer decoder module comprises 6 decoder layers, each Transformer decoder layer comprises a first discrete Fourier transform process, a second discrete Fourier transform process and a feedforward neural network process in sequence.

[0050] The first discrete Fourier transform process comprises in parallel: a step of directly adding and normalizing features of N target query vectors, and a step of first performing discrete Fourier transform and then adding and normalizing features, to form a first decoder output.

[0051] The second discrete Fourier transform process comprises in parallel: a step of directly adding and normalizing features of the first decoder output, and a step of first performing second discrete Fourier transform and then adding and normalizing features, to form a second decoder output.

[0052] The feedforward neural network process comprises in parallel: a step of directly adding and normalizing features of the second decoder output, and a step of first performing feedforward neural network and then adding and normalizing features, to form an output of the Transformer decoder module.

[0053] The Transformer decoder module is similar to the Transformer encoder module, and the self-attention sublayer in the Transformer decoder module is replaced by a discrete Fourier sublayer, and the replaced decoder structure diagram using discrete Fourier is as follows: Figure 5As shown: the N target query vectors inputted by the decoder are closely related to the performance of the algorithm. After each iteration of the decoder, the N target query vectors will be updated. The final result is predicted by the feedforward neural network according to the target query vector. In DETR, the N target query vectors are randomly initialized at the beginning of training. This randomly initialized target query vector does not provide a good initial state for the decoder, which reduces the convergence speed of the model. Therefore, the target query vector initialization method based on prior information can improve the performance of the algorithm. The technical solution generates N target query vectors with prior information, inputs them into the decoder, and then decodes them into box coordinates and class labels through the feedforward neural network, thereby generating N final predictions. A 1-dimensional discrete Fourier transform along the sequence direction and a 1-dimensional discrete Fourier transform along the hidden feature direction are used on these picture features, the model uses the entire image as context information when predicting, and all targets are implemented globally through parallel box output, which greatly improves the timeliness of detection and the convergence speed of the model.

[0054] The feedforward neural network in the Transformer encoder module and the Transformer decoder module is a forward propagation linear layer. The feedforward neural network predicts the standardized center coordinates and the boundary box height and width, and the linear layer uses the activation function to predict the class label. Since the improved DETR module predicts N boundary boxes, where N is usually much larger than the actual number of objects of interest in the image, an additional special class label is used to represent that no object is detected in the slot. This class has a similar function to the background class in standard object detection methods.

[0055] According to the application, the discrete Fourier sublayer includes a parameter-free discrete Fourier transform:

[0056]

[0057] In formula (III), given a picture feature sequence x n , where n∈[0, N-1]; for the kth sequence, i represents the imaginary part; X k represents x n After the discrete Fourier transform, the feature representation is

[0058] The improved DETR is a non-attention Transformer architecture, where each layer is composed of a discrete Fourier hybrid sublayer and a feedforward sublayer.

[0059] Replace the self-attention sublayer in each Transformer encoder module and Transformer decoder module with a discrete Fourier sublayer, process the picture feature matrix by using a 2-dimensional discrete Fourier transform to decompose into a 1-dimensional discrete Fourier transform Fseq that extracts the interaction information between features along the sequence direction and a 1-dimensional discrete Fourier transform Fh that extracts the intra-feature information along the hidden feature direction:

[0060]

[0061] In formula (IV), y is the picture feature output extracted after the discrete Fourier sublayer change, represents the discrete Fourier sublayer, and only the real part of the result is finally retained, so there is no need to modify the nonlinear feedforward sublayer or the output layer to process complex numbers.

[0062] According to the application, the bipartite graph matching module is used to match the predicted target bounding box and the ground truth after the feedforward neural network:

[0063] The improved DETR model outputs a fixed number of prediction boxes, each of which has a predicted target position and a category of the target in the box. For cases where the number of prediction values N is not enough, the is filled in, i.e. no category prediction, the ground truth of the label is consistent with the small number of output values, and the is filled in.

[0064] According to the application, in the bipartite graph matching module, in order to solve some problems related to bipartite graph matching, the Hungarian algorithm is used to calculate the optimal arrangement: the predicted target bounding box and the ground truth are matched.

[0065] According to the application, the loss calculation module is used to calculate the loss function of the predicted target bounding box and the ground truth

[0066]

[0067] In formula (V), refers to finding the matching arrangement with the minimum loss when the bipartite graph is matched using the Hungarian algorithm; each label y i =(ci, bi) in the ground truth set, ci is the category label, and bi is a 4-dimensional normalized vector containing the horizontal coordinate of the target position, the vertical coordinate of the target position, the height of the target, and the width of the target; is the label of the optimal sequence.

[0068] The improved DETR model infers a fixed-size N prediction set during a one-pass decoder process, where N is set to be significantly larger than the typical number of objects in an image. One of the main difficulties in training is to score the predicted objects, i.e., class, location, and size, according to the actual situation; the loss of DETR produces the best bipartite graph matching between the predicted objects and the real objects, and optimizes the object boundary box-specific loss; the bipartite graph matching loss is the loss of two-by-two matching between the real value and the predicted value.

[0069] The boundary box loss directly uses the L1 loss, which is unfair to small targets, so the combination of L1 loss and GIOU general intersection over union loss is used to make the loss insensitive to the size of the target, and the main idea is to measure the coincidence degree of two enclosing boxes.

[0070] According to the application, the loss calculation module is further used to calculate the boundary box loss

[0071]

[0072] In formula (VI), λ iou , λ L1 is a hyperparameter used to normalize the L1 loss and the GIOU loss; b i is a 4-dimensional normalized vector of the real label; is the predicted matching permutation position label; is the GIOU loss.

[0073] According to the application, the loss calculation module is further used to calculate the final loss function by combining the target class loss and the position loss

[0074]

[0075] In formula (VII), y is the real target label manually labeled; is the predicted target label; represents the predicted target class probability when the bipartite graph loss is minimized; is the predicted permutation position label when the bipartite graph loss is minimized.

[0076] The technical advantages of the application are:

[0077] 1. The application is based on the improved DETR for wood defect detection, and the DETR does not generate anchor frames and does not need non-maximum suppression for post-processing operation, providing a new end-to-end wood defect detection method.

[0078] 2、The application proposes to use a standard non-parametric discrete Fourier transform to replace the attention sublayer in DETR, which greatly improves the detection speed of the wood detection model. In addition to the speed gain, a balance between accuracy and memory usage is ensured. The improved DETR model is easier to deploy, meeting the real-time and fast requirements of wood detection.

[0079] 3、The application proposes to generate a target query in combination with prior information, so that the target query vector naturally has target feature information, has better learning ability, speeds up the convergence of the model, and further improves the detection speed of the wood detection model. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 Fig. 1 is a flowchart of wood defect detection based on an improved DETR model.

[0081] Figure 2 Fig. 2 is a schematic diagram of wood defect types.

[0082] Figure 3 Fig. 3 is a flowchart of the improved DETR detection model in the application.

[0083] Figure 4 Fig. 4 is a structural schematic diagram of the improved Transformer encoder module in the application.

[0084] Figure 5 Fig. 5 is a structural schematic diagram of the improved Transformer decoder module in the application.

[0085] Figure 6 Fig. 6 is a detection effect diagram of the embodiment.

[0086] Figure 7 Fig. 7 is a detection effect diagram of the embodiment.

[0087] Label 1 represents knuckles, label 2 represents bark, label 3 represents cracks, label 4 represents notches, and label 5 represents dry scars. DETAILED DESCRIPTION

[0088] The application will be described in detail below in conjunction with the drawings of the specification, but is not limited thereto.

[0089] Embodiment 1,

[0090] A wood defect detection method based on an improved DETR model, comprising:

[0091] (1) Obtain images containing wood defects as a data set, wherein the wood defects include but are not limited to knuckles, knuckles, bark, cracks, notches, and dry scars;

[0092] (2) Data augmentation is performed on the image, and data augmentation is performed on the sample by flipping, rotating, cropping and adding noise to the image, so as to improve the quality of the sample and further improve the robustness and generalization ability of the model;

[0093] (3) The labeled data is used to form the types and positions of wood defects, and the training set, the validation set and the test set are divided in the proportion of 7:2:1;

[0094] (4) The improved DETR model is trained using a graphics card, in the embodiment, one graphics card, two graphics cards, four graphics cards and eight graphics cards are used to train the improved DETR model and save the model parameters with the highest recognition accuracy, that is, the optimal model parameters, when the model is trained;

[0095] (5) The optimal model parameters are loaded into the improved DETR model, and then the wood defects are detected;

[0096] The improved DETR model comprises a picture feature extraction part and a Transformer structure part.

[0097] The picture feature extraction part comprises a convolutional neural network and position encoding, and the Transformer structure part comprises a Transformer encoder module, a Transformer decoder module, a feedforward neural network, a bipartite graph matching module and a loss calculation module.

[0098] The original attention sublayer in the Transformer encoder module is replaced by a parameter-free discrete Fourier sublayer, and the original randomly generated target query vector is improved into a target query vector with prior information. After processing by an alpha layer encoder and an alpha layer decoder, the result of target detection is output by using a feedforward neural network twice, wherein alpha is 6, and alpha' is determined according to actual scene experience and does not belong to the content protected by the application. Figure 4 、 5 During training, bipartite matching assigns a unique prediction to the ground truth, and a mismatched prediction produces a "no class" prediction.

[0099] The backbone network of the improved DETR model is a pre-trained residual convolutional network resnet50, and the Figure 1 target detection result is generally large, so a convolutional neural network is generally used for feature extraction and size reduction. The feature map output by the convolutional neural network contains 2048 channels, and the original image is reduced by 32 times in length and width. Each channel is used to extract different features of the image.

[0100] The Transformer encoder module comprises 6 layers of encoders, each layer of encoder comprising a discrete Fourier transform procedure and a feedforward neural network procedure in sequence;

[0101] The discrete Fourier transform procedure comprises, in parallel, a feature addition and normalization processing step directly performed on the image with position encoding processed by the convolutional neural network, and a feature addition and normalization processing step performed after the image is processed by the discrete Fourier transform, forming an input of the feedforward neural network;

[0102] The feedforward neural network procedure comprises, in parallel, a feature addition and normalization processing step directly performed on the input of the feedforward neural network, and a feature addition and normalization processing step performed after the input of the feedforward neural network is processed by the feedforward neural network, forming an input of the Transformer decoder.

[0103] The method for processing the image with position encoding processed by the convolutional neural network comprises the following steps:

[0104] a. Dimension compression: the C*H*W-dimensional feature map output by the backbone network is first processed by a 1*1 convolution kernel to reduce the dimension, compressing the number of channels from C to d, thereby obtaining a new d*H*W-dimensional feature map; wherein C is the number of convolution channels; H is the height of the feature map; W is the width of the feature map; and d is the number of compressed channels;

[0105] b. Conversion into serialized data: since the encoder module requires an input sequence, the spatial dimensions (height and width) are compressed into one dimension, i.e., the d*H*W-dimensional new feature map obtained in the previous step is deformed into a d*HW-dimensional feature vector, i.e., a d*E-dimensional feature vector, wherein E=HW;

[0106] c. Adding position encoding: since the Transformer encoder model is order-independent, the E-dimensional feature vector in the d*HW-dimensional feature vector is related to the position of the original image, and therefore the d*HW-dimensional feature vector is added with position encoding to reflect the position information.

[0107] The method for generating the target query vector with prior information is as follows:

[0108] The input of the Transformer encoder module is X e wherein wherein, is a real number set; it is mapped into a latent target query vector with prior information through a linear layer formula, and the linear layer formula is as follows:

[0109] X o=tanh(X) e W m (I)

[0110] In formula (I), tanh is the activation function; It is the weight matrix for extracting prior information; q represents the dimension of the real number set, which is the potential target query vector; m is a subscript with no practical meaning, used to distinguish different weight matrices;

[0111] Calculate the score for each potential target query vector. The top N high-scoring potential target query vectors are selected as the final N target query vectors, and the scoring function is as follows:

[0112] x s =sigmoid(X o W s (II)

[0113] In formula (II), sigmoid is the activation function; q and 1 represent the dimension of the real number set and the weight matrix of the scoring function, respectively.

[0114] The Transformer decoder module includes 6 layers of decoders, each of which includes a first discrete Fourier transform process, a second discrete Fourier transform process, and a feedforward neural network process in sequence.

[0115] The first discrete Fourier transform process includes the following parallel steps: directly adding and normalizing the features of N target query vectors, and then performing a discrete Fourier transform followed by feature addition and normalization to form the output of the first decoder.

[0116] The second discrete Fourier transform process includes the following parallel steps: first, the output of the first decoder is directly subjected to feature addition and normalization; second, the output of the second decoder is subjected to the second discrete Fourier transform and then feature addition and normalization.

[0117] The feedforward neural network process includes the following parallel steps: the second decoder output is directly subjected to feature addition and normalization processing, and the second decoder output is first processed through the feedforward neural network and then subjected to feature addition and normalization processing to form the output of the Transformer decoder module.

[0118] Example 2

[0119] As described in Example 1, the discrete Fourier leaf layer includes a parameterless discrete Fourier transform:

[0120]

[0121] In formula (III), given a picture feature sequence x n , where n∈[0, N-1]; for the kth sequence, i represents the imaginary part; X k represents the real part of x n After the discrete Fourier transform, the feature representation is obtained.

[0122] Replace the self-attention sublayer in each Transformer encoder module and Transformer decoder module with a discrete Fourier sublayer, process the picture feature matrix by using a 2-dimensional discrete Fourier transform to decompose into a 1-dimensional discrete Fourier transform Fseq that extracts interaction information between features along the sequence direction and a 1-dimensional discrete Fourier transform Fh that extracts intra-feature information along the hidden feature direction:

[0123]

[0124] In formula (IV), y is the picture feature output extracted after the discrete Fourier sublayer, represents the discrete Fourier sublayer, and only the real part of the result is finally retained, so there is no need to modify the nonlinear feedforward sublayer or the output layer to process complex numbers.

[0125] The bipartite graph matching module is used to match the predicted target bounding box and the ground truth after the feedforward neural network:

[0126] The improved DETR model outputs a fixed number of prediction boxes, each of which has a predicted target position and the category of the target in the box. For cases where the number of prediction values N is not enough, the padding, i.e., no category prediction, the ground truth of the label is consistent with the number of output values, and the padding is used.

[0127] In the bipartite graph matching module, in order to solve some problems related to bipartite graph matching, the Hungarian algorithm is used to calculate the optimal arrangement: the predicted target bounding box and the ground truth are matched.

[0128] The loss calculation module is used to calculate the loss function of the predicted target bounding box and the ground truth

[0129]

[0130] In formula (V), refers to finding the matching arrangement with the smallest loss when implementing the matching of the bipartite graph using the Hungarian algorithm; each label y i= (ci, bi), ci is a class label, bi is a 4-dimensional normalized vector containing the horizontal coordinate of the target position, the vertical coordinate of the target position, the height of the target, and the width of the target; is the label of the optimal sequence.

[0131] The improved DETR model infers a fixed-size N of predictions in one pass through the decoder, where N is set to be significantly larger than the typical number of objects in an image. One of the main difficulties in training is to score the predicted objects, i.e., class, position, and size, according to the actual situation; the loss of DETR produces the best bipartite matching between predicted objects and real objects, optimizing the loss specific to the object bounding box; the bipartite graph matching loss is the loss of pairwise matching between the true value and the predicted value.

[0132] The bounding box loss directly uses the L1 loss, which is unfair to small targets, so a combination of L1 loss and GIOU general intersection over union loss is used to make the loss insensitive to the size of the target, the main idea is to measure the degree of coincidence of two enclosing boxes.

[0133] The loss calculation module is further configured to calculate a bounding box loss

[0134]

[0135] In formula (VI), lambda iou , lambda L1 is a hyperparameter used to normalize the L1 loss and the GIOU loss; b i is a 4-dimensional normalized vector of the real label; is the predicted matching permutation position label; is the GIOU loss.

[0136] Embodiment 3,

[0137] The loss calculation module is further configured to calculate a final loss function by combining the target class loss and the target position loss as described in embodiment 2

[0138]

[0139] In formula (VII), y is the real target label manually labeled; is the predicted target label; represents the predicted target class probability when the bipartite graph loss is minimized; is the predicted permutation position label when the bipartite graph loss is minimized.

[0140] In the practice of Embodiment 1, Embodiment 2 and Embodiment 3, when obtaining the data set containing wood defects on site, a large number of defective wood boards are collected from the wood factory in multiple times, and the wood boards collected on site: in the actual processing of wood, a single picture often contains multiple different kinds of defects, such as knots, live knots, bark, cracks, notches and dry scars.

[0141] During the labeling of the above data set, it is difficult to distinguish between some knots and live knots, and it is difficult to distinguish the light-colored bark texture on the wood board, which leads to slow progress in the early labeling. In order to solve the above technical problems, the data set is screened by experienced workers, and a unified labeling standard is developed as shown in Figure 2

[0142] Knot: the label when labeling the knot is “0”, the knot on the wood board is mostly black, which is annular or semi-annular separated from the surrounding structure, and is easy to shrink and leak after drying;

[0143] Live knot: the label when labeling the live knot is “1”, the live knot on the wood board is usually brown, which is connected with the surrounding structure with complete texture, and does not form a broken and pith structure;

[0144] No labeling: the knot and live knot smaller than 300 pixels in the wood board are not labeled;

[0145] Bark: the label when labeling the bark is “2”, the bark on the wood board is usually mixed color, mainly gray-brown and dark brown, the place where the bark is located is thin, usually rectangular, and the paper or small piece shape is peeled off;

[0146] Crack: the label when labeling the crack is “3”, the crack on the wood board is a narrow crack, which usually appears on the edge of the wood board;

[0147] Notch: the label when labeling the notch is “4”, the notch on the wood board is usually caused by the bark, which causes the wood board to be incomplete;

[0148] Dry scar: the label when labeling the dry scar is “5”, the dry scar on the wood board is the knot scar produced by the growth of the wood branch, which has no dead knot or decay, and the dry scar is connected by multiple live knots or dead knots.

[0149] Experimental results

[0150] In the experiment, the number of GPUs is set to 1, 2, 4 and 8 respectively, the number of training rounds is 300, 8 pictures are loaded on each GPU, and the experimental results are shown in Table 1 as follows:

[0151] Table 1 DETR and experimental results based on improved DETR

[0152] ​

[0153] AP in Table 1 represents average precision, AP50 represents the average precision of the IOU between the predicted bounding box and the real labeled box, i.e. the average precision when the intersection over union threshold is greater than 0.5, AP75 represents the average precision when the IOU threshold between the predicted bounding box and the real labeled box is greater than 0.75, APs represents the average precision when detecting small targets, and APm represents the average precision when detecting medium targets. It is found from Table 1 that when the number of GPUs increases from 1 to 4, the performance of the two models also improves, because with the increase of the number of graphics cards, the generalization performance of the parameters after each adjustment is enhanced; when the number of GPUs is 8, the performance decreases, because when training multiple cards, data interaction between multiple cards needs to be considered, which causes a slight decrease in accuracy. Therefore, the number of cards and the amount of data for each batch of training should be considered during training. Finally, when the number of graphics cards is 4, the experimental results of the two models are the best. The improved DETR replaces the self-attention sublayer with a parameter-free discrete Fourier sublayer, which greatly reduces the parameter amount. As can be seen from Table 1, the parameter amount is reduced to about 70% of the original; on the other hand, the introduction of prior information enables the target query vector to have the feature information of the target in the initialization stage, which speeds up the convergence of the model, so the training time is also reduced to 67% of the original, significantly improving the running speed. As can be seen from the table, the replaced discrete Fourier sublayer has strong ability to extract image features. Under the same experimental conditions, the improved DETR based on the original model can achieve 97% of the accuracy of the original model with 70% of the parameter amount of the original model. Overall, the improved DETR has relatively fast running speed and high recognition effect, and takes into account the speed and accuracy. The experimental results prove the effectiveness of the improvement.

[0154] The test effect diagram of the final model based on the improved DETR is shown in Figure 6 、 Figure 7 . Figure 6 、 Figure 7 In which each defect is marked, and the mark 0 represents dead knot, the mark 1 represents live knot, the mark 2 represents bark, the mark 3 represents crack, the mark 4 represents gap, and the mark 5 represents dry knot.

Claims

1. A method for detecting wood defects based on an improved DETR model, characterized in that, include: (1) Obtain images containing wood defects at the site as a dataset; (2) Perform data augmentation on the image; (3) Label the data to form the types and locations of wood defects, and divide the data into training set, validation set and test set; (4) Training with the improved DETR model: The improved DETR model is trained using a graphics card, and the model parameters with the highest recognition accuracy are saved, i.e., the optimal model parameters; (5) Load the optimal model parameters into the improved DETR model, and then perform defect detection on the wood; The improved DETR model includes an image feature extraction part and a Transformer structure part; The image feature extraction part includes a convolutional neural network and position encoding; the Transformer structure part includes a Transformer encoder module, a Transformer decoder module, a feedforward neural network, a bipartite graph matching module, and a loss calculation module; The original attention sublayer in the Transformer encoder module is replaced with a parameterless discrete Fourier leaf layer, and the original randomly generated target query vector is improved to a target query vector with prior information. Layer encoder and After processing by the layer decoder, the target detection result is finally output using two feedforward neural networks. Number of floors; The method for generating the target query vector with prior information is as follows: The output of the Transformer encoder module is ,in ,in, It is the set of real numbers; it is mapped into a potential target query vector with prior information through a linear layer formula, which is as follows: （I） In formula (I), tanh is the activation function; It is a weight matrix for extracting prior information; represents the dimension of the real number set, which is the potential target query vector; m is used to distinguish different weight matrices; Calculate the score for each potential target query vector. The top N high-scoring potential target query vectors are selected as the final N target query vectors. The scoring function is as follows: (II) In formula (II), sigmoid is the activation function; The weight matrix represents the score function. Both 1 and 1 represent the dimension of the real number set.

2. The wood defect detection method based on the improved DETR model according to claim 1, characterized in that, The backbone network of the improved DETR model is a pre-trained residual convolutional network, resnet50.

3. The wood defect detection method based on the improved DETR model according to claim 2, characterized in that, The Transformer encoder module includes 6 encoder layers, each of which includes a discrete Fourier transform process and a feedforward neural network process in sequence. The discrete Fourier transform process includes the following steps in parallel: the image after being processed by the convolutional neural network and then having positional encoding added is directly subjected to feature addition and normalization, and the image is first subjected to discrete Fourier transform and then to feature addition and normalization, forming the input of the feedforward neural network. The feedforward neural network process includes the following parallel steps: the input of the feedforward neural network is directly subjected to feature addition and normalization processing, and the input of the feedforward neural network is first processed by the feedforward neural network and then subjected to feature addition and normalization processing to form the input of the Transformer decoder.

4. The wood defect detection method based on the improved DETR model according to claim 3, characterized in that, The method for adding positional encoding to an image after processing by a convolutional neural network involves converting the feature map output by the backbone network into serialized data that can be processed by the Transformer encoder module. Specifically, this includes: a. Dimensionality Compression: The C * H * W dimensional feature map output by the backbone network is first reduced in dimension using a 1 * 1 convolutional kernel, compressing the number of channels from C to d, resulting in a new d * H * W dimensional feature map; where C is the number of convolutional channels; H is the height of the feature map; W is the width of the feature map; and d is the number of channels after compression. b. Convert to serialized data: The resulting d * H * W dimensional feature map is transformed into a d * HW dimensional feature vector, i.e. a d * E dimensional feature vector, where E = HW; c. Add positional encoding: Add positional encoding to the d * HW dimensional feature vector to reflect positional information.

5. The wood defect detection method based on the improved DETR model according to claim 1, characterized in that, The Transformer decoder module includes 6 layers of decoders, each of which includes a first discrete Fourier transform process, a second discrete Fourier transform process, and a feedforward neural network process in sequence. The first discrete Fourier transform process includes the following parallel steps: directly adding and normalizing the features of N target query vectors, and then performing a discrete Fourier transform followed by feature addition and normalization to form the output of the first decoder. The second discrete Fourier transform process includes the following parallel steps: first, the output of the first decoder is directly subjected to feature addition and normalization; second, the output of the second decoder is subjected to the second discrete Fourier transform and then feature addition and normalization. The feedforward neural network process includes the following parallel steps: the second decoder output is directly subjected to feature addition and normalization processing, and the second decoder output is first processed by the feedforward neural network and then subjected to feature addition and normalization processing to form the output of the Transformer decoder module.

6. The wood defect detection method based on the improved DETR model according to claim 1, characterized in that, The discrete Fourier leaf layer includes a parameter-free discrete Fourier transform: (III) In formula (III), given an image feature sequence x n， Where n∈[0,N-1]; for the th k There are sequences, where i represents the imaginary part; X k Represents x n Feature representation after Discrete Fourier Transform; The self-attention sublayers in each Transformer encoder and decoder module are replaced with discrete Fourier transform layers. The image feature matrix is ​​processed using a 2D discrete Fourier transform to decompose it into a 1D discrete Fourier transform Fseq that extracts interaction information between features along the sequence direction and a 1D discrete Fourier transform Fh that extracts intra-feature information along the hidden feature direction. (IV) In formula (IV), y is the image feature output after being extracted by the Discrete Fourier Leaf Layer Transform. This represents a discrete Fourier leaf layer.

7. The wood defect detection method based on the improved DETR model according to claim 1, characterized in that, The bipartite graph matching module is used to match the predicted target bounding box with the ground truth value after the feedforward neural network. The improved DETR model outputs a fixed number of prediction boxes, each containing the predicted target location and the category of the target within the box. In the bipartite graph matching module, the Hungarian algorithm is used to calculate the optimal permutation, which matches the predicted target bounding box with the ground truth.

8. The wood defect detection method based on the improved DETR model according to claim 1, characterized in that, The loss calculation module is used to calculate the loss function between the predicted target bounding box and the ground truth value. : （V） In formula (V), This refers to using the Hungarian algorithm to find the matching permutation with the minimum loss when performing matching on a bipartite graph; each label in the ground truth set y i =(ci,bi), where ci is the category label and bi is a 4-dimensional normalized vector containing the x-coordinate of the target position, the y-coordinate of the target position, the height of the target, and the width of the target. It is the label of the optimal sequence.

9. The wood defect detection method based on the improved DETR model according to claim 8, characterized in that, The loss calculation module is also used to calculate the bounding box loss. : (WE) In formula (VI), , Hyperparameters are used to normalize both L1 loss and GIOU loss. It is a 4-dimensional normalized vector of the real label; These are the predicted matching arrangement position labels; It is a GIOU loss; The loss calculation module is also used to calculate the final loss function. : (VII) In formula (VII), These are real target labels manually labeled; It predicts the target label; This represents the predicted target class probability when the bipartite graph loss is minimized. It is the predicted permutation position label when the bipartite graph loss is minimized.

Images