Wood surface defect detection method based on multi-modal data

Abstract

The invention discloses a multi-modal data-based wood surface defect detection method, which belongs to the technical field of image recognition, and specifically comprises the following steps of: acquiring a wood surface image, marking the wood surface image as a primary screening image and preprocessing the primary screening image, converting the primary screening image into a gray level image, calculating a gray level mean value and a variance of the gray level image, and setting a sliding window and an abnormal threshold value; calculating the mean value and variance of the gray values of the areas in the sliding window, judging whether the gray value distribution of the areas in the sliding window is abnormal or not according to an abnormal threshold value, marking the abnormal areas as abnormal areas, marking the normal areas as to-be-checked areas, screening out areas with abnormal confusion circles from the to-be-checked areas, marking the areas as the abnormal areas, and determining whether the confusion circles are abnormal or not. The image of the abnormal area is cut off from the fine check image and marked as the ROI image, the ROI image is input into the recognition model to accurately judge the defect type, the judgment result is visually output, and the speed of image detection and recognition of large-batch wood can be increased.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a wood surface defect detection method based on multimodal data. Background Art

[0002] Wood is an important raw material that plays an important role in industrial production and daily life. The economic value of wood is closely related to its quality. Therefore, timely detection of defects on the wood surface is of great significance for subsequent production and processing.

[0003] At present, image recognition technology is often used when detecting surface defects of wood. By collecting image data of wood, image recognition models are used to replace manual surface defect detection of wood, and better recognition effects can be achieved.

[0004] However, when detecting wood surface defects, traditional methods require overall recognition of the image of each piece of wood. This recognition method limits the recognition speed. The production process of the wood industry often requires rapid recognition and detection of large quantities of wood. Therefore, the detection method that only relies on a single image recognition model cannot meet actual production requirements. Summary of the invention

[0005] The purpose of the present invention is to provide a wood surface defect detection method based on multimodal data to solve the following technical problems:

[0006] When detecting wood surface defects, traditional image recognition detection methods need to perform overall recognition of each wood surface image, which limits the recognition speed. However, in the production process of the wood industry, it is often necessary to quickly identify a large number of wood surface defects to ensure production efficiency. Therefore, the traditional method that relies on a single image recognition model cannot meet production needs.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A wood surface defect detection method based on multimodal data comprises the following steps:

[0009] The image sensor is placed directly above the wood, and the parameters are adjusted so that the background deep point plane of the image sensor is aligned with the wood surface, and the wood surface image is collected and marked as the primary screening image;

[0010] Preprocess the initial screening image, convert the preprocessed initial screening image into a grayscale image, and calculate the mean V1 and variance Z1 of the grayscale value of the grayscale image;

[0011] Use a sliding window to detect the grayscale image, calculate the mean V2 and variance Z2 of the grayscale value in each sliding window, and compare V2 and Z2 with the mean V1 and variance Z1 respectively. Use the abnormal threshold to determine whether the grayscale value distribution in the window area is abnormal. Mark the window area with abnormal grayscale value distribution as an abnormal area, and mark the area with normal grayscale value distribution as a pending area.

[0012] Perform confusion circle detection on the area to be determined to determine whether there is an abnormal confusion circle in the area to be determined. When an abnormal confusion circle exists in the area to be determined, the minimum circumscribed rectangular area of ​​the abnormal confusion circle is marked as an abnormal area.

[0013] Increase the depth of field, take the wood surface image again and mark it as a detailed inspection image, crop the image area corresponding to the abnormal area in the detailed inspection image and mark it as a ROI image, input the ROI image into the preset wood surface defect recognition model, and visualize the output recognition result on the detailed inspection image.

[0014] As a further solution of the present invention: when collecting the preliminary screening image and the detailed inspection image on the same wood surface, a neutral light source is used to provide uniform illumination for the wood surface, and the relative position of the image sensor and the wood surface and the focus of the image sensor remain unchanged during the two collection processes.

[0015] As a further solution of the present invention: the preprocessing includes image denoising and image enhancement, and the image denoising method includes mean filtering, median filtering or Gaussian filtering;

[0016] The image enhancement method is to use histogram equalization to enhance the contrast of the image.

[0017] As a further solution of the present invention: the process of judging whether the gray value distribution of the area within the sliding window is abnormal is as follows:

[0018] The abnormal thresholds include α and β, both α and β are positive numbers, and the ratio of Z2 to Z1 is marked as R. Only when |V1-V2|>α and |R-1|>β exists, the grayscale value distribution of the area in the sliding window is judged to be abnormal, and the area is marked as an abnormal area. Otherwise, it is judged to be normal and the area is marked as a pending area.

[0019] As a further solution of the present invention: the process of judging whether there is an abnormal circle of confusion is as follows:

[0020] The grayscale image in the area to be determined is binarized to obtain a binarized image of the area to be determined. All circular contours in the binarized image are detected according to the circularity, and the circular contours with an internal grayscale value of 255 are screened and marked as diffuse circles. The straight-line distance from the center point of the diffuse circle contour to the edge of the contour is calculated and marked as the radius r. A radius threshold interval k is set. If there is a diffuse circle whose value of r is not within the radius threshold interval k, the diffuse circle is judged to be an abnormal diffuse circle, and the minimum circumscribed rectangular area of ​​the abnormal diffuse circle is marked as an abnormal area.

[0021] As a further solution of the present invention: the setting process of the radius threshold interval k is:

[0022] After marking the circular contours in the binary image using the minimum enclosing rectangle, the mean radius of all circular contours is calculated and marked as r 均 , k = [r 均 -a,r 均+ a], where a is the preset error value.

[0023] As a further solution of the present invention, the specific process of visually outputting the recognition result on the refined inspection image is as follows:

[0024] According to the abnormal area, the regional image of the corresponding position is cut out on the detailed examination image, the cut image is marked as the ROI image, and the size of all ROI images is unified;

[0025] The ROI image is input into the preset wood surface defect recognition model for recognition, to detect whether there are real defects in the corresponding area of ​​the wood surface and to determine the specific type of defects. After the recognition of all ROI images is completed, the recognition results are marked in the corresponding areas on the detailed inspection image.

[0026] As a further solution of the present invention: the training process of the wood surface defect model is:

[0027] Constructing a data set, wherein the data set includes normal wood surface image data and wood surface image data with known defects and marked;

[0028] The data set is divided into a training set, a validation set and a test set, wherein each of the training set, the validation set and the test set contains normal wood surface image data and wood surface image data with known defects and marked;

[0029] Using the image data in the training set, the wood surface defect recognition model is trained through the back propagation algorithm and optimizer, and the recognition effect of the model is verified using the validation set;

[0030] The model is adjusted and optimized according to the verification results fed back by the verification set to obtain the final wood surface defect recognition model, and the test set is used to evaluate the recognition effect of the final wood surface defect recognition model trained.

[0031] Beneficial effects of the present invention:

[0032] The present invention utilizes a grading strategy and multimodal data, and realizes a rapid preliminary screening of wood surface defects by gray value and confusion circle detection before defect recognition on the wood surface. In the preliminary screening stage, the present invention also utilizes the depth of field principle, blurs the defective area in the image acquisition stage, highlights the difference between the defective area and the normal area, and makes the defective area easier to be screened out. At the same time, the present invention utilizes the characteristics that the gray value distribution of the blurred defective area is different from that of the normal area, and the confusion circle formed by the defective area is also different from that of the normal area. The abnormal area is determined by detecting the area with abnormal gray value distribution and the area with abnormal confusion circle through a sliding window, thereby improving the speed of preliminary screening. After the preliminary screening, a ROI image with possible defects is obtained according to the abnormal area, and only the ROI image is input into the recognition model without the need to recognize the complete image, thereby reducing the workload of model recognition and thereby improving the image recognition speed. Compared with the method that only relies on the image recognition model, the present invention has better performance in detection speed when facing large-scale wood surface defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below in conjunction with the accompanying drawings.

[0034] Figure 1 It is a schematic diagram of the process of the present invention;

[0035] Figure 2 It is a flow chart for identifying abnormal areas in primary screening images;

[0036] Figure 3 It is a flow chart for detecting abnormal diffuse circles in the undetermined area. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1-3 As shown, the present invention is a wood surface defect detection method based on multimodal data, comprising the following steps:

[0039] Set the image sensor directly above the wood, and adjust the parameters to align the far point plane of the image sensor's back depth of field with the wood surface. The back depth of field is the range in which the rear of the subject can be clearly imaged, and the far point of the back depth of field represents the farthest distance that the back depth of field imaging range can reach. Objects beyond this distance cannot be clearly imaged. This step is to blur the image of the defective part on the wood surface and highlight the difference between it and the normal area. Collect the image of the wood surface and mark it as a preliminary screening image. When collecting the image, a neutral light source needs to be used to provide uniform illumination for the wood surface to prevent ambient light from interfering with the wood surface. At the same time, providing uniform illumination can also highlight the characteristics of the defective area to a certain extent.

[0040] The initial screening image is preprocessed, and the preprocessing includes image denoising and image enhancement. The image denoising method includes mean filtering, median filtering or Gaussian filtering. The image enhancement method uses histogram equalization to enhance the contrast of the image. The image preprocessing is performed to remove the interference left on the image by some external factors during shooting, improve the image quality, and make the image features more prominent and easier to identify.

[0041] Convert the pre-processed primary screening image into a grayscale image, calculate the mean V1 and variance Z1 of the grayscale value of the grayscale image, and use a sliding window to detect the grayscale image. The size of the sliding window can be set slightly larger to quickly detect the grayscale image, but the window setting should not be too small. A too small window will reduce the number of pixels in the area covered by the window, so the variance of the grayscale value will also decrease accordingly, which will increase the misjudgment rate. The step size of the window is generally set to half or one-quarter of the window size, calculate the mean V2 and variance Z2 of the grayscale value in each sliding window, and compare V2 and Z2 with the mean V1 and variance Z1 respectively, and judge whether the grayscale value distribution in the window area is abnormal through the abnormal threshold.

[0042] The abnormal thresholds include α and β, both of which are positive numbers. The ratio of Z2 to Z1 is marked as R. Only when |V1-V2|>α and |R-1|>β exists, the grayscale value distribution of the area in the sliding window is judged to be abnormal, otherwise it is judged to be normal. The grayscale value distribution can be used for judgment because the defective area on the surface of the wood has been blurred in advance using the depth of field principle, and the grayscale value of the blurred part in the image is more uniform than that of the normal part, and the variance is smaller than that of the normal area. Because the blurred area is close to the background area, it is possible to judge whether the area is abnormal based on the distribution characteristics of the grayscale value in the area.

[0043] The parameters selected to judge whether the gray value distribution is abnormal are the mean and variance, because the gray value mean and variance of most areas on the wood surface are determined by the normal area, and the normal area is not blurred, so the original color difference and texture difference are retained, so the gray value mean is higher, and the gray value variance in the area is larger. Therefore, the regional mean and variance in the sliding window area are compared with the overall mean and variance to obtain the gray value distribution in the window area. The window area with abnormal gray value distribution is marked as an abnormal area, and the area with normal gray value distribution is marked as a pending area.

[0044] The area to be determined is subjected to a confusion circle detection to determine whether there is an abnormal confusion circle in the area to be determined. The process of detecting the confusion circle is as follows: the grayscale image in the area to be determined is binarized to obtain a binary image of the area to be determined, all circular contours in the binary image are detected according to the circularity, and the circular contours with an internal grayscale value of 255 are screened and marked as confusion circles. First, confusion circles are generated because the defective area cannot be imaged clearly. Although the normal area can be imaged clearly, some confusion circles will still be formed. Moreover, since uniform illumination is provided to the wood surface when the image is collected, the confusion circles formed by the defective area and the normal area will have a higher brightness. In the binary image, the color appears as white, and the grayscale value corresponding to white is 255.

[0045] Calculate the straight-line distance from the center point of the diffuse circle contour to the edge of the contour and mark it as the radius r, set the radius threshold interval k, and set the radius threshold interval k as follows: after marking the circular contour in the binary image with the minimum circumscribed rectangle, calculate the mean radius of all circular contours and mark it as r 均 , k = [r 均 -a,r 均+ a], a is the preset error value. If there is a certain confusion circle whose value of r is not in the interval k, the confusion circle is judged to be an abnormal confusion circle, and the minimum circumscribed rectangular area of ​​the abnormal confusion circle is marked as an abnormal area. The basis for judgment is that the size of the confusion circle in the normal area is relatively uniform, while the size of the confusion circle in the defect area is significantly different from that in the normal area. For example, larger defects such as nodules, scars or spots will form larger confusion circles after blurring.

[0046] Increase the depth of field, take another picture of the wood surface and mark it as a detailed inspection image. The purpose of collecting detailed inspection images is that the image that is finally input into the image recognition model should ensure that all details of the wood surface can be clearly imaged. If the image details are not enough, the recognition accuracy of the model will also be affected. During the shooting process, it is necessary to ensure that the focus of the image sensor is the same as when the initial screening image is collected, and the position of the image sensor and the wood surface remains unchanged during the two image collection processes. This step is to ensure that the positional relationship between the detailed inspection image and the different areas in the initial screening image can accurately correspond to each other, so as to facilitate the subsequent cropping operation according to the abnormal area.

[0047] In the detailed inspection image, the image area corresponding to the abnormal area is cropped and marked as the ROI image to train the wood surface defect recognition model. The main training process is:

[0048] The model can use a CNN neural network model or a YOLO model to construct a data set, wherein the data set includes normal wood surface image data and wood surface image data with known defects and marked. The data set can be constructed by collecting images on site and manually annotating them, or by using an existing public data set;

[0049] The data set is divided into a training set, a validation set and a test set. The training set, the validation set and the test set each contain normal wood surface image data and wood surface image data with known defects and marked. The image data in the training set accounts for the largest proportion of the total image data in the data set. The number of images in the training set generally accounts for about 96% of the total number of images in the data set.

[0050] The wood surface defect recognition model is trained using the image data in the training set through the back propagation algorithm and optimizer, and the recognition effect of the model is verified using the validation set.

[0051] The process of training the model using the training set and verifying the effect using the validation set is a repeated feedback process. The model architecture or internal parameters are continuously adjusted and optimized according to the verification results of the validation set to obtain the final wood surface defect recognition model. The purpose of using the test set is to evaluate the performance of the trained wood surface defect recognition model when recognizing unseen images.

[0052] After unifying the size of all ROI images, they are input into the preset wood surface defect recognition model for accurate recognition, to determine whether there are real defects in the corresponding area of ​​the wood surface, and to determine the type of defects. The ROI image size is unified because the input image size acceptable to the image recognition model is generally fixed. At the same time, the unified ROI image allows the model to support batch recognition. After the model recognizes all ROI images, the recognition results will be marked at the corresponding position on the refined image.

[0053] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0054] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0055] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A wood surface defect detection method based on multimodal data, characterized in that: The following steps are involved: The image sensor is placed directly above the wood, and the parameters are adjusted so that the background deep point plane of the image sensor is aligned with the wood surface, and the wood surface image is collected and marked as the primary screening image; Preprocess the initial screening image, convert the preprocessed initial screening image into a grayscale image, and calculate the mean V1 and variance Z1 of the grayscale value of the grayscale image; Use a sliding window to detect the grayscale image, calculate the mean V2 and variance Z2 of the grayscale value in each sliding window, and compare V2 and Z2 with the mean V1 and variance Z1 respectively. Use the abnormal threshold to determine whether the grayscale value distribution in the window area is abnormal. Mark the window area with abnormal grayscale value distribution as an abnormal area, and mark the area with normal grayscale value distribution as a pending area. Perform confusion circle detection on the area to be determined to determine whether there is an abnormal confusion circle in the area to be determined. When an abnormal confusion circle exists in the area to be determined, the minimum circumscribed rectangular area of ​​the abnormal confusion circle is marked as an abnormal area. Increase the depth of field, take the wood surface image again and mark it as a detailed inspection image, crop the image area corresponding to the abnormal area in the detailed inspection image and mark it as a ROI image, input the ROI image into the preset wood surface defect recognition model, and visualize the output recognition result on the detailed inspection image.

2. A wood surface defect detection method based on multimodal data according to claim 1, characterized in that: When collecting the preliminary screening image and the detailed inspection image on the same wood surface, a neutral light source is used to provide uniform illumination for the wood surface, and the relative position of the image sensor and the wood surface and the focus of the image sensor remain unchanged during the two collection processes.

3. The wood surface defect detection method based on multimodal data according to claim 1 is characterized in that: The preprocessing includes image denoising and image enhancement, and the image denoising method includes mean filtering, median filtering or Gaussian filtering; The image enhancement method is to use histogram equalization to enhance the contrast of the image.

4. The wood surface defect detection method based on multimodal data according to claim 1 is characterized in that: The process of judging whether the gray value distribution in the sliding window area is abnormal is as follows: The abnormal thresholds include α and β, both α and β are positive numbers, and the ratio of Z2 to Z1 is marked as R. Only when |V1-V2|>α and |R-1|>β exists, the grayscale value distribution of the area in the sliding window is judged to be abnormal, and the area is marked as an abnormal area. Otherwise, it is judged to be normal and the area is marked as a pending area.

5. The method for detecting wood surface defects based on multimodal data according to claim 4, characterized in that: The process of judging whether there is an abnormal circle of confusion is as follows: The grayscale image in the area to be determined is binarized to obtain a binarized image of the area to be determined. All circular contours in the binarized image are detected according to the circularity, and the circular contours with an internal grayscale value of 255 are screened and marked as diffuse circles. The straight-line distance from the center point of the diffuse circle contour to the edge of the contour is calculated and marked as the radius r. A radius threshold interval k is set. If there is a diffuse circle whose value of r is not within the radius threshold interval k, the diffuse circle is judged to be an abnormal diffuse circle, and the minimum circumscribed rectangular area of ​​the abnormal diffuse circle is marked as an abnormal area.

6. The method for detecting wood surface defects based on multimodal data according to claim 5, characterized in that: The setting process of the radius threshold interval k is: After marking the circular contours in the binary image using the minimum enclosing rectangle, the mean radius of all circular contours is calculated and marked as r 均 , k = [r 均 -a,r 均+ a], where a is the preset error value.

7. The method for detecting wood surface defects based on multimodal data according to claim 6, characterized in that: The specific process of visualizing the output recognition results on the refined inspection image is as follows: According to the abnormal area, the regional image of the corresponding position is cut out on the detailed examination image, the cut image is marked as the ROI image, and the size of all ROI images is unified; The ROI image is input into the preset wood surface defect recognition model for recognition, to detect whether there are real defects in the corresponding area of ​​the wood surface and to determine the specific type of defects. After the recognition of all ROI images is completed, the recognition results are marked in the corresponding areas on the detailed inspection image.

8. The method for detecting wood surface defects based on multimodal data according to claim 7, characterized in that: The training process of the wood surface defect model is: Constructing a data set, wherein the data set includes normal wood surface image data and wood surface image data with known defects and marked; The data set is divided into a training set, a validation set and a test set, wherein each of the training set, the validation set and the test set contains normal wood surface image data and wood surface image data with known defects and marked; Using the image data in the training set, the wood surface defect recognition model is trained through the back propagation algorithm and optimizer, and the recognition effect of the model is verified using the validation set; The model is adjusted and optimized according to the verification results fed back by the verification set to obtain the final wood surface defect recognition model, and the test set is used to evaluate the recognition effect of the final wood surface defect recognition model trained.

Images