Improved YOLOv4-based self-adaptive broken tooth surface inclusion judgment method and system

Abstract

The invention discloses a self-adaptive broken tooth surface inclusion judgment method and system based on improved YOLOv4. The method comprises the steps that 1, a gear broken tooth surface image is collected; 2, converting the image into a grayscale image; 3, obtaining a to-be-fused feature matrix of the gray-level co-occurrence matrix feature quantity of the single picture; 4, obtaining model training sample data; 5, constructing a broken tooth surface inclusion detection model which comprises a trunk feature extraction network, a feature pyramid network, a feature merging module and a detection head; 6, inputting a to-be-fused feature matrix of the gray-level co-occurrence matrix feature quantity of the single picture and model training sample data into the broken tooth surface inclusion detection model, and performing training and verification to obtain a broken tooth surface inclusion detection model which is verified to be qualified; and step 7, processing the to-be-detected broken tooth surface image by using the model which is verified to be qualified to obtain a gear broken tooth detection result. According to the invention, the detection of the inclusions on the broken tooth surface is more accurate.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an adaptive broken tooth surface inclusion judgment method and system based on improved YOLOv4. Background Art

[0002] At present, gear transmission is the most widely used form of transmission in mechanical transmission. It is also an important part of the automobile transmission system and plays an important role in the operation of the automobile. Gear failure is also an important cause of failure in the automobile transmission system. Among them, gear tooth breakage is the main failure mode of gears, and stress concentration is one of the main causes of gear tooth breakage. Due to the presence of inclusions in the gear, the gear stress will be concentrated, which makes the gear more prone to tooth breakage. Therefore, analyzing the problem of gear inclusions is extremely important to improve gear quality and reduce gear failures.

[0003] At present, the main method to determine whether there are inclusions on the gear cross section is to magnify the broken tooth surface through a high-power electronic magnifying glass and judge based on the experience of the gear engineer. The equipment is not portable and the labor cost is relatively high. The cycle from on-site tooth breaking to analysis is relatively long. There are certain technical barriers to analyzing the causes of gear tooth breaking. Inexperienced engineers find it difficult to determine the causes of gear tooth breaking and cannot provide timely feedback to the manufacturer, which affects product quality. Summary of the invention

[0004] In view of the problems that the cause of gear tooth breakage is mainly dependent on the experience of engineers, there are many equipment limitations, high labor costs, long analysis cycles, and certain technical barriers in the analysis of the cause of gear tooth breakage, the present invention proposes a method for determining inclusions on the broken tooth surface based on deep neural network adaptation, so as to reduce the difficulty of determining inclusions on the broken tooth surface and more accurately and conveniently determine the inclusions on the broken tooth surface and their location and size.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: On the one hand, the present invention provides an adaptive broken tooth surface inclusion judgment method based on improved YOLOv4, which specifically includes the following steps: Step 1: Collect the image of the broken tooth surface of the gear; Step 2: converting the acquired gear broken tooth surface image into a grayscale image; Step 3: Adjust the grayscale image to a uniform size and perform data enhancement processing to obtain a training image data set; then calculate the grayscale co-occurrence matrix for each image in the training image data set to obtain a feature matrix to be fused of the grayscale co-occurrence matrix feature quantity of a single image; Step 4: For the training image dataset obtained in step 3, mark the center position and size of the inclusions on each image, and mark whether there are inclusions to obtain model training sample data; Step 5: Construct a broken tooth surface inclusion detection model, including a backbone feature extraction network, a feature pyramid network, a feature merging module and a detection head; wherein the backbone feature extraction network is used to extract the spatial features and semantic features of the input image to obtain three-scale broken tooth surface feature maps; the feature pyramid network is used to fuse the three-scale broken tooth surface feature maps output from the backbone feature extraction network, and output three-scale feature images of broken tooth surface inclusions; the feature merging module is used to correspondingly splice the feature matrix to be fused of the gray level co-occurrence matrix feature quantity of a single image with the three-scale feature images of the broken tooth surface inclusions obtained by the feature pyramid network to obtain a feature merging matrix of three scales; the detection head is used to detect the broken tooth surface inclusions in the image through the input feature merging matrix of three scales; Step 6: Input the to-be-fused feature matrix of the gray-level co-occurrence matrix feature quantity of the single image obtained in step 3 and the model training sample data obtained in step 4 into the broken tooth surface inclusion detection model constructed in step 5, train and verify the model, and obtain a verified qualified broken tooth surface inclusion detection model; Step seven, use the qualified model verified in step six to process the image of the broken tooth surface to be detected, and obtain the gear broken tooth detection result.

[0006] Furthermore, step three specifically includes the following sub-steps: Step 31, adjust the grayscale image into image data of uniform size; use data enhancement to expand the training data, and then perform normalization processing; Step 32, extract features from each image obtained in step 31 using a gray level co-occurrence matrix to obtain a gray level co-occurrence matrix, perform normalization processing, calculate the eigenvalues ​​of the direction matrix in four directions of 0°, 45°, 90°, and 135°, and use the average value as the eigenvalue co-occurrence matrix; Step 33, calculate the following five feature quantities of the grayscale co-occurrence matrix according to the eigenvalue co-occurrence matrix obtained in step 32: energy, contrast, homogeneity, entropy and inverse difference, and then expand each feature quantity of the single image into feature matrices of 52×52×1, 26×26×1, and 13×13×1 respectively, stack the matrices of these five feature quantities, and obtain the feature matrices to be fused of the grayscale co-occurrence matrix feature quantities of the single image: 52×52×5, 26×26×5, and 13×13×5.

[0007] Furthermore, in step five: (1) The backbone feature extraction network includes a Focus layer and five residual modules connected in sequence; Focus layer: used to convert the input 416×416×1 image into a 208×208×32 feature map through Focus operation and 32 3×3 convolutions in sequence; Residual module: used to implement the following operations: first, a convolution with a kernel size of 3×3 and a stride of 2 is performed to divide the input feature map into two equal parts, and then each part enters a 1×1 convolution to obtain the output results of the two parts. The output results of one part are processed by the residual block and a 1×1 convolution in turn, and the processed results are stacked and merged with the output results of the other part; (2) The feature pyramid network is used to implement the following operations: ① Obtain high-resolution low-level feature images, intermediate feature images, and low-resolution high-level feature images output by the third residual module, the fourth residual module, and the fifth residual module of the backbone feature extraction network, respectively; ② Upsample half of the high-level feature image to obtain an upsampled data set, and perform five convolutions on the image obtained by combining the data set with the intermediate feature image to obtain an intermediate fused feature image; ③ Upsample half of the intermediate fused feature image to obtain an upsampled data set, and perform five convolutions on the image obtained by combining the data set with the low-level feature image to obtain a low-level fused feature image; ④ Output the low-level fused feature image to obtain a low-level fused feature image output set; At the same time, downsample the low-level fused feature image to obtain a downsampled data set, and stack and merge it with the intermediate fused feature image to obtain an image, and then perform five convolutions to obtain an intermediate fused feature image output set; ⑤ Stack and merge the image obtained by downsampling the intermediate fused feature image output set with the high-level feature image to obtain an image, and then perform five convolutions on it to obtain a high-level fused feature image output set; (3) Feature merging module: used to concatenate the low-level fused feature image output set, the intermediate-level fused feature image output set, and the high-level fused feature image output set output by the feature pyramid network with the feature matrices to be fused of the gray-level co-occurrence matrix feature quantities of the single images of 52×52×5, 26×26×5, and 13×13×5 obtained in step 3, one by one, to obtain a feature merging matrix after feature merging; (4) Detection head: It is used to merge the input feature matrices of three scales (13×13×517, 26×26×261, and 52×52×133) and compress the number of channels through multiple convolutions to obtain the multi-scale feature maps of YOLO Head: 13×13×21, 26×26×21, and 52×52×21.

[0008] Furthermore, step seven includes the following process: (1) Preprocessing the image of the broken tooth surface to be detected; (2) Inputting the preprocessed image into the qualified broken tooth surface inclusion detection model obtained in step 6 for detection; (3) Output the gear broken tooth detection results. If there are inclusions, the detection results include the location and size of the inclusions.

[0009] Furthermore, the method further comprises step eight: evaluating the gear tooth breakage detection result outputted from step seven to form an evaluation report of the gear tooth breakage surface.

[0010] Furthermore, it also includes step nine: using a storage server to store the collected broken tooth surface images, preprocessed images, model iteration training results, broken tooth surface inclusion detection models, gear broken tooth detection results and evaluation reports.

[0011] Furthermore, the method further includes step ten: displaying the content stored in step nine.

[0012] On the other hand, the present invention also provides an adaptive broken tooth surface inclusion judgment system based on improved YOLOv4, which specifically includes the following modules: An image acquisition module is used to acquire images of broken gear tooth surfaces; A grayscale image conversion module is used to convert the acquired gear broken tooth surface image into a grayscale image; The module for calculating the feature matrix to be fused is used to adjust the grayscale image to a uniform size and perform data enhancement processing to obtain a training image data set; then each image in the training image data set is calculated through the grayscale co-occurrence matrix to obtain the feature matrix to be fused of the grayscale co-occurrence matrix feature quantity of a single image; Training sample acquisition module: used to implement the following process: for the training image data set obtained by the feature matrix calculation module to be fused, mark the center position and size of the inclusions on each image, and mark whether there are inclusions, to obtain model training sample data; Model construction module: used to construct a broken tooth surface inclusion detection model, including a backbone feature extraction network, a feature pyramid network, a feature merging module and a detection head; wherein the backbone feature extraction network is used to extract the spatial features and semantic features of the input image to obtain three-scale broken tooth surface feature maps; the feature pyramid network is used to fuse the three-scale broken tooth surface feature maps output from the backbone feature extraction network, and output three-scale feature images of broken tooth surface inclusions; the feature merging module is used to correspondingly splice the feature matrix to be fused of the gray level co-occurrence matrix feature quantity of a single image with the three-scale feature images of the broken tooth surface inclusions obtained by the feature pyramid network to obtain the three-scale feature merging matrix; the detection head is used to detect the broken tooth surface inclusions in the image through the input three-scale feature merging matrix; The training and verification module is used to input the feature matrix to be fused of the gray-level co-occurrence matrix feature quantity of a single image obtained by the feature matrix calculation module to be fused and the model training sample data obtained by the training sample acquisition module into the broken tooth surface inclusion detection model constructed by the model construction module, train and verify the model, and obtain a verified qualified broken tooth surface inclusion detection model; The gear broken tooth detection module is used to process the broken tooth surface image to be detected using the model verified by the training and verification module to obtain the gear broken tooth detection result.

[0013] Compared with the prior art, the present invention has the following beneficial effects: Based on the YOLOv4 model, the present invention proposes a method of combining the texture features obtained with the grayscale co-occurrence matrix, and applies it to the detection of broken tooth surface inclusions, so that the detection of broken tooth surface inclusions is more accurate. An adaptive broken tooth surface inclusion judgment system is established based on this method, which reduces the technical difficulty of broken tooth surface inclusion judgment, makes it easier for non-professionals to judge the cause of gear broken tooth failure, and promptly feedbacks product quality issues, effectively improving the reliability of new products. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural schematic diagram of the broken tooth surface inclusion detection model; Figure 2 It is a structural diagram of the residual module (Resblock_body); Figure 3 is a flow chart of the method of the present invention; Figure 4 The sample pictures and the enlarged pictures of inclusions of the embodiments of the present invention are shown.

[0015] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0016] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.

[0017] Embodiment 1: This embodiment provides an adaptive broken tooth surface inclusion judgment method based on improved YOLOv4, which specifically includes the following steps: Step 1: Collect the image of the broken tooth surface of the gear; Specifically, an industrial camera is used to photograph the broken tooth surface of the gear. The industrial camera photographs the broken tooth surface of the gear vertically at a fixed distance, and the photographed image must include the entire broken tooth surface.

[0018] Step 2: converting the acquired gear broken tooth surface image into a grayscale image; Specifically, the collected three-channel color gear broken tooth surface image is converted into a single-channel grayscale image, which can reduce the amount of calculation for subsequent broken tooth surface identification.

[0019] Step 3: Adjust the grayscale image to a uniform size and perform data enhancement processing to obtain a training image dataset; then calculate each image in the training image dataset through GLCM (gray-level co-occurrence matrix) to obtain the feature matrix to be fused of the gray-level co-occurrence matrix feature quantity of a single image; Specifically, the following sub-steps are included: Step 31, adjust the grayscale image into image data of uniform size (such as 416×416) to be suitable for the training of the broken tooth surface inclusion detection model; and use data enhancement technology (such as random cropping, flipping, etc.) to expand the training data, and then perform normalization processing, as shown in the following formula. Since the model training is highly correlated with the key information in the image, the image is required to contain a complete broken tooth surface, and finally a training image data set is obtained.

[0020]

[0021] in is the normalization result of pixel gray value, is the pixel gray value, is the maximum grayscale value of the pixel, is the minimum pixel grayscale value.

[0022] Step 32, each image obtained in step 31 is subjected to feature extraction using GLCM (gray-level co-occurrence matrix), the gray level is selected as 8, the window matrix size selected for calculating the eigenvalue is 5, the step size d=1, and an 8×8 gray-level co-occurrence matrix is ​​obtained. The matrix is ​​normalized, and the eigenvalues ​​of the direction matrix are obtained in the four directions of 0°, 45°, 90°, and 135°, and the average value is used as the eigenvalue co-occurrence matrix; Step 33, according to the eigenvalue co-occurrence matrix obtained in step 32, the following five feature quantities of the gray-level co-occurrence matrix are calculated: energy, contrast, homogeneity, entropy and inverse difference, and then each feature quantity of a single image is expanded into a feature matrix of 52×52×1, 26×26×1, and 13×13×1 respectively, and the matrices of these five feature quantities are stacked to obtain the feature matrix to be fused of the gray-level co-occurrence matrix feature quantity of a single image: 52×52×5, 26×26×5, and 13×13×5. Among them, the five feature quantities are as follows: ① Energy (ASM) is the sum of the squares of the elements of the gray-level co-occurrence matrix, which reflects the uniformity of the gray-level distribution and the coarseness of the texture of the image.

[0023] ②Contrast is the moment of inertia near the main diagonal of the grayscale co-occurrence matrix, which reflects the clarity of the image and the depth of the texture grooves.

[0024] ③ Inverse variance (Homogeneity) reflects the homogeneity of image texture and measures the amount of local changes in image texture.

[0025] ④Entropy, which reflects the randomness of image texture.

[0026] ⑤Correlation (Corrln) reflects the similarity of the elements of the spatial grayscale co-occurrence matrix in the row or column direction, and reflects the local grayscale correlation of the image.

[0027] Step 4: For the training image dataset obtained in step 3, mark the center position (x, y coordinate values) and size (including the width w and height h of the bounding box) of the inclusions on each image, and mark whether there are inclusions to obtain model training sample data; Step 5: Construct a broken tooth surface inclusion detection model.

[0028] The YOLOv4 model is an end-to-end target detection network that can simultaneously detect the position and category of multiple objects in a picture. Therefore, the present invention constructs a broken tooth surface inclusion detection model based on the YOLOv4 model, and then obtains a broken tooth surface inclusion detection module, which is used to determine the position and size of inclusions in the newly obtained broken tooth surface image. Figure 1 As shown in the figure, the structure of the constructed broken tooth surface inclusion detection model is as follows: Specifically, the broken tooth surface inclusion detection model is as follows: Figure 1 As shown, it includes a backbone feature extraction network (CSPDarknet53), a feature pyramid network (PANet), a feature merging module and a detection head (YOLO Head). Among them, the backbone feature extraction network is used to extract the spatial features and semantic features of the input broken tooth surface image through multi-layer convolution to obtain three-scale broken tooth surface feature maps; the feature pyramid network is used to fuse the three-scale broken tooth surface feature maps output from the backbone feature extraction network, and output three-scale feature images of broken tooth surface inclusions; the feature merging module is used to splice the feature matrix to be fused (obtained from step 3) of the gray level co-occurrence matrix feature quantity of a single image with the three-scale feature images of the broken tooth surface inclusions obtained by the feature pyramid network, and obtain the three-scale feature merging matrix; the detection head is used to detect the broken tooth surface inclusions in the image through the input three-scale feature merging matrix.

[0029] (1) Backbone feature extraction network (CSPDarknet53), including Focus layer and 5 residual modules (Resblock_body) connected in sequence. Among them: Focus layer: It is used to convert the input 416×416×1 image (Inputs) into a 208×208×32 feature map by sequentially passing through the Focus operation (decomposing each 2×2 pixel block into 4 channels) and 32 3×3 convolutions. Using the Focus layer can reduce the spatial size while maintaining the image information density.

[0030] Residual module: such as Figure 2 As shown in the figure, the residual module is used to implement the following operations: first, a convolution (Conv) with a convolution kernel size of 3×3 and a stride of 2 is performed, and the input feature map is divided into two equal parts, and then each part enters a 1×1 convolution to obtain the output results of the two parts. The output results of one part are processed by the residual block (containing two convolution layers, namely 1×1 convolution and 3×3 convolution) and a 1×1 convolution in turn, and the processed results are stacked and merged with the output results of the other part. The residual module designed as above makes it easier to optimize the input feature map, and can improve the confidence of detecting inclusions on the broken tooth surface by increasing the depth. The internal residual block uses jump connections to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0031] Specifically, the structures of the 5 residual modules (Resblock_body_1~Resblock_body_5) are: ①1 residual block + 64 3×3 convolutions; ②2 residual blocks + 128 3×3 convolutions; ③8 residual blocks + 256 3×3 convolutions; ④8 residual blocks + 512 3×3 convolutions; ⑤4 residual blocks + 1024 3×3 convolutions.

[0032] (2) After the feature pyramid network (PANet) completes the feature extraction from bottom to top, it is also necessary to realize the feature extraction from top to bottom. Specifically: ① Obtain the high-resolution low-level feature image (52×52×256), intermediate feature image (26×26×512), and low-resolution high-level feature image (13×13×1024) output by the 3rd residual module, the 4th residual module, and the 5th residual module of the backbone feature extraction network respectively; ② Upsample half of the high-level feature image (13×13×512) to obtain the upsampled data set (26×26×256), and perform 5 convolutions on the image obtained by combining the data set with the intermediate feature image (26×26×768) to obtain the intermediate fused feature image (26×26×256); ③ Upsample half of the intermediate fused feature image (13×13×128) to obtain the upsampled data set (52×52×128), and perform 5 convolutions on the image obtained by combining the data set with the low-level feature image (5 2×52×384) is convolved 5 times to obtain a low-level fusion feature image (52×52×128); ④ The low-level fusion feature image is output to obtain a low-level fusion feature image output set (52×52×128); at the same time, the low-level fusion feature image is downsampled to obtain a downsampled data set (26×26×256), which is stacked and merged with the intermediate fusion feature image to obtain an image (26×26×512), and then convolved 5 times to obtain an intermediate fusion feature image output set (26×26×256); ⑤ The image (13×13×512) obtained by downsampling the intermediate fusion feature image output set is stacked and merged with the high-level feature image (13×13×512) to obtain an image (13×13×1024), which is convolved 5 times to obtain a high-level fusion feature image output set (13×13×512).

[0033] (3) Feature merging module: It is used to concatenate the low-level fused feature image output set (52×52×128), the intermediate fused feature image output set (26×26×256), and the high-level fused feature image output set (13×13×512) output by the feature pyramid network (PANet) with the feature matrices to be fused of the gray-level co-occurrence matrix feature quantities of the single images of 52×52×5, 26×26×5, and 13×13×5 obtained in step 3, to obtain the feature merging matrix after feature merging.

[0034] (4) Detection head (YOLO Head): It is used to merge the input feature matrices of three scales (13×13×517, 26×26×261, and 52×52×133) through multiple convolutions to compress the number of channels, and obtain the multi-scale feature maps of the YOLO Head: 13×13×21, 26×26×21, and 52×52×21; since the feature layer prediction results correspond to the positions of the three prediction boxes, the outputs of the single prediction boxes of the broken tooth surface inclusions are 13×13×7, 26×26×7, and 52×52×7, respectively.

[0035] Step six: Input the feature matrix to be fused of the grayscale co-occurrence matrix feature quantity of the single image obtained in step three and the model training sample data obtained in step four into the broken tooth surface inclusion detection model constructed in step five, train and verify the model, and obtain a verified qualified broken tooth surface inclusion detection model.

[0036] (1) Input the model training sample data obtained in step 4 into the broken tooth surface inclusion detection model constructed in step 5; (2) When training the broken tooth surface inclusion detection model constructed in step 5, the model training sample data obtained in step 4 is used as input, and the x, y coordinate values ​​of the center position of the inclusion, the inclusion width w, the inclusion height h, and whether there is an inclusion are output. The goal is to minimize the LOSS value of the loss function shown in the following formula, and the Adam gradient descent optimization algorithm is used for iterative training, and the cosine annealing scheduler is used to adjust the learning rate.

[0037]

[0038] Among them, CIoU Loss stands for CIoU loss, as shown in the following formula, which is used for bounding box regression to optimize the overlap, center point distance and aspect ratio between the predicted box and the real box;

[0039]

[0040] In the formula, IoU represents the overlap rate between the predicted box and the real box. Represents the prediction box With real box The Euclidean distance between the center points, Represents the diagonal length of the minimum enclosing box that surrounds the predicted box and the true box, is the aspect ratio similarity penalty term, which is used to measure the consistency of the aspect ratio of the predicted box and the aspect ratio of the real box. and are the width and height of the prediction box, respectively. and is the width and height of the real box, is the weight.

[0041] Among them, Confidence Loss represents the confidence loss, and uses binary cross entropy loss, as shown in the following formula, to optimize confidence prediction;

[0042] In the formula, is the true label, is the probability predicted by the model; is the binary cross entropy loss; Among them, Classification Loss represents the category loss, and binary cross entropy loss is used to optimize the prediction of the existence probability of inclusions. , , are the corresponding weighting coefficients, usually selected , , .

[0043] (3) When verifying the broken tooth surface inclusion detection model, mAP (mean Average Precision) is used as an evaluation indicator to measure the detection performance of the model on the validation set; (4) Store the verified model.

[0044] Step seven, use the qualified model verified in step six to process the image of the broken tooth surface to be detected, and obtain the gear broken tooth detection result.

[0045] (1) Preprocessing the image of the broken tooth surface to be inspected, including converting it into a grayscale image and adjusting it to a uniform size; (2) Inputting the preprocessed image into the qualified broken tooth surface inclusion detection model obtained in step 6 for detection; (3) Output the gear broken tooth detection results. If there are inclusions, the detection results include the location and size of the inclusions.

[0046] The method of the present invention further comprises the steps of: Step 8: Evaluate the gear tooth breakage detection result output in step 7 to form an evaluation report of the gear tooth breakage surface; (1) Evaluate the gear tooth breakage detection results output in step 7; (2) Produce an evaluation report on the gear tooth surface.

[0047] Step 9: Store the data, training process, verification model and results; The storage server is used to store the collected broken tooth surface images, preprocessed images (obtained from steps two and three), model iteration training results (obtained from step six), broken tooth surface inclusion detection model (obtained from step six), gear broken tooth detection results (obtained from step seven) and evaluation report (obtained from step eight).

[0048] Step 10: Display the results.

[0049] Display the content stored in step nine.

[0050] Embodiment 2: The adaptive broken tooth surface inclusion judgment system based on improved YOLOv4 provided in this embodiment specifically includes the following modules: An image acquisition module is used to acquire images of broken gear tooth surfaces; A grayscale image conversion module is used to convert the acquired gear broken tooth surface image into a grayscale image; The module for calculating the feature matrix to be fused is used to adjust the grayscale image to a uniform size and perform data enhancement processing to obtain a training image data set; then each image in the training image data set is calculated through the grayscale co-occurrence matrix to obtain the feature matrix to be fused of the grayscale co-occurrence matrix feature quantity of a single image; Training sample acquisition module: used to implement the following process: for the training image data set obtained by the feature matrix calculation module to be fused, mark the center position and size of the inclusions on each image, and mark whether there are inclusions, to obtain model training sample data; Model construction module: used to construct a broken tooth surface inclusion detection model, including a backbone feature extraction network, a feature pyramid network, a feature merging module and a detection head; wherein the backbone feature extraction network is used to extract the spatial features and semantic features of the input image to obtain three-scale broken tooth surface feature maps; the feature pyramid network is used to fuse the three-scale broken tooth surface feature maps output from the backbone feature extraction network, and output three-scale feature images of broken tooth surface inclusions; the feature merging module is used to correspondingly splice the feature matrix to be fused of the gray level co-occurrence matrix feature quantity of a single image with the three-scale feature images of the broken tooth surface inclusions obtained by the feature pyramid network to obtain the three-scale feature merging matrix; the detection head is used to detect the broken tooth surface inclusions in the image through the input three-scale feature merging matrix; The training and verification module is used to input the feature matrix to be fused of the gray-level co-occurrence matrix feature quantity of a single image obtained by the feature matrix calculation module to be fused and the model training sample data obtained by the training sample acquisition module into the broken tooth surface inclusion detection model constructed by the model construction module, train and verify the model, and obtain a verified qualified broken tooth surface inclusion detection model; The gear broken tooth detection module is used to process the broken tooth surface image to be detected using the model verified by the training and verification module to obtain the gear broken tooth detection result.

Claims

1. An adaptive broken tooth surface inclusion judgment method based on improved YOLOv4, characterized in that: The specific steps include: Step 1: Collect the image of the broken tooth surface of the gear; Step 2: converting the acquired gear broken tooth surface image into a grayscale image; Step 3: Adjust the grayscale image to a uniform size and perform data enhancement processing to obtain a training image data set; then calculate the grayscale co-occurrence matrix for each image in the training image data set to obtain a feature matrix to be fused of the grayscale co-occurrence matrix feature quantity of a single image; Step 4: For the training image dataset obtained in step 3, mark the center position and size of the inclusions on each image, and mark whether there are inclusions to obtain model training sample data; Step 5: Construct a broken tooth surface inclusion detection model, including a backbone feature extraction network, a feature pyramid network, a feature merging module and a detection head; wherein the backbone feature extraction network is used to extract the spatial features and semantic features of the input image to obtain three-scale broken tooth surface feature maps; the feature pyramid network is used to fuse the three-scale broken tooth surface feature maps output from the backbone feature extraction network, and output three-scale feature images of broken tooth surface inclusions; the feature merging module is used to correspondingly splice the feature matrix to be fused of the gray level co-occurrence matrix feature quantity of a single image with the three-scale feature images of the broken tooth surface inclusions obtained by the feature pyramid network to obtain a feature merging matrix of three scales; the detection head is used to detect the broken tooth surface inclusions in the image through the input feature merging matrix of three scales; Step 6: Input the to-be-fused feature matrix of the gray-level co-occurrence matrix feature quantity of the single image obtained in step 3 and the model training sample data obtained in step 4 into the broken tooth surface inclusion detection model constructed in step 5, train and verify the model, and obtain a verified qualified broken tooth surface inclusion detection model; Step seven, use the qualified model verified in step six to process the image of the broken tooth surface to be detected, and obtain the gear broken tooth detection result.

2. The adaptive broken tooth surface inclusion judgment method based on improved YOLOv4 as claimed in claim 1, characterized in that: Step 3 specifically includes the following sub-steps: Step 31, adjust the grayscale image into image data of uniform size; use data enhancement to expand the training data, and then perform normalization processing; Step 32, extract features from each image obtained in step 31 using a gray level co-occurrence matrix to obtain a gray level co-occurrence matrix, perform normalization processing, calculate the eigenvalues ​​of the direction matrix in four directions of 0°, 45°, 90°, and 135°, and use the average value as the eigenvalue co-occurrence matrix; Step 33, calculate the following five feature quantities of the grayscale co-occurrence matrix according to the eigenvalue co-occurrence matrix obtained in step 32: energy, contrast, homogeneity, entropy and inverse difference, and then expand each feature quantity of the single image into feature matrices of 52×52×1, 26×26×1, and 13×13×1 respectively, stack the matrices of these five feature quantities, and obtain the feature matrices to be fused of the grayscale co-occurrence matrix feature quantities of the single image: 52×52×5, 26×26×5, and 13×13×5.

3. The adaptive broken tooth surface inclusion judgment method based on improved YOLOv4 as claimed in claim 1, characterized in that: In step five: (1) The backbone feature extraction network includes a Focus layer and five residual modules connected in sequence; Focus layer: used to convert the input 416×416×1 image into a 208×208×32 feature map through Focus operation and 32 3×3 convolutions in sequence; Residual module: used to implement the following operations: first, a convolution with a kernel size of 3×3 and a stride of 2 is performed to divide the input feature map into two equal parts, and then each part enters a 1×1 convolution to obtain the output results of the two parts. The output results of one part are processed by the residual block and a 1×1 convolution in turn, and the processed results are stacked and merged with the output results of the other part; (2) The feature pyramid network is used to implement the following operations: ① Obtain high-resolution low-level feature images, intermediate feature images, and low-resolution high-level feature images output by the third residual module, the fourth residual module, and the fifth residual module of the backbone feature extraction network, respectively; ② Upsample half of the high-level feature image to obtain an upsampled data set, and perform five convolutions on the image obtained by combining the data set with the intermediate feature image to obtain an intermediate fused feature image; ③ Upsample half of the intermediate fused feature image to obtain an upsampled data set, and perform five convolutions on the image obtained by combining the data set with the low-level feature image to obtain a low-level fused feature image; ④ Output the low-level fused feature image to obtain a low-level fused feature image output set; At the same time, downsample the low-level fused feature image to obtain a downsampled data set, and stack and merge it with the intermediate fused feature image to obtain an image, and then perform five convolutions to obtain an intermediate fused feature image output set; ⑤ Stack and merge the image obtained by downsampling the intermediate fused feature image output set with the high-level feature image to obtain an image, and then perform five convolutions on it to obtain a high-level fused feature image output set; (3) Feature merging module: used to concatenate the low-level fused feature image output set, the intermediate-level fused feature image output set, and the high-level fused feature image output set output by the feature pyramid network with the feature matrices to be fused of the gray-level co-occurrence matrix feature quantities of the single images of 52×52×5, 26×26×5, and 13×13×5 obtained in step 3, one by one, to obtain a feature merging matrix after feature merging; (4) Detection head: It is used to merge the input feature matrices of three scales (13×13×517, 26×26×261, and 52×52×133) and compress the number of channels through multiple convolutions to obtain the multi-scale feature maps of YOLO Head: 13×13×21, 26×26×21, and 52×52×21.

4. The adaptive broken tooth surface inclusion judgment method based on improved YOLOv4 as claimed in claim 1, characterized in that: Step 7 includes the following process: (1) Preprocessing the image of the broken tooth surface to be detected; (2) Inputting the preprocessed image into the qualified broken tooth surface inclusion detection model obtained in step 6 for detection; (3) Output the gear broken tooth detection results. If there are inclusions, the detection results include the location and size of the inclusions.

5. The adaptive broken tooth surface inclusion judgment method based on improved YOLOv4 as claimed in claim 4 is characterized in that: The method further includes step eight: evaluating the gear tooth breakage detection result outputted from step seven to form an evaluation report of the gear tooth breakage surface.

6. The adaptive broken tooth surface inclusion judgment method based on improved YOLOv4 as claimed in claim 5, characterized in that: It also includes step nine: using a storage server to store the collected broken tooth surface images, preprocessed images, model iteration training results, broken tooth surface inclusion detection models, gear broken tooth detection results and evaluation reports.

7. The adaptive broken tooth surface inclusion judgment method based on improved YOLOv4 as claimed in claim 6, characterized in that: The method further includes step ten: displaying the content stored in step nine.

8. An adaptive broken tooth surface inclusion judgment system based on improved YOLOv4, characterized in that: The modules include: An image acquisition module is used to acquire images of broken gear tooth surfaces; A grayscale image conversion module is used to convert the acquired gear broken tooth surface image into a grayscale image; Feature matrix calculation module to be fused: used to adjust the grayscale image to a uniform size and perform data enhancement processing to obtain a training image data set; Then, each image in the training image data set is calculated through the gray level co-occurrence matrix to obtain the feature matrix to be fused of the gray level co-occurrence matrix feature quantity of a single image; Training sample acquisition module: used to implement the following process: for the training image data set obtained by the feature matrix calculation module to be fused, mark the center position and size of the inclusions on each image, and mark whether there are inclusions, to obtain model training sample data; Model construction module: used to construct a broken tooth surface inclusion detection model, including a backbone feature extraction network, a feature pyramid network, a feature merging module and a detection head; wherein the backbone feature extraction network is used to extract the spatial features and semantic features of the input image to obtain three-scale broken tooth surface feature maps; the feature pyramid network is used to fuse the three-scale broken tooth surface feature maps output from the backbone feature extraction network, and output three-scale feature images of broken tooth surface inclusions; the feature merging module is used to correspondingly splice the feature matrix to be fused of the gray level co-occurrence matrix feature quantity of a single image with the three-scale feature images of the broken tooth surface inclusions obtained by the feature pyramid network to obtain the three-scale feature merging matrix; the detection head is used to detect the broken tooth surface inclusions in the image through the input three-scale feature merging matrix; The training and verification module is used to input the to-be-fused feature matrix of the gray-level co-occurrence matrix feature quantity of a single image obtained by the to-be-fused feature matrix calculation module and the model training sample data obtained by the training sample acquisition module into the broken tooth surface inclusion detection model constructed by the model construction module, train and verify the model, and obtain a verified qualified broken tooth surface inclusion detection model; The gear broken tooth detection module is used to process the broken tooth surface image to be detected using the model verified by the training and verification module to obtain the gear broken tooth detection result.

Images