Control box assembly defect detection method and device based on YOLOv4 neural network and medium
The control box assembly defect detection method based on YOLOv4 neural network solves the problems of low detection accuracy and low efficiency in the existing technology, and realizes fast and efficient control box assembly defect detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN INST OF TECH
- Filing Date
- 2022-06-27
- Publication Date
- 2026-06-02
AI Technical Summary
Existing methods for detecting assembly defects in control boxes are inaccurate and inefficient, failing to meet the needs of intelligent industry upgrades.
A control box assembly defect detection method based on YOLOv4 neural network is adopted. By taking pictures of the components inside the control box, performing data augmentation processing and information annotation, an initial convolutional neural network detection model is constructed. The final convolutional neural network detection model is obtained through training and testing, which is used to detect assembly defects in the control box.
It enables rapid and efficient detection of assembly defects in control boxes under complex environments, improving the accuracy and efficiency of the detection.
Smart Images
Figure CN115330997B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of control box assembly defect detection technology, specifically to a control box assembly defect detection method, device and storage medium based on YOLOv4 neural network. Background Technology
[0002] Currently, with the continuous development of my country's manufacturing industry, intelligent assembly systems are gradually being applied. The detection and identification of assembly defects in control box parts is a crucial aspect of intelligent assembly systems, involving research fields such as computer vision, deep learning, and image recognition. Currently, traditional assembly systems rely on repetitive manual labor, which is prone to errors due to human fatigue and limited human visual resolution. Intelligent systems are not yet widely used, and there are few application-level methods for detecting specific targets in industrial parts. Therefore, to avoid wasting manpower and time and adapting to unstable factors such as lighting in factory environments, intelligent control box assembly defect detection that ensures both accuracy and speed is imperative. However, existing control box assembly defect detection methods suffer from low accuracy and low efficiency, failing to meet the practical needs of intelligent industry upgrading. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a method, device and storage medium for detecting assembly defects of control boxes based on YOLOv4 neural network, which addresses the shortcomings of the prior art.
[0004] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A method for detecting assembly defects in control boxes based on YOLOv4 neural networks, comprising the following steps:
[0005] Multiple images of various types of components inside the control box were captured.
[0006] Data augmentation processing was performed on each component image to obtain multiple images of industrial parts;
[0007] Each industrial part image is labeled with information, and the labeled industrial part images are divided into a sample training set and a sample test set according to a preset ratio;
[0008] An initial convolutional neural network detection model is constructed based on the YOLOv4 neural network. The initial convolutional neural network detection model is then trained using the sample training set to obtain a trained convolutional neural network detection model.
[0009] The trained convolutional neural network detection model is tested using the sample test set to obtain the final convolutional neural network detection model.
[0010] Images are taken of multiple categories of components in the correctly assembled control box, and standard parameters are obtained from the component images;
[0011] Images of multiple categories of components inside the control box to be inspected are captured to obtain images of the components to be inspected;
[0012] The image of the component to be inspected is input into the final convolutional neural network detection model, which outputs the detection parameters of the component image. The detection parameters are verified by the standard parameters to obtain the detection result of whether the control box to be inspected has assembly defects.
[0013] The beneficial effects of this invention are as follows: an initial convolutional neural network detection model is constructed based on the YOLOv4 neural network, and the initial convolutional neural network detection model is trained using industrial part images after data augmentation to obtain the final convolutional neural network detection model. The detection parameters output by the convolutional neural network detection model are verified by the correct standard parameters to obtain the detection result of whether the control box under test has assembly defects. This invention can quickly and efficiently complete the task of detecting assembly defects in control boxes in complex environments.
[0014] Based on the above technical solution, the present invention can be further improved as follows.
[0015] Furthermore, the process of capturing images of multiple categories of components within the control box to obtain multiple component images specifically involves:
[0016] Images of multiple types of components inside the control box were captured by taking pictures of them under different lighting conditions, angles, and positions, resulting in multiple images of the components.
[0017] Furthermore, the data augmentation processing of each component image is performed to obtain multiple industrial component images, specifically as follows:
[0018] The images of each component are mirrored and flipped. The original component images and their mirrored counterparts are then flipped vertically, horizontally, and diagonally to obtain multiple images of industrial parts.
[0019] Furthermore, the process of annotating each industrial part image specifically involves:
[0020] Each industrial part image is named according to its serial number, and each named industrial part image is labeled with the four positional information (x, y, w, and h) and category information of the label box. Each labeled industrial part image is used as a sample image.
[0021] The beneficial effects of adopting the above-mentioned further technical solution are: it can obtain sample images for training the initial convolutional neural network detection model, and the sample images are relatively rich and comprehensive, which can improve the accuracy of training the initial convolutional neural network detection model.
[0022] Furthermore, the step of training the initial convolutional neural network detection model using the sample training set specifically involves:
[0023] The initial convolutional neural network detection model divides each industrial part image in the input sample training set into an S×S grid.
[0024] Boundary boxes are defined by fixing prior boxes through dimensional clustering. The initial convolutional neural network predicts four positional information (x, y, w, and h) and a confidence score for the boundary boxes, resulting in predicted boxes. The confidence score is:
[0025]
[0026] Among them, P r (object) represents the probability of containing an object, with a value of 1 or 0. The intersection-union ratio (IUGR) represents the ratio of the actual values of the labeled boxes to the predicted values. A value of 1 indicates that the labeled boxes and predicted boxes coincide, while a value of 0 indicates that the labeled boxes and predicted boxes do not coincide.
[0027] The confidence level is scored using a scoring formula, which is as follows:
[0028]
[0029] Among them, P r (class) represents the probability that the grid belongs to a certain class, P r (class|object) represents the C class probabilities of the grid prediction containing the target;
[0030] Based on the confidence scores of the probabilities of C categories, a non-maximum suppression algorithm is used to identify each type of target.
[0031] The prediction boxes are set to multiple different sizes, and feature extraction training is performed on the prediction boxes of each size through a residual network, thereby completing the training of the initial convolutional neural network detection model.
[0032] The beneficial effect of adopting the above-mentioned further technical solution is that a convolutional neural network detection model with high detection accuracy can be trained by setting the confidence level of the prediction box.
[0033] Furthermore, the process of taking images of multiple categories of components within the correctly assembled control box and obtaining standard parameters from the component images specifically involves:
[0034] Take an image above the center of the correctly assembled control box according to the set resolution and preset distance;
[0035] Information is labeled from multiple categories of components in the correctly assembled control box. The labeling includes location information and category information, and the labeled location information and category information are used as standard parameters.
[0036] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A control box assembly defect detection device based on YOLOv4 neural network, comprising:
[0037] The imaging module is used to capture images of multiple types of components inside the control box, obtaining multiple images of the components.
[0038] The preprocessing module is used to perform data augmentation processing on each component image to obtain multiple industrial component images;
[0039] Each industrial part image is labeled with information, and the labeled industrial part images are divided into a sample training set and a sample test set according to a preset ratio;
[0040] The training module is used to construct an initial convolutional neural network detection model based on the YOLOv4 neural network, and to train the initial convolutional neural network detection model using the sample training set to obtain a trained convolutional neural network detection model.
[0041] The testing module is used to test the trained convolutional neural network detection model using the sample test set to obtain the final convolutional neural network detection model.
[0042] The detection module is used to call the imaging module to take images of multiple categories of parts in the correctly assembled control box and obtain standard parameters from the part images;
[0043] Images of multiple categories of components inside the control box to be inspected are captured to obtain images of the components to be inspected;
[0044] The image of the component to be inspected is input into the final convolutional neural network detection model, which outputs the detection parameters of the component image. The detection parameters are verified by the standard parameters to obtain the detection result of whether the control box to be inspected has assembly defects.
[0045] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a control box assembly defect detection device based on YOLOv4 neural network, including a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the control box assembly defect detection method based on YOLOv4 neural network as described above is implemented.
[0046] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the control box assembly defect detection method based on YOLOv4 neural network as described above. Attached Figure Description
[0047] Figure 1 A schematic flowchart of the control box assembly defect detection method provided in an embodiment of the present invention;
[0048] Figure 2 This is a functional block diagram of the control box assembly defect detection device provided in an embodiment of the present invention. Detailed Implementation
[0049] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.
[0050] Example 1:
[0051] like Figure 1 As shown, a method for detecting assembly defects in a control box based on a YOLOv4 neural network includes the following steps:
[0052] Multiple images of various types of components inside the control box were captured.
[0053] Data augmentation processing was performed on each component image to obtain multiple images of industrial parts;
[0054] Each industrial part image is labeled with information, and the labeled industrial part images are divided into a sample training set and a sample test set according to a preset ratio;
[0055] An initial convolutional neural network detection model is constructed based on the YOLOv4 neural network. The initial convolutional neural network detection model is then trained using the sample training set to obtain a trained convolutional neural network detection model.
[0056] The trained convolutional neural network detection model is tested using the sample test set to obtain the final convolutional neural network detection model.
[0057] Images are taken of multiple categories of components in the correctly assembled control box, and standard parameters are obtained from the component images;
[0058] Images of multiple categories of components inside the control box to be inspected are captured to obtain images of the components to be inspected;
[0059] The image of the component to be inspected is input into the final convolutional neural network detection model, which outputs the detection parameters of the component image. The detection parameters are verified by the standard parameters to obtain the detection result of whether the control box to be inspected has assembly defects.
[0060] In the above embodiments, an initial convolutional neural network detection model is constructed based on the YOLOv4 neural network, and the initial convolutional neural network detection model is trained using data-augmented images of industrial parts to obtain the final convolutional neural network detection model. The detection parameters output by the convolutional neural network detection model are verified using correct standard parameters to obtain the detection result of whether the control box to be detected has assembly defects. The present invention can quickly and efficiently complete the task of detecting assembly defects in control boxes in complex environments.
[0061] Preferably, the step of taking images of multiple categories of components inside the control box to obtain multiple component images specifically involves:
[0062] Images of multiple types of components inside the control box were captured by taking pictures of them under different lighting conditions, angles, and positions, resulting in multiple images of the components.
[0063] The selection of shooting angle and orientation should take into account various situations. The captured images should reflect the two-dimensional image information of each part as much as possible. The images should be taken under different lighting conditions to obtain multiple images of the parts, thus constructing the basic dataset.
[0064] Preferably, the step of performing data augmentation processing on each component image to obtain multiple industrial component images specifically involves:
[0065] The images of each component are mirrored and flipped. The original component images and their mirrored counterparts are then flipped vertically, horizontally, and diagonally to obtain multiple images of industrial parts.
[0066] Preferably, the step of annotating each industrial part image separately involves:
[0067] Each industrial part image is named according to its serial number, and each named industrial part image is labeled with the four positional information (x, y, w, and h) and category information of the label box. Each labeled industrial part image is used as a sample image.
[0068] In the above embodiments, sample images for training the initial convolutional neural network detection model can be obtained, and the sample images are relatively rich and comprehensive, which can improve the accuracy of training the initial convolutional neural network detection model.
[0069] Preferably, training the initial convolutional neural network detection model using the sample training set specifically involves:
[0070] The initial convolutional neural network detection model divides each industrial part image in the input sample training set into an S×S grid.
[0071] Boundary boxes are defined by fixing prior boxes through dimensional clustering. The initial convolutional neural network predicts four positional information (x, y, w, and h) and a confidence score for the boundary boxes, resulting in predicted boxes. The confidence score is:
[0072]
[0073] Among them, P r (object) represents the probability of containing an object, with a value of 1 or 0. The intersection-union ratio (IUGR) represents the ratio of the actual values of the labeled boxes to the predicted values. A value of 1 indicates that the labeled boxes and predicted boxes coincide, while a value of 0 indicates that the labeled boxes and predicted boxes do not coincide.
[0074] The confidence level is scored using a scoring formula, which is as follows:
[0075]
[0076] Among them, P r (class) represents the probability that the grid belongs to a certain class, P r (class|object) represents the C class probabilities of the grid prediction containing the target;
[0077] Based on the confidence scores of the probabilities of C categories, a non-maximum suppression algorithm is used to identify each type of target.
[0078] The prediction boxes are set to multiple different sizes, and feature extraction training is performed on the prediction boxes of each size through a residual network, thereby completing the training of the initial convolutional neural network detection model.
[0079] To understand, the network takes a single-frame sample image of an industrial part as input and divides it into an S×S grid. Then, using dimensional clustering to fix the prior boxes, bounding boxes are selected. The neural network predicts four location parameters (x, y, w, h) and a confidence score for each bounding box (i.e., the label box), extracting features from three different bounding boxes to form a pyramid network. Finally, a residual network is used to extract features, generating a trained convolutional neural network detection model for the control box part. x and y are the coordinates of the center point of the label box, and w and h are the width and height of the label box.
[0080] In the above embodiments, a convolutional neural network detection model with high detection accuracy can be trained by setting the confidence level of the prediction box.
[0081] Preferably, the step of taking images of multiple categories of components within the correctly assembled control box and obtaining standard parameters from the component images specifically involves:
[0082] Take an image above the center of the correctly assembled control box according to the set resolution and preset distance;
[0083] Information is labeled from multiple categories of components in the correctly assembled control box. The labeling includes location information and category information, and the labeled location information and category information are used as standard parameters.
[0084] Specifically, an industrial camera with a resolution of 4000*4000 was used to capture images at a fixed height of 50 cm above the center of the correctly assembled control box. Then, the pixel position information of each type of part in the captured images was labeled. Finally, the generated pixel position information serves as standard information for subsequent assembly defect verification.
[0085] Finally, an industrial camera with a resolution of 4000*4000 was used to capture images at a fixed height of 50 cm above the center of the control box to be inspected.
[0086] The generated image is loaded into a convolutional neural network detection model for detecting control box parts. The detection outputs the type of each part and the pixel position of each part in the image.
[0087] The output information is compared with the acquired standard parameters (i.e., the marked position information and category information). If there are missing parts or large differences in the pixel position of parts, it can be determined that there is an assembly defect in the control box.
[0088] Example 2:
[0089] like Figure 2 As shown, a control box assembly defect detection device based on YOLOv4 neural network includes:
[0090] The imaging module is used to capture images of multiple types of components inside the control box, obtaining multiple images of the components.
[0091] The preprocessing module is used to perform data augmentation processing on each component image to obtain multiple industrial component images;
[0092] Each industrial part image is labeled with information, and the labeled industrial part images are divided into a sample training set and a sample test set according to a preset ratio;
[0093] The training module is used to construct an initial convolutional neural network detection model based on the YOLOv4 neural network, and to train the initial convolutional neural network detection model using the sample training set to obtain a trained convolutional neural network detection model.
[0094] The testing module is used to test the trained convolutional neural network detection model using the sample test set to obtain the final convolutional neural network detection model.
[0095] The detection module is used to call the imaging module to take images of multiple categories of parts in the correctly assembled control box and obtain standard parameters from the part images;
[0096] Images of multiple categories of components inside the control box to be inspected are captured to obtain images of the components to be inspected;
[0097] The image of the component to be inspected is input into the final convolutional neural network detection model, which outputs the detection parameters of the component image. The detection parameters are verified by the standard parameters to obtain the detection result of whether the control box to be inspected has assembly defects.
[0098] Preferably, in the preprocessing module, information annotation is performed on each industrial part image, specifically as follows:
[0099] Each industrial part image is named according to its serial number, and each named industrial part image is labeled with the four positional information (x, y, w, and h) and category information of the label box. Each labeled industrial part image is used as a sample image.
[0100] Example 3:
[0101] A control box assembly defect detection device based on YOLOv4 neural network includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the control box assembly defect detection method based on YOLOv4 neural network as described above.
[0102] Example 4:
[0103] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the control box assembly defect detection method based on the YOLOv4 neural network as described above.
[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for detecting assembly defects in a control box based on a YOLOv4 neural network, characterized in that, Includes the following steps: Multiple images of various types of components inside the control box were captured. Data augmentation processing was performed on each component image to obtain multiple images of industrial parts; Each industrial part image is labeled with information, and the labeled industrial part images are divided into a sample training set and a sample test set according to a preset ratio; An initial convolutional neural network detection model is constructed based on the YOLOv4 neural network. The initial convolutional neural network detection model is then trained using the sample training set to obtain a trained convolutional neural network detection model. The trained convolutional neural network detection model is tested using the sample test set to obtain the final convolutional neural network detection model. Images are taken of multiple categories of components in the correctly assembled control box, and standard parameters are obtained from the component images; Images of multiple categories of components inside the control box to be inspected are captured to obtain images of the components to be inspected; The image of the component to be detected is input into the final convolutional neural network detection model, which outputs the detection parameters of the component to be detected. The detection parameters are verified by the standard parameters to obtain the detection result of whether the control box to be detected has assembly defects. The process of annotating each industrial part image separately involves the following steps: Each industrial part image is named according to its serial number, and each named industrial part image is labeled with the four positional information of x, y, w and h of the label box and the category information. Each labeled industrial part image is used as a sample image. The training of the initial convolutional neural network detection model using the sample training set specifically involves: The initial convolutional neural network detection model divides each industrial part image in the input sample training set into an S×S grid. Boundary boxes are defined by fixing prior boxes through dimensional clustering. The initial convolutional neural network predicts four positional information (x, y, w, and h) and a confidence score for the boundary boxes, resulting in predicted boxes. The confidence score is: ， in, Indicates the probability of the included target, with a value of 1 or 0. The intersection-union ratio (IUGR) represents the ratio of the actual values of the labeled boxes to the predicted values. A value of 1 indicates that the labeled boxes and predicted boxes coincide, while a value of 0 indicates that the labeled boxes and predicted boxes do not coincide. The confidence level is scored using a scoring formula, which is as follows: ， in, This represents the probability that the grid belongs to one of the categories. This represents the C class probabilities of the grid prediction containing the target; Based on the confidence scores of the probabilities of C categories, a non-maximum suppression algorithm is used to identify each type of target. The prediction boxes are set to multiple different sizes, and feature extraction training is performed on the prediction boxes of each size through a residual network, thereby completing the training of the initial convolutional neural network detection model.
2. The method for detecting assembly defects in a control box according to claim 1, characterized in that, The process of capturing images of multiple categories of components inside the control box to obtain multiple component images is as follows: Images of multiple types of components inside the control box were captured by taking pictures of them under different lighting conditions, angles, and positions, resulting in multiple images of the components.
3. The method for detecting assembly defects in a control box according to claim 1, characterized in that, The process of performing data augmentation on each component image to obtain multiple industrial component images is as follows: The images of each component are mirrored and flipped. The original component images and their mirrored counterparts are then flipped vertically, horizontally, and diagonally to obtain multiple images of industrial parts.
4. The method for detecting assembly defects in a control box according to claim 1, characterized in that, The process of taking images of multiple categories of components within the correctly assembled control box and obtaining standard parameters from the component images specifically involves: Take an image above the center of the correctly assembled control box according to the set resolution and preset distance; Information is labeled from multiple categories of components in the correctly assembled control box. The labeling includes location information and category information, and the labeled location information and category information are used as standard parameters.
5. A control box assembly defect detection device based on YOLOv4 neural network, comprising: The imaging module is used to capture images of multiple types of components inside the control box, obtaining multiple images of the components. The preprocessing module is used to perform data augmentation processing on each component image to obtain multiple industrial component images; Each industrial part image is labeled with information, and the labeled industrial part images are divided into a sample training set and a sample test set according to a preset ratio; The training module is used to construct an initial convolutional neural network detection model based on the YOLOv4 neural network, and to train the initial convolutional neural network detection model using the sample training set to obtain a trained convolutional neural network detection model. The testing module is used to test the trained convolutional neural network detection model using the sample test set to obtain the final convolutional neural network detection model. The detection module is used to call the imaging module to take images of multiple categories of parts in the correctly assembled control box and obtain standard parameters from the part images; Images of multiple categories of components inside the control box to be inspected are captured to obtain images of the components to be inspected; The image of the component to be detected is input into the final convolutional neural network detection model, which outputs the detection parameters of the component to be detected. The detection parameters are verified by the standard parameters to obtain the detection result of whether the control box to be detected has assembly defects. The process of annotating each industrial part image separately involves the following steps: Each industrial part image is named according to its serial number, and each named industrial part image is labeled with the four positional information of x, y, w and h of the label box and the category information. Each labeled industrial part image is used as a sample image. The training of the initial convolutional neural network detection model using the sample training set specifically involves: The initial convolutional neural network detection model divides each industrial part image in the input sample training set into an S×S grid. Boundary boxes are defined by fixing prior boxes through dimensional clustering. The initial convolutional neural network predicts four positional information (x, y, w, and h) and a confidence score for the boundary boxes, resulting in predicted boxes. The confidence score is: ， in, Indicates the probability of the included target, with a value of 1 or 0. The intersection-union ratio (IUGR) represents the ratio of the actual values of the labeled boxes to the predicted values. A value of 1 indicates that the labeled boxes and predicted boxes coincide, while a value of 0 indicates that the labeled boxes and predicted boxes do not coincide. The confidence level is scored using a scoring formula, which is as follows: ， in, This represents the probability that the grid belongs to one of the categories. This represents the C class probabilities of the grid prediction containing the target; Based on the confidence scores of the probabilities of C categories, a non-maximum suppression algorithm is used to identify each type of target. The prediction boxes are set to multiple different sizes, and feature extraction training is performed on the prediction boxes of each size through a residual network, thereby completing the training of the initial convolutional neural network detection model.
6. A control box assembly defect detection device based on YOLOv4 neural network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the control box assembly defect detection method based on YOLOv4 neural network as described in any one of claims 1 to 4.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the control box assembly defect detection method based on the YOLOv4 neural network as described in any one of claims 1 to 4.