Mobile terminal fastener classified counting method and system based on improved YOLOv5

Abstract

The invention relates to the technical field of image processing, and provides an improved YOLOv5-based mobile terminal fastener classified counting method and system, and the method comprises the steps: constructing a fastener data set; constructing an original YOLOv5 model, and obtaining a plurality of different improved YOLOv5 models according to the original YOLOv5 model; the original YOLOv5 model and the multiple different improved YOLOv5 models are trained through the fastener data set; evaluating the trained original YOLOv5 model and the improved YOLOv5 model through a plurality of performance indexes, and obtaining the improved YOLOv5 model with the optimal comprehensive performance; based on the improved YOLOv5 model with the optimal comprehensive performance, deployment is carried out on a mobile terminal to carry out visual image detection on the to-be-detected fasteners, and classified counting of the to-be-detected fasteners is achieved. According to the method, photo album photo detection, photo taking and real-time detection can be realized through the camera of the mobile terminal, the fasteners to be detected are counted in a classified manner, and the problem that the fasteners such as bolts, nuts and gaskets cannot be counted in a classified manner in a weighing type counting method is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a mobile terminal fastener classification and counting method and system based on improved YOLOv5. Background Art

[0002] Commonly used fasteners such as bolts, nuts, washers, bolts, nuts and washers, etc., as indispensable industrial parts in daily life, their production in my country is already very large. With the expansion of China's fastener market, the production and export volume of bolts, nuts and washers continues to increase.

[0003] At present, in the packaging production lines of most bolt, nut and washer manufacturers, the weighing counting method is mainly used to calculate the quantity, but it is not possible to classify and count fasteners such as bolts, nuts and washers. Therefore, it is necessary to develop a method that can classify and count fasteners such as bolts, nuts and washers. Summary of the invention

[0004] Based on this, in order to solve the problem that the weighing counting method cannot classify and count fasteners such as bolts, nuts and washers, the present invention provides a mobile terminal fastener classification counting method and system based on improved YOLOv5, and its specific technical solution is as follows:

[0005] A mobile terminal fastener classification and counting method based on improved YOLOv5 comprises the following steps:

[0006] S1, build fastener dataset;

[0007] S2, constructing an original YOLOv5 model, and obtaining a plurality of different improved YOLOv5 models according to the original YOLOv5 model;

[0008] S3, training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset;

[0009] S4, evaluating the trained original YOLOv5 model and the improved YOLOv5 model through multiple performance indicators to obtain the improved YOLOv5 model with the best comprehensive performance;

[0010] S5, based on the improved YOLOv5 model with the best comprehensive performance, it is deployed on the mobile terminal to perform visual image detection on the fasteners to be detected, so as to realize classification and counting of the fasteners to be detected.

[0011] The mobile terminal fastener classification and counting method based on improved YOLOv5 utilizes the improved YOLOv5 model with the best comprehensive performance to be deployed on the mobile terminal to perform visual image detection on the fasteners to be detected. The method is based on visual image detection technology and can realize album photo detection, photo taking and real-time detection through the camera of the mobile terminal. The classification and counting of the fasteners to be detected solves the problem that the weighing counting method cannot classify and count fasteners such as bolts, nuts and washers.

[0012] Preferably, in step S1, the specific method of constructing the fastener data set includes the following steps:

[0013] S10, collecting images of fasteners of different types and sizes;

[0014] S11, performing data enhancement on the collected fastener image to increase the fastener data set;

[0015] S12, annotating the fastener image after data enhancement;

[0016] S13, dividing the annotated fastener images into a training set, a verification set and a test set according to a preset ratio.

[0017] Preferably, in step S2, an original YOLOv5 model is constructed, and a specific method for obtaining a plurality of different improved YOLOv5 models according to the original YOLOv5 model comprises the following steps:

[0018] S20, improving the structure of the original YOLOv5 model to obtain a YOLOv5_Mobile model;

[0019] S21, improving the structure of the original YOLOv5 model to obtain a YOLOv5_MobileNetV2 model;

[0020] S22, improving the structure of the original YOLOv5 model to obtain a YOLOv5_MobileNetV4 model;

[0021] Among them, multiple different improved YOLOv5 models include YOLOv5_MobileVi t model, YOLOv5_MobileNetV2 model and YOLOv5_MobileNetV4 model.

[0022] Preferably, the specific method of training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset comprises the following steps:

[0023] The original YOLOv5 model and a plurality of different improved YOLOv5 models are trained using the training set.

[0024] Preferably, in step S4, the trained original YOLOv5 model and the improved YOLOv5 model are evaluated by multiple performance indicators, and a specific method for obtaining the improved YOLOv5 model with the best comprehensive performance comprises the following steps:

[0025] S40, obtaining the original YOLOv5 model and generating a confusion matrix and a PR curve during training of the improved YOLOv5 model;

[0026] S41, obtaining precision, recall, average precision, frames per second, GFLOPs, and model parameter quantity according to the confusion matrix and the PR curve;

[0027] S42, evaluating the trained original YOLOv5 model and the improved YOLOv5 model according to the precision, the recall, the average precision mean, the frames per second, the GFLOPs, and the model parameter quantity, to obtain the improved YOLOv5 model with the best overall performance.

[0028] Preferably, in step S5, based on the improved YOLOv5 model with the best comprehensive performance, the model is deployed on the mobile terminal to perform visual image detection on the fasteners to be detected, and the specific method for implementing the classification and counting of the fasteners to be detected includes the following steps:

[0029] S50, performing file format conversion and optimization on the improved YOLOv5 model with the best comprehensive performance;

[0030] S51, deploying the improved YOLOv5 model with the best comprehensive performance after file format conversion and optimization on a mobile terminal;

[0031] S52, generating an APP application installation package and installing it in the mobile terminal;

[0032] S53, performing visual image detection on the fasteners to be detected through the APP in the mobile terminal to achieve classification and counting of the fasteners to be detected.

[0033] A mobile terminal fastener classification and counting system based on improved YOLOv5, comprising:

[0034] A first building module is used to build a fastener dataset;

[0035] A second construction module is used to construct an original YOLOv5 model, and obtain a plurality of different improved YOLOv5 models according to the original YOLOv5 model;

[0036] A training module, used for training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset;

[0037] A performance evaluation module is used to evaluate the trained original YOLOv5 model and the improved YOLOv5 model through multiple performance indicators to obtain an improved YOLOv5 model with the best comprehensive performance;

[0038] A detection module is deployed, which is used to perform visual image detection on the fasteners to be detected based on the improved YOLOv5 model with the best comprehensive performance, and realize classification and counting of the fasteners to be detected.

[0039] Preferably, the multiple different improved YOLOv5 models include YOLOv5_MobileNetVit model, YOLOv5_MobileNetV2 model and YOLOv5_MobileNetV4 model.

[0040] Preferably, the fasteners include bolts, nuts and washers, and the fastener data set includes a training set, a validation set and a test set.

[0041] Preferably, the mobile terminal is an Android terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but the emphasis is placed on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0043] Figure 1 It is a schematic diagram of the overall process of an Android-side fastener classification and counting method based on improved YOLOv5 in one embodiment of the present invention;

[0044] Figure 2 is a flowchart of a specific method for constructing a fastener data set in one embodiment of the present invention;

[0045] Figure 3 is a flowchart of a specific method for obtaining multiple different improved YOLOv5 models in one embodiment of the present invention;

[0046] Figure 4 It is a flowchart of a specific method for obtaining an improved YOLOv5 model with optimal comprehensive performance in one embodiment of the present invention;

[0047] Figure 5 It is a flowchart of a specific method for implementing classification and counting of fasteners to be inspected in one embodiment of the present invention;

[0048] Figure 6 It is a schematic diagram of the overall process of an Android-side fastener classification and counting method based on improved YOLOv5 in another embodiment of the present invention;

[0049] Figure 7 is a schematic diagram of the original image of a bolt, a nut and a gasket in one embodiment of the present invention;

[0050] Figure 8 is a schematic diagram of the original images of bolts, nuts and washers after translation transformation in one embodiment of the present invention;

[0051] Fig. 9 is a schematic diagram of the original image of the bolt, nut and gasket after mirror transformation in one embodiment of the present invention;

[0052] Fig.10 is a schematic diagram of the original images of bolts, nuts and washers after affine transformation in one embodiment of the present invention;

[0053] Fig.11 1 is a schematic diagram of the improved YOLOv5_Mobile network structure in one embodiment of the present invention;

[0054] Fig.12 Schematic diagram of the structure of ConvBNReLU and InvertedRes idual network modules in one embodiment of the present invention

[0055] Fig.13 This is a schematic diagram of the improved YOLOv5_MobileNetV2 network structure in one embodiment of the present invention.

[0056] Fig.14 Schematic diagram of the improved YOLOv5_MobileNetV4 network structure in one embodiment of the present invention

[0057] Fig.15 It is a schematic diagram of the final classification and counting effect of bolts, nuts and washers based on the YOLOv5_MobileNetV2 model in one embodiment of the present invention;

[0058] Fig.16 It is a schematic diagram of the final classification and counting effect of bolts, nuts and washers based on the original YOLOv5 model in one embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with its embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0060] The “first” and “second” mentioned in the present invention do not represent specific quantities and orders, but are merely used to distinguish names.

[0061] Before describing the mobile terminal fastener classification and counting method and system based on improved YOLOv5 according to the present invention, a brief description of the prior art is first given.

[0062] Commonly used fasteners such as bolts, nuts, washers, etc., as indispensable industrial parts in daily life, have a huge output in my country. With the expansion of China's fastener market, the output and export volume of bolts, nuts and washers are increasing. Therefore, in the production process of bolts, nuts and washers, it is of great significance to classify and count the number of bolts, nuts and washers in real time.

[0063] Existing fastener counting methods such as manual counting, photoelectric counting and weighing counting have problems with efficiency and accuracy. Manual detection counting, as the most traditional detection method, has been widely used in this field, but it requires a lot of manpower and time costs. Photoelectric counting is easily affected by the intensity and color of ambient light. If the ambient light is too strong, it may cause the photoelectric sensor to misjudge and produce incorrect counting results. For example, in an environment with direct sunlight or in a workplace with strong light, the accuracy of the photoelectric counter may be affected. The weighing counting method cannot realize the classified counting of fasteners such as bolts, nuts and washers. It only calculates the number based on weight and cannot distinguish between objects such as color, shape, quality defects, etc. For example, in a pile of objects mixed with qualified and unqualified products, the weighing counting cannot separate the bolts, nuts and washers and can only calculate the total number.

[0064] Computer vision has developed rapidly in recent years and has become one of the important means to assist smart factories. Based on computer vision, the classification and counting of fasteners such as bolts, nuts and washers through target detection technology is an effective way to realize automatic counting of fasteners. Machine vision has been widely used in many fields, including agriculture, biological research, industry, etc., and in the field of target counting, crowd counting, cell counting, and vehicle counting are the main research directions. In the classification and counting of fasteners such as bolts, nuts and washers, machine vision faces difficulties such as image acquisition in motion, overlapping recognition, complex background interference, and the non-portability of the counting system. In addition, the machine vision counting system involves industrial cameras, lighting systems, high-performance computers or image processing hardware, and complex software licenses. The initial investment cost is high, and the maintenance and upgrade of the system also requires certain expenses, including software updates and hardware repairs.

[0065] At present, in the packaging production lines of most bolt, nut and gasket manufacturers, the weighing counting method is mainly used to calculate the quantity, but it is unable to classify and count fasteners such as bolts, nuts and gaskets. In addition, the weighing counting method is easily affected by oil pollution, has poor accuracy, low automation, and is prone to leakage. The production and packaging of fasteners such as bolts, nuts and gaskets urgently need to develop a high-precision and highly automated fastener counting method.

[0066] In order to solve the problem that the weighing counting method cannot classify and count fasteners such as bolts, nuts and washers, the present invention provides a mobile terminal fastener classification counting method based on improved YOLOv5, such as Figure 1 As shown, it includes the following steps:

[0067] S1, build fastener dataset.

[0068] Preferably, in step S1, Figure 2 As shown in the figure, the specific method of constructing the fastener dataset includes:

[0069] S10, collecting fastener images of different types and sizes.

[0070] The fasteners include but are not limited to bolts, nuts and washers.

[0071] S11, performing data enhancement on the collected fastener images to increase the fastener dataset.

[0072] The data enhancement is performed through the computer vision library Open CV, and the data enhancement includes but is not limited to brightness enhancement, contrast enhancement, rotation transformation, flip transformation, affine transformation, shear transformation, HSV data enhancement, and translation transformation.

[0073] S12, labeling the fastener image after data enhancement.

[0074] S13, dividing the annotated fastener images into a training set, a validation set, and a test set according to a preset ratio to facilitate model training.

[0075] S2, constructing an original YOLOv5 model, and obtaining a plurality of different improved YOLOv5 models according to the original YOLOv5 model, such as Figure 3 As shown, the specific steps include:

[0076] S20, improving the structure of the original YOLOv5 model to obtain a YOLOv5_Mobile model.

[0077] S21, improve the structure of the original YOLOv5 model to obtain a YOLOv5_MobileNetV2 model.

[0078] S22, improve the structure of the original YOLOv5 model to obtain a YOLOv5_MobileNetV4 model.

[0079] Among them, multiple different improved YOLOv5 models include YOLOv5_MobileVi t model, YOLOv5_MobileNetV2 model and YOLOv5_MobileNetV4 model.

[0080] S3: training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset.

[0081] Specifically, the original YOLOv5 model and a plurality of different improved YOLOv5 models are trained using the training set.

[0082] S4, evaluating the trained original YOLOv5 model and the improved YOLOv5 model through multiple performance indicators to obtain the improved YOLOv5 model with the best comprehensive performance, such as Figure 4 As shown, the specific steps include:

[0083] S40, obtaining the original YOLOv5 model and generating a confusion matrix and a PR curve during training of the improved YOLOv5 model.

[0084] S41, according to the confusion matrix and the PR curve, obtain the precision, recall, mean Average Precision (mAP), frames per second (FPS), GFLOPs (gigafloating-point operations per second) and model parameters Parameters.

[0085] S42, evaluating the trained original YOLOv5 model and the improved YOLOv5 model according to the precision, the recall, the average precision mean, the frames per second, the GFLOPs, and the model parameter quantity, to obtain the improved YOLOv5 model with the best overall performance.

[0086] S5, based on the improved YOLOv5 model with the best comprehensive performance, it is deployed on the mobile terminal to perform visual image detection on the fasteners to be detected, so as to realize classification and counting of the fasteners to be detected.

[0087] Based on the improved YOLOv5 model with the best comprehensive performance, it is deployed on a mobile terminal to improve the overall efficiency and accuracy of classification and counting of fasteners to be detected.

[0088] Preferably, if Figure 5 As shown, in step S5, based on the improved YOLOv5 model with the best comprehensive performance, the model is deployed on the mobile terminal to perform visual image detection on the fasteners to be detected, and the specific method for implementing the classification and counting of the fasteners to be detected includes the following steps:

[0089] S50, performing file format conversion and optimization on the improved YOLOv5 model with the best comprehensive performance.

[0090] S51, deploying the improved YOLOv5 model with the best comprehensive performance after file format conversion and optimization on a mobile terminal.

[0091] S52, generating an APP application installation package and installing it in the mobile terminal.

[0092] S53, performing visual image detection on the fasteners to be detected through the APP in the mobile terminal to achieve classification and counting of the fasteners to be detected.

[0093] Based on visual image detection technology, an improved YOLOv5 model is deployed on the mobile terminal. The camera of the mobile terminal can be used to take real-time photos of the fasteners to be inspected. The photos in the album can also be directly called and detected, and the images of the fasteners to be inspected can be detected in real time. Finally, the classification and counting of the fasteners to be inspected can be achieved.

[0094] In summary, the mobile terminal fastener classification and counting method based on improved YOLOv5 utilizes the improved YOLOv5 model with the best comprehensive performance to be deployed on the mobile terminal to perform visual image detection on the fasteners to be detected. It is based on visual image detection technology and can realize album photo detection, photo taking and real-time detection through the camera of the mobile terminal, and automatically classify and count the fasteners to be detected. It has a good degree of automation and counting accuracy, and solves the problem that the weighing counting method cannot classify and count fasteners such as bolts, nuts and washers, and the problem that the weighing counting method is easily affected by oil pollution and has poor accuracy and low degree of automation.

[0095] In addition, compared with the manual detection and counting method that consumes a lot of manpower and time costs, the present invention is based on the improved YOLOv5 model with the best comprehensive performance, deployed on the mobile terminal, and uses the mobile terminal to realize the classification and counting of the fasteners to be detected, which can realize rapid classification and counting and save manpower and time costs. Compared with the photoelectric counting method that is easily disturbed by the intensity and color of the ambient light and leads to erroneous counting results, the present invention can improve the counting accuracy and reduce misjudgment. Compared with the high initial investment cost and the maintenance and upgrade costs including software updates and hardware repairs, the present invention has the characteristics of low cost and strong robustness.

[0096] The present invention also provides a mobile terminal fastener classification and counting system based on improved YOLOv5, which includes a first building module, a second building module, a training module, a performance evaluation module and a deployment detection module.

[0097] The first construction module is used to construct a fastener data set; the second construction module is used to construct an original YOLOv5 model, and obtain multiple different improved YOLOv5 models based on the original YOLOv5 model; the training module is used to train the original YOLOv5 model and multiple different improved YOLOv5 models through the fastener data set; the performance evaluation module is used to evaluate the trained original YOLOv5 model and the improved YOLOv5 model through multiple performance indicators to obtain the improved YOLOv5 model with the best comprehensive performance; the deployment detection module is used to deploy the improved YOLOv5 model with the best comprehensive performance on a mobile terminal to perform visual image detection on the fasteners to be detected, so as to realize the classification and counting of the fasteners to be detected.

[0098] Preferably, the multiple different improved YOLOv5 models include YOLOv5_MobileVi t model, YOLOv5_MobileNetV2 model and YOLOv5_MobileNetV4 model.

[0099] The fasteners include bolts, nuts and washers, the fastener data set includes a training set, a verification set and a test set, and the mobile terminal is an Android terminal.

[0100] The mobile terminal fastener classification and counting system based on improved YOLOv5 is deployed on a mobile terminal by setting a first building module, a second building module, a training module, a performance evaluation module and a deployment detection module, and utilizing the improved YOLOv5 model with the best comprehensive performance to perform visual image detection on the fasteners to be detected. Based on the visual image detection technology, the system can realize album photo detection, photo taking and real-time detection through the camera of the mobile terminal, automatically classify and count the fasteners to be detected, and has a good degree of automation and counting accuracy, thereby solving the problem that the weighing counting method cannot classify and count fasteners such as bolts, nuts and washers, and the problem that the weighing counting method is easily affected by oil pollution and has poor accuracy and low degree of automation.

[0101] As a preferred technical solution, Figure 6 As shown, the mobile terminal fastener classification and counting method based on improved YOLOv5 described in the present invention includes the following steps:

[0102] S1, build fastener dataset.

[0103] Specifically, fastener images including bolts, nuts and washers of different sizes are collected. For the collected fastener images, in order to reduce the amount of calculation and relieve the pressure on the equipment, the collected fastener images are converted into images with a resolution of 640*640 using the computer vision library Open CV. The collected fastener images such as bolts, nuts and washers are enhanced by the computer vision library Open CV. The data enhancement includes but is not limited to brightness enhancement, contrast enhancement, rotation transformation, flip transformation, affine transformation, shear transformation, HSV data enhancement and translation transformation, such as Figure 7 , Figure 8 , Fig. 9 as well as Fig.10 As shown. This increases the data set, reduces overfitting, and improves the generalization ability of the model.

[0104] LabelImg software is used to label fasteners such as bolts, nuts and washers, including the categories of fasteners such as bolts, nuts and washers and the labeling of location information files. At the same time, duplicate data, missing value data and outlier data are removed to ensure the accuracy, completeness, consistency and availability of the data, so as to ensure the quality and credibility of the data set.

[0105] Finally, the fastener dataset is divided into training set, validation set and test set in a ratio of 6:3:1 to facilitate model training.

[0106] S2: construct an original YOLOv5 model, and obtain multiple different improved YOLOv5 models according to the original YOLOv5 model.

[0107] To improve YOLOv5, three different network structures were used.

[0108] Method 1: Use Mobi leVit to replace the backbone network of YOLOv5 to obtain the YOLOv5_MobileVit model. Specifically, the Mobi leVit network realizes efficient image recognition and processing on mobile devices and other resource-constrained platforms. Use the Mobi leVit network to replace the original head feature extraction network C3 as the backbone network of the head network to obtain the YOLOv5_Mobi leVit model, as shown in Fig.11 shown.

[0109] Principle and features: Mobi leVit is a lightweight, general-purpose visual transformer for mobile devices. It combines the advantages of convolutional neural networks (CNNs) and visual transformers (ViTs), and can not only utilize the spatial local information processing capabilities of CNNs, but also the global information processing capabilities of transformers. It models the local and global information in the input tensor through a special Mobi leVitblock structure. The structure first applies a standard convolutional layer to encode local spatial information, and then uses a point convolutional layer to project the tensor into a high-dimensional space in order to model long-distance non-local dependencies.

[0110] Performance advantage: With a similar number of parameters, Mobi leVit outperforms existing lightweight CNNs in different mobile vision tasks, with better generalization and robustness. For example, on the ImageNet-1k dataset, Mobi leVit achieves 78.4% top-1 accuracy with about 6 million parameters, which is 3.2% and 6.2% higher than Mobi lenetV3 and DEIT, respectively.

[0111] Application scenarios: Mobi leVit is suitable for mobile device vision tasks with limitations on model size and computing resources, such as image classification, target detection and segmentation on mobile devices.

[0112] Method 2: Use MobileNetV2 to replace the backbone network of YOLOv5 to obtain the YOLOv5_MobileNetV2 model. Specifically, MobileNetV2 is a lightweight neural network architecture that is mainly used for computer vision tasks on mobile devices. In mobile phone camera applications, it can classify photos taken in real time. Use the MobileNetV2 network to replace the original head feature extraction network C3. As the backbone network backbone of the head network, MobileNetV2 mainly uses deep separable convolution. On the basis of MobileNetV1, it references the residual module and the activation function of Relu6 f(x) = min(max(0,x), 6). The idea of ​​1*n and n*1 is used instead of the n*n matrix, which will reduce the amount of calculation. First, modify the backbone network: add MobileNetV2 as the backbone in models / common; secondly, add two functions ConvBNReLU and InvertedResidual in common in yolo.py, such as Fig.12 As shown; Finally, make the YAML configuration file of MobileNetV2 to get the YOLOv5_MobileNetV2 model, as shown Fig.13 shown.

[0113] Principle and characteristics: The MobileNetV2 model introduces a linear bottleneck and an inverted residual structure. First, 1x1 convolution is used to increase the dimension, and deep convolution is used to process the information in the length and width directions in the high-dimensional space, and then 1x1 convolution is used to reduce the dimension. When reducing the dimension, a linear activation function is used instead of the common ReLU6 activation function, which can reduce information loss, especially in low-dimensional space.

[0114] Performance advantage: Compared with traditional convolutional neural networks, MobileNetV2 greatly reduces the number of model parameters and computational complexity while maintaining high accuracy, making it very suitable for running on mobile devices. For example, it has only about 3 million parameters, far fewer than some large neural network models.

[0115] Application scenarios: MobileNetV2 is widely used in various computer vision tasks on mobile devices, such as real-time image recognition, video surveillance and other scenarios, and can process visual information quickly and accurately in resource-constrained environments.

[0116] Method three: Use MobileNetV4 to replace the backbone network of YOLOv5 to obtain the YOLOv5_MobileNetV4 model. Specifically, compared with the MobileNetV2 network, the MobileNetV4 network provides flexibility in terms of spatial and channel mixing, can choose to expand the receptive field, and improve computational efficiency. Use the MobileNetV4 network to replace the original head feature extraction network C3 as the backbone network of the head network, download the MobileNetV4 code and put it in the folder under the YOLOv5 project, and then import the MobileNetV4 related modules in the yolo.py file to obtain the YOLOv5_MobileNetV4 model, as shown in the figure. Fig.14 shown.

[0117] Principle and features: The core of MobileNetV4 is the universal inverted bottleneck (UIB) search block, which is a unified and flexible structure that combines the inverted bottleneck, ConvNext, feedforward network (FFN) and a novel extra depth separable (ExtraDW) variant. In addition, MobileMQA is introduced, which is an attention block designed for mobile accelerators to improve the running speed of the model.

[0118] Performance advantage: MobileNetV4 can achieve good performance on a variety of hardware platforms, such as mobile CPUs, DSPs, GPUs, and dedicated accelerators, close to Pareto optimality. By introducing novel distillation technology, the accuracy of the model can be further improved.

[0119] Application scenarios: MobileNetV4 is suitable for mobile device application scenarios that have high requirements for model performance and accuracy, such as artificial intelligence applications in high-end smartphones and visual perception of mobile robots.

[0120] S3: training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset.

[0121] The collected image data of bolts, nuts and washers are saved in the Dataset folder under the original YOLOv5 model and multiple different improved YOLOv5 models for model training. The training environment is shown in Table 1, including CUDA11.8 (Compute Unified Device Architecture, unified computing structure 11.8), Pytorch 1.9.0 and Python3.7.

[0122] Hardware / Software CPU GPU CUDA Pytorch Python Model / Version i7-13850HX 2080 16G 11.8 1.9.0 3.7

[0123] Table 1 Model training environment parameters

[0124] S4, evaluating the trained original YOLOv5 model and the improved YOLOv5 model through multiple performance indicators to obtain an improved YOLOv5 model with the best comprehensive performance.

[0125] According to the confusion matrix, LOSS curve, PR curve, etc. generated by model training, the trained original YOLOv5 model and the improved YOLOv5 model are evaluated using multiple performance indicators such as precision, recall, mean Average Precision (mAP), FPS, GFLOPs (gigafloating-point operations per second), and model parameters.

[0126] GFLOPs and model parameters can be calculated by running the Python script yolo.py in the models folder of the YOLOv5 project. The GFLOPs calculation reflects the computing requirements of the model at runtime. A higher amount of calculation may require more powerful hardware devices to achieve real-time reasoning, especially in processing large-scale image data or in application scenarios with high real-time requirements.

[0127] C is the total number of categories, S PRi is the area under the PR curve of the i-th category, TP represents the positive sample predicted by the model as the positive class, FP represents the negative sample predicted by the model as the positive class, and FN represents the positive sample predicted by the model as the negative class. pre_process is the image preprocessing time, including image aspect ratio scaling and padding, channel transformation (HWC-CHW) and dimensionality increase processing, etc., in ms; inference is the inference time, which refers to the time from the image input model after preprocessing to the model output result, in ms; NMS0 is the non-maximum suppression processing time, which converts the model output result, etc., in ms.

[0128] The performance indicators of the original YOLOv5 model and the improved YOLOv5 model are shown in Table 2.

[0129]

[0130] Table 2 Changes of indicators before and after improvement

[0131] The pre_process, inference, and NMS parameters of the original YOLOv5 model and the improved YOLOv5 model are shown in Table 3.

[0132] pre_process inference NMS0 FPS YOLOv5s 0.6ms 12.3ms 1.5ms 69.44 YOLOv5_MobileVit 0.4ms 24.5ms 1.4ms 38.02 YOLOv5_MobileNetV2 0.4ms 15.2ms 1.6ms 58.13 YOLOv5_MobileNetV4 0.4ms 19.3ms 2.2ms 45.66

[0133] Table 3 Comparison of pre_process, inference, and NMS

[0134] S5, deploying the improved YOLOv5 model with the best comprehensive performance on a mobile terminal to perform visual image detection on the fasteners to be detected, thereby achieving classification and counting of the fasteners to be detected.

[0135] Combined with Table 2, for the bolt, nut and gasket data set of the present invention, it can be seen from the comparison of the performance indicators of the original YOLOv5 model and the three different improved YOLOv5 models that the average precision of the improved YOLOv5 model is improved compared with the original YOLOv5 model, and the performance indicators of the YOLOv5_MobileNetV4 model are poor compared with the other two improved YOLOv5 models. Comparing the YOLOv5_MobileNetV2 model with the YOLOv5_MobileVi t model, the average precision of the YOLOv5_MobileNetV2 model is 3% lower, but its FPS is significantly higher by 26%, and GFLOPs is reduced by 3 times. Small GFLOPs is conducive to deployment in mobile devices with low computing power. The larger the FPS value, the faster the detection speed of the model, the smaller the computing resources occupied, and the flash back of the APP will not be caused by insufficient computing power during the calculation process of the Android device. The performance indicators of various improved YOLOv5 models were comprehensively compared, and finally the YOLOv5_MobileNetV2 model was selected for deployment.

[0136] Convert the yolov5s.pt and yolov5_mobi lenetv2.pt files generated by training into yolov5s.torchscript and yolov5_mobi lenetv2.torchscript files respectively, and then use the optimizer module torch.ut i ls.mobile_optimizer in PyTorch to optimize the model, reduce the model size, and reduce the computational complexity to obtain two Pytorch Lite model files, yolov5.torchscript.pt l and yolov5_mobi lenetv2.torchscript.pt l.

[0137] Import yolov5_mobi lenetv2.torchscript.pt l into the ObjectDetect ion project based on the Pytorch Adroid Lite framework (an open source Android APP that implements simple target detection by calling the Pytorch Lite model), improve ObjectDetect ion through Android Studio, add functional interfaces (open album, take photos with camera, real-time detection) and result display interfaces.

[0138] Specifically, use Android Studio to open the ObjectDetection project of Pytorch, import the yolov5_mobi lenetv2.torchscript.pt l file into the project, and improve the project. First, improve the function interface and add buttons for opening the album, taking photos with the camera, and real-time detection. Secondly, add a display window for counting results in the detection result interface. The specific implementation steps are as follows: (1) Create a HashSet to store the three category indexes of the detected bolts, nuts, and washers. (2) Traverse the detection result list results and add the category index of each result to the HashSet. (3) Convert the HashSet to an ArrayList and sort the ArrayList. (4) Traverse the sorted ArrayList and count the number of each category in the detection results. (5) Add the statistical results to mResultString. (6) Set mResultString to the text of TextView. Finally, add the real-time detection interface. The specific steps are as follows: (1) Load the model yolov5_mobi lenetv2.torchscript.pt l file and the class name file classes.txt. (2) Initialize the camera preview view and result display view. (3) Obtain camera permissions and start the camera. (4) Process the image captured by the camera, convert it into a bitmap, and rotate and scale it. (5) Convert the bitmap into a Tensor and use the machine learning model for forward propagation to obtain the output result. (6) Process the output result and display the number of different categories detected in real time on the interface.

[0139] Name the APP as "Fastener Counting App" through Android Studio and generate the application installation package through Build APK, and install the APP in the Android phone Oneplus 8T.

[0140] The Android test environment parameter table is shown in Table 4.

[0141] Hardware / Software Android Phone CPU Camera Android Model / Version Oneplus 8T Snapdragon 865 4800 pixels 14

[0142] Table 4 Android test environment parameters

[0143] After successful installation, click the button on the function interface to realize the classification and counting of bolts, nuts and washers. The test results are as follows: Fig.15 shown.

[0144] Preferably, the mobile terminal fastener classification and counting method based on improved YOLOv5 also includes the following steps: deploying the original YOLOv5 model on the Android terminal, importing yolov5.torchscript.pt l into the ObjectDetection project based on the Pytorch AdroidLi te framework, improving ObjectDetection through Android Studio, adding a functional interface (opening the album, taking photos with the camera, real-time detection) and a result display interface; installing the APP on the Android terminal, classifying and counting the bolts, nuts and washers, and the detection results are as follows: Fig.16 shown.

[0145] Combination Fig.15 as well as Fig.16 It is known that in Fig.16 In the figure, there is a false detection in the upper right corner. Fig.15 By comparing the detection results of the YOLOv5s original model and the YOLOv5_MobileNetV2 model, it can be clearly seen that the model improvement has achieved remarkable results.

[0146] According to the confusion matrix, LOSS curve, PR curve, etc. generated by model training, multiple performance index parameters including precision, recall, average precision, FPS, GFLOPs, model parameter quantity Parameters, etc. can be obtained, and the advantages and disadvantages of the trained original YOLOv5 model and the improved YOLOv5 model can be evaluated based on multiple performance index parameters. However, in the prior art, technicians need to comprehensively compare the performance indicators of various YOLOv5 models based on personal experience to obtain a YOLOv5 model with better comprehensive performance indicators. When selecting the YOLOv5 model with the best comprehensive performance, the above method requires technicians to spend a certain amount of time to repeatedly compare and measure, which is inefficient and not smart enough.

[0147] In order to solve the above problem, in one embodiment, the mobile terminal fastener classification and counting method based on improved YOLOv5 also includes the following steps: assigning corresponding weight values ​​to different performance index parameters respectively, and calculating the first comprehensive performance values ​​of multiple different improved YOLOv5 models relative to the original YOLOv5 model according to the performance index parameter values ​​of the original YOLOv5 model and multiple different improved YOLOv5 models respectively; based on the first comprehensive performance value, taking the improved YOLOv5 model with the largest first comprehensive performance value as the YOLOv5 model with the best comprehensive performance.

[0148] In this way, by assigning corresponding weight values ​​to different performance indicator parameters and calculating the first comprehensive performance values ​​of multiple different improved YOLOv5 models relative to the original YOLOv5 model, the YOLOv5 model with the best comprehensive performance can be automatically and quickly obtained. It does not require technicians to spend time repeatedly comparing and measuring, and has a high degree of intelligent automation.

[0149] Preferably, the first comprehensive performance value Among them, PV i ' represents the i-th performance indicator parameter value in the original YOLOv5 model, where PV i represents the i-th performance index parameter value in the improved YOLOv5 model, n represents the number of performance index parameters, α i It represents the first adjustment coefficient corresponding to the i-th performance indicator parameter value in the improved YOLOv5 model, which can be set by the technician. The first comprehensive performance value can be quickly calculated.

[0150] The above method for automatically and quickly obtaining the YOLOv5 model with the best comprehensive performance is based on the first comprehensive performance values ​​of multiple different improved YOLOv5 models relative to the original YOLOv5 model, and is not measured and evaluated based on its own performance indicator parameter values. Since it evaluates the overall performance values ​​of different improved YOLOv5 models through a single dimension, it has problems of singleness and limitations.

[0151] Preferably, in order to further improve the intelligence and accuracy of obtaining the YOLOv5 model with the best comprehensive performance, and solve the problems of singleness and limitations of evaluating the overall performance values ​​of different improved YOLOv5 models through a single dimension, the mobile terminal fastener classification and counting method based on improved YOLOv5 also includes the following steps: according to the performance indicator parameter values ​​of the original YOLOv5 model and multiple different improved YOLOv5 models, calculating the comprehensive performance values ​​of multiple different improved YOLOv5 models relative to the original YOLOv5 model from multiple different dimensions, and calculating the overall performance value according to the comprehensive performance values ​​under multiple different dimensions Among them, CPV i represents the i-th comprehensive performance value under multiple different dimensions, β i Represents the second adjustment coefficient of the i-th comprehensive performance value. In this way, the overall performance value is calculated by calculating the average of the comprehensive performance values ​​under multiple different dimensions, and the overall performance values ​​of different improved YOLOv5 models relative to the original YOLOv5 model are calculated and evaluated by integrating multiple different dimensions. This can avoid the problems of singleness and limitations in evaluating the overall performance values ​​of different improved YOLOv5 models through a single dimension, and improve the intelligence and accuracy of obtaining the YOLOv5 model with the best comprehensive performance.

[0152] Specifically, the performance indicator parameter values ​​include precision pre, recall rec, average precision mAP, FPS, GFLOPs and model parameter par. Before performing the above operations, the performance indicator parameter values ​​may be normalized to simplify the calculation and improve the efficiency of the algorithm.

[0153] CPV total =(β 1 ×CPV 1 +β 2 ×CPV 2 ) / 2, CPV 1 Represents the first comprehensive performance value, β 1 The first adjustment factor representing the first performance comprehensive value, CPV 2 Represents the second comprehensive performance value, β 2 A second adjustment factor representing a second comprehensive performance value.

[0154] Among them, PV i " represents the quantized i-th performance index parameter value in the improved YOLOv5 model, n represents the number of performance index parameters, β i It represents the third adjustment coefficient corresponding to the quantized i-th performance indicator parameter value in the improved YOLOv5 model, which can be set by a technician, and e represents a natural constant.

[0155] The first comprehensive performance value Among them, PV i ' represents the i-th performance indicator parameter value in the original YOLOv5 model, PV i " represents the quantized i-th performance index parameter value in the improved YOLOv5 model, n represents the number of performance index parameters, α i It represents the first adjustment coefficient corresponding to the quantized i-th performance indicator parameter value in the improved YOLOv5 model, which can be set by a technician.

[0156] In summary, the mobile terminal fastener classification and counting method based on improved YOLOv5 of the present invention has the following advantages:

[0157] 1. The present invention adopts an improved YOLOv5 model to classify and count bolts, nuts and washers. By using the Pytorch Android Lite framework to load the improved YOLOv5 model, it can be used in bolt, nut and washer classification and counting situations. It can realize the classification and counting of bolts, nuts and washers through a mobile phone APP, which greatly reduces the labor cost and time cost, and is much faster than manual classification and counting. The improved YOLOv5 model can be deployed on Android devices to maintain high detection accuracy and speed, and can accurately identify target objects, so as to achieve fast classification and counting of bolts, nuts and washers.

[0158] 2. The present invention uses data enhancement to enhance the collected images of bolts, nuts and gaskets through the computer vision library Open CV, and performs operations such as brightness enhancement, contrast enhancement, rotation transformation, mirror transformation, affine transformation, shear transformation, HSV data enhancement, and translation transformation, so as to improve the generalization ability of the model, reduce the amount of calculation and relieve equipment pressure.

[0159] 3. The present invention can not only classify and count bolts, nuts and washers by opening the album and selecting photos through the APP, but also classify and count bolts, nuts and washers by opening the camera and taking photos.

[0160] 4. Since MobileNetV2 is widely used in various computer vision tasks on mobile devices, such as real-time image recognition, video surveillance and other scenarios, it can process visual information quickly and accurately in resource-constrained environments. The present invention also performs classification and counting of bolts, nuts and washers in real-time video.

[0161] The present invention can realize the classification counting of the target in corresponding occasions by training other target data sets.

[0162] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A mobile terminal fastener classification and counting method based on improved YOLOv5, characterized in that: The mobile terminal fastener classification and counting method based on improved YOLOv5 comprises the following steps: S1, build fastener dataset; S2, constructing an original YOLOv5 model, and obtaining a plurality of different improved YOLOv5 models according to the original YOLOv5 model; S3, training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset; S4, evaluating the trained original YOLOv5 model and the improved YOLOv5 model through multiple performance indicators to obtain the improved YOLOv5 model with the best comprehensive performance; S5, based on the improved YOLOv5 model with the best comprehensive performance, it is deployed on the mobile terminal to perform visual image detection on the fasteners to be detected, so as to realize classification and counting of the fasteners to be detected.

2. A mobile terminal fastener classification and counting method based on improved YOLOv5 as claimed in claim 1, characterized in that: In step S1, the specific method of constructing the fastener dataset includes the following steps: S10, collecting images of fasteners of different types and sizes; S11, performing data enhancement on the collected fastener image to increase the fastener data set; S12, annotating the fastener image after data enhancement; S13, dividing the annotated fastener images into a training set, a verification set and a test set according to a preset ratio.

3. A mobile terminal fastener classification and counting method based on improved YOLOv5 as claimed in claim 2, characterized in that: In step S2, an original YOLOv5 model is constructed, and a specific method for obtaining a plurality of different improved YOLOv5 models according to the original YOLOv5 model comprises the following steps: S20, improving the structure of the original YOLOv5 model to obtain a YOLOv5_MobileVit model; S21, improving the structure of the original YOLOv5 model to obtain a YOLOv5_MobileNetV2 model; S22, improving the structure of the original YOLOv5 model to obtain a YOLOv5_MobileNetV4 model; Among them, multiple different improved YOLOv5 models include YOLOv5_MobileVit model, YOLOv5_MobileNetV2 model and YOLOv5_MobileNetV4 model.

4. A mobile terminal fastener classification and counting method based on improved YOLOv5 as claimed in claim 3, characterized in that: The specific method for training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset comprises the following steps: The original YOLOv5 model and a plurality of different improved YOLOv5 models are trained using the training set.

5. A mobile terminal fastener classification and counting method based on improved YOLOv5 as claimed in claim 4, characterized in that: In step S4, the trained original YOLOv5 model and the improved YOLOv5 model are evaluated by multiple performance indicators, and the specific method for obtaining the improved YOLOv5 model with the best comprehensive performance includes the following steps: S40, obtaining the original YOLOv5 model and generating a confusion matrix and a PR curve during training of the improved YOLOv5 model; S41, obtaining precision, recall, average precision, frames per second, GFLOPs, and model parameter quantity according to the confusion matrix and the PR curve; S42, evaluating the trained original YOLOv5 model and the improved YOLOv5 model according to the precision, the recall, the average precision mean, the frames per second, the GFLOPs, and the model parameter quantity, to obtain the improved YOLOv5 model with the best overall performance.

6. A mobile terminal fastener classification and counting method based on improved YOLOv5 as claimed in claim 5, characterized in that: In step S5, based on the improved YOLOv5 model with the best comprehensive performance, the model is deployed on the mobile terminal to perform visual image detection on the fasteners to be detected. The specific method for implementing classification and counting of the fasteners to be detected includes the following steps: S50, performing file format conversion and optimization on the improved YOLOv5 model with the best comprehensive performance; S51, deploying the improved YOLOv5 model with the best comprehensive performance after file format conversion and optimization on a mobile terminal; S52, generating an APP application installation package and installing it in the mobile terminal; S53, performing visual image detection on the fasteners to be detected through the APP in the mobile terminal to achieve classification and counting of the fasteners to be detected.

7. A mobile terminal fastener classification and counting system based on improved YOLOv5, characterized in that: The mobile terminal fastener classification and counting system based on improved YOLOv5 includes: A first building module is used to build a fastener dataset; A second construction module is used to construct an original YOLOv5 model, and obtain a plurality of different improved YOLOv5 models according to the original YOLOv5 model; A training module, used for training the original YOLOv5 model and a plurality of different improved YOLOv5 models using the fastener dataset; A performance evaluation module is used to evaluate the trained original YOLOv5 model and the improved YOLOv5 model through multiple performance indicators to obtain an improved YOLOv5 model with the best comprehensive performance; A detection module is deployed, which is used to perform visual image detection on the fasteners to be detected based on the improved YOLOv5 model with the best comprehensive performance, and realize classification and counting of the fasteners to be detected.

8. A mobile terminal fastener classification and counting system based on improved YOLOv5 as claimed in claim 7, characterized in that: Multiple different improved YOLOv5 models include the YOLOv5_MobileVit model, the YOLOv5_MobileNetV2 model, and the YOLOv5_MobileNetV4 model.

9. A mobile terminal fastener classification and counting system based on improved YOLOv5 as claimed in claim 8, characterized in that: The fasteners include bolts, nuts and washers, and the fastener data set includes a training set, a validation set and a test set.

10. A mobile terminal fastener classification and counting system based on improved YOLOv5 as claimed in claim 9, characterized in that: The mobile terminal is an Android terminal.

Images