Improvement method of YOLOv5 neural network based on within-class and between-class correlation

Abstract

The application provides a YOLOv5 neural network improvement method based on intra-class and inter-class correlation, comprising the following steps: obtaining a feature map and a prediction box through a YOLOv5 network model, and then calculating target spectral information and gradient information according to the input image, the feature map and the prediction box; using a fully connected network to abstract target essential features from the target spectrum and gradient information and representing the target essential features as a vector; representing the difference between different target features by calculating the distance measure between the two vectors; combining a YOLOv5 loss function to construct a loss function based on the intra-class and inter-class measure difference to train the network model, so as to force the network to learn the universal features that are conducive to distinguishing different targets. The application adds a loss function based on the intra-class and inter-class correlation to the original loss function of the network to constrain, thereby strengthening the classification ability of the network to objects and improving the precision of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing, and particularly relates to a YOLOv5 neural network improvement method based on intra-class and inter-class correlation. BACKGROUND

[0002] Target detection is a task of finding and classifying targets in images or videos. As a hot spot in the field of computer vision, it has wide application prospects in intelligent driving, vehicle detection, crop yield estimation, etc. However, due to the problems such as small target being difficult to detect, the same target showing significant different features in different images, and different targets having similar features, the development of target detection is brought many difficulties and challenges. Traditional target detection algorithms based on deep learning use end-to-end network training method, and the model feature learning mainly depends on the number and quality of training samples. When the target shows diversified spectral or structural features, the existing network model can only sacrifice the recognition accuracy of a small number of targets inconsistent with the main target features to ensure that the overall recognition accuracy is optimal.

[0003] Patent 202111497382.X gives an improved yolov5 target detection algorithm model, which combines the convolution module, C3 module, multi-scale fusion module, up-sampling module and fusion module in the network model to obtain a new network model; improves the network detection accuracy.

[0004] Patent 202111303025.5 replaces the backbone network of YOLOv5 algorithm with MobileNetV3 network which removes the last average pooling layer and two pointwise convolution layers, and adjusts the modules in MobileNetV3 network model and YOLOv5 network model; while retaining the detection accuracy, improves the detection speed.

[0005] The above two patents mainly improve the network model to improve the performance of the network. However, for target detection with diversified spectral and structural features, improving the feature learning module cannot effectively improve the accuracy of this type of target detection. Fully considering the intra-class and inter-class correlation information can effectively control the learning preference of the network model, and then achieve the purpose of improving the feature learning ability of the model for this type of target. SUMMARY

[0006] To solve the above technical problems, the application provides a YOLOv5 neural network improvement method based on intra-class and inter-class correlation. According to the differences between different class targets, the spectral and gradient information of different class targets are integrated, different class information is compared, and a model with higher accuracy is obtained by making the network learn the comparison result.

[0007] To achieve the above objectives, this invention provides an improved YOLOv5 neural network method based on intra- and inter-class correlations, comprising the following steps:

[0008] Obtain the target spectral information and target gradient information based on the predicted bounding box;

[0009] By using a fully connected network, the essential features of the target are abstracted from the target spectral information and target gradient information and represented as a vector, that is, each class is represented by a vector;

[0010] The difference between different target features is represented by calculating the distance metric between two vectors;

[0011] By combining the YOLOv5 loss function, a loss function based on intra-class and inter-class metric differences is constructed to train the network model, forcing the network to learn universal features that are beneficial for distinguishing different targets.

[0012] Alternatively, the method for obtaining the target spectral information is as follows:

[0013] Obtain the predicted bounding boxes and feature maps obtained from network training;

[0014] The mean and variance of the data within the prediction box are calculated to obtain the target spectral information.

[0015] Optionally, the method for obtaining target gradient information is as follows:

[0016] Obtain the input image and the predicted bounding boxes obtained from network training;

[0017] Based on the input image, an image containing gradient information is obtained;

[0018] Place the prediction box on the graph containing gradient information;

[0019] The mean and variance of the data placed within the prediction box on the graph containing gradient information are calculated to obtain the target gradient information.

[0020] Optionally, the comparison of the vectors includes intra-class differences and inter-class differences;

[0021] The method for calculating intra-class differences is as follows: the intra-class differences of the same target in different images can be obtained by calculating the distance measure of the vectors in two iterations.

[0022] The method for calculating inter-class differences is as follows: in the iteration where there are differences among targets of the same class, the distance measure between the vectors of this class and all other classes saved in the previous iterations can be used to characterize the differences between the features of different targets.

[0023] Optionally, the formula for calculating the intra- and inter-class differences is the cosine similarity S. a,b :

[0024]

[0025] In a certain iteration, a new vector appears in a certain class. In calculating the intra-class difference, the new vector in this iteration is A in formula (1), and B is the old vector of the same class as A generated in the previous iteration. In calculating the inter-class difference, the new vector with intra-class difference in this iteration is A in formula (1), and B is the vector of all other classes saved in the previous iteration.

[0026] Optionally, the expression for the loss function based on target information is:

[0027]

[0028] in, as well as The intra-class and inter-class differences are calculated using formula (1). For intra-class differences, To represent inter-class differences, 'a' is a new vector of a certain class in a certain iteration, 'a0' is an old vector of the same class as 'a' generated in previous iterations, and 'b'... m is the vector of all other classes saved in the previous iteration, n is the number of all other class vectors saved in the previous iteration, and j is the number of classes in the new vector calculated in a certain iteration that are the same as the classes in the old vector calculated in the previous iteration.

[0029] Optionally, the method of combining the loss function based on target information with the binary cross-entropy loss function and CIOU loss function of the YOLOv5 neural network itself is as follows:

[0030] loss = L YOLOv5 +λL intra-inter

[0031] Where λ is the adjustment parameter.

[0032] Compared with the prior art, the present invention has the following advantages and technical effects:

[0033] This invention introduces intra- and inter-class correlations on top of the network's own loss function, integrates information from different target classes, and constructs a loss function from the comparison results of information from different target classes. This allows the network to learn more detailed classification information, enhances the network's ability to classify objects, and enables the generation of a more accurate model. Attached Figure Description

[0034] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 This is a schematic diagram of the improved YOLOv5 neural network method based on intra- and inter-class correlations according to Embodiment 1 of the present invention. Detailed Implementation

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0038] Example 1

[0039] like Figure 1 As shown, this invention provides an improved YOLOv5 neural network method based on intra- and inter-class correlations, comprising the following steps:

[0040] Obtain target spectral information and target gradient information;

[0041] By using a fully connected network, the essential features of the target are abstracted from the target's spectral and gradient information and represented as a vector, that is, each class is represented by a vector;

[0042] The differences between different target features can be characterized by calculating the distance measure between two vectors. Similarly, the intra-class differences of the same target in different images can be obtained by calculating the distance measure between vectors in two iterations.

[0043] By combining the YOLOv5 loss function, a loss function based on intra-class and inter-class metric differences is constructed to train the network model, forcing the network to learn universal features that are beneficial for distinguishing different targets.

[0044] Calculation of spectral and gradient information

[0045] Furthermore, the YOLOv5 neural network can obtain three sets of feature maps of different sizes after training. These three sets of feature maps are upsampled and concatenated to obtain a set of feature maps. Then, convolution is performed to obtain the desired feature map. The prediction box is converted to the size of the feature map and placed on the feature map. The mean and variance of the data in the prediction box are calculated to obtain spectral information.

[0046] Furthermore, the input image is obtained, and gradient calculation is performed on the input image to obtain a graph containing gradient information. The predicted bounding boxes obtained from network training are obtained, and the predicted bounding boxes are placed on the graph containing gradient information. The mean and variance of the data within the predicted bounding boxes are calculated to obtain the target gradient information.

[0047] Integration of spectral and gradient information

[0048] Furthermore, a fully connected network is used to abstract the essential features of the target from the target's spectral and gradient information and represent them as a vector, that is, each class is represented by a vector.

[0049] Intra-class and inter-class comparisons

[0050] Furthermore, the difference calculation for each class of vectors includes intra-class difference and inter-class difference. Intra-class difference is obtained by calculating the distance measure between the vectors of two iterations to obtain the intra-class difference of the same target in different images. Inter-class difference is obtained by calculating the distance measure between the vectors of the same class and all other classes saved in the previous iterations during the iteration where there are differences among targets of the same class, thus representing the difference between the features of different targets.

[0051]

[0052] In one iteration, a new vector appeared in a certain class.

[0053] In calculating intra-class differences, the new vector in this iteration is A in formula (1), and B is the old vector of the same class as A generated in the previous iteration.

[0054] In calculating inter-class differences, the new vector with intra-class differences in this iteration is A in formula (1), and B is the vector of all other classes saved in the previous iteration.

[0055] Construction of loss function based on target information

[0056] Furthermore, since we want the distance between pixels of the same type to be as small as possible, and the distance between pixels of different types to be as large as possible, we construct the following loss function:

[0057]

[0058] in, as well as The intra-class and inter-class differences are calculated using formula (1). For intra-class differences, To represent inter-class differences, 'a' is a new vector of a certain class in a certain iteration, 'a0' is an old vector of the same class as 'a' generated in previous iterations, and 'b'... m is the vector of all other classes saved in the previous iteration, n is the number of all other class vectors saved in the previous iteration, and j is the number of classes in the new vector calculated in a certain iteration that are the same as the classes in the old vector calculated in the previous iteration.

[0059] The loss function based on target information is combined with the binary cross-entropy loss function and CIOU loss function of the YOLOv5 neural network itself to achieve the goal of enabling the network to learn intra-class and inter-class information:

[0060] loss = L YOLOv5 +λL intra-inter (3)

[0061] Where λ is an adjustable parameter that allows the loss function based on target information to have an appropriate impact on the binary cross-entropy loss function and CIOU loss function of the YOLOv5 neural network itself, avoiding the loss function based on target information from being too large or too small.

[0062] Furthermore, the vectors from each iteration are saved, with only one vector saved for each class. When a new vector of the same class appears, its average value is taken with the old vector of that class and then saved for subsequent calculation of intra-class and inter-class differences.

[0063] This invention adds a loss function based on intra- and inter-class correlation to the original loss function of the network to constrain it, thereby enhancing the network's ability to classify objects and improving the accuracy of the model.

[0064] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An improved YOLOv5 neural network method based on intra- and inter-class correlations, characterized in that, Includes the following steps: Obtain the target spectral information and target gradient information based on the predicted bounding box; By using a fully connected network, the essential features of the target are abstracted from the target spectral information and target gradient information and represented as a vector, that is, each class is represented by a vector; The difference between different target features is represented by calculating the distance metric between two vectors; By combining the YOLOv5 loss function, a loss function based on intra-class and inter-class measure differences is constructed to train the network model, forcing the network to learn universal features that are beneficial for distinguishing different targets. Comparisons of vectors include intra-class differences and inter-class differences; The method for calculating intra-class differences is as follows: the intra-class differences of the same target in different images can be obtained by calculating the distance measure of the vectors in two iterations. The method for calculating inter-class differences is as follows: in the iteration for differences between targets of the same class, the distance measure between the vectors of the current class and all other classes saved in the previous iterations can be used to characterize the differences between the features of different targets. The formula for calculating the intra- and inter-class differences is the cosine similarity S. a,b : In a certain iteration, a new vector appears in a certain class. In calculating intra-class differences, the new vector in this iteration is A in formula (1), and B is the old vector of the same class as A generated in the previous iteration. In calculating inter-class differences, the new vector with intra-class differences in this iteration is A in formula (1), and B is the vector of all other classes saved in the previous iteration. The expression for the loss function based on target information is: in, as well as The intra-class and inter-class differences are calculated using formula (1). For intra-class differences, To represent inter-class differences, 'a' is a new vector of a certain class in a certain iteration, 'a0' is an old vector of the same class as 'a' generated in previous iterations, and 'b'... m is the vector of all other classes saved in the previous iteration, n is the number of all other class vectors saved in the previous iteration, and j is the number of classes in the new vector calculated in a certain iteration that are the same as the classes in the old vector calculated in the previous iteration. The method of combining the target information-based loss function with the YOLOv5 neural network's own binary cross-entropy loss function and CIOU loss function is as follows: loss=L YOLOv5 +λL intra-inter Where λ is the adjustment parameter; The method for obtaining target spectral information is as follows: After training, the YOLOv5 neural network can obtain three sets of feature maps of different sizes. These three sets of feature maps are upsampled and concatenated to obtain a set of feature maps. Then, convolution is performed to obtain the desired feature map. The prediction box is converted to the size of the feature map and placed on the feature map. The mean and variance of the data in the prediction box are calculated to obtain spectral information. The method for obtaining target gradient information is as follows: Obtain the input image, perform gradient calculation on the input image to obtain a graph containing gradient information; obtain the prediction boxes obtained by network training, place the prediction boxes on the graph containing gradient information, and perform mean and variance calculation on the data in the prediction boxes to obtain the target gradient information. Save the vectors from each iteration, saving only one vector for each class. When a new vector of the same class appears, average it with the old vector of that class and save it for later use in calculating the differences between classes. A loss function based on intra- and inter-class correlation is added to the original network loss function to constrain it.

Images