An improved object intelligent detection method and system based on Yolo

Abstract

The present invention provides an object intelligent detection system based on yolo improvement, which solves the problem that the existing yolo network cannot detect the angle of object placement. The method includes: obtaining a picture of the object to be tested; inputting the picture of the object to be tested into a trained intelligent recognition network model, and obtaining the detection result and the corresponding object angle. The intelligent recognition network model is constructed based on the deep learning framework Pytorch, based on the yolo network model, and adopts the Darknet‑53 model structure, including 53 convolutional layers, 23 residual modules, 5 downsamplings, 3 prediction result output terminals at different scales, and an attention module arranged on each prediction result output terminal. On the basis of the original yolov3 network, the present invention adds an attention module, completes the detection of the object angle without reducing the object recognition accuracy, and is conducive to the mechanical gripper to accurately grasp the object according to the angle.

Description

Technical Field

[0001] The present invention relates to an object intelligent detection method and system based on Yolo improvement, belonging to the technical field of intelligent detection. Background Art

[0002] Since 2006, under the leadership of Hinton, Bengio, Lecun and others, a large number of deep learning papers have been published. In particular, in 2012, AlexNet won the championship in the ImageNet image recognition competition. Since then, neural networks have received widespread attention. Image detection is one of the basic tasks in the field of computer vision. Traditional image detection algorithms generally require three stages: region selection, feature extraction, and classification. They are not very ideal in terms of prediction accuracy and speed. With the rise of deep learning, image detection has ushered in rapid development. The mainstream detection algorithms are divided into two categories, namely one-stage detection algorithms and two-stage detection algorithms.

[0003] The one-stage detection algorithm can directly generate category probabilities and location coordinate values ​​in one stage without the need for the region proposal stage. Typical algorithms include the yolo series, SSD, and centernet. The two-stage detection algorithm divides the detection problem into two stages. The first stage generates candidate regions, including the approximate location information of the target object, and then classifies and adjusts the location of the candidate regions in the second stage. Typical algorithms include R-CNN, Fast R-CNN, and FasterR-CNN. In terms of speed, the one-stage detection algorithm is superior to the two-stage detection algorithm, but slightly inferior in accuracy.

[0004] The current image detection algorithm can achieve the recognition and positioning of objects, but it does not detect the angle of the object. When you need to use a mechanical gripper or other equipment to grasp an object, the angle of the object is very important for the placement of the mechanical gripper.

[0005] In summary, how to detect the angle of an object while ensuring the accuracy of the object detection is a problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide an improved object intelligent detection method and system based on YOLO, which can detect the angle of an object without reducing the accuracy of object recognition.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0008] In a first aspect, the present invention provides an object intelligent detection method based on yolo improvement, comprising the following steps:

[0009] Get the image of the object to be tested;

[0010] Input the image of the object to be tested into the trained intelligent recognition network model to obtain the detection result and the corresponding object angle;

[0011] The intelligent recognition network model is based on the YOLO network model, and an attention module is provided on the output end to recognize the angle of the object.

[0012] Furthermore, the intelligent recognition network model is built based on the deep learning framework Pytorch, based on the yolo network model, and adopts the Darknet-53 model structure, including 53 convolutional layers, 23 residual modules, 5 downsamplings, 3 prediction result output terminals at different scales, and an attention module set on each prediction result output terminal.

[0013] Furthermore, the output scales of the prediction result output terminals at the three different scales are 52*52, 26*26, and 13*13 respectively.

[0014] Furthermore, the training method of the intelligent recognition network model includes:

[0015] Collecting object images to form a data set, expanding the data set using a data enhancement method, annotating each image in the data set with the class, position, and angle information of the object, and dividing the data set into a training set and a test set;

[0016] The images of the training set and the test set are input into the intelligent recognition network model for training and testing to obtain a trained intelligent recognition network model.

[0017] Furthermore, the method of collecting object images to form a data set and expanding the data set using a data enhancement method includes:

[0018] Collect pictures of objects in different environments, sizes, angles, and lighting conditions as data sets;

[0019] The data set is expanded by random vertical flipping, random rotation of any angle, random scaling, random horizontal flipping, horizontal translation and vertical translation.

[0020] Further, the method of dividing the data set into a training set and a test set includes:

[0021] The data set is shuffled and randomly divided into training set and test set in a ratio of 9:1.

[0022] Furthermore, the images of the training set and the test set are input into the intelligent recognition network model, and the method for training and testing includes:

[0023] The images in the training set are uniformly scaled to a size of 416*416, and each pixel value in the image is multiplied by 1 / 255 so that the pixel value is between 0 and 1;

[0024] Initialize the weights in the network model and load the pre-trained weights of the yolo network model;

[0025] The scaled image data is input into the network model, the initial value of the learning rate is set to 0.001, the number of batch samples is set to 16, the training iteration is 100 rounds, the best weight model is retained, and the trained intelligent recognition network model is obtained.

[0026] Furthermore, the method of inputting the image of the object to be tested into the trained intelligent recognition network model includes:

[0027] The image of the living object to be detected is scaled to 416*416 and input into the trained intelligent recognition network model.

[0028] In a second aspect, the present invention provides an object intelligent detection system based on yolo improvement, the system comprising:

[0029] Image acquisition module: used to obtain images of the object to be tested;

[0030] Detection output module: used to input the image of the object to be tested into the trained intelligent recognition network model to obtain the detection results and the corresponding object angle;

[0031] The intelligent recognition network model is built based on the deep learning framework Pytorch, based on the yolo network model, and adopts the Darknet-53 model structure, including 53 convolutional layers, 23 residual modules, 5 downsampling, 3 prediction result output terminals at different scales, and an attention module set on each prediction result output terminal.

[0032] In a third aspect, the present invention provides an object intelligent detection system based on Yolo improvement, including a processor and a storage medium;

[0033] The storage medium is used to store instructions;

[0034] The processor is used to operate according to the instructions to execute the steps of the method according to the first aspect.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Based on the original yolo network model, the present invention adds an attention module to complete the detection of object angles without reducing the accuracy of object recognition, which is conducive to the mechanical gripper to accurately grasp the object according to the angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 (a) is a picture to be tested, and (b) is a schematic diagram of the output of the relevant annotation information of (a);

[0038] Figure 1 (c) is another image to be tested, and (d) is a schematic diagram of the output of the relevant annotation information of (c);

[0039] Figure 2 This is a schematic diagram of the yolo network structure;

[0040] Figure 3 Schematic diagram of the attention module;

[0041] Figure 4 Schematic diagram of the Yolo network with the attention module added;

[0042] Figure 5 is a flow chart of the method of the present invention;

[0043] Figure 6 This is the block diagram of the intelligent detection system. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0045] Embodiment 1:

[0046] This embodiment provides an object intelligent detection method based on Yolo improvement, comprising the following steps:

[0047] Get the image of the object to be tested;

[0048] Input the image of the object to be tested into the trained intelligent recognition network model to obtain the detection results and the corresponding object angle.

[0049] like Figure 1As shown in the figure, this example collects pictures of common daily necessities in different sizes, angles, and lighting conditions, such as beverage bottles and cartons. The data set is expanded using data enhancement methods, including random rotation of any angle, random upside-down flipping, and random horizontal flipping. The position, category, and angle information of the objects in each picture are annotated using open source tools such as labelme and labelbox, and an XML file is generated. Finally, the data set pictures are shuffled and divided into a training set and a test set in a ratio of 9:1.

[0050] The angle information of the object in the image is based on the horizontal line. If the object is tilted upward, it is a positive angle, and if it is tilted downward, it is a negative angle. The angle range is between -90 degrees and 90 degrees, with 0 degrees for horizontal and -90 degrees or 90 degrees for vertical.

[0051] like Figure 2 As shown, the intelligent recognition network model of this embodiment uses the yolo network as the basic network, and uses the deep learning framework Pytorch to build the Darknet-53 network structure, which contains 53 convolutional layers, 23 residual modules, and 5 downsampling. Figure 2 It can be seen that the YOLO network absorbs the advantages of the FPN network and has three scales of output, 52*52, 26*26, and 13*13. Different output scales correspond to objects of different sizes, which enhances the detection effect of small objects to a certain extent. For example, the 52*52 scale output is used to detect large objects, the 26*26 scale output is used to detect medium-sized objects, and the 13*13 scale output is used to detect small objects.

[0052] The network uses a large number of BatchNormalization layers and residual modules, and uses LeakyReLu as the activation function. There is a tensor splicing operation on the prediction branch, which is implemented by splicing through the upsampling of the middle layer of the network and a layer after the middle layer. The update of the Yolo network weights uses gradient descent and back propagation like the BP neural network. The following is the gradient formula for the convolution layer and the pooling layer:

[0053]

[0054] Where E represents the result of matrix product, w ij Represents each element in the matrix, v ij Specific corresponding w ij The matrix elements of δ ij For v ij is the gradient size of the result.

[0055] Find δ ij and The expression of is as follows:

[0056] δ l-1 =conv2(rot180(w l ),δ l ,'full')φ l (v l-1 )

[0057] δ, w, v are in matrix form. In the formula, the rot180() function represents the matrix w l Rotate 180 degrees counterclockwise, the parameter full indicates a full convolution operation. The calculation of the BatchNormalization layer is as follows:

[0058]

[0059]

[0060] Where input x=(x (1) …x (d) ), is the output of batch normalization, are the expectation and mean of x respectively. γ and β are two parameters introduced to control the effect of normalization. ε is a value greater than 0 to prevent the division by 0. i is the output result of the BatchNormalization layer.

[0061] The relative position method is used to predict the position, predicting the relative coordinates of the center point of the box relative to the upper left corner of the grid unit, and directly predicting (t x ,t y ,t w ,t h ), and then calculate the position and confidence of the box through the following offset formula.

[0062]

[0063]

[0064]

[0065]

[0066] t x ,t y ,t w ,t h is the predicted output of the model, c x 、c y They represent the coordinates of the grids respectively. For example, if the size of the feature map is 13*13, then there are 13*13 grids. The grid coordinates c in the first row and first column arex 、c y It is 1, 1; p w and p h Indicates the box size before prediction. b x 、b y 、b w 、b h Represents the center coordinates and corresponding width and height of the predicted object box.

[0067] like Figure 3 As shown, this example uses the attention module as the successor of the yolo network output. On the basis of yolo completing the position and category detection of the object category, the angle information of the object is detected. Attention improves the disadvantage of slow RNN training and uses the self-attention mechanism for fast parallel training. Attention can be used to replace the new network structure of RNN and CNN, which can directly obtain global information. The structure of attention mainly consists of two parts, namely the encoding module and the decoding module. This example uses the attention module to detect the angle information of objects in the picture. As shown in the figure, for the attention module, the three matrices Q (query), K (key), and V (value) come from the same input, which corresponds to the input matrix entering the attention module. First, calculate the dot product of Q and K. In order to prevent the result from being too large, divide it by a scale. where d k is the dimension of query and key vectors, and finally the result is normalized by softmax and then multiplied by matrix V to get the weighted sum representation. As shown below:

[0068]

[0069] like Figure 4 The figure shows a schematic diagram of the YOLO network model with an attention module added. During training, first freeze the attention module and load the pre-trained weights on the coco or voc dataset. You can also start training the model from scratch, which is slightly slower than the model with pre-trained weights loaded. After loading the pre-trained model, scale the images in the training set to a size of 416*416, and multiply each pixel value in the image by 1 / 255 so that the pixel value is between 0 and 1. Input the scaled image data into the network model, set the initial learning rate to 0.001, the batch size to 16, train for 100 iterations, and retain the best weight model. During network training, the loss function is as follows:

[0070]

[0071] Assume that there are S*S grids in total, and each grid generates B candidate boxes. In the formula, the parameter Respectively represent the center coordinates x, y, the true value and the predicted value. Indicates whether the jth box of the i-th grid is responsible for detecting this object. If it is responsible for detecting the object, the value is 1. Similarly, I ij nobj It means that the jth box in the i-th grid is not responsible for detecting objects. The parameter confidence C in training represents the true value. The value of C is determined by whether the box in the grid detects an object. If it is responsible for detection, the value is 1, otherwise, the value is 0. The loss function is mainly divided into two parts, with objects and without objects. A weight coefficient is also added to the loss part without objects. The weight coefficient is added because most of the content in a picture does not include objects. In order to make the contribution of the calculation part without objects less than the part with objects, a suitable weight system must be selected so that the network tends to be inclined to the part with objects during training. The general value can be 0.5, 0.6, etc. For the prediction of angles, the square loss function is used for calculation. Because the marked angle size is between -90 degrees and 90 degrees, 90 degrees are added during training, and finally multiplied by 1 / 180 to make the angle value between 0 and 1. When the actual prediction result is multiplied by 180, then 90 degrees are subtracted to invert the true predicted angle size. During training, the training images are uniformly scaled to a size of 416*416, and each pixel value in the image is multiplied by 1 / 255, so that the pixel value is between 0 and 1. The scaled image data is input into the network model, the initial value of the learning rate is set to 0.001, the number of batch samples is 16, the training iteration is 100 rounds, and the best weight model is retained.

[0072] After the model is trained, the image to be detected is scaled to a size of 416*416, the network model is input, the optimal weight model is loaded, the detection results are obtained, and the angle information of each object detected is printed on the terminal.

[0073] Embodiment 2:

[0074] This embodiment provides an object intelligent detection system based on Yolo improvement, the system comprising:

[0075] Image acquisition module: used to obtain images of the object to be tested;

[0076] Detection output module: used to input the image of the object to be tested into the trained intelligent recognition network model to obtain the detection results and the corresponding object angle.

[0077] The intelligent recognition network model is built based on the deep learning framework Pytorch, based on the yolo network model, and adopts the Darknet-53 model structure, including 53 convolutional layers, 23 residual modules, 5 downsampling, 3 prediction result output terminals at different scales, and an attention module set on each prediction result output terminal.

[0078] Embodiment three:

[0079] The embodiment of the present invention also provides an object intelligent detection system based on Yolo improvement, including a processor and a storage medium;

[0080] The storage medium is used to store instructions;

[0081] The processor is used to operate according to the instructions to execute the steps of the method described in embodiment 1.

[0082] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0083] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An object intelligent detection method based on yolo improvement, characterized in that: The following steps are involved: Get the image of the object to be tested; Input the image of the object to be tested into the trained intelligent recognition network model to obtain the detection result and the corresponding object angle; The intelligent recognition network model is based on the YOLO network model, and an attention module is provided on the output end to recognize the angle of the object; The training method of the intelligent recognition network model includes: Collecting object images to form a data set, expanding the data set using a data enhancement method, annotating each image in the data set with the class, position, and angle information of the object, and dividing the data set into a training set and a test set; Input the images of the training set and the test set into the intelligent recognition network model for training and testing to obtain a trained intelligent recognition network model; The training set and test set images are input into the intelligent recognition network model. The training and testing methods include: The images in the training set are uniformly scaled to a size of 416*416, and each pixel value in the image is multiplied by 1 / 255 so that the pixel value is between 0 and 1; Initialize the weights in the network model and load the pre-trained weights of the yolo network model; Input the scaled image data into the network model, set the initial value of the learning rate to 0.001, the number of batch samples to 16, train for 100 rounds, retain the best weight model, and obtain the trained intelligent recognition network model; During network training, the loss function is as follows: There are S*S grids in total, and each grid generates B candidate boxes. In the formula, the parameters x, y, Respectively represent the center coordinates x, y, the true value and the predicted value; parameter Indicates whether the jth box of the i-th grid is responsible for detecting this object. If it is responsible for detecting the object, the value is 1, otherwise it is 0; ij nobj It means that the jth box in the i-th grid is not responsible for detecting objects. If it is not responsible for detecting objects, the value is 1, otherwise it is 0; The parameter confidence C in training represents the true value. The value of C is determined by whether the box in the grid detects an object. If it is responsible for detection, the value is 1, otherwise, the value is 0; The intelligent recognition network model is built based on the deep learning framework Pytorch, based on the yolo network model, and adopts the Darknet-53 model structure, including 53 convolutional layers, 23 residual modules, 5 downsampling, 3 prediction result output terminals at different scales, and an attention module set on each prediction result output terminal; The output scales of the prediction result output terminals at the three different scales are 52*52, 26*26, and 13*13 respectively; The method of collecting object images to form a data set and expanding the data set using a data enhancement method includes: Collect pictures of objects in different environments, sizes, angles, and lighting conditions as data sets; Expanding the data set by randomly flipping vertically, randomly rotating the data set at any angle, randomly scaling the data set, randomly flipping horizontally, horizontally translating the data set, and vertically translating the data set; Methods for splitting a dataset into training and testing sets include: The data set is shuffled and randomly divided into training set and test set in a ratio of 9:1; The method of inputting the image of the object to be tested into the trained intelligent recognition network model includes: The image of the living object to be detected is scaled to 416*416 and input into the trained intelligent recognition network model.

2. A Yolo-based intelligent object detection system for executing the method as claimed in claim 1, characterized in that: The system comprises: Image acquisition module: used to obtain images of the object to be tested; Detection output module: used to input the image of the object to be tested into the trained intelligent recognition network model to obtain the detection results and the corresponding object angle; The intelligent recognition network model is built based on the deep learning framework Pytorch, based on the yolo network model, and adopts the Darknet-53 model structure, including 53 convolutional layers, 23 residual modules, 5 downsampling, 3 prediction result output terminals at different scales, and an attention module set on each prediction result output terminal.

3. An object intelligent detection system based on yolo improvement, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to claim 1 .

Images