Multi-target localization method based on yolov3 and convolutional neural network

Abstract

The present invention provides a multi-target positioning method based on YOLOv3 and a convolutional neural network, comprising an offline training phase and an online positioning phase. The offline training phase divides sub-areas to obtain training data; an RGB color map is trained using a YOLOv3 network to establish a target detection model; depth images are segmented and fused according to the target detection results to obtain a training data fingerprint; finally, a position estimation model is obtained using a convolutional neural network; in the online positioning phase, image fusion is performed using the target detection model to obtain an online data fingerprint; and the position estimation model of each target is determined to obtain the target position. The present invention uses a YOLOv3 network and a convolutional neural network to establish a position estimation model in the offline training phase, and then segments and fuses the depth images to extract the training positioning fingerprint. In the online phase, the online positioning fingerprint and the position estimation model are used to complete multi-target positioning, thereby solving the problems of insufficient prediction flexibility and accuracy of existing target positioning methods.

Description

Technical Field

[0001] The present invention relates to a multi-target positioning method based on YOLOv3 and a convolutional neural network, and belongs to the field of deep learning. Background Art

[0002] The research on multi-target recognition and localization technology has always been one of the most basic and challenging research topics in the field of Computer Vision (CV).

[0003] Currently, there are two main types of multi-target recognition technologies: radar target recognition and image target recognition. The former has garnered significant attention in scenarios such as autonomous driving and military applications because the point clouds captured by lidar (LiDAR) can retain rich, original geometric information about objects. However, because lidar point clouds are sparse and unstructured, processing them requires more computing power than image recognition. Therefore, image target recognition is often used in everyday scenarios.

[0004] In recent years, image object recognition technology has achieved significant breakthroughs. Popular algorithms can be divided into two categories. One is based on Region Proposal (R-CNN) algorithms. These are two-stage algorithms that require: 1) generating candidate object bounding boxes (i.e., the object locations); and 2) performing classification and regression on these bounding boxes. Region proposals (RPs) are generated first, representing a preselected bounding box that may contain the object to be detected. A convolutional neural network is then used to classify the samples. Common two-stage object recognition algorithms include R-CNNs, SPP-Net, Fast R-CNNs, Faster R-CNNs, and R-FCNs. The other category is one-stage algorithms such as YOLO and SSD, which use a single convolutional neural network to directly predict the class and location of different objects. They do not utilize RPs, but instead extract features directly from the network to predict object class and location. Common one-stage object recognition algorithms include OverFeat, YOLOV1, YOLOV2, YOLOV3, SSD, and RetinaNet. The first type of method is more accurate but slower, while the second type of method is faster but less accurate.

[0005] Multi-target positioning has a long history of development. Existing positioning technologies include infrared, ultrasonic, Bluetooth, ZigBee, ultra-wideband (UWB), and WiFi, each with its own advantages and disadvantages. With the advancement of various cameras and deep learning technologies, multi-target positioning is also moving towards deep learning-based visual positioning.

[0006] Compared to traditional positioning techniques, which suffer from poor stability and significant noise interference, convolutional neural networks offer a wider range of potential predictor variables and richer functional form specifications. This flexibility has pushed the boundaries of positioning technology, and academic interest in deep learning methods for positioning has also increased. Among deep learning techniques, convolutional neural networks offer increased flexibility in data prediction using various high-dimensional models for statistical prediction, which can better approximate the complex environmental processes of real life. However, with this increased flexibility comes a greater potential for overfitting the data. Therefore, several so-called "regularization" methods for model selection and mitigation of overfitting have been developed to improve the deep learning fitting process. These improvements emphasize stable out-of-sample performance, explicitly preventing overfitting. Finally, for many predictor variables, efficient search algorithms can be used to simulate potential model operations, approximating the optimal prediction specification while keeping costs low.

[0007] In view of this, it is necessary to provide a multi-target positioning method based on yolov3 and convolutional neural network to solve the above problems. Summary of the Invention

[0008] The purpose of the present invention is to provide a multi-target positioning method based on YOLOv3 and convolutional neural network to solve the problems of insufficient prediction flexibility and accuracy of existing target positioning methods.

[0009] To achieve the above objectives, the present invention provides a multi-target positioning method based on yolov3 and convolutional neural network, which includes two stages: offline training stage and online positioning stage.

[0010] The offline training phase includes:

[0011] Step 1: Divide the positioning interval into several sub-intervals and use the camera to obtain training data for each sub-interval. The training data includes point cloud information, RGB color map and depth image;

[0012] Step 2: Use the YOLOv3 network to perform offline training on the RGB color map for multi-target detection and recognition, obtain the training results, and establish a multi-target detection model;

[0013] Step 3: Based on the target recognition results output by the multi-target detection model, the depth image is cut and fused to extract the training positioning fingerprint of each target;

[0014] Step 4: Use convolutional neural network to perform offline regression training to obtain the position estimation model of each sub-interval;

[0015] The online orientation phase includes:

[0016] Step 5: Use the camera to collect online data, which includes point cloud information, RGB color map and depth image;

[0017] Step 6: Send the RGB color map obtained in step 5 to the multi-target detection model in step 2, output the target detection result, and complete the target recognition;

[0018] Step 7: Using the target detection results obtained in step 6, segment and fuse the depth image obtained in step 5 to obtain the online positioning fingerprint of each target;

[0019] Step 8: Using the point cloud information of each target obtained in step 5, determine the position estimation model of step 4 applicable to each target, and combine it with the online positioning fingerprint described in step 7 to complete multi-target positioning.

[0020] As a further improvement of the present invention, step 1 specifically includes:

[0021] Step 11: Divide the positioning interval into several sub-intervals based on the point cloud information of different positions in the camera;

[0022] Step 12: In each subinterval, the target is located at different reference points, and the camera is used to obtain training information. The training information includes at least the point cloud information, RGB color map, and depth image of the target.

[0023] As a further improvement of the present invention, step 2 specifically includes:

[0024] Step 21: Label the collected RGB color map with the target label to create a training data set;

[0025] Step 22: Use the yolov3 network to perform classification learning training on the training data set in step 21, obtain the training results, and establish a multi-target detection model.

[0026] As a further improvement of the present invention, step 3 specifically includes:

[0027] Step 31: Based on the output of the box information of each target by the multi-target detection model in step 2, the position information of each target in the RGB color map of the training data is obtained;

[0028] Step 32: According to the position of the target in the RGB color map, for each target recognition result, find the position of the target corresponding to the depth image in step 1, and set the pixel to 1, and set the pixels at other positions of the depth image to 0, completing the segmentation of the depth image and obtaining a separated depth map;

[0029] Step 33: Add the separated depth map based on target segmentation to the pixel values ​​of the corresponding positions of the original depth image obtained in step 1, perform pixel-level fusion, integrate the target position features into the original depth image, and form a training positioning fingerprint for each target.

[0030] As a further improvement of the present invention, step 4 specifically includes:

[0031] Step 41: The training positioning fingerprint and corresponding location information of each target obtained in step 3 are fed into a convolutional neural network for feature extraction. After passing through the convolutional layer and the pooling layer, the tensor after the last pooling layer is extracted and flattened into a 1024-dimensional feature vector.

[0032] Step 42: Send the 1024-dimensional feature vector to the fully connected network, and use the fully connected network to perform offline regression learning based on the position of each sub-interval to obtain a regression prediction model.

[0033] As a further improvement of the present invention, in step 41, the convolutional neural network includes four convolutional layers and three pooling layers.

[0034] As a further improvement of the present invention, in step 42, the fully connected network of the convolutional neural network includes four fully connected layers.

[0035] As a further improvement of the present invention, step 5 specifically includes:

[0036] Step 51: using the camera to divide the positioning interval into several sub-intervals according to the point cloud information of different positions in the camera;

[0037] Step 52: In each subinterval, the target is located at a different reference point, and a camera is used to obtain online information, which includes at least point cloud information, RGB color map, and depth image of the target.

[0038] As a further improvement of the present invention, step 7 specifically includes:

[0039] Step 71: Based on the output of the box information of each target by the multi-target detection model in step 6, the position information of each target in the RGB color map of the online data is obtained;

[0040] Step 72: According to the position of the target in the RGB color map, for each target recognition result, find the position of the target corresponding to the depth image in step 5, and set the pixel to 1, and set the pixels at other positions in the depth image to 0, completing the segmentation of the depth image to obtain a separated depth map;

[0041] Step 73: Add the segmented depth map based on target segmentation to the pixel values ​​at the corresponding positions of the original depth image obtained in step 5, perform pixel-level fusion, integrate the target position features into the original depth image, and form an online positioning fingerprint for each target.

[0042] As a further improvement of the present invention, step 8 specifically includes:

[0043] Step 81: Using the box information of each target output by the multi-target detection model, obtain the center pixel of the box and extract its corresponding depth information in the point cloud information;

[0044] Step 82: In order to avoid the phenomenon of holes in the depth information of pixels in the RGB color image in the point cloud information, assume that the center pixel position of the box is selected as (x, y), and select the middle value of the depth information of five pixels with position coordinates (x, y), (x-1, y), (x+1, y), (x, y-1), and (x, y+1) in the RGB color image in the point cloud information.

[0045] The beneficial effects of the present invention are as follows: compared with the existing technology, the multi-target positioning method based on YOLOv3 and convolutional neural network of the present invention establishes a position estimation model through the YOLOv3 network and the convolutional neural network in an offline training stage, and cuts and fuses the depth image, extracts the training positioning fingerprint, and uses the online positioning fingerprint and the position estimation model in the online stage to complete multi-target positioning, thereby solving the problems of insufficient prediction flexibility and insufficient accuracy of the existing target positioning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a step diagram of a multi-target positioning method based on yolov3 and convolutional neural network in a preferred embodiment of the present invention.

[0047] Figure 2 It is a flow chart of the present invention.

[0048] Figure 3 It is a schematic diagram of the structure of the yolov3 network of the present invention.

[0049] Figure 4 This is a schematic diagram of the target result output based on yo l ov3 of the present invention.

[0050] Figure 5 It is a schematic diagram of the color depth image segmentation result of the present invention.

[0051] Figure 6 2 is a schematic diagram of image fusion of color depth images of the present invention.

[0052] Figure 7 It is a schematic diagram of the image fusion result of the color depth image of the present invention.

[0053] Figure 8 It is a structural diagram of the convolutional neural network of the present invention.

[0054] Figure 9 It is a schematic diagram of describing target depth information of the present invention.

[0055] Figure 10 Schematic diagram of target depth information based on point cloud measurement values ​​of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] It should be noted here that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0058] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0059] See also Figure 1 and Figure 2 As shown, the present invention provides a multi-target positioning method based on yolov3 and convolutional neural network, including two stages: offline training stage and online positioning stage.

[0060] The offline training phase includes:

[0061] Step 1: Use the camera to collect training data, which includes point cloud information, RGB color map and depth image;

[0062] Step 2: Use the Yo L OV3 network to perform offline training for multi-target detection and recognition, obtain the training results, and establish a multi-target detection model;

[0063] Step 3: Use the convolutional neural network to perform offline regression training of multiple target positions based on sub-intervals on the training results obtained in step 2;

[0064] Step 4: Use convolutional neural network to perform offline regression training to obtain the position estimation model of each sub-interval;

[0065] Step 1 specifically includes:

[0066] Step 11: Use the camera to divide the positioning interval into several sub-intervals based on the point cloud information of different positions in the camera;

[0067] Step 12: In each subinterval, the target is located at different reference points, and the camera is used to obtain training information. The training information includes at least the point cloud information, RGB color map, and depth image of the target.

[0068] Step 2 specifically includes:

[0069] Step 21: Label the collected RGB color map with the target label to create a training data set. Specifically, in this application, the label Img tool is used to label the RGB color map collected by the camera with label to create a training data set for target recognition.

[0070] Step 22: Use the Yolov3 network to perform classification learning training on the training data set, obtain training results, and establish a multi-target detection model. Specifically, the training data set prepared in step 21 is fed into the Yolov3 network in a ratio of 8:1:1 (training set: validation set: test set) to establish a multi-target detection model.

[0071] For details, see Figure 3 As shown in the figure, the Yolov3 algorithm is a comprehensive improvement based on Yolov1 and Yolov2. This algorithm has achieved good advantages in both running time and detection accuracy. Compared with the previous two versions, the Yolov3 algorithm adjusted the network structure and adopted the Darknet-53 network structure.

[0072] The backbone of Yolov3 has evolved from Darknet-19 in the Yolov2 era to Darknet-53, which has deepened the number of network layers and introduced the cross-layer addition operation in Resnet. Figure 2 The difference between the concat operation and the sum operation is that the sum operation, derived from the ResNet concept, adds the input feature map to the corresponding dimension of the output feature map, i.e., y = f(x) + x. The concat operation, derived from the design of the DenseNet network, directly concatenates the feature maps according to the channel dimension. Upsampling (unsampling) converts a small-scale feature map into a large-scale image through interpolation or other methods. Upsampling does not change the number of channels in the feature map.

[0073] The input image resolution of Darknet-53 is 416×416. After being downsampled by 32 times, 16 times, and 8 times, the sizes of the three feature layers used for prediction are 13×13, 26×26, and 52×52 respectively. These three feature output maps are convolved and predicted by Num 1×1 convolution kernels. The calculation formula of Num is as follows:

[0074] Num=(4+1+class)×k

[0075] Among them, k is the number of preset bounding boxes, which generally defaults to 9, and class is the number of categories of the predicted target.

[0076] Compared with the previous two versions, the yolov3 algorithm has three improvements: 1. The network structure is adjusted and the Darknet-53 network structure is adopted; 2. Three feature maps of different scales are used for target detection, as shown in Table 1; 3. The logistic regression algorithm is used to predict the target bounding box.

[0077] Table 1 Feature map preset bounding box parameters

[0078]

[0079] See also Figure 4 As shown in the figure, the present invention is based on the output of the Yolov3 target result diagram, the dotted rectangular box in the figure is the preset bounding box, and the solid rectangular box is the predicted bounding box calculated by the offset predicted by the network. x ,c y ) is the center coordinate of the preset bounding box on the feature map, (p w ,p h ) is the width and height of the preset bounding box on the feature map, (t x ,t y ,t w ,t h ) are the center offsets of the bounding box predicted by the network (t x ,t y ) and aspect ratio (t w ,t h ), (b x ,b y ,b w ,b h ) is the final predicted target bounding box. The conversion process from the preset bounding box to the final predicted bounding box is shown in the following formula, where the σ(x) function is a sigmoid function whose purpose is to scale the predicted offset to between 0 and 1:

[0080] b x =σ(t x )+c x

[0081] b y =σ(t y )+c y

[0082]

[0083]

[0084] In addition, the loss function is improved in the yolov3 algorithm, and the binary cross entropy loss function is used to represent the category loss. The loss function of yolov3 is mainly divided into three parts: target positioning offset loss L loc (l,g), target confidence loss L conf (o,c) and target classification loss L cla (O,C), where λ1, λ2, λ3 are balance coefficients:

[0085] L(O,o,C,c,l,g)=λ1L conf (o,c)+λ2L cla (O,C)+λ3L loc (l,g)

[0086] The target confidence can be understood as the probability of the target existing in the predicted target rectangle. The target confidence loss L conf (o,c) uses binary cross entropy loss (Binary Cross Entropy), where o i ∈{0,1}, indicates whether the target actually exists in the predicted target bounding box i, 0 means no target exists, and 1 means yes target exists. Indicates the Sigmoid probability of predicting whether there is a target in the target rectangle i:

[0087]

[0088]

[0089] Target category loss L cla (O, C) also uses binary cross entropy loss, where O ij ∈{0,1}, indicates whether the j-th type of target actually exists in the predicted target bounding box i, 0 means no, 1 means yes. Indicates the Sigmoid probability of the jth type of target in the network prediction target bounding box i:

[0090]

[0091]

[0092] Target localization loss L loc (l,g) uses the sum of the squares of the difference between the true deviation value and the predicted deviation value, where Indicates the predicted rectangular frame coordinate offset, Indicates the coordinate offset between the matching GTbox and the default box, (b x,b y ,b w ,b h ) is the predicted target rectangle parameter, (c x ,c y ,c w ,c h ) is the default rectangular frame parameter, (g x ,g y ,g w ,g h ) are the parameters of the true target rectangle that matches it, and these parameters are mapped on the predicted feature map:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] See also Figures 5 to 7 As shown, step 3 specifically includes:

[0099] Step 31: Based on the multi-target recognition results, the depth image is cut and fused to extract the training positioning fingerprint of each target;

[0100] Step 31 specifically includes:

[0101] Step 311: Based on the output of the multi-target detection model in step 2 for each target box information, the position information of each target in the RGB color map of the training data is obtained, where the output target box information includes key information such as the upper left and lower right coordinates of the target, confidence rate, target category, etc.

[0102] Step 312: For each target recognition result, find the position of the target in the depth image obtained in step 1 according to the position of the target in the RGB color map, set the pixel to 1, and set the pixels at other positions in the depth image to 0, completing the segmentation of the depth image and obtaining a segmented depth map;

[0103] Step 312 primarily involves image fusion, using the OpenCV masking technique for image segmentation. The specific steps are as follows: First, based on the information output by the target detection model after inputting the test image, the coordinates of the segmented regions are extracted. Then, a mask is manually created based on the segmented region coordinates. Pixel values ​​within the mask are set to 255, and pixel values ​​outside the mask are set to 0. Finally, the mask is bitwise ANDed with the target depth map to perform image segmentation.

[0104] Step 313: Add the segmented depth map based on target segmentation to the pixel values ​​at the corresponding positions of the original depth image obtained in step 1, perform pixel-level fusion, integrate the target position features into the original depth image, and form a training positioning fingerprint for each target.

[0105] Step 313 mainly performs image fusion, which is to fuse the separated depth map with the original depth image obtained in step 1. The image fusion technology used in the present invention is the linear mixing technology in OpenCV, and its principle is as follows: Assuming that there are two image matrices f0(x) and f1(x), when the two images are fused, their respective weights are α and β, then the calculation formula for the pixel channel value in the fused target image g(x) is:

[0106] g(x)=αf0(x)+βf1(x)+γ

[0107] There are no mandatory requirements for the weights α and β of the two images in the above formula, but it is generally recommended that α + β = 1. γ is the correction coefficient and is set to 0 by default.

[0108] Step 4 specifically includes:

[0109] Step 41: The training positioning fingerprint and corresponding location information of each target obtained in step 31 are fed into a convolutional neural network for feature extraction. After passing through the convolutional layer and the pooling layer, the tensor after the last pooling layer is extracted and flattened into a 1024-dimensional feature vector.

[0110] Step 42: Send the 1024-dimensional feature vector to the fully connected network, and use the fully connected network to perform offline regression learning based on the position of each sub-interval to obtain a regression prediction model.

[0111] In step 41, the convolutional neural network includes four convolutional layers and three pooling layers.

[0112] In step 42, the fully connected network of the convolutional neural network includes four fully connected layers.

[0113] For details, see Figure 8As shown in the figure, the structure of the convolutional neural network adopted in the specific implementation of the present invention is described. Convolutional neural network (CNN) is a widely used deep learning technology with excellent image recognition performance. Convolutional neural networks and ordinary neural networks have many similarities. They both imitate the structure of human nerves and are composed of neurons with learnable weights and bias constants. Each neuron can receive an input signal and output the score of each classification after calculation. However, the input of a convolutional neural network is generally an image. The convolutional network performs convolution pooling, successfully reducing the dimensionality of images with large amounts of data, and ultimately enabling it to be trained. The characteristic of a convolutional neural network is that it is composed of one or more convolutional layers and pooling layers. Convolutional neural networks have three unique concepts: local filters, maximum pooling, and weight sharing. There are mainly three types of layers in a convolutional neural network, namely convolutional layers, pooling layers, and fully connected layers.

[0114] Assume that a picture X in the training set is the input of the convolutional neural network, s(i, j) is the feature matrix of the i-th layer of the convolutional neural network, and s(i, j) can be expressed as the following formula:

[0115]

[0116] Where n is the number of input matrices, or the dimension of the last dimension of the tensor. k Represents the kth input matrix. W k Represents the kth subconvolution kernel matrix of the convolution kernel. s(i,j) is the value of the corresponding position element of the output matrix corresponding to the convolution kernel W, and b represents the offset matrix of the i-th layer.

[0117] After the convolutional layer, the convolutional neural network enters the pooling layer. This layer downsamples the feature maps calculated by the convolutional layer, reducing the complexity of the network model and balancing rapidly changing feature maps. After multiple convolution and sampling operations, the fully connected layer of the convolutional neural network finally generates a one-dimensional feature vector for subsequent processing.

[0118] The two activation functions ReLU and Elu that are often used in the entire convolution pooling process are as follows:

[0119] ReLU activation function:

[0120]

[0121] Elu activation function:

[0122]

[0123] The ReLU activation function is a piecewise linear function with unilateral inhibition, which gives neurons sparse activation. The ReLU activation function has negative values, which push the output mean of the activated unit closer to 0, achieving batch normalization and reducing computational complexity. Convolutional training is prone to overfitting and produces simplistic models. Therefore, a Dropout layer should be added after the fully connected layer, with a typical value range of 0.4-0.6. The model should be trained using the backpropagation algorithm.

[0124] Take a single picture as an example: the input picture data format is 128×128×3. After the first convolution (3×3×32, stride=1), the data becomes 128×128×32; after the second convolution (3×3×32, stride=2), the data becomes 64×64×32. After the first pooling, the data becomes 32×32×32. After the third convolution (3×3×64, stride=2), the data becomes 16×16×64. After the second pooling, the data becomes 8×8×64; after the fourth convolution (3×3×128, stride=2), the data becomes 4×4×128, and after the third pooling, the data becomes 2×2×128; after straightening, it is connected through the first fully connected layer, and then 1024 features are output, and after the second fully connected layer, 256 features are output, and then after the third fully connected layer, 64 features are output, and finally after the fourth fully connected layer, 1 feature is output, that is, the final feature vector.

[0125] The online orientation phase includes:

[0126] Step 5: Use the camera to collect online data, which includes point cloud information, RGB color map and depth image;

[0127] Step 6: Send the RGB color map obtained in step 5 to the multi-target detection model in step 2, output the target detection result, and complete the target recognition;

[0128] Step 7: Using the target detection results obtained in step 6, segment and fuse the depth image obtained in step 5 to obtain the online positioning fingerprint of each target;

[0129] Step 8: Using the point cloud information of each target obtained in step 5, determine the position estimation model applicable to each target in step 4, and combine it with the online positioning fingerprint in step 7 to complete multi-target positioning.

[0130] The collection process of step 5 is similar to that of step 1, including:

[0131] Step 51: using the camera to divide the positioning interval into several sub-intervals according to the point cloud information of different positions in the camera;

[0132] Step 52: In each subinterval, the target is located at a different reference point, and a camera is used to obtain online information, which includes at least point cloud information, RGB color map, and depth image of the target.

[0133] Step 7 specifically includes:

[0134] Step 71: Based on the output of the box information of each target by the multi-target detection model in step 6, the position information of each target in the RGB color map of the online data is obtained;

[0135] Step 72: According to the position of the target in the RGB color map, for each target recognition result, find the position of the target corresponding to the depth image in step 5, and set the pixel to 1, and set the pixels at other positions in the depth image to 0, completing the segmentation of the depth image to obtain a separated depth map;

[0136] Step 73: Add the segmented depth map based on target segmentation to the pixel values ​​at the corresponding positions of the original depth image obtained in step 5, perform pixel-level fusion, integrate the target position features into the original depth image, and form an online positioning fingerprint for each target.

[0137] Step 8 specifically includes:

[0138] Step 81: Using the box information of each target output by the multi-target detection model, obtain the center pixel of the box and extract its corresponding depth information in the point cloud information;

[0139] Step 82: In order to avoid the phenomenon of holes in the depth information of pixels in the RGB color image in the point cloud information, assume that the center pixel position of the box is selected as (x, y), and select the middle value of the depth information of five pixels with position coordinates (x, y), (x-1, y), (x+1, y), (x, y-1), and (x, y+1) in the RGB color image in the point cloud information.

[0140] like Figure 9 As shown, the two cameras are placed on the same horizontal line, Zl represents the optical axis of the left camera, Zr represents the optical axis of the right camera, the focal length of the camera is f, b represents the distance between the coordinate origins of the two cameras, d is the distance from point P on the image to the horizontal line of the camera, and the imaging points of any point P on the left and right cameras are Pl and Pr, and the coordinates of Pl and Pr on the x-axis are x1 and x2 respectively. Then, according to the similar triangle theorem, we can deduce:

[0141]

[0142] After transformation, we get:

[0143]

[0144] In the three-dimensional distance measurement in space, the relationship between similar triangles and combined with observation can be concluded:

[0145]

[0146] It can be seen from the above formula that at the final distance In [1], B is the baseline distance between the two cameras, and f is the focal length of the cameras. Therefore, B and f are both fixed values ​​and are only related to d. In other words, the depth calculated by the camera is only the absolute distance from point P to the camera horizon, not the actual distance.

[0147] like Figure 10 As shown in , the point m on the depth map is converted into the point M of the world coordinate, which is the sub-process of converting the depth image to the point cloud. We assume that the center of the depth map is O, and we can use the similarity between the triangle OmO' and OMA in the figure, so we can get Reflect this similarity relationship on the internal reference, that is:

[0148]

[0149]

[0150] In fact, this is also the derivation process of the intrinsic parameter matrix. However, to convert the pixel coordinates into homogeneous coordinates:

[0151]

[0152] Habitually Written on the left, the above formula becomes:

[0153]

[0154] The matrix It is called the internal parameter matrix K. Let P(X, Y, Z), then the above formula can be written as:

[0155] P uv =KTP w

[0156] The present invention uses the above principle to map the coordinates m(u, v) of the center point of the box output by the target recognition model to the point cloud coordinate system M(x, y, z), that is, from the camera coordinate system to the world coordinate system. Since the coordinate origins of the camera coordinate system and the world coordinate system coincide, the same object in the camera coordinate system and the world coordinate system has the same depth. Therefore, in order to prevent the void phenomenon caused by the sudden change of depth information of the pixel points in the RGB color image in the point cloud, it is assumed that the position of the center pixel point of the box is (x, y), and the five pixel points with position coordinates of (x, y), (x-1, y), (x+1, y), (x, y-1), and (x, y+1) in the RGB color image are selected, and the points without void phenomenon are averaged, and their depth information is used for the subsequent determination of the target position estimation model.

[0157] The present invention utilizes a camera to realize the recognition and positioning of multiple targets, fully utilizes the measurement function of existing equipment, does not need to add any measurement hardware, and has the advantages of low hardware cost and high system integration.

[0158] This paper combines the YOLOv3 network with a convolutional neural network to identify and locate multiple targets. The YOLOv3 network improves target detection accuracy, especially for small targets. It also provides location information for target segmentation in depth maps.

[0159] The present invention highlights the target position fingerprint information and improves the target positioning performance by segmenting and fusing the target depth map. At the same time, the positioning interval is divided to obtain multiple positioning interval position estimation models. The depth value of the target point cloud is extracted by using the camera point cloud measurement value, and a more appropriate position estimation model is selected for positioning to improve the positioning accuracy.

[0160] In summary, the multi-target positioning method based on YOLOv3 and convolutional neural network of the present invention establishes a position estimation model through the YOLOv3 network and convolutional neural network through an offline training phase, cuts and fuses the depth image, extracts the training positioning fingerprint, and uses the online positioning fingerprint and position estimation model in the online phase to complete multi-target positioning, thereby solving the problems of insufficient prediction flexibility and insufficient accuracy of existing target positioning methods.

[0161] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-target positioning method based on YOLOv3 and convolutional neural network, including two stages: offline training stage and online positioning stage, characterized by: The offline training phase includes: Step 1: Divide the positioning interval into several sub-intervals and use the camera to obtain training data for each sub-interval. The training data includes point cloud information, RGB color map and depth image; Step 2: Use the YOLOv3 network to perform offline training on the RGB color map for multi-target detection and recognition, obtain the training results, and establish a multi-target detection model; Step 3: Based on the target recognition results output by the multi-target detection model, the depth image is cut and fused to extract the training positioning fingerprint of each target; Step 4: The training positioning fingerprint and corresponding location information of each target obtained in step 3 are fed into the convolutional neural network for offline regression training to obtain the location estimation model of each subinterval; The online orientation phase includes: Step 5: Use the camera to collect online data, which includes point cloud information, RGB color map and depth image; Step 6: Send the RGB color map obtained in step 5 to the multi-target detection model in step 2, output the target detection result, and complete the target recognition; Step 7: Using the target detection results obtained in step 6, segment and fuse the depth image obtained in step 5 to obtain the online positioning fingerprint of each target; Step 8: Using the point cloud information of each target obtained in step 5, determine the position estimation model of step 4 applicable to each target, and combine it with the online positioning fingerprint of step 7 to complete multi-target positioning; Step 3 specifically includes: Step 31: Based on the output of the box information of each target by the multi-target detection model in step 2, the position information of each target in the RGB color map of the training data is obtained; Step 32: According to the position of the target in the RGB color map, for each target recognition result, find the position of the target corresponding to the depth image in step 1, and set the pixel to 1, and set the pixels at other positions of the depth image to 0, completing the segmentation of the depth image and obtaining a separated depth map; Step 33: Add the separated depth map based on target segmentation to the pixel values ​​of the corresponding positions of the original depth image obtained in step 1, perform pixel-level fusion, integrate the target position features into the original depth image, and form a training positioning fingerprint for each target.

2. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 1, characterized in that, Step 1 specifically includes: Step 11: Divide the positioning interval into several sub-intervals based on the point cloud information of different positions in the camera; Step 12: In each subinterval, the target is located at different reference points, and the camera is used to obtain training information. The training information includes at least the point cloud information, RGB color map, and depth image of the target.

3. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 1, characterized in that, Step 2 specifically includes: Step 21: Label the collected RGB color map with the target label to create a training data set; Step 22: Use the yolov3 network to perform classification learning training on the training data set in step 21, obtain the training results, and establish a multi-target detection model.

4. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 1, characterized in that, Step 4 specifically includes: Step 41: The training positioning fingerprint and corresponding location information of each target obtained in step 3 are fed into a convolutional neural network for feature extraction. After passing through the convolutional layer and the pooling layer, the tensor after the last pooling layer is extracted and flattened into a 1024-dimensional feature vector. Step 42: Send the 1024-dimensional feature vector to the fully connected network, and use the fully connected network to perform offline regression learning based on the position of each sub-interval to obtain a regression prediction model.

5. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 4, characterized in that: In step 41, the convolutional neural network includes four convolutional layers and three pooling layers.

6. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 4, characterized in that: In step 42, the fully connected network of the convolutional neural network includes four fully connected layers.

7. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 1, characterized in that, Step 5 specifically includes: Step 51: using the camera to divide the positioning interval into several sub-intervals according to the point cloud information of different positions in the camera; Step 52: In each subinterval, the target is located at a different reference point, and a camera is used to obtain online information, which includes at least point cloud information, RGB color map, and depth image of the target.

8. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 5, characterized in that, Step 7 specifically includes: Step 71: Based on the output of the box information of each target by the multi-target detection model in step 6, the position information of each target in the RGB color map of the online data is obtained; Step 72: According to the position of the target in the RGB color map, for each target recognition result, find the position of the target corresponding to the depth image in step 5, and set the pixel to 1, and set the pixels at other positions in the depth image to 0, completing the segmentation of the depth image to obtain a separated depth map; Step 73: Add the segmented depth map based on target segmentation to the pixel values ​​at the corresponding positions of the original depth image obtained in step 5, perform pixel-level fusion, integrate the target position features into the original depth image, and form an online positioning fingerprint for each target.

9. The multi-target positioning method based on yolov3 and convolutional neural network according to claim 1, characterized in that, Step 8 specifically includes: Step 81: Using the box information of each target output by the multi-target detection model, obtain the center pixel of the box and extract its corresponding depth information in the point cloud information; Step 82: In order to avoid the phenomenon of holes in the depth information of pixels in the RGB color image in the point cloud information, assume that the center pixel position of the box is selected as (x, y), and select the middle value of the depth information of five pixels with position coordinates (x, y), (x-1, y), (x+1, y), (x, y-1), and (x, y+1) in the RGB color image in the point cloud information.

Images