Mobile robot heterogeneous information fusion target positioning and tracking method and system

Abstract

The invention relates to the technical field of target tracking and positioning algorithms, in particular to a mobile robot heterogeneous information fusion target positioning and tracking method and system. According to the technical scheme, the system comprises a control unit MCU, a laser radar, a visual sensor, a visual sensor driver connected with the visual sensor, a visual target tracking and positioning system and a laser radar target positioning system. Tracking and positioning of the target are achieved by fusing the 2D laser radar positioning result and the visual tracking and positioning result, the target is tracked and positioned in a laser radar and visual fusion mode, the sensing ability of the system to the environment is improved, the limitation of a single sensor is avoided, and the system reliability is improved. And meanwhile, HOG features and depth features extracted by a VGG19 convolutional neural network are fused. In a depth feature extraction process, an attention mechanism is inserted into a VGG19 convolutional neural network to improve the reliability of depth features.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking and positioning algorithms, and in particular to a mobile robot heterogeneous information fusion target positioning and tracking method and system. Background Art

[0002] With the update and iteration of various sensors and target positioning and tracking algorithms, people are committed to pursuing the fusion of multiple sensors, the accuracy of each algorithm and the overall stability of the system.

[0003] Patent No. CN117011378A discloses a mobile robot target positioning and tracking method and related equipment. By constructing a map to locate itself and using the YOLO algorithm to detect the target, the detection results are input into StrongSORT to track the target using the appearance features of the target, effectively solving the occlusion problem. However, the actual position of the target is not well handled, nor is the limitation of a single sensor such as the visual sensor being easily affected by light considered.

[0004] Patent No. CN106997205B discloses a multi-sensor fusion positioning method, device, mobile robot and medium. It constructs a visual inertial odometer by acquiring the pre-integration error in the IMU data and the reprojection error in the depth image data, and then fuses the visual inertial odometer and UWB data to obtain the target positioning data. However, the calculation related to the depth information in this design greatly consumes the computing power of the system, and is affected by the access method of each sensor in the system. If any of the sensors has a problem, the incorrect fusion of the visual inertial odometer or UWB data will increase the final target positioning result.

[0005] Patent No. CN110766170B uses the fusion of RGB camera and single-channel multi-line laser radar to realize the positioning and tracking of personnel. Through deep learning, the image target is detected and the center point coordinates of the detection frame are calculated, and this coordinate is used as the ranging value of the laser radar. The image ranging value and the laser ranging value are passed through Kalman filtering to obtain the final ranging value. This method has high real-time performance, is simple, and can quickly locate personnel, but when it is challenged by occlusion, background clutter, and lighting changes, the positioning and tracking of targets in the image are easily affected. If the tracking fails due to the external environment, the ranging value obtained by the tracking frame is inaccurate, which in turn affects the overall performance of the system.

[0006] In recent years, target tracking and positioning tasks have occupied an important position, mainly in the following aspects:

[0007] 1. Wide application: Target tracking and positioning are widely used in many fields such as military, civil, aviation, aerospace, etc., such as photoelectric target detection and tracking, personnel identification and tracking in large places such as terminal buildings, video surveillance, automatic driving, smart home, etc. By accurately tracking the target in real time, the intelligence of the system and the effect of the application can be improved.

[0008] 2. Providing key information: Target tracking and positioning can provide key information for other visual tasks. For example, target tracking and positioning can provide information such as the location and motion trajectory of the target for behavior recognition, so as to better understand and analyze the target's behavior.

[0009] 3. Solve practical problems: Target tracking and positioning can help solve some practical problems, such as anomaly detection in video surveillance. By tracking and positioning abnormal targets, security issues can be discovered and handled in a timely manner.

[0010] Therefore, the research and development of target tracking and positioning tasks is of great significance, which can promote the development of related fields and improve the performance and effects of various applications.

[0011] After a long period of development, various target tracking and positioning algorithms have emerged one after another, but problems such as internal rotation, illumination change, background clutter, and scale change still need to be developed. Therefore, in response to the above problems, this application proposes a mobile robot heterogeneous information fusion target positioning and tracking method and system. Summary of the invention

[0012] The purpose of the present invention is to propose a mobile robot heterogeneous information fusion target positioning and tracking method and system in view of the problems of in-plane rotation, illumination change, background clutter, scale change, and the limitations of single sensors in environmental perception in existing target tracking and positioning algorithms in the background technology.

[0013] In a first aspect, the present invention discloses a mobile robot heterogeneous information fusion target positioning and tracking system, comprising a control unit MCU, a laser radar, a visual sensor, and a visual sensor driver connected to the visual sensor, wherein the input end of the control unit MCU is electrically connected to the laser radar and the visual sensor driver, respectively, and also comprises a visual target tracking and positioning system and a laser radar target positioning system, wherein:

[0014] The visual target tracking and positioning system includes a feature calculation module, a target tracking and positioning module connected to the output end of the feature calculation module, the feature calculation module includes a video frame preprocessing submodule, a feature extraction submodule and a multi-feature fusion submodule, the control unit MCU is bidirectionally data-connected to the video frame preprocessing submodule, and the output end of the video frame preprocessing submodule is sequentially connected to the feature extraction submodule and the multi-feature fusion submodule;

[0015] The laser radar target positioning system includes a median and Gaussian fusion filtering module, an adaptive DBSCAN clustering module, an adaptive threshold abnormal cluster filling module, a target cluster centroid calculation module, and a heterogeneous information fusion target positioning and tracking module, which are connected in sequence. The median and Gaussian fusion filtering module is connected to the output end of the control unit MCU, and the heterogeneous information fusion target positioning and tracking module is bidirectionally data-connected to the target tracking and positioning module.

[0016] Optionally, it also includes a display module, which includes a display module driver and a display, the display module driver and the display are bidirectionally data connected, and the output end of the heterogeneous information fusion target positioning and tracking module is connected to the display module driver.

[0017] Optionally, an output end of the control unit MCU is connected to an IO controller, and an output end of the IO controller is electrically connected to a fan.

[0018] In a second aspect, the present invention further proposes a mobile robot heterogeneous information fusion target positioning and tracking method, which is applied to the mobile robot heterogeneous information fusion target positioning and tracking system described in the first aspect, and the visual target tracking and positioning system comprises the following steps:

[0019] Step 1: calibrate the monocular camera and convert the image plane coordinates into pixel coordinates;

[0020] The visual sensor collects video data and transmits the video data to the visual target tracking and positioning system through the visual sensor driver. The DCF framework in the visual target tracking and positioning system achieves high frame rate tracking by minimizing the loss between labels and the cyclic correlation error between filters.

[0021] Step 2: The visual target tracking and positioning system inputs a frame of data, introduces spatial context information, samples the position information around the target, and uses the formula to update the regression function;

[0022] Step 3: The multi-feature fusion module in the visual target tracking and positioning system extracts multiple feature information from the video data. In the multi-feature extraction process, the HOG feature and the four deep features extracted by the VGG19 network are used simultaneously, and the CBAM model is introduced to increase the weight of the target area.

[0023] Step 4: The multi-feature fusion module arranges and combines the five features extracted in step 3;

[0024] Step 5: The multi-feature fusion module selects the best fusion feature by arranging and combining the five features in step 4, and evaluates the best fusion feature by calculating the intersection-over-union ratio and introducing weights;

[0025] In step six, the multi-feature fusion module verifies the reliability of the sample by calculating the PSR and reliability score in the best fused feature, and adjusts the learning factor according to the reliability score to adaptively update the fused feature.

[0026] Optionally, in the step 3, the multi-feature fusion module uses the underlying information of the HOG feature and the four deep features extracted by the convolutional neural network VGG19 to introduce a convolutional block attention model into the VGG19 convolutional neural network;

[0027] CBAM consists of a channel attention mechanism and a spatial attention mechanism in series, which increases the weight of the target area and reduces the weight of other areas. The input of CBAM is marked as x∈R C×H×W , C is the number of channels, H and W represent the height and width of the image respectively;

[0028] Input the feature map, multiply it by the channel weight Mc in an element-wise manner to obtain the feature map x1, input it into the spatial attention mechanism and the spatial weight Ms according to the multiplication of the corresponding elements to obtain the final output x2;

[0029] The two feature maps are added together through multi-layer perception to obtain the final channel weight;

[0030] Mc is normalized, the MLP consists of two fully connected layers and a ReLU nonlinear activation function, and the channel attention mechanism searches the image for useful information through the relationship between features.

[0031] Optionally, in step 4, the multi-feature fusion module arranges and combines the five features extracted in step 3, specifically including arranging and combining the five extracted features HOG, POOL2, POOL3, POOL4, and POOL5 Increase the base number of fused features.

[0032] Optionally, the step 5 is specifically as follows: the multi-feature fusion module uses the fusion features to track a certain frame and draw different tracking results, applies the robustness selection of different trace results to the current frame, and uses the IOU intersection-over-union formula to express the relationship between two different bounding boxes in the same frame, f1,f2,…,f p ,…f n Features used to represent fusion fusion1,fusion2,…,fusion p ,…fusion n ; In the tth frame, the tracking result corresponding to each fusion feature is replaced by The intersection-over-combination ratio of the fused features p and n is:

[0033]

[0034] In order to alleviate the gap between low IOU and high IOU, a nonlinear Gaussian function is used for processing

[0035]

[0036] use Calculate and f p The average IOU between fused features, where m represents the number of fused features, and the standard deviation in a shorter period of time is used to indicate the degree of fluctuation of the fused features. The formula is as follows:

[0037]

[0038] Where Δt represents a shorter time period, and the fused features and time need to be stable. By introducing weights G = ρ1, ρ2, …, ρ Δt ,(ρ>1), we get the formula:

[0039]

[0040] Under the above conditions, the evaluation of p fusion features in t frame is obtained:

[0041]

[0042] γ is used to avoid the denominator being zero. In two adjacent frames, the Euclidean distance between the center points of the bounding box of the best tracking result directly reflects the degree of its trajectory oscillation:

[0043]

[0044] use represents the Euclidean distance, represents the trajectory smoothness, represents the center of the bounding box, and furthermore, It also represents the average length and height of the bounding box, similar to score1. Considering the stability of time, the weight W is introduced to obtain the fusion feature self-assessment score:

[0045]

[0046] Finally, score1 and score2 are linearly combined to obtain the final evaluation, where ε represents the trade-off parameter: score(f p )=ε·score1+(1-ε)·score2.

[0047] Optionally, in step 6, when the multi-feature fusion module calculates the PSR in the best fused feature, when the object is severely occluded or deformed, the reliability of the sample is verified by the peak sidelobe ratio, which is expressed as follows:

[0048]

[0049] In the above formula, α is the mean, β is the standard deviation, M is the response, and max(M) represents the maximum confidence. The average PSR value of different feature responses is calculated:

[0050]

[0051] In the above formula and Represent the PSR of the pooling layer responses in HOG and neural network respectively. The reliability score is obtained by integrating the average robustness score and the average PSR score:

[0052]

[0053] In the above formula represents the average robustness score. When the reliability score Ct shows a significant decrease, the learning factor in (6) is reduced by expression (26):

[0054]

[0055] In the above formula, S represents the learning factor of DCF, ψ represents the reliability bound, and δ represents the power exponent.

[0056] Optionally, the laser radar target positioning system includes the following steps:

[0057] Step 1: The collected lidar data is processed by median and Gaussian fusion filtering, using median filtering to remove sharp noise and Gaussian filtering to smooth the data;

[0058] Step 2: The adaptive DBSCAN clustering module improves the clustering algorithm, adaptively determines the parameters α and MinPts, calculates the distance distribution matrix D={Dist(i,j)|1≤i≤n,1≤j≤n}, and sorts in ascending order;

[0059] Polynomial curve fitting is used to determine the peak point of the distribution matrix;

[0060] Traverse all sample points S = 1, 2, ..., n, find the nearest neighbor of a given point, and when N (S n )={S n ∈D|Dis(S i ,S j )≤α}, S j YesSi The nearest neighbor of Dis(S i ,S j ) indicates S i With S j The Euclidean distance between them is expressed as:

[0061] Then traverse all sample points S = 1, 2, ..., n, and find the core point. If the number of sample points | N (S n )|≥MinPts, then it is recorded as a core point and stored in the core point set ρ; if |N(S n )|<MinPts, it is recorded as an edge point; the others are recorded as noise; iterate all the core points that have not been visited in the core point set, find the sample points with density reachable, render and color them into a cluster H.

[0062] Finally, when the target appears or approaches the critical position of the laser beam of the lidar, or the target has a slender shape, the adaptive threshold abnormal cluster filling module is used to suppress abnormal clusters by filling the missing data;

[0063] Step 3: The centroids of all clusters after clustering are used as the coordinates of the target. The calculation process of the centroid calculation module of the target cluster is as follows: Calculate the local density of a cluster k:

[0064] Find S i With S j The minimum distance between i =min(H k (S i ,S j )), and calculate ε i The maximum value in

[0065] The local density is related to ε i Multiply them together to get a decision point, namely δ i =ρ i ×ε i ;

[0066] Using context decision point δ i -1,δ i ,δ i +1 for further accurate selection of centroids, input of decision points into In the above example, we get the final center of mass Q i , where b1, b2, b3 represent δ i -1,δ i ,δ i +1The length of each side of the connected triangle area;

[0067] Step 4: Fuse the positioning result of the laser radar with the tracking result of the image. The point cloud data of the laser radar has different data space vectors than the image. The fusion process is the mapping transformation from the laser radar coordinate system to the image coordinate system. The mapping transformation relationship is:

[0068]

[0069] In the above formula, γ represents the scaling factor, [IM 1] T is the homogeneous coordinate of a point in space in the image coordinate system, [xy 1] T is the homogeneous coordinate of a point in space in the laser radar coordinate system, c j (j=1,…,8) is the parameter of the transformation matrix. Eliminating the scaling factor γ yields the following formula:

[0070]

[0071] Calculate the confidence of LiDAR target positioning and the confidence of visual positioning, and assign weights to perform fusion positioning of the two.

[0072] Compared with the prior art, the present invention has the following beneficial technical effects:

[0073] 1. The present invention integrates the contextual information around the target into the tracking framework, so that the convolutional neural network can learn more useful information. The integrated tracking algorithm is committed to fusing various features together, using HOG features and deep features extracted by VGG19 convolutional neural network for fusion. In the process of deep feature extraction, the attention mechanism is inserted into the VGG19 convolutional neural network to improve the reliability of deep features.

[0074] 2. By fusing context information with the DCF framework and embedding the attention mechanism in the convolutional neural network, a multi-feature cascade target positioning and tracking strategy and system is developed to improve the tracking accuracy under uncertain interference conditions as well as the robustness and efficiency of the tracker, alleviating the limitations of a single sensor.

[0075] 3. The laser radar and vision fusion method is used to track and locate the target, which improves the system's ability to perceive the environment.

[0076] 4. Utilize the high resolution of vision and the high precision of 2D lidar to enhance the environmental adaptability and safety of the system, avoiding the limitations of a single sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A block diagram of a mobile robot heterogeneous information fusion target positioning and tracking system of the present invention is given;

[0078] Figure 2Schematic diagram of the imaging process of the monocular ranging model;

[0079] Figure 3 A flow chart of a mobile robot heterogeneous information fusion target positioning and tracking method of the present invention is given;

[0080] Figure 4 It is the CBAM process diagram;

[0081] Figure 5 Process diagram generated for channel weight Mc;

[0082] Figure 6 Process diagram generated for the spatial weight Ms;

[0083] Figure 7 This is the result map of visual target positioning and tracking;

[0084] Figure 8 This is the laser radar target positioning and tracking result map;

[0085] Fig. 9 This is the result of the heterogeneous information fusion target positioning and tracking experiment. DETAILED DESCRIPTION

[0086] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0087] A mobile robot heterogeneous information fusion target positioning and tracking method and system requires the monocular camera to be calibrated in advance to provide the necessary parameters for subsequent visual positioning, that is, the positioning of the target in the target frame. The vision-based ranging model is similar to the process of convex lens imaging. A monocular ranging model is established to achieve vision-based target positioning by locating the midpoint of the bottom of the tracking frame. Figure 2 , assuming that the coordinates of R and R' are (x, y, z) and (x', y', z') respectively, and the focal length between the image plane and the lens is M', according to the properties of similar triangles, we can get Further, the image plane coordinates are converted to pixel coordinates (p, q) = (λ1x′ + C x ,λ2y′+C y ), where λ1,λ2,C x ,C y are the translation and scaling amounts respectively. The solution is

[0088] See also Figure 1A mobile robot heterogeneous information fusion target positioning and tracking system, comprising a control unit MCU, a laser radar, a visual sensor, a visual sensor driver connected to the visual sensor, a visual target tracking and positioning system and a laser radar target positioning system, the input end of the control unit MCU is electrically connected to the laser radar and the visual sensor driver respectively, the visual target tracking and positioning system comprises a feature calculation module, a target tracking and positioning module connected to the output end of the feature calculation module, the feature calculation module comprises a video frame preprocessing submodule, a feature extraction submodule and a multi-feature fusion submodule, the control unit MCU is bidirectionally data-connected to the video frame preprocessing submodule, and the output end of the video frame preprocessing submodule is sequentially connected to the feature extraction submodule and the multi-feature fusion submodule.

[0089] The laser radar target positioning system includes a median and Gaussian fusion filtering module, an adaptive DBSCAN clustering module, an adaptive threshold abnormal cluster filling module, a target cluster centroid calculation module, and a heterogeneous information fusion target positioning and tracking module, which are connected in sequence. The median and Gaussian fusion filtering module is connected to the output end of the control unit MCU, and the heterogeneous information fusion target positioning and tracking module is bidirectionally data-connected to the target tracking and positioning module.

[0090] Since the laser radar is sensitive to the target's position and posture information, the laser radar is used to locate the target. The external environment is collected by the laser radar and transmitted to the control unit MCU, and then the median and Gaussian fusion filters are used to remove noise points to improve data reliability. The point cloud data is then clustered into clusters using the adaptive DBSCAN clustering algorithm. Then, the adaptive threshold abnormal cluster is used to fill in the abnormal clusters. Finally, the position and posture information of the centroid of the target cluster is calculated as the position and posture information of the target.

[0091] Furthermore, the system also includes a display module, which includes a display module driver and a display, the display module driver and the display are bidirectionally data connected, and the output end of the heterogeneous information fusion target positioning and tracking module is connected to the display module driver.

[0092] Among them, the laser radar is responsible for collecting 2D point cloud data of the external environment and providing data for the target positioning system. The radar driver package facilitates the control unit MCU to drive the laser radar and convert the outer contour of the scanned target into a digital signal and transmit it to the display module for display. The visual sensor is responsible for collecting external video images and providing input data for the target tracking system. The visual sensor driver converts the analog video signal collected by the sensor into a digital signal for processing by the control unit MCU. The connection between the visual sensor and the visual sensor driver: The visual sensor is responsible for collecting external video images and transmitting them to the visual sensor driver. The visual sensor driver is an interface device that converts the video signal collected by the sensor into a digital signal that can be processed by the control unit MCU. The control unit MCU realizes the control and coordination of the entire system through the control unit MCU and the IO controller. It receives the video data transmitted by the visual sensor driver, interacts with the multi-feature fusion module, and sends the final tracking and positioning results to the display module driver.

[0093] It is worth mentioning that the multi-feature fusion module: by processing video data, it uses multiple feature fusion algorithms to track and locate the target, thereby improving the accuracy and stability of tracking. The connection between the control unit MCU and the multi-feature fusion module: the control unit MCU is the core of the entire system, responsible for controlling and coordinating the work of each module. The multi-feature fusion module is an important part of the target tracking system, responsible for processing visual data and tracking and locating the target. The control unit MCU sends the collected video data to the multi-feature fusion module for processing through the connection with the multi-feature fusion module.

[0094] Further, the display module driver and the display: the display module driver converts the display signal sent by the control unit MCU into a signal that can be recognized by the display, and the display is used to display the tracked and located target in real time. The connection between the display module driver and the display: the display module driver is a device used to control the display, which receives the display signal from the control unit MCU and converts it into a form that the display can understand. The display is used to display the tracked and located target.

[0095] This mobile robot heterogeneous information fusion target positioning and tracking system can collect external video images from visual sensors, and after processing and analysis, it can track and locate the target. The core of the system is the control unit MCU and the multi-feature fusion module, which work together to improve the performance and accuracy of tracking. Finally, through the display module driver and display, the results of the system can be visualized in real time and provided to users for use and observation. The external video image is collected by the visual sensor, and then transmitted to the control unit MCU through the visual sensor driver. Then, the above mobile robot heterogeneous information fusion target positioning and tracking method collects the image sequence in the video, tracks and locates the target in the image, and finally displays it on the display through the display driver.

[0096] In the evaluation benchmark, accuracy and success rate are mainly used to evaluate tracking algorithms. First, the accuracy is defined as the percentage of the distance between the center point of the target bounding box predicted by the tracking algorithm and the standard bounding box that is less than a threshold. Different thresholds lead to different results. Second, the bounding box tracked by the tracking algorithm is represented as A, and the standard bounding box is represented as B The overlap rate between the two is When one of the video frames is larger than the threshold, the tracking is considered successful, and the percentage of successful frames to the total number of frames in the dataset is defined as the success rate.

[0097] In addition, the output end of the control unit MCU is connected to the IO controller, and the output end of the IO controller is electrically connected to the fan to dissipate heat for the entire system.

[0098] In this embodiment, the visual target tracking and positioning system includes the following parts:

[0099] 1. Discriminative Correlation Filter (DCF) Framework

[0100] See also Figure 3 As shown, the visual sensor collects video data and transmits the video data to the target tracking system through the visual sensor driver. The DCF framework in the target tracking system minimizes the loss between labels and the cyclic correlation error between filters. It uses a simple structure to achieve a high frame rate. The main goal of the DCF tracker is to use the F(x) function to eliminate the error between the sample X and the regression target Y, which can be expressed as:

[0101]

[0102] F(X)=Xμ (1)

[0103] In (1) above, λ is a regularization parameter to prevent overfitting, making the gradient of equation (1) zero, and the solution of μ is as follows:

[0104] μ=(X T X+λI) -1 X T Y (2)

[0105] The formula can be derived from the properties of circulant matrix and Fourier transform:

[0106]

[0107] Where ∧ represents discrete Fourier transform (DFT), * represents conjugate complex operation, and · is the multiplication of corresponding elements (i.e. Hadamard product), where Hadamard product: g×h matrix A=[a kr ] and the matrix B = [b kr ], denoted as A·B, is still a g×h matrix, defined as The Hadamard product is also called the corresponding element product. When the image z is input, the regression detection stage begins:

[0108]

[0109] In (4), D -1 is the inverse discrete Fourier transform. Overfitting can be avoided by using linear interpolation to update the filter model, which is as follows:

[0110]

[0111] ν is defined as the learning rate, t is the current frame, and the model update process is as follows, N t and Q t Represent the numerator and denominator of the model respectively:

[0112]

[0113] 2. Spatial context information

[0114] In order to make the tracker model more robust, the target tracking system inputs a frame of data and introduces spatial context information, samples the position information around the target, and uses the formula to update the regression function. The information around the target (up, down, left, and right) is sampled to provide enough samples for the subsequent VGG19 network, and the sampling results are used as negative samples to train the model and cyclically shift. The regression function adding spatial context information can be written as:

[0115]

[0116] F′(X i )=X i μ (7)

[0117] In the above formula, X∈R n and X i ∈R n (i=1,2,…,n) represent the feature map and spatial context information respectively, λ′ is the new regularization parameter, then equation (7) can be rewritten as:

[0118]

[0119] Do the following:

[0120]

[0121] The closed-form solution of formula (8) can be obtained:

[0122]

[0123] Therefore, the closed-form solution in the Fourier domain is obtained from equation (9):

[0124]

[0125] 3. Multi-feature extraction

[0126] The multi-feature fusion module in the target tracking system extracts multiple feature information from the video data. In the multi-feature extraction process, the underlying information of the HOG feature and the four deep features extracted by the convolutional neural network VGG19 are used simultaneously. The convolution block attention model (CBAM) is introduced into the VGG19 convolutional neural network to increase the weight of the target area and filter out important information. Figure 4 As shown in Figure 2, CBAM is mainly composed of a channel attention mechanism and a spatial attention mechanism in series, which can increase the weight of the target area and reduce the weight of other areas. The input of CBAM is marked as x∈R C×H×W C is the number of channels, H and W represent the height and width of the image respectively. Input the feature map, multiply it by the channel weight Mc in an element-wise manner to get the feature map x1, and then x1 is input into the spatial attention mechanism and the spatial weight Ms to get the final output x2 according to the multiplication of the corresponding elements. This process can be expressed as:

[0127]

[0128] Is the Kronecker product, that is, the Kronnecker product, where the Kronnecker product is the right Kronecker product of the s×v matrix A and the p×q matrix B Defined as: Left Kronecker product: the left Kronecker product of the s×v matrix A and the p×q matrix B Defined as

[0129] like Figure 5 As shown in the figure, the generation process of channel weights is as follows: the input x is subjected to global maximum pooling and global average pooling respectively to compress the spatial information and efficiently extract the channel attention features to obtain two feature maps R1 and R2 with a dimension of 1×1×C. Then, the two feature maps are added together through multi-layer perception (MLP) to obtain the final channel weights. Finally, in order to reduce the calculation parameters, Mc is normalized. MLP consists of two fully connected layers (FC1 and FC2) and a ReLU nonlinear activation function. The channel attention mechanism searches the image for useful information through the relationship between features so that the network knows what to focus on. In short, the Mc calculation process is:

[0130] M c (x)=ζ(MLP(AVGpool(x))+MLP(MAXpool(x))) (13)

[0131] In equation (13), ζ represents the sigmoid function, AVGpool represents global average pooling, and MAXpool represents global maximum pooling.

[0132] like Figure 6 As shown in the figure, the spatial attention mechanism considers the relationship within the space and is able to find the location of the focus area. The process of spatial weighting is to take x1 as input, perform average pooling and maximum pooling in the channel dimension without compressing the spatial information, and obtain two feature maps G1 and G2 with dimensions of 1×H×W. The two are then overlapped and the final spatial weight is generated through a 7×7 convolution to reduce the parameters of the normalization operation again. The whole process is summarized as follows:

[0133] M s (x1)=ζ(f 7×7 (AVGpool(x1);MAXpool(x1))) (14)

[0134] 4. Multi-feature fusion

[0135] The multi-feature fusion module arranges and combines the five features (HOG / POOL2 / POOL3 / POOL4 / POOL5) extracted above To increase the basic number of fused features. Therefore, considering the diversity of tracking, fused features can be obtained, which lays the foundation for subsequent screening of excellent features.

[0136] 5. Best fusion feature selection

[0137] Select the best one among 31 fused features. The most direct way is to use these fused features to track a certain frame and plot different tracking results. The robust selection of different tracking results is applied to the current frame. The Intersection Over Union (IOU) formula is very useful for the relationship between two different bounding boxes in the same frame. f1,f2,…,f p ,…f n Features used to represent fusion fusion1,fusion2,…,fusion p ,…fusion n In the t frame, the tracking result corresponding to each fusion feature is replaced by The intersection-over-combination ratio of the fused features p and n is:

[0138]

[0139] In order to alleviate the gap between low IOU and high IOU, a nonlinear Gaussian function is used for processing

[0140]

[0141] use Calculate and f p The average IOU between fused features. Where m represents the number of fused features. The standard deviation in a shorter period of time is used to indicate the degree of fluctuation of the fused features. The formula is as follows:

[0142]

[0143] Where Δt represents a shorter time period. The fused features and time need to be stable. By introducing weights G = ρ1, ρ2, …, ρ Δt ,(ρ>1), we can get the formula:

[0144]

[0145] Under the above conditions, the evaluation of p fusion features in t frames can be obtained.

[0146]

[0147] γ is used to avoid the denominator being zero. In two adjacent frames, the Euclidean distance between the center points of the bounding box of the best tracking result directly reflects the degree of its trajectory oscillation:

[0148]

[0149] use represents the Euclidean distance, represents the trajectory smoothness, represents the center of the bounding box. In addition, It also represents the average length and height of the bounding box. Similar to score1, considering the stability of time, the weight W is also introduced to obtain the fusion feature self-assessment score.

[0150]

[0151] Finally, score1 and score2 are linearly combined to get the final evaluation, where ε represents the trade-off parameter.

[0152] score(f p )=ε·score1+(1-ε)·score2 (22)

[0153] 6. Adaptive update of fusion features

[0154] When the multi-feature fusion module calculates the PSR in the best fused feature, the reliability of the sample is difficult to guarantee when the object is severely occluded or deformed. Therefore, the reliability of the sample is verified by the peak sidelobe ratio (PSR), which is expressed as follows:

[0155]

[0156] In the above formula, α is the mean, β is the standard deviation, M is the response, and max(M) represents the maximum confidence level.

[0157] Calculate the average PSR value of different feature responses.

[0158]

[0159] In the above formula and denote the PSR of the pooling layer responses in HOG and neural networks, respectively. However, due to the presence of similar features in the target, the PSR values ​​may be indistinguishable. To address the above problem, the reliability score is obtained by integrating the average robustness score and the average PSR score.

[0160]

[0161] In the above formula Represents the average robustness score. When the reliability score C t When a significant decrease is shown, the learning factor in (6) is reduced by expression (26).

[0162]

[0163] In the above formula, S represents the learning factor of DCF, ψ represents the reliability bound, and δ represents the power exponent.

[0164] In addition to the above-mentioned visual target tracking and positioning system, the laser radar target positioning system of this embodiment includes:

[0165] 7. Locate the visual target box

[0166] Positioning the target within the target frame. The vision-based ranging model is similar to the process of convex lens imaging. The model for establishing monocular ranging is shown in the figure above. r ,O r ,Y r is the real coordinate system, X l ,Y l ,Z l is the camera coordinate system, X c ,Y c ,Z c is the image coordinate system, O c ,O l represents the focal length f, O l ,O r H is the height of the monocular camera, R is the intersection of the target and the ground, O l The angle between R and the ground is μ. The above tracking algorithm is used to obtain the pixel value of the target bottom in the target frame of the image plane, and the coordinate Center of the optical center in the image plane in the y direction. Through the calibration of the above camera, we can get From the figure, we can see the following relationship Solved

[0167] 8. LiDAR Adaptive DBSCAN Clustering

[0168] First, the collected lidar data is filtered by median and Gaussian fusion, and the sharp noise is removed by median filtering, and the data is further smoothed by Gaussian filtering. Specifically, the window size W and the standard deviation σ of the Gaussian filter are defined first, and the median M of each radar data S = 1, 2, ..., n within the window range is calculated. i And replace the original data point Then, the data after median filtering is subjected to Gaussian filtering to reduce high-frequency noise. To M i A convolution operation is performed to smooth the data after median filtering.

[0169] Secondly, the traditional DBSCAN clustering algorithm has high computational complexity and requires manual adjustment of the neighborhood radius size α and the minimum number of points in the neighborhood that become core points MinPts. Improve the clustering algorithm and adaptively determine the parameters α and MinPts. Calculate the distance distribution matrix D = {Dist(i,j)|1≤i≤n,1≤j≤n} and sort it in ascending order. Use polynomial curve fitting with low fitting order, high precision and low complexity to find the peak point of the distribution matrix. The process is expressed by the formula: g(x) = a1x 4 +a2x 3 +a3x 2 +a4x+a5. Taking the second derivative, we get g(x)″=12a1x 2 +6a2x+2a3, the solution is The smaller value of the two solutions is discarded near the boundary point. Then substitute the larger x into g(x) to get α. MinPts takes the average number of points in the neighborhood, that is:

[0170] Again, traverse all sample points S = 1, 2, ..., n to find the nearest neighbors of the given point. When N (S n )={S n ∈D|Dis(S i ,S j )≤α}, S j YesS i The nearest neighbor of Dis(S i ,S j ) indicates S i With S j The Euclidean distance between them is expressed as: Then, traverse all sample points S = 1, 2, ..., n to find the core point. When the number of sample points | N (S n )|≥MinPts is recorded as a core point and stored in the core point set ρ; if |N(S n )|<MinPts is recorded as an edge point; the others are recorded as noise. Iterate all the core points that have not been visited in the core point set, find the sample points with density reachable, and render them into a cluster H.

[0171] Finally, when the target appears or approaches the critical position of the laser radar beam or the target is slender, the laser point cloud will also cluster the point cloud into a cluster, but a certain cluster will flash frequently, affecting the positioning result. In this case, the adaptive threshold abnormal cluster filling method is used to suppress abnormal clusters by filling the missing data. Specifically, a difference threshold φ is set, and the clustering cluster of a certain frame is compared with the clustering cluster of the previous frame. When the difference threshold is exceeded, it is defined as an abnormal cluster, that is, |H t (i)-Ht-1 (i)|>φ, use Lagrange interpolation to calculate the filling points Fill the missing data, the filling range is E = |H t (i)-H t-1 (i) All sample points S′1, S′2, …, S′ in | n ∈E.

[0172] 9. Calculation of the center of mass

[0173] The centroid of all clusters after clustering is used as the coordinate of the target. In real life, the shapes of clusters are mostly irregular, and for clusters of certain shapes, if the centroid is calculated using the overall sum of squares method, the centroid will be mistakenly located outside a cluster. According to the phenomenon that the centroid is usually surrounded by points with low local density and far away from points with high local density, the centroid calculation process of each cluster is improved as follows: Step 1, calculate the local density: The second step is to find S i With S j The minimum distance between i = min(Dis(S i ,S j )) and calculate ε i The maximum value in The third step is to transform the local density into ε i Multiply them together to get a decision point, namely δ i =ρ i ×ε i Finally, the context decision point δ i -1,δ i ,δ i +1 for further accurate selection of centroids, input of decision points into The final center of mass is obtained from the equation, where b1, b2, and b3 represent δ i -1,δ i ,δ i +1 The length of each side of the connected triangle area.

[0174] 10. Mapping point cloud data to images

[0175] The positioning results of the LiDAR are combined with the tracking results of the image. The point cloud data of the LiDAR has different data space vectors than the image, so the fusion process can be regarded as a mapping transformation from the LiDAR coordinate system to the image coordinate system. The mapping transformation relationship is:

[0176]

[0177] In the above formula, γ represents the scaling factor, [IM 1] T is the homogeneous coordinate of a point in space in the image coordinate system, [xy 1]T is the homogeneous coordinate of a point in space in the laser radar coordinate system. j (j=1,…,8) are the parameters of the transformation matrix. Eliminating the scaling factor γ yields the following formula:

[0178]

[0179] Finally, the confidence of the laser radar target positioning is calculated and combined with the confidence of the visual positioning, and the weights are assigned to perform the fusion positioning of the two. First, the local density ρ i The normalized value is used as the confidence U of the laser radar target positioning l ; The visual tracking frame overlap rate OS is used as the confidence U of visual target positioning c . Secondly, assign the weights of LiDAR and visual positioning, and record is the weight of the laser radar; is the visual weight. Then F i =ω D D i +ω Dis Dis i The target positioning result in the i-th frame is obtained, which enhances the robustness and stability of the system and alleviates the limitations of a single sensor.

[0180] In the above, the position information obtained by the laser radar is mapped to the image, and the positioning results of the laser radar and vision are calculated and weighted to complete the joint tracking and positioning of the target. The fusion of laser radar and vision can improve the accuracy and robustness of the mobile robot's heterogeneous information fusion target positioning and tracking system, and alleviate the limitations of a single sensor.

[0181] Experimental results:

[0182] The proposed multi-feature target localization and tracking method experiments were tested on a computer configured with AMD Ryzen 5-3500U 2.10GHz CPU and 12GB RAM.

[0183] The public datasets used to evaluate tracking algorithms are: OTB2013, OTB2015, TC128, and UAV123. This embodiment selects 11 different tracking algorithms including this embodiment, and evaluates them on the following 4 public datasets to compare the performance of the tracking algorithms. The OTB2013 dataset divides the video into 11 attributes, including illumination variation (IV), out of view (OV), occlusion (OCC), fast motion (FM), scale variation (SV), in-plane rotation (IPR), out of plane rotation (OPR), low resolution (LR), motion blur (MB), deformation (DEF), and background clutter (BC). OTB2015 is a supplement to the OTB2013 dataset. The TC128 dataset has 128 sequences, which also include sequences from various OTB series datasets. UAV123 is a long sequence tracking drone shooting. It has 123 sequences, a large number of scenes and actions, and can also be divided into stable controllable camera capture, unstable camera capture and composite video synthesis. In order to evaluate the tracking performance of the designed mobile robot heterogeneous information fusion target positioning and tracking method, the distance precision (DP) is introduced to evaluate the tracking algorithm's tracking target accuracy. The larger the DP value, the higher the tracking accuracy. In addition, the area under the curve (AUC) value is introduced to evaluate the performance of the classifier. The larger the AUC value, the better the classifier performance.

[0184] Experimental comprehensive scoring results

[0185] In order to evaluate and compare the effectiveness of the tracking performance of the multi-feature fusion method with attention mechanism and context information (MFAC) for mobile robot target localization and tracking, it was verified on four datasets with three other current advanced tracking algorithms: kernel correlation filter (KCF), spatiotemporal regularized correlation filter (STRCF), and multi-cue correlation filters for robust visual tracking (MCCT).

[0186] Table 1 Average DP scores of five tracking algorithms.

[0187]

[0188]

[0189] Table 2 Average AUC scores of five tracking algorithms

[0190]

[0191] In Tables 1 and 2, the accuracy and success rate of the four tracking algorithms on the four datasets are recorded to obtain the overall average. The DP and AUC comprehensive scores of MFAC are 0.843 and 0.771, respectively, which are the highest among all algorithms. Compared with other tracking algorithms, MFAC has an overall advantage.

[0192] In the process of deep feature extraction of the present invention, the attention mechanism is inserted into the VGG19 convolutional neural network to improve the reliability of deep features. The multi-feature cascade target positioning and tracking strategy and system combining deep features with Histogram of Oriented Gradients (HOG) features improves the tracking accuracy and robustness of the tracker under uncertain interference conditions. The fusion of context information and DCF framework improves the efficiency of the tracker and the frame rate of tracking.

[0193] The heterogeneous information fusion target positioning and tracking method was experimentally verified on a mobile robot physical platform. Both visual target tracking and positioning and lidar positioning are performed in the robot operating system Ros Operating System (ROS) under Ubuntu 18.04. In the image display window, first use the mouse to select the target to be tracked and positioned, locate the target from the target frame of the first frame, and then perform target tracking based on the MFAC algorithm. Then, read and process the next frame of the video stream to locate and track the target until the end of the video stream. The visual positioning and tracking test results are shown in Figure 2. Figure 7 As shown in the figure, the target is selected as a carton, and its visual tracking and positioning result is 2.74m. Figure 8 ,The laser radar target positioning algorithm processes the scanned data and obtains the laser radar positioning result of 2.88m.

[0194] The results of the target positioning and tracking algorithm using heterogeneous information fusion are as follows: Fig. 9 As shown in the figure, the positioning result is 2.86m. The visual positioning window updates and displays the heterogeneous information fusion positioning result. The result is as follows: Fig. 9 As shown in the figure. From the positioning results, the camera provides rich visual information such as color and texture, while the lidar also provides high-precision positioning information. The combination of the two can better alleviate the limitations of single sensor failure or error. The fusion of lidar and vision can improve the accuracy and robustness of the mobile robot heterogeneous information fusion target positioning and tracking system. Dynamically adjusting the weights of each sensor can make the mobile robot heterogeneous information fusion target positioning and tracking method and system adapt to different scenes and environments.

[0195] The above specific embodiments are only several preferred embodiments of the present invention. Based on the technical solutions of the present invention and the relevant inspirations of the above embodiments, those skilled in the art may make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A mobile robot heterogeneous information fusion target positioning and tracking system, comprising a control unit MCU, a laser radar, a visual sensor and a visual sensor driver connected to the visual sensor, wherein the input end of the control unit MCU is electrically connected to the laser radar and the visual sensor driver respectively, and is characterized in that: It also includes a visual target tracking and positioning system and a laser radar target positioning system, among which: The visual target tracking and positioning system includes a feature calculation module, a target tracking and positioning module connected to the output end of the feature calculation module, the feature calculation module includes a video frame preprocessing submodule, a feature extraction submodule and a multi-feature fusion submodule, the control unit MCU is bidirectionally data-connected to the video frame preprocessing submodule, and the output end of the video frame preprocessing submodule is sequentially connected to the feature extraction submodule and the multi-feature fusion submodule; The laser radar target positioning system includes a median and Gaussian fusion filtering module, an adaptive DBSCAN clustering module, an adaptive threshold abnormal cluster filling module, a target cluster centroid calculation module, and a heterogeneous information fusion target positioning and tracking module, which are connected in sequence. The median and Gaussian fusion filtering module is connected to the output end of the control unit MCU, and the heterogeneous information fusion target positioning and tracking module is bidirectionally data-connected to the target tracking and positioning module.

2. A mobile robot heterogeneous information fusion target positioning and tracking system according to claim 1, characterized in that: It also includes a display module, which includes a display module driver and a display. The display module driver and the display are bidirectionally data connected, and the output end of the heterogeneous information fusion target positioning and tracking module is connected to the display module driver.

3. The mobile robot heterogeneous information fusion target positioning and tracking system according to claim 1, characterized in that: The output end of the control unit MCU is connected to the IO controller, and the output end of the IO controller is electrically connected to the fan.

4. A mobile robot heterogeneous information fusion target positioning and tracking method, characterized in that: The mobile robot heterogeneous information fusion target positioning and tracking system applied to any one of claims 1 to 3, the visual target tracking and positioning system comprises the following steps: Step 1: calibrate the monocular camera and convert the image plane coordinates into pixel coordinates; The visual sensor collects video data and transmits the video data to the visual target tracking and positioning system through the visual sensor driver. The DCF framework in the visual target tracking and positioning system achieves high frame rate tracking by minimizing the loss between labels and the cyclic correlation error between filters. Step 2: The visual target tracking and positioning system inputs a frame of data, introduces spatial context information, samples the position information around the target, and uses the formula to update the regression function; Step 3: The multi-feature fusion module in the visual target tracking and positioning system extracts multiple feature information from the video data. In the multi-feature extraction process, the HOG feature and the four deep features extracted by the VGG19 network are used simultaneously, and the CBAM model is introduced to increase the weight of the target area. Step 4: The multi-feature fusion module arranges and combines the five features extracted in step 3; Step 5: The multi-feature fusion module selects the best fusion feature by arranging and combining the five features in step 4, and evaluates the best fusion feature by calculating the intersection-over-union ratio and introducing weights; In step six, the multi-feature fusion module verifies the reliability of the sample by calculating the PSR and reliability score in the best fused feature, and adjusts the learning factor according to the reliability score to adaptively update the fused feature.

5. A mobile robot heterogeneous information fusion target positioning and tracking method according to claim 4, characterized in that: In the step 3, the multi-feature fusion module uses the underlying information of the HOG feature and the four deep features extracted by the convolutional neural network VGG19, and introduces a convolutional block attention model into the VGG19 convolutional neural network; CBAM consists of a channel attention mechanism and a spatial attention mechanism in series, which increases the weight of the target area and reduces the weight of other areas. The input of CBAM is marked as x∈R C×H×W , C is the number of channels, H and W represent the height and width of the image respectively; Input the feature map, multiply it by the channel weight Mc in an element-wise manner to obtain the feature map x1, input it into the spatial attention mechanism and the spatial weight Ms according to the multiplication of the corresponding elements to obtain the final output x2; The two feature maps are added together through multi-layer perception to obtain the final channel weight; Mc is normalized, the MLP consists of two fully connected layers and a ReLU nonlinear activation function, and the channel attention mechanism searches the image for useful information through the relationship between features.

6. A mobile robot heterogeneous information fusion target positioning and tracking method according to claim 5, characterized in that: In step 4, the multi-feature fusion module arranges and combines the five features extracted in step 3, specifically including arranging and combining the five extracted features HOG, POOL2, POOL3, POOL4, and POOL5 Increase the base number of fused features.

7. A mobile robot heterogeneous information fusion target positioning and tracking method according to claim 6, characterized in that: Specifically, the multi-feature fusion module uses the fusion features to track a certain frame and draw different tracking results, applies the robustness selection of different trace results to the current frame, and uses the IOU intersection-over-union formula to express the relationship between two different bounding boxes in the same frame, f1,f2,…,f p ,…f n Features used to represent fusion fusion1,fusion2,…,fusion p ,…fusion n ; In the tth frame, the tracking result corresponding to each fusion feature is replaced by TRt fn , the intersection and union ratio of the fused features p and n is: In order to alleviate the gap between low IOU and high IOU, a nonlinear Gaussian function is used for processing use Calculate and f p The average IOU between fused features, where m represents the number of fused features, and the standard deviation in a shorter period of time is used to indicate the degree of fluctuation of the fused features. The formula is as follows: Where Δt represents a shorter time period, and the fused features and time need to be stable. By introducing weights G = ρ1, ρ2, …, ρ Δt ,(ρ>1), we get the formula: Under the above conditions, the evaluation of p fusion features in t frame is obtained: γ is used to avoid the denominator being zero. In two adjacent frames, the Euclidean distance between the center points of the bounding box of the best tracking result directly reflects the degree of its trajectory oscillation: use represents the Euclidean distance, represents the trajectory smoothness, represents the center of the bounding box, and furthermore, It also represents the average length and height of the bounding box, similar to score1. Considering the stability of time, the weight W is introduced to obtain the fusion feature self-assessment score: Finally, score1 and score2 are linearly combined to obtain the final evaluation, where ε represents the trade-off parameter: score(f p )=ε·score1+(1-ε)·score2.

8. The method for mobile robot heterogeneous information fusion target positioning and tracking according to claim 4, characterized in that: In step 6, when the multi-feature fusion module calculates the PSR in the best fused feature, when the object is severely occluded or deformed, the reliability of the sample is verified by the peak sidelobe ratio, which is expressed as follows: In the above formula, α is the mean, β is the standard deviation, M is the response, and max(M) represents the maximum confidence. The average PSR value of different feature responses is calculated: In the above formula and Represent the PSR of the pooling layer responses in HOG and neural network respectively. The reliability score is obtained by integrating the average robustness score and the average PSR score: In the above formula Represents the average robustness score, when the reliability score C t When a significant decrease is shown, the learning factor in (6) is reduced by expression (26): In the above formula, S represents the learning factor of DCF, ψ represents the reliability bound, and δ represents the power exponent.

9. A mobile robot heterogeneous information fusion target positioning and tracking method according to claim 4, characterized in that: The laser radar target positioning system comprises the following steps: Step 1: The collected lidar data is processed by median and Gaussian fusion filtering, using median filtering to remove sharp noise and Gaussian filtering to smooth the data; Step 2: The adaptive DBSCAN clustering module improves the clustering algorithm, adaptively determines the parameters α and MinPts, calculates the distance distribution matrix D={Dist(i,j)|1≤i≤n,1≤j≤n}, and sorts in ascending order; Polynomial curve fitting is used to determine the peak point of the distribution matrix; Traverse all sample points S = 1, 2, ..., n, find the nearest neighbor of a given point, and when N (S n )={S n ∈D|Dis(S i ,S j )≤α}, S j YesS i The nearest neighbor of Dis(S i ,S j ) indicates S i With S j The Euclidean distance between them is expressed as: Then traverse all sample points S = 1, 2, ..., n, and find the core point. If the number of sample points | N (S n )|≥MinPts, then it is recorded as a core point and stored in the core point set ρ; if |N(S n )|<MinPts, it is recorded as an edge point; the others are recorded as noise; iterate all the core points that have not been visited in the core point set, find the sample points with density reachable, render and color them into a cluster H. Finally, when the target appears or approaches the critical position of the laser beam of the lidar, or the target has a slender shape, the adaptive threshold abnormal cluster filling module is used to suppress abnormal clusters by filling the missing data; Step 3: The centroids of all clusters after clustering are used as the coordinates of the target. The calculation process of the centroid calculation module of the target cluster is as follows: Calculate the local density of a cluster k: Find S i With S j The minimum distance between i =min(H k (S i ,S j )), and calculate ε i The maximum value in The local density is related to ε i Multiply them together to get a decision point, namely δ i =ρ i ×ε i ; Using context decision point δ i -1,δ i ,δ i +1 for further accurate selection of centroids, input of decision points into In the above example, we get the final center of mass Q i , where b1, b2, b3 represent δ i -1,δ i ,δ i +1The length of each side of the connected triangle area; Step 4: Fuse the positioning result of the laser radar with the tracking result of the image. The point cloud data of the laser radar has different data space vectors than the image. The fusion process is the mapping transformation from the laser radar coordinate system to the image coordinate system. The mapping transformation relationship is: In the above formula, γ represents the scaling factor, [IM 1] T is the homogeneous coordinate of a point in space in the image coordinate system, [x y1] T is the homogeneous coordinate of a point in space in the laser radar coordinate system, c j (j=1,…,8) is the parameter of the transformation matrix. Eliminating the scaling factor γ yields the following formula: Calculate the confidence of LiDAR target positioning and the confidence of visual positioning, and assign weights to perform fusion positioning of the two.

Images