Visual positioning method based on self-learning

Abstract

The invention discloses a self-learning visual positioning method. The method comprises the following steps: S1, acquiring image data; s2, image enhancement; s3, dividing the obtained image information into grids on the basis of a YOLOv5 algorithm, extracting feature information and label information from the grid information, then adjusting weight information of the feature information and the label information through a Bayesian adaptive learning method, and obtaining a target image; combining the feature information according to the adjusted weight information, and converting the combined image information into three-dimensional coordinate information; s4, adjusting a predicted speed value and predicted three-dimensional coordinate information through a least square method, S5, predicting attitude data through the Markov data transition probability matrix, the predicted speed information, the predicted three-dimensional coordinate information and target frame information of a current position in the image information in the step S3, and then performing positioning movement according to the predicted attitude data. And the positioning data reliability is good.

Description

Technical Field

[0001] The present invention belongs to the field of positioning technology, and in particular relates to a visual positioning method based on self-learning. Background Art

[0002] At present, the more common mobile robots include household service robots, AGV logistics robots, etc. Household service robots mainly include educational robots, cleaning robots, robots that help the elderly and the disabled, etc. In special occasions, such as exhibitions, restaurants, welcoming guests, shopping guides, security and monitoring, special robots are playing an increasingly important role. Although mobile robots with complex functions have brought disruptive changes to people's production and life, there are still limitations to robot functions. In complex scenarios and fields with high performance requirements, there is still much room for improvement in robot functions and performance. For autonomous mobile robots, robot positioning and navigation are key issues, involving precise positioning, scene recognition, semantic understanding, obstacle analysis and decision-making, etc., which require our in-depth analysis and research.

[0003] Existing positioning technologies generally use a single sensor such as a visual sensor, laser sensor or other sensor for positioning. However, when the surrounding environment has less texture or the texture is regularly distributed, the positioning effect of a pure visual sensor is poor and track drift is prone to occur. Similarly, when there are fewer obstacles in the surrounding environment or the environment is regularly distributed, the pure laser sensor will not be able to determine the surrounding environment when there are no obstacles around, resulting in positioning drift. In order to solve the problem of a single sensor losing positioning in certain environments, many studies have combined visual sensors and laser sensors with other sensors (such as IMU, encoders, etc.). Compared with pure vision or pure laser sensor odometers, fusion sensors have higher positioning accuracy, are not easy to lose accuracy in complex environments, and have better robustness.

[0004] In order to solve the above problems, a patent document with Chinese patent application number 202311372884.9 and publication date 2024.01.30 discloses a visual laser sensor fusion positioning method based on deep learning, which specifically includes the following steps: collecting real-time data through the laser radar, camera and IMU sensor on the robot; using the deep learning model to extract features and identify the environment of the preprocessed data; according to the feature extraction and environment recognition results, weighted fusion of different sensor data to obtain accurate pose estimation; using IMU data and GPS data to optimize the back-end factor graph to obtain accurate pose. The above documents use deep learning models to extract point and line features and identify the environment, thereby improving feature quality and positioning accuracy in low-texture environments; through multi-sensor data fusion, the positioning accuracy and robustness of the mobile robot SLAM system can be improved in low-texture environments, and real-time estimation of the robot's position and posture can be achieved.

[0005] However, this document only determines the overlap of data between multiple sensors through weighted values, and does not consider that the image data obtained by the visual sensor may be contaminated by noise, which may lead to deviations in the subsequent three-dimensional visual images, thereby causing inaccuracy in subsequent positioning. In addition, the posture control is only determined by the IMU sensor, and the final posture cannot be determined by comparing the detected posture with the predicted posture to ensure posture stability, thereby achieving reliability in positioning. Summary of the invention

[0006] The purpose of the present invention is to provide a visual positioning method based on self-learning. The present invention can accurately identify and classify images, use the fusion data of visual sensors and IMU inertial sensors to achieve high-precision positioning of the robot, and reliably realize posture control.

[0007] To achieve the above purpose, a visual positioning method based on self-learning includes the following steps: S1 obtains image data; obtains image information of the surrounding environment through visual sensors; S2 image enhancement: first extract brightness information from the image, then use the histogram equalization method to remove point data whose brightness exceeds the preset brightness range, and form image information; S3 divides the acquired image information into grids based on the YOLOv5 algorithm, and extracts feature information and label information from the grid information. The label information is the location and category of the target object in the labeled image. Then, the weight information of the feature information and label information is adjusted through the Bayesian adaptive learning method. Then, the feature information is merged according to the adjusted weight information, and the merged image information is converted into three-dimensional coordinate information. S4 obtains the expected speed information within a preset time through the IMU sensor, and then determines the expected posture information by scanning the Aruco code, and then uses the least square method to adjust the error value between the expected speed information and the predicted speed information to determine the predicted speed information, and uses the least square method to adjust the error between the expected three-dimensional coordinate information and the predicted three-dimensional coordinate information to determine the predicted three-dimensional coordinate information; S5 predicts the posture data through the Markov chain data transfer probability matrix, the predicted speed information, the predicted three-dimensional coordinate information, and the target frame information of the current position in the image information in step S3, and then performs positioning and movement according to the predicted posture data.

[0008] The above method collects images by acquiring visual sensors, and then uses the histogram equalization method to eliminate the information of points with too high or too low brightness, thereby improving the effective recognition of the entire image, and then uses the YOLOv5 algorithm to divide the image into multiple small units and extract feature information and label information. Since the label information is used to mark the feature position and category in the image, the Bayesian adaptive learning method is then used to determine the weight value based on the feature information and weight information, and finally the weight used for merging the graphics is determined based on the adjusted weight value. Since the existing YOLOv5 algorithm generally uses a complex distribution function, the weight parameters cannot be determined in the final merging process of the neural network. The Bayesian adaptive learning method can determine the distribution function as a Gaussian distribution characteristic, because everything in nature is It conforms to the characteristics of Gaussian distribution, so this distribution can meet the requirements of most images. At the same time, the distribution has a certain degree of certainty and can form a certain weight function to ensure the accuracy of the image. In addition, the expected speed information and the expected three-dimensional coordinate information of the current position in the three-dimensional image are determined by the IMU sensor, and then the deviation between the predicted value and the expected value is adjusted by the least squares method to adjust the predicted value. Then, the adjusted predicted value, the posture information obtained by scanning the Aruco code, the information of the target frame in the image information, and the Markov chain data transfer probability matrix are used to predict the posture data, so that multiple data such as speed, position, posture, image target frame information are fused, so that the posture is associated with the speed, position, posture and target frame information in the image, ensuring the reliability of the posture data.

[0009] Furthermore, the label information includes the category ID, bounding box coordinates and possible confidence of the target object.

[0010] In the above settings, the label information includes the category of the target object, the bounding box coordinates, and the confidence information, so that it is convenient to determine the loss situation and finally determine the prediction result during the subsequent prediction.

[0011] Furthermore, the S2 comprises the steps of: S21 extracts the brightness information in the color channel through the formula Gray = A* R + C * G +D * B, where R, G, and B represent the three colors of red, green, and blue respectively; S22 scans each pixel of the original grayscale image in sequence and calculates the grayscale histogram of the image; each pixel only contains the grayscale value; S23 by formula Calculate the cumulative histogram of the grayscale histogram, where q and p are the brightness information corresponding to the output image and the input image respectively; S23 by formula , calculate the mapping relationship between input and output, H and W are the height and width of the image respectively, and hist is the histogram.

[0012] The above settings utilize a grayscale conversion and histogram equalization processing mechanism to eliminate pixel points with too high or too low brightness in the image, making the detection target more distinguishable.

[0013] Furthermore, the step S3 includes: S31 Extract features through the backbone network part, continuously reduce the feature map through the convolution module, and fuse the feature information into the channels; S32 The Neck module is responsible for fusing with the features of the backbone network; S33 Output multi-scale detection boxes through the head module, and the detection boxes contain label information.

[0014] S34 The Bayesian adaptive learning method includes given known feature information and label information on the premise that ; where f(k) is the Monte Carlo sampling function, which takes a finite number of random variable samples to approximate the complex integral; x(k) is the k-th new sample, y(k) is the k-th new label; q(w) is the Gaussian distribution function, w is the weight information, q(y(k) 丨 x(k)) is the output feature information and label information, p(y(k) 丨 f(k)) is the sampling information corresponding to the feature label information in the input image, and p(y(k) 丨 x(k), w) is the feature information, label information in the input image, and the weight information under the corresponding Gaussian distribution.

[0015] The above settings ensure the accuracy of identifying the target category, introduce a Bayesian-based network weight adaptive adjustment method to reduce the uncertainty in model detection.

[0016] Furthermore, after S34, it also includes S35: Evaluate the Bayesian adaptive learning situation through the overall loss function, and the loss function includes confidence loss, classification loss, and bounding box regression loss.

[0017] The above settings can mainly evaluate through the confidence loss via label information, the classification loss can be evaluated through the classification information in the target box, and the bounding box regression loss can be evaluated through the size of the bounding box, etc., so as to determine the learning situation through the loss function and ensure the reliability of learning.

[0018] Furthermore, the step S5 includes: S51 If the Markov chain is in state j at time (t - 1) and the state transitions to i at time t, then the conditional probability distribution matrix is: ; is the state at time t; The state at time t-1; each state includes: the center point, size and category information of the target frame; the depth information of the image information; the current coordinate information, current speed information and current posture information; The Markov chain transition probability matrix finally formed by S52 is ; S53 estimates the output posture information of the mobile robot in the next state through the Markov chain transition probability matrix in the current state.

[0019] The above settings can predict future states by analyzing the current state and transition probability, and can be trained and adjusted based on actual data to adapt to different situations.

[0020] Furthermore, after step S53, it also includes: S54 determines the action in the current situation through a reward and penalty mechanism. If there is an obstacle on the right side, if it is estimated that the output posture information of the next state is to move to the right for collision, then a penalty process will be performed, so that the mobile robot will not consider the posture information that has been punished in the next posture information estimation process; if it is estimated that the output posture information of the next state is to move to the left for obstacle avoidance, then a reward process will be performed, so that the mobile robot will consider the posture information that has been rewarded in the next posture information estimation process.

[0021] The above settings can facilitate the mobile robot to adjust the estimated posture information in turn according to the subsequent actual posture information operation process, thereby ensuring the reliability of the estimated posture information.

[0022] Furthermore, S4 including S41 uses the least square method to adjust the error, such as the formula: ; Among them, N records the IMU safety data within a period of time, F is the expected data fed back by scanning the Aruco code, P is the predicted value, which contains the x, y, z position coordinates and speed information in the x, y, z directions respectively, and E is the error value.

[0023] S42 adjusts the predicted value of the mobile robot according to the error value.

[0024] The above settings may have complex motion and environmental conditions in actual applications. By establishing a nonlinear mapping relationship, these complex scenarios can be better handled, and the adaptability and robustness of the model can be improved. By establishing a nonlinear mapping relationship, the relationship between the inertial sensor measurement value and the actual physical quantity can be better handled, the measurement accuracy can be improved, the error can be reduced, and different sensor models and complex scenarios can be adapted. The specific benefits will vary depending on the application scenario and requirements. In actual applications, it is necessary to select appropriate mapping functions and models according to the specific situation to achieve the best effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is the work flow chart of the present invention. DETAILED DESCRIPTION

[0026] The invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] like Figure 1 As shown, a visual positioning method based on self-learning includes the following steps: S1 obtains image data; obtains image information of the surrounding environment through visual sensors; S2 image enhancement: first extract brightness information from the image, then use the histogram equalization method to remove point data that exceeds the preset brightness range, and form image information; S21 extracts the brightness information in the color channel through the formula Gray = A * R + C * G + D * B, where R, G, and B represent the three colors of red, green, and blue, respectively. In this embodiment, A=0.299; C=0.587; D=0.114.

[0028] S22 scans each pixel of the original grayscale image in sequence and calculates a grayscale histogram of the image; each pixel only contains a grayscale value.

[0029] S23 by formula Calculate the cumulative histogram of the grayscale histogram, where q and p are the brightness information corresponding to the output image and the input image respectively.

[0030] S23 by formula , calculate the mapping relationship between input and output, H and W are the height and width of the image respectively, and hist is the histogram.

[0031] S3 divides the acquired image information into grids based on the YOLOv5 algorithm, extracts feature information and label information from the grid information. The label information is the position and category of the target object in the annotated image. Then, the weight information of the feature information and the label information is adjusted through the Bayesian adaptive learning method. After that, the feature information is merged according to the adjusted weight information, and the merged image information is converted into three-dimensional coordinate information. S31 specifically includes S31 - S35; S31 extracts features through the backbone part, continuously shrinks the feature map through the convolutional module, and integrates the feature information into the channels; S32 is responsible for the fusion of features with the backbone network through the Neck module; S33 outputs detection frames of multiple scales through the head module, and the detection frames contain label information.

[0032] S34 The Bayesian adaptive learning method includes given known feature information and label information on the premise that ; where f(k) is the Monte Carlo sampling function, which takes a finite number of random variable samples to approximate the complex integral; x(k) is the k-th new sample, y(k) is the k-th new label; q(w) is the Gaussian distribution function, w is the weight information, q(y(k)|x(k)) is the output feature information and label information, p(y(k)|f(k)) is the sampling information corresponding to the feature label information in the input image, and p(y(k)|x(k), w) is the feature information, label information in the input image, and the weight information under the corresponding Gaussian distribution.

[0033] S35 evaluates the Bayesian adaptive learning situation through the overall loss function. The loss function includes confidence loss, classification loss, and regression loss of the bounding box. In this embodiment, the confidence loss is mainly determined by the difference between the confidence information in the label information and the preset confidence information. The classification loss determines whether there is a loss by comparing the classification situation in the target box with the classification situation in the actual target box. The regression loss of the bounding box is determined by whether the learned bounding box matches the actual bounding box. If the matching degree exceeds a preset size, such as 80%, it is considered that there is no loss. If the evaluation shows that the loss function has no loss, the image information is determined according to the current weight. If the evaluation shows that the loss function has a loss, it is prompted to adjust the weight information, which can be adjusted manually or in other ways.

[0034] S4 obtains the expected speed information within a preset time through the IMU sensor, and then determines the expected posture information by scanning the Aruco code, and then uses the least square method to adjust the error value between the expected speed information and the predicted speed information to determine the predicted speed information, and uses the least square method to adjust the error between the expected three-dimensional coordinate information and the predicted three-dimensional coordinate information to determine the predicted three-dimensional coordinate information. Step S4 includes S41 and S42; S41 uses the least squares method to adjust the error, as shown in the formula: ; Among them, N records the IMU safety data within a period of time, F is the expected data fed back by scanning the Aruco code, P is the predicted value, which contains the x, y, z position coordinates and speed information in the x, y, z directions respectively, and E is the error value.

[0035] S42 adjusts the predicted value of the mobile robot according to the error value.

[0036] S5 predicts the posture data through the Markov chain data transfer probability matrix, the predicted speed information, the predicted three-dimensional coordinate information, and the target frame information of the current position in the image information in step S3, and then performs positioning and movement according to the predicted posture data.

[0037] S5 includes S51-S53; S51 If the Markov chain is in state j at time (t-1) and changes to state i at time t, then the conditional probability distribution matrix is: ; is the state at time t; The state at time t-1; each state includes: the center point, size and category information of the target frame; the depth information of the image information; the current coordinate information, current speed information and current posture information; The Markov chain transition probability matrix finally formed by S52 is ; S53 estimates the output posture information of the mobile robot in the next state through the Markov chain transition probability matrix in the current state.

[0038] S54 determines the action in the current situation through a reward and penalty mechanism. If there is an obstacle on the right side, if it is estimated that the output posture information of the next state is to move to the right for collision, a penalty will be performed, so that the mobile robot will not consider the posture information that has been punished during the next posture information estimation process. If it is estimated that the output posture information of the next state is to move to the left for obstacle avoidance, a reward will be performed, so that the mobile robot will consider the posture information that has been rewarded during the next posture information estimation process.

[0039] The backbone network part, namely the backbone, is used to extract information features, and continuously shrinks the feature map through the convolution module to integrate the feature information into the channel; then the Neck module is responsible for integrating with the backbone network features of the front, on the one hand to reduce the probability of feature loss, and on the other hand the deep semantics obtained can greatly improve the level of model detection. Finally, the head module outputs multi-scale detection boxes, thereby realizing the network's detection effect for targets of different sizes.

[0040] The backbone is usually composed of a convolution module, a C3 model and an SPPF module. The convolution module is mainly a 3*3 convolution kernel. Its main purpose is to obtain the characteristics of the local area, compress the size of the original image, and abstract the prior knowledge of the original image to reach a level that the computer can understand.

[0041] The C3 module is an enhanced version of the convolution module, which has a more outstanding ability in feature extraction; in this embodiment, after the feature map enters the C3 module, it will be divided into two paths, the left path passes through the convolution layer and a Bottleneck module, and the right path only passes through a convolution layer. Finally, the two paths are connected and then pass through another convolution layer; the three convolution layer modules in C3 are all 1*1 convolutions, which play a role in dimensionality reduction or dimensionality increase. The Bottleneck is a residual connection module, which includes two convolution layers. The first Conv is a 1*1 convolution, which reduces the channel to half of the original, and the second is a 3*3 convolution, which doubles the number of channels. Reducing the dimension first helps the convolution kernel to better understand the feature information, and increasing the dimension will help to extract more and more detailed features. Finally, the residual structure is used to add the input and output to avoid the problem of gradient disappearance.

[0042] The SPPF module is spatial pyramid pooling. Its function is to convert feature maps with non-fixed scales into a unified scale. The implementation principle is to connect three parallel maximum pooling layers and inputs together, and use three kernels of different sizes to represent three scales. In this way, the maximum representative eigenvalue is obtained at different scales of the image and fused; through the Neck module data fusion, three multi-scale detection frames are output through the head module. In this embodiment, the YOLOv5 algorithm is an existing algorithm, and the specific neck module, head module, and backbone network part are existing modules in the YOLOv5 algorithm, and the specific working principle is no longer repeated again.

[0043] The working principle of the present invention is as follows: by acquiring a visual sensor to collect images, and then using a histogram equalization method to eliminate information of points with too high or too low brightness, thereby improving the effective recognition of the entire image, and then using the YOLOv5 algorithm to divide the image into multiple small units and extract feature information and label information. Since the label information is used to mark the feature position and category in the image, a Bayesian adaptive learning method is used to determine the weight value according to the feature information and weight information, and finally the weight used for merging the graphics is determined according to the adjusted weight value. Since the existing YOLOv5 algorithm generally uses a complex distribution function, the weight parameters cannot be determined in the final merging process of the neural network. The Bayesian adaptive learning method can determine the distribution function as a Gaussian distribution characteristic, because things in nature All items conform to the characteristics of Gaussian distribution, so this distribution can meet the requirements of most images. At the same time, the distribution has a certain degree of certainty and can form a certain weight function to ensure the accuracy of the image. In addition, the expected speed information and the expected three-dimensional coordinate information of the current position in the three-dimensional image are determined by the IMU sensor, and then the deviation between the predicted value and the expected value is adjusted by the least squares method. The predicted value is adjusted, and then the adjusted predicted value, the posture information obtained by scanning the Aruco code, the information of the target frame in the image information, and the Markov chain data transfer probability matrix are used to predict the posture data, so that multiple data such as speed, position, posture, image target frame information are fused, so that the posture is associated with the speed, position, posture and target frame information in the image, ensuring the reliability of the posture data.

Claims

1. A visual positioning method based on self-learning, characterized in that: The following steps are involved: S1 obtains image information of the surrounding environment through visual sensors; S2 image enhancement: first extract brightness information from the image, then use the histogram equalization method to remove point data whose brightness exceeds the preset brightness range, and form image information; S3 divides the acquired image information into grids based on the YOLOv5 algorithm, and extracts feature information and label information from the grid information. The label information is the location and category of the target object in the labeled image. Then, the weight information of the feature information and label information is adjusted through the Bayesian adaptive learning method. Then, the feature information is merged according to the adjusted weight information, and the merged image information is converted into three-dimensional coordinate information. S4 obtains the expected speed information within a preset time through the IMU sensor, and then determines the expected posture information by scanning the Aruco code, and then uses the least square method to adjust the error value between the expected speed information and the predicted speed information to determine the predicted speed information, and uses the least square method to adjust the error between the expected three-dimensional coordinate information and the predicted three-dimensional coordinate information to determine the predicted three-dimensional coordinate information; S5 predicts the posture data through the Markov chain data transfer probability matrix, the predicted speed information, the predicted three-dimensional coordinate information, and the target frame information of the current position in the image information in step S3, and then performs positioning and movement according to the predicted posture data.

2. The visual positioning method based on self-learning according to claim 1, characterized in that: The label information contains the category ID of the target object, the bounding box coordinates, and possible confidence.

3. The visual positioning method based on self-learning according to claim 1, characterized in that: The S2 comprises the steps of: S21 extracts the brightness information in the color channel through the formula Gray = A* R + C * G +D * B, where R, G, and B represent the three colors of red, green, and blue respectively; S22 scans each pixel of the original grayscale image in sequence and calculates the grayscale histogram of the image; Each pixel contains only grayscale values; S23 by formula Calculate the cumulative histogram of the grayscale histogram, where q and p are the brightness information corresponding to the output image and the input image respectively; S23 by formula , calculate the mapping relationship between input and output, H and W are the height and width of the image respectively, and hist is the histogram.

4. The visual positioning method based on self-learning according to claim 1, characterized in that: The step S3 comprises: S31 extracts features through the backbone network and continuously reduces the feature map through the convolution module, integrating the feature information into the channel; S32 is responsible for integrating features with the backbone network through the Neck module; S33 outputs a multi-scale detection frame through the head module, and the detection frame contains label information. S34 The Bayesian adaptive learning method includes given known feature information and label information On the premise of ; where f(k) is a Monte Carlo sampling function that takes a finite number of random variable samples to approximate a complex integral; x(k) is the k-th new sample, y(k) is the k-th new label; q(w) is a Gaussian distribution function, w is the weight information, q(y(k)|x(k)) is the output feature information and label information, p(y(k)|f(k)) is the sampling information corresponding to the feature label information in the input image, and p(y(k)|x(k), w) is the feature information, label information in the input image, and the weight information under the corresponding Gaussian distribution.

5. The self-learning based visual positioning method according to claim 4, characterized in that: S34 also includes S35: evaluating the adaptive learning of Bayesian through the overall loss function, the loss function includes confidence loss, classification loss and bounding box regression loss.

6. The self-learning based visual positioning method according to claim 1, characterized in that: The step S5 comprises: S51 If the Markov chain is in state j at time (t-1) and changes to state i at time t, then the conditional probability distribution matrix is: ; is the state at time t; The state at time t-1; each state includes: the center point, size and category information of the target frame; the depth information of the image information; the current coordinate information, current speed information and current posture information; The Markov chain transition probability matrix finally formed by S52 is ； S53 estimates the output posture information of the mobile robot in the next state through the Markov chain transition probability matrix in the current state.

7. The visual positioning method based on self-learning according to claim 6, characterized in that: Step S53 also includes: S54 determines the action in the current situation through a reward and penalty mechanism. If there is an obstacle on the right side, if it is estimated that the output posture information of the next state is to move to the right for collision, then a penalty will be performed, so that the mobile robot will not consider the posture information that has been punished in the next posture information estimation process. If it is estimated that the output posture information of the next state is to move to the left for obstacle avoidance, then a reward will be performed, so that the mobile robot will consider the posture information that has been rewarded in the next posture information estimation process.

Images