A visual positioning method based on self-learning

Abstract

The application discloses a self-learning visual positioning method, comprising the following steps: S1, acquiring image data; S2, image enhancement; first, extracting brightness information from the picture, S3, dividing the acquired image information into grids based on the YOLOv5 algorithm, and extracting feature information and label information in the grid information, then adjusting the weight information of the feature information and the label information through the adaptive learning method of Bayes, then merging the feature information according to the adjusted weight information, and converting the merged image information into three-dimensional coordinate information; S4, adjusting the predicted speed value and the predicted three-dimensional coordinate information through the least square method, S5, predicting the posture data through the Markov data transition probability matrix, the predicted speed information, the predicted three-dimensional coordinate information and the target frame information of the current position in the image information in the S3 step, and then positioning and moving according to the predicted posture data, and the positioning data is good in reliability.

Description

Technical Field

[0001] The present invention belongs to the field of positioning technology, and in particular relates to a visual positioning method based on self-learning. Background Art

[0002] Currently, common mobile robots include home service robots and AGV logistics robots. Home service robots primarily include educational robots, cleaning robots, and robots that assist the elderly and disabled. Specialized robots are playing an increasingly important role in specialized settings, such as exhibitions, restaurants, welcoming guests, shopping guides, security, and surveillance. Although complex mobile robots have revolutionized production and life, their capabilities still have limitations. In complex scenarios and areas with high performance requirements, there is still significant room for improvement in robot functionality and performance. For autonomous mobile robots, positioning and navigation are key issues, involving precise positioning, scene recognition, semantic understanding, obstacle analysis, and decision-making, requiring in-depth analysis and research.

[0003] Existing positioning technologies generally use a single sensor such as a visual sensor, laser sensor, or other sensor for positioning. However, when the surrounding environment has few textures or the texture distribution is regular, the positioning effect of a pure visual sensor is poor and track drift is prone to occur. Similarly, when there are few obstacles in the surrounding environment or the environment is regularly distributed, the pure laser sensor will not be able to determine the surrounding environment when there are no obstacles around, resulting in positioning drift. In order to solve the problem of single sensors losing positioning in certain environments, many studies have combined visual sensors and laser sensors with other sensors (such as IMU, encoders, etc.). Compared with pure vision or pure laser sensor odometers, fusion sensors have higher positioning accuracy, are not easy to lose accuracy in complex environments, and have better robustness.

[0004] To address these issues, a patent document, such as Chinese patent application number 202311372884.9, published on January 30, 2024, discloses a deep learning-based visual laser sensor fusion positioning method. The method specifically includes the following steps: collecting real-time data from a robot's lidar, camera, and IMU sensors; performing feature extraction and environmental recognition on the preprocessed data using a deep learning model; performing weighted fusion of the different sensor data based on the feature extraction and environmental recognition results to obtain an accurate pose estimate; and utilizing IMU and GPS data through back-end factor graph optimization to obtain an accurate pose estimate. The above document utilizes a deep learning model for point and line feature extraction and environmental recognition, thereby improving feature quality and positioning accuracy in low-texture environments. Through multi-sensor data fusion, the positioning accuracy and robustness of a mobile robot's SLAM system can be improved in low-texture environments, and real-time estimation of the robot's position and pose can be achieved.

[0005] However, this document only determines the overlap of data between multiple sensors through weighted values, and does not consider that the image data obtained by the visual sensor may be contaminated by noise, which may lead to deviations in the subsequent three-dimensional visual images, thereby causing inaccurate subsequent positioning. In addition, the posture control is only determined by the IMU sensor, and the final posture cannot be determined by comparing the detected posture with the predicted posture to ensure posture stability, thereby achieving reliable positioning. Summary of the Invention

[0006] The purpose of the present invention is to provide a visual positioning method based on self-learning, which can accurately identify and classify images, use the fusion data of visual sensors and IMU inertial sensors to achieve high-precision positioning of robots, and reliably realize posture control.

[0007] To achieve the above purpose, a visual positioning method based on self-learning includes the following steps:

[0008] S1 acquires image data; acquires image information of the surrounding environment through visual sensors;

[0009] S2 image enhancement: first extract brightness information from the image, then use histogram equalization to remove point data with brightness exceeding the preset brightness range, and form image information;

[0010] S3 divides the acquired image information into grids based on the YOLOv5 algorithm, and extracts feature information and label information from the grid information. The label information is the location and category of the target object in the marked image. Then, the weight information of the feature information and label information is adjusted through the Bayesian adaptive learning method. The feature information is then merged according to the adjusted weight information, and the merged image information is converted into three-dimensional coordinate information.

[0011] S4 obtains the expected speed information within a preset time through the IMU sensor, and then determines the expected posture information by scanning the Aruco code. Then, the least squares method is used to adjust the error value between the expected speed information and the predicted speed information to determine the predicted speed information. The least squares method is used to adjust the error value between the expected three-dimensional coordinate information and the predicted three-dimensional coordinate information to determine the predicted three-dimensional coordinate information.

[0012] S5 predicts the posture data through the Markov chain data transfer probability matrix, the predicted speed information, the predicted three-dimensional coordinate information, and the target frame information of the current position in the image information in step S3, and then performs positioning and movement according to the predicted posture data.

[0013] The above method collects images by acquiring visual sensors, and then uses the histogram equalization method to eliminate the information of points with too high or too low brightness, thereby improving the overall effective recognition of the image, and then uses the YOLOv5 algorithm to divide the image into multiple small units and extract feature information and label information. Since the label information is used to mark the feature position and category in the image, the Bayesian adaptive learning method is used to determine the weight value based on the feature information and weight information, and finally the weight used for merging the graphics is determined based on the adjusted weight value. Since the existing YOLOv5 algorithm generally uses a complex distribution function, the weight parameters cannot be determined in the final merging process of the neural network. The Bayesian adaptive learning method can determine the distribution function as a Gaussian distribution characteristic, because things in nature are all It conforms to the characteristics of Gaussian distribution, so the use of this distribution can meet the requirements of most images. At the same time, the distribution has a certain degree of certainty and can form a certain weight function to ensure the accuracy of the image. In addition, the expected speed information and the expected three-dimensional coordinate information of the current position in the three-dimensional image are determined by the IMU sensor, and then the deviation between the predicted value and the expected value is adjusted by the least squares method to adjust the predicted value. Then, the adjusted predicted value, the posture information obtained by scanning the Aruco code, the information of the target frame in the image information, and the Markov chain data transfer probability matrix are used to predict the posture data, so that multiple data such as speed, position, posture, image target frame information are fused, so that the posture is associated with the speed, position, posture and target frame information in the image, ensuring the reliability of the posture data.

[0014] Furthermore, the label information includes the category ID of the target object, the bounding box coordinates, and possible confidence.

[0015] With the above settings, the label information includes the category of the target object, the bounding box coordinates, and the confidence information, which makes it easier to determine the loss situation and finally determine the prediction result during the subsequent prediction.

[0016] Furthermore, the S2 comprises the steps of:

[0017] S21 extracts brightness information in the color channel using the formula Gray = A* R + C * G +D * B, where R, G, and B represent red, green, and blue, respectively.

[0018] S22 scans each pixel of the original grayscale image in sequence and calculates the grayscale histogram of the image; each pixel only contains the grayscale value;

[0019] S23 by formula Calculate the cumulative histogram of the grayscale histogram, where q and p are the brightness information corresponding to the output image and the input image respectively;

[0020] S23 by formula , calculate the mapping relationship between input and output, H and W are the height and width of the image respectively, and hist is the histogram.

[0021] The above settings use grayscale and histogram equalization processing mechanisms to remove pixels with too high or too low brightness in the image, making the detected targets more recognizable.

[0022] Furthermore, step S3 includes:

[0023] S31 extracts features through the backbone network and continuously reduces the feature map through the convolution module, integrating the feature information into the channel;

[0024] S32 is responsible for integrating features with the backbone network through the Neck module;

[0025] S33 outputs a multi-scale detection frame through the head module, and the detection frame contains label information.

[0026] The Bayesian adaptive learning method described in S34 includes given known feature information and label information Under the premise of

[0027] ; Wherein f(k) is the Monte Carlo sampling function, which takes a finite number of random variable samples to approximate the complex integral; x(k) is the kth new sample, y(k) is the kth new label; q(w) is the Gaussian distribution function, w is the weight information, q(y(k)丨x(k)) is the output feature information and label information, p(y(k)丨f(k)) is the sampling information corresponding to the feature label information in the input image, p(y(k)丨x(k), w) is the feature information, label information and weight information under the corresponding Gaussian distribution in the input image.

[0028] The above settings ensure the accuracy of identifying target categories and introduce a Bayesian-based network weight adaptive adjustment method to reduce the uncertainty in model detection.

[0029] Furthermore, S34 also includes S35: evaluating the Bayesian adaptive learning through the overall loss function, which includes confidence loss, classification loss and bounding box regression loss.

[0030] With the above settings, the confidence loss can be evaluated mainly through label information, the classification loss can be evaluated through the classification information in the target box, and the bounding box regression loss can be evaluated through the size of the bounding box, etc., so that the learning situation can be determined through the loss function to ensure the reliability of learning.

[0031] Furthermore, the step S5 includes:

[0032] S51 If the Markov chain is in state j at time (t-1) and changes to state i at time t, then the conditional probability distribution matrix is: ; is the state at time t; The state at time t-1; each state includes: the center point, size and category information of the target box; the depth information of the image information; the current coordinate information, current speed information and current posture information;

[0033] The Markov chain transition probability matrix finally formed by S52 is

[0034] ;

[0035] S53 estimates the output posture information of the mobile robot in the next state through the Markov chain transition probability matrix in the current state.

[0036] The above settings can predict future states by analyzing the current state and transition probability, and can be trained and adjusted based on actual data to adapt to different situations.

[0037] Furthermore, step S53 also includes: S54 determines the action in the current situation through a reward and punishment mechanism. If there is an obstacle on the right side, if it is estimated that the output posture information of the next state is to move to the right for collision, then a penalty process will be performed, so that the mobile robot will not consider the posture information that has been punished in the next posture information estimation process; if it is estimated that the output posture information of the next state is to move to the left for obstacle avoidance, then a reward process will be performed, so that the mobile robot will consider the posture information that has been rewarded in the next posture information estimation process.

[0038] The above settings can facilitate the mobile robot to adjust the estimated posture information according to the subsequent actual posture information operation process, thereby ensuring the reliability of the estimated posture information.

[0039] Furthermore, S4 including S41 uses the least square method to adjust the error, such as the formula:

[0040] ;

[0041] Among them, N records the IMU safety data over a period of time, F is the expected data fed back by scanning the Aruco code, P is the predicted value, which includes the x, y, z position coordinates and speed information in the x, y, z directions respectively, and E is the error value.

[0042] S42 adjusts the predicted value of the mobile robot according to the error value.

[0043] The above settings may have complex motion and environmental conditions in actual applications. By establishing a nonlinear mapping relationship, these complex scenarios can be better handled, and the adaptability and robustness of the model can be improved. By establishing a nonlinear mapping relationship, the relationship between the inertial sensor measurement value and the actual physical quantity can be better handled, and the measurement accuracy can be improved, the error can be reduced, and it can adapt to different sensor models and complex scenarios. The specific benefits will vary depending on the application scenario and requirements. In actual applications, it is necessary to select appropriate mapping functions and models according to the specific situation to achieve the best effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the workflow diagram of the present invention. DETAILED DESCRIPTION

[0045] The invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 1 As shown, a visual positioning method based on self-learning includes the following steps:

[0047] S1 acquires image data; obtains image information of the surrounding environment through visual sensors;

[0048] S2 image enhancement: first extract brightness information from the image, then use histogram equalization to remove point data that exceeds the preset brightness range, and form image information;

[0049] S21 extracts the brightness information in the color channel using the formula Gray = A * R + C * G + D * B, where R, G, and B represent red, green, and blue, respectively. In this embodiment, A = 0.299; C = 0.587; and D = 0.114.

[0050] S22 scans each pixel of the original grayscale image in sequence and calculates the grayscale histogram of the image; each pixel only contains the grayscale value.

[0051] S23 by formula Calculate the cumulative histogram of the grayscale histogram, where q and p are the brightness information corresponding to the output image and the input image respectively.

[0052] S23 by formula , calculate the mapping relationship between input and output, H and W are the height and width of the image respectively, and hist is the histogram.

[0053] S3 divides the acquired image information into grids based on the YOLOv5 algorithm, and extracts feature information and label information from the grid information. The label information is the location and category of the target object in the marked image. Then, the weight information of the feature information and label information is adjusted through the Bayesian adaptive learning method. The feature information is then merged according to the adjusted weight information, and the merged image information is converted into three-dimensional coordinate information.

[0054] S31 specifically includes S31-S35;

[0055] S31 extracts features through the backbone network and continuously reduces the feature map through the convolution module to integrate the feature information into the channel;

[0056] S32 is responsible for integrating features with the backbone network through the Neck module;

[0057] S33 outputs a multi-scale detection frame through the head module, and the detection frame contains label information.

[0058] The Bayesian adaptive learning method described in S34 includes given known feature information and label information Under the premise of

[0059] ; Wherein f(k) is the Monte Carlo sampling function, which takes a finite number of random variable samples to approximate the complex integral; x(k) is the kth new sample, y(k) is the kth new label; q(w) is the Gaussian distribution function, w is the weight information, q(y(k)丨x(k)) is the output feature information and label information, p(y(k)丨f(k)) is the sampling information corresponding to the feature label information in the input image, p(y(k)丨x(k), w) is the feature information, label information and weight information under the corresponding Gaussian distribution in the input image.

[0060] S35 evaluates the adaptive learning of Bayesian through the overall loss function, and the loss function includes confidence loss, classification loss and regression loss of the bounding box. In this embodiment, the confidence loss is mainly determined by the difference between the confidence information in the label information and the preset confidence information. The classification loss is determined by the classification situation in the target box and the classification situation in the actual target box to determine whether there is a loss. The regression loss of the bounding box is determined by whether the learned bounding box matches the actual bounding box. If the matching degree exceeds the preset size, such as 80%, it is determined that there is no loss. If the loss function is evaluated to have no loss, the image information is determined according to the current weight. If the loss function is evaluated to have a loss, it is prompted to adjust the weight information, which can be adjusted by thinking or other methods.

[0061] S4 obtains the expected speed information within a preset time through the IMU sensor, then determines the expected posture information by scanning the Aruco code, and then uses the least squares method to adjust the error value between the expected speed information and the predicted speed information to determine the predicted speed information. The least squares method is used to adjust the error value between the expected three-dimensional coordinate information and the predicted three-dimensional coordinate information to determine the predicted three-dimensional coordinate information. Step S4 includes S41 and S42;

[0062] S41 uses the least squares method to adjust the error, as shown in the formula:

[0063] ;

[0064] Among them, N records the IMU safety data over a period of time, F is the expected data fed back by scanning the Aruco code, P is the predicted value, which includes the x, y, z position coordinates and speed information in the x, y, z directions respectively, and E is the error value.

[0065] S42 adjusts the predicted value of the mobile robot according to the error value.

[0066] S5 predicts the posture data through the Markov chain data transfer probability matrix, the predicted speed information, the predicted three-dimensional coordinate information, and the target frame information of the current position in the image information in step S3, and then performs positioning and movement according to the predicted posture data.

[0067] S5 includes S51-S53;

[0068] S51 If the Markov chain is in state j at time (t-1) and changes to state i at time t, then the conditional probability distribution matrix is: ; is the state at time t; The state at time t-1; each state includes: the center point, size and category information of the target box; the depth information of the image information; the current coordinate information, current speed information and current posture information;

[0069] The Markov chain transition probability matrix finally formed by S52 is

[0070] ;

[0071] S53 estimates the output posture information of the mobile robot in the next state through the Markov chain transition probability matrix in the current state.

[0072] S54 determines the action in the current situation through a reward and punishment mechanism. If there is an obstacle on the right side, if it is estimated that the output posture information of the next state is to move to the right for collision, a penalty process will be performed, so that the mobile robot will not consider the posture information that has been punished in the next posture information estimation process. If it is estimated that the output posture information of the next state is to move to the left for obstacle avoidance, a reward process will be performed, so that the mobile robot will consider the posture information that has been rewarded in the next posture information estimation process.

[0073] The backbone network part, namely the backbone, is used to extract information features, and continuously shrinks the feature map through the convolution module to integrate the feature information into the channel; then the Neck module is responsible for integrating with the backbone network features of the front, on the one hand reducing the probability of feature loss, and on the other hand the deep semantics obtained can greatly improve the level of model detection. Finally, the head module outputs multi-scale detection boxes, thereby realizing the network's detection effect for targets of different sizes.

[0074] The backbone is usually composed of a convolution module, a C3 model, and an SPPF module. The convolution module is mainly a 3*3 convolution kernel. Its main purpose is to obtain the characteristics of the local area, compress the size of the original image, and abstract the prior knowledge of the original image to a level that the computer can understand.

[0075] The C3 module is an enhanced version of the convolution module, which has more outstanding capabilities in feature extraction. In this embodiment, after the feature map enters the C3 module, it will be divided into two paths. The left path passes through the convolution layer and a Bottleneck module, and the right path only passes through one convolution layer. Finally, the two paths are connected and then pass through another convolution layer. The three convolution layer modules in C3 are all 1*1 convolutions, which play the role of dimensionality reduction or dimensionality increase. The Bottleneck is a residual connection module, which contains two convolution layers. The first Conv is a 1*1 convolution, which reduces the channel to half of the original, and the second is a 3*3 convolution, which doubles the number of channels. Reducing the dimension first helps the convolution kernel better understand the feature information, and increasing the dimension will help extract more and more detailed features. Finally, the residual structure is used to add the input and output to avoid the problem of gradient disappearance.

[0076] The SPPF module is spatial pyramid pooling. Its function is to convert feature maps with non-fixed scales into a unified scale. Its implementation principle is to connect three parallel maximum pooling layers and inputs together, and use three kernels of different sizes to represent three scales. In this way, the maximum representative eigenvalues ​​are obtained at different scales of the image and fused; through the Neck module data fusion, three multi-scale detection frames are output through the head module. In this embodiment, the YOLOv5 algorithm is an existing algorithm, and the specific neck module, head module, and backbone network part are existing modules in the YOLOv5 algorithm. The specific working principle will not be repeated again.

[0077] The working principle of the present invention is as follows: by acquiring a visual sensor to collect images, and then using the histogram equalization method to eliminate the information of points with too high or too low brightness, thereby improving the effective recognition of the entire image, and then using the YOLOv5 algorithm to divide the image into multiple small units and extract feature information and label information. Since the label information is used to mark the feature position and category in the image, the Bayesian adaptive learning method is used to determine the weight value according to the feature information and weight information, and finally the weight used for merging the graphics is determined according to the adjusted weight value. Since the existing YOLOv5 algorithm generally uses a complex distribution function, the weight parameters cannot be determined in the final merging process of the neural network. The Bayesian adaptive learning method can determine the distribution function as a Gaussian distribution characteristic, because things in nature are The items all conform to the Gaussian distribution characteristics, so this distribution can meet the requirements of most images. At the same time, the distribution has a certain degree of certainty and can form a certain weight function to ensure the accuracy of the image. In addition, the expected speed information and the expected three-dimensional coordinate information of the current position in the three-dimensional image are determined by the IMU sensor, and then the deviation between the predicted value and the expected value is adjusted by the least squares method to adjust the predicted value. Then, the adjusted predicted value, the posture information obtained by scanning the Aruco code, the information of the target frame in the image information, and the Markov chain data transfer probability matrix are used to predict the posture data, so that multiple data such as speed, position, posture, and image target frame information are fused, so that the posture is associated with the speed, position, posture, and target frame information in the image, ensuring the reliability of the posture data.

Claims

1. A visual positioning method based on self-learning, characterized by: The following steps are involved: S1 obtains image information of the surrounding environment through visual sensors; S2 image enhancement: first extract brightness information from the image, then use histogram equalization to remove point data with brightness exceeding the preset brightness range, and form image information; S3 divides the acquired image information into grids based on the YOLOv5 algorithm, and extracts feature information and label information from the grid information. The label information is the location and category of the target object in the marked image. Then, the weight information of the feature information and label information is adjusted through the Bayesian adaptive learning method. The feature information is then merged according to the adjusted weight information, and the merged image information is converted into three-dimensional coordinate information. S4 obtains the expected speed information within a preset time through the IMU sensor, and then determines the expected posture information by scanning the Aruco code. Then, the least squares method is used to adjust the error value between the expected speed information and the predicted speed information to determine the predicted speed information. The least squares method is used to adjust the error value between the expected three-dimensional coordinate information and the predicted three-dimensional coordinate information to determine the predicted three-dimensional coordinate information. S5 predicts the posture data through the Markov chain data transfer probability matrix, the predicted speed information, the predicted three-dimensional coordinate information, and the target frame information of the current position in the image information in step S3, and then performs positioning and movement according to the predicted posture data.

2. The self-learning-based visual positioning method according to claim 1, wherein: The label information contains the target object’s category ID, bounding box coordinates, and possible confidence.

3. The self-learning-based visual positioning method according to claim 1, wherein: The S2 comprises the steps of: S21 extracts brightness information in the color channel using the formula Gray = A* R + C * G +D * B, where R, G, and B represent red, green, and blue, respectively. S22 scans each pixel of the original grayscale image in sequence and calculates the grayscale histogram of the image; Each pixel contains only grayscale values; S23 by formula Calculate the cumulative histogram of the grayscale histogram, where q and p are the brightness information corresponding to the output image and the input image respectively; S23 by formula , calculate the mapping relationship between input and output, H and W are the height and width of the image respectively, and hist is the histogram.

4. The self-learning-based visual positioning method according to claim 1, wherein: The step S3 comprises: S31 extracts features through the backbone network and continuously reduces the feature map through the convolution module, integrating the feature information into the channel; S32 is responsible for integrating features with the backbone network through the Neck module; S33 outputs a multi-scale detection frame through the head module, and the detection frame contains label information; The Bayesian adaptive learning method described in S34 includes given known feature information and label information Under the premise of ; where f(k) is a Monte Carlo sampling function that takes a finite number of random variable samples to approximate a complex integral; x(k) is the k-th new sample, y(k) is the k-th new label; q(w) is a Gaussian distribution function, w is weight information, q(y(k)|x(k)) is the output feature information and label information, p(y(k)|f(k)) is the sampling information corresponding to the feature label information in the input image, and p(y(k)|x(k), w) is the feature information, label information in the input image, and the weight information under the corresponding Gaussian distribution.

5. The self-learning-based visual positioning method according to claim 4, characterized in that: S34 also includes S35: Evaluating the Bayesian adaptive learning through the overall loss function, which includes confidence loss, classification loss and bounding box regression loss.

6. The self-learning-based visual positioning method according to claim 1, wherein: The step S5 comprises: S51 If the Markov chain is in state j at time (t-1) and changes to state i at time t, then the conditional probability distribution matrix is: ; is the state at time t; The state at time t-1; each state includes: the center point, size and category information of the target box; the depth information of the image information; the current coordinate information, current speed information and current posture information; The Markov chain transition probability matrix finally formed by S52 is ； S53 estimates the output posture information of the mobile robot in the next state through the Markov chain transition probability matrix in the current state.

7. The self-learning-based visual positioning method according to claim 6, characterized in that: Step S53 also includes: S54 determines the action in the current situation through a reward and punishment mechanism. If there is an obstacle on the right side, if it is estimated that the output posture information of the next state is moving to the right for collision, then a penalty process will be performed, so that the mobile robot will not consider the posture information that has been punished in the next posture information estimation process; if it is estimated that the output posture information of the next state is moving to the left for obstacle avoidance, then a reward process will be performed, so that the mobile robot will consider the posture information that has been rewarded in the next posture information estimation process.

Images