Obstacle avoidance and bypassing method and system based on multi-sensor fusion for obstacle perception

Abstract

The application provides an obstacle avoidance and obstacle bypassing method and system based on multi-sensor fusion for obstacle perception, fuses acquired mobile robot vision data to obtain a vision data preliminary fusion result; fuses acquired mobile robot laser radar data to obtain a laser radar data preliminary fusion result; fuses acquired mobile robot ultrasonic data to obtain an ultrasonic data preliminary fusion result; based on a multi-sensor fusion algorithm of D-S evidence theory, globally fuses multi-sensor data of the vision data preliminary fusion result, the laser radar data preliminary fusion result and the ultrasonic data preliminary fusion result to obtain final robot surrounding obstacle information; controls robot obstacle bypassing according to the obtained obstacle information; the application realizes hierarchical and zoned perception of obstacles, can accurately perceive obstacles and make decisions on mobile robot obstacle bypassing and avoidance actions, so that the robot can more accurately avoid or bypass obstacles, and collision with obstacles and pedestrians is avoided.

Description

Technical Field

[0001] This invention relates to the field of robot navigation and control technology, and in particular to an obstacle avoidance and bypass method and system based on multi-sensor fusion for obstacle perception. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] During the daily operations of mobile robots, we usually pre-set the walking route for the robot. Since the working environment of the robot is relatively complex, unpredictable obstacles may appear to block the robot's path during the task. If the robot does not have the function of obstacle avoidance and obstacle bypass, it may collide with pedestrians or other obstacles, thereby causing accidents and unnecessary consequences.

[0004] In the field of robotics, the use of multi-sensor fusion for environmental perception has become a trend. The various sensors used by robots have their own advantages and disadvantages. For example, 2D cameras can effectively identify objects and clearly define the edges of targets, but they are sensitive to ambient lighting conditions and lack environmental depth information. LiDAR (2D-Lidar) has poor detection performance for transparent and dark-colored objects and is prone to missed detections. The rapid development of multi-sensor perception fusion algorithms has promoted the application of multi-sensor fusion to identify and perceive the environment of mobile robots.

[0005] The purpose of multi-sensor information fusion is to use reasonable analysis and solutions to complement the advantages and disadvantages of multiple sensors, and to make up for the accuracy and missed detection problems of single sensor data. Through multi-sensor fusion strategies, robots can make accurate decisions on obstacle avoidance and bypass based on the results of multi-sensor information fusion. When a robot is performing a task in a narrow corridor, its left and right sides are close to the walls. In this case, the robot should be able to perceive the surrounding obstacles and choose whether to bypass them from the left or the right, thus making the robot's operation process more intelligent.

[0006] The inventors discovered that existing multi-sensor fusion strategies are generally divided into centralized fusion and distributed fusion. Centralized fusion has a large computational load and high communication costs, while distributed fusion systems have long response times and low efficiency. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method and system for obstacle avoidance and bypass based on multi-sensor fusion. By fusing information from multiple sensors, it achieves hierarchical and zoned perception of obstacles, enabling precise perception of obstacles and decision-making for the mobile robot's obstacle avoidance and bypass actions. This allows the robot to perform more accurate obstacle avoidance or bypass, thus preventing collisions with obstacles.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides an obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception.

[0010] An obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception includes the following steps:

[0011] The acquired visual data of the mobile robot is initially fused to obtain the preliminary visual data fusion result;

[0012] The acquired LiDAR data of the mobile robot is initially fused to obtain the preliminary fusion result of the LiDAR data;

[0013] The acquired ultrasonic data from the mobile robot were initially fused to obtain preliminary fusion results.

[0014] Based on the DS evidence theory, a multi-sensor fusion algorithm is used to perform global fusion of visual data, lidar data, and ultrasonic data to obtain the final obstacle information around the robot.

[0015] The robot controls obstacle avoidance based on the obtained obstacle information.

[0016] As one possible implementation of the first aspect of the present invention, a first set range in front of the robot is set as a first-level obstacle avoidance area, a second set range to the left front of the first-level obstacle avoidance area is set as a left front obstacle avoidance area, and a third set distance range to the right front of the first-level obstacle avoidance area is set as a right front obstacle avoidance area.

[0017] The primary obstacle avoidance area, the left front obstacle avoidance area, and the right front obstacle avoidance area are adjacent to each other. The left front obstacle avoidance area and the right front obstacle avoidance area are symmetrically arranged along the extension of the center line of the primary obstacle avoidance area. The center line of the primary obstacle avoidance area is collinear with the center line of the robot's forward direction.

[0018] As a further limitation of the first aspect of the present invention, the acquired mobile robot visual data is initially fused to obtain a preliminary visual data fusion result, including:

[0019] Preprocess the point cloud data of the robot from multiple directions;

[0020] Based on the preprocessed point cloud data, point cloud feature points are obtained at the left front and right front of the robot. If the point cloud feature points are not in the obstacle avoidance area of ​​the robot, they are not processed. If the point cloud feature points are in the obstacle avoidance area, the point cloud feature points are compared with the set obstacle feature point threshold.

[0021] When the number of point cloud feature points of the obstacle on the left front side is greater than the set threshold, and when the number of point cloud feature points of the obstacle on the left front side is not greater than the set threshold, obstacle avoidance control in the left front obstacle avoidance area is executed; otherwise, obstacle avoidance or obstacle bypass is not triggered.

[0022] When the number of point cloud feature points of the obstacle on the right front side is greater than the set threshold, and the number of point cloud feature points of the obstacle on the left front side is not greater than the set threshold, obstacle avoidance control in the right front obstacle avoidance area is executed; otherwise, obstacle avoidance or obstacle bypass is not triggered.

[0023] When the number of point cloud feature points of the obstacle on the left front side is greater than a given threshold, the center coordinates of the obstacle on the left front side and the center coordinates of the obstacle on the right front side are calculated. The first closest distance between the center coordinates of the obstacle on the left front side and the front and rear of the robot and the second closest distance between the center coordinates of the obstacle on the right front side and the front and rear of the robot are calculated. The obstacle avoidance area corresponding to the minimum value of the first closest distance and the second closest distance is selected for obstacle avoidance control. The obstacle avoidance control results are used as the preliminary visual data fusion results.

[0024] As a further limitation of the first aspect of the present invention, when the first nearest distance is less than the distance of the first obstacle avoidance area along the robot axis, or the second nearest distance is less than the distance of the first obstacle avoidance area along the robot axis, the size of the obstacle on the left and the obstacle on the right are compared, and the obstacle avoidance area with the smaller obstacle is selected for obstacle avoidance control.

[0025] As one possible implementation of the first aspect of the present invention, a first set range in front of the robot is set as a first obstacle avoidance area, a second set range to the left front of the first obstacle avoidance area is set as a first obstacle bypass area, a third set distance range to the right front of the first obstacle avoidance area is set as a second obstacle bypass area, the left side of the first obstacle bypass area is set as a third obstacle bypass area, and the right side of the second obstacle bypass area is set as a fourth obstacle bypass area.

[0026] The first, second, third, and fourth obstacle avoidance areas are all located in front of the primary obstacle avoidance area. The first and second obstacle avoidance areas are symmetrically arranged along the extension of the centerline of the primary obstacle avoidance area, and the third and fourth obstacle avoidance areas are symmetrically arranged along the extension of the centerline of the primary obstacle avoidance area. The centerline of the primary obstacle avoidance area is collinear with the central axis of the robot's forward direction.

[0027] As a further limitation of the first aspect of the present invention, a radar obstacle avoidance distance threshold is set, with the robot's forward direction as the positive X direction and the left side of the X direction as the positive Y direction.

[0028] When the distance between the obstacle and the robot is greater than the radar obstacle avoidance distance threshold, and the distance between the obstacle and the robot is within the obstacle avoidance range in the X direction;

[0029] If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the first obstacle avoidance area, the obstacle triggering cumulative number in the first obstacle avoidance area is incremented by 1.

[0030] If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the second obstacle avoidance area, then the obstacle is in the robot's second obstacle avoidance area, and the obstacle triggering cumulative increment in the second obstacle avoidance area is 1.

[0031] If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the third obstacle avoidance area, the obstacle triggering cumulatively in the third obstacle avoidance area is incremented by 1.

[0032] If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the fourth obstacle avoidance area, then the obstacle is in the robot's fourth obstacle avoidance area, and the obstacle triggering cumulatively in the fourth obstacle avoidance area is incremented by 1.

[0033] If the cumulative number of obstacle triggers in the obstacle avoidance area is N after N consecutive frames of detection, then a valid obstacle is considered to have been detected, and obstacle avoidance control is triggered, where N is an integer greater than or equal to 3; otherwise, it is considered that no valid obstacle has been detected, and obstacle avoidance control is not triggered.

[0034] As a further definition of the first aspect of the present invention, the area in front of the primary obstacle avoidance area is the secondary obstacle avoidance area, and the area in front of the secondary obstacle avoidance area is the tertiary obstacle avoidance area.

[0035] The first obstacle avoidance area, the second obstacle avoidance area, the third obstacle avoidance area, and the fourth obstacle avoidance area are all divided into secondary obstacle avoidance areas and tertiary obstacle avoidance areas along the robot's direction of movement;

[0036] When the road conditions ahead of the robot do not meet the conditions for obstacle avoidance, obstacle avoidance control is implemented, including:

[0037] When the robot is in the level 3 obstacle avoidance zone, it does not slow down but triggers an alarm. When the robot is in the level 2 obstacle avoidance zone, it begins to slow down and triggers an alarm. When the robot is in the level 1 obstacle avoidance zone, it stops moving forward and triggers an alarm.

[0038] The second aspect of the present invention provides an obstacle avoidance and bypass system based on multi-sensor fusion for obstacle perception.

[0039] An obstacle avoidance and bypass system based on multi-sensor fusion for obstacle perception includes:

[0040] The visual data fusion module is configured to perform preliminary fusion of the acquired mobile robot visual data to obtain preliminary visual data fusion results.

[0041] The lidar data fusion module is configured to: perform preliminary fusion of the acquired lidar data of the mobile robot to obtain preliminary lidar data fusion results;

[0042] The ultrasonic data fusion module is configured to perform preliminary fusion of the acquired ultrasonic data from the mobile robot to obtain preliminary ultrasonic data fusion results.

[0043] The multi-sensor data fusion module is configured to: perform global fusion of multi-sensor data based on the DS evidence theory multi-sensor fusion algorithm, and perform preliminary fusion of visual data, LiDAR data, and ultrasonic data to obtain the final obstacle information around the robot;

[0044] The obstacle avoidance control module is configured to control the robot to avoid obstacles based on the obtained obstacle information.

[0045] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception as described in the first aspect of the present invention.

[0046] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception as described in the first aspect of the present invention.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] (1) This invention improves the distributed and centralized fusion methods, analyzes their respective advantages and disadvantages, and uses a hybrid multi-sensor data fusion method to achieve multi-sensor data fusion. First, the data of each sensor is preprocessed and initially fused, which greatly reduces the amount of data to be processed and the speed of data fusion.

[0049] (2) By dividing the areas detected by different sensors into separate zones, the present invention makes the robot's obstacle perception more accurate, thereby making the robot's judgment of the direction of obstacle avoidance and obstacle bypass more precise.

[0050] (3) By fusing data from cameras, lidar, and ultrasonic sensors, obstacle perception is achieved, thereby enabling the selection of the robot's obstacle avoidance direction and level, and improving the accuracy and efficiency of obstacle avoidance control.

[0051] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0052] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0053] Figure 1 This is a schematic diagram of the multi-sensor data fusion model provided in Embodiment 1 of the present invention;

[0054] Figure 2 This is a schematic diagram of the evidence information fusion process of the DS evidence theory provided in Embodiment 1 of the present invention;

[0055] Figure 3 This is a schematic diagram of the obstacle avoidance area partitioning of the mobile robot's visual sensor provided in Embodiment 1 of the present invention;

[0056] Figure 4 This is a schematic diagram of the obstacle perception process of the visual sensor provided in Embodiment 1 of the present invention;

[0057] Figure 5 This is a schematic diagram of the obstacle perception and obstacle avoidance area partitioning provided by the lidar sensor in Embodiment 1 of the present invention;

[0058] Figure 6 This is a schematic diagram of the three-level obstacle avoidance area partitioning for the lidar sensor obstacle perception provided in Embodiment 1 of the present invention;

[0059] Figure 7 This is a schematic diagram of the obstacle perception process of the lidar sensor provided in Embodiment 1 of the present invention;

[0060] Figure 8 This is a schematic diagram of the installation position of the ultrasonic sensor provided in Embodiment 1 of the present invention.

[0061] Figure 9 This is a schematic diagram of the obstacle avoidance and bypass system based on multi-sensor fusion for obstacle perception provided in Embodiment 2 of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0064] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0065] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0066] Example 1:

[0067] Embodiment 1 of the present invention provides an obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception. By installing lidar, camera and four sets of ultrasonic sensors at different positions of the robot, the robot can perceive obstacles around the robot during operation, thereby realizing accurate three-level obstacle avoidance and bypass of the robot according to a preset path.

[0068] This invention analyzes the advantages and disadvantages of distributed fusion and centralized fusion, and proposes a hybrid multi-sensor fusion framework for indoor mobile robots, such as... Figure 1 As shown, data from multiple sensors are first preprocessed and initially fused at their respective intermediate stations, then fused together to finally output the final decision information.

[0069] The multi-sensor fusion obstacle perception method described in this embodiment is divided into four main modules: hybrid multi-sensor information fusion, visual sensor obstacle perception, lidar obstacle perception, and ultrasonic sensor obstacle perception. By using multi-sensor fusion to fuse information from multiple sensors, obstacle classification and zoning perception are achieved. This enables accurate perception and decision-making for the mobile robot's obstacle avoidance and bypass actions, allowing the robot to perform more precise obstacle avoidance or bypass and avoid collisions with obstacles.

[0070] Specifically, it includes the following:

[0071] S1: Install a lidar sensor on the top of the mobile robot's chassis, install a camera vision sensor on the vehicle body above the chassis, and install a total of 8 ultrasonic sensors in 4 groups in the four directions of front, back, left, and right of the mobile robot's chassis.

[0072] S2: The data obtained from the camera sensor, lidar sensor, and ultrasonic sensor are preprocessed at the intermediate station. Appropriate methods are used to delete and filter the sensor information. The visual sensor information is filtered using PCL point cloud pass-through filtering to filter the point cloud information within the set limit area. Data that is not considered or has low support is deleted to reduce the amount of computation.

[0073] S3: Establish the lidar coordinate system, vision coordinate system, and ultrasonic coordinate system. The robot coordinate system follows the right-hand coordinate system, with the center of the lidar as the origin, the positive X-axis in front, and the positive Y-axis to the left of the robot lidar. The lidar coordinate system, vision coordinate system, and ultrasonic coordinate system are unified through rigid body transformation.

[0074] S4: At the intermediate station, the data from multiple sensors that have undergone preprocessing are initially fused using the least squares method to obtain the optimal fused data from a single sensor.

[0075] S5: Then, the DS evidence theory algorithm is used to transmit the locally optimal fusion data obtained by the three sensors to the fusion center for global fusion of multi-sensor information to obtain the final fusion result. The fusion result is used to determine the obstacles around the robot, thereby enabling the robot to make the correct obstacle avoidance or obstacle bypass judgment.

[0076] Among them, visual obstacle perception mainly obtains obstacle point cloud information through visual sensors, and uses PCL point cloud pass-through filtering to filter the point cloud information in the set area. It uses feature extraction methods to extract feature points of obstacles, processes the obtained point cloud pose, calculates the midpoint coordinates of obstacles, and determines whether the robot triggers obstacle avoidance or obstacle bypass by comparing the number of feature points in each direction and the distance of the obstacle relative to the robot, and which direction the obstacle avoidance or obstacle bypass is triggered.

[0077] LiDAR obstacle perception mainly obtains the pose of the obstacle relative to the robot's coordinate system through LiDAR, sets the obstacle avoidance area and three levels of obstacle avoidance area for the robot, and divides the area in front of the robot that needs to be detected for obstacles into four obstacle avoidance zones: left-left, left-front, right-front, and right-right; so that the robot can make the correct obstacle avoidance or obstacle avoidance judgment.

[0078] Ultrasonic obstacle perception mainly involves installing a set of ultrasonic sensors at the front, back, left, and right of the robot chassis. By using the ranging function of the ultrasonic sensors, obstacles around the robot can be detected. Objects that are missed by cameras and lidar can be detected again, thereby achieving the purpose of precise obstacle avoidance.

[0079] By processing and fusing information from multiple sensors through hybrid multi-sensor fusion, and performing tasks according to a preset path, the robot can perceive obstacles. This improves the accuracy and robustness of obstacle perception during operation, thereby enabling the robot to accurately and intelligently avoid and bypass obstacles.

[0080] A: Multi-sensor fusion includes:

[0081] The purpose of multi-sensor information fusion is to use reasonable methods and strategies to complement the disadvantages of cameras, laser sensors, and ultrasonic sensors by fusing redundant and complementary data. This process compensates for the low accuracy and missed detections of data from a single sensor, enabling the robot to perceive obstacles and make correct action decisions.

[0082] A1: Install multiple sensors appropriately in suitable locations; install a LiDAR sensor on the top of the mobile robot's chassis, a camera on the upper part of the vehicle body above the chassis, and a total of 8 ultrasonic sensors in 4 groups in the four directions (front, rear, left, and right) of the mobile robot's chassis. Detect vehicles ahead using cameras, LiDAR, and ultrasonic sensors. Since each sensor has different operating methods, installation positions, coordinate systems, and data transmission / reception methods, multi-sensor data fusion is required.

[0083] A2: Achieve spatial synchronization of multi-sensor data; To achieve spatial synchronization of multi-sensor data, the target information acquired by all sensors must first be transformed into a unified world coordinate system. Establish a lidar coordinate system, a vision coordinate system, and an ultrasonic coordinate system. The robot coordinate system follows a right-handed coordinate system, with the center of the lidar as the origin, the positive X-axis directly in front, and the positive Y-axis to the left of the robot's lidar. Unify the lidar coordinate system, the camera coordinate system, and the various ultrasonic coordinate systems through coordinate transformation.

[0084] A3: Achieve time synchronization of multi-sensor data; since each sensor operates in a different way, time synchronization is required. Given the operating frequency of each sensor, the LiDAR is used as a reference, and interpolation is employed to synchronize the time.

[0085] A4: By synchronizing multiple sensors spatially and temporally, we obtain the detection results of multiple sensors at the same time under a unified world coordinate system. The data from the cameras, LiDAR sensors, and ultrasonic sensors are preprocessed separately at their respective intermediate stations. After preprocessing at these intermediate stations, the data is initially fused using the least squares method to obtain the optimal fused data from each sensor. For visual sensor information, PCL point cloud pass-through filtering is used to filter point cloud information within a defined area, deleting data that is not considered or has low support to reduce computational load.

[0086] A5: After the information from each sensor is pre-processed at the intermediate station, since the processing of the data differs and the data are relatively independent, it is necessary to link and associate the data from multiple sensors with respect to the same target.

[0087] A6: Then, using the DS evidence theory algorithm, the multi-sensor decision-level fusion algorithm transmits the locally optimal fusion data obtained from the three sensors to the fusion center for global fusion of multi-sensor information to obtain the final fusion result. The robot can then perceive obstacles around it and make correct judgments and decisions.

[0088] Specifically, the DS evidence theory includes:

[0089] (1) Identification Framework

[0090] In the DS evidence theory, the identification frame is a set containing all possible outcomes of a given problem. The identification frame is represented by a set u, where all elements are mutually exclusive and no two elements can be simultaneously represented. This set is also called the hypothesis space, and it is the complete set u consisting of these mutually exclusive events.

[0091] Its form can be expressed as:

[0092] u={θ1, θ2, θ3,…,θ n}

[0093] Where θ i (i = 1, 2, 3, ..., n) are elements in the set u. All possible combinations of elements in u are called the power set of the recognition frame u, mathematically expressed as 2. u The power set contains all elements of u and their multiple subsets.

[0094] (2) Basic Probability Assignment (BPA) function

[0095] Assign a number between 0 and 1 to all subsets θ composed of elements within the recognition framework u. That is, let m(θ) be the number that maps the subset θ to the interval [0, 1]. This mapping relationship is called the mass function. The mass (basic probability assignment function) function satisfies the following conditions:

[0096]

[0097] From the above equation, we can see that the value of the basic probability assignment function m(θ) represents the subset θ in the power set 2. u The probability of the set is such that the sum of the values ​​m(θ) of all the fundamental probability assignment functions is 1, where the probability of the empty set is 0. In the power set 2... u If the probability value m(θ) of an element is greater than 0, then that element is called a focal element. The value of m(θ) reflects the overall trust in that subset θ; the larger the value, the higher the credibility of the focal element, and the smaller the value, the lower the credibility of the focal element. Therefore, how to assign the basic probability values ​​of focal elements is a difficult point in DS evidence theory, and the reliability of the assignment will largely determine the reliability of the final result of the algorithm.

[0098] (3) Belief Function

[0099] Suppose u is a recognition frame, and m(θ) satisfies mapping 2 u → Assign values ​​to the basic probabilities on u in [0, 1], and define the function Bel: 2 u →[0,1] is:

[0100] Bel(A) = ∑ B∈A m(B)

[0101] The above formula is the mathematical expression for the trust function, representing the sum of the BPAs of all subsets of an event A, where:

[0102]

[0103] The degree of support for a piece of evidence can be represented by the Bel function, while the degree of non-opposition to a piece of evidence is described by the likelihood function.

[0104] (4) Likelihood function

[0105] In the recognition framework u, the basic probability function value m satisfies the mapping: 2 u →[0, 1], define function Pl: 2 u →[0,1] is

[0106]

[0107] Where Pl(A) represents the likelihood function; express The level of trust.

[0108] If multiple sources of evidence exist for a given proposition, it is necessary to fuse the probabilities of these sources. The DS evidence theory combination rule is as follows:

[0109] The number of pieces of evidence is n, and the identification framework is u, m1, m2, m3, ..., m n Let each piece of evidence represent its basic probability assignment function. Then, the expression for the combination rule is:

[0110]

[0111] Where K is the conflict coefficient.

[0112] The conflict coefficient K describes the inconsistency in the support of the proposition among various pieces of evidence. The larger the coefficient K is, the more inconsistent the degree of support of the proposition by the evidence. When K=1, it indicates that the degree of support of the proposition by the various pieces of evidence is completely opposite, the evidence is completely conflicting, and the combination rule fails.

[0113] The evidence information fusion process of the DS evidence theory can be represented as Figure 2 As shown.

[0114] B: Perception of visual obstacles, such as Figure 3 and Figure 4 As shown, the process includes the following:

[0115] B1: Initiate obstacle detection and acquire point cloud data of the mobile robot in five directions (up, down, left, right, and front) through the visual sensor;

[0116] B2: Use PCL pass-through filtering to downsample the point cloud data within the given value range set in five directions. By filtering the point cloud, the amount of computation can be reduced and the influence of noise on obstacle perception can be removed. Only the required point cloud data needs to be processed.

[0117] B3: Then, using the ORB feature extraction method, extract the FAST feature points in the left front and right front of the robot. Transform the extracted FAST feature points into the robot coordinate system. If the obstacle feature point is not in the robot's obstacle avoidance area, it is not processed. If the obstacle feature point is in the robot's obstacle avoidance area, it is compared with the set FAST obstacle feature point threshold to determine which side of the robot has an obstacle or which side has a larger obstacle target.

[0118] B4: Obstacle detection using a visual sensor can yield the maximum point cloud pose (x) in all directions. max y max θmax ) and minimum point cloud pose (x min y min θ min The maximum point cloud pose (x) is obtained. max y max θ max ) and minimum point cloud pose (x min y min θ min This reflects the boundary data information of the obstacle;

[0119] B5: Analyze whether the robot triggers obstacle avoidance and in which direction it does so based on the above data: When the number of feature points of the obstacle on the left front side is greater than a given threshold, calculate the midpoint of the obstacle's coordinates using the obstacle boundary information obtained in B4, i.e.:

[0120] X = (x max +x min )*0.5, Y=(y max +y min )*0.5;

[0121] (x left y left The coordinates of the center of the obstacle on the left front side are the coordinates; otherwise, obstacle avoidance or bypass will not be triggered; the same applies to the right side.

[0122] B6: Based on B5, when the number of feature points of the obstacle on the left front side or the obstacle on the right front side exceeds a set threshold, the coordinates (x, y, y) of the midpoint of the obstacle on the left front side are calculated using B5. left y left ) and the coordinates (x) of the midpoint of the obstacle on the right front side. right y right By comparing the distance y between the obstacle on the left front side and the robot, left The distance y between the robot and the obstacle on the right front side right *1.1, determines which area's obstacle avoidance or obstacle bypass is triggered (in y). left and y right *1.1 corresponds to the area where the minimum value is used for obstacle avoidance or obstacle bypass;

[0123] B6.1: When the number of point cloud feature points of the obstacle on the left front side is greater than the set threshold, and when the number of point cloud feature points of the obstacle on the left front side is not greater than the set threshold, obstacle avoidance control in the left front obstacle avoidance area is executed; otherwise, obstacle avoidance or obstacle bypass is not triggered.

[0124] B6.2: When the number of point cloud feature points of the obstacle on the right front side is greater than the set threshold, and the number of point cloud feature points of the obstacle on the left front side is not greater than the set threshold, obstacle avoidance control in the right front obstacle avoidance area is executed; otherwise, obstacle avoidance or obstacle bypass is not triggered.

[0125] B7: When the obstacle on the left front side is at its closest distance x from the robot's front and rear sides. leftmin The closest distance x between the obstacle and the robot in the first-level obstacle avoidance zone or the obstacle on the right front side and the robot's front and rear sides. rightmin If the obstacle is smaller than the first-level obstacle avoidance area, then compare the size of the obstacle on the left front side with that on the right front side (perform obstacle avoidance or obstacle bypass based on the area corresponding to the smaller obstacle).

[0126] C: LiDAR sensor for obstacle detection, such as Figure 5 , Figure 6 and Figure 7 As shown

[0127] C1: Obtain the coordinate pose information of obstacles around the robot using a LiDAR sensor. Record the LiDAR pose as (x, y), and the pose of the obstacle furthest from the robot on the left as (x, y). leftmax y leftmax The pose of the nearest obstacle to the robot on the left is (x). leftmin y leftmin The pose of the obstacle furthest from the robot on the right is (x). rightmax y rightmax The pose of the obstacle on the right closest to the robot is (x). rightmin y rightmin );

[0128] C2: Divide the area where the robot needs to perceive obstacles into four directions: left-left (i.e., the first obstacle avoidance area), right-right (i.e., the second obstacle avoidance area), left-front (i.e., the third obstacle avoidance area), and right-front (i.e., the fourth obstacle avoidance area). Determine the position of the obstacle. Then, compare the distance between the obstacle and the robot with the distance the robot needs to avoid the obstacle. If the obstacle is within the robot's obstacle avoidance or obstacle avoidance level, convert the pose to the world coordinate system and accumulate the obstacle count.

[0129] C3: If the detected feature points fall in the same pose for 3 consecutive frames, it is considered that a valid obstacle has been detected; if the obstacle is triggered, it is determined whether there is more space on the left and right sides for the robot to pass through, so as to avoid errors in judging only the obstacle in front; if the same feature points are not detected for 3 consecutive frames, it is considered that there is no obstacle in that direction.

[0130] More specifically, including:

[0131] Define a radar obstacle avoidance parameter `radar_obs_x`. When the distance between the obstacle and the robot is greater than `radar_obs_x` and the distance between the obstacle and the robot is within the obstacle avoidance range in the x-direction, and the distance between the obstacle and the robot is within the obstacle avoidance range in the y-direction, it means that the obstacle is on the robot's left or right front side. Since the robot's pose information follows the right-hand rule, with the left side being the positive y-direction and the right side being the negative y-direction, when the obstacle coordinate y < 0 and is within the left front obstacle avoidance area, it means that the obstacle is on the robot's left front side, and the left front obstacle trigger accumulation is incremented by 1. Similarly, when the obstacle coordinate y > 0 and is within the right front obstacle avoidance area, it means that the obstacle is on the robot's right front side, and the right front obstacle trigger accumulation is incremented by 1.

[0132] When the distance between the obstacle and the robot is greater than the radar obstacle avoidance parameter radar_obs_x and the distance between the obstacle and the robot is within the obstacle avoidance range in the x-direction, and the y-value of the obstacle's pose relative to the robot is within the left-left region, the robot is triggered to avoid obstacles on the left-left side, and the cumulative count of obstacle avoidance on the left-left side is incremented by 1; when the y-value of the obstacle's pose relative to the robot is within the right-right region, the robot is triggered to avoid obstacles on the right-right side, and the cumulative count of obstacle avoidance on the right-right side is incremented by 1.

[0133] If the detected feature points fall in the same pose for three consecutive frames, a valid obstacle is considered to have been detected. If an obstacle is triggered, it is determined whether there is more space on the left and right sides for the robot to pass through, avoiding errors in judging only the obstacle in front (the laser data will show which direction the obstacle avoidance area can allow the robot to pass normally. The robot's own length and width are written in the configuration, and the laser data can be used to determine which side of the space has the conditions for the robot to successfully avoid the obstacle). If the same feature point is not detected for three consecutive frames, it is considered that there is no obstacle in that direction.

[0134] The robot's obstacle avoidance and obstacle avoidance are divided into deceleration zones and braking zones. When the robot is in the level 3 obstacle avoidance zone, it will not decelerate but will trigger a level 3 alarm. When the robot is in the level 2 obstacle avoidance zone, it will start to decelerate and trigger a level 2 alarm. When it is in the level 1 obstacle avoidance zone, it will trigger obstacle stopping and trigger a level 1 alarm. When the road conditions ahead of the robot do not meet the conditions for obstacle avoidance, the robot will determine which obstacle avoidance level it is in based on the distance between the obstacle and the robot, so that the robot can avoid colliding with the obstacle.

[0135] By dividing the area in front of the robot that needs to be detected into zones, obstacle information obtained by the LiDAR sensor is used to perceive obstacles in four directions: left-left, left-front, right-front, and right-right. Based on the relationship between the obstacle and the detection area, the robot performs operations such as speed reduction control and turning around obstacles. Based on the relative distance of the obstacle obtained by the LiDAR data, the robot can achieve the purpose of accurately bypassing or avoiding obstacles.

[0136] D: Ultrasonic sensor for obstacle detection, such as Figure 8 As shown

[0137] D1: Since visual sensors can only see obstacles in front of the robot, while lidar sensors are based on optical detection, lidar can pass through transparent glass and is not good at detecting objects with low reflectivity, which will cause a certain degree of missed obstacle detection during the robot's movement.

[0138] D2: In some actual robot working environments, there may be glass doors, glass walls, dark-colored objects, insufficient light sources, etc. in the environment where the robot is performing tasks. In this case, it is very necessary to install ultrasonic sensors. This can prevent the robot from mistakenly thinking that there are no obstacles in front of it due to the failure of the lidar to detect transparent glass, which may cause the robot to crash into the glass door or glass wall.

[0139] D3: Obstacle detection is achieved by installing four sets of eight ultrasonic sensors on the front, back, left, and right sides of the mobile robot chassis. The ultrasonic sensors can measure the distance between themselves and obstacles and can detect obstacles such as transparent glass and low reflectivity objects that cannot be detected by lidar.

[0140] Example 2:

[0141] like Figure 9 As shown, Embodiment 2 of the present invention provides an obstacle avoidance and bypass system based on multi-sensor fusion for obstacle perception, comprising:

[0142] The visual data fusion module is configured to perform preliminary fusion of the acquired mobile robot visual data to obtain preliminary visual data fusion results.

[0143] The lidar data fusion module is configured to: perform preliminary fusion of the acquired lidar data of the mobile robot to obtain preliminary lidar data fusion results;

[0144] The ultrasonic data fusion module is configured to perform preliminary fusion of the acquired ultrasonic data from the mobile robot to obtain preliminary ultrasonic data fusion results.

[0145] The multi-sensor data fusion module is configured to: perform global fusion of multi-sensor data based on the DS evidence theory multi-sensor fusion algorithm, and perform preliminary fusion of visual data, LiDAR data, and ultrasonic data to obtain the final obstacle information around the robot;

[0146] The obstacle avoidance control module is configured to control the robot to avoid obstacles based on the obtained obstacle information.

[0147] The working method of the system is the same as the obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception provided in Embodiment 1, and will not be repeated here.

[0148] Example 3:

[0149] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it implements the steps in the obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception as described in Embodiment 1 of the present invention.

[0150] Example 4:

[0151] Embodiment 4 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception as described in Embodiment 1 of the present invention.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception, characterized in that, Includes the following processes: The acquired visual data of the mobile robot is initially fused to obtain the preliminary visual data fusion result; The acquired LiDAR data of the mobile robot is initially fused to obtain the preliminary fusion result of the LiDAR data; The acquired ultrasonic data from the mobile robot were initially fused to obtain preliminary fusion results. Based on the DS evidence theory, a multi-sensor fusion algorithm is used to perform global fusion of visual data, lidar data, and ultrasonic data to obtain the final obstacle information around the robot. The robot performs obstacle avoidance control based on the obtained obstacle information; The first set range in front of the robot is set as the first obstacle avoidance zone, the second set range to the left front of the first obstacle avoidance zone is set as the first obstacle bypass zone, the third set range to the right front of the first obstacle avoidance zone is set as the second obstacle bypass zone, the left side of the first obstacle bypass zone is set as the third obstacle bypass zone, and the right side of the second obstacle bypass zone is set as the fourth obstacle bypass zone. The first obstacle avoidance area, the second obstacle avoidance area, the third obstacle avoidance area, and the fourth obstacle avoidance area are all located in front of the first-level obstacle avoidance area. The first obstacle avoidance area and the second obstacle avoidance area are arranged symmetrically along the extension of the center line of the first-level obstacle avoidance area. The third obstacle avoidance area and the fourth obstacle avoidance area are arranged symmetrically along the extension of the center line of the first-level obstacle avoidance area. The center line of the first-level obstacle avoidance area is collinear with the center line of the robot's forward direction. Set a radar obstacle avoidance distance threshold, with the robot's forward direction as the positive X direction and the left side of the X direction as the positive Y direction; When the distance between the obstacle and the robot is greater than the radar obstacle avoidance distance threshold, and the distance between the obstacle and the robot is within the obstacle avoidance range in the X direction; If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the first obstacle avoidance area, the obstacle triggering cumulative number in the first obstacle avoidance area is incremented by 1. If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the second obstacle avoidance area, then the obstacle is in the robot's second obstacle avoidance area, and the obstacle triggering cumulative increment in the second obstacle avoidance area is 1. If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the third obstacle avoidance area, the obstacle triggering cumulatively in the third obstacle avoidance area is incremented by 1. If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the fourth obstacle avoidance area, then the obstacle is in the robot's fourth obstacle avoidance area, and the obstacle triggering cumulative increment in the fourth obstacle avoidance area is 1. If the cumulative number of obstacle triggers in the obstacle avoidance area is N after N consecutive frames of detection, then a valid obstacle is considered to have been detected, and obstacle avoidance control is triggered, where N is an integer greater than or equal to 3; otherwise, no valid obstacle is considered to have been detected, and obstacle avoidance control is not triggered. The first level obstacle avoidance zone leads to the second level obstacle avoidance zone, and the second level obstacle avoidance zone leads to the third level obstacle avoidance zone. The first obstacle avoidance area, the second obstacle avoidance area, the third obstacle avoidance area, and the fourth obstacle avoidance area are all divided into secondary obstacle avoidance areas and tertiary obstacle avoidance areas along the robot's direction of movement; When the road conditions ahead of the robot do not meet the conditions for obstacle avoidance, obstacle avoidance control is implemented, including: When the robot is in the level 3 obstacle avoidance zone, it does not slow down but triggers an alarm. When the robot is in the level 2 obstacle avoidance zone, it begins to slow down and triggers an alarm. When the robot is in the level 1 obstacle avoidance zone, it stops moving forward and triggers an alarm.

2. An obstacle avoidance and bypass system based on multi-sensor fusion for obstacle perception, characterized in that, include: The visual data fusion module is configured to perform preliminary fusion of the acquired mobile robot visual data to obtain preliminary visual data fusion results. The lidar data fusion module is configured to: perform preliminary fusion of the acquired lidar data of the mobile robot to obtain preliminary lidar data fusion results; The ultrasonic data fusion module is configured to perform preliminary fusion of the acquired ultrasonic data from the mobile robot to obtain preliminary ultrasonic data fusion results. The multi-sensor data fusion module is configured to: perform global fusion of multi-sensor data based on the DS evidence theory multi-sensor fusion algorithm, and perform preliminary fusion of visual data, LiDAR data, and ultrasonic data to obtain the final obstacle information around the robot; The obstacle avoidance control module is configured to control the robot to avoid obstacles based on the obtained obstacle information. The first set range in front of the robot is set as the first obstacle avoidance zone, the second set range to the left front of the first obstacle avoidance zone is set as the first obstacle bypass zone, the third set range to the right front of the first obstacle avoidance zone is set as the second obstacle bypass zone, the left side of the first obstacle bypass zone is set as the third obstacle bypass zone, and the right side of the second obstacle bypass zone is set as the fourth obstacle bypass zone. The first obstacle avoidance area, the second obstacle avoidance area, the third obstacle avoidance area, and the fourth obstacle avoidance area are all located in front of the first-level obstacle avoidance area. The first obstacle avoidance area and the second obstacle avoidance area are arranged symmetrically along the extension of the center line of the first-level obstacle avoidance area. The third obstacle avoidance area and the fourth obstacle avoidance area are arranged symmetrically along the extension of the center line of the first-level obstacle avoidance area. The center line of the first-level obstacle avoidance area is collinear with the center line of the robot's forward direction. Set a radar obstacle avoidance distance threshold, with the robot's forward direction as the positive X direction and the left side of the X direction as the positive Y direction; When the distance between the obstacle and the robot is greater than the radar obstacle avoidance distance threshold, and the distance between the obstacle and the robot is within the obstacle avoidance range in the X direction; If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the first obstacle avoidance area, the obstacle triggering cumulative number in the first obstacle avoidance area is incremented by 1. If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the second obstacle avoidance area, then the obstacle is in the robot's second obstacle avoidance area, and the obstacle triggering cumulative increment in the second obstacle avoidance area is 1. If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the third obstacle avoidance area, the obstacle triggering cumulatively in the third obstacle avoidance area is incremented by 1. If the distance between the obstacle and the robot is within the obstacle avoidance range in the Y direction and the obstacle's Y-direction coordinate is located in the fourth obstacle avoidance area, then the obstacle is in the robot's fourth obstacle avoidance area, and the obstacle triggering cumulative increment in the fourth obstacle avoidance area is 1. If the cumulative number of obstacle triggers in the obstacle avoidance area is N after N consecutive frames of detection, then a valid obstacle is considered to have been detected, and obstacle avoidance control is triggered, where N is an integer greater than or equal to 3; otherwise, no valid obstacle is considered to have been detected, and obstacle avoidance control is not triggered. The first level obstacle avoidance zone leads to the second level obstacle avoidance zone, and the second level obstacle avoidance zone leads to the third level obstacle avoidance zone. The first obstacle avoidance area, the second obstacle avoidance area, the third obstacle avoidance area, and the fourth obstacle avoidance area are all divided into secondary obstacle avoidance areas and tertiary obstacle avoidance areas along the robot's direction of movement; When the road conditions ahead of the robot do not meet the conditions for obstacle avoidance, obstacle avoidance control is implemented, including: When the robot is in the level 3 obstacle avoidance zone, it does not slow down but triggers an alarm. When the robot is in the level 2 obstacle avoidance zone, it begins to slow down and triggers an alarm. When the robot is in the level 1 obstacle avoidance zone, it stops moving forward and triggers an alarm.

3. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception as described in claim 1.

4. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the obstacle avoidance and bypass method based on multi-sensor fusion for obstacle perception as described in claim 1.

Images