Obstacle avoidance method based on obstacle recognition, terminal and storage medium

Abstract

The invention discloses an obstacle avoidance method based on obstacle recognition, a terminal and a storage medium, and the method comprises the steps: obtaining camera data and laser radar data, carrying out the position calibration and type recognition of an obstacle according to the camera data and the laser radar data, and obtaining the type of the obstacle and the relative coordinates of the obstacle relative to a vehicle; vehicle GPS coordinates at the same moment are obtained, and obstacle GPS coordinates are calculated through a rotation matrix based on the relative coordinates; judging whether obstacle avoidance is needed or not according to the type of the obstacle, and if the obstacle avoidance is needed, generating an obstacle avoidance route according to the GPS coordinates of the obstacle; the camera and the laser radar are used for carrying out calibration and type recognition on the obstacle, obstacle detection is more comprehensive, the type of the obstacle can be recognized, whether obstacle avoidance is needed or not is judged, better obstacle recognition is achieved, unmanned driving safety is improved, and unnecessary obstacle avoidance can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of unmanned tracking driving technology, and in particular to an obstacle avoidance method, terminal and storage medium based on obstacle recognition. Background Art

[0002] Unmanned tracking driving is a very common way for unmanned vehicles to drive autonomously. Tracking driving refers to the automatic tracking driving of the car according to a pre-set route. When there are obstacles on the unmanned driving path and the vehicle cannot actively avoid the obstacles, accidents such as collisions and rollovers will occur, which will not only cause economic losses, but also cause mission failures, limiting the scope of use and service life of unmanned vehicles. At present, the mainstream obstacle recognition equipment includes millimeter wave radar, ultrasonic radar, etc., but these devices are all restricted in use. How to better identify obstacles and thus improve the safety of unmanned driving is a technical problem that needs to be solved at present. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an obstacle avoidance method, terminal and storage medium based on obstacle recognition, which can better identify and distinguish obstacles and improve the safety and efficiency of unmanned driving.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: An obstacle avoidance method based on obstacle recognition comprises the following steps: S1. Obtain camera data and laser radar data, and perform position calibration and type identification of obstacles according to the camera data and laser radar data to obtain the type of the obstacle and the relative coordinates of the obstacle relative to the vehicle; S2. Obtain the GPS coordinates of the vehicle at the same time, and calculate the GPS coordinates of the obstacle through the rotation matrix based on the relative coordinates; S3. Determine whether obstacle avoidance is required based on the obstacle type. If obstacle avoidance is required, generate an obstacle avoidance route based on the GPS coordinates of the obstacle.

[0005] In order to solve the above technical problems, another technical solution adopted by the present invention is: An obstacle avoidance terminal based on obstacle recognition includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above obstacle avoidance method based on obstacle recognition are implemented.

[0006] In order to solve the above technical problems, another technical solution adopted by the present invention is: A storage medium stores a computer program, which implements the steps of the above obstacle avoidance method based on obstacle recognition when executed.

[0007] The beneficial effects of the present invention are as follows: the obstacle avoidance method, terminal and storage medium based on obstacle recognition of the present invention calibrate and identify the type of obstacles through cameras and laser radars, so that obstacle detection is more comprehensive and the type of obstacles can be identified, thereby determining whether obstacle avoidance is required, achieving better obstacle recognition, improving the safety of unmanned driving, and avoiding unnecessary obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of an obstacle avoidance method based on obstacle recognition according to an embodiment of the present invention; Figure 2 This is a structural diagram of an obstacle avoidance terminal based on obstacle recognition according to an embodiment of the present invention; Figure 3 A schematic diagram of a camera calibration principle of an obstacle avoidance method based on obstacle recognition according to an embodiment of the present invention; Figure 4 A schematic diagram of generating an obstacle avoidance route for an obstacle avoidance method based on obstacle recognition according to an embodiment of the present invention; Description of labels: 1. An obstacle avoidance terminal based on obstacle recognition; 2. A processor; 3. A memory. DETAILED DESCRIPTION

[0009] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0010] Please refer to Figure 1 , an obstacle avoidance method based on obstacle recognition, comprising the steps of: S1. Obtain camera data and laser radar data, and perform position calibration and type identification of obstacles according to the camera data and laser radar data to obtain the type of the obstacle and the relative coordinates of the obstacle relative to the vehicle; S2. Obtain the GPS coordinates of the vehicle at the same time, and calculate the GPS coordinates of the obstacle through the rotation matrix based on the relative coordinates; S3. Determine whether obstacle avoidance is required based on the obstacle type. If obstacle avoidance is required, generate an obstacle avoidance route based on the GPS coordinates of the obstacle.

[0011] From the above description, it can be seen that the beneficial effects of the present invention are: an obstacle avoidance method, terminal and storage medium based on obstacle recognition of the present invention calibrates and identifies the type of obstacles through cameras and lidars, and the obstacle detection is more comprehensive, and the type of obstacles can be identified, and then it is determined whether obstacle avoidance is needed, thereby achieving better obstacle recognition, improving the safety of unmanned driving, and avoiding unnecessary obstacle avoidance.

[0012] Furthermore, the step S2 and the step S3 further include the following steps: S21. Compare the relative coordinates of all the obstacles identified by the laser radar and the camera, classify the obstacle coordinates whose deviations are within a preset range as the same obstacle, and use the relative coordinates identified by the laser radar as the relative coordinates of the obstacle relative to the vehicle.

[0013] From the above description, it can be seen that the obstacles identified by the lidar and the camera are clustered to avoid repeated consideration of the same obstacle in the future.

[0014] Furthermore, the principle of calibrating the position of obstacles based on the camera data is as follows: A point P=(X,Y,Z) in three-dimensional space T The projection point p=(x,y) on the image T , is the intersection of the line connecting the optical center and point P and the image plane, P^=(X,Y,Z,t) T and p^=(x,y,t) T Represent the secondary coordinates of points P and p respectively. In homogeneous coordinates, the relationship between a three-dimensional point P and its projection point p can be expressed as follows: ; Where k is any non-zero scale factor, [R|t] represents the external parameters of the camera, R is the rotation matrix, t is the translation vector, K represents the camera internal parameter matrix, (U 0, V0) is the principal point coordinate, f u and f v They represent the scale factors in the X-axis and Y-axis directions respectively, and S represents the calibration of the distortion of the image coordinate system and the world coordinate system.

[0015] It can be seen from the above description that, through the above method, the camera is used to achieve the calibration of obstacles.

[0016] Furthermore, whether obstacle avoidance is required is determined as follows: Determine whether the type of the obstacle is in a preset avoidance type list, if so, obstacle avoidance is required, otherwise, obstacle avoidance is not required; It is determined whether any edge point of the obstacle is less than a preset safety distance from the tracking route of the vehicle. If so, obstacle avoidance is required; otherwise, obstacle avoidance is not required.

[0017] From the above description, it can be seen that whether obstacle avoidance is required is determined based on the distance between the obstacle edge point and the vehicle tracking route and the type of obstacle.

[0018] Furthermore, the step S2 is specifically as follows: Get the current GPS data, and the relative coordinates (x, y) of the obstacle calibrated by the laser radar and the camera, according to the rotation matrix: x′ = xcosθ − ysinθ; y′= xsinθ+ycosθ; The first coordinate (x', y') can be obtained, where the units of x, y, x' and y' are all meters, and according to 1m = 0.00054054054' and the initial GPS information (lon, lat), the longitude and latitude coordinates of the obstacle can be obtained: lon_x= lon+ x'×0.00054054054′; lat_y =lat+ y'×0.00054054054′; Among them, lat represents the longitude of the vehicle's current position, lon represents the latitude of the vehicle's current position, and θ represents the heading angle.

[0019] It can be seen from the above description that obstacle coordinate transformation is performed through the above steps.

[0020] Furthermore, the step S1 further includes the following steps: Obtaining the longitude and latitude data of each tracking point of the tracking route collected in advance; According to the current GPS data of the vehicle, the next tracking point is selected, and the direction of the straight line between the two points in the GPS coordinate system is calculated using trigonometric functions. The difference is made with the current heading angle of the vehicle to obtain the angle at which the front of the vehicle needs to turn, thereby controlling the steering and driving of the vehicle.

[0021] From the above description, it can be seen that the tracking route is collected in advance, and the vehicle calculates the steering angle to control steering and driving based on the current position and the coordinates of the next tracking point to achieve unmanned tracking driving.

[0022] Please refer to Figure 2 An obstacle avoidance terminal based on obstacle recognition includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above obstacle avoidance method based on obstacle recognition are implemented.

[0023] A storage medium stores a computer program, which implements the steps of the above obstacle avoidance method based on obstacle recognition when executed.

[0024] The present invention discloses an obstacle avoidance method, terminal and storage medium based on obstacle recognition, which are suitable for obstacle avoidance driving during unmanned tracking driving.

[0025] Please refer to Figure 1 , Figure 3 and Figure 4 , Embodiment 1 of the present invention is: An obstacle avoidance method based on obstacle recognition comprises the following steps: Obtaining the longitude and latitude data of each tracking point of the tracking route collected in advance; According to the current GPS data of the vehicle, the next tracking point is selected, and the direction of the straight line between the two points in the GPS coordinate system is calculated using trigonometric functions. The difference is made with the current heading angle of the vehicle to obtain the angle at which the front of the vehicle needs to turn, thereby controlling the steering and driving of the vehicle.

[0026] S1. Obtain camera data and laser radar data, and perform position calibration and type identification of obstacles according to the camera data and laser radar data to obtain the type of the obstacle and the relative coordinates of the obstacle relative to the vehicle; In this embodiment, the principle of calibrating the position of the obstacle based on the camera data is: A point P=(X,Y,Z) in three-dimensional space T The projection point p=(x,y) on the image T , is the intersection of the line connecting the optical center and point P and the image plane, P^=(X,Y,Z,t) T and p^=(x,y,t) T Represent the secondary coordinates of points P and p respectively. In homogeneous coordinates, the relationship between a three-dimensional point P and its projection point p can be expressed as follows: ; Where k is any non-zero scale factor, [R|t] represents the external parameters of the camera, R is the rotation matrix, t is the translation vector, K represents the camera internal parameter matrix, (U 0, V0) is the principal point coordinate, f u and f v They represent the scale factors in the X-axis and Y-axis directions respectively, and S represents the calibration of the distortion of the image coordinate system and the world coordinate system.

[0027] In other equivalent embodiments, a binocular camera may be used to calibrate obstacles to improve accuracy.

[0028] S2. Obtain the GPS coordinates of the vehicle at the same moment, and calculate the GPS coordinates of the obstacle through the rotation matrix based on the relative coordinates.

[0029] The step S2 is specifically as follows: Get the current GPS data, and the relative coordinates (x, y) of the obstacle calibrated by the laser radar and the camera, according to the rotation matrix: x′ = xcosθ − ysinθ; y′= xsinθ+ycosθ; The first coordinate (x', y') can be obtained, where the units of x, y, x' and y' are all meters, and according to 1m = 0.00054054054' and the initial GPS information (lon, lat), the longitude and latitude coordinates of the obstacle can be obtained: lon_x= lon+ x'×0.00054054054′; lat_y =lat+ y'×0.00054054054′; Among them, lat represents the longitude of the vehicle's current position, lon represents the latitude of the vehicle's current position, and θ represents the heading angle.

[0030] S21. Compare the relative coordinates of all the obstacles identified by the laser radar and the camera, classify the obstacle coordinates whose deviations are within a preset range as the same obstacle, and use the relative coordinates identified by the laser radar as the relative coordinates of the obstacle relative to the vehicle.

[0031] In this embodiment, the longitude and latitude information of the two sets of data are compared respectively, and the data with consistency (the error is less than a preset threshold) are classified as the same object.

[0032] S3. Determine whether obstacle avoidance is required based on the obstacle type. If obstacle avoidance is required, generate an obstacle avoidance route based on the GPS coordinates of the obstacle.

[0033] Whether obstacle avoidance is needed is determined as follows: Determine whether the type of the obstacle is in a preset avoidance type list, if so, obstacle avoidance is required, otherwise, obstacle avoidance is not required; It is determined whether any edge point of the obstacle is less than a preset safety distance from the tracking route of the vehicle. If so, obstacle avoidance is required; otherwise, obstacle avoidance is not required.

[0034] In this embodiment, the type of obstacle is judged. If the obstacle type is not in the preset avoidance list, such as flowers and plants, it is ignored and no obstacle avoidance is required. For the remaining obstacles, it is judged whether the object is on the tracking track or in the danger range of the tracking track (less than the safety distance) based on its distance from the tracking track. In this embodiment, the safety distance is half the width of the vehicle. If the width of the vehicle is 4m, the tracking track is within 2m to the left and right and belongs to the danger range. If there is no obstacle in the danger range, the unmanned vehicle continues to perform the tracking task, otherwise it enters the obstacle avoidance step and generates an obstacle avoidance route according to the obstacle avoidance algorithm.

[0035] like Figure 4 As shown, in this embodiment, the obstacle avoidance algorithm uses the equation of a circle: (xa)×(xa)+(yb)(yb)=r×r.

[0036] The curve represented by the equation of a circle is a circle with O(a, b) as its center and r as its radius.

[0037] In this equation, r is the turning radius of the car, so we can determine a=r, b=0. Substituting equally spaced x values ​​into the formula yields a sequence of (x, y) points. This sequence of points is the obstacle avoidance trajectory points (A->B), while trajectories B->C and C->D are both straight line segments.

[0038] Follow the tracking trajectory generated by the above steps to drive around the obstacle.

[0039] In addition, in this embodiment, the identification of obstacle types is specifically as follows: For obstacles not recognized by the camera, the obstacle type is obtained by inputting the point cloud data of the obstacle collected by the laser radar into a pre-trained first obstacle recognition model; For obstacles not identified by the laser radar, the image of the obstacle captured by the camera is input into a pre-trained second obstacle recognition model to obtain the type of the obstacle; For an obstacle recognized by both the laser radar and the camera, the point cloud data of the obstacle and the image of the obstacle are input into a pre-trained third obstacle recognition model to obtain the obstacle type.

[0040] In this embodiment, three obstacle recognition models are preset. Among them, the training data of the first obstacle recognition model focuses on obstacles that are difficult for some cameras to capture, such as transparent glass.

[0041] Similarly, the laser radar may miss detection due to unexpected factors. Therefore, the camera is used to supplement the data missed by the laser radar, and the obstacle type is identified through the second obstacle recognition model.

[0042] For obstacles detected by both the lidar and the camera, comprehensive identification is performed using the data from both. The third obstacle model is trained for this purpose and has higher recognition accuracy.

[0043] In other equivalent embodiments, considering that laser radar still has a certain possibility of missing detection of transparent objects, the laser radar can be replaced with ultrasonic radar, etc. in specific use, and the recognition model can be trained specifically. Ultrasonic radar is not based on optical principles and can effectively complement the camera to ensure the accuracy of recognition.

[0044] Please refer to Figure 2 , Embodiment 2 of the present invention is: An obstacle avoidance terminal 1 based on obstacle recognition includes a processor 2, a memory 3, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps of the obstacle avoidance method based on obstacle recognition in the above embodiment 1 are implemented.

[0045] Embodiment 3 of the present invention is: A storage medium stores a computer program thereon, and when the computer program is executed, the steps of the obstacle avoidance method based on obstacle recognition in the above embodiment 1 are implemented.

[0046] In summary, the obstacle avoidance method, terminal and storage medium based on obstacle recognition provided by the present invention calibrate and identify the type of obstacles through cameras and lidars. The obstacle detection is more comprehensive and the type of obstacle can be identified to determine whether obstacle avoidance is needed, thereby achieving better obstacle recognition, improving the safety of unmanned driving, and avoiding unnecessary obstacle avoidance.

[0047] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An obstacle avoidance method based on obstacle recognition, characterized in that: Includes steps: S1. Obtain camera data and laser radar data, and perform position calibration and type identification of obstacles according to the camera data and laser radar data to obtain the type of the obstacle and the relative coordinates of the obstacle relative to the vehicle; The obstacle type identification includes the following steps: For obstacles not recognized by the camera, the obstacle type is obtained by inputting the point cloud data of the obstacle collected by the laser radar into a pre-trained first obstacle recognition model; For obstacles not identified by the laser radar, the image of the obstacle captured by the camera is input into a pre-trained second obstacle recognition model to obtain the obstacle type; For an obstacle identified by both the laser radar and the camera, the point cloud data of the obstacle and the image of the obstacle are input into a pre-trained third obstacle recognition model to obtain the obstacle type; S2. Obtain the GPS coordinates of the vehicle at the same time, and calculate the GPS coordinates of the obstacle through the rotation matrix based on the relative coordinates; S3. Determine whether obstacle avoidance is required based on the obstacle type. If obstacle avoidance is required, generate an obstacle avoidance route based on the GPS coordinates of the obstacle.

2. The obstacle avoidance method based on obstacle recognition according to claim 1, characterized in that: The steps between step S1 and step S2 also include: S11. Compare the relative coordinates of all the obstacles identified by the laser radar and the camera, classify the obstacle coordinates whose deviations are within a preset range as the same obstacle, and use the relative coordinates identified by the laser radar as the relative coordinates of the obstacle relative to the vehicle.

3. The obstacle avoidance method based on obstacle recognition according to claim 2, characterized in that: The principle of calibrating the position of obstacles based on the camera data is as follows: A point P=(X,Y,Z) in three-dimensional space T The projection point p=(x,y) on the image T , is the intersection of the line connecting the optical center and point P and the image plane, P^=(X,Y,Z,t) T and p^=(x,y,t) T Represent the secondary coordinates of points P and p respectively. In homogeneous coordinates, the relationship between a three-dimensional point P and its projection point p can be expressed as follows: ； Where k is any non-zero scale factor, [R|t] represents the external parameters of the camera, R is the rotation matrix, t is the translation vector, K represents the camera internal parameter matrix, (U 0, V0) is the principal point coordinate, f u and f v They represent the scale factors in the X-axis and Y-axis directions respectively, and S represents the calibration of the distortion of the image coordinate system and the world coordinate system.

4. The obstacle avoidance method based on obstacle recognition according to claim 1, characterized in that: Whether obstacle avoidance is needed is determined as follows: Determine whether the type of the obstacle is in a preset avoidance type list, if so, obstacle avoidance is required, otherwise, obstacle avoidance is not required; It is determined whether any edge point of the obstacle is less than a preset safety distance from the tracking route of the vehicle. If so, obstacle avoidance is required; otherwise, obstacle avoidance is not required.

5. The obstacle avoidance method based on obstacle recognition according to claim 1, characterized in that: The step S2 is specifically as follows: Get the current GPS data, and the relative coordinates (x, y) of the obstacle calibrated by the laser radar and the camera according to the rotation matrix: x′ = xcosθ − ysinθ; y′= xsinθ+ycosθ; The first coordinate (x', y') can be obtained, where the units of x, y, x' and y' are all meters, and according to 1m = 0.00054054054' and the initial GPS information (lon, lat), the longitude and latitude coordinates of the obstacle can be obtained: lon_x= lon+ x'×0.00054054054′; lat_y =lat+ y'×0.00054054054′; Among them, lat represents the longitude of the vehicle's current position, lon represents the latitude of the vehicle's current position, and θ represents the heading angle.

6. The obstacle avoidance method based on obstacle recognition according to claim 1, characterized in that: The step S1 also includes the following steps: Obtaining the longitude and latitude data of each tracking point of the tracking route collected in advance; According to the current GPS data of the vehicle, the next tracking point is selected, and the direction of the straight line between the two points in the GPS coordinate system is calculated using trigonometric functions. The difference is made with the current heading angle of the vehicle to obtain the angle at which the front of the vehicle needs to turn, thereby controlling the steering and driving of the vehicle.

7. An obstacle avoidance terminal based on obstacle recognition, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of any one of the obstacle avoidance methods based on obstacle recognition according to claims 1-6 are implemented.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the steps in the obstacle avoidance method based on obstacle recognition according to any one of claims 1 to 6 are implemented.

Images