Occupied grid map generation method fusing fish eye passable space detection

Abstract

The invention discloses an occupation grid map generation method fused with fisheye passable space detection, and the method comprises the steps: fusing the fisheye passable space detection with laser radar passable space detection, and generating a more accurate and stable passable space occupation grid map; the fisheye image information is rich, rain, fog, dust and the like are not detected to be impassable, the grids updated by laser detection are reversely projected into the fisheye image, the occupation probability of the grids detected to be impassable by the fisheye is reduced, laser false detection is inhibited, and the influence that the dust, the rain, the fog and the like are falsely detected to be impassable by the laser is reduced; meanwhile, the horizontal and vertical field angles of view of the fisheye camera are 180 degrees, so that no blind area exists basically, the passable space condition in the laser blind area can be detected, the occupancy probability of grids which are located in the blind area and are detected to be impassable by the fisheye is maintained, and the safety of automatic driving is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent driving, and relates to the generation of occupancy grid maps in scenarios such as mining areas and bulk terminals. Specifically, it relates to a method for generating an occupancy grid map that integrates fisheye traversable space detection. Background Art

[0002] In the field of autonomous driving, the traversable space generally refers to the area around the vehicle where it can safely drive. During the vehicle's driving process, the traversable space information can be used to determine whether the vehicle is about to deviate from the safe area or collide with an obstacle. If it is detected that the vehicle is about to enter a non-traversable space, the system can promptly take safety measures such as braking or avoidance. At the same time, the traversable space provides constraint conditions for the path planning algorithm. By accurately perceiving the traversable space, the path planning module can generate a safer and more efficient driving path.

[0003] Due to the advantages of lidar such as directly obtaining three-dimensional information and being unaffected by light, the current generation of traversable space mainly relies on lidar detection. However, lidar also has problems such as being easily interfered by rain, fog, dust, etc. and resulting in false detections, and lacking information such as color texture and semantics. In addition, currently, in order to take into account long-distance target detection, lidars on autonomous driving vehicles are usually installed at a relatively high position. Due to the limited vertical field of view angle, there will be a certain lidar perception blind area near the vehicle body. If an obstacle enters this blind area, it cannot be detected by lidar, posing a greater collision risk. Summary of the Invention

[0004] Aiming at the above problems, the main purpose of the present invention is to design a method for generating an occupancy grid map that integrates fisheye traversable space detection, introducing a fisheye camera for traversable space detection to solve the problems of large lidar blind areas and false detections caused by being easily affected by rain, fog, and dust.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: A method for generating an occupancy grid map that integrates fisheye traversable space detection, which is used for a vehicle equipped with a lidar and a fisheye camera, includes the following steps: Establish a reference occupancy grid map in combination with lidar blind area information; Obtain lidar traversable space detection and fisheye traversable space detection as input data; Based on the timestamp of the current frame of lidar traversable space detection received and the reference occupancy grid map, establish the occupancy grid map at the current moment; According to the motion information of the vehicle itself, convert the occupancy grid map information at the previous moment to the occupancy grid map at the current moment; Subtract a prior value from the occupancy probabilities of all grids in the occupancy grid map at the current moment, so that the occupancy probabilities of the grids not updated by the lidar traversable space detection can be reduced; Project the lidar traversable space detection of the current frame into the occupancy grid map at the current moment, fuse the detection results of the fisheye traversable space, and update the grid occupancy probabilities; Through the fisheye traversable space detection, determine whether to maintain the grid occupancy probability for the grids in the lidar blind area that are not updated by the lidar traversable space detection; Moving objects are not expressed in the form of an occupancy grid map, and the occupancy probabilities of the grids within the target contour are cleared through the received target tracking results; Extract the indices of the grids in the occupancy grid map at the current moment whose occupancy probabilities are greater than the threshold, and output them for downstream use.

[0006] As a further description of the present invention, the lidar traversable space detection is the point cloud information of static or moving obstacles around the vehicle at the current moment; The lidar traversable space detection at the moment is , composed of a series of points, and each point contains position and type information ; where is the axis, axis and axis coordinates of the point in the ego-vehicle coordinate system. The ego-vehicle coordinate system takes the center of the ego-vehicle as the origin, the forward direction is the positive direction of the axis, the left direction is the positive direction of the axis, and the upward direction is the positive direction of the axis, represents the type information of the point.

[0007] As a further description of the present invention, the fisheye traversable space detection consists of a set of horizontally continuous pixel points on the image information; the pixel points represent the boundary of the traversable space on the image, the upper side of the boundary is the non-traversable area, and the lower side of the boundary is the traversable area; The detection of the fisheye traversable space boundary at the moment is expressed as: , , ; where is the height of the fisheye image, that is, the number of pixel points of the fisheye image in the longitudinal direction, is the width of the fisheye image, that is, the number of pixel points of the fisheye image in the horizontal direction; is the abscissa of the detection point on the fisheye image pixel coordinate system, continuously increasing from 1 to , that is, there are a total of traversable space boundary points on the fisheye image piece; Indexed by it is possible to query the ordinate of the passable boundary on the corresponding column, i.e., that is, , .

[0008] As a further description of the present invention, the occupied grid map includes a plurality of grids, each grid being square, and the physical size of the occupied grid map is , where is the side length of the grid, representing the resolution of the grid, is the number of rows of the grid map, is the number of columns of the grid map, is the total number of grids in the grid map; The starting point of the occupied grid map is the upper left corner of the grid map, i.e., the grid in the 0th row and 0th column , and the origin of the vehicle is located at the grid with the index of ; the grid in the th row and th column is represented as , where ; The attribute of each grid is , where is the position of the grid in the vehicle system, represented by , is the axis, axis coordinates of the grid position in the vehicle system, is the probability that the grid is impassable, i.e., the grid occupancy probability, is the flag bit indicating whether the grid is in the lidar blind area, indicates that the grid is in the lidar blind area, indicates that the grid is not in the lidar blind area, is whether the grid occupancy probability is updated by lidar points, is whether the grid occupancy probability is maintained.

[0009] As a further description of the present invention, the establishment of the reference occupied grid map includes the following steps: Initialize a row, column reference occupied grid map , traverse each grid, and initialize the grid attributes; for the in the th row and th column grid initialize the attributes: Grid Occupancy probability initialization, grid The occupancy probability is set to the initial value , that is, the grid Occupancy probability ; Grid The position in the ego vehicle system Is initialized to: ; ; Blind spot flag calculation, the blind spot is a fixed area around the vehicle, represented by a closed area If the grid The position in the ego vehicle system Is within the ego vehicle's blind spot , then the blind spot flag of the grid Is initialized to ; If the grid The position in the ego vehicle system Is not within the ego vehicle's blind spot , then the blind spot flag of the grid Is initialized to ; Grid Other flag initialization: ; ; Among them, Indicates whether the occupancy probability of the grid is updated by lidar points. The reference occupancy grid map defaults that the occupancy probability of all grids has not been updated by lidar points, and the default setting is , Indicates whether the occupancy probability of the grid is maintained. The reference occupancy grid map defaults that the occupancy probability of all grids has not been maintained, and the default setting is .

[0010] As a further description of the present invention, establishing an occupancy grid map at the current moment includes the following steps: Detecting the passable space of the lidar input at the current moment The timestamp of which is , and establishing a new occupancy grid map at the moment according to the reference occupancy grid map , .

[0011] As a further description of the present invention, the occupancy grid map at the previous frame moment Convert information to an occupancy grid map and reduce the occupancy probability of the entire map grid, including the following steps: According to the position of the ego vehicle in the world coordinate system at time and the orientation angle , and the position of the ego vehicle in the world coordinate system at time and the orientation angle , calculate to obtain to the translation vector and the rotation matrix of the ego vehicle at time; The expression of the rotation matrix is: ; Among them, is to the difference in orientation angles at time, ; The expression of the translation vector is: ; According to the translation vector and the rotation matrix , convert the position of the grid in the occupancy grid map at time in the th row and th column to the current time ego vehicle coordinate system, and obtain the position of the grid in the ego vehicle coordinate system at time , and the expression is: ; ; Among them, is the position of the grid in the ego vehicle coordinate system at time; Calculate the index in the occupancy grid map at the current time , and the expression is: ; ; Among them, is the horizontal index, is the horizontal index of the grid at the origin of the ego vehicle, is the vertical index, is the longitudinal index of the origin grid of the vehicle itself, is the side length of the grid, representing the resolution of the grid, indicates rounding the result; Judge whether the index is within the occupied grid map inside, that is, the index satisfies and , where, is the number of rows of the grid map, is the number of columns of the grid map; If the index is within the occupied grid map inside, then at the current moment occupies the new grid corresponding to the grid map is: ; ; If , then assign the occupancy probability of the original grid to the new grid, that is ; where, is the occupancy probability of the new grid, is the occupancy probability of the original grid; After the conversion is completed, subtract the prior value from the occupancy probabilities of all grids in the occupied grid map .

[0012] As a further description of the present invention, update the grid occupancy probability of the occupied grid map at the current moment, including updating the grid occupancy probability by lidar passable space detection and updating the grid occupancy probability by fisheye passable space detection. The specific steps are as follows: Traverse the lidar passable space detection at the moment , calculate the index of the th lidar point in the occupied grid map at the current moment , and the expression is: ; ; where, is the axis coordinate of the lidar point in the vehicle's own coordinate system, is the axis coordinate of the lidar point in the vehicle's own coordinate system, is the lateral index, is the lateral index of the origin grid of the vehicle itself, is the longitudinal index, is the longitudinal index of the origin grid of the vehicle itself, is the side length of the grid, representing the resolution of the grid, denotes rounding the result; judge whether the index is within the occupied grid map range, that is, the index satisfies and ; where is the number of rows of the grid map, is the number of columns of the grid map; If the index is within the occupied grid map , then the corresponding grid of this laser point in the occupied grid map is: ; Update the occupancy probability of the grid : If the update flag of the grid is , that is , it means that the grid has been updated by other laser points before and does not need to be updated again. Continue to process the next laser point for lidar free space detection ; If the update flag of the grid is , that is , it means that the grid has not been updated by other laser points and the occupancy probability of the grid needs to be updated. Then the occupancy probability of the grid is added with a fixed value , indicating that the grid has its occupancy probability increased due to being hit by a laser detection point; Update the position information of the grid, that is ; and set the grid update flag to , that is , indicating that the occupancy probability of this grid has been updated; Project the grid updated by lidar free space detection onto the fisheye image to obtain the coordinates of the pixel point corresponding to the grid on the fisheye image, ; where is the abscissa of this pixel point in the fisheye image pixel coordinate system, is the ordinate of this pixel point in the fisheye image pixel coordinate system; Through the abscissa , query and obtain from the boundary of the current fisheye free space detection the ordinate of the fisheye free space detection boundary at ; If , then this grid is within the passable space detected by the fisheye, that is, the fisheye passable space detection considers the grid passable. At this time, subtract a fixed value from the occupancy probability of the grid to reduce the grid occupancy probability; if , then this grid is not within the passable space detected by the fisheye, and the occupancy probability of the grid is not adjusted.

[0013] As a further description of the present invention, the determination of whether to maintain the grid occupancy probability for the grids not updated by the lidar within the lidar blind area includes the following steps: Traverse all grids in the occupancy grid map at the current moment . For the grid in the th row and th column , if is within the blind area of the lidar passable space and has not been updated by the lidar passable space detection, that is, is , is , and the grid occupancy probability has not been maintained, that is, is , then project the grid onto the fisheye image to obtain the coordinates of the pixel point corresponding to the grid on the fisheye image, ; where, is the abscissa of this pixel point in the fisheye image pixel coordinate system, is the ordinate of this pixel point in the fisheye image pixel coordinate system; Through 's abscissa , query and obtain from the boundary of the fisheye passable space detection at the current moment of the ordinate at the fisheye passable space detection boundary at ; if , then this grid is within the non - passable space detected by the fisheye, then restore the globally reduced prior probability, that is, the current grid occupancy probability plus ; if , then the grid is within the passable space detected by the fisheye, and the occupancy probability of the grid is not maintained.

[0014] As a further description of the present invention, the moving object is not expressed in the form of occupying a grid map. The grid occupancy probability within the target contour is cleared through the target tracking result and output to the downstream, including the following steps: Represent the contour of the target using a closed area and traverse all the grids in the occupancy grid map at the current moment For the grid at the th row and the th column , if the grid is located within the target contour in the vehicle's own coordinate system, then set the occupancy probability of this grid to the initial value of the grid occupancy probability , that is ; Traverse all the grids in the occupancy grid map at the current moment For the grid at the th row and the th column , if its occupancy probability is greater than the output occupancy probability threshold , then put the index into the output index set; After the traversal is completed, the index set obtained is the occupancy grid map composed of grids with high occupancy probability. Send the index set to the downstream. The downstream combines the index with the resolution of the grid to restore the position of the occupied grid in the vehicle's own coordinate system.

[0015] Compared with the prior art, the technical effect of the present invention is as follows: The present invention provides a method for generating an occupancy grid map that fuses fisheye passable space detection, which fuses fisheye passable space detection and lidar passable space detection to generate a more accurate and stable occupancy grid map of the passable space; because the fisheye image has rich information and will not detect rain, fog, dust, etc. as impassable, the updated grids by lidar detection are back-projected onto the fisheye image, and the occupancy probability of the grids detected as passable by the fisheye is reduced, which can suppress lidar misdetection and reduce the impact of lidar misdetecting dust and rain, fog, etc. as impassable; at the same time, since the horizontal and vertical field of view angles of the fisheye camera are 180 degrees and there is basically no blind area, it can detect the passable space situation in the lidar blind area. For the grids in the blind area and detected as impassable by the fisheye, maintain the occupancy probability, so that the obstacle grid will not be determined as passable due to the rapid decrease of the occupancy probability after the lidar detection disappears when it enters the blind area, ensuring the safety of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS ​

[0016] Figure 1 Schematic diagram of the overall method flow of the present invention; Figure 2 Schematic diagram of the passable space detection of the lidar of the present invention; Figure 3 Schematic diagram of the passable space detection of the fisheye of the present invention; Figure 4 Schematic diagram of the occupancy grid map in the present invention; Figure 5 Schematic diagram of updating the grid occupancy probability for the passable space detection of the fisheye of the present invention. Detailed implementation manner

[0017] The present invention will be described in detail below with reference to the accompanying drawings: In an embodiment of the present invention, a method for generating an occupancy grid map integrating fisheye passable space detection is disclosed. This method is to reduce the blind area of the lidar and reduce the influence of the lidar misdetecting rain, fog, and dust as impassable. A fisheye camera is introduced for passable space detection. The semantic information of the passable space detection of the fisheye camera is rich, which can accurately distinguish water mist and dust, and can suppress the misdetection of rain, fog, and dust by the lidar. At the same time, the fisheye camera has a large vertical field of view, usually up to 180 degrees, which can well make up for the blind area of the lidar at close range. Specifically, referring to Figures 1-5 As shown, this method is used for a vehicle equipped with a lidar and a fisheye camera, and includes the following steps: Establish a reference occupancy grid map in combination with the lidar blind area information; Obtain the lidar passable space detection and the fisheye passable space detection as input data; According to the timestamp of the current frame lidar passable space detection received, and based on the reference occupancy grid map, establish the occupancy grid map at the current moment; According to the motion information of the vehicle itself, convert the occupancy grid map information at the previous moment to the occupancy grid map at the current moment; Subtract a prior value from the occupancy probability of all grids in the occupancy grid map at the current moment, so that the occupancy probability of the grids not updated by the lidar passable space detection can be reduced; Project the current frame lidar passable space detection onto the occupancy grid map at the current moment, fuse the detection results of the fisheye passable space, and update the grid occupancy probability; Through the fisheye passable space detection, determine whether to maintain the grid occupancy probability for the grids in the lidar blind area that are not updated by the lidar passable space detection; Moving objects are not expressed in the form of an occupancy grid map, and the occupancy probability of the grids within the target contour is cleared through the received target tracking results; Extract the indices of the grids in the occupancy grid map at the current moment with an occupancy probability greater than the threshold and output them for downstream use.

[0018] It should be noted that in this embodiment, the above operation process is based on the occupancy grid map at the current moment; among them, operations such as converting the information of the previous frame to the current frame, reducing the global probability, updating the occupancy probability, and maintaining the blind area are all performed on the occupancy grid map at the current moment. Specifically, an operation is performed on a certain grid among them.

[0019] In this embodiment, the method of fusing the detection of the navigable space of the fisheye camera and the detection of the navigable space of the lidar will be described in detail as follows: I. Input description of lidar and fisheye navigable space detection 1. Lidar navigable space detection As Figure 2 shown, the lidar navigable space detection is the point cloud information of the static or moving obstacles around the vehicle at the current moment; The lidar navigable space detection at the moment is , composed of a series of points, and each point contains position and type information ; among them, is the axis, axis and axis coordinates of the point in the vehicle coordinate system. The vehicle coordinate system takes the center of the vehicle as the origin, the forward direction is the positive direction of the axis, the left direction is the positive direction of the axis, and the upward direction is the positive direction of the axis. represents the type information of the point.

[0020] 2. Fisheye navigable space detection As Figure 3 shown, the fisheye navigable space detection is composed of a group of horizontally continuous pixel points on the image information; the pixel points represent the boundaries of the navigable space on the image. The upper side of the navigable boundary on the fisheye image is the non-navigable area, and the lower side of the boundary is the navigable area. Specifically, the upper side of the boundary is a fence, which is a non-navigable area.

[0021] In this embodiment, the detection points of the fisheye navigable space boundary are represented as: , , ; Among them, is the height of the fisheye image, that is, the number of pixel points of the fisheye image in the vertical direction, is the width of the fisheye image, i.e., the number of pixels of the fisheye image in the horizontal direction; is the abscissa of the detection point in the pixel coordinate system of the fisheye image, increasing continuously from 1 to ; that is, there are a total of passable space boundary points on the fisheye image; taking as the index, the ordinate of the passable boundary on the corresponding column can be queried , that is , . .

[0022] II. Occupancy Grid Map Description 1. Definition of Occupancy Grid In this embodiment, the passable space is represented in the form of an occupancy grid map. As shown in Figure 4 , the occupancy grid map includes a number of grids, each grid is square, and the physical size of the occupancy grid map is , where is the side length of the grid, representing the resolution of the grid, is the number of rows of the grid map, is the number of columns of the grid map, is the total number of grids in the grid map. The starting point of the occupancy grid map is the upper left corner of the grid map, that is, the grid at the 0th row and 0th column , and the origin of the vehicle is located at the grid with the index .

[0023] The grid at the th row and th column is represented as , where ; the attribute of each grid is ; where is the position of the grid in the vehicle coordinate system, usually represented by , is the axis and axis coordinates of the grid position in the vehicle coordinate system, is the probability that the grid is impassable, that is, the grid occupancy probability, is the flag bit indicating whether the grid is in the blind area of the lidar. When it is , it means the grid is in the blind area of the lidar, and when it is , it means the grid is not in the blind area of the lidar, is whether the grid occupancy probability has been updated by the lidar points, is whether the grid occupancy probability has been maintained.

[0024] 2. Reference Occupancy Grid Map Establish Initialize a row, column reference occupancy grid map , and traverse each grid to initialize the grid attributes.

[0025] Specifically, in this embodiment, for the row, the column grid is represented by , and the grid (1) Grid occupancy probability initialization The occupancy probability of grid is set to the initial value , that is, the occupancy probability of grid occupancy probability ; (2) The position of grid in the ego-vehicle system is initialized as: ; ; (3) Blind spot flag calculation The blind spot is a fixed area around the vehicle, determined by the lidar installation position and the lidar field of view. Therefore, the index of the blind spot grid in each occupancy grid map is determined and will not change according to the movement of the ego-vehicle. Therefore, the flag only needs to be calculated once according to the actual blind spot situation and can be reused when creating each new occupancy grid map.

[0026] The ego-vehicle blind spot is represented by the closed area . The closed area is usually a polygon; if the grid 's position in the ego-vehicle system is within the ego-vehicle blind spot , then the blind spot flag of this grid is initialized to ; if the grid 's position is not within the ego-vehicle blind spot , then the blind spot flag of this grid is initialized to Grid other flag initializations: ; ; Among them, Whether the occupancy probability of the grid is updated by the lidar points, the reference occupancy grid map assumes that the occupancy probability of all grids has not been updated by the lidar points, and the default setting is , Whether the occupancy probability of the grid is maintained, the reference occupancy grid map assumes that the occupancy probability of all grids has not been maintained, and the default setting is .

[0027] It should be noted that for any occupancy grid map newly created at a subsequent time , , simply copy , that is .

[0028] III. Processing Flow of the Occupancy Grid Map Generation Method According to the timestamp of the current frame lidar free space detection received, initialize the occupancy grid map at the current time, and transform the occupancy grid map information of the previous frame to the occupancy grid map at the current time according to the vehicle's pose.

[0029] Subtract the fixed prior probability from the occupancy probability of all grids to ensure that the occupancy probability of grids not updated by the lidar free space detection can be reduced.

[0030] Project the current frame lidar free space detection onto the occupancy grid map at the current time, and at the same time combine the fisheye free space detection to update the occupancy probability of the grids.

[0031] Combine the fisheye free space detection to determine whether it is necessary to maintain the occupancy probability of grids in the blind area that have not been updated by the lidar free space detection.

[0032] Express moving objects in the form of targets rather than in the form of occupancy grids, which is more conducive to the prediction and planning module to accurately predict the trajectories of the targets. Specifically, according to the input target tracking results, operate on the occupancy grid map, and reduce the occupancy probability of the grids inside the contour of the moving target to the initial value without external output.

[0033] IV. Establishing the Occupancy Grid Map at the Current Time In this embodiment, to establish the occupancy grid map at the current time, transform the occupancy grid map information at the previous time to the occupancy grid map , and reduce the occupancy probability of all grids in the map, including the following steps: 1. Establish the occupancy grid map at

[0034] The timestamp of the lidar traversable space detection input at the current moment is , and based on the reference occupancy grid map create a new occupancy grid map at the moment, .

[0035] 2. Convert the information to the

[0036] occupancy grid map at the previous frame moment. According to the position of the vehicle in the world coordinate system at the moment and the orientation angle , and the position of the vehicle in the world coordinate system at the moment and the orientation angle , calculate the to translation vector and rotation matrix of the vehicle at the moment; The expression of the rotation matrix is: ; where is the to orientation angle difference at the moment, ; The translation vector is obtained by subtracting the position at the moment from the position at the moment. The expression is: ; According to the to translation vector and rotation matrix of the vehicle at the moment, convert the occupancy grid map at the moment to the occupancy grid map at the current moment ; The specific conversion steps are: (1) According to the translation vector and rotation matrix , convert the position of the grid in the occupancy grid map at the moment to the vehicle coordinate system at the current moment , and obtain the grid in the vehicle coordinate system at the current moment Position of the ego vehicle series at a moment , the expression is: ; ; wherein, is the grid at the position under the ego vehicle series at the moment; (2) Calculate the index occupied in the grid map at the current moment, the expression is: ; ; wherein, is the horizontal index, is the horizontal index of the grid at the origin of the ego vehicle, is the vertical index, is the vertical index of the grid at the origin of the ego vehicle, is the side length of the grid, representing the resolution of the grid, indicating rounding the result; (3) Determine whether the index is within the range of the occupied grid map , that is, the index satisfies and , wherein, is the number of rows of the grid map, is the number of columns of the grid map; if the index is within the range of the occupied grid map , then in the current grid map the corresponding new grid is ; if , then assign the original grid occupancy probability to the new grid, that is ; wherein, is the new grid occupancy probability, is the original grid occupancy probability.

[0037] 3. Global occupancy probability reduction After the conversion, subtract a prior value from the occupancy probability of all grids in the occupied grid map ; ensure that the occupancy probability of grids without lidar passable space detection and update can be reduced.

[0038] V. Update grid occupancy probability Update the occupied grid map at the current moment The grid occupancy probability includes: the lidar passable space detection updated grid occupancy probability and the fisheye passable space detection updated grid occupancy probability; calculate the grid corresponding to the lidar passable space detection point in the current occupancy grid map; according to the update flag bit of the grid, confirm whether the grid occupancy probability needs to be updated; if the occupancy probability needs to be updated, then update the grid occupancy probability by combining the lidar passable space detection and the fisheye passable space detection; if the grid occupancy probability does not need to be updated, then do nothing. The specific steps are as follows: 1. Calculate the grid corresponding to the lidar passable space detection point in the current occupancy grid map Traverse The lidar passable space detection at time , calculate the th laser point in the current occupancy grid map index of , the expression is: ; ; where is the axis coordinate of the laser point in the vehicle's own coordinate system, is the axis coordinate of the laser point in the vehicle's own coordinate system, is the horizontal index, is the horizontal index of the grid at the origin of the vehicle, is the vertical index, is the vertical index of the grid at the origin of the vehicle, is the side length of the grid, representing the resolution of the grid, means rounding the result; Judge whether the index is within the range of the occupancy grid map , that is, the index satisfies and ; where is the number of rows of the grid map, is the number of columns of the grid map; if the index is within the range of the occupancy grid map , then the grid corresponding to this laser point in the occupancy grid map is: .

[0039] 2. According to the update flag bit of the grid, confirm whether the grid occupancy probability needs to be updated If the update flag bit of the grid is , that is , indicating that the grid has been updated by other laser points before and does not need to be updated again. Continue to process the passable space detection of the lidar for the next laser point.

[0040] If the update flag bit of the grid is , that is , indicating that the grid has not been updated by other laser points and the occupancy probability of the grid needs to be updated.

[0041] 3. Update the occupancy probability of the grid by combining the passable space detection of the lidar and the passable space detection of the fisheye (1) Update the occupancy probability of the grid according to the passable space detection of the lidar If the update flag bit of the grid is , then the occupancy probability of the grid is added with a fixed value , indicating that the grid has been hit by a laser detection point, and the occupancy probability of the grid rises; then update the position information of the grid, that is ; and set the grid update flag bit to , that is , indicating that the occupancy probability of this grid has been updated; even if there are subsequent laser points hitting this grid, it will not cause the occupancy probability of the grid to rise, preventing this grid from being updated repeatedly.

[0042] (2) Calculate the projection coordinates of the grid in the fisheye image Project the grid updated by the passable space detection of the lidar onto the passable space detection of the fisheye to obtain the coordinates of the corresponding pixel point of the grid in the fisheye image.

[0043] In this embodiment, according to the installation angle of the fisheye camera, the rotation matrix from the camera system to the vehicle system is calculated, and the expression is: ; where are the yaw, pitch and roll angles of the camera installed in the vehicle system respectively.

[0044] The occupancy grid map is established on the plane with the vehicle system height of 0, so the position of the grid in the vehicle system: , where is the position of the grid in the vehicle system. According to the installation position of the fisheye camera in the vehicle system, and the rotation matrix from the camera system to the vehicle system , the position of the grid in the fisheye camera coordinate system is calculated as: ; It is expressed as , then The included angle with the optical axis axis is , and the expression is: .

[0045] Calculate the projection radius according to the fisheye projection model . Assuming that the fisheye camera projection model is an equidistant projection model, then , where represents the focal length; considering the distortion coefficient of the fisheye camera , perform distortion correction on the projection radius . Using the radial distortion model, the corrected projection radius is .

[0046] Use the internal parameter matrix of the camera to project the corrected onto the fisheye image plane to obtain the grid back-projected to the pixel coordinates of the fisheye image : ; Among them, and are the abscissa and ordinate of this pixel point in the fisheye image pixel coordinate system; is the focal length scaling factor in the horizontal direction, with the unit of pixel, is the focal length scaling factor in the vertical direction, with the unit of pixel; and are the center points of the fisheye image pixel coordinate system.

[0047] (3) Combine the fisheye passable space detection to update the grid occupancy probability In this embodiment, through the abscissa of , it is possible to query from the boundary of the fisheye passable space detection at the current moment to obtain the ordinate of the fisheye passable space detection boundary at , that is . If , then the grid is within the passable space detected by the fisheye, that is, the fisheye passable space detection considers the grid passable. At this time, the occupancy probability of the grid needs to be subtracted by a fixed value to reduce the grid occupancy probability; if , then this grid is not within the passable space for fish-eye detection, and the occupancy probability of the grid is not adjusted, as Figure 5 shown.

[0048] VI. Determination of maintaining the occupancy probability of blind-zone grids Since there are blind zones around the lidar near the vehicle body, as the host vehicle or the target moves, when an obstacle enters the lidar blind zone, due to the lack of laser observations, the occupancy probability of the grids in this area will decrease, resulting in being judged as passable and generating a collision risk. Therefore, for the grids in the blind zone, it is necessary to combine the fish-eye passable space detection to maintain the occupancy probability.

[0049] Specifically, in this embodiment, the determination of whether to perform the maintenance of the grid occupancy probability for the grids in the lidar blind zone that have not been updated by the lidar includes the following steps: Traverse all the grids in the occupancy grid map at the current moment . For the grid at the th row and the th column , if it is within the blind zone of the lidar passable space and has not been updated by the lidar passable space detection, that is is , is , and the grid occupancy probability has not been maintained, that is is , then project the grid onto the fish-eye image to obtain the coordinates of the pixel point corresponding to the grid on the fish-eye image, ; where is the abscissa of the pixel point in the fish-eye image pixel coordinate system, is the ordinate of the pixel point in the fish-eye image pixel coordinate system; Through the abscissa of, the ordinate of the fish-eye passable space detection boundary at can be queried from the boundary of the fish-eye passable space detection at the current moment, that is . If , then this grid is within the non-passable space of the fish-eye detection, then add back the globally reduced prior probability, that is, the current grid occupancy probability plus , and at the same time, the occupancy probability maintenance flag needs to be set to ; If then the grid is within the passable space for fisheye detection, and the occupancy probability of the grid is not maintained.

[0050] VII. Clear the occupancy probability of the grids within the target contour and output the occupancy grid map 1. Clear the occupancy probability of the grids within the target contour Expressing a moving object in the form of a target rather than an occupancy grid is more conducive to the vehicle prediction and planning module to accurately predict the trajectory of the target. Therefore, it is necessary to operate on the occupancy grid map according to the input target tracking result, reduce the occupancy probability of the grids inside the moving target contour to the initial value, and do not output it externally.

[0051] Specifically, represent the contour of the target using a closed area , and the closed area is usually a polygon; traverse all the grids in the occupancy grid map at the current moment. For the grid in the th row and the th column , if the position of the grid in the vehicle's own coordinate system is within the target contour , then set the occupancy probability of this grid to the initial value of the grid occupancy probability , that is

[0052] 2. Output of the occupancy grid map Traverse all the grids in the occupancy grid map at the current moment. For the grid in the th row and the th column , if its occupancy probability is greater than the output occupancy probability threshold , then put the index into the output index set; after the traversal is completed, the obtained index set is the occupancy grid map composed of grids with high occupancy probability, and send the index set to the downstream. The downstream restores the position of the occupancy grid in the vehicle's own coordinate system according to the index in combination with the resolution of the grid.

[0053] Through the above content, specific embodiments of the present invention are disclosed. It should be noted that the above fisheye passable space detection is not limited to being performed on the original fisheye image, but also includes being performed on the cylindrical unfolding diagram of the original fisheye image. The processing method is similar, and the grid is back-projected onto the fisheye unfolding diagram to perform relevant verification and blind area maintenance work. In addition, the calculation of the above grid occupancy probability is the cumulative addition and subtraction of probabilities, but is not limited to this method, and also includes other methods for calculating the occupancy probability, such as the occupancy probability accumulation methods of Bayesian theorem, evidence theory, etc.

[0054] Based on the above content, a method for generating an occupancy grid map that fuses fisheye passable space detection and lidar passable space detection is disclosed. It is applied to scenarios such as mining areas and bulk terminals. Compared with the prior art, the present invention has the following advantages: 1. The present invention establishes an occupancy grid map template under the vehicle system as the reference occupancy grid map. The flag bit indicating whether each grid in the template is a blind area only needs to be calculated once, and can be copied later, improving the generation efficiency of the occupancy grid map; 2. According to the change of the vehicle's motion state, the present invention converts the occupancy grid map at the previous moment to the current moment and reduces the global occupancy probability to ensure that the occupancy probability of the grid without lidar detection update can be reduced; 3. The present invention back-projects the grid updated by lidar detection onto the fisheye image. For the grid detected as passable by the fisheye, its occupancy probability is reduced, which can suppress lidar false detection and reduce the influence of lidar misdetecting dust, rain, fog, etc. as non-passable; 4. The present invention combines fisheye passable space detection to maintain the occupancy probability of the grid in the blind area and detected as non-passable by the fisheye. After the obstacle grid enters the blind area, it will not be judged as passable due to the rapid decrease of the occupancy probability after the lidar detection disappears, ensuring the safety of autonomous driving; 5. The output of the present invention is the integer grid coordinates, rather than directly transmitting the floating-point position information of the grid points, which can save the transmission bandwidth during transmission.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Any other modifications or equivalent replacements made by those of ordinary skill in the art to the technical solutions of the present invention should be covered within the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating an occupancy grid map that integrates fisheye passable space detection, characterized in that, The method is used for a vehicle equipped with a lidar and a fisheye camera, and includes the following steps: Establish a reference occupancy grid map in combination with the lidar blind area information; Obtain the lidar passable space detection and the fisheye passable space detection as input data; According to the timestamp of the current frame lidar passable space detection received and based on the reference occupancy grid map, establish the occupancy grid map at the current moment; According to the motion information of the vehicle itself, convert the occupancy grid map information at the previous moment into the occupancy grid map at the current moment; Subtract a prior value from the occupancy probability of all grids in the occupancy grid map at the current moment, so that the occupancy probability of the grids not updated by the lidar passable space detection can be reduced; Project the current frame lidar passable space detection into the occupancy grid map at the current moment, fuse the detection results of the fisheye passable space, and update the grid occupancy probability; Through the fisheye passable space detection, determine whether to maintain the grid occupancy probability for the grids in the lidar blind area that are not updated by the lidar passable space detection; Moving objects are not expressed in the form of an occupancy grid map, and the occupancy probability of the grids within the target contour is cleared through the received target tracking results; Extract the indices of the grids in the occupancy grid map at the current moment whose occupancy probability is greater than the threshold, and output them for downstream use.

2. A method for generating an occupancy grid map integrating fisheye passable space detection according to claim 1, characterized in that: The lidar passable space detection is the point cloud information of static or moving obstacles around the vehicle at the current moment; The lidar passable space detection at a moment is , composed of a series of points, each point including position and type information ; among them, is the axis, axis and axis coordinates of the point in the vehicle's own coordinate system. The vehicle's own coordinate system takes the center of the vehicle as the origin, the forward direction is the positive direction of the axis, the left direction is the positive direction of the axis, and the upward direction is the positive direction of the axis, indicating the type information of the point.

3. A method for generating an occupancy grid map that fuses fisheye passable space detection, as claimed in claim 1, wherein: The fisheye passable space detection consists of a set of horizontally continuous pixel points on the image information; the pixel points represent the boundary of the passable space on the image, with the non-passable area above the boundary and the passable area below the boundary; The detection of the fisheye passable space boundary at a moment is expressed as: ， ， ； Among them, is the height of the fisheye image, that is, the number of pixels of the fisheye image in the vertical direction, is the width of the fisheye image, that is, the number of pixels of the fisheye image in the horizontal direction; is the abscissa of the detection point on the pixel coordinate system of the fisheye image, increasing continuously from 1 to That is, there are a total of passable space boundary points on the fisheye image; Indexed by , the ordinate of the passable boundary on the column corresponding to can be queried, that is , namely , .

4. A method for generating an occupancy grid map integrating fisheye passable space detection according to claim 1, characterized in that: The occupancy grid map consists of a number of grids, each grid is square, and the physical size of the occupancy grid map is , where is the side length of the grid, representing the resolution of the grid, is the number of rows of the grid map, is the number of columns of the grid map, is the total number of grids in the grid map; The starting point of the occupancy grid map is the upper left corner of the grid map, that is, the grid at the 0th row and the 0th column , and the ego-vehicle origin is located at the grid with the index ; the grid at the th row and the th column is represented as , where ; The attributes of each grid are , where is the position of the grid in the vehicle's own system, represented by . is the axis and axis coordinates of the grid position in the vehicle's own system. is the probability that the grid is impassable, i.e., the occupancy probability of the grid. is the flag indicating whether the grid is in the blind area of the lidar. indicates that the grid is in the lidar blind area. indicates that the grid is not in the lidar blind area. is whether the occupancy probability of the grid is updated by lidar points. is whether the occupancy probability of the grid is maintained.

5. A method for generating an occupancy grid map integrating fisheye passable space detection, characterized in that: The establishment of the reference occupancy grid map includes the following steps: Initialize a row, column reference occupancy grid map , traverse each grid and initialize the grid attributes; for the in the row, the column grid attributes are initialized as follows: Grid Occupancy probability initialization, grid The occupancy probability is set to the initial value , that is, the grid Occupancy probability ; Grid Position under the vehicle's own series Initialized to: ； ； Blind Spot Flag Calculation. The blind spot is a fixed area around the vehicle, represented by a closed area If the grid is within the position of the ego vehicle in the ego vehicle's blind spot then initialize the blind spot flag of the grid to ; If the grid is not within the position of the ego vehicle in the ego vehicle's blind spot then initialize the blind spot flag of the grid to ; Grid Initialization of other flag bits: ； ； Among them, indicates whether the grid occupancy probability is updated by lidar points. By default, all grid occupancy probabilities in the reference occupancy grid map have not been updated by lidar points and are default set to , indicates whether the grid occupancy probability is maintained. By default, all grid occupancy probabilities in the reference occupancy grid map have not been maintained and are default set to .

6. A method for generating an occupancy grid map integrating fisheye passable space detection according to claim 5, characterized in that: The establishment of the occupancy grid map at the current moment includes the following steps: Current moment input lidar passable space detection The timestamp of is, according to the reference occupancy grid map To establish a New occupancy grid map at the moment , .

7. A method for generating an occupancy grid map integrating fisheye passable space detection according to claim 6, characterized in that: The occupancy grid map at the previous frame is converted to the occupancy grid map and the grid occupancy probability of the whole map is reduced, including the following steps: and the grid occupancy probability of the whole map is reduced, including the following steps: According to The position of the ego vehicle in the world coordinate system at a certain moment and the orientation angle as well as The position of the ego vehicle in the world coordinate system at a certain moment and the orientation angle compute to obtain to The translation vector of the ego vehicle at a certain moment and the rotation matrix ; Rotation matrix The expression is: ； Among them, is from the orientation angle difference at the moment to ; Translation vector The expression is: ； According to the translation vector and the rotation matrix , transform the position of the grid occupying the grid map at the -th row and -th column at time to the current time in the ego-vehicle coordinate system, and obtain the position of this grid in the ego-vehicle coordinate system at time . The expression is: ； ； Among them, is the grid at the position of the vehicle itself at a certain moment; Calculation The index that occupies the grid map at the current moment in is expressed as: ； ； Among them, is the horizontal index, is the horizontal index of the grid at the origin of the vehicle itself, is the vertical index, is the vertical index of the grid at the origin of the vehicle itself, is the side length of the grid, representing the resolution of the grid, indicates rounding the result; Determine whether the index is within the occupied grid map That is, the index satisfies and where is the number of rows of the grid map, and is the number of columns of the grid map; If the index is within the occupied grid map then at the current moment, the new grid corresponding to the occupied grid map is: ; If , then assign the original grid occupancy probability to the new grid, i.e., ; where is the new grid occupancy probability, and is the original grid occupancy probability. After the conversion is completed, the occupancy probability of all grids in the grid map will be subtracted by the prior value .​ 8. A method for generating an occupancy grid map integrating fisheye passable space detection according to claim 1, characterized in that: Update the occupancy probability of the grid map at the current moment including updating the occupancy probability of the grid by lidar passable space detection and updating the occupancy probability of the grid by fisheye passable space detection. The specific steps are as follows: Traverse Passable space detection of lidar at a moment , calculate the th laser point occupying the grid map at the current moment index in , the expression is: ； ； Among them, is the laser point under the ego vehicle system axis coordinate, is the laser point under the ego vehicle system axis coordinate, is the lateral index, is the lateral index of the ego vehicle origin grid, is the longitudinal index, is the longitudinal index of the ego vehicle origin grid, is the side length of the grid, representing the resolution of the grid, indicates rounding the result; Determine whether the index is within the occupied grid map range, that is, the index satisfies and ; where is the number of rows of the grid map, is the number of columns of the grid map; If the index is within the occupied grid map then the corresponding grid of this laser point in the occupied grid map is: ; Updated grid Occupancy probability: If the update flag bit of the grid is , that is , it means that the grid has been updated by other laser points before and does not need to be updated again. Continue to process the next laser point for lidar traversable space detection ; If the update flag bit of the grid is , that is , indicating that the grid has not been updated by other laser points and the occupancy probability of the grid needs to be updated. Then the occupancy probability of the grid is added with a fixed value , indicating that the grid due to being hit by a laser detection point, the occupancy probability of the grid rises; Update the position information of the grid, i.e., ; and set the grid update flag to , i.e., , indicating that the occupancy probability of this grid has been updated; The grid updated by lidar passable space detection , is projected onto the fisheye image to obtain the pixel points corresponding to the grid on the fisheye image coordinates, wherein, is the abscissa of the pixel point in the fisheye image pixel coordinate system, is the ordinate of the pixel point in the fisheye image pixel coordinate system; pass The horizontal axis , the boundary detected from the fisheye traversable space at the current moment Get The vertical coordinate of the fisheye traversable space detection boundary ,Right now ;like , then the grid It is in the passable space of fisheye detection, that is, the fisheye passable space detection considers the grid Passable, at this time the grid occupancy probability Subtract a fixed value , reducing the grid occupancy probability; if , then the grid If the grid is not in the passable space of fisheye detection, the occupancy probability of the grid will not be adjusted.

9. A method for generating an occupancy grid map integrating fisheye passable space detection according to claim 1, characterized in that: The determination of whether to maintain the grid occupancy probability for the grids in the lidar blind area that are not updated by the lidar includes the following steps: Traverse all the grids in the occupancy grid map at the current moment For the -th row and the -th column of the grid , if it is in the blind area of the lidar traversable space and has not been updated by the lidar traversable space detection, that is is , is , and the occupancy probability of the grid has not been maintained, that is is , then project the grid onto the fisheye image to obtain the coordinates of the pixel point corresponding to the grid on the fisheye image, ; where is the abscissa of the pixel point in the fisheye image pixel coordinate system, is the ordinate of the pixel point in the fisheye image pixel coordinate system; pass The horizontal axis , the boundary detected from the fisheye traversable space at the current moment Get The vertical coordinate of the fisheye traversable space detection boundary ,Right now ;like , then the grid If the grid is in an inaccessible space detected by the fisheye, the globally reduced prior probability is restored, that is, the current grid occupancy probability Plus ;like , then the grid Within the traversable space of fisheye detection, the occupancy probability of the grid is not maintained.

10. A method for generating an occupancy grid map integrating fisheye passable space detection according to claim 1, characterized in that: Moving objects are not expressed in the form of an occupancy grid map, and the occupancy probability of the grids within the target contour is cleared through the target tracking results and output to the downstream, including the following steps: Use a closed area to represent the outline of the target Traverse all the grids in the occupancy grid map at the current moment For the grid in the th row and th column , if the grid is within the position of the ego vehicle system and within the target outline , then set the occupancy probability of this grid to the initial value of the grid occupancy probability , that is ; Traverse all the grid cells in the occupancy grid map at the current moment Among them, for the grid cell at the th row and the th column , if its occupancy probability is greater than the output occupancy probability threshold , then put the index into the output index set; The index set obtained after traversal is the occupancy grid map composed of high-occupancy probability grids. Send the index set to the downstream, and the downstream restores the position of the occupancy grid in the vehicle's own coordinate system according to the index combined with the resolution of the grid , and restores the position of the occupancy grid in the vehicle's own coordinate system.

Images