Vehicle surrounding environment sensing method, device and equipment and storage medium

Abstract

The invention provides a vehicle surrounding environment sensing method and device, equipment and a storage medium, and relates to the technical field of intelligent vehicles, and the method comprises the steps: collecting and processing a vehicle surrounding environment image through a vehicle-mounted fisheye camera, and obtaining a panoramic top view image; the method comprises the steps of obtaining a panoramic top view image with grid points, obtaining coordinates of the grid points of a vehicle in a first state and coordinates of the grid points of the vehicle in a second state at any moment, obtaining pixel displacement, judging the height of an object according to the pixel displacement and the height of a fisheye camera, and classifying the grid points. And obtaining the obstacle type in the surrounding environment of the vehicle. Thus, the obstacle type in the surrounding environment of the vehicle is obtained by analyzing and processing the image collected by the fisheye camera, then perception of the surrounding environment of the vehicle is achieved, and due to the fact that the fisheye camera low in cost is used, the problem that in the prior art, when the surrounding environment of the vehicle is perceived, the equipment cost is high is solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle technology, and in particular to a method, device, equipment and storage medium for sensing the surrounding environment of a vehicle. Background Art

[0002] With the continuous improvement of automobile assisted driving technology, drivers have an increasing demand for vehicle surrounding environment perception and danger warning during driving. The existing technical solutions for vehicle surrounding environment perception are mainly divided into two categories: using ultrasonic radar, millimeter wave radar, lidar and other sensors to perceive the surrounding space, among which the solution based on ultrasonic radar to provide reversing obstacle warning function is the most widely used; using visible light vision sensors to visually detect the vehicle surrounding environment, and classify the vehicle surrounding environment through pre-trained semantic segmentation models or target detection models to achieve the purpose of vehicle surrounding environment perception.

[0003] In the above-mentioned methods for sensing the surrounding environment of the vehicle, when radar is used for sensing the surrounding environment of the vehicle, there is a disadvantage of high hardware cost. When visible light vision sensors are used for sensing the surrounding environment, there are high requirements for model training data quality, large model calculation amount, high requirements for vehicle processor computing power, and also high cost problems. Therefore, the equipment used in the vehicle surrounding environment sensing method in the prior art makes the cost required to realize vehicle surrounding environment sensing high. Summary of the invention

[0004] In view of this, the present application provides a method, device, equipment and storage medium for sensing the vehicle's surrounding environment, aiming to solve the problem of high equipment cost corresponding to the method used in the prior art for sensing the vehicle's surrounding environment.

[0005] In a first aspect, the present application provides a method for sensing the surrounding environment of a vehicle, comprising:

[0006] When the vehicle is in the first state, an image of the vehicle's surrounding environment is collected by a vehicle-mounted fisheye camera, and the image of the vehicle's surrounding environment is processed to obtain a panoramic bird's-eye view image of the image of the vehicle's surrounding environment;

[0007] Dividing the panoramic bird's-eye view image to obtain a panoramic bird's-eye view image with grid points;

[0008] Obtaining the coordinates of the grid points of the vehicle in the first state as first coordinates; obtaining the coordinates of the grid points of the vehicle at any time in the second state as second coordinates;

[0009] Calculate the pixel displacement of the vehicle in the panoramic bird's-eye view image according to the first coordinate and the second coordinate;

[0010] The height of the object corresponding to the second coordinate is determined according to the pixel displacement and the height of the fisheye camera, the grid points are classified according to the height, and the type of obstacles in the surrounding environment of the vehicle is obtained according to the classification result.

[0011] Optionally, classifying the grid points according to the heights includes:

[0012] Acquire a first grid point and a second grid point, wherein the second grid point is adjacent to the first grid point in the panoramic overhead view image having grid points; calculate a height difference between a height of an object corresponding to the first grid point and a height of an object corresponding to the second grid point, and when the height difference is not greater than a preset height difference threshold, regard the second grid point and the first grid point as grid points of the same type; when the height difference is greater than a preset height difference threshold, regard the second grid point as a new type of grid point.

[0013] Optionally, the method further includes: connecting grid points belonging to the same category; and displaying an area formed by the connected lines in the panoramic bird's-eye view image.

[0014] Optionally, the method further includes: calculating the distance between a point on the contour line of the display area and the vehicle, and issuing an alarm prompt to the user when the distance is less than a preset alarm threshold.

[0015] Optionally, after the grid points are classified, the method further includes:

[0016] Determine the number of grid points in each type of grid point and remove grid point groups that contain only one grid point.

[0017] Optionally, the method further includes: storing the obstacle type in the surrounding environment of the vehicle as a result of perceiving the surrounding environment of the vehicle.

[0018] In a second aspect, the present application provides a vehicle surrounding environment perception method and device, comprising:

[0019] An image acquisition unit, configured to acquire an image of the vehicle's surroundings through a vehicle-mounted fisheye camera when the vehicle is in the first state, and process the image of the vehicle's surroundings to obtain a panoramic bird's-eye view image of the image of the vehicle's surroundings;

[0020] A division unit, used for dividing the panoramic bird's-eye view image to obtain a panoramic bird's-eye view image with grid points;

[0021] An acquisition unit, configured to acquire the coordinates of the grid points of the vehicle in the first state as first coordinates; and acquire the coordinates of the grid points of the vehicle at any time in the second state as second coordinates;

[0022] a calculation unit, configured to calculate a pixel displacement of the vehicle in the panoramic bird's-eye view image according to the first coordinate and the second coordinate;

[0023] A judgment unit is used to judge the height of the object corresponding to the second coordinate according to the pixel displacement and the height of the fisheye camera, classify the grid points according to the height, and obtain the type of obstacles in the surrounding environment of the vehicle according to the classification result.

[0024] Optionally, the judgment unit is specifically used to obtain a first grid point and a second grid point, wherein the second grid point is adjacent to the first grid point in the panoramic overhead view image with grid points; calculate the height difference between the height of the corresponding object on the first grid point and the height of the corresponding object on the second grid point, and when the height difference is not greater than a preset height difference threshold, regard the second grid point and the first grid point as the same type of grid points; when the height difference is greater than a preset height difference threshold, regard the second grid point as a new type of grid point.

[0025] Optionally, the device further includes: an image processing unit, configured to connect grid points belonging to the same category; and display an area formed by the connected lines in the panoramic bird's-eye view image.

[0026] Optionally, the device further includes: an alarm unit, configured to calculate the distance between a point on the contour line of the display area and the vehicle, and to issue an alarm prompt to the user when the distance is less than a preset alarm threshold.

[0027] Optionally, the judgment unit is further used to, after the grid points are classified, judge the number of grid points in each category of grid points and remove grid point groups containing only one grid point.

[0028] Optionally, the device further includes: a storage unit, configured to store the types of obstacles in the surrounding environment of the vehicle as a result of sensing the surrounding environment of the vehicle.

[0029] In a third aspect, the present application provides a computing device, comprising a memory and a processor, wherein the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the vehicle surrounding environment perception method described in any one of the first aspects above.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium, in which a code is stored. When the code is executed, a device executing the code implements the vehicle surrounding environment perception method described in any one of the first aspects above.

[0031] The present application provides a method for perceiving the surrounding environment of a vehicle. When executing the method, firstly, an image of the surrounding environment of the vehicle is collected by a vehicle-mounted fisheye camera, the image of the surrounding environment of the vehicle is processed to obtain a panoramic overhead image of the image of the surrounding environment of the vehicle, and then the panoramic overhead image is divided to obtain a panoramic overhead image with grid points, the coordinates of the grid points of the vehicle in the first state are obtained as the first coordinates, the coordinates of the grid points of the vehicle at any time in the second state are obtained as the second coordinates, the pixel displacement of the vehicle in the panoramic overhead image is calculated according to the first coordinates and the second coordinates, the height of the object corresponding to the second coordinate is determined according to the pixel displacement and the height of the fisheye camera, the grid points are classified according to the height, and the obstacle type in the surrounding environment of the vehicle is obtained according to the classification result. In this way, the obstacle type in the surrounding environment of the vehicle can be obtained by analyzing and processing the image collected by the fisheye camera, thereby realizing the perception of the surrounding environment of the vehicle. Since a low-cost fisheye camera is used, the problem of high equipment cost in perceiving the surrounding environment of the vehicle in the prior art is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of a vehicle hardware structure provided in an embodiment of the present application;

[0034] Figure 2 A flow chart of a vehicle surrounding environment perception method provided in an embodiment of the present application;

[0035] Figure 3 An image captured by a fisheye camera and its corresponding undistorted top view image provided in an embodiment of the present application;

[0036] Figure 4 A panoramic overhead view image of a vehicle's surrounding environment provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a panoramic overhead view image with grid points provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of pixel displacement calculation provided in an embodiment of the present application;

[0039] Figure 7A road environment perception principle diagram provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of obstacle categories provided in an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of vehicle surrounding environment perception and distance calculation provided in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of obtaining the surrounding environment of a vehicle provided in an embodiment of the present application;

[0043] Fig.11 A schematic diagram of the structure of a vehicle surrounding environment sensing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] As described in the background technology of this application, in the prior art methods for perceiving the surrounding environment of a vehicle, when radar is used for perceiving the surrounding environment of the vehicle, there is a disadvantage of high hardware cost. When visible light vision sensors are used for perceiving the surrounding environment, there are high requirements for the quality of model training data, large amount of model calculation, high requirements for the computing power of the on-board processor, and also high cost problems. Therefore, the equipment used in the prior art methods for perceiving the surrounding environment of the vehicle makes the cost required to achieve the perception of the surrounding environment of the vehicle relatively high.

[0045] In order to solve the above technical problems, an embodiment of the present application provides a vehicle surrounding environment perception method, the method comprising:

[0046] First, a vehicle surrounding environment image is collected by a vehicle-mounted fisheye camera, and the vehicle surrounding environment image is processed to obtain a panoramic overhead image of the vehicle surrounding environment image, and then the panoramic overhead image is divided to obtain a panoramic overhead image with grid points, and the coordinates of the grid points of the vehicle in a first state are obtained as first coordinates, and the coordinates of the grid points of the vehicle at any time in a second state are obtained as second coordinates, and the pixel displacement of the vehicle in the panoramic overhead image is calculated according to the first coordinates and the second coordinates. According to the pixel displacement and the height of the fisheye camera, the height of the object corresponding to the second coordinate is determined, the grid points are classified according to the height, and the obstacle type in the surrounding environment of the vehicle is obtained according to the classification result.

[0047] In this way, the types of obstacles in the vehicle's surroundings can be obtained by analyzing and processing the images collected by the fisheye camera, thereby realizing perception of the vehicle's surroundings. Since a low-cost fisheye camera is used, the problem of high equipment cost in the prior art for perceiving the vehicle's surroundings is solved.

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] Figure 1 A schematic diagram of a vehicle hardware structure provided in an embodiment of the present application is shown below in combination with Figure 1 The simple hardware structure of the vehicle in the embodiment of the present application is introduced as follows: Figure 1 As shown, the vehicle provided in the embodiment of the present application has four fisheye cameras, and the field of view of each fisheye camera is greater than 180 degrees. The fisheye cameras are respectively installed at the front bumper, the rear bumper or the trunk door, the left rearview mirror and the right rearview mirror of the vehicle to ensure that the field of view of the four fisheye cameras can cover all the surroundings of the vehicle, achieving a 360-degree effect without blind spots. After the four fisheye cameras collect images in the surrounding environment of the vehicle, the images are sent to the processor for further processing. The fisheye camera is directly connected to the on-board processor, which at least includes a central processing unit and a graphics processor. The graphic display module and the alarm module are also connected to the processor and are directly controlled by the processor. The processor sends the image processing results to the display module for display to the user. When the distance between the vehicle and the object in the surrounding environment is less than the preset alarm threshold, an alarm prompt is issued to the user through the alarm module.

[0050] It should be understood that in order to demonstrate the low cost and technical advantages of the present application, the hardware and connection method described in this embodiment are configured in most cars, without the need for additional hardware and cost.

[0051] The above-mentioned fisheye cameras have all passed the standard calibration site before leaving the factory to obtain the camera distortion intrinsic parameters and the extrinsic parameters between each camera. The camera distortion intrinsic parameters are used to describe the camera distortion caused by the manufacturing process, installation errors and the fisheye camera lens. The camera extrinsic parameters are used to describe the relative position relationship between each camera, which is represented by the rotation matrix R and the translation matrix T.

[0052] The above is a brief introduction to the hardware devices of the vehicle in the embodiment of the present application. Figure 2 The vehicle surrounding environment perception method of the present application is specifically introduced. Figure 2 A flow chart of a vehicle surrounding environment perception method provided in an embodiment of the present application. Figure 2 As shown, the vehicle surrounding environment perception method provided by the embodiment of the present application may include:

[0053] S201. When the vehicle is in a first state, an image of the vehicle's surroundings is collected by a vehicle-mounted fisheye camera, and the image of the vehicle's surroundings is processed to obtain a panoramic bird's-eye view image of the image of the vehicle's surroundings.

[0054] Since the vehicle is equipped with four fisheye cameras, images of the vehicle's surroundings captured by the four fisheye cameras will be obtained when capturing images. After capturing images of the vehicle's surroundings, the fisheye camera sends the images to the processor. After the processor receives the images from the fisheye camera, it uses the internal and external parameters of the camera to correct each fisheye image into an undistorted overhead view image according to the preset field of view around the vehicle. The following takes the image captured by the fisheye camera installed on the trunk as an example to introduce the image captured by the fisheye camera and the undistorted overhead view image after de-distortion processing. Figure 3 As shown, Figure 3 An image captured by a fisheye camera and its corresponding undistorted overhead view image provided in an embodiment of the present application, the fisheye camera illustrated in the figure is a fisheye camera installed on the trunk door, which captures a rear-view fisheye image and its corresponding undistorted overhead view image, wherein the left picture is an image of the vehicle's surroundings captured by the fisheye camera, and the right picture is an undistorted overhead view image obtained after processing. As an example, the rear-view range is preset to 3 meters, and the scale between the actual distance and the pixel distance is scale, that is, the actual distance of 1m is equal to the unit pixel of scale. Similarly, the images captured by the other three fisheye cameras and the obtained undistorted overhead view image are the same as those obtained. Figure 3 The process is the same and will not be repeated here.

[0055] The first state of the vehicle may be a stationary state or a moving state. The specific first state of the vehicle being stationary or in moving state may be set by those skilled in the art according to actual conditions, that is, images of the vehicle's surroundings may be captured when the vehicle is stationary or in moving state.

[0056] After dedistortion processing is performed on the images collected by the four fisheye cameras, four distortion-free overhead images are obtained. Then, each distortion-free overhead image is converted to the HSV color space, and the "V" channel information is counted to complete the brightness mean statistics of the distortion-free overhead images. The brightness levels of the overhead images with high brightness mean and the overhead images with low brightness mean are adjusted to a certain range to obtain four distortion-free overhead images with high brightness consistency. The four distortion-free overhead images that have undergone brightness equalization processing are further spliced ​​into a 2D panoramic overhead image of the vehicle's surroundings to obtain a panoramic overhead image of the vehicle's surroundings. The specific panoramic overhead image of the vehicle's surroundings is as follows. Figure 4 As shown, Figure 4A panoramic overhead view image of the vehicle's surrounding environment is provided in an embodiment of the present application. The boxed area in the figure is the area that needs to be stitched. The pixel values ​​of the two overlapping images are weightedly fused using the distance weight method to improve the stitching consistency of the 2D panoramic overhead view image.

[0057] The second state of the vehicle can be a stationary state or a moving state. When the first state of the vehicle is that the vehicle is in a stationary state, the second state is a moving state; when the first state of the vehicle is that the vehicle is in a moving state, the second state can be a stationary state or a moving state.

[0058] S202: Divide the panoramic bird's-eye view image to obtain a panoramic bird's-eye view image with grid points.

[0059] The panoramic bird's-eye view image is divided into m*n grids. The grid point spacing and sparsity are set by those skilled in the art according to actual conditions and are not limited here. Figure 5 As shown, Figure 5 A schematic diagram of a panoramic overhead image with grid points provided in an embodiment of the present application, Figure 5 The panoramic bird's-eye view image in has a grid with grid points at the intersections of the lines.

[0060] S203, obtaining the coordinates of the grid points of the vehicle in the first state as first coordinates; obtaining the coordinates of the grid points of the vehicle at any time in the second state as second coordinates.

[0061] Since the pixel displacement of the vehicle during movement needs to be calculated, for the convenience of calculation, when the vehicle is started but not moved, the vehicle coordinate system and the two-dimensional plane world coordinate system are immediately established with the geometric center point of the vehicle as the origin, with the front direction of the vehicle as the positive direction of the Y axis, the right side of the vehicle as the positive direction of the X axis, and the geometric center of the vehicle as the coordinate origin. When the vehicle does not move, the vehicle coordinate system coincides with the world coordinate system. No matter how the vehicle moves, the world coordinate system remains unchanged, while the vehicle coordinate system follows the movement of the vehicle and remains relatively still with the vehicle, always taking the geometric center of the vehicle as the origin, the front direction of the vehicle as the positive direction of the Y axis, and the right side of the vehicle as the positive direction of the X axis. The moment when the vehicle has started but has not moved is recorded as time t, and the position of the grid point at time t on the world coordinate system is recorded as the first coordinate. When the vehicle starts to travel, any moment of the vehicle in the second state is taken as time t+1, and the panoramic overhead view image of the vehicle surrounding environment image at time t+1 will change, and the coordinates of all grid points corresponding to the panoramic overhead view image at time t+1 are obtained as the second coordinate. Since the coordinates of the grid point at time t are known variables, and the coordinates of the grid point at time t+1 are unknown variables, it is necessary to use SIFT, ORB or optical flow method to find the coordinates of the grid point at time t+1 to obtain the second coordinate.

[0062] S204: Calculate the pixel displacement of the vehicle in the panoramic bird's-eye view image according to the first coordinates and the second coordinates.

[0063] After obtaining the first coordinate and the second coordinate, the pixel displacement of the vehicle in the panoramic bird's-eye view image is obtained by calculating the pixel displacement of the grid points. Specifically, Figure 6 As shown, Figure 6 A schematic diagram of pixel displacement calculation provided in an embodiment of the present application, assuming that point O is the origin of the vehicle coordinate system, and point A is any grid point; t is the origin of the vehicle coordinate system at time t, O t+1 is the origin of the vehicle coordinate system at time t+1, A t is the grid point at time t, A t+1 is the position of the grid point at time t+1, find the grid point A at time t+1 t+1 After that, vector That is, the displacement of the grid point in the world coordinate system. In order to improve the efficiency of obstacle height calculation, the displacement vectors of all grid points need to be projected on the Y-axis direction of the vehicle coordinate system. The projection in the Y-axis direction of the vehicle coordinate system is b t b t+1 In order to make the calculation process of pixel displacement clearer, the embodiment of the present application also provides a cross-sectional view of the Y axis of the vehicle coordinate system, such as Figure 7A road environment perception principle diagram provided for an embodiment of the present application, at time t, during the process of correcting the image captured by the fisheye camera into a distortion-free overhead view image, the object A1B1 vertical to the ground will be projected to the ground plane and become A1b1; at time t+1, during the vehicle's driving, the object A1B1 vertical to the ground moves to A2B2, and its projection on the ground is A2b2. Assuming that point A1 and point b1 are grid points to be tracked in the 2D panoramic overhead view image, the displacements of point A1 and point b1 at time t to t+1 are A1A2 and b1b2 respectively. All grid points are tracked, and the pixel displacements of all grid points at time t to t+1 are obtained and saved.

[0064] S205. Determine the height of the object corresponding to the second coordinate according to the pixel displacement and the height of the fisheye camera, classify the grid points according to the height, and obtain the type of obstacles in the surrounding environment of the vehicle according to the classification result.

[0065] Figure 7 OC in the schematic diagram of a road environment perception is the installation height of the fisheye camera. The pixel displacement length b1b2 of the non-ground point b1 during the time period dt is correlated with the height of the object A1B1 vertical to the ground, and the following relationship exists:

[0066]

[0067] It can be easily obtained from the above formula that when the vehicle moving distance converted to pixel distance is equal to the length of b1b2, A1B1=0, which means that the height of the grid point is 0; and when the vehicle moving distance converted to pixel distance is less than the length of b1b2, A1B1>0, which means that there is height information at the location of the grid point, and the height value is the calculated value of A1B1. Use this method to calculate the height of each grid point in turn, obtain the height of the object corresponding to the second coordinate, and save it. In addition, when the vehicle is moving, the processor obtains the current vehicle wheel speed information V from the vehicle bus signal, then within the time period dt between two environmental perceptions, the vehicle moving distance S=V×dt, unit m. In the dt time period, the vehicle moving distance S multiplied by the scale scale should be equal to the pixel displacement length A1A2 of the ground point A1 during the dt time period, and should be less than the pixel displacement length b1b2 of the non-ground point b1 during the dt time period, that is:

[0068]

[0069] Where K is the pixel threshold allowed for error.

[0070] The classifying of the grid points according to the height includes: acquiring a first grid point and a second grid point, the second grid point being adjacent to the first grid point in the panoramic overhead view image having grid points; calculating the height difference between the height of the object corresponding to the first grid point and the height of the object corresponding to the second grid point, and when the height difference is not greater than a preset height difference threshold, treating the second grid point and the first grid point as grid points of the same type; and when the height difference is greater than a preset height difference threshold, treating the second grid point as a new type of grid point.

[0071] It is understandable that the height of a single object in the real world is continuous, so the grid points are classified according to the calculated height information and height continuity, points with a certain continuity are divided into the same object, and the ground and obstacles are instance-segmented along the grid points. The classification method in the embodiment of the present application is as follows:

[0072] For the panoramic bird's-eye view image with grid points at time t+1, start from the upper left of the image and traverse all grid points in a row-by-column manner. The coordinates of the grid point in the i-th row and j-th column are marked as A(i,j), and its corresponding height is H(i,j). The specific height value is the height of the object corresponding to the second coordinate obtained by the above calculation. So the starting point is A(0,0) and the ending point is A(m,n). Then determine whether the current grid point is the starting point A(0,0). If the current grid point is A(0,0), initialize the classification and put A(0,0) into the first category G1. The inherent attributes of the first category G1 include: grid point coordinate set {A(0,0)}, grid point height set {H(0,0)}.

[0073] If the grid point A(i,j) is not the starting point, then there is at least one class G1. Assume that there are two classes G1 and G2, and the inherent attributes of G1 are: grid point coordinate set {A(0,0), A(k,n)}, grid point height set {H(0,0), H(k,n)}. At this time, it is necessary to calculate the distance between the grid point A(i,j) and each grid point coordinate in the G1 attribute grid point coordinate set. If calculated with A(k,n), the calculation method is L=|ik|+|jn|. If L<2 is calculated, the height corresponding to this point is judged again. When the height difference H d=|H(i,j)-H(k,n)| is also less than or equal to the set height difference threshold T, then the grid point A(i,j) is added to the first category G1, and the inherent attributes of G1 become: grid point coordinate set {A(0,0), A(k,n), A(i,j)}, grid point height set {H(0,0), H(k,n), H(i,j)}. If the grid point A(i,j) still cannot be classified after traversing the first category grid points G1 and the second category grid points G2, then a new third category grid point G3 is created and the grid point data is saved. The inherent attributes of G3 are: grid point coordinate set {A(i,j)}, grid point height set {H(i,j)}. Traverse all grid points and classify them according to the above method, and assign them to the appropriate category until the end point A(m,n) also completes the classification task.

[0074] After the classification is completed, it is necessary to determine the number of grid points in each type of grid point and remove grid point groups containing only one grid point. After filtering the grid point groups, the grid points belonging to the same type are connected; and the area formed by the connected lines is displayed in the panoramic bird's-eye view image.

[0075] like Figure 8 As shown, Figure 8 A schematic diagram of obstacle categories provided in an embodiment of the present application is provided. Assume that after clustering and classification, two groups, i.e., two obstacle targets, are separated. Obstacle 1 is an obstacle such as a curb or a bush, and obstacle 2 is a pedestrian obstacle. Then, the grid points of each type of area retained by the cluster are connected, and the maximum contour of the ground area is drawn and rendered in a graphics display device through a graphics processor. The final image is as follows: Fig. 9 As shown, Fig. 9 A schematic diagram of vehicle surrounding environment perception and distance calculation provided in an embodiment of the present application is provided, which calculates the distance between the point on the contour line of the display area and the vehicle, and when the distance is less than a preset alarm threshold, an alarm prompt is issued to the user. Specifically, Fig. 9 As shown in the figure, the road area forms the maximum contour of the road area around the vehicle body, which is expressed by a dotted line in the figure. Traverse the pixel points on the dotted line, set the current traversal point as point A, the center point of the vehicle is point O, and start from point A. Search the outer contour of the vehicle in the AO vector direction to obtain the outer contour point B, calculate the pixel distance between AB, and use the scale of the actual distance and the pixel distance as scale, then the spatial distance D can be calculated:

[0076]

[0077] Each point on the road area contour can be traversed to obtain its corresponding real space distance from the vehicle body. When the real space distance of some points on the contour is less than the alarm setting threshold, the driver is warned through the alarm module, and the closer the real distance is, the higher the alarm intensity is. In addition, after the alarm information is generated, the alarm log information can also be stored through the storage device on the vehicle, such as recording and retaining information such as the alarm distance, alarm time, alarm intensity, and alarm location coordinates.

[0078] The above embodiment provides a specific process of the vehicle sensing obstacles in the surrounding environment during driving and issuing an alarm based on the distance between the vehicle and the obstacle. In another implementation provided by the embodiment of the present application, the vehicle's surrounding environment sensing results at multiple moments can also be stored. For details, see Fig.10 , Fig.10 A schematic diagram of obtaining the vehicle surrounding environment provided in an embodiment of the present application is provided. At the time 1 when the vehicle starts, a world coordinate system and a vehicle coordinate system are established. According to the vehicle speed and steering wheel angle information, the movement trajectory of the vehicle at time 2-4 is tracked in the world coordinate system. In the above steps, all the vehicle surrounding environments and obstacles are sensed in the vehicle coordinate system, and the sensed results are converted from the vehicle coordinate system to the world coordinate system for environmental memory and storage, and a two-dimensional plane vehicle surrounding environment map is established.

[0079] The above are some specific implementations of a vehicle surrounding environment perception method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.

[0080] Fig.11 A schematic diagram of the structure of a vehicle surrounding environment sensing device provided in an embodiment of the present application. Fig.11 As shown, the vehicle surrounding environment perception device 110 provided in the embodiment of the present application includes:

[0081] The image acquisition unit 111 is used for acquiring an image of the vehicle's surroundings through a vehicle-mounted fisheye camera when the vehicle is in the first state, and processing the image of the vehicle's surroundings to obtain a panoramic bird's-eye view image of the image of the vehicle's surroundings;

[0082] A division unit 112 is used to divide the panoramic bird's-eye view image to obtain a panoramic bird's-eye view image with grid points;

[0083] The acquisition unit 113 is used to acquire the coordinates of the grid points of the vehicle in the first state as the first coordinates; and acquire the coordinates of the grid points of the vehicle at any time in the second state as the second coordinates;

[0084] A calculation unit 114, configured to calculate a pixel displacement of the vehicle in the panoramic bird's-eye view image according to the first coordinate and the second coordinate;

[0085] The judgment unit 115 is used to judge the height of the object corresponding to the second coordinate according to the pixel displacement and the height of the fisheye camera, classify the grid points according to the height, and obtain the obstacle type in the surrounding environment of the vehicle according to the classification result.

[0086] In one implementation of the embodiment of the present application, the judgment unit is specifically used to obtain a first grid point and a second grid point, wherein the second grid point is adjacent to the first grid point in the panoramic overhead view image with grid points; calculate the height difference between the height of the corresponding object on the first grid point and the height of the corresponding object on the second grid point, and when the height difference is not greater than a preset height difference threshold, treat the second grid point and the first grid point as the same type of grid points; when the height difference is greater than the preset height difference threshold, treat the second grid point as a new type of grid point.

[0087] In one implementation of the embodiment of the present application, the device further includes: an image processing unit, configured to connect grid points belonging to the same category; and display an area formed by the connected lines in the panoramic overhead view image.

[0088] In one implementation of the embodiment of the present application, the device further includes: an alarm unit, configured to calculate the distance between a point on the contour line of the display area and the vehicle, and to issue an alarm prompt to the user when the distance is less than a preset alarm threshold.

[0089] In an implementation of the embodiment of the present application, the judgment unit is further used to, after the grid points are classified, judge the number of grid points in each category of grid points and remove grid point groups that only include one grid point.

[0090] In an implementation of the embodiment of the present application, the device further includes: a storage unit, configured to store the obstacle type in the surrounding environment of the vehicle as a result of sensing the surrounding environment of the vehicle.

[0091] The embodiments of the present application also provide corresponding computing devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0092] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the method described in any embodiment of the present application.

[0093] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the method described in any embodiment of the present application.

[0094] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0095] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0096] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and apparatus embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for sensing vehicle surrounding environment, characterized in that: The method comprises: When the vehicle is in the first state, an image of the vehicle's surrounding environment is collected by a vehicle-mounted fisheye camera, and the image of the vehicle's surrounding environment is processed to obtain a panoramic bird's-eye view image of the image of the vehicle's surrounding environment; Dividing the panoramic bird's-eye view image to obtain a panoramic bird's-eye view image with grid points; Obtaining the coordinates of the grid points of the vehicle in the first state as first coordinates; obtaining the coordinates of the grid points of the vehicle at any time in the second state as second coordinates; Calculate the pixel displacement of the vehicle in the panoramic bird's-eye view image according to the first coordinates and the second coordinates; The height of the object corresponding to the second coordinate is determined according to the pixel displacement and the height of the fisheye camera, the grid points are classified according to the height, and the type of obstacles in the surrounding environment of the vehicle is obtained according to the classification result.

2. The method according to claim 1, characterized in that The classifying the grid points according to the height comprises: Acquire a first grid point and a second grid point, wherein the second grid point is adjacent to the first grid point in the panoramic overhead view image having grid points; calculate a height difference between a height of an object corresponding to the first grid point and a height of an object corresponding to the second grid point, and when the height difference is not greater than a preset height difference threshold, regard the second grid point and the first grid point as grid points of the same type; when the height difference is greater than a preset height difference threshold, regard the second grid point as a new type of grid point.

3. The method according to claim 1, characterized in that: The method further comprises: The grid points belonging to the same category are connected; and the area formed by the connected lines is displayed in the panoramic bird's-eye view image.

4. The method according to claim 3, characterized in that The method further comprises: The distance between the point on the contour line of the display area and the vehicle is calculated, and when the distance is less than a preset alarm threshold, an alarm prompt is issued to the user.

5. The method according to claim 2, characterized in that: After the grid points are classified, the method further includes: Determine the number of grid points in each type of grid point and remove grid point groups that contain only one grid point.

6. The method according to claim 1, characterized in that The method further comprises: The obstacle types in the surrounding environment of the vehicle are stored as the perception results of the surrounding environment of the vehicle.

7. A vehicle surrounding environment sensing device, characterized in that: The device comprises: An image acquisition unit, configured to acquire an image of the vehicle's surroundings through a vehicle-mounted fisheye camera when the vehicle is in the first state, and process the image of the vehicle's surroundings to obtain a panoramic bird's-eye view image of the image of the vehicle's surroundings; A division unit, used for dividing the panoramic bird's-eye view image to obtain a panoramic bird's-eye view image with grid points; An acquisition unit, configured to acquire the coordinates of the grid points of the vehicle in the first state as first coordinates; and acquire the coordinates of the grid points of the vehicle at any time in the second state as second coordinates; a calculation unit, configured to calculate a pixel displacement of the vehicle in the panoramic bird's-eye view image according to the first coordinate and the second coordinate; A judgment unit is used to judge the height of the object corresponding to the second coordinate according to the pixel displacement and the height of the fisheye camera, classify the grid points according to the height, and obtain the type of obstacles in the surrounding environment of the vehicle according to the classification result.

8. The device according to claim 7, characterized in that The device also includes: The image processing unit is used to connect the grid points belonging to the same category; and display the area formed by the connected lines in the panoramic bird's-eye view image.

9. A computing device, characterized in that The computing device includes: a memory and a processor; The memory is used to store computer programs; The processor is used to implement the vehicle surrounding environment perception method as described in any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the vehicle surrounding environment perception method according to any one of claims 1 to 6 is implemented.

Images