Vehicle-based three-dimensional panoramic look-around image construction method and related device

Abstract

The invention discloses a vehicle-based three-dimensional panoramic look-around image construction method and a related device, and the method comprises the steps: obtaining a plurality of two-dimensional look-around images around a vehicle through a vehicle-mounted look-around camera which is used for providing dead-angle-free look-around images around the vehicle; performing movement analysis on a plurality of key grid points in a two-dimensional panoramic top view image spliced based on the plurality of two-dimensional panoramic images according to a preset time period to obtain a plurality of movement distances of the plurality of key grid points in the preset time period; clustering the plurality of key grid points into a plurality of clusters according to the plurality of moving distances; and determining a bottom plane contour through a plurality of key grid points in the clustering cluster corresponding to a preset moving distance in the plurality of moving distances. Constructing a three-dimensional panoramic looking-around model through the bottom plane grid points corresponding to the bottom plane contour, the chamfer connection grid points and the model wall grid points; and filling the plurality of two-dimensional look-around images into the three-dimensional panoramic look-around model through the calibrated internal and external parameters of the vehicle-mounted look-around camera to obtain a three-dimensional panoramic look-around image.

Description

Technical Field

[0001] The present application relates to the technical field of image construction, and particularly relates to a method and related device for constructing a three-dimensional panoramic surround view image based on a vehicle. Background Art

[0002] With the rapid increase in the installation rate of the imaging function of the three-dimensional panoramic surround view system of vehicles, during the driving process of a vehicle, the three-dimensional panoramic surround view image around the vehicle provided by the three-dimensional panoramic surround view system can greatly improve the driving safety of the vehicle.

[0003] In the related art, the three-dimensional panoramic surround view system needs to sense the environmental information around the vehicle by additionally adding ranging sensors such as ultrasonic radars and depth cameras to assist in constructing the three-dimensional panoramic surround view image, so as to provide the three-dimensional panoramic surround view image for the safe driving of the vehicle.

[0004] However, the additional ranging sensors such as ultrasonic radars and depth cameras have relatively high hardware costs and cannot be applied to various types of vehicles, resulting in that the three-dimensional panoramic surround view system may not be constructed. The three-dimensional panoramic surround view image is provided for the safe driving of the vehicle, thereby reducing the driving safety of the vehicle. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a method and related device for constructing a three-dimensional panoramic surround view image based on a vehicle. Without additionally adding hardware sensors, only by obtaining multiple two-dimensional surround view images around the vehicle based on on-vehicle surround view cameras, the three-dimensional panoramic surround view image around the vehicle can be constructed, which is applicable to the three-dimensional panoramic surround view systems of various types of vehicles, constructs and provides the three-dimensional panoramic surround view image for the safe driving of the vehicle, thereby improving the driving safety of the vehicle.

[0006] On the one hand, the embodiments of the present application provide a method for constructing a three-dimensional panoramic surround view image based on a vehicle, and the method includes:

[0007] Obtaining multiple two-dimensional surround view images around the vehicle through on-vehicle surround view cameras; the on-vehicle surround view cameras are used to provide a panoramic surround view image without dead angles around the vehicle;

[0008] Performing movement analysis on multiple key grid points in a two-dimensional panoramic top view image stitched based on the multiple two-dimensional surround view images according to a preset time period to obtain multiple movement distances of the multiple key grid points in the preset time period;

[0009] Performing clustering processing on the multiple key grid points according to the multiple movement distances to obtain multiple clustering clusters;

[0010] Determine the bottom plane contour according to multiple key grid points in the clustering cluster corresponding to the preset moving distance among the multiple moving distances; the preset moving distance is determined according to one or more of the minimum moving distance and the estimated displacement distance, and the estimated displacement distance is estimated according to the vehicle speed and the steering wheel angle within the preset time period;

[0011] Construct a three-dimensional panoramic surround view model according to the bottom plane grid points, chamfer connection grid points and model wall grid points corresponding to the bottom plane contour;

[0012] Fill the multiple two-dimensional surround view images into the three-dimensional panoramic surround view model according to the internal and external calibration parameters of the vehicle-mounted surround view camera to obtain a three-dimensional panoramic surround view image.

[0013] Optionally, the determining the bottom plane contour according to multiple key grid points in the clustering cluster corresponding to the preset moving distance among the multiple moving distances includes:

[0014] Determine multiple key grid points in the clustering cluster corresponding to the preset moving distance among the multiple moving distances as multiple ground grid points;

[0015] Determine the maximum contour of the multiple ground grid points as the bottom plane contour.

[0016] Optionally, the determining step of the multiple key grid points includes:

[0017] Perform grid division on the two-dimensional panoramic top view image to obtain multiple image grids in the two-dimensional panoramic top view image;

[0018] Determine multiple intersection points between the multiple image grids in the two-dimensional panoramic top view image as the multiple key grid points.

[0019] Optionally, the constructing a three-dimensional panoramic surround view model according to the bottom plane grid points, chamfer connection grid points and model wall grid points corresponding to the bottom plane contour includes:

[0020] Determine the bottom plane grid points according to the bottom plane radius corresponding to the bottom plane contour;

[0021] Determine the chamfer connection grid points according to the bottom plane grid points and the chamfer circle radius;

[0022] Determine the model wall grid points according to the chamfer connection grid points and the model wall inclination angle;

[0023] Construct the three-dimensional panoramic surround view model according to the bottom plane grid points, the chamfer connection grid points and the model wall grid points.

[0024] Optionally, filling the multiple two-dimensional omnidirectional images into the three-dimensional panoramic omnidirectional model according to the calibrated internal and external parameters of the vehicle-mounted omnidirectional camera to obtain a three-dimensional panoramic omnidirectional image includes:

[0025] Determine the conversion relationship between the image coordinate system and the vehicle coordinate system according to the calibrated internal and external parameters;

[0026] Determine multiple key pixel points in the multiple two-dimensional omnidirectional images corresponding to multiple key grid points in the three-dimensional panoramic omnidirectional model according to the conversion relationship;

[0027] Fill multiple key pixel points in the multiple two-dimensional omnidirectional images into multiple key grid points in the three-dimensional panoramic omnidirectional model to obtain a three-dimensional filled omnidirectional image;

[0028] Perform interpolation filling on the three-dimensional filled omnidirectional image according to the multiple two-dimensional omnidirectional images to obtain the three-dimensional panoramic omnidirectional image.

[0029] Optionally, the stitching steps of the two-dimensional panoramic overhead image include:

[0030] Perform image correction on the multiple two-dimensional omnidirectional images according to the calibrated internal and external parameters to obtain multiple corrected two-dimensional omnidirectional images;

[0031] Perform brightness equalization on the multiple corrected two-dimensional omnidirectional images to obtain multiple equalized two-dimensional omnidirectional images;

[0032] Stitch the multiple equalized two-dimensional omnidirectional images into the two-dimensional panoramic overhead image.

[0033] Optionally, the vehicle-mounted omnidirectional camera includes vehicle-mounted fisheye cameras with a field of view angle greater than 180 degrees on the front side, rear side, left side, and right side of the vehicle; the multiple two-dimensional omnidirectional images include two-dimensional front fisheye images, two-dimensional rear fisheye images, two-dimensional left fisheye images, and two-dimensional right fisheye images.

[0034] On the other hand, an embodiment of the present application provides a device for constructing a three-dimensional panoramic omnidirectional image based on a vehicle, and the device includes: an acquisition unit, an analysis unit, a clustering unit, a determination unit, a construction unit, and a filling unit;

[0035] The acquisition unit is configured to acquire multiple two-dimensional omnidirectional images around the vehicle through a vehicle-mounted omnidirectional camera; the vehicle-mounted omnidirectional camera is configured to provide a panoramic omnidirectional image without dead angles around the vehicle;

[0036] The analysis unit is configured to perform movement analysis on a plurality of key grid points in a two-dimensional panoramic bird's-eye view image stitched based on the plurality of two-dimensional surround-view images according to a preset time period, and obtain a plurality of movement distances of the plurality of key grid points during the preset time period;

[0037] The clustering unit is configured to perform clustering processing on the plurality of key grid points according to the plurality of movement distances to obtain a plurality of clustering clusters;

[0038] The determination unit is configured to determine a bottom plane contour according to a plurality of key grid points in a clustering cluster corresponding to a preset movement distance among the plurality of movement distances; the preset movement distance is determined according to one or more of a minimum movement distance and an estimated displacement distance, and the estimated displacement distance is estimated according to the vehicle speed and the steering wheel angle of the vehicle during the preset time period;

[0039] The construction unit is configured to construct a three-dimensional panoramic surround-view model according to the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour;

[0040] The filling unit is configured to fill the plurality of two-dimensional surround-view images into the three-dimensional panoramic surround-view model according to the calibrated internal and external parameters of the vehicle-mounted surround-view camera to obtain a three-dimensional panoramic surround-view image.

[0041] Optionally, the determination unit is specifically configured to:

[0042] Determine a plurality of key grid points in a clustering cluster corresponding to a preset movement distance among the plurality of movement distances as a plurality of ground grid points;

[0043] Determine the maximum contour of the plurality of ground grid points as the bottom plane contour.

[0044] Optionally, the determination unit is further configured to:

[0045] Perform grid division on the two-dimensional panoramic bird's-eye view image to obtain a plurality of image grids in the two-dimensional panoramic bird's-eye view image;

[0046] Determine a plurality of intersections between the plurality of image grids in the two-dimensional panoramic bird's-eye view image as the plurality of key grid points.

[0047] Optionally, the construction unit is specifically configured to:

[0048] Determine the bottom plane grid points according to the bottom plane radius corresponding to the bottom plane contour;

[0049] Determine the chamfer connection grid points according to the bottom plane grid points and the chamfer circle radius;

[0050] Determine the model wall grid points according to the chamfer connection grid points and the model wall inclination angle;

[0051] Construct the three-dimensional panoramic surround view model according to the bottom plane grid points, the chamfer connection grid points and the model wall grid points.

[0052] Optionally, the filling unit is specifically configured to:

[0053] Determine the conversion relationship between the image coordinate system and the vehicle coordinate system according to the calibrated internal and external parameters;

[0054] Determine the multiple key pixel points in the multiple two-dimensional surround view images corresponding to the multiple key grid points in the three-dimensional panoramic surround view model according to the conversion relationship;

[0055] Fill the multiple key pixel points in the multiple two-dimensional surround view images into the multiple key grid points in the three-dimensional panoramic surround view model to obtain a three-dimensional filled surround view image;

[0056] Perform interpolation filling on the three-dimensional filled surround view image according to the multiple two-dimensional surround view images to obtain the three-dimensional panoramic surround view image.

[0057] Optionally, the device further includes: a splicing unit;

[0058] The splicing unit is configured to:

[0059] Perform image correction on the multiple two-dimensional surround view images according to the calibrated internal and external parameters to obtain the corrected multiple two-dimensional surround view images;

[0060] Perform brightness equalization on the corrected multiple two-dimensional surround view images to obtain the equalized multiple two-dimensional surround view images;

[0061] Splice the equalized multiple two-dimensional surround view images into the two-dimensional panoramic top view image.

[0062] Optionally, the vehicle-mounted surround view camera includes vehicle-mounted fisheye cameras with a field of view angle greater than 180 degrees on the front side, rear side, left side, and right side of the vehicle; the multiple two-dimensional surround view images include two-dimensional front fisheye images, two-dimensional rear fisheye images, two-dimensional left fisheye images, and two-dimensional right fisheye images.

[0063] On the other hand, an embodiment of the present application provides a vehicle, and the vehicle includes a processor and a memory:

[0064] The memory is used to store a computer program and transmit the computer program to the processor;

[0065] The processor is configured to execute the method described in the above aspect according to the instructions in the computer program.

[0066] On the other hand, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, and when the computer program is executed by a processor, it is used to implement the method described in the above aspect.

[0067] Compared with the prior art, the present application has at least the following advantages:

[0068] Adopting the technical solution of the embodiment of the present application, first, through an in-vehicle surround-view camera for providing a panoramic surround-view image around the vehicle, a plurality of two-dimensional surround-view images around the vehicle are obtained; this method can collect panoramic surround-view images around the vehicle and provide accurate image data for subsequent construction of a three-dimensional panoramic surround-view image. Then, moving analysis is performed on a plurality of key grid points in a two-dimensional panoramic overhead image stitched based on a plurality of two-dimensional surround-view images according to a preset time period to obtain a plurality of moving distances of the plurality of key grid points in the preset time period; the plurality of key grid points are clustered into a plurality of clustering clusters according to the plurality of moving distances; on the basis of estimating a predicted displacement distance through the vehicle speed and steering wheel angle of the vehicle in the preset time period and determining a preset moving distance through one or more of the minimum moving distance and the predicted displacement distance, a bottom plane contour is determined by a plurality of key grid points in the clustering cluster corresponding to the preset moving distance among the plurality of moving distances; this method can determine the bottom plane contour formed by a plurality of key grid points in the ground area based on the fact that the moving distance of the key grid points in the ground area within the same time period is less than that of the key grid points in the non-ground area, and provide an accurate bottom plane for subsequent construction of a three-dimensional panoramic surround-view image. Finally, a three-dimensional panoramic surround-view model is constructed through the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour; a three-dimensional panoramic surround-view image is obtained by filling a plurality of two-dimensional surround-view images into the three-dimensional panoramic surround-view model through the internal and external calibration parameters of the in-vehicle surround-view camera; this method can construct a three-dimensional panoramic surround-view model by considering the bottom plane grid points, chamfer connection grid points, and model wall grid points on the basis of the bottom plane contour, and fill the three-dimensional panoramic surround-view model with a plurality of two-dimensional surround-view images to realize the construction of a relatively accurate three-dimensional panoramic surround-view image.

[0069] It can be seen that this method does not require additional hardware sensors, and only based on the in-vehicle surround-view camera to obtain a plurality of two-dimensional surround-view images around the vehicle, a three-dimensional panoramic surround-view image around the vehicle can be constructed, which is applicable to the three-dimensional panoramic surround-view systems of various types of vehicles, constructs and provides a three-dimensional panoramic surround-view image for safe driving of the vehicle, thereby improving the driving safety of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments of the present application. Obviously, the accompanying drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0071] Figure 1 It is a schematic diagram of the system framework involved in an application scenario in the embodiments of the present application;

[0072] Figure 2 It is a schematic flowchart of a method for constructing a three-dimensional panoramic surround view image based on a vehicle provided by the embodiments of the present application;

[0073] Figure 3 It is a schematic diagram for performing movement analysis on ground grid points and non-ground grid points in the embodiments of the present application to obtain the movement distances of the ground grid points and non-ground grid points in a preset time period;

[0074] Figure 4 It is a schematic diagram of an in-vehicle surround view camera provided by the embodiments of the present application, including in-vehicle fisheye cameras with a field of view angle greater than 180 degrees on the front side, rear side, left side, and right side of the vehicle;

[0075] Figure 5 It is a schematic diagram of a two-dimensional panoramic top view image and multiple key grid points in the two-dimensional panoramic top view image provided by the embodiments of the present application;

[0076] Figure 6 It is a schematic diagram of a bottom plane contour provided by the embodiments of the present application;

[0077] Figure 7 It is a schematic diagram of bottom plane grid points, chamfer connection grid points, and model wall grid points provided by the embodiments of the present application;

[0078] Figure 8 It is a specific flowchart of a method for constructing a three-dimensional panoramic surround view image based on a vehicle provided by the embodiments of the present application;

[0079] Figure 9 It is a schematic structural diagram of a device for constructing a three-dimensional panoramic surround view image based on a vehicle provided by the embodiments of the present application. Detailed implementation manners

[0080] In order to enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope of protection of this application.

[0081] At present, during the driving of a vehicle, in order to improve the driving safety of the vehicle, a three-dimensional panoramic surround view system is required to provide a three-dimensional panoramic surround view image around the vehicle. Usually, the three-dimensional panoramic surround view system needs to sense the environmental information around the vehicle by additionally adding ranging sensors such as ultrasonic radars and depth cameras to assist in constructing the three-dimensional panoramic surround view image, so as to provide the three-dimensional panoramic surround view image for the safe driving of the vehicle. However, it has been found through research that the hardware costs of the additionally added ranging sensors such as ultrasonic radars and depth cameras are relatively high and cannot be applied to various types of vehicles, resulting in that the three-dimensional panoramic surround view system may not be able to construct and provide the three-dimensional panoramic surround view image for the safe driving of the vehicle, thereby reducing the driving safety of the vehicle.

[0082] To solve this problem, in the embodiments of the present application, a vehicle surround-view camera for providing a panoramic surround-view image around the vehicle is used to obtain a plurality of two-dimensional surround-view images around the vehicle; this method can collect panoramic surround-view images around the vehicle and provide accurate image data for subsequent construction of a three-dimensional panoramic surround-view image. Moving analysis is performed on a plurality of key grid points in a two-dimensional panoramic top-view image stitched based on the plurality of two-dimensional surround-view images according to a preset time period to obtain a plurality of moving distances of the plurality of key grid points in the preset time period; the plurality of key grid points are clustered into a plurality of clustering clusters according to the plurality of moving distances; on the basis of estimating a predicted displacement distance through the vehicle speed and steering wheel angle of the vehicle in the preset time period and determining a preset moving distance through one or more of the minimum moving distance and the predicted displacement distance, a bottom plane contour is determined through the plurality of key grid points in the clustering cluster corresponding to the preset moving distance among the plurality of moving distances; based on the fact that the moving distance of the key grid points in the ground area is less than that of the key grid points in the non-ground area in the same time period, this method can determine the bottom plane contour formed by the plurality of key grid points in the ground area and provide an accurate bottom plane for subsequent construction of a three-dimensional panoramic surround-view image. A three-dimensional panoramic surround-view model is constructed through the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour; through the internal and external calibration parameters of the vehicle surround-view camera, the plurality of two-dimensional surround-view images are filled into the three-dimensional panoramic surround-view model to obtain a three-dimensional panoramic surround-view image; this method can construct a three-dimensional panoramic surround-view model by considering the bottom plane grid points, chamfer connection grid points, and model wall grid points on the basis of the bottom plane contour, and fill the three-dimensional panoramic surround-view model with the plurality of two-dimensional surround-view images to realize the construction of a relatively accurate three-dimensional panoramic surround-view image.

[0083] See Figure 1 , Figure 1 FIG. is a schematic diagram of a system framework involved in an application scenario provided by an embodiment of the present application. The application scenario includes a vehicle surround-view camera 101 and a processor 102. The processor 102 includes a central processing unit (CPU) 103 and a graphics processing unit (GPU) 104. The vehicle camera 101 is connected to the processor 102, and the CPU 103 is connected to the GPU 104; the vehicle surround-view camera 101 is used to provide a panoramic surround-view image around the vehicle, and the CPU 103 and GPU 104 in the processor 102 are used to construct a three-dimensional panoramic surround-view image around the vehicle.

[0084] First, in the above application scenario, although the action description of the implementation manner provided by the embodiments of the present application is performed by the processor 102; however, the embodiments of the present application are not limited in terms of the execution subject, as long as the actions disclosed in the implementation manner provided by the embodiments of the present application are executed.

[0085] Secondly, the above scenario is only a scenario example provided by the embodiments of the present application, and the embodiments of the present application are not limited to this scenario.

[0086] Next, in conjunction with the accompanying drawings, the specific implementation manners of the method and related devices for constructing a three-dimensional panoramic surround view image based on a vehicle in the embodiments of the present application will be described in detail through embodiments.

[0087] First, the specific implementation manner of the method for constructing a three-dimensional panoramic surround view image based on a vehicle in the embodiments of the present application will be described in detail through an embodiment.

[0088] See Figure 2 , Figure 2 , which is a schematic flowchart of a method for constructing a three-dimensional panoramic surround view image based on a vehicle provided by the embodiments of the present application. In this embodiment, the method may include the following steps, for example:

[0089] S201: Obtain a plurality of two-dimensional surround view images around the vehicle through an in-vehicle surround view camera; the in-vehicle surround view camera is used to provide a non-blind-spot surround view image around the vehicle.

[0090] In the related art, a three-dimensional panoramic surround view system needs to additionally increase ranging sensors such as ultrasonic radars and depth cameras to sense the environmental information around the vehicle to assist in constructing a three-dimensional panoramic surround view image, so as to provide the three-dimensional panoramic surround view image for the safe driving of the vehicle. However, the hardware costs of the additionally increased ultrasonic radars, depth cameras and other ranging sensors are relatively high and cannot be applied to various types of vehicles, resulting in that the three-dimensional panoramic surround view system may not be able to construct and provide the three-dimensional panoramic surround view image for the safe driving of the vehicle, thereby reducing the driving safety of the vehicle.

[0091] Therefore, in the embodiments of the present application, in order to solve the above problems, there is no need to additionally increase hardware sensors, and only based on the in-vehicle surround view camera that the vehicle itself has, a non-blind-spot surround view image around the vehicle is collected to provide accurate image data for subsequent construction of a three-dimensional panoramic surround view image; based on this, through the in-vehicle surround view camera used to provide a non-blind-spot surround view image around the vehicle, a plurality of two-dimensional surround view images around the vehicle can be obtained.

[0092] S202: Perform movement analysis on a plurality of key grid points in a two-dimensional panoramic overhead image stitched based on a plurality of two-dimensional surround view images according to a preset time period to obtain a plurality of movement distances of the plurality of key grid points in the preset time period.

[0093] S203: Perform clustering processing on the plurality of key grid points according to the plurality of movement distances to obtain a plurality of clustering clusters.

[0094] S204: Determine the bottom plane contour based on multiple key grid points in the clustering cluster corresponding to the preset moving distance among multiple moving distances; the preset moving distance is determined based on one or more of the minimum moving distance and the estimated displacement distance, and the estimated displacement distance is estimated based on the vehicle speed and the steering wheel angle within a preset time period.

[0095] In the embodiment of the present application, considering that constructing a three-dimensional panoramic view image around the vehicle requires constructing a three-dimensional panoramic view model around the vehicle, and the core of the three-dimensional panoramic view model around the vehicle is the bottom plane contour composed of ground grid points; based on this, after executing S201 to obtain multiple two-dimensional panoramic view images around the vehicle, the multiple two-dimensional panoramic view images around the vehicle can be stitched into a two-dimensional panoramic overhead view image around the vehicle. For multiple key grid points in the two-dimensional panoramic overhead view image, based on the basic principle that the moving distance of ground grid points within the same time period is less than that of non-ground grid points (the moving distance of key grid points in the ground area within the same time period is less than that of key grid points in the non-ground area), multiple key grid points in the ground area can be determined, that is, the bottom plane contour formed by multiple ground grid points, so as to construct a three-dimensional panoramic view model subsequently and thus construct a three-dimensional panoramic view image.

[0096] Based on this, after executing S201 to obtain multiple two-dimensional panoramic view images around the vehicle, first, perform a movement analysis on multiple key grid points in the two-dimensional panoramic overhead view image stitched based on multiple two-dimensional panoramic view images according to a preset time period to obtain multiple moving distances of multiple key grid points within the preset time period; then, cluster multiple key grid points into multiple clustering clusters according to multiple moving distances; finally, on the basis of estimating the estimated displacement distance through the vehicle speed and the steering wheel angle of the vehicle within a preset time period and determining the preset moving distance through one or more of the minimum moving distance and the estimated displacement distance, determine the bottom plane contour through multiple key grid points in the clustering cluster corresponding to the preset moving distance among multiple moving distances, that is, multiple ground grid points.

[0097] See Figure 3 , Figure 3A schematic diagram of a movement analysis of ground grid points and non-ground grid points provided in an embodiment of the present application to obtain the movement distance of ground grid points and non-ground grid points in a preset time period; wherein point O is a vehicle-mounted surround view camera, point A1 is a ground grid point at time T, point B1 is a non-ground grid point at time T, the vehicle is in a forward state, point A1 moves to point A2 at time T+1, and point B1 moves to point B2 at time T+1; in the above-mentioned two-dimensional panoramic overhead view image, point B1 at time T is projected on Point B1 is projected on the ground area at time T+1, and point B2 is projected on the ground area at time T+1 to become point B2. Since point A1 and point A2 are originally on the ground plane, point A1 is projected on the ground area at time T and coincides with point A1, and point A2 is projected on the ground area at time T+1 and coincides with point A2; the moving distance of point A1 in the preset time period from time T to time T+1 is A1A2, and the moving distance of point B1 in the preset time period from time T to time T+1 is b1b2, b1b2>A1A2. Based on this, it can be analyzed that in the same time period, the moving distance of the ground grid point is smaller, the moving distance of the non-ground grid point is larger, and the moving distance is proportional to the height of the ground grid point from the ground. The higher the ground grid point is from the ground, the larger the moving distance. That is, for multiple key grid points in a two-dimensional panoramic overhead image, the moving distance of the ground grid point in the same time period is the smallest.

[0098] S205: Constructing a three-dimensional panoramic surround model according to the bottom plane grid points, chamfer connection grid points and model wall grid points corresponding to the bottom plane contour.

[0099] S206: Filling the plurality of two-dimensional surround view images into the three-dimensional panoramic surround view model according to the calibrated internal and external parameters of the vehicle-mounted surround view camera to obtain a three-dimensional panoramic surround view image.

[0100] In the embodiment of the present application, after executing S202-S204 to determine the bottom plane contour, since the bottom plane contour is the core of constructing a three-dimensional panoramic surround view model around the vehicle, considering the bottom plane grid points, chamfer connection grid points and model wall grid points on the basis of the bottom plane contour, a three-dimensional panoramic surround view model can be constructed, and the three-dimensional panoramic surround view model can be filled with multiple two-dimensional surround view images to achieve the construction of a more accurate three-dimensional panoramic surround view image.

[0101] Based on this, after executing S202-S204 to determine the bottom plane contour, first, a three-dimensional panoramic surround view model is constructed through the bottom plane grid points, chamfer connection grid points and model wall grid points corresponding to the bottom plane contour; then, considering the coordinate system differences between multiple two-dimensional surround view images and the three-dimensional panoramic surround view model, the three-dimensional panoramic surround view image can be obtained by filling multiple two-dimensional surround view images into the three-dimensional panoramic surround view model through the calibrated internal and external parameters of the vehicle-mounted surround view camera.

[0102] Through various embodiments provided by this embodiment, first, a vehicle surround camera for providing a panoramic view image around the vehicle is used to obtain a plurality of two-dimensional surround view images around the vehicle; this method can collect panoramic view images around the vehicle without dead angles, providing accurate image data for subsequent construction of a three-dimensional panoramic surround view image. Then, movement analysis is performed on a plurality of key grid points in a two-dimensional panoramic overhead view image stitched based on the plurality of two-dimensional surround view images according to a preset time period, obtaining a plurality of movement distances of the plurality of key grid points in the preset time period; the plurality of key grid points are clustered into a plurality of clustering clusters according to the plurality of movement distances; the bottom plane contour is determined by the plurality of key grid points in the clustering cluster corresponding to the minimum movement distance among the plurality of movement distances; based on the fact that the movement distances of the key grid points in the ground area within the same time period are less than those of the key grid points in the non-ground area, this method can determine the bottom plane contour formed by the plurality of key grid points in the ground area, providing an accurate bottom plane for subsequent construction of a three-dimensional panoramic surround view image. Finally, a three-dimensional panoramic surround view model is constructed through the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour; the plurality of two-dimensional surround view images are filled into the three-dimensional panoramic surround view model through the calibrated internal and external parameters of the vehicle surround camera to obtain a three-dimensional panoramic surround view image; by considering the bottom plane grid points, chamfer connection grid points, and model wall grid points on the basis of the bottom plane contour, this method can construct a three-dimensional panoramic surround view model and fill the three-dimensional panoramic surround view model with a plurality of two-dimensional surround view images to achieve the construction of a relatively accurate three-dimensional panoramic surround view image.

[0103] It can be seen that this method does not require additional hardware sensors. By only obtaining a plurality of two-dimensional surround view images around the vehicle based on the vehicle surround camera, a three-dimensional panoramic surround view image around the vehicle can be constructed, which is applicable to the three-dimensional panoramic surround view systems of various types of vehicles, constructs and provides a three-dimensional panoramic surround view image for safe driving of the vehicle, thereby improving the driving safety of the vehicle.

[0104] In the embodiments of the present application, considering that the vehicle surround-view camera needs to provide a panoramic view image around the vehicle without blind spots, vehicle-mounted fisheye cameras with a field of view angle greater than 180 degrees can be respectively arranged on the front side, rear side, left side, and right side of the vehicle to form the vehicle surround-view camera, so that there is no blind spot in the field of view angle of the vehicle surround-view camera. Correspondingly, the two-dimensional surround-view image around the vehicle obtained by the vehicle-mounted fisheye camera located on the front side of the vehicle is a two-dimensional front fisheye image, the two-dimensional surround-view image around the vehicle obtained by the vehicle-mounted fisheye camera located on the rear side of the vehicle is a two-dimensional rear fisheye image, the two-dimensional surround-view image around the vehicle obtained by the vehicle-mounted fisheye camera located on the left side of the vehicle is a two-dimensional left fisheye image, and the two-dimensional surround-view image around the vehicle obtained by the vehicle-mounted fisheye camera located on the right side of the vehicle is a two-dimensional right fisheye image. Therefore, in an optional implementation manner of the embodiments of the present application, the vehicle surround-view camera includes vehicle-mounted fisheye cameras with a field of view angle greater than 180 degrees located on the front side, rear side, left side, and right side of the vehicle; the multiple two-dimensional surround-view images include two-dimensional front fisheye images, two-dimensional rear fisheye images, two-dimensional left fisheye images, and two-dimensional right fisheye images.

[0105] See Figure 4 , Figure 4 FIG. is a schematic diagram of a vehicle surround-view camera provided in the embodiments of the present application, which includes vehicle-mounted fisheye cameras with a field of view angle greater than 180 degrees located on the front side, rear side, left side, and right side of the vehicle; among them, the vehicle-mounted fisheye camera located on the front side of the vehicle is installed on the front bumper of the vehicle, the vehicle-mounted fisheye camera located on the rear side of the vehicle is installed on the rear bumper of the vehicle, the vehicle-mounted fisheye camera located on the left side of the vehicle is installed on the left rearview mirror of the vehicle, and the vehicle-mounted fisheye camera located on the right side of the vehicle is installed at the right rearview mirror of the vehicle. On this basis, a vehicle coordinate system is established, with the horizontal line where the connection line between the vehicle-mounted fisheye camera located on the left side of the vehicle and the vehicle-mounted fisheye camera located on the right side of the vehicle is the X-axis, and the vertical line where the connection line between the vehicle-mounted fisheye camera located on the front side of the vehicle and the vehicle-mounted fisheye camera located on the rear side of the vehicle is the Y-axis.

[0106] In the embodiments of the present application, the specific implementation method of stitching multiple two-dimensional surround-view images into a two-dimensional panoramic overhead view image is as follows: First, considering that there are certain image distortions in the multiple two-dimensional surround-view images obtained by the vehicle-mounted surround-view camera, and the multiple two-dimensional surround-view images conform to the image coordinate system, while the two-dimensional panoramic overhead view image needs to conform to the vehicle coordinate system, it is necessary to perform image correction on the multiple two-dimensional surround-view images through the internal and external calibration parameters of the vehicle-mounted surround-view camera for adjusting image distortion and converting the image coordinate system and the vehicle coordinate, so as to obtain multiple corrected two-dimensional surround-view images, that is, multiple two-dimensional overhead view images without image distortion; Then, in order to make the image brightness of the multiple corrected two-dimensional surround-view images consistent when stitching them into a two-dimensional panoramic overhead view image later, it is also necessary to perform brightness equalization on the multiple corrected two-dimensional surround-view images to obtain multiple brightness-equalized two-dimensional surround-view images. Finally, stitching the multiple brightness-equalized two-dimensional surround-view images can obtain the two-dimensional panoramic overhead view image. Therefore, in an alternative embodiment of the present application, the stitching step of the two-dimensional panoramic overhead view image in S202 includes the following S1-S3:

[0107] S1: According to the internal and external calibration parameters, perform image correction on multiple two-dimensional surround-view images to obtain multiple corrected two-dimensional surround-view images.

[0108] Among them, when the vehicle leaves the factory, the internal and external parameters of the vehicle-mounted surround-view camera are pre-calibrated to obtain the internal and external calibration parameters, and the internal and external calibration parameters are saved; the internal parameters in the internal and external calibration parameters include the image distortion coefficient and optical axis offset generated by the vehicle-mounted surround-view camera due to the installation of the sensing element, manufacturing tolerance, and lens distortion, and the external parameters in the internal and external calibration parameters include the rotation and translation relationship of the vehicle-mounted surround-view camera relative to the vehicle coordinate system, so the conversion relationship between the image coordinate system and the vehicle coordinate system can be obtained through the internal and external calibration parameters.

[0109] S2: Perform brightness equalization on the multiple corrected two-dimensional surround-view images to obtain multiple brightness-equalized two-dimensional surround-view images.

[0110] S3: Stitch the multiple brightness-equalized two-dimensional surround-view images into a two-dimensional panoramic overhead view image.

[0111] In the embodiments of the present application, the determination method of multiple key grid points in the two-dimensional panoramic overhead view image is as follows: First, divide the two-dimensional panoramic overhead view image into multiple image grids by means of grid division; Then, use the multiple intersection points between the multiple image grids as multiple key grid points in the two-dimensional panoramic overhead view image. Therefore, in an alternative embodiment of the present application, the determination step of multiple key grid points in S202 includes the following S4-S5:

[0112] S4: Divide the two-dimensional panoramic overhead view image into grids to obtain multiple image grids in the two-dimensional panoramic overhead view image.

[0113] S5: Determine multiple intersection points between multiple image grids in the two-dimensional panoramic top-down image as multiple key grid points.

[0114] See Figure 5 , Figure 5 which is a schematic diagram of a two-dimensional panoramic top-down image and multiple key grid points in the two-dimensional panoramic top-down image provided by an embodiment of the present application; among them, the two-dimensional panoramic top-down image obtained by stitching multiple two-dimensional surround-view images around the vehicle is like the entire image shown in Figure 5 Perform grid division on the two-dimensional panoramic top-down image to obtain multiple image grids in the two-dimensional panoramic top-down image like the multiple grids shown in Figure 5 Take multiple intersection points between multiple image grids as multiple key grid points.

[0115] In the embodiment of the present application, when specifically implementing S204, considering the basic principle that for multiple key grid points in the two-dimensional panoramic top-down image, the moving distance of ground grid points within the same time period is less than that of non-ground grid points (the moving distance of key grid points in the ground area within the same time period is less than that of key grid points in the non-ground area), first, from the multiple clusters obtained by clustering multiple key grid points according to multiple moving distances in S203, select the cluster corresponding to the preset moving distance among the multiple moving distances. The multiple key grid points in this cluster are multiple ground grid points; then, take the maximum contour of the multiple ground grid points as the bottom plane contour. Therefore, in an optional implementation manner of the embodiment of the present application, S204 includes the following S2041 - S2042:

[0116] S2041: Determine multiple key grid points in the cluster corresponding to the preset moving distance among the multiple moving distances as multiple ground grid points.

[0117] As an example, when there are N clusters among the multiple clusters, that is, P1, P2,..., P N , determine the cluster corresponding to the preset moving distance from P1, P2,..., P N as P min , and take multiple key grid points in P min as multiple ground grid points.

[0118] S2042: Determine the maximum contour of the multiple ground grid points as the bottom plane contour.

[0119] See Figure 6 , Figure 6 which is a schematic diagram of a bottom plane contour provided by an embodiment of the present application; among them, the maximum contour of multiple ground grid points is determined as the bottom plane contour like Figure 6In the irregular solid line outline shown, multiple key grid points other than multiple ground grid points among the multiple key grid points are used as multiple non-ground grid points.

[0120] In the embodiment of the present application, when S205 is specifically implemented, first, the bottom plane grid points can be determined by the bottom plane radius from the origin of the vehicle coordinate system to the bottom plane contour; secondly, considering that the three-dimensional panoramic surround model has a certain chamfer circle radius, the chamfer connection grid points can be calculated through the bottom plane grid points and the chamfer circle radius; then, further considering that the three-dimensional panoramic surround model has a certain model wall inclination angle, the model wall grid points can be calculated through the chamfer connection grid points and the model wall inclination angle; finally, based on the bottom plane grid points, the chamfer connection grid points and the model wall grid points, the three-dimensional panoramic surround model can be constructed. Therefore, in an optional implementation of the embodiment of the present application, S205 includes the following S2051-S2054:

[0121] S2051: Determine the bottom plane grid points according to the bottom plane radius corresponding to the bottom plane contour.

[0122] S2052: Determine chamfer connection grid points according to the bottom plane grid points and the chamfer circle radius.

[0123] S2053: Determine the model wall grid points according to the chamfer connection grid points and the inclination angle of the model wall.

[0124] See also Figure 7 , Figure 7 A schematic diagram of a bottom plane grid point, a chamfer connection grid point and a model wall grid point provided in an embodiment of the present application; wherein, Figure 6 Based on the bottom plane contour shown, point O is the origin of the vehicle coordinate system, point A is the bottom plane grid point determined by the bottom plane radius OA corresponding to the bottom plane contour, point B is the chamfer connection grid point determined by the bottom plane grid point point A and the chamfer circle radius, and point C is the model wall grid point determined by the chamfer connection grid point point B and the model wall inclination angle.

[0125] S2054: Construct a three-dimensional panoramic surround model according to the bottom plane grid points, the chamfer connection grid points and the model wall grid points.

[0126] In the embodiment of the present application, when specifically implementing S206, considering that multiple two-dimensional surround-view images conform to the image coordinate system and the three-dimensional panoramic surround-view model conforms to the vehicle coordinate system, first, it is necessary to calibrate the internal and external parameters of the in-vehicle surround-view camera to determine the conversion relationship between the image coordinate system and the vehicle coordinate system; secondly, determine multiple key pixel points in multiple two-dimensional surround-view images corresponding to multiple key grid points in the three-dimensional panoramic surround-view model through the conversion relationship; then, fill multiple key pixel points in multiple two-dimensional surround-view images into multiple key grid points in the three-dimensional panoramic surround-view model to obtain a three-dimensional filled surround-view image; finally, perform interpolation filling on other grid points in the three-dimensional filled surround-view image according to multiple two-dimensional surround-view images to obtain a three-dimensional panoramic surround-view image. Therefore, in an alternative implementation manner of the embodiment of the present application, S206 includes the following S2061-S2064:

[0127] S2061: Determine the conversion relationship between the image coordinate system and the vehicle coordinate system according to the calibrated internal and external parameters.

[0128] S2062: Determine multiple key pixel points in multiple two-dimensional surround-view images corresponding to multiple key grid points in the three-dimensional panoramic surround-view model according to the conversion relationship.

[0129] S2063: Fill multiple key pixel points in multiple two-dimensional surround-view images into multiple key grid points in the three-dimensional panoramic surround-view model to obtain a three-dimensional filled surround-view image.

[0130] S2064: Perform interpolation filling on the three-dimensional filled surround-view image according to multiple two-dimensional surround-view images to obtain a three-dimensional panoramic surround-view image.

[0131] As shown above, refer to Figure 8 , Figure 8Specific flowchart of a method for constructing a three-dimensional panoramic surround view image based on a vehicle provided by an embodiment of the present application; the specific process includes: obtaining a plurality of two-dimensional surround view images around the vehicle through an in-vehicle surround view camera; performing image correction on the plurality of two-dimensional surround view images according to calibrated internal and external parameters to obtain the corrected plurality of two-dimensional surround view images; performing brightness equalization on the corrected plurality of two-dimensional surround view images to obtain the equalized plurality of two-dimensional surround view images; stitching the equalized plurality of two-dimensional surround view images into a two-dimensional panoramic top view image. Performing movement analysis on a plurality of key grid points in the two-dimensional panoramic top view image according to a preset time period to obtain a plurality of movement distances of the plurality of key grid points in the preset time period; performing clustering processing on the plurality of key grid points according to the plurality of movement distances to obtain a plurality of clustering clusters; after estimating an estimated displacement distance based on the vehicle speed and steering wheel angle of the vehicle according to the preset time period and determining a preset movement distance according to one or more of the minimum movement distance and the estimated displacement distance, determining a bottom plane contour according to the plurality of key grid points in the clustering cluster corresponding to the preset movement distance among the plurality of movement distances. Constructing a three-dimensional panoramic surround view model according to the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour; filling the plurality of two-dimensional surround view images into the three-dimensional panoramic surround view model according to the calibrated internal and external parameters of the in-vehicle surround view camera to obtain a three-dimensional panoramic surround view image.

[0132] Next, a specific implementation manner of a three-dimensional panoramic surround view image construction device based on a vehicle in an embodiment of the present application will be described in detail through another embodiment.

[0133] See Figure 9 , which shows a schematic structural diagram of a three-dimensional panoramic surround view image construction device based on a vehicle in an embodiment of the present application. In this embodiment, the device may specifically include: an acquisition unit 901, an analysis unit 902, a clustering unit 903, a determination unit 904, a construction unit 905, and a filling unit 906;

[0134] The acquisition unit 901 is configured to obtain a plurality of two-dimensional surround view images around the vehicle through an in-vehicle surround view camera; the in-vehicle surround view camera is used to provide a panoramic surround view image without dead angles around the vehicle;

[0135] The analysis unit 902 is configured to perform movement analysis on a plurality of key grid points in a two-dimensional panoramic top view image stitched based on a plurality of two-dimensional surround view images according to a preset time period to obtain a plurality of movement distances of the plurality of key grid points in the preset time period;

[0136] The clustering unit 903 is configured to perform clustering processing on the plurality of key grid points according to the plurality of movement distances to obtain a plurality of clustering clusters;

[0137] A determination unit 904, configured to determine a bottom plane contour according to a plurality of key grid points in a clustering cluster corresponding to a preset moving distance among a plurality of moving distances; the preset moving distance is determined according to one or more of a minimum moving distance and an estimated displacement distance, and the estimated displacement distance is estimated according to the vehicle speed and the steering wheel angle within a preset time period;

[0138] A construction unit 905, configured to construct a three-dimensional panoramic surround view model according to the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour;

[0139] A filling unit 906, configured to fill a plurality of two-dimensional surround view images into the three-dimensional panoramic surround view model according to the calibrated internal and external parameters of the vehicle-mounted surround view camera to obtain a three-dimensional panoramic surround view image.

[0140] In an alternative implementation manner of the embodiment of the present application, the determination unit 904 is specifically configured to:

[0141] Determine a plurality of key grid points in a clustering cluster corresponding to a preset moving distance among a plurality of moving distances as a plurality of ground grid points;

[0142] Determine the maximum contour of the plurality of ground grid points as the bottom plane contour.

[0143] In an alternative implementation manner of the embodiment of the present application, the determination unit 904 is further configured to:

[0144] Perform grid division on the two-dimensional panoramic top view image to obtain a plurality of image grids in the two-dimensional panoramic top view image;

[0145] Determine a plurality of intersection points between the plurality of image grids in the two-dimensional panoramic top view image as a plurality of key grid points.

[0146] In an alternative implementation manner of the embodiment of the present application, the construction unit 905 is specifically configured to:

[0147] Determine bottom plane grid points according to the bottom plane radius corresponding to the bottom plane contour;

[0148] Determine chamfer connection grid points according to the bottom plane grid points and the chamfer circle radius;

[0149] Determine model wall grid points according to the chamfer connection grid points and the model wall inclination angle;

[0150] Construct a three-dimensional panoramic surround view model according to the bottom plane grid points, chamfer connection grid points, and model wall grid points.

[0151] In an alternative implementation manner of the embodiment of the present application, the filling unit 906 is specifically configured to:

[0152] Determine the conversion relationship between the image coordinate system and the vehicle coordinate system based on the calibrated internal and external parameters;

[0153] According to the conversion relationship, a plurality of key pixel points in a plurality of two-dimensional surround view images corresponding to a plurality of key grid points in a three-dimensional surround view model are determined;

[0154] Filling a plurality of key pixel points in a plurality of two-dimensional surround view images into a plurality of key grid points in a three-dimensional panoramic surround view model to obtain a three-dimensional filled surround view image;

[0155] The three-dimensional filled surround view image is interpolated and filled according to a plurality of two-dimensional surround view images to obtain a three-dimensional panoramic surround view image.

[0156] In an optional implementation of the embodiment of the present application, the device further includes: a splicing unit;

[0157] Splicing unit for:

[0158] According to the calibrated internal and external parameters, image correction is performed on the multiple two-dimensional surround view images to obtain multiple corrected two-dimensional surround view images;

[0159] Performing brightness equalization on the corrected multiple two-dimensional surround view images to obtain multiple equalized two-dimensional surround view images;

[0160] The multiple balanced two-dimensional surround images are stitched together into a two-dimensional panoramic bird's-eye view image.

[0161] In an optional implementation of the embodiment of the present application, the vehicle-mounted surround-view camera includes a vehicle-mounted fisheye camera with a field of view angle greater than 180 degrees located on the front, rear, left and right sides of the vehicle; the multiple two-dimensional surround-view images include a two-dimensional front fisheye image, a two-dimensional rear fisheye image, a two-dimensional left fisheye image and a two-dimensional right fisheye image.

[0162] Through various embodiments provided in this embodiment, a vehicle surround-view camera for providing a panoramic view image around the vehicle is used to obtain a plurality of two-dimensional surround-view images around the vehicle; it can collect a panoramic view image without dead angles around the vehicle, providing accurate image data for subsequent construction of a three-dimensional panoramic surround-view image. Perform movement analysis on a plurality of key grid points in the two-dimensional panoramic top-down image stitched based on the plurality of two-dimensional surround-view images according to a preset time period to obtain a plurality of movement distances of the plurality of key grid points in the preset time period; cluster the plurality of key grid points into a plurality of clusters according to the plurality of movement distances; on the basis of determining a preset movement distance through one or more of the vehicle speed and steering wheel angle of the vehicle in the preset time period, the minimum movement distance, and the estimated displacement distance, determine the bottom plane contour through the plurality of key grid points in the cluster corresponding to the preset movement distance among the plurality of movement distances; based on the fact that the movement distance of the key grid points in the ground area within the same time period is less than that of the key grid points in the non-ground area, it is possible to determine the bottom plane contour formed by the plurality of key grid points in the ground area, providing an accurate bottom plane for subsequent construction of a three-dimensional panoramic surround-view image. Construct a three-dimensional panoramic surround-view model through the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour; fill the three-dimensional panoramic surround-view model with the plurality of two-dimensional surround-view images through the internal and external calibration parameters of the vehicle surround-view camera to obtain a three-dimensional panoramic surround-view image; on the basis of the bottom plane contour, considering the bottom plane grid points, chamfer connection grid points, and model wall grid points, a three-dimensional panoramic surround-view model can be constructed, and the three-dimensional panoramic surround-view model can be filled with the plurality of two-dimensional surround-view images to realize the construction of a relatively accurate three-dimensional panoramic surround-view image.

[0163] It can be seen that this device does not require additional hardware sensors. By only obtaining a plurality of two-dimensional surround-view images around the vehicle based on the vehicle surround-view camera, a three-dimensional panoramic surround-view image around the vehicle can be constructed, which is applicable to the three-dimensional panoramic surround-view systems of various types of vehicles, constructs and provides a three-dimensional panoramic surround-view image for the safe driving of the vehicle, thereby improving the driving safety of the vehicle.

[0164] In addition, an embodiment of the present application further provides a vehicle, and the vehicle includes a processor and a memory:

[0165] The memory is used to store a computer program and transmit the computer program to the processor;

[0166] The processor is used to execute the method described in the above embodiments of the claims according to the instructions in the computer program.

[0167] An embodiment of the present application further provides a computer-readable storage medium, and the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, it is used to implement the method described in the above embodiments.

[0168] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference may be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For related parts, reference may be made to the description in the method part.

[0169] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered as exceeding the scope of this application.

[0170] It should be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0171] As described above, the above are only the preferred embodiments of this application and do not impose any form of limitation on this application. Although this application has been disclosed above with the preferred embodiments, it is not intended to limit this application. Any person skilled in the art can make many possible changes and modifications to the technical solution of this application, or modify it into equivalent embodiments with equivalent changes, without departing from the scope of the technical solution of this application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of this application without departing from the content of the technical solution of this application still fall within the scope of the protection of the technical solution of this application.

Claims

1. A method for constructing a three-dimensional panoramic surround view image based on a vehicle, characterized in that, it includes: Obtaining a plurality of two-dimensional surround view images around the vehicle through in-vehicle surround view cameras; the in-vehicle surround view cameras are used to provide a non-blind-spot surround view image around the vehicle; Performing movement analysis on a plurality of key grid points in a two-dimensional panoramic top view image stitched based on the plurality of two-dimensional surround view images according to a preset time period, and obtaining a plurality of movement distances of the plurality of key grid points in the preset time period; Performing clustering processing on the plurality of key grid points according to the plurality of movement distances to obtain a plurality of clustering clusters; Determining a bottom plane contour according to a plurality of key grid points in a clustering cluster corresponding to a preset movement distance among the plurality of movement distances; the preset movement distance is determined according to one or more of a minimum movement distance and an estimated displacement distance, and the estimated displacement distance is estimated according to the vehicle speed and the steering wheel angle of the vehicle in the preset time period; Constructing a three-dimensional panoramic surround view model according to the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour; Filling the plurality of two-dimensional surround view images into the three-dimensional panoramic surround view model according to the calibrated internal and external parameters of the in-vehicle surround view cameras to obtain a three-dimensional panoramic surround view image.

2. The method according to claim 1, characterized in that, The determining the bottom plane contour according to a plurality of key grid points in a clustering cluster corresponding to a preset movement distance among the plurality of movement distances includes: Determining a plurality of key grid points in a clustering cluster corresponding to a preset movement distance among the plurality of movement distances as a plurality of ground grid points; Determining the maximum contour of the plurality of ground grid points as the bottom plane contour.

3. The method according to claim 1, characterized in that, The determining step of the plurality of key grid points includes: Dividing the two-dimensional panoramic top view image into grids to obtain a plurality of image grids in the two-dimensional panoramic top view image; Determining a plurality of intersections between the plurality of image grids in the two-dimensional panoramic top view image as the plurality of key grid points.

4. The method according to claim 1, characterized in that, The constructing a three-dimensional panoramic surround view model according to the bottom plane grid points, chamfer connection grid points, and model wall grid points corresponding to the bottom plane contour includes: Determining the bottom plane grid points according to the bottom plane radius corresponding to the bottom plane contour; Determining the chamfer connection grid points according to the bottom plane grid points and the chamfer circle radius; Determining the model wall grid points according to the chamfer connection grid points and the model wall inclination angle; Constructing the three-dimensional panoramic surround view model according to the bottom plane grid points, the chamfer connection grid points, and the model wall grid points.

5. The method according to claim 1, characterized in that, The filling the plurality of two-dimensional surround view images into the three-dimensional panoramic surround view model according to the calibrated internal and external parameters of the in-vehicle surround view cameras to obtain a three-dimensional panoramic surround view image includes: Determining the conversion relationship between the image coordinate system and the vehicle coordinate system according to the calibrated internal and external parameters; According to the conversion relationship, determining a plurality of key pixel points in the plurality of two-dimensional surround view images corresponding to a plurality of key grid points in the three-dimensional surround view model; Filling a plurality of key pixel points in a plurality of two-dimensional surround view images into a plurality of key grid points in the three-dimensional panoramic surround view model to obtain a three-dimensional filled surround view image; The three-dimensional filled surround view image is interpolated and filled according to the multiple two-dimensional surround view images to obtain the three-dimensional panoramic surround view image.

6. The method according to claim 1, It is characterized in that The step of stitching the two-dimensional panoramic bird's-eye view image comprises: According to the calibrated internal and external parameters, image correction is performed on the multiple two-dimensional surround view images to obtain multiple corrected two-dimensional surround view images; Performing brightness equalization on the corrected multiple two-dimensional surround view images to obtain multiple equalized two-dimensional surround view images; The equalized multiple two-dimensional surround view images are stitched into the two-dimensional panoramic bird's-eye view image.

7. The method according to any one of claims 1 to 6, It is characterized in that The vehicle-mounted surround-view camera includes a vehicle-mounted fisheye camera with a field of view angle greater than 180 degrees located at the front, rear, left and right sides of the vehicle; the multiple two-dimensional surround-view images include a two-dimensional front fisheye image, a two-dimensional rear fisheye image, a two-dimensional left fisheye image and a two-dimensional right fisheye image.

8. A vehicle-based three-dimensional panoramic image construction device, It is characterized in that include: Acquisition unit, analysis unit, clustering unit, determination unit, construction unit and filling unit; The acquisition unit is used to acquire multiple two-dimensional surround view images around the vehicle through the on-board surround view camera; the on-board surround view camera is used to provide a surround view image without blind spots around the vehicle; The analysis unit is used to perform movement analysis on a plurality of key grid points in the two-dimensional panoramic overhead view image stitched based on the plurality of two-dimensional surround view images according to a preset time period, and obtain a plurality of movement distances of the plurality of key grid points in the preset time period; The clustering unit is used to perform clustering processing on the multiple key grid points according to the multiple moving distances to obtain multiple cluster clusters; The determining unit is used to determine the bottom plane contour according to a plurality of key grid points in a cluster corresponding to a preset moving distance among the plurality of moving distances; the preset moving distance is determined according to one or more of a minimum moving distance and an estimated displacement distance, and the estimated displacement distance is estimated according to a vehicle speed and a steering wheel angle of the vehicle in the preset time period; The construction unit is used to construct a three-dimensional panoramic surround model according to the bottom plane grid points, chamfer connection grid points and model wall grid points corresponding to the bottom plane contour; The filling unit is used to fill the multiple two-dimensional surround view images into the three-dimensional panoramic surround view model according to the calibrated internal and external parameters of the vehicle-mounted surround view camera to obtain a three-dimensional panoramic surround view image.

9. A vehicle, It is characterized in that The vehicle comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1-7 based on the instructions in the computer program.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium is configured to store a computer program, and when the computer program is executed by a processor, it is configured to implement the method according to any one of claims 1-7.

Images