Road image processing method based on inverse perspective transformation and electronic device

Abstract

This application discloses a road image processing method and electronic device based on inverse perspective transformation. The method involves acquiring LiDAR point cloud data and a first vehicle-mounted image, determining a first conversion parameter between the LiDAR point cloud data and the first vehicle-mounted image, associating the LiDAR point cloud data and the first vehicle-mounted image based on the first conversion parameter to obtain road surface point cloud data, and determining the road surface slope variation data using the road surface point cloud data. Based on the slope variation data, the road surface contained in the first vehicle-mounted image is segmented to obtain at least one road segment, wherein adjacent road segments have different slopes. A second conversion parameter corresponding to each road segment is determined based on inverse perspective transformation. Using the second conversion parameters of each road segment, the first vehicle-mounted image is transformed to obtain a first horizontal road image corresponding to the first vehicle-mounted image. The road image processing method based on inverse perspective transformation provided in this application can improve the accuracy of BEV (Battery on the Vehicle) viewpoint images.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a road image processing method and electronic device based on inverse perspective transformation. Background Technology

[0002] In the field of autonomous driving, after a vehicle captures road images using a forward-facing camera, the two-dimensional pixel information in the image needs to be converted into geometric structure information in three-dimensional space to achieve accurate perception of environmental elements such as roads, lane lines, and obstacles. Inverse Perspective Mapping (IPM), as a key technology connecting two-dimensional images and three-dimensional space, aims to eliminate perspective effects and project the road surface area in the forward-facing image captured by the camera onto a bird's-eye view (BEV) perspective, thereby providing intuitive geometric information for decision-making modules such as path prediction and obstacle avoidance.

[0003] Currently, in related technologies, IPM technology typically relies on manually labeled reference points for calibration, calculating transformation parameters through the correspondence between single-camera images and reference points. However, real-world road environments are complex, and traditional IPM methods struggle to dynamically adapt to these complexities. Relying on fixed, manually labeled reference points leads to geometric distortion in the resulting BEV-view images, failing to meet the high-precision, continuous environmental perception requirements of autonomous driving systems. Summary of the Invention

[0004] This application provides a road image processing method and electronic device based on inverse perspective transformation, which can improve the accuracy of BEV perspective images.

[0005] This application provides a road image processing method based on inverse perspective transformation, including:

[0006] Acquire multi-source sensor data, wherein the multi-source sensor data includes at least lidar point cloud data and a first vehicle-mounted image;

[0007] Determine the first conversion parameters between the lidar point cloud data and the first vehicle-mounted image;

[0008] Based on the first transformation parameters, the lidar point cloud data is correlated with the first vehicle-mounted image to obtain road surface point cloud data;

[0009] By using road surface point cloud data, the slope variation data of the road surface can be determined;

[0010] Based on slope variation data, the road surface contained in the first vehicle image is segmented to obtain at least one road segment, wherein adjacent road segments have different slopes.

[0011] The second transformation parameters corresponding to each road segment are determined based on inverse perspective transformation;

[0012] The first vehicle image is converted using the second conversion parameters for each road segment to obtain the first horizontal road image corresponding to the first vehicle image.

[0013] This application also provides a road image processing apparatus based on inverse perspective transformation, comprising:

[0014] The acquisition module is used to acquire multi-source sensor data, wherein the multi-source sensor data includes at least lidar point cloud data and a first vehicle-mounted image;

[0015] The processing module is used to determine the first conversion parameters between the lidar point cloud data and the first vehicle-mounted image;

[0016] The processing module is also used to associate the lidar point cloud data with the first vehicle image based on the first conversion parameters to obtain road surface point cloud data;

[0017] The processing module is also used to determine the slope change data of the road surface through the road surface point cloud data;

[0018] The processing module is also used to segment the road surface contained in the first vehicle image based on the slope change data to obtain at least one road segment, wherein adjacent road segments have different slopes.

[0019] The processing module is also used to determine the second transformation parameters corresponding to each road segment based on the inverse perspective transformation;

[0020] The processing module is also used to convert the first vehicle image using the second conversion parameters of each road segment to obtain the first horizontal road image corresponding to the first vehicle image.

[0021] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of the road image processing method based on inverse perspective transformation described above when executing the computer program.

[0022] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the road image processing method based on inverse perspective transformation described above.

[0023] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the road image processing method based on inverse perspective transformation described above.

[0024] The road image processing method and electronic device based on inverse perspective transformation provided in this application can obtain road surface point cloud data by combining LiDAR point cloud data and vehicle-mounted images. The road surface point cloud data can then be used to determine the slope variation data. Furthermore, the road surface is divided into segments according to slope using this slope variation data. For segments with different slopes, the transformation parameters adapted to the current slope are dynamically calculated. This overcomes the limitations of traditional IPM calibration relying on manual annotation. Simultaneously, it dynamically adapts to undulating road surface scenarios, improving the geometric accuracy of converting vehicle-mounted images. Thus, vehicle-mounted images can be converted into horizontal road images that accurately reflect actual road surface information. Therefore, the road image processing method based on inverse perspective transformation provided in this application can improve the precision and accuracy of horizontal road image conversion. Attached Figure Description

[0025] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0027] Figure 2 A schematic diagram illustrating the acquisition of conversion parameters for related technologies;

[0028] Figure 3 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 1 ;

[0029] Figure 4 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 2 ;

[0030] Figure 5 This is a schematic diagram illustrating the projection of LiDAR point cloud data onto an onboard image.

[0031] Figure 6 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 3 ;

[0032] Figure 7 A schematic diagram of the actual road surface condition for example;

[0033] Figure 8 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 4 ;

[0034] Figure 9 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 5 ;

[0035] Figure 10 Grid division diagram for example Figure 1 ;

[0036] Figure 11 Grid division diagram for example Figure 2 ;

[0037] Figure 12 Grid division diagram for example Figure 3 ;

[0038] Figure 13 A schematic diagram of the projection of the target road surface point cloud as an example;

[0039] Figure 14 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 6 ;

[0040] Figure 15 Lane lines as an example from the perspective of a BEV;

[0041] Figure 16 This is a schematic diagram illustrating how lane lines are determined based on coordinate data.

[0042] Figure 17 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 7 ;

[0043] Figure 18 A schematic diagram illustrating the transformation of a 3D road image as an example;

[0044] Figure 19 A schematic diagram of a 3D road image for example;

[0045] Figure 20 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 8 ;

[0046] Figure 21 A schematic diagram of a vehicle-mounted camera as an example;

[0047] Figure 22 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 9 ;

[0048] Figure 23A schematic diagram of the structure of the road image processing device based on inverse perspective transformation provided in the embodiments of this application;

[0049] Figure 24 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, other embodiments obtained by those of ordinary skill in the art without creative effort are all within the protection scope of this application.

[0051] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0052] It should also be noted that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. The terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; they can be internal connections between two elements. The terms "parallel," "perpendicular," and "equal" include the described situation and situations similar to the described situation, the range of which is within an acceptable deviation range, wherein the acceptable deviation range is determined by those skilled in the art taking into account the measurement under discussion and the error associated with the measurement of a particular quantity (i.e., the limitations of the measurement system). For example, "parallel" includes absolute parallelism and approximate parallelism, where an acceptable deviation range for approximate parallelism can be, for example, within 5°; "perpendicular" includes absolute perpendicularity and approximate perpendicularity, where an acceptable deviation range for approximate perpendicularity can also be, for example, within 5°. "Equal" includes absolute equality and approximate equality, where an acceptable deviation range for approximate equality can be, for example, the difference between the two equal items being less than or equal to 5% of either one. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0053] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application, such as... Figure 1 As shown, in the field of autonomous driving, after a vehicle captures road images through a forward-facing camera, it needs to convert the two-dimensional pixel information in the image into geometric structure information in three-dimensional space to achieve accurate perception of environmental elements such as roads, lane lines, and obstacles. IPM, as a key technology connecting two-dimensional images and three-dimensional space, aims to eliminate perspective effects by projecting the road surface area in the forward-facing image captured by the camera onto the BEV's perspective, thereby providing intuitive geometric information for decision-making modules such as path prediction and obstacle avoidance.

[0054] Currently, in related technologies, IPM technology usually relies on manually labeled reference points for calibration, and calculates transformation parameters through the correspondence between single-camera images and reference points. Figure 2 A schematic diagram illustrating the acquisition of conversion parameters for related technologies, such as... Figure 2As shown, the traditional IPM method requires marking at least four non-collinear reference points on the road surface to obtain the transformation parameters. Then, the corresponding points of these reference points are extracted from the forward-view image captured by the camera. Finally, the transformation parameters are calculated by comparing the spatial reference points with their corresponding points in the image. However, real-world road environments are complex, and the traditional IPM method struggles to dynamically adapt to these complexities, leading to distance perception biases. For example... Figure 1 As shown, region 1, being closer in distance, shows better results after transformation. However, region 2, being farther away, shows poorer results after transformation, exhibiting geometric distortion and disrupting lane line parallelism. Therefore, the relevant technology struggles to meet the high-precision, continuous environmental perception requirements of autonomous driving systems. Furthermore, when vehicles travel on inclines or declines, fixed IPM parameters also lead to geometric distortion in the projected area.

[0055] The road image processing method and electronic device based on inverse perspective transformation provided in this application combine LiDAR point cloud data and vehicle-mounted images to obtain road surface point cloud data. This road surface point cloud data allows for the determination of road slope variation data. The road surface is then divided into segments based on slope variation data. For segments with different slopes, transformation parameters adapted to the current slope are dynamically calculated. This overcomes the limitations of traditional IPM calibration relying on manual annotation. Simultaneously, it dynamically adapts to undulating road surface scenarios, improving the geometric accuracy of converting vehicle-mounted images. This allows the vehicle-mounted image to be converted into a horizontal road image that accurately reflects the actual road surface information. Therefore, the road image processing method based on inverse perspective transformation provided in this application can improve the precision and accuracy of horizontal road image conversion.

[0056] The embodiments of this application provide a road image processing method based on inverse perspective transformation. The system is described in detail below: Figure 3 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 1 ,like Figure 3 As shown, it includes:

[0057] S301. Acquire multi-source sensor data, wherein the multi-source sensor data includes at least lidar point cloud data and a first vehicle-mounted image.

[0058] Based on the scenario example, the execution entity in this embodiment can be the vehicle's control system. Multi-source sensor data refers to a data set from different types of sensors. LiDAR point cloud data: LiDAR acquires three-dimensional point cloud data of the road surface by emitting laser pulses, such as three-dimensional spatial coordinate information. The first vehicle-mounted image is two-dimensional image data of the road area acquired by the vehicle-mounted camera through optical imaging, including two-dimensional pixel information for each pixel.

[0059] S302, Determine the first conversion parameter between the lidar point cloud data and the first vehicle-mounted image.

[0060] In the case of a scenario example, the installation positions of the LiDAR and the vehicle camera on the vehicle are fixed. Joint calibration can be performed by the LiDAR and the vehicle camera to obtain calibration parameters such as the internal parameters of the vehicle camera and the transformation parameters between the LiDAR coordinate system and the vehicle camera coordinate system. Based on the above calibration parameters, the first transformation parameters for projecting the LiDAR point cloud data onto the first vehicle image can be determined.

[0061] Optional, Figure 4 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 2 ,like Figure 4 As shown, S302 includes:

[0062] S401. Based on preset spatial corresponding points, establish the lidar coordinate system and the camera coordinate system.

[0063] Using scenario examples, for the same object, the LiDAR coordinate system and the camera coordinate system use the same name, such as "telephone pole" or "street lamp." Under the same name, the LiDAR coordinate system and the camera coordinate system are established based on the same object.

[0064] S402. Determine the mapping relationship between the lidar coordinate system and the camera coordinate system to obtain the first transformation parameters.

[0065] Based on a scenario example and pre-defined parameters of the LiDAR and camera, such as the camera's center point and focal length, and the LiDAR's center point, the mapping relationship between the LiDAR coordinate system and the camera coordinate system is determined. This involves identifying the transformation parameters between the two coordinate systems to obtain the first transformation parameters mapping the LiDAR point cloud data to the first vehicle-mounted image. Using the method provided in this example, pre-defined spatial corresponding points can be used as a reference to accurately determine the mapping relationship between the two coordinate systems, thereby accurately obtaining the first transformation parameters.

[0066] S303. Based on the first conversion parameters, the lidar point cloud data is correlated with the first vehicle-mounted image to obtain road surface point cloud data.

[0067] Combined with scenario examples, Figure 5 This is an example of projecting LiDAR point cloud data onto an onboard image, such as... Figure 5As shown, based on the first conversion parameter, the LiDAR point cloud data can be projected onto the first vehicle image to obtain the three-dimensional spatial coordinate information corresponding to each pixel in the first vehicle image. The first vehicle image combined with the LiDAR point cloud data can be determined as road surface point cloud data. Therefore, the road surface point cloud data includes the three-dimensional spatial coordinate information of each pixel in the first vehicle image.

[0068] Optional, Figure 6 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 3 ,like Figure 6 As shown, S303 includes:

[0069] S601. Obtain the installation height information of the lidar.

[0070] Based on the scenario example, the installation height information of the LiDAR is the fixed height when the vehicle leaves the factory. It can be the height of the center point of the LiDAR from the ground. The installation height of the LiDAR can be defined as H.

[0071] S602. Based on the installation height information of the lidar and the preset height threshold, filter the lidar point cloud data to obtain the target lidar point cloud data within the preset height range.

[0072] Based on the scenario example, the preset height threshold can be determined according to the actual situation. This height threshold can be defined as δ, and the preset height range can be a height range of H±δ. The lidar point cloud data within this height range of H±δ can be determined as the target lidar point cloud data.

[0073] S603. Based on the first conversion parameters, the target lidar point cloud data is projected onto the first vehicle-mounted image to obtain road surface point cloud data.

[0074] Using a scenario example, the first transformation parameters obtained through joint calibration of LiDAR and vehicle-mounted camera are used to correlate the road area in the first vehicle-mounted image with the target LiDAR point cloud data to obtain road surface point cloud data. This road surface point cloud data can be defined as... Combining Figure 5 , Figure 5 The point cloud data in the example only includes the target LiDAR point cloud data within the height range of H±δ. By projecting the target LiDAR point cloud data within the height range of H±δ onto the first vehicle image for association, the amount of data to be processed can be reduced, thereby improving the processing efficiency of associating road areas with point cloud data.

[0075] S304. Determine the slope change data of the road surface using road surface point cloud data.

[0076] Combined with scenario examples, based on road surface point cloud data Point clouds of roads with different slopes can be identified through 3DHough transform. ,pass By determining the three-dimensional spatial coordinate data corresponding to each point cloud on the road surface, the height information of each pixel in the first vehicle-mounted image is determined. Arranging the height information of each pixel in the first vehicle-mounted image in order from near to far, it can be determined whether the road ahead is uphill or downhill. For example, along the vehicle's direction of travel, if the height of a pixel gradually increases, it is determined to be an uphill section; conversely, if the height of a pixel gradually decreases, it is determined to be a downhill section. Simultaneously, based on the ratio between the height change of different pixels and their distance, the sine value of the slope can be determined, thus obtaining the slope change data corresponding to the road surface.

[0077] S305. Based on slope change data, the road surface contained in the first vehicle image is segmented to obtain at least one road segment, wherein adjacent road segments have different slopes.

[0078] Combined with scenario examples, Figure 7 This is a schematic diagram of the actual road surface as an example, such as... Figure 7 As shown, the road surface displayed in the first vehicle image includes three different slopes, namely α1, α2 and α3. Dividing the road surface according to the slope, we can get three road segments with different slopes. That is, the road segments with slopes of α1 and α3 are downhill road segments, and the road segment with slope of α2 is an uphill road segment. Any two adjacent road segments have different slopes.

[0079] S306. Determine the second transformation parameters corresponding to each road segment based on inverse perspective transformation.

[0080] With a scenario example, inverse perspective transformation refers to the technology of converting the vehicle image captured by the vehicle camera into a BEV perspective image. The second transformation parameter refers to the mapping parameters that map each pixel in the first vehicle image to the BEV perspective image, which is also known as the IPM transformation parameter.

[0081] S307. Using the second conversion parameters for each road segment, the first vehicle-mounted image is converted to obtain the first horizontal road image corresponding to the first vehicle-mounted image.

[0082] In the context of this example, the horizontal road image refers to the aforementioned bird's-eye view, i.e., the BEV perspective image. The first horizontal road image corresponding to the first vehicle-mounted image refers to the BEV perspective image converted from the first vehicle-mounted image. Based on the second conversion parameters, each pixel in the first vehicle-mounted image is converted to obtain the first horizontal road image.

[0083] Based on the method provided in this example, road surface point cloud data can be obtained by combining LiDAR point cloud data and vehicle-mounted images. The road surface slope change data can be determined from the road surface point cloud data. Then, the road surface is divided into road segments according to the slope based on the road surface slope change data. For road segments with different slopes, the transformation parameters adapted to the current slope are dynamically calculated. This solves the limitation of traditional IPM calibration relying on manual annotation. At the same time, it dynamically adapts to road surface scenarios with undulating slopes, improving the geometric accuracy of transforming vehicle-mounted images. Thus, vehicle-mounted images can be transformed into horizontal road images that conform to actual road conditions, so that the horizontal road images can accurately reflect the actual road surface information. Therefore, the accuracy and precision of horizontal road image transformation can be improved.

[0084] Optional, Figure 8 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 4 ,like Figure 8 As shown, S306 includes:

[0085] S801. For each road segment, the road segment is divided to obtain at least one grid corresponding to the road segment.

[0086] In the context of a scenario, if a road segment is too long, directly performing an inverse perspective transformation on the segment may cause data distortion. Therefore, the road segment can be further divided to obtain one or more grids corresponding to that segment. Performing an inverse perspective transformation on the grid unit can improve the accuracy of data processing.

[0087] Optional, Figure 9 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 5 ,like Figure 9 As shown, S801 includes:

[0088] S901. Determine the length of each road segment.

[0089] Example, combination Figure 7 For each road segment, such as a downhill segment with a gradient of α1, the first coordinate of the point cloud furthest from the front of the vehicle is determined based on the road surface point cloud on that segment. and the second coordinate of the nearest point cloud According to the first coordinate Second coordinate Determine the length of this section. Similarly, the lengths of the other two sections can be obtained in the same way. For example, the final length of the downhill section with slope α1 is 30 meters, the length of the uphill section with slope α2 is 22 meters, and the length of the downhill section with slope α3 is 8 meters.

[0090] S902. For each road segment, if the length of the road segment is less than a preset threshold, the road segment is divided according to its length to obtain a grid corresponding to the road segment.

[0091] For example, the preset threshold can be determined according to the actual situation, such as 10 meters. For a downhill section with a slope of α3, if the length of the section is less than or equal to 10 meters, that is... - ≤10 meters, for example, - =8 meters, then the road segment is divided into grids with a length of 8 meters.

[0092] S903. For each road segment, if the length of the road segment is greater than a preset threshold, the road segment is divided at a preset interval to obtain at least one grid corresponding to the road segment; or, the target interval corresponding to the road segment is determined based on the length of the road segment, and the road segment is divided at the target interval corresponding to the road segment to obtain at least one grid corresponding to the road segment.

[0093] For example, the downhill section with a slope of α1 and the uphill section with a slope of α2 are both longer than 10 meters, that is, the downhill section with a slope of α1 and the uphill section with a slope of α2 are both longer than 10 meters. - For sections longer than 10 meters, when using the first method, each section is divided according to a preset interval d. The preset interval d can be determined based on the actual situation, for example, it can be 10 meters. For a downhill section with a slope of α1, dividing the section at a fixed interval of 10 meters will result in three grids. For an uphill section with a slope of α2, dividing the section at a fixed interval of 10 meters will result in two grids, but two meters will remain undivided. Therefore, for sections whose length cannot be divided evenly by the preset interval d, the grid interval can be adjusted adaptively according to the actual length of the section. For example, adjusting the interval to 11 meters will divide the section into two grids. Figure 10 Grid division diagram for example Figure 1 , Figure 10 The example illustrates the final number of grid cells obtained: a downhill section with a slope of α1 is divided into three grid cells, an uphill section with a slope of α2 is divided into two grid cells, and a downhill section with a slope of α3 is divided into one grid cell. Based on the method provided in this example, an appropriate grid division method can be selected according to the actual length of the road segment to ensure that the final grid cells adapt to the actual road segment length.

[0094] S802. Determine the number of road surface point clouds in each grid.

[0095] Based on a scenario example and the principle of four points being coplanar, to transform a road surface into an image from a BE (Browser-Based Image) perspective, the number of non-collinear road surface point clouds included in the road surface must be no less than four. Therefore, the first step is to determine the number of road surface point clouds in each grid cell.

[0096] S803. If the number of non-collinear road surface point clouds in a grid is less than four, the grid is merged with the adjacent grid to obtain the target grid corresponding to the road segment. The target grid includes at least four non-collinear target road surface point clouds.

[0097] Combined with scenario examples, Figure 10 Taking three grid cells on a downhill road segment with a slope of α1 as an example, the number of non-collinear road surface point clouds in the three grid cells are 6, 5, and 3, respectively. Road surface point clouds on grid lines can be simultaneously calculated in the grid cells on either side. Therefore, for a grid cell with 3 road surface point clouds, it needs to be merged with the adjacent grid cells to obtain... Figure 11 , Figure 11 Grid division diagram for example Figure 2 ,like Figure 11 As shown, the three grids of the downhill section with a slope of α1 can be merged into two target grids. Figure 12 Grid division diagram for example Figure 3 ,like Figure 12 As shown, if the number of road surface point clouds in the three grids of the downhill section with slope α1 is still less than four after merging, then they are merged with the grids on the adjacent uphill section with slope α2 until the number of non-collinear road surface point clouds in the grid is not less than four.

[0098] S804. For each target grid in the road segment, project each target road surface point cloud in the target grid onto the horizontal plane to obtain the horizontal projection point corresponding to the target road surface point cloud.

[0099] Combined with scenario examples, Figure 13 A projection diagram of the target road surface point cloud as an example, such as... Figure 13 As shown, taking a downhill road surface with a slope of α1 as an example, the target road surface point cloud in this target grid is defined as... ,Will Projecting this onto a horizontal plane yields point clouds of each target road surface. The horizontal projection point from the BEV perspective is defined as... .

[0100] S805. Based on the first transformation parameters, determine the first target pixel point of the target road surface point cloud on the first vehicle image.

[0101] Based on the scenario example and the first transformation parameters obtained above, the point cloud of each target road surface is determined. The pixels corresponding to each target road surface point cloud in the first vehicle image are used to determine the first target pixels. .

[0102] S806. Determine the first correspondence between the horizontal projection point and the first target pixel.

[0103] Combined with scenario examples, using target road surface point clouds As an intermediary, the target road surface point cloud With horizontal projection point There is a vertical mapping relationship, the first target pixel. From the target road surface point cloud Based on the first transformation parameter, the target road surface point cloud can be used as a basis. With horizontal projection point The vertical mapping relationship between them and the first transformation parameter are used to obtain the first target pixel. With horizontal projection point The first correspondence between them.

[0104] S807. Based on the first correspondence, determine the second transformation parameter corresponding to the target raster.

[0105] In the context of the scenario example, the second transformation parameter corresponding to the target raster refers to the second transformation parameter used to transform the region corresponding to the target raster in the first vehicle-mounted image into a horizontal road image under the BEV perspective. This second transformation parameter is also the first IPM transformation parameter used to transform the first vehicle-mounted image into a horizontal road image under the BEV perspective. The first IPM transformation parameter can be defined as... .

[0106] Based on the method provided in this example, the target road point cloud can be used as a transformation medium to obtain the second transformation parameters corresponding to each target raster, so that the area corresponding to each target raster on the first vehicle image can be transformed into a horizontal road image under the BEV perspective.

[0107] Optional, Figure 14 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 6 ,like Figure 14 As shown, S307 includes:

[0108] S1401. Determine the target image area involved by each target grid in the first vehicle image.

[0109] Using scenario examples, the target road point cloud in the target raster for road surfaces with different slopes is shown. Based on the first target pixel point corresponding to the first vehicle image Calculate the maximum encompassing region corresponding to the target raster on the first vehicle image, that is, the target image region corresponding to the target raster on the first vehicle image. In this way, the area encompassed by road surfaces of different distances and slopes can be determined on the image.

[0110] S1402. Obtain the image features corresponding to each target image region.

[0111] Based on scenario examples, an image backbone network can be used to determine image features, which can be the color features of each pixel in the image.

[0112] S1403. Based on the image features corresponding to the target image region, determine the lane line corresponding to the target image region, as well as multiple second target pixels near the lane line.

[0113] Based on scenario examples, lane line recognition networks can be used to determine the corresponding lane lines in the target image region based on image features, and the pixels near the lane lines can be identified as the second target pixels.

[0114] S1404. Based on the second transformation parameter corresponding to the target raster, transform each pixel in the target image region corresponding to the target raster to obtain the horizontal view image of the region corresponding to the target image region.

[0115] Using a scenario example, and taking the target grid as a unit, the target image region corresponding to each target grid on the first vehicle-mounted image is... Each pixel in the raster is based on the second transformation parameter corresponding to the target raster obtained above. The transformation yields the BEV view image corresponding to each target image region, which is the horizontal view image of the region corresponding to each target image region.

[0116] S1405. Based on the second transformation parameter corresponding to the target grid, project the lane line corresponding to the target image region corresponding to the target grid onto the horizontal view image of the region corresponding to the target image region to obtain the horizontal view lane line corresponding to the horizontal view image of the region.

[0117] Similarly, for the lane line portion corresponding to each target grid, the second transformation parameter of the target grid is also used for transformation to obtain the corresponding horizontal view lane line in the BEV view image.

[0118] S1406. The horizontal view images of each region are stitched together, and the horizontal view lane lines corresponding to the horizontal view lane lines of each region are stitched together according to the coordinate data corresponding to the horizontal view lane lines, so as to obtain the first horizontal road image of the first vehicle image in the horizontal view.

[0119] Based on the scenario example, the horizontal view images of the regions corresponding to each target grid are stitched together sequentially according to the order of the target grids. At the same time, referring to the relative positions of the horizontal view lane lines in the horizontal view images of each region, the corresponding horizontal view lane lines are stitched together to obtain a complete first horizontal road image of the first vehicle image from a horizontal view. The first horizontal road image includes the stitched lane lines from the BEV view. Figure 15 The lane lines from the perspective of a BEV are shown in the example. Figure 15 It can be seen that the lane lines in the BEV perspective are obtained by converting the lane lines on the road surface with different slopes into BEV perspective and stitching them together.

[0120] Based on the method provided in this example, the horizontal view image of the region can be transformed in units of grids according to the second transformation parameters corresponding to each target grid, and then stitched together. The lane lines involved in each target image region can be identified, the lane lines can be transformed horizontally, and the images can be stitched together according to the coordinate data of the lane lines. This can ensure the accuracy of lane line stitching and ensure that the obtained first horizontal road image can reflect the real road surface conditions.

[0121] Optionally, it also includes: determining the coordinate data corresponding to the lane lines from a horizontal perspective.

[0122] Based on the scenario example, and using the second transformation parameter The IPM transformation is only a transformation between different image pixels and does not have dimensions. In other words, the transformed BEV view image does not have the real spatial coordinates of the spatial scene. Therefore, it is necessary to assign real spatial coordinates to the pixels of the lane lines in the horizontal view. Figure 16 The example is a schematic diagram showing how lane lines are determined based on coordinate data fitting. Figure 16 As shown, the point cloud spatial coordinates corresponding to the lane lines can be defined as follows: Determine using the first conversion parameter The coordinates of the corresponding projected pixel on the first vehicle image are By setting a distance threshold Determine the distance from the lane line Coordinates of each pixel within the range Based on the second conversion parameters obtained above It can be used to measure the distance from the lane line. Coordinates of each pixel within the range The transformation is performed to obtain the coordinates of the target horizontal projection points corresponding to each pixel in the BEV view. The coordinates of each target horizontal projection point By fitting the data, the lane lines under the BEV perspective are obtained. Since the lane lines are obtained by fitting the coordinates of each target horizontal projection point, the coordinate data of each point on the lane line under the horizontal perspective can be obtained indirectly.

[0123] Optional, combined Figure 16 The coordinate data corresponding to the horizontal view lane line can also be determined based on the first transformation parameters obtained above. This allows for the determination of the target point cloud corresponding to each second target pixel. Based on the aforementioned LiDAR point cloud data, the spatial coordinate data corresponding to each target point cloud can be obtained. Each target point cloud is then vertically projected onto a horizontal plane to obtain the target horizontal projection point. By setting the vertical coordinates of the target point cloud to zero, the planar coordinate data corresponding to each target horizontal projection point can be obtained. By fitting each target horizontal projection point, the relative position of the horizontal view lane line in the first horizontal road image is obtained, thus yielding the coordinate data corresponding to the horizontal view lane line. The coordinate data corresponding to the horizontal view lane line can characterize the relative position of the horizontal view lane line in the first horizontal road image.

[0124] Optional, Figure 17 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 7 ,like Figure 17 As shown, it also includes:

[0125] S1701. For each target grid, determine the second correspondence between the target road surface point cloud in the target grid and the corresponding horizontal projection point.

[0126] Combined with scenario examples, Figure 13 The mapping relationship between each target road surface point cloud in the target grid and the corresponding horizontal projection point under the BEV view is determined, and the target road surface point cloud is obtained sequentially. With the corresponding horizontal projection point The second correspondence between them.

[0127] S1702. Determine the third transformation parameter corresponding to the target raster based on the second correspondence.

[0128] Using a scenario example, the third transformation parameter refers to the transformation parameter that converts the corresponding horizontal projection points from the BEV's perspective into a 3D road image, based on... and The correspondence is used to calculate the IPM transformation parameters from the BEV viewpoint road surface area to the actual three-dimensional road surface, which is the third transformation parameter. The third transformation parameter can be defined as follows: .

[0129] S1703. Based on the third transformation parameter corresponding to the target raster, the horizontal view image of the area corresponding to the target raster is transformed into a three-dimensional road image of the area.

[0130] Combined with scenario examples, Figure 18 A schematic diagram illustrating the transformation of a 3D road image, as shown below. Figure 18As shown, based on the second transformation parameter After converting the target image region corresponding to each target raster on the first vehicle image into a regional horizontal road image, the third transformation parameter corresponding to the target raster can be used as the basis for the transformation. The corresponding horizontal road image of the region is then converted into a three-dimensional road image of the region.

[0131] S1704. Determine the slope, first elevation coordinate, and second elevation coordinate corresponding to each target grid.

[0132] Using scenario examples, for road sections with different gradients, the gradient of each section can be obtained, and the gradient of each road surface can be defined as... Determine the first and second elevation coordinates for each slope section. The first elevation coordinate refers to the lowest coordinate of the 3D road image of that area. The second elevation coordinate refers to the highest coordinate of the 3D road image of that area. .

[0133] S1705. Based on the slope, first height coordinate and second height coordinate corresponding to each target grid, the horizontal view lane lines of each target grid in the corresponding area horizontal view image are transformed into three-dimensional lane lines.

[0134] Based on a scenario example, the generation of the corresponding 3D lane lines in a regional 3D road image can be achieved by determining the angle of the 3D lane lines based on the slope of the regional 3D road image, and determining the upper and lower positions of the 3D lane lines based on the first and second height coordinates, thereby converting the horizontal view lane lines into 3D lane lines.

[0135] S1706. The stereoscopic road images and stereoscopic lane lines of each region are stitched together to obtain the first stereoscopic road image corresponding to the first vehicle-mounted image.

[0136] Combined with scenario examples, Figure 19 A schematic diagram of a 3D road image for example, such as Figure 19 As shown, the stereo road images of each region and the corresponding stereo lane lines are stitched together according to the first height coordinate and the second height coordinate to obtain the first stereo road image corresponding to the first vehicle image. The first stereo road image contains stereo lane lines.

[0137] Based on the method provided in this example, a horizontal bird's-eye view can be transformed into a three-dimensional road image. The three-dimensional road image can more intuitively reflect the actual road conditions and improve the safety of autonomous driving.

[0138] Optional, Figure 20 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 8The multi-source sensor data also includes a second vehicle-mounted image. The first vehicle-mounted image was captured by a first vehicle-mounted camera, and the second vehicle-mounted image was captured by a second vehicle-mounted camera. The focal length of the first vehicle-mounted camera is shorter than that of the second vehicle-mounted camera. Accordingly, as... Figure 20 As shown, it also includes:

[0139] S2001. Determine the second horizontal road image of the second vehicle-mounted image from a horizontal perspective, and the corresponding second stereo road image.

[0140] Combined with scenario examples, Figure 21 A schematic diagram of an example vehicle-mounted camera, such as... Figure 21 As shown, the vehicle-mounted camera includes multiple cameras with different focal lengths. Figure 21 The example illustrates three cameras with different focal lengths. A camera with a shooting range of 52° or 28° can be identified as the first vehicle-mounted camera; the first vehicle-mounted image captured by the first camera is taken from a relatively short distance. A camera with a shooting range of 150° can be identified as the second vehicle-mounted camera; the second vehicle-mounted image captured by the second camera is taken from a longer distance. Following the same method as processing the first vehicle-mounted image, a second horizontal road image and a second stereoscopic road image corresponding to the second vehicle-mounted image are obtained.

[0141] S2002. Determine the pixel correspondence between the overlapping areas of the first vehicle image and the second vehicle image.

[0142] Based on the scenario example, there is an overlapping area between the first vehicle image and the second vehicle image. The correspondence between pixels in this area is determined.

[0143] S2003. Based on the pixel correspondence, the first horizontal road image and the second horizontal road image are stitched together, and the first three-dimensional road image and the second three-dimensional road image are stitched together.

[0144] Based on a scenario example, the first and second horizontal road images are stitched together according to the correspondence between pixels in the repeating areas to achieve a transition from the near first horizontal road image to the distant second horizontal road image. Simultaneously, the first and second stereoscopic road images are stitched together to obtain a transition from the near first stereoscopic road image to the distant second stereoscopic road image, thereby expanding the captured road surface range and allowing for advance prediction of road conditions in the distance.

[0145] Optional, Figure 22 A flowchart illustrating the road image processing method based on inverse perspective transformation provided in this application embodiment. Figure 9 ,like Figure 22 As shown, S2001 includes:

[0146] S2201. Obtain the first focal length corresponding to the first vehicle-mounted camera, and obtain the second focal length corresponding to the second vehicle-mounted camera.

[0147] Based on the scenario example and the parameters of the vehicle-mounted camera, determine the first focal length and the second focal length.

[0148] S2202. Determine the preset fourth conversion parameter between the first focal length and the second focal length.

[0149] Based on the scenario example, determine the fourth conversion parameter that corresponds from the second focal length to the first focal length, according to the first focal length and the second focal length.

[0150] S2203. Based on the fourth conversion parameter, the pixels on the second vehicle image are projected onto the first vehicle image to obtain a third correspondence between the pixels of the first vehicle image and the pixels of the second vehicle image.

[0151] Based on the scenario example and according to the fourth conversion parameter, it is determined that the pixels on the second vehicle image will be projected onto the first vehicle image, and the pixels on the second vehicle image will be mapped to the pixels on the first vehicle image to obtain the third mapping relationship.

[0152] S2204. Based on the third correspondence, determine the second horizontal road image of the second vehicle image from a horizontal perspective, and the corresponding second stereo road image.

[0153] Based on the scenario example and the aforementioned content, the pixels of the first vehicle-mounted image can be converted to obtain the corresponding first horizontal road image and first three-dimensional road image. Now, according to the third correspondence, the correspondence between the pixels of the first vehicle-mounted image and the pixels of the second vehicle-mounted image can be known. Therefore, according to the third correspondence, the pixel information corresponding to the pixels of the second vehicle-mounted image on the first vehicle-mounted image can be determined. Then, according to the same method as described above, the second horizontal road image and the second three-dimensional road image corresponding to the pixels of the second vehicle-mounted image can be obtained.

[0154] Based on the method provided in this example, a more distant vehicle image can be projected as a closer vehicle image before determining the corresponding horizontal road image and stereo road image. This can reduce distance perception bias and improve the accuracy of the horizontal road image and stereo road image corresponding to the actual road at a distance.

[0155] Figure 23 This is a schematic diagram of the structure of the road image processing device based on inverse perspective transformation provided in the embodiments of this application, as shown below. Figure 23 As shown, it includes:

[0156] The acquisition module 231 is used to acquire multi-source sensor data, wherein the multi-source sensor data includes at least lidar point cloud data and a first vehicle image;

[0157] Processing module 232 is used to determine the first conversion parameters between the lidar point cloud data and the first vehicle image;

[0158] The processing module 232 is also used to associate the lidar point cloud data with the first vehicle image based on the first conversion parameters to obtain road surface point cloud data;

[0159] The processing module 232 is also used to determine the actual road condition information through road surface point cloud data;

[0160] The processing module 232 is also used to determine the corresponding second transformation parameters based on the inverse perspective transformation, with reference to the actual road condition information;

[0161] The processing module 232 is further configured to use the second conversion parameters to convert the first vehicle image to obtain the first horizontal road image corresponding to the first vehicle image.

[0162] The description of the features in the embodiment of the road image processing device based on inverse perspective transformation provided in this embodiment can be found in the relevant description of the embodiment of the road image processing method based on inverse perspective transformation, and will not be repeated here.

[0163] Figure 24 A schematic diagram of the structure of the electronic device provided in this application. Figure 24 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus.

[0164] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-described embodiment of the road image processing method based on inverse perspective transformation.

[0165] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0166] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0167] The memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk storage device.

[0168] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0169] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the road image processing method embodiments based on inverse perspective transformation described above.

[0170] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0171] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the road image processing method based on inverse perspective transformation.

[0172] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the road image processing method based on inverse perspective transformation.

[0173] It should be noted that the division of units is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0174] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0176] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0178] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0179] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation.

[0180] The foregoing has provided a detailed description of a road image processing method and electronic device based on inverse perspective transformation provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A road image processing method based on inverse perspective transformation, characterized in that, include: Acquire multi-source sensor data, wherein the multi-source sensor data includes at least lidar point cloud data and a first vehicle-mounted image; Determine the first conversion parameter between the lidar point cloud data and the first vehicle image; Based on the first conversion parameters, the lidar point cloud data is associated with the first vehicle image to obtain road surface point cloud data; The slope variation data of the road surface is determined using the road surface point cloud data. Based on the slope change data, the road surface contained in the first vehicle image is segmented to obtain at least one road segment, wherein adjacent road segments have different slopes. The second transformation parameters corresponding to each road segment are determined based on inverse perspective transformation; The first vehicle image is converted using the second conversion parameters of each road segment to obtain the first horizontal road image corresponding to the first vehicle image.

2. The method according to claim 1, characterized in that, The determination of the first conversion parameter between the lidar point cloud data and the first vehicle-mounted image includes: Based on preset spatial corresponding points, establish the lidar coordinate system and the camera coordinate system; The mapping relationship between the established lidar coordinate system and the camera coordinate system is determined to obtain the first transformation parameters.

3. The method according to claim 1, characterized in that, Based on the first conversion parameters, the lidar point cloud data is correlated with the first vehicle-mounted image to obtain road surface point cloud data, including: Obtain the installation height information of the lidar; Based on the installation height information of the lidar and the preset height threshold, the lidar point cloud data is filtered to obtain target lidar point cloud data within the preset height range. Based on the first conversion parameters, the target lidar point cloud data is projected onto the first vehicle-mounted image to obtain the road surface point cloud data.

4. The method according to claim 1, characterized in that, The determination of the second transformation parameters corresponding to each road segment based on inverse perspective transformation includes: For each road segment, the road segment is divided to obtain at least one grid corresponding to the road segment; Determine the number of road surface point clouds in each grid cell; If the number of non-collinear road surface point clouds in the grid is less than four, the grid is merged with the adjacent grid to obtain the target grid corresponding to the road segment, wherein the target grid includes at least four non-collinear target road surface point clouds. For each target grid in the road segment, the target road surface point cloud in the target grid is projected onto the horizontal plane to obtain the horizontal projection point corresponding to the target road surface point cloud; Based on the first conversion parameters, the first target pixel of the target road surface point cloud on the first vehicle image is determined; Determine the first correspondence between the horizontal projection point and the first target pixel; Based on the first correspondence, the second transformation parameter corresponding to the target raster is determined.

5. The method according to claim 4, characterized in that, The step of dividing each road segment to obtain at least one grid corresponding to each road segment includes: Determine the length of each road segment; For each road segment, if the length of the road segment is less than a preset threshold, the road segment is divided according to its length to obtain a grid corresponding to the road segment. For each road segment, if the length of the road segment is greater than or equal to a preset threshold, the road segment is divided at a preset interval to obtain at least one grid corresponding to the road segment; or, the target interval corresponding to the road segment is determined based on the length of the road segment, and the road segment is divided at the target interval corresponding to the road segment to obtain at least one grid corresponding to the road segment.

6. The method according to claim 4, characterized in that, The step of converting the first vehicle-mounted image using the second conversion parameters of each road segment to obtain the first horizontal road image corresponding to the first vehicle-mounted image includes: Determine the target image region involved by each target grid in the first vehicle-mounted image; Obtain the image features corresponding to each target image region; Based on the image features corresponding to the target image region, determine the lane line corresponding to the target image region, and a plurality of second target pixels near the lane line; Based on the second conversion parameter corresponding to the target grid, each pixel in the target image region corresponding to the target grid is converted to obtain the horizontal view image of the region corresponding to the target image region; Based on the second transformation parameter corresponding to the target grid, the lane line corresponding to the target image region corresponding to the target grid is projected onto the horizontal view image of the region corresponding to the target image region to obtain the horizontal view lane line corresponding to the horizontal view image of the region. Determine the coordinate data corresponding to the lane lines from the horizontal perspective; The horizontal view images of each region are stitched together, and the horizontal view lane lines corresponding to the horizontal view lane lines of each region are stitched together according to the coordinate data corresponding to the horizontal view lane lines, so as to obtain the first horizontal road image of the first vehicle image under the horizontal view.

7. The method according to claim 6, characterized in that, Also includes: For each target grid, a second correspondence is determined between the target road surface point cloud in the target grid and the corresponding horizontal projection point; The third transformation parameter corresponding to the target raster is determined based on the second correspondence; Based on the third transformation parameter corresponding to the target grid, the horizontal view image of the region corresponding to the target grid is transformed into a regional stereo road image; Determine the slope, first elevation coordinate, and second elevation coordinate corresponding to each target grid; Based on the slope, first height coordinate and second height coordinate corresponding to each target grid, the horizontal view lane line of each target grid in the corresponding area horizontal view image is transformed into a three-dimensional lane line. The stereoscopic road images and stereoscopic lane lines of each region are stitched together to obtain the first stereoscopic road image corresponding to the first vehicle-mounted image.

8. The method according to claim 7, characterized in that, The multi-source sensor data also includes a second vehicle-mounted image, wherein the first vehicle-mounted image is captured by a first vehicle-mounted camera, the second vehicle-mounted image is captured by a second vehicle-mounted camera, and the focal length of the first vehicle-mounted camera is smaller than the focal length of the second vehicle-mounted camera. Correspondingly, it also includes: Determine the second horizontal road image of the second vehicle-mounted image from the horizontal perspective, and the corresponding second stereo road image; Determine the pixel correspondence between the overlapping areas of the first vehicle image and the second vehicle image; Based on the pixel correspondence, the first horizontal road image and the second horizontal road image are stitched together, and the first three-dimensional road image and the second three-dimensional road image are stitched together.

9. The method according to claim 8, characterized in that, Determining the second vehicle-mounted image as a second horizontal road image in the horizontal view, and the corresponding second stereo road image, includes: Obtain the first focal length corresponding to the first vehicle-mounted camera, and obtain the second focal length corresponding to the second vehicle-mounted camera; Determine a preset fourth conversion parameter between the first focal length and the second focal length; Based on the fourth conversion parameter, the pixels on the second vehicle image are projected onto the first vehicle image to obtain a third correspondence between the pixels of the first vehicle image and the pixels of the second vehicle image. Based on the third correspondence, the second vehicle image is determined as a second horizontal road image under the horizontal viewpoint, and the corresponding second stereo road image is determined.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the road image processing method based on inverse perspective transformation as described in any one of claims 1 to 9.

Images