An off-road road slope prediction method and device based on multi-modal data fusion

Abstract

The application provides a cross-country road slope prediction method and device based on multi-modal data fusion, wherein the method comprises: segmenting an RGB image by using an image segmentation model to obtain a binary mask image of a road region; projecting point cloud data to an RGB image coordinate system to obtain a point cloud image, and calculating a spatial feature vector of each pixel of the point cloud image; determining an effective area in a driving direction of the ego vehicle based on the binary mask image and the spatial feature vector of each pixel; determining a mean value in the driving direction of the ego vehicle and a height mean value of each row of the point cloud image based on the effective area in the driving direction of the ego vehicle; fitting a height mean value curve with the mean value in the driving direction of the ego vehicle as an independent variable based on the mean value in the driving direction of the ego vehicle and the height mean value of each row; uniformly sampling the height mean value curve to obtain a plurality of sampling points; and calculating a slope value in the driving direction of the ego vehicle based on two adjacent sampling points. The application improves the accuracy and error stability of slope calculation in a complex cross-country scene.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for predicting the slope of off-road roads based on multimodal data fusion. Background Technology

[0002] Current road slope prediction methods are mostly designed for roads with regular surfaces and use traditional accelerometers for slope calculation. They lack multimodal information fusion strategies to adequately address the real-time, accurate perception and prediction needs in complex environments. Practical applications present several challenges: off-road environments feature diverse road surface types, abrupt slope changes, and numerous natural obstacles, placing extremely high demands on slope perception tasks. Traditional single-sensor perception schemes are prone to failure in scenarios with sandstorms or sudden changes in lighting; while slope estimation methods based on pre-stored maps cannot handle the "mapless, unstructured" terrain features of off-road environments. These issues result in perception models exhibiting low prediction accuracy and limited recognition capabilities when facing complex dynamic scenes, failing to meet practical needs.

[0003] Therefore, how to sense the surface condition and identify slope changes in real time and accurately in complex off-road environments has become a challenge for current off-road intelligent equipment technology. Summary of the Invention

[0004] In view of this, this application provides a method and apparatus for predicting the slope of off-road roads based on multimodal data fusion, in order to solve the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a method for predicting the slope of off-road terrain based on multimodal data fusion, including: Acquire RGB images and point cloud data of the vehicle at the current moment while it is on an off-road road; The RGB image is segmented using a pre-trained image segmentation model to obtain a binary mask image of the road region; The point cloud data is projected onto the RGB image coordinate system to obtain a point cloud image, and the spatial feature vector of each pixel in the point cloud image is calculated. Based on the binary mask image of the road area and the spatial feature vector of each pixel in the point cloud image, the effective area for the vehicle's driving direction is determined; based on the effective area for the vehicle's driving direction, the mean value of the vehicle's driving direction and the mean value of the height of each row in the point cloud image are determined. Based on the mean values ​​of the vehicle's driving direction and the mean values ​​of the height in each row, a height mean curve with the mean value of the vehicle's driving direction as the independent variable is obtained by fitting. Uniform sampling is performed on the height mean curve to obtain multiple sampling points; based on two adjacent sampling points, the slope value in the vehicle's driving direction is calculated.

[0006] In one possible implementation, the binary mask image of the road region is represented as: When the pixels of the binary mask image If it is a drivable area, then the pixel mask value ;otherwise, .

[0007] In one possible implementation, the point cloud data is projected onto an RGB image coordinate system to obtain a point cloud image, and the spatial feature vector of each pixel in the point cloud image is calculated; including: Homogeneous processing is performed on each point in the point cloud data to obtain multiple homogeneous points; all homogeneous points are stacked column-wise to obtain a matrix. : Obtain the homogeneous extrinsic parameter matrix from the radar coordinate system to the camera coordinate system. : in, Let be the rotation matrix from the radar coordinate system to the camera coordinate system. This is the translation vector from the radar coordinate system to the camera coordinate system; Using homogeneous extrinsic matrices For matrix Perform extrinsic parameter transformation to obtain the matrix in the camera coordinate system. : Using the correction matrix For matrix Apply correction to obtain the matrix : Using projection matrix matrix Projecting onto the RGB image coordinate system yields the matrix. : Among them, matrix The i-th column These represent the homogeneous components of the projection point in the RGB coordinate system; Calculate the pixel coordinates of the i-th point : Two-dimensional pixel coordinates Encoded as a one-dimensional index : in, The width of the point cloud image; For the same one-dimensional index From multiple points, only the point with the lowest depth is retained, thus obtaining a point cloud image; Set the pixels of the point cloud image The spatial feature vector is ,in, For pixels The x-component of the projection point in the radar coordinate system; For pixels The y-component of the projection point in the radar coordinate system; For pixels The z-component of the projection point in the radar coordinate system.

[0008] In one possible implementation, the effective area for the vehicle's travel direction is determined based on a binary mask image of the road region and the spatial features of each pixel; including: Effective area in the direction of vehicle travel for: From the effective area Extract the height values ​​of all pixels and determine the minimum height value from them. ; For the effective area Height value of spatial feature vector Normalization is performed to obtain the normalized height value. : .

[0009] In one possible implementation, based on the effective area of ​​the vehicle's driving direction, the mean value of the vehicle's driving direction and the mean value of its height for each row of the point cloud image are determined; including: Obtain the set of valid points for each row of the point cloud image. : Calculate the mean value of the vehicle's direction of travel for each row. : in, Indicates the number of valid points; Calculate the average height of each row : .

[0010] In one possible implementation, based on the mean of the vehicle's travel direction and the mean of its height in each row, a height mean curve is fitted with the mean of the vehicle's travel direction as the independent variable; including: Set the height mean curve with the mean of the vehicle's direction of travel as the independent variable as follows: in, This is the average value in the direction of the vehicle's travel. The average height All are coefficients; Based on the mean values ​​of the vehicle's driving direction and height for each row, coefficients are fitted to obtain the coefficients. .

[0011] In one possible implementation, the method further includes: Calculate the slope angle corresponding to each slope value to obtain a slope angle sequence; Starting from the k-th slope angle in the slope angle sequence, construct a window of a preset length. ; Window The mean of the statistic is determined as the k-th candidate statistic. Sort the candidate statistics from largest to smallest to obtain the candidate statistics sequence; Select a predetermined number of candidate statistics from the candidate statistic sequence, and calculate the mean of the predetermined number of candidate statistics as the slope angle estimate. .

[0012] In one possible implementation, the method further includes: The pitch angle at the current moment is obtained from the IMU on the vehicle. ; The estimated slope angle is compensated to obtain the compensated slope angle. : .

[0013] Secondly, embodiments of this application provide an off-road road slope prediction device based on multimodal data fusion, comprising: The acquisition unit is used to acquire RGB images and point cloud data collected by the vehicle on the off-road road at the current moment; The image segmentation unit is used to segment the RGB image using a pre-trained image segmentation model to obtain a binary mask image of the road region; The first calculation unit is used to project the point cloud data onto the RGB image coordinate system to obtain a point cloud image, and to calculate the spatial feature vector of each pixel in the point cloud image. The determination unit is used to determine the effective area of ​​the vehicle's driving direction based on the spatial feature vector of each pixel in the binary mask image of the road area and the point cloud image; and based on the effective area of ​​the vehicle's driving direction, to determine the mean value of the vehicle's driving direction and the mean value of the height of each row of the point cloud image. The fitting unit is used to fit a height mean curve with the mean of the vehicle's driving direction as the independent variable, based on the mean of the vehicle's driving direction and the mean of the height in each row. The second calculation unit is used to perform uniform sampling on the height mean curve to obtain multiple sampling points; based on two adjacent sampling points, the slope value of the vehicle's driving direction is calculated.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods of embodiments of this application.

[0016] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method of embodiments of this application.

[0017] This application improves the accuracy of slope calculation and the stability of errors in complex off-road scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the off-road road slope prediction method based on multimodal data fusion provided in this application embodiment; Figure 2 A functional structure diagram of the off-road road slope prediction device based on multimodal data fusion provided in the embodiments of this application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0023] This application proposes a multinomial slope fitting method based on multimodal information fusion. It utilizes the point cloud height information in the segmented road area, and then integrates IMU attitude data to compensate for errors and a sliding window to calculate the slope trend of the entire slope, thereby achieving the ability to accurately and quickly calculate slope data within the slope area.

[0024] The advantages of this application are: 1. By constructing a mapping relationship from point cloud to image pixels based on a calibration matrix, the LiDAR point cloud information is aligned with the camera image at the pixel level. A nearest depth point preservation mechanism is introduced at the pixel level, effectively resolving multi-point projection conflicts and occlusion interference issues. By outputting a fixed-size pixel 3D attribute mapping table, subsequent geometric calculations can directly obtain the corresponding spatial distance and height information within the image domain, avoiding repetitive point cloud projection calculations and improving computational efficiency and system real-time performance. Simultaneously, this mapping method enhances the spatial consistency between the point cloud and the image, providing a stable and reliable 3D data foundation for subsequent geometric analysis based on road semantic regions.

[0025] 2. Based on the point cloud-image mapping results, a road semantic mask constraint is introduced to extract only valid 3D points within the road area. By aggregating points along the image row direction, the 2D sparse height distribution is reduced to a 1D height variation curve, thus avoiding the instability caused by directly fitting the sparse point cloud to a plane. By smoothing and differentiating the aggregated height curve, a continuous slope sequence is obtained, effectively reducing the impact of outliers, local noise, and point cloud sparsity on the slope estimation results. This method can stably reflect the overall slope trend of the road in complex road environments, improving the robustness and reliability of slope estimation.

[0026] 3. The slope sequence is further angled and subjected to robust statistical processing. By using sliding window averaging, significant window filtering, and fusion calculation, the problem of slope results being diluted by local noise or large flat areas is avoided, making the final output more representative of the real slope characteristics. Simultaneously, vehicle IMU data pitch angle priors are introduced into the longitudinal slope estimation process for compensation, effectively eliminating the influence of vehicle attitude changes on the slope results. This post-processing strategy significantly improves the stability, consistency, and engineering applicability of the slope output in dynamic driving scenarios, making the estimation results closer to the true geometric slope of the road. After multiple tests, this road slope estimation algorithm demonstrates high accuracy and stable error in calculating different slopes in complex off-road scenarios, providing reliable slope data support for off-road vehicle power control and safe driving.

[0027] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0028] like Figure 1 As shown in the figure, this application provides a method for predicting the slope of off-road roads based on multimodal data fusion, including: Step 101: Acquire the RGB image and point cloud data of the vehicle at the current moment while it is on the off-road road; Step 102: Segment the RGB image using a pre-trained image segmentation model to obtain a binary mask image of the road region; Step 103: Project the point cloud data onto the RGB image coordinate system to obtain a point cloud image, and calculate the spatial feature vector of each pixel in the point cloud image; Step 104: Based on the binary mask image of the road area and the spatial feature vector of each pixel in the point cloud image, determine the effective area of ​​the vehicle's driving direction; based on the effective area of ​​the vehicle's driving direction, determine the mean value of the vehicle's driving direction and the mean value of the height of each row in the point cloud image. Step 105: Based on the mean value of the vehicle's driving direction and the mean value of the height in each row, fit the mean value of the height with the mean value of the vehicle's driving direction as the independent variable. Step 106: Perform uniform sampling on the height mean curve to obtain multiple sampling points; calculate the slope value of the vehicle's driving direction based on two adjacent sampling points.

[0029] This embodiment utilizes the image detection and segmentation capabilities of the BiSeNetv2 image segmentation algorithm to obtain a road segmentation mask from the input image as the subsequent slope calculation region. Then, it integrates image data, point cloud data, and the road segmentation mask, overcoming the limitations of a single sensor. Through processes such as heightmap transformation and filtering / denoising, and with the aid of a sliding window and polynomial fitting, it outputs the local slope values ​​of the drivable area. By leveraging the complementary advantages of different sensors, it overcomes interference from off-road environments such as sand and bumps, constructing a stable and reliable slope estimate.

[0030] The method in this embodiment improves the accuracy of slope calculation and the stability of errors in complex off-road scenarios, providing reliable slope data support for off-road vehicle power control and safe driving.

[0031] In some embodiments, typical off-road road images of multiple categories are collected as training data. These typical off-road road categories include unstructured off-road surface types such as gravel roads, grasslands, and dirt roads, and cover image samples under off-road scenarios such as sandstorm obstruction, sudden changes in lighting, and different slope angles, to ensure the comprehensiveness and representativeness of the training data. The collected image samples are preprocessed and labeled, with the labels indicating the boundaries between road and non-road areas for each category. The preprocessed image samples are then input into the BiSeNetv2 model for training. The BiSeNetv2 model structure diagram is shown below. Figure 1 As shown.

[0032] After training, the off-road road images to be detected are input into the optimized BiSeNetv2 model. The model accurately segments different types of road surfaces in the images, as follows: Figure 2 As shown, the model's input is the original image, and the target is a binary mask image of the road region in the output segmentation result. In the mask image, regions with a pixel value of 1 correspond to drivable road surfaces, and regions with a pixel value of 0 correspond to non-road surfaces.

[0033] In some embodiments, the binary mask image of the road region is represented as: When the pixels of the binary mask image If it is a drivable road area, then the pixel mask value ;otherwise, .

[0034] In some embodiments, the point cloud data is projected onto an RGB image coordinate system to obtain a point cloud image, and the spatial feature vector of each pixel in the point cloud image is calculated; including: Homogeneous processing is performed on each point in the point cloud data to obtain multiple homogeneous points; all homogeneous points are stacked column-wise to obtain a matrix. : Obtain the homogeneous extrinsic parameter matrix from the radar coordinate system to the camera coordinate system. : in, Let be the rotation matrix from the radar coordinate system to the camera coordinate system. This is the translation vector from the radar coordinate system to the camera coordinate system; Using homogeneous extrinsic matrices For matrix Perform extrinsic parameter transformation to obtain the matrix in the camera coordinate system. : Using the correction matrix For matrix Apply correction to obtain the matrix : Using projection matrix matrix Projecting onto the RGB image coordinate system yields the matrix. : Among them, matrix The i-th column These represent the homogeneous components of the projection point in the RGB coordinate system; Calculate the pixel coordinates of the i-th point : Two-dimensional pixel coordinates Encoded as a one-dimensional index : in, The width of the point cloud image; For the same one-dimensional index From multiple points, only the point with the lowest depth is retained, thus obtaining a point cloud image; Set the pixels of the point cloud image The spatial feature vector is ,in, For pixels The x-component of the projection point in the radar coordinate system; For pixels The y-component of the projection point in the radar coordinate system; For pixels The z-component of the projection point in the radar coordinate system.

[0035] The specific calculation process is as follows: Analyzing the calibration matrix calib file: The calib file contains calibration data from sensors such as cameras, radars, and inertial navigation systems. Each line in the file represents a parameter name, and the parameter matrix follows a colon, as shown below: Camera projection matrix for: Where i = 0, 1, 2, 3; i represents the camera number, 0 indicates the left grayscale camera, 1 indicates the right grayscale camera, 2 indicates the left color camera, and 3 indicates the right color camera. This represents a reference value (in meters) relative to reference camera 0. This embodiment only uses P2.

[0036] In actual calculations, it is necessary to... The matrix is ​​expanded to The matrix: add a vector of all zeros in the fourth row and fourth column, and set the index of (4,4) to 1.

[0037] This is the rotation and translation matrix from radar to camera: In actual calculations, this is achieved by adding a fourth row vector [0,0,0,1]. The matrix is ​​expanded to The matrix. In fact, this matrix contains two parts: rotation matrix and The translation vector is given by the following formula: The above matrix completes the projection link of "radar coordinates → camera coordinates → pixel coordinates".

[0038] Organize the discrete point set into a matrix form and perform homogeneous expansion so that the coordinate transformation and projection of all points can be completed at once using matrix multiplication, thereby improving efficiency and consistency.

[0039] First, construct homogeneous points: For point Construct homogeneous coordinates : The constant 1 represents the homogeneous coordinate extension term, which enables translation to be uniformly expressed through matrix multiplication.

[0040] Stack all homogeneous points column-wise to obtain a matrix. : This organization method allows a subsequent matrix chain to be computed in parallel on all points without needing to iterate point by point.

[0041] Perform extrinsic transformation from radar to camera coordinates. To project the point cloud onto the image plane, the points must first be transformed from the radar coordinate system to the camera coordinate system, because both the projection matrix and the camera model are referenced to the camera coordinate system. This embodiment uses the homogeneous extrinsic matrix provided in the calibration file: in, Represents the rotation matrix. This represents the translation vector. Then, an extrinsic transformation is performed on all points: This yields the point set in the camera coordinate system. If no extrinsic parameter transformation is performed, the point will remain in the radar coordinate system, and directly using the camera projection matrix will result in severe misprojection.

[0042] In many automotive camera systems or KITTI-style data, the projection matrix is ​​often not directly oriented towards the original camera coordinate system, but rather towards the rectified camera coordinate system. Therefore, this embodiment introduces a correction matrix: And extend it to homogeneous form: Then, apply correction to the camera coordinate point set: This aligns the point coordinates with the "camera model of the projection matrix," ensuring correct subsequent projection.

[0043] After aligning the coordinate systems, use the projection matrix. Homogeneous coordinate representation of a 3D homogeneous point projected onto the image plane: Each column These represent the homogeneous components of the pixel coordinates.

[0044] Homogeneous normalization yields continuous pixel coordinates, and depth measurements are extracted for subsequent constraints. Normalization of the homogeneous coordinates then yields the actual continuous pixel coordinates: Because after normalization and It only describes the pixel location, but cannot reflect the spatial relationships between multiple points at that pixel. Therefore, it is retained. This serves as a key quantity for subsequent validity screening and occlusion conflict resolution. Depth constraints can quickly eliminate points located behind the camera, preventing back projection.

[0045] Invalid points are removed from the projection results, and valid points are converted into discrete pixel indices to ensure that the points written into the mapping table satisfy geometric meaning and image boundary constraints. The image boundary constraints are obtained by projecting... Treated as continuous pixel coordinates, they must fall within the valid area of ​​the image: Only points that satisfy the boundary constraints are retained. This ensures that subsequent writes will result in valid indexes and will not exceed the limits. Simultaneously, a positive depth constraint is applied: requiring... Ensure the point is in front of the camera. If the depth is negative, it means the point is behind the camera or the projection chain produces a reversed result, which lacks practical imaging significance and should be discarded. The width of the RGB image. The height of the RGB image.

[0046] Due to the density of point clouds, occlusion, and discretization of projection, multiple 3D points may map to the same pixel. Direct overwrite would cause distant points to overwrite nearby points or randomly overwrite, thus compromising geometric consistency. Therefore, this embodiment employs a strategy of "retaining only the nearest depth point per pixel" to achieve conflict resolution and occlusion suppression.

[0047] First, construct a pixel linear index: This involves converting the two-dimensional pixel coordinates... Encoded as a one-dimensional index : This encoding makes the same pixel correspond to the same... It can be used for grouping and deduplication.

[0048] Then, sort by "pixel index + depth", that is, sort all valid points according to the following rules: first by... Group pixels of the same type from smallest to largest, then sort them by depth d from smallest to largest. This way, for each pixel, the first pixel in the sorted sequence is the nearest neighbor. Finally, remove duplicate pixels (keeping only the first one), i.e., deduplicate the sorted pixels... The first position is taken as the representative point. According to the sorting property, this representative point has the minimum depth, thus achieving "nearest point preservation".

[0049] The filtered and conflict-resolved projection results are written into a fixed-size pixel array, resulting in a "pixel-to-3D point attribute" mapping that can be directly indexed by downstream algorithms. Based on the image dimensions (H, W), an `im_map(H, W, 3)` function is constructed, with all initial values ​​set to 0. The three channels have fixed meanings: Channel 0: radar coordinate system x (forward distance), Channel 1: radar coordinate system y (lateral distance), and Channel 2: radar coordinate system z (height). Simultaneously, a `valid_mask(H, W)` function is constructed, with an initial value of 0. Pixels with projection points are set to 1, allowing downstream algorithms to quickly determine whether a pixel contains 3D information and avoiding miscalculations based on values ​​of 0.

[0050] This embodiment projects the 3D point cloud data acquired by LiDAR onto the camera image coordinate system, establishing a sparse correspondence between "pixels and 3D points," thereby providing spatial point-based attribute information for each pixel in subsequent 3D geometric calculations of the road area. The mapping result output by this component can suppress occlusion and multi-point conflicts while ensuring projection accuracy, through a "retain nearest point per pixel" strategy, thus improving the stability of subsequent geometric calculations.

[0051] In some embodiments, the effective area for the vehicle's driving direction is determined based on a binary mask image of the road area and the spatial features of each pixel; including: Effective area in the direction of vehicle travel for: Effective area The calculation range is limited to 3 to 50 meters in front of the target area to exclude extremely close-range noise and sparse / unstable points that are too far away, as well as points in the road mask that are only in the road area. From the effective area Extract the height values ​​of all pixels and determine the minimum height value from them. ; For the effective area Height value of spatial feature vector Normalization is performed to obtain the normalized height value. : .

[0052] This treatment does not change the slope (the derivative remains unchanged), but it can reduce the numerical bias, making the fitted curve take the lowest point of the road as the reference zero point, thus improving the stability of the fit.

[0053] This embodiment uses a binary mask image of the road area to restrict subsequent calculations to be performed only within the road area, thus avoiding interference from road edges, vehicles, and non-ground points on slope fitting.

[0054] In some embodiments, based on the effective area of ​​the vehicle's driving direction, the mean value of the vehicle's driving direction and the mean value of its height for each row of the point cloud image are determined; including: Obtain the set of valid points for each row of the point cloud image. : Calculate the mean value of the vehicle's direction of travel for each row. : in, Indicates the number of valid points; Calculate the average height of each row : .

[0055] This embodiment constructs a vertical curve point set using an "image row aggregation" method. The core idea is to collect the effective 3D points of the road within each row of pixels and use the average value to form a representative point. This compresses the sparse, noisy set of points into a one-dimensional curve, making it easier to fit and differentiate.

[0056] In some embodiments, based on the mean of the vehicle's driving direction and the mean of the height in each row, a height mean curve is fitted with the mean of the vehicle's driving direction as the independent variable; including: Set the height mean curve with the vehicle's direction of travel as the independent variable as follows: in, This is the average value in the direction of the vehicle's travel. The average height All are coefficients; Based on the mean values ​​of the vehicle's driving direction and height for each row, coefficients are fitted to obtain the coefficients. .

[0057] In some embodiments, the slope value is calculated as follows: For the kth sampling point and the (k+1)th sampling point Calculate the k-th slope value : This embodiment combines road semantic masking to extract three-dimensional geometric sampling points within the road area, constructs a height change curve along the vehicle's direction of travel, and obtains the longitudinal and lateral slopes of the road through curve fitting and derivative calculation, outputting them in the form of slope, thereby providing a geometric basis for subsequent slope angle estimation and vehicle control / perception decision-making.

[0058] In some embodiments, the method further includes: Calculate the slope angle corresponding to each slope value to obtain the slope angle sequence: Starting from the k-th slope angle in the slope angle sequence, construct a window of a preset length. ; Window The mean of the statistic is determined as the k-th candidate statistic. Sort the candidate statistics from largest to smallest to obtain the candidate statistics sequence; Select a predetermined number of candidate statistics from the candidate statistic sequence, and calculate the mean of the predetermined number of candidate statistics as the slope angle estimate. .

[0059] In this embodiment, the slope sequence is converted into a slope angle sequence, and a stable slope estimate is obtained by selecting the Top-N significant windows and fusing the mean by sorting the sliding window mean and absolute value.

[0060] In some embodiments, the method further includes: The pitch angle at the current moment is obtained from the IMU on the vehicle. ; The estimated slope angle is compensated to obtain the compensated slope angle. : .

[0061] Since the gradient output is affected by the vehicle's own pitch attitude and systemic zero bias, in order to make the output closer to the actual road gradient, this embodiment uses IMU data for angle compensation to improve the robustness and engineering usability of the gradient output.

[0062] Based on the same inventive concept, this application provides an off-road road slope prediction device based on multimodal data fusion, see reference. Figure 2 As shown, the off-road road slope prediction device 200 based on multimodal data fusion provided in this application embodiment includes at least: The acquisition unit 201 is used to acquire the RGB image and point cloud data of the vehicle at the current moment collected on the off-road road; Image segmentation unit 202 is used to segment the RGB image using a pre-trained image segmentation model to obtain a binary mask image of the road region; The first calculation unit 203 is used to project the point cloud data onto the RGB image coordinate system to obtain a point cloud image, and calculate the spatial feature vector of each pixel of the point cloud image. The determining unit 204 is used to determine the effective area of ​​the vehicle's driving direction based on the spatial feature vector of each pixel of the binary mask image of the road area and the point cloud image; and to determine the mean value of the vehicle's driving direction and the mean value of the height of each row of the point cloud image based on the effective area of ​​the vehicle's driving direction. Fitting unit 205 is used to fit a height mean curve with the mean of the vehicle's driving direction as the independent variable based on the mean of the vehicle's driving direction and the mean of the height in each row. The second calculation unit 206 is used to perform uniform sampling on the height mean curve to obtain multiple sampling points; based on two adjacent sampling points, the slope value of the vehicle's driving direction is calculated.

[0063] It should be noted that the principle of the off-road slope prediction device 200 based on multimodal data fusion provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the off-road slope prediction device 200 based on multimodal data fusion provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0064] Based on the same inventive concept, embodiments of this application also provide an electronic device, such as... Figure 3 As shown, it includes a memory and a processor. The memory stores an executable program, and the processor executes the executable program to implement the steps of the off-road road slope prediction method based on multimodal data fusion provided in the above embodiments.

[0065] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0066] Since the electronic device described in this application embodiment is an electronic device equipped with a memory for implementing the off-road slope prediction method based on multimodal data fusion disclosed in this application embodiment, those skilled in the art can understand the structure and variations of the electronic device described in this application embodiment based on the off-road slope prediction method based on multimodal data fusion disclosed in this application embodiment, and therefore will not be described again here.

[0067] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it implements the steps of the off-road road slope prediction method based on multimodal data fusion provided in the above embodiments.

[0068] The storage medium in this embodiment may be included in an electronic device; or it may exist independently and not be assembled into an electronic device. The storage medium carries one or more computer programs, which, when executed, implement the steps of the off-road road slope prediction method based on multimodal data fusion provided in the above embodiment.

[0069] It should be understood that the various solutions in this embodiment have the same technical effects as those in the above method embodiments, and will not be repeated here.

[0070] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. Optionally, specific examples in this embodiment can refer to the examples described in any embodiment of this application, which will not be repeated here. Obviously, those skilled in the art should understand that the various modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular hardware and software combination.

[0071] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the off-road road slope prediction method based on multimodal data fusion provided in the above embodiments.

[0072] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions targeted in the blocks may occur in a different order than those targeted in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0073] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A method for predicting the slope of off-road terrain based on multimodal data fusion, characterized in that, include: Acquire RGB images and point cloud data of the vehicle at the current moment while it is on an off-road road; The RGB image is segmented using a pre-trained image segmentation model to obtain a binary mask image of the road region; The point cloud data is projected onto the RGB image coordinate system to obtain a point cloud image, and the spatial feature vector of each pixel in the point cloud image is calculated. Based on the binary mask image of the road area and the spatial feature vector of each pixel in the point cloud image, the effective area for the vehicle's driving direction is determined; based on the effective area for the vehicle's driving direction, the mean value of the vehicle's driving direction and the mean value of the height of each row in the point cloud image are determined. Based on the mean values ​​of the vehicle's driving direction and the mean values ​​of the height in each row, a height mean curve with the mean value of the vehicle's driving direction as the independent variable is obtained by fitting. Uniform sampling is performed on the height mean curve to obtain multiple sampling points; based on two adjacent sampling points, the slope value in the direction of vehicle travel is calculated. Set the pixels of the point cloud image The spatial feature vector is ,in, For pixels The x-component of the projection point in the radar coordinate system; For pixels The y-component of the projection point in the radar coordinate system; For pixels The z-component of the projection point in the radar coordinate system; Based on the binary mask image of the road area and the spatial features of each pixel, the effective area for determining the vehicle's driving direction is determined; including: Effective area in the direction of vehicle travel for: in, For pixels The mask value; From the effective area Extract the height values ​​of all pixels and determine the minimum height value from them. ; For the effective area Height value of spatial feature vector Normalization is performed to obtain the normalized height value. : Based on the effective area of ​​the vehicle's driving direction, determine the mean value of the vehicle's driving direction and the mean value of its height for each row of the point cloud image; including: Obtain the set of valid points for each row of the point cloud image. : Calculate the mean value of the vehicle's direction of travel for each row. : in, Indicates the number of valid points; Calculate the average height of each row : 。 2. The method according to claim 1, characterized in that, The binary mask image of the road area is represented as follows: When the pixels of the binary mask If it is a drivable area, then the pixel mask value ; otherwise, .

3. The method according to claim 2, characterized in that, Projecting the point cloud data onto the RGB image coordinate system to obtain a point cloud image, and calculating the spatial feature vector of each pixel in the point cloud image; including: Homogeneous processing is performed on each point in the point cloud data to obtain multiple homogeneous points; all homogeneous points are stacked column-wise to obtain a matrix. : Obtain the homogeneous extrinsic parameter matrix from the radar coordinate system to the camera coordinate system. : in, Let be the rotation matrix from the radar coordinate system to the camera coordinate system. This is the translation vector from the radar coordinate system to the camera coordinate system; Using homogeneous extrinsic matrices For matrix Perform extrinsic parameter transformation to obtain the matrix in the camera coordinate system. : Using the correction matrix For matrix Apply correction to obtain the matrix : Using projection matrix matrix Projecting onto the RGB image coordinate system yields the matrix. : Among them, matrix The i-th column These represent the homogeneous components of the projection point in the RGB coordinate system; Calculate the pixel coordinates of the i-th point : Two-dimensional pixel coordinates Encoded as a one-dimensional index : in, The width of the point cloud image; For the same one-dimensional index From multiple points, only the point with the minimum depth is retained, thus obtaining a point cloud image.

4. The method according to claim 1, characterized in that, Based on the mean values ​​of the vehicle's driving direction and the mean values ​​of the height in each row, a height mean curve with the mean value of the vehicle's driving direction as the independent variable is obtained by fitting. include: Set the height mean curve with the mean of the vehicle's direction of travel as the independent variable as follows: in, This is the average value in the direction of the vehicle's travel. The average height All are coefficients; Based on the mean values ​​of the vehicle's driving direction and height for each row, coefficients are fitted to obtain the coefficients. .

5. The method according to claim 1, characterized in that, The method further includes: Calculate the slope angle corresponding to each slope value to obtain a slope angle sequence; Starting from the k-th slope angle in the slope angle sequence, construct a window of a preset length. ; Window The mean of the statistic is determined as the k-th candidate statistic. Sort the candidate statistics from largest to smallest to obtain the candidate statistics sequence; Select a predetermined number of candidate statistics from the candidate statistic sequence, and calculate the mean of the predetermined number of candidate statistics as the slope angle estimate. .

6. The method according to claim 1, characterized in that, The method further includes: The pitch angle at the current moment is obtained from the IMU on the vehicle. ; The estimated slope angle is compensated to obtain the compensated slope angle. : 。 7. A device for predicting the slope of off-road terrain based on multimodal data fusion, characterized in that, include: The acquisition unit is used to acquire RGB images and point cloud data collected by the vehicle on the off-road road at the current moment; The image segmentation unit is used to segment the RGB image using a pre-trained image segmentation model to obtain a binary mask image of the road region; The first calculation unit is used to project the point cloud data onto the RGB image coordinate system to obtain a point cloud image, and to calculate the spatial feature vector of each pixel in the point cloud image. The determination unit is used to determine the effective area of ​​the vehicle's driving direction based on the spatial feature vector of each pixel in the binary mask image of the road area and the point cloud image; and based on the effective area of ​​the vehicle's driving direction, to determine the mean value of the vehicle's driving direction and the mean value of the height of each row of the point cloud image. The fitting unit is used to fit a height mean curve with the mean of the vehicle's driving direction as the independent variable, based on the mean of the vehicle's driving direction and the mean of the height in each row. The second calculation unit is used to perform uniform sampling on the height mean curve to obtain multiple sampling points; based on two adjacent sampling points, the slope value in the direction of vehicle travel is calculated. Set the pixels of the point cloud image The spatial feature vector is ,in, For pixels The x-component of the projection point in the radar coordinate system; For pixels The y-component of the projection point in the radar coordinate system; For pixels The z-component of the projection point in the radar coordinate system; The specific unit is used for: Effective area in the direction of vehicle travel for: in, For pixels The mask value; From the effective area Extract the height values ​​of all pixels and determine the minimum height value from them. ; For the effective area Height value of spatial feature vector Normalization is performed to obtain the normalized height value. : Obtain the set of valid points for each row of the point cloud image. : Calculate the mean value of the vehicle's direction of travel for each row. : in, Indicates the number of valid points; Calculate the average height of each row : 。 8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

Images