Image processing method and vehicle

Abstract

This application provides an image processing method and a vehicle. The method includes: acquiring image data of the road where the vehicle is located, and target calibration parameters of a vehicle acquisition device, wherein the image data is acquired by the vehicle acquisition device; extracting features from the image data to obtain image distribution features and depth distribution features, wherein the image distribution features represent the positional distribution of different objects in the image data, and the depth distribution features represent the distance between different objects and the vehicle; fusing the image distribution features and depth distribution features based on the target calibration parameters to obtain target image features; and predicting a road elevation map of the road where the vehicle is currently located based on the target image features, wherein the road elevation map represents the height distribution of the road. This application solves the technical problem of low accuracy in elevation estimation in vehicle image processing.

Description

Technical Field

[0001] This application relates to the field of vehicle image processing, and more specifically, to an image processing method and a vehicle. Background Technology

[0002] In autonomous driving environmental perception, monocular vision bird's-eye views are widely used for 3D road reconstruction. Vehicles need to accurately acquire road surface elevation information in complex road conditions to support safety functions such as suspension pre-adjustment and bump warning. However, related technologies for bird's-eye view elevation estimation generally suffer from problems such as insufficient detail in depicting the 3D geometric structure of the road and ineffective fusion of visual semantic information and spatial structural information. This results in low accuracy of elevation prediction results in complex scenarios, failing to meet the basic requirements of high-reliability autonomous driving perception for continuous, accurate, and robust 3D road structure modeling.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides an image processing method and a vehicle, which at least solves the technical problem of low accuracy in elevation estimation in vehicle image processing in related technologies.

[0005] According to one aspect of the embodiments of this application, an image processing method is provided, comprising: acquiring image data of the road where a vehicle is located, and target calibration parameters of a vehicle acquisition device, wherein the image data is acquired by the vehicle acquisition device; extracting features from the image data to obtain image distribution features and depth distribution features, wherein the image distribution features are used to represent the positional distribution of different objects in the image data, and the depth distribution features are used to represent the distance between different objects and the vehicle; fusing the image distribution features and depth distribution features based on the target calibration parameters to obtain target image features; and predicting a road elevation map of the road where the vehicle is currently located based on the target image features, wherein the road elevation map is used to represent the height distribution of the road.

[0006] Furthermore, feature fusion is performed on image distribution features and depth distribution features based on target calibration parameters to obtain target image features, including: spatial projection of image distribution features based on target calibration parameters to obtain first image features; multi-plane projection of depth distribution features based on target calibration parameters to obtain second image features, wherein multi-plane projection is used to represent the process of projection onto planes at different heights; and target image features are obtained based on the first image features and the second image features.

[0007] Further, the image distribution features are spatially projected based on the target calibration parameters to obtain the first image features, including: spatially projecting the image distribution features based on the target calibration parameters to obtain the first projection coordinate points; mapping the first projection coordinate points to a first space and a second space respectively to obtain the first feature and the second feature, wherein the first space is composed of multiple rectangular units and the second space is composed of multiple sector units; and obtaining the first image features based on the first feature and the second feature.

[0008] Further, based on the first feature and the second feature, a first image feature is obtained, including: mapping the second feature to a first space to obtain a third feature; and performing weighted fusion of the first feature and the third feature to obtain the first image feature.

[0009] Furthermore, the depth distribution features are projected onto multiple planes based on the target calibration parameters to obtain the second image features, including: projecting the depth distribution features onto multiple planes based on the target calibration parameters to obtain multiple second projection coordinate points, wherein different second projection coordinate points are located on different height planes; and encoding the positions of the multiple second projection coordinate points to obtain the second image features.

[0010] Further, based on the first image features and the second image features, the target image features are obtained, including: stitching the first image features and the second image features together to obtain the first stitched image features; performing multi-scale feature enhancement on the first stitched image features to obtain the target enhanced image features; obtaining the second stitched image features based on the first stitched image features and the target enhanced image features; and performing weighted fusion on the first stitched image features, the second stitched image features, the first image features, and the second image features to obtain the target image features.

[0011] Further, based on the first stitched image features and the target enhancement image features, a second stitched image feature is obtained, including: projecting the first stitched image features onto a preset grid cell to obtain multiple image pixels; determining a distance confidence map based on the average variance of the multiple image pixels, wherein the distance confidence map is used to represent the confidence level of the distance represented by different image pixels; and stitching the distance confidence map and the target enhancement image features to obtain the second stitched image feature.

[0012] Further, multi-scale feature enhancement is performed on the first stitched image features to obtain target enhanced image features, including: multi-scale feature enhancement is performed on the first stitched image features to obtain multiple enhanced image features, wherein the number of features of different enhanced image features is different; global average pooling is performed on the multiple enhanced image features to obtain the fusion weight of the multiple enhanced image features; and weighted fusion is performed on the multiple enhanced image features based on the fusion weight to obtain target enhanced image features.

[0013] Furthermore, the method also includes: obtaining the initial calibration parameters of the vehicle acquisition device; performing depth prediction on the pixels in the image data to obtain the depth parameters of the pixels, wherein the depth parameters are used to represent the distance between the object corresponding to the pixel and the vehicle; and correcting the initial calibration parameters based on the depth parameters to obtain the target calibration parameters.

[0014] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0015] The aforementioned memory can refer to devices inside a computer used to store data and programs, including RAM, hard disks, etc. RAM can be used to temporarily store running programs and data, while hard disks can be used to store programs and data long-term. Memory enables the computer to read and write data and execute programs. The aforementioned processor is responsible for executing instructions in computer programs and performing data processing. It can also be responsible for controlling and executing various operations, including arithmetic operations, logical operations, and data transmission.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0018] The aforementioned computer program products can refer to software programs that have been written, tested, and released, and can run on computers or other devices. Computer program products can include application programs, operating systems, utility software, etc., used to achieve specific functions or solve specific problems.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0020] The aforementioned non-volatile computer-readable storage medium can refer to a medium for storing data. Non-volatile computer-readable storage media can retain data without loss when power is off and can be used to store long-term data, such as operating systems, applications, and user files. Non-volatile storage media can include hard disk drives, solid-state drives, optical disks, and flash memory storage devices, etc.

[0021] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0022] The aforementioned computer program can refer to a set of instructions used to tell the computer to perform specific tasks or operations. Computer programs can be written by programmers using specific programming languages ​​and can include algorithms, data structures, logic, and control flow. Computer programs can be used for a variety of purposes, including application software, operating systems, etc.

[0023] In this embodiment, image data of the road where the vehicle is located and target calibration parameters of the vehicle acquisition device are acquired. The image data is acquired by the vehicle acquisition device. Feature extraction is performed on the image data to obtain image distribution features and depth distribution features. The image distribution features represent the positional distribution of different objects in the image data, and the depth distribution features represent the distance between different objects and the vehicle. Based on the target calibration parameters, the image distribution features and depth distribution features are fused to obtain target image features. Based on the target image features, a road elevation map of the road where the vehicle is currently located is predicted, whereby the road elevation map represents the height distribution of the road. This embodiment provides a data foundation for visual and geometric modeling by acquiring image data of the road where the vehicle is located and target calibration parameters of the vehicle acquisition device. Subsequently, this embodiment extracts image distribution features and depth distribution features from the image data to obtain semantic and distance perception information. Next, this embodiment of the application utilizes target calibration parameters to construct a dual-path modeling architecture of explicit geometric projection branch and implicit ray attention branch to generate target image features, thereby predicting road elevation maps. This achieves the goal of constructing a unified geometric representation space and accurately depicting the vertical height distribution of the road. As a result, it realizes the technical effect of directly outputting high-precision, spatially consistent road elevation information from monocular images without the need for external sensors, thus solving the technical problem of low accuracy in elevation estimation in vehicle image processing in related technologies. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a flowchart of an image processing method according to an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of an image processing method according to an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of an image processing algorithm framework according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of an optional image processing method according to an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of an optional image processing method according to an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of an image processing apparatus according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] According to an embodiment of this application, an image processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This embodiment provides an image processing method. Figure 1 This is a flowchart of an image processing method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0035] Step 102: Acquire image data of the road where the vehicle is located, as well as the target calibration parameters of the vehicle acquisition device. The image data is acquired by the vehicle acquisition device.

[0036] The road mentioned above refers to the actual ground environment that the vehicle comes into contact with while driving, which has three-dimensional elevation variation characteristics (such as slope, potholes, shoulders, etc.).

[0037] The image data mentioned above refers to images acquired in real time by the vehicle's data acquisition device. Image data includes, but is not limited to, road texture, structure, and visual information.

[0038] The aforementioned vehicle acquisition device refers to a monocular camera system installed on a vehicle. A monocular camera system possesses intrinsic and extrinsic parameters. The intrinsic parameters refer to the inherent imaging geometric parameters of the monocular camera system, used to describe the projection relationship between image pixel coordinates and the camera coordinate system (such as lens focal length, principal point position, and pixel distortion characteristics). The extrinsic parameters refer to the spatial pose parameters of the monocular camera system relative to the vehicle's motion coordinate system, used to describe the camera's position and orientation in the vehicle's three-dimensional space.

[0039] The aforementioned target calibration parameters refer to the camera intrinsic and extrinsic parameters of the vehicle acquisition device after online adaptive improvement. Extrinsic parameters include corrected pitch angle, roll angle, and translation components, used to express the geometric mapping relationship between image pixels and three-dimensional points in the vehicle coordinate system. Intrinsic parameters include, but are not limited to, focal length, principal point coordinates, and distortion coefficients (such as radial distortion and tangential distortion).

[0040] In one optional embodiment, the vehicle elevation map generation system acquires image data of the road ahead using a monocular camera deployed at the front of the vehicle. The image data undergoes distortion correction and color normalization by an onboard image preprocessing module. Simultaneously, the system utilizes a built-in Inertial Measurement Unit (IMU) and wheel speed sensors to obtain the vehicle's current pitch angle, roll angle, and speed. Combining this with a depth estimation network to predict the discrete depth distribution and variance of each pixel in the image, a reprojection error function is constructed. Using the road plane assumption as a geometric constraint, an iterative algorithm is used to dynamically solve for the correction of the camera's intrinsic and extrinsic parameters, ultimately outputting target calibration parameters that match the current vehicle attitude and environment.

[0041] The aforementioned inertial measurement unit is a sensor module used to measure the angular velocity and linear acceleration of an object in three-dimensional space. It is typically composed of a gyroscope and an accelerometer and can acquire real-time attitude change information of a vehicle (such as pitch angle and roll angle).

[0042] The aforementioned depth estimation network is used to predict the scene depth value corresponding to each pixel in image data, that is, the distance between the real-world object and the camera corresponding to each pixel. Scene depth values ​​are usually discrete depth distributions, providing a crucial distance information foundation for subsequent geometric projection and 3D reconstruction.

[0043] The aforementioned discrete depth distribution refers to dividing a continuous depth space into several preset depth intervals and predicting the probability value of each pixel in the image belonging to each depth interval, thus forming a probability distribution vector.

[0044] In another optional embodiment, the vehicle elevation map generation system acquires road image data through an onboard camera. The generation of target calibration parameters incorporates road segment-level calibration recommendations provided by Vehicle-to-Everything (V2X) services. When the vehicle enters a high-precision map area, it receives improved parameters (such as camera pitch compensation angle, vertical offset, and illumination correction factor) from historical road segment calibrations pushed by road infrastructure or a cloud platform. The vehicle elevation map generation system weights and fuses the cloud-recommended parameters with real-time attitude data from the vehicle's onboard inertial measurement unit (IMU), and fine-tunes them by combining local image depth consistency checks to generate high-precision, low-latency target calibration parameters.

[0045] This application embodiment acquires image data and target calibration parameters through a data acquisition device in the vehicle, enabling subsequent elevation estimation to adaptively compensate for changes in vehicle attitude, improving the stability and accuracy of 3D spatial modeling under complex road conditions, and providing reliable input for subsequent modeling.

[0046] Step 104: Extract features from the image data to obtain image distribution features and depth distribution features. The image distribution features are used to represent the positional distribution of different objects in the image data, and the depth distribution features are used to represent the distance between different objects and the vehicle.

[0047] The aforementioned image distribution features refer to the visual features extracted from image data that represent the spatial location and structural distribution of various objects (such as lane lines, vehicles, curbs, etc.) in a scene, reflecting the semantics and geometric layout of the objects.

[0048] The aforementioned depth distribution features refer to the probability distribution of the distance between each pixel in the image and the object, reflecting the three-dimensional structural information of the scene.

[0049] In one optional embodiment, the vehicle elevation map generation system extracts image distribution features containing semantic and spatial location information from image data step by step using a multi-layer convolutional network. Simultaneously, the system outputs the probability distribution of each pixel across multiple preset depth intervals through discrete depth prediction, and then calculates the weighted mean and variance of the depth based on these probability distributions to form a depth distribution feature reflecting the distance between the object and the vehicle and its confidence level.

[0050] The aforementioned multi-layer convolutional network takes image data as input and extracts multi-level feature maps from low-level edge textures to high-level semantic objects through hierarchical convolution and downsampling operations, outputting image distribution features containing spatial structure and semantic category information.

[0051] The discrete depth prediction described above outputs a discrete probability distribution vector for each pixel in the image, representing the probability that the true depth of each pixel falls within a preset depth range. By performing a weighted average and variance calculation on these distributions, depth distribution features are generated.

[0052] In another alternative embodiment, the vehicle elevation map generation system uses a coding structure based on a self-attention mechanism to globally model the image data, generating image distribution features with semantic consistency, while the depth distribution features are directly predicted by continuous probability modeling to predict the depth value and its uncertainty parameters corresponding to each pixel.

[0053] The encoding structure of the aforementioned self-attention mechanism is based on a transformer or variant architecture. It models long-range dependencies between pixels in an image through self-attention, achieving unified modeling of the global semantic context. This generates image distribution features with high semantic consistency and spatial structural integrity, effectively alleviating the limitations of traditional convolutional networks in representing distant object associations and occluded regions. The aforementioned transformer is a deep neural network architecture based on a self-attention mechanism. The core idea of ​​the transformer is to dynamically aggregate contextual information by calculating the global association weights between elements in the input feature sequence, thereby modeling long-range dependencies.

[0054] The aforementioned continuous probability modeling directly models the depth value of each pixel using a continuous probability distribution (such as Gaussian distribution, Laplace distribution, etc.), outputting the depth mean and uncertainty parameters (such as variance or scale parameters), thereby more accurately representing the distance between objects and vehicles in the real world and their estimation confidence, and improving the robustness and continuity of depth estimation in texture-deficient or distant regions.

[0055] This application embodiment achieves parallel acquisition of visual semantics and geometric structure by simultaneously extracting image distribution features and depth distribution features, providing a dual data foundation for subsequent elevation map output and enhancing robust modeling capabilities for complex scenes such as occlusion and weak textures.

[0056] Step 106: Based on the target calibration parameters, perform feature fusion on the image distribution features and depth distribution features to obtain the target image features.

[0057] The aforementioned target image features refer to a unified bird's-eye view (BEV) spatial representation generated in the vehicle coordinate system by geometrically aligning and semantically fusing image distribution features and depth distribution features through target calibration parameters. This BEV spatial representation combines semantic positional accuracy with three-dimensional elevation reliability.

[0058] In one optional embodiment, the vehicle elevation map generation system projects the image distribution features along the visual ray according to the target calibration parameters to generate the first image features, and projects the depth distribution features onto multiple height planes to form the second image features. Then, through stitching and multi-scale feature enhancement processing, the original image distribution features and the enhanced features are combined and weighted to finally output the target image features.

[0059] The aforementioned multi-scale feature enhancement processing refers to introducing parallel multi-scale convolutional branches (such as 3×3 convolutional kernels with different dilation rates or feature pyramids with different downsampling scales) into the stitched image features (such as the stitched result of the first image features and the second image features) to capture local fine-grained structures (such as road surface cracks and curb bulges) and global macroscopic morphology (such as slope and arched road surfaces). Subsequently, the vehicle elevation map generation system adaptively learns the fusion weights of features at each scale through global context aggregation (such as global average pooling), realizing dynamic weighted fusion of multi-scale geometric semantics, and finally outputting target enhanced image features with stronger representational capabilities.

[0060] In another optional embodiment, the vehicle elevation map generation system spatially projects the image distribution features based on target calibration parameters to obtain first image features, and performs multi-plane projection on the depth distribution features to generate second image features. The vehicle elevation map generation system further introduces confidence information from the depth distribution features as a spatial adaptive weight to guide the fusion process of the first and second image features. This allows the system to strengthen geometric projection information in regions with high depth confidence and enhance semantic reasoning capabilities in regions with low confidence, ultimately obtaining the target image features through stitching and joint improvement.

[0061] This application embodiment utilizes target calibration parameters to achieve the fusion of visual features and depth information, breaking through the limitations of single-modal modeling, effectively improving the geometric accuracy and scene adaptability of road elevation estimation, and providing core feature data with rigorous structure and rich semantics for the subsequent output of road elevation maps.

[0062] Step 108: Based on the features of the target image, predict the road elevation map of the road where the vehicle is currently located, where the road elevation map is used to represent the height distribution of the road.

[0063] The aforementioned road elevation map refers to a two-dimensional grid-like representation of the vertically changing elevation distribution of the road surface in the BEV plane. Each grid cell corresponds to an elevation value, reflecting the road surface undulations, slope, and local unevenness.

[0064] In one optional embodiment, the vehicle elevation map generation system inputs the target image features into a multi-layer convolutional prediction head, divides the road height into multiple discrete intervals through classification and regression, outputs the probability distribution of each height interval for each spatial location, and calculates continuous height values ​​based on weighted average, thereby generating a road elevation map that represents the longitudinal undulation of the road.

[0065] The aforementioned multi-layer convolutional prediction head is a lightweight neural network module composed of several consecutive convolutional layers, used to map high-dimensional target image features (such as BEV spatial features) into road elevation prediction results.

[0066] In another alternative embodiment, the vehicle elevation map generation system directly spatially maps the target image features through an end-to-end regression structure, outputting continuous height values. At the same time, it introduces local smoothing constraints and global slope consistency regularization terms, so that the prediction results satisfy the macroscopic road geometry while maintaining local details, and finally generate a road elevation map with physical consistency and strong noise resistance.

[0067] The aforementioned local smoothing constraint refers to applying a norm penalty to the height difference between adjacent BEV grid cells during the elevation prediction process, encouraging gradual changes in elevation values ​​in adjacent areas, thereby suppressing local abnormal fluctuations caused by depth noise, texture loss, or occlusion.

[0068] The aforementioned global slope consistency regularization term is a structured regularization constraint based on the macroscopic geometric laws of roads. By modeling the overall tilt trend of roads over long distances (such as uphill, downhill, and curve cross slope), it forces the predicted elevation map to be consistent with the preset road topology priors (such as road design standards and curvature-slope relationships).

[0069] This application embodiment predicts elevation maps through target image features, achieving end-to-end, continuous, and spatially consistent 3D road morphology reconstruction. This avoids the accuracy loss caused by discretization or grid interpolation, and improves the accuracy and practicality of perceiving key terrain features such as slopes, bumps, and shoulders.

[0070] In this embodiment, image data of the road where the vehicle is located and target calibration parameters of the vehicle acquisition device are acquired. The image data is acquired by the vehicle acquisition device. Feature extraction is performed on the image data to obtain image distribution features and depth distribution features. The image distribution features represent the positional distribution of different objects in the image data, and the depth distribution features represent the distance between different objects and the vehicle. Based on the target calibration parameters, the image distribution features and depth distribution features are fused to obtain target image features. Based on the target image features, a road elevation map of the road where the vehicle is currently located is predicted, whereby the road elevation map represents the height distribution of the road. This embodiment provides a data foundation for visual and geometric modeling by acquiring image data of the road where the vehicle is located and target calibration parameters of the vehicle acquisition device. Subsequently, this embodiment extracts image distribution features and depth distribution features from the image data to obtain semantic and distance perception information. Next, this embodiment of the application utilizes target calibration parameters to construct a dual-path modeling architecture of explicit geometric projection branch and implicit ray attention branch to generate target image features, thereby predicting road elevation maps. This achieves the goal of constructing a unified geometric representation space and accurately depicting the vertical height distribution of the road. As a result, it realizes the technical effect of directly outputting high-precision, spatially consistent road elevation information from monocular images without the need for external sensors, thus solving the technical problem of low accuracy in elevation estimation in vehicle image processing in related technologies.

[0071] Optionally, feature fusion is performed on image distribution features and depth distribution features based on target calibration parameters to obtain target image features, including: spatial projection of image distribution features based on target calibration parameters to obtain first image features; multi-plane projection of depth distribution features based on target calibration parameters to obtain second image features, wherein multi-plane projection is used to represent the process of projection onto planes at different heights; and target image features are obtained based on the first image features and the second image features.

[0072] The aforementioned first image feature refers to the explicit geometric representation obtained by mapping the image distribution features along the real camera projection geometry to the bird's-eye view plane based on the target calibration parameters. The first image feature is used to reflect the structural layout of objects in the scene on the horizontal plane.

[0073] The aforementioned second image feature refers to a multi-layer implicit height response map formed by projecting and encoding depth distribution features onto multiple vertical height planes based on target calibration parameters. The second image feature is used to characterize the structural confidence and spatial distribution of different elevation regions.

[0074] In one optional embodiment, when the vehicle elevation map generation system performs feature fusion on image distribution features and depth distribution features based on target calibration parameters, it first spatially maps the image distribution features along visual projection rays according to the target calibration parameters to obtain a first image feature representing the distribution of objects in the image on a bird's-eye view plane. Simultaneously, the depth distribution features are projected onto multiple preset height planes to form a second image feature reflecting distance information at different elevation levels, where each height plane corresponds to a vertical slice, thus constructing a multi-layered depth response. Subsequently, the vehicle elevation map generation system concatenates the first and second image features and aggregates local and global contextual information through multi-scale feature enhancement processing. Finally, the vehicle elevation map generation system adaptively weights and fuses the first and second image features with the aforementioned image enhancement features to generate a target image feature that integrates geometric structure and depth semantics.

[0075] This application embodiment captures road planar structure and vertical elevation change information through collaborative modeling of spatial projection and multi-plane projection, realizing the expression of geometric constraints and high sensitivity, and improving the accuracy and robustness of road elevation estimation under complex terrain of target image features.

[0076] Optionally, spatial projection of image distribution features based on target calibration parameters to obtain first image features includes: spatial projection of image distribution features based on target calibration parameters to obtain first projection coordinate points; mapping the first projection coordinate points to a first space and a second space respectively to obtain first features and second features, wherein the first space is composed of multiple rectangular units and the second space is composed of multiple sector units; and obtaining first image features based on the first features and second features.

[0077] The aforementioned first projection coordinate point is a three-dimensional spatial coordinate point obtained by back-calculating the two-dimensional image coordinates and their predicted depth values ​​corresponding to each pixel in the image distribution features to the vehicle coordinate system based on the target calibration parameters of the vehicle acquisition device through a three-dimensional spatial projection model.

[0078] The aforementioned first space refers to a two-dimensional planar coordinate system composed of uniformly distributed rectangular units. This first space characterizes the regular, orthogonal spatial structure of the road within the horizontal range in front of vehicles, and is suitable for describing features with straight or regular geometric shapes, such as lanes and curbs. The uniformly distributed rectangular units can be Cartesian grids, rectangular grids, parallelogram grids, etc.

[0079] The aforementioned second space refers to a two-dimensional radial space composed of sector-shaped units. This second space is radially distributed around the vehicle, with each unit covering a specific angular and radial distance range. It excels at capturing long-distance, non-uniformly distributed road structures (such as curves and ramp edges) and exhibits higher angular resolution under spatial projection. The aforementioned sector-shaped units can be polar coordinate grids, hierarchical sector-shaped grids, or logarithmic polar coordinate grids.

[0080] The first feature mentioned above refers to the spatial feature obtained by projecting the image distribution features onto the first space through the target calibration parameters and then accumulating and aggregating them within each rectangular unit. The first feature is used to reflect the semantic and geometric distribution of the road in the regular coordinate system.

[0081] The second feature mentioned above refers to the feature generated by aggregating the image distribution features along the radial and angular dimensions after being mapped to the second space by the same projection transformation, emphasizing the structural continuity and angular sensitivity of the road at long distances and under non-uniform viewing angles.

[0082] In one optional embodiment, when the vehicle elevation map generation system projects the image distribution features spatially based on target calibration parameters to obtain the first image features, it first projects the image distribution features along the visual ray in three-dimensional space according to the target calibration parameters, generating first projected coordinate points corresponding to the road plane. Subsequently, the vehicle elevation map generation system maps these first projected coordinate points to two different bird's-eye view representation spaces. The first space is composed of rectangular units divided by a uniform grid, used to capture the regular structure of the road in the Cartesian coordinate system in the horizontal and vertical directions. The second space is composed of fan-shaped units divided by radial sectors, used to preserve the natural perspective distribution characteristics from the vehicle's perspective, especially enhancing the ability to express distant and large-angle areas. The vehicle elevation map generation system accumulates and normalizes the projected points in the two spaces respectively to obtain the corresponding first and second features. After remapping the second feature to the first space, it performs weighted fusion with the first feature, thereby combining the uniformity of the rectangular grid and the perspective consistency of the fan-shaped grid, and finally generating a first image feature with stronger geometric adaptability and spatial integrity.

[0083] This application embodiment obtains first features and second features by simultaneously mapping image distribution features to a first space and a second space, thereby achieving adaptive complementarity between the two spatial features and enabling the obtained first image features to have high accuracy.

[0084] Optionally, based on the first feature and the second feature, a first image feature is obtained, including: mapping the second feature to a first space to obtain a third feature; and performing weighted fusion of the first feature and the third feature to obtain the first image feature.

[0085] The third feature mentioned above refers to the alignment feature generated by interpolating or sampling within each rectangular cell after reprojecting the second feature into the first space through a coordinate mapping algorithm (such as the inverse transformation from polar coordinates to Cartesian coordinates).

[0086] In one optional embodiment, when the vehicle elevation map generation system obtains the first image feature based on the first and second features, it first remaps the second feature from the second space composed of sector units to the first space composed of rectangular units through coordinate transformation, generating a third feature consistent with the structure of the first space. Subsequently, the vehicle elevation map generation system adaptively weights and fuses the first and third features according to the geometric consistency and depth confidence during the projection process. This allows the regular structural information retained in the rectangular grid and the viewpoint consistency information enhanced by the sector mapping to compensate for each other, thereby improving the accuracy of representing edge and distant regions while preserving the overall geometric shape of the road. Finally, it outputs a first image feature that integrates the advantages of multiple spaces.

[0087] This application embodiment achieves comparability and fusion of two different representations in the same coordinate system by uniformly mapping the second feature to the first space, thus avoiding alignment errors caused by directly splicing heterogeneous spaces.

[0088] Optionally, the depth distribution features are projected onto multiple planes based on the target calibration parameters to obtain the second image features, including: projecting the depth distribution features onto multiple planes based on the target calibration parameters to obtain multiple second projection coordinate points, wherein different second projection coordinate points are located on different height planes; and encoding the positions of the multiple second projection coordinate points to obtain the second image features.

[0089] The aforementioned multi-plane projection, based on the vehicle's target calibration parameters, samples and projects the depth information corresponding to each pixel in the image along its corresponding three-dimensional projection ray onto multiple preset vertical height planes, thereby constructing a hierarchical feature representation with height dimension information in the BEV space.

[0090] The aforementioned second projection coordinate point refers to a set of three-dimensional spatial coordinate points obtained by sampling the depth distribution information corresponding to each pixel in the image along its projection ray in three-dimensional space on multiple preset vertical height planes based on the target calibration parameters. Each of these coordinate points represents the geometric position of the image features in the bird's-eye view space at a specific height level. Different coordinate points are distributed on different height planes, forming a multi-level spatial sampling of the vertical structure of the scene.

[0091] The aforementioned positional encoding refers to injecting information related to the relative position in three-dimensional space into each of the second projection coordinate points to enhance the feature's ability to perceive height hierarchy, spatial orientation, and geometric structure.

[0092] In one optional embodiment, the vehicle elevation map generation system performs multi-plane projection on the depth distribution features based on target calibration parameters to obtain multiple second projection coordinate points, where each second projection coordinate point corresponds to a different height plane. The vehicle elevation map generation system further performs position encoding on the multiple second projection coordinate points to introduce spatial relative relationship information of each height level, and aggregates the encoded features to generate second image features. The second image features characterize the multi-layer geometric expression of depth semantics distributed along the vertical direction in bird's-eye view space, providing implicit geometric clues with height dimension perception capabilities for subsequent fusion with image distribution features.

[0093] This application embodiment achieves multi-layer perception of the road's vertical structure by encoding the position of the second projection coordinate point, thereby improving the accuracy and robustness of the elevation map output in complex terrain and depth uncertainty scenarios.

[0094] Optionally, obtaining target image features based on the first image features and the second image features includes: stitching the first image features and the second image features together to obtain a first stitched image feature; performing multi-scale feature enhancement on the first stitched image feature to obtain a target enhanced image feature; obtaining a second stitched image feature based on the first stitched image feature and the target enhanced image feature; and performing weighted fusion on the first stitched image feature, the second stitched image feature, the first image feature, and the second image feature to obtain the target image feature.

[0095] The first stitched image feature mentioned above is an initial fusion feature formed by combining the first image feature and the second image feature along the channel dimension, preserving the original information of both types of features. The channel dimension refers to the number of channels in the first stitched image feature, the second stitched image feature, and the stitched image feature used to represent different semantic or geometric information; each channel is equivalent to a feature response map.

[0096] The aforementioned target enhancement image features refer to the high-order features extracted after multi-scale contextual modeling of the features of the first stitched image, which enhances the ability to express local and global structures.

[0097] The second stitched image feature mentioned above further integrates the first stitched image feature with the target enhancement feature to form a composite representation containing multi-level semantic and structural details.

[0098] In one optional embodiment, when the vehicle elevation map generation system obtains the target image features based on the first image features and the second image features, it first concatenates the first image features and the second image features along the channel dimension to form an initial fusion representation containing spatial distribution and multi-height layer depth response, namely the first concatenated image features. Subsequently, the vehicle elevation map generation system performs multi-scale feature enhancement on the first concatenated image features. It extracts contextual information at different scales through parallel multi-dilution convolutional paths and combines global pooling to adaptively learn the fusion weights for each scale, generating target enhanced image features rich in multi-layered road structure information. Based on this, the vehicle elevation map generation system concatenates the first concatenated image features and the target enhanced image features again to form the second concatenated image features, fusing the original fusion information with the enhanced semantic expression. Finally, the vehicle elevation map generation system performs joint weighted fusion of the first concatenated image features, the second concatenated image features, and the first image features and the second image features. The fusion weights are derived from the confidence information represented by the depth distribution features, ensuring that the dominance of geometric structure is strengthened in depth-reliable regions and the semantic compensation capability is enhanced in depth-blurred or occluded regions.

[0099] This application embodiment effectively integrates the geometric accuracy of the original features and the semantic depth of the enhanced features through multi-level feature aggregation and weighted fusion, thereby improving the responsiveness of the target image features to complex road structures.

[0100] Optionally, based on the first stitched image features and the target enhancement image features, a second stitched image feature is obtained, including: projecting the first stitched image features onto a preset grid cell to obtain multiple image pixels; determining a distance confidence map based on the average variance of the multiple image pixels, wherein the distance confidence map is used to represent the confidence level of the distance represented by different image pixels; and stitching the distance confidence map and the target enhancement image features to obtain the second stitched image feature.

[0101] The aforementioned preset grid unit refers to the pre-divided regular spatial grid points under the bird's-eye view, which are used to carry the feature response after projection and constitute the discrete sampling unit of the two-dimensional BEV space.

[0102] The aforementioned multiple image pixels refer to the corresponding local spatial representation formed after mapping the feature vectors in the first stitched image features to the aforementioned spatial grid points, reflecting the joint geometric and semantic information of the grid unit positions.

[0103] The aforementioned distance confidence map is a two-dimensional map generated by statistically analyzing the depth variance of projected pixels within each spatial grid point. It is used to quantify the reliability of depth estimation at each spatial location; the smaller the variance, the higher the confidence level.

[0104] In one optional embodiment, when the vehicle elevation map generation system obtains the second stitched image features based on the first stitched image features and the target augmented image features, it first maps the first stitched image features to preset grid cells and extracts multiple image pixels corresponding to each grid cell through interpolation and spatial aggregation operations. Subsequently, based on the depth distribution features associated with the spatial locations corresponding to these image pixels, the vehicle elevation map generation system calculates the average variance among the pixels, forming a distance confidence map reflecting the stability of distance prediction. This confidence map characterizes the reliability of depth estimation within each grid cell. Next, the vehicle elevation map generation system stitches the distance confidence map and the target augmented image features along the channel dimension, combining the enhanced multi-scale semantic features with spatial confidence information. Finally, it generates the second stitched image features that integrate geometric semantics and confidence guidance information.

[0105] The aforementioned interpolation and spatial aggregation operations refer to the vehicle elevation map generation system mapping the features of the first stitched image from the discrete pixel coordinates of the original feature map to the continuous spatial positions of preset grid cells through bilinear interpolation, in order to compensate for non-integer coordinate offsets during the projection process. Subsequently, within the neighborhood covered by each grid cell, multiple interpolated feature points are weighted and aggregated (e.g., average, maximum, or Gaussian weighted summation) to generate a unified feature response representing that grid cell.

[0106] This application embodiment introduces depth uncertainty into the feature fusion process in a spatially perceptible form by constructing a distance confidence map, and expresses it jointly with higher-order enhanced features, enabling the network to adaptively distinguish between reliable and suspicious regions, improving the physical rationality and anti-interference ability of the fusion process, and enhancing the stability and accuracy of the elevation map in complex scenarios such as occlusion and weak texture.

[0107] Optionally, multi-scale feature enhancement is performed on the first stitched image features to obtain target enhanced image features, including: multi-scale feature enhancement is performed on the first stitched image features to obtain multiple enhanced image features, wherein the number of features of different enhanced image features is different; global average pooling is performed on the multiple enhanced image features to obtain the fusion weight of the multiple enhanced image features; and weighted fusion is performed on the multiple enhanced image features based on the fusion weight to obtain target enhanced image features.

[0108] The aforementioned multiple enhanced image features refer to multi-scale feature maps extracted in parallel from the features of the first stitched image using convolutional kernels with different receptive fields. Each feature map captures structural information in different spatial ranges, such as local details, mid-range textures, and global slope trends. The aforementioned convolutional kernels with receptive fields refer to convolutional kernel structures with different effective receptive ranges used in the convolutional operation for parallel extraction of multi-scale spatial features.

[0109] The aforementioned global average pooling refers to the average aggregation of all spatial locations of each enhanced feature map to generate a global vector representing the overall semantic strength of the scale, which is used to measure the contribution of each scale to the current task.

[0110] The aforementioned fusion weights are dynamic coefficients obtained by normalizing the global average pooling results, reflecting the relative importance of features at different scales in the current scenario, thus achieving adaptive feature selection and combination.

[0111] In one optional embodiment, when the vehicle elevation map generation system performs multi-scale feature enhancement on the first stitched image features to obtain the target enhanced image features, it first performs differential processing on the first stitched image features through parallel multi-scale convolutional paths, generating multiple enhanced image features with different receptive fields and channel numbers. Each path captures multi-level road morphology information from local details to global structure. Subsequently, the vehicle elevation map generation system performs a global average pooling operation on each enhanced image feature to extract a global semantic response vector, and calculates adaptive fusion weights for each scale feature based on the global semantic response vector, enabling the vehicle elevation map generation system to dynamically allocate contribution ratios according to the semantic richness of the input features. Finally, the vehicle elevation map generation system performs a weighted summation of the multiple enhanced image features based on the obtained fusion weights, achieving adaptive aggregation of multi-scale information, and generating the target enhanced image features that combine fine-grained structural sensitivity and global contextual consistency.

[0112] This application embodiment uses a multi-scale feature enhancement mechanism to enable the vehicle elevation map generation system to simultaneously extract local details and global structural information of the road surface, and adaptively allocate fusion weights based on the semantic importance of features at each scale, thereby achieving intelligent feature selection and collaborative improvement, effectively enhancing the elevation map's ability to perceive complex road morphologies and its consistency in expression.

[0113] Optionally, the method further includes: obtaining initial calibration parameters of the vehicle acquisition device; performing depth prediction on pixels in the image data to obtain depth parameters of the pixels, wherein the depth parameters are used to represent the distance between the object corresponding to the pixel and the vehicle; and correcting the initial calibration parameters based on the depth parameters to obtain target calibration parameters.

[0114] The aforementioned initial calibration parameters refer to the internal and external parameters preset by the vehicle acquisition device at the factory or during installation. These internal and external parameters include, but are not limited to, lens focal length, principal point position, and installation angle, and are used to describe the geometric relationship between the camera and the vehicle coordinate system, serving as the fundamental basis for image projection into three-dimensional space.

[0115] The aforementioned pixel refers to the smallest unit in an image collected by a sensor. Each pixel corresponds to the light information in a certain direction in the real world and is a basic component of image data.

[0116] The aforementioned depth parameter refers to the estimated actual distance between each pixel and the vehicle, inferred by analyzing the image content, reflecting the three-dimensional spatial location information of each point in the scene.

[0117] In one optional embodiment, before estimating road elevation, the vehicle elevation map generation system first acquires the initial calibration parameters of the vehicle acquisition device as the basis for geometric projection. Subsequently, the system performs depth prediction on each pixel in the input image data, outputting depth parameters representing the distance between the corresponding scene object and the vehicle. These depth parameters are calculated from discrete depth probability values ​​using a weighted average and variance, providing both center estimation and confidence information. Based on the obtained depth parameters, the system analyzes the geometric consistency deviation when pixels are projected onto the bird's-eye view plane in the image space. By minimizing projection-reprojection errors or spatial smoothing constraints on the depth distribution, it dynamically corrects the initial calibration parameters, achieving online compensation for systematic deviations such as vehicle pitch and roll angles. Finally, the system uses the corrected parameters as the target calibration parameters.

[0118] The aforementioned minimization of projection reprojection error refers to the process by which the vehicle elevation map generation system, based on the current initial calibration parameters, projects the depth parameters of the pixels in reverse to three-dimensional space and then reprojects them back to the image plane, calculating the deviation between the original pixel position and the original pixel position.

[0119] The aforementioned spatial smoothness constraint of depth distribution refers to the vehicle elevation map generation system applying a consistency constraint on the depth parameters of adjacent pixels in the image space. This encourages smooth changes in depth values ​​and their confidence in semantically continuous regions (such as road surfaces and curbs), and suppresses abnormal fluctuations in local depth caused by texture loss, occlusion, or noise.

[0120] This application embodiment analyzes the depth information in the image to dynamically correct the parameters of the vehicle acquisition device, making the projection geometry more consistent with the real road environment and effectively eliminating calibration drift problems caused by vehicle vibration, assembly errors or long-term use.

[0121] Optionally, depth prediction is performed on pixels in the image data to obtain depth parameters for the pixels, including: performing depth prediction on pixels in the image data to obtain discrete depth probability values ​​for the pixels, wherein the discrete depth probability values ​​are used to represent the probability of the predicted pixels in different preset depth intervals; based on the discrete depth probability values, a weighted average is performed on the center values ​​of different preset depth intervals to obtain a depth mean; based on the discrete depth probability values ​​and the depth mean, a depth variance is determined, wherein the depth variance is used to represent the confidence level of the discrete depth probability values; and based on the depth mean and the depth variance, a depth parameter is determined.

[0122] The aforementioned discrete depth probability value refers to dividing the depth range into multiple fixed intervals and predicting the probability of each pixel falling into each interval.

[0123] The aforementioned preset depth intervals refer to a series of depth ranges pre-defined for ease of calculation. Each interval has a center value, which is used to discretize continuous depth values ​​to facilitate neural network learning and inference.

[0124] The aforementioned mean depth is the average depth estimate obtained by weighting and summing the center values ​​of each depth interval based on the probability of each interval, representing the depth position of the current pixel.

[0125] The aforementioned depth variance reflects the degree of concentration of the depth probability distribution. The smaller the variance, the more certain the model is of the true depth of the pixel. The larger the variance, the more ambiguous the prediction and the lower the confidence level. It is a key indicator for measuring the reliability of depth prediction.

[0126] The aforementioned depth parameters constitute a complete depth representation composed of the depth mean and depth variance, and are a type of depth information with confidence.

[0127] In one optional embodiment, when the vehicle elevation map generation system performs depth prediction on pixels in image data to obtain depth parameters, it first outputs a discrete depth probability value for each pixel through a depth estimation network. The discrete depth probability value represents the distribution probability of the pixel within several preset depth intervals. Subsequently, based on these discrete depth probability values, the vehicle elevation map generation system performs a weighted average calculation on the center values ​​of each depth interval to obtain a depth mean reflecting the expected distance of the pixel. On this basis, the vehicle elevation map generation system calculates the depth variance by combining the discrete depth probability values ​​and the depth mean. The depth variance is the weighted sum obtained by multiplying the square of the difference between the center value and the depth mean of each depth interval by the corresponding probability value. This weighted sum is used to quantify the dispersion of the pixel's depth estimation and serves as a measure of its prediction confidence level. Finally, the vehicle elevation map generation system jointly encodes the depth mean and depth variance to form the pixel's depth parameter. The depth parameter is a two-dimensional vector, where the first component represents the center estimate of the scene distance, and the second component represents the degree of uncertainty of the corresponding estimate.

[0128] This application's embodiments model depth in a probabilistic manner, providing not only a depth estimate for each point but also quantifying the reliability of that estimate. This enables subsequent geometric projection and feature fusion processes to avoid misleading elevation results with inaccurate information, thereby improving the stability of the vehicle elevation map generation system in complex scenarios.

[0129] This application provides a method for estimating monocular bird's-eye view elevation based on explicit-implicit hybrid geometric modeling. Figure 2 This is a schematic diagram of an image processing method according to an embodiment of this application, such as...Figure 2 As shown, monocular images undergo feature encoding and depth / pose estimation, displaying a geometric bird's-eye view. This includes camera parameters, virtual camera normalization, and implicit ray bird's-eye view. Multi-scale uncertainty perception fusion, elevation prediction head, and elevation estimation are also performed. Specifically, the steps include:

[0130] The network provided in this embodiment has a dual-input, dual-output structure. The input is a single-view RGB image. and camera calibration parameters (such as internal references) With external references ), the above 3 represents the image height, W represents the image width, and 3 represents the RGB three channels. The output is a predicted BEV elevation map. With shared BEV features The above. The number of horizontal grids in the BEV plane. Here, C represents the number of vertical grid cells in the BEV plane and the number of feature channels. The network, based on its processing flow, can be divided into a forward-looking coding module, a virtual camera normalization module, an explicit geometric branch, an implicit ray branch, a multi-scale uncertainty perception fusion module, and an elevation prediction head. The camera calibration parameters mentioned above refer to the target calibration parameters.

[0131] Forward-looking feature encoding and depth / pose estimation. This embodiment uses EfficientNet-B3 as the backbone network to extract multi-scale image features, which are then fused into a uniform-resolution forward-looking feature map through a cascaded feature pyramid network structure. .in, , , This application embodiment specifies each pixel. Discrete depth distribution is predicted using an independent depth head (3 layers of 3×3 convolutions). The depth distribution satisfies D represents the total number of depth intervals (discrete granularity). For pixels In the The predicted probabilities over k depth intervals, and the numbering sequence of k depth intervals. The EfficientNet-B3 described above is an efficient and lightweight convolutional neural network architecture that improves computational resource consumption and model parameter count while maintaining high recognition accuracy. The feature pyramid network structure fuses high-semantic low-resolution feature maps and low-semantic high-resolution feature maps output from the backbone network (such as EfficientNet-B3) at different depths through top-down and lateral connections, thereby generating a set of multi-scale feature maps with uniform resolution and rich semantic and spatial details.

[0132] Mean depth With variance Calculated from the distribution:

[0133]

[0134] Where D is the total number of depth intervals. For pixels The weighted average of the predicted depth, For pixels The variance of the predicted depth, For pixels In the Predicted probabilities over depth intervals The center value of the k-th depth interval is given. The pose estimation branch uses the feature vector after global average pooling. Regression is performed to predict the pitch and roll corrections relative to the calibration extrinsic parameters. ,in, It consists of two fully connected layers. The aforementioned corrections are used to construct virtual camera parameters to compensate for the interference of vehicle dynamic attitude changes on geometric projection. This is the roll angle correction amount. This is the pitch angle correction amount.

[0135] Virtual camera normalization. The core idea of ​​virtual camera normalization is to dynamically adjust the camera's extrinsic parameters using attitude corrections, making the projected geometry approximate the ideal assumption. Correction matrix. The construction is as follows:

[0136]

[0137] Wherein, Δψ is the roll angle correction amount, Δ This is the pitch angle correction amount. The operation converts Euler angles into a rotation matrix. (Internal parameters) Can be maintained The principal point offset can remain unchanged, or it can be improved during training to adapt to lens distortion. , This is the extrinsic parameter after normalization for the virtual camera. All subsequent 3D coordinate transformations will use this parameter. .

[0138] Explicit geometric bird's-eye view branching. The core of explicit branching lies in the radial-Cartesian hybrid mesh design. First, standard upscaling, flattening, and shooting projection (Lift, Splat, Shoot, LSS) are performed, i.e., pixel-wise... With depth bin Calculate world coordinates :

[0139]

[0140] in, Let be the coordinates of the 3D point corresponding to the k-th depth interval in the world coordinate system. The center value of the k-th depth interval. This is the inverse of the virtual camera intrinsic parameter matrix. Let be the homogeneous coordinates of the image pixels. 3D ray point in camera coordinate system Translation vector in the extrinsic parameters of a virtual camera. The inverse matrix of the rotation matrix in the extrinsic parameters of the virtual camera. This embodiment of the application is based on the BEV plane. A dual-grid system is constructed on top of this. The LSS projection described above is a method for mapping two-dimensional image features to three-dimensional space, thereby constructing a bird's-eye view (BEV).

[0141] Cartesian grid coverage Resolution 0.1m, size Radial grid polar coordinates , (Diagonal radius), resolution 0.1m, (Front-view sector), resolution 1°, size .

[0142] Projection operator and Map the three-dimensional points to the Cartesian unit respectively. With radial unit Deep uncertainty modulation weights are introduced during feature accumulation. :

[0143]

[0144] in, To preset hyperparameters, The term is used to compensate for scale attenuation in perspective projection. For depth variance.

[0145] Descartes BEV characteristics We obtain the following through weighted accumulation and normalization:

[0146]

[0147] in, The explicit eigenvector of the (i,j)th BEV cell in the Cartesian grid. For Cartesian projection functions, It is a linear mapping feature. For deep uncertainty modulation weights, Normalization factor. Radial BEV characteristics. Calculate similarly.

[0148] Radial features are remapped to a Cartesian grid via a lookup table. :

[0149]

[0150] in, The radial distance in the world coordinate system corresponding to the Cartesian mesh cell (i,j). For index mapping of angular direction, This represents the azimuth angle in the world coordinate system corresponding to the Cartesian grid cell (i,j). The final explicit features are then fused using layer normalization. :

[0151]

[0152] in, For layer normalization, Characterized by Cartesian BEV The radial BEV features after remapping The fusion weight matrix is ​​learnable. Implicit ray BEV branch. The implicit branch is for each BEV unit. Construct a ray-indexing mechanism. Set world coordinates. :

[0153]

[0154] in, This represents the reference height of the lateral position coordinates of the i-th horizontal grid cell in the BEV diagram in the world coordinate system. This represents the vertical position coordinates of the j-th vertical grid cell in the world coordinate system in the bird's-eye view. Meters. Projected onto the pixel plane. :

[0155]

[0156] in, Pixel coordinates (homogeneous coordinates) in the image plane. For camera projection function, This represents the three-dimensional coordinates of the position of the (i,j)th element in the BEV plane within the world coordinate system. (Internal parameters) For internal reference, , To obtain the normalized extrinsic parameters for the virtual camera, samples are taken along the ray in the height dimension. Points: , exist Uniformly distributed across intervals. Projecting onto each sampling point and using bilinear interpolation to obtain the forward-looking features. .

[0157] Location coding and geometric coding enhance query capabilities:

[0158]

[0159] in, For projection to All pixel depth statistics, This represents the reference height of the lateral position coordinates of the i-th horizontal grid cell in the BEV diagram in the world coordinate system. This represents the vertical position coordinates of the j-th vertical grid cell in the world coordinate system in the bird's-eye view. For position encoding, sine and cosine functions are used for the BEV unit. , For geometric encoding, reference height is fused using a multilayer perceptron. average depth Depth Variance Injecting regional depth statistics enhances the geometric semantic consistency of ray sampling.

[0160] The transformer attention calculation uses the standard scaled dot product form and outputs implicit BEV features. This embodiment uses a 4-head attention mechanism with a hidden layer dimension of 256.

[0161] The Multi-scale Uncertainty-aware Fusion Module (MUF) integrates multi-scale structural coding with uncertainty gating in a unified design.

[0162] First, concatenate the bi-branch features. :

[0163]

[0164] Where C is the number of feature channels in a single branch. It is an explicit feature. These are implicit features. Parallel dilated convolutions achieve multi-scale context aggregation:

[0165]

[0166] The convolution kernels are all 3×3, and the number of output channels is... , For a 3×3 convolution operation with a hole rate of k, Enhance feature maps for local details, Mid-range structural enhancement feature map Macro trend enhancement feature map and As a bibranching feature, the above , and These represent expansion rates of 1, 2, and 3, respectively.

[0167] Global context guides scale weight generation:

[0168]

[0169] in, This represents a two-branch feature, where C is the number of channels in a single-branch feature. This is the global average pooling function. Let g be the learnable weight transformation matrix, and g be the global context feature vector. This involves normalizing the global feature vector after linear transformation, converting it into a set of non-negative weight coefficients that sum to 1. Multi-scale feature weighted fusion:

[0170]

[0171] Deep uncertainty aggregation for BEV units Collect the set of all pixels projected onto this unit. , These are the weighting coefficients. Enhance feature maps for local details, Mid-range structural enhancement feature map Macroeconomic trend enhancement feature map, calculate average variance :

[0172]

[0173] in, Let be the set of pixels projected onto BEV cell (i,j). Let (u, v) be the depth variance of pixels (u, v) in the front view image. This forms an uncertainty heatmap. Adaptive weighted full convolutional prediction:

[0174]

[0175] in, This indicates that in the BEV space, the (i,j)th cell will have multi-scale augmented features. With distance confidence plot The joint feature vector is formed by concatenating the features along the channel dimension. This indicates that the joint features are obtained through a 1×1 convolution. After nonlinear mapping, the output of the sigmoid function is a spatially adaptive fusion weight between 0 and 1. Final fusion features:

[0176]

[0177] in, It is an explicit feature. These are implicit features. This design causes the network to favor explicit geometry in deep, reliable regions and automatically switch to implicit inference in uncertain regions.

[0178] BEV elevation prediction head, final BEV characteristics The data is fed into a lightweight prediction head and processed using EfficientNet-B0 style 2D convolutions. An elevation map is generated using an interval classification and regression strategy, outputting a height feature map. ,in, This represents the number of discrete height intervals. The EfficientNet-B0 described above is a lightweight, high-efficiency convolutional neural network architecture that maintains low computational overhead while possessing excellent image feature extraction capabilities, and is commonly used for real-time perception tasks deployed in vehicles. After Softmax normalization of each grid channel, continuous height prediction is obtained through soft argmin:

[0179]

[0180] in, Indicates the position in a bird's-eye view space. Elevation prediction value at the location The soft-argmin method is used, which involves using representative values ​​from discrete height ranges. Perform a weighted summation to output a continuous and smooth elevation estimate. For the first Each interval represents an elevation value.

[0181] Training strategy and loss function. This application's embodiments employ a phased training strategy. The first phase freezes the MUF module, training only the two-branch architecture until convergence. The second phase jointly fine-tunes the entire network, with the loss function including depth loss, i.e., a loss based on the predicted distribution. Sparse depth truth with LiDAR projection A binary cross-entropy method is used. The elevation loss is calculated based on the predicted elevation. Compared with LiDAR rasterized elevation true values A smoothed L1 loss is used. Uncertainty regularization is applied, i.e., for depth variance. Apply L2 regularization to prevent overconfidence. The total loss is a weighted sum, i.e. The aforementioned LiDAR projection refers to the process of mapping the three-dimensional point cloud data acquired by the LiDAR radar onto a two-dimensional image plane based on the camera's intrinsic and extrinsic parameters.

[0182] The above Depth-distribution cross-entropy loss, To smooth the L1 loss for elevation prediction, This is the L2 regularization term for the depth variance. The smoothed L1 loss described above is a robust regression loss function that combines the characteristics of L1 and L2 losses. It is used to calculate the error between the predicted and true values. Its core advantage lies in its insensitivity to outliers while maintaining differentiability and gradient stability in small error regions. The L2 regularization described above is a regularization method that suppresses model overfitting and overconfidence by adding a penalty term to the sum of squared model parameters in the loss function.

[0183] Figure 3 This is a schematic diagram of an image processing algorithm framework according to an embodiment of this application, such as... Figure 3 As shown, the vehicle elevation map generation system is based on monocular image input. It extracts image features through two convolutional layers in the backbone network and predicts the depth distribution to represent the discrete depth probability of each pixel. Simultaneously, a pose estimation module calculates camera pose corrections to obtain improved parameters to replace the initial camera parameters. In the explicit geometry branch, the improved parameters are used to perform feature mapping on the image features, generating Cartesian and polar coordinate features. Weighted fusion and layer normalization are then used to achieve stable fusion of the multi-grid structure. In the implicit ray branch, the vehicle elevation map generation system performs position and geometric encoding on the projection of each sampling point, and combines a standard transform to aggregate semantic information across the elevation layers. Subsequently, the system concatenates the explicit and implicit features and extracts multi-scale context through multi-scale dilated convolution and a perceptron. The features output by the perceptron are then processed by soft maximum normalization to generate spatial adaptive fusion weights. Global average pooling is used to extract global context features, which are further weighted and fused. Combined with a distance confidence map constructed from the average variance, the system is again processed through concatenation and convolutional layers. Ultimately, the vehicle elevation map generation system adaptively fuses explicit and implicit features using a weighted fusion module to output a continuous road elevation map.

[0184] Figure 4 This is a schematic diagram of an optional image processing method according to an embodiment of this application, such as... Figure 4 As shown, image data of the road where the vehicle is located and the target calibration parameters of the vehicle acquisition device are acquired. Feature extraction is performed on the image data to obtain image distribution features and depth distribution features. Based on the target calibration parameters, the image distribution features are spatially projected to obtain the first image feature. At the same time, based on the target calibration parameters, the depth distribution features are multi-plane projection to obtain the second image feature. The first image feature and the second image feature are combined to generate the target image feature. Finally, the road elevation map of the road where the vehicle is currently located is predicted based on the target image feature.

[0185] Figure 5This is a schematic diagram of an optional image processing method according to an embodiment of this application, such as... Figure 5 As shown, image data of the road where the vehicle is located and the target calibration parameters of the vehicle acquisition device are acquired. Feature extraction is performed on the image data to obtain image distribution features and depth distribution features. Based on the target calibration parameters, the image distribution features are spatially projected to obtain the first image feature. At the same time, based on the target calibration parameters, the depth distribution features are multi-plane projection to obtain the second image feature. The first image feature and the second image feature are combined to generate the target image feature. Finally, the road elevation map of the road where the vehicle is currently located is predicted based on the target image feature.

[0186] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0187] According to another aspect of the embodiments of this application, an image processing method is also provided, the image processing method comprising the following steps:

[0188] First, image data of the road where the vehicle is located, as well as the target calibration parameters of the vehicle acquisition device, are acquired. The image data is acquired by the vehicle acquisition device.

[0189] Next, feature extraction is performed on the image data to obtain image distribution features and depth distribution features. The image distribution features are used to represent the positional distribution of different objects in the image data, and the depth distribution features are used to represent the distance between different objects and vehicles.

[0190] Next, the image distribution features are spatially projected based on the target calibration parameters to obtain the first image features; the depth distribution features are multi-plane projected based on the target calibration parameters to obtain the second image features, wherein the multi-plane projection is used to represent the process of projecting onto planes at different heights; and the target image features are obtained based on the first image features and the second image features.

[0191] Next, the image distribution features are spatially projected based on the target calibration parameters to obtain the first projection coordinate points; the first projection coordinate points are mapped to the first space and the second space respectively to obtain the first feature and the second feature, wherein the first space is composed of multiple rectangular units and the second space is composed of multiple sector units; based on the first feature and the second feature, the first image feature is obtained.

[0192] Next, the second feature is mapped to the first space to obtain the third feature; the first feature and the third feature are then weighted and fused to obtain the first image feature.

[0193] Finally, a road elevation map of the road where the vehicle is currently located is predicted based on the features of the target image. The road elevation map is used to represent the height distribution of the road.

[0194] According to another aspect of the embodiments of this application, an image processing method is also provided, the image processing method comprising the following steps:

[0195] First, image data of the road where the vehicle is located and target calibration parameters of the vehicle acquisition device are acquired, wherein the image data is acquired by the vehicle acquisition device;

[0196] Next, feature extraction is performed on the image data to obtain image distribution features and depth distribution features. The image distribution features are used to represent the positional distribution of different objects in the image data, and the depth features are used to represent the distance between the different objects and the vehicle.

[0197] Next, the image distribution features are spatially projected based on the target calibration parameters to obtain first image features; the depth distribution features are multi-plane projected based on the target calibration parameters to obtain multiple second projection coordinate points, wherein different second projection coordinate points are located on different height planes; the multiple second projection coordinate points are positionally encoded to obtain second image features. Based on the first image features and the second image features, the target image features are obtained.

[0198] Finally, a road elevation map of the road where the vehicle is currently located is predicted based on the target image features, wherein the road elevation map is used to represent the height distribution of the road.

[0199] According to an embodiment of this application, an image processing apparatus is provided. It should be noted that the apparatus can be used to perform the image processing method described above. Figure 6 This is a schematic diagram of an image processing apparatus according to an embodiment of this application, such as... Figure 6 As shown, the device includes: an acquisition module 602, a feature extraction module 606, a fusion module 606, and a prediction module 608.

[0200] The acquisition module 602 is used to acquire image data of the road where the vehicle is located, as well as the target calibration parameters of the vehicle acquisition device, wherein the image data is acquired by the vehicle acquisition device; the feature extraction module 604 is used to extract features from the image data to obtain image distribution features and depth distribution features, wherein the image distribution features are used to represent the positional distribution of different objects in the image data, and the depth distribution features are used to represent the distance between different objects and the vehicle; the fusion module 606 is used to fuse the image distribution features and depth features based on the target calibration parameters to obtain target image features; the prediction module 608 is used to predict the road elevation map of the road where the vehicle is currently located based on the target image features, wherein the road elevation map is used to represent the height distribution of the road.

[0201] The fusion module is also used to perform spatial projection on the image distribution features based on the target calibration parameters to obtain the first image features; to perform multi-plane projection on the depth distribution features based on the target calibration parameters to obtain the second image features, wherein multi-plane projection is used to represent the process of projection onto planes at different heights; and to obtain the target image features based on the first image features and the second image features.

[0202] The fusion module is also used to spatially project the image distribution features based on the target calibration parameters to obtain the first projection coordinate points; map the first projection coordinate points to the first space and the second space respectively to obtain the first feature and the second feature, wherein the first space is composed of multiple rectangular units and the second space is composed of multiple sector units; and obtain the first image feature based on the first feature and the second feature.

[0203] The fusion module is also used to map the second feature to the first space to obtain the third feature; and to perform weighted fusion of the first feature and the third feature to obtain the first image feature.

[0204] The fusion module is also used to perform multi-plane projection on the depth distribution features based on the target calibration parameters to obtain multiple second projection coordinate points, wherein different second projection coordinate points are located on different height planes; and to perform position encoding on the multiple second projection coordinate points to obtain second image features.

[0205] The fusion module is also used to stitch together the first image features and the second image features to obtain the first stitched image features; to perform multi-scale feature enhancement on the first stitched image features to obtain the target enhanced image features; to obtain the second stitched image features based on the first stitched image features and the target enhanced image features; and to perform weighted fusion on the first stitched image features, the second stitched image features, the first image features, and the second image features to obtain the target image features.

[0206] The fusion module is also used to project the first stitched image features onto a preset grid cell to obtain multiple image pixels; determine a distance confidence map based on the average variance of the multiple image pixels, wherein the distance confidence map is used to represent the confidence level of the distance represented by different image pixels; and stitch the distance confidence map and the target enhanced image features to obtain a second stitched image feature.

[0207] The fusion module is also used to perform multi-scale feature enhancement on the first stitched image features to obtain multiple enhanced image features, wherein the number of features of different enhanced image features is different; to perform global average pooling on the multiple enhanced image features to obtain the fusion weights of the multiple enhanced image features; and to perform weighted fusion on the multiple enhanced image features based on the fusion weights to obtain the target enhanced image features.

[0208] The device also includes: acquiring the initial calibration parameters of the vehicle acquisition device; performing depth prediction on the pixels in the image data to obtain the depth parameters of the pixels, wherein the depth parameters are used to represent the distance between the object corresponding to the pixel and the vehicle; and correcting the initial calibration parameters based on the depth parameters to obtain the target calibration parameters.

[0209] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0210] The aforementioned memory can refer to devices inside a computer used to store data and programs, including RAM, hard disks, etc. RAM can be used to temporarily store running programs and data, while hard disks can be used to store programs and data long-term. Memory enables the computer to read and write data and execute programs. The aforementioned processor is responsible for executing instructions in computer programs and performing data processing. It can also be responsible for controlling and executing various operations, including arithmetic operations, logical operations, and data transmission.

[0211] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0212] The aforementioned computer storage media can refer to the media used in computer memory to store certain discontinuous physical quantities. Computer storage media mainly include semiconductors, magnetic cores, magnetic drums, magnetic tapes, laser discs, etc. Computer-readable storage media include stored programs, which can be a set of instructions that a computer can recognize and execute, running on an electronic computer to meet certain information needs.

[0213] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0214] The aforementioned computer program products can refer to software programs that have been written, tested, and released, and can run on computers or other devices. Computer program products can include application programs, operating systems, utility software, etc., used to achieve specific functions or solve specific problems.

[0215] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0216] The aforementioned non-volatile computer-readable storage medium can refer to a medium for storing data. Non-volatile computer-readable storage media can retain data without loss when power is off and can be used to store long-term data, such as operating systems, applications, and user files. Non-volatile storage media can include hard disk drives, solid-state drives, optical disks, and flash memory storage devices, etc.

[0217] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0218] The aforementioned computer program can refer to a set of instructions used to tell the computer to perform specific tasks or operations. Computer programs can be written by programmers using specific programming languages ​​and can include algorithms, data structures, logic, and control flow. Computer programs can be used for a variety of purposes, including application software, operating systems, etc.

[0219] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0221] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0222] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0223] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0224] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, include: The image data of the road where the vehicle is located, and the target calibration parameters of the vehicle acquisition device are acquired, wherein the image data is acquired by the vehicle acquisition device; Feature extraction is performed on the image data to obtain image distribution features and depth distribution features, wherein the image distribution features are used to represent the positional distribution of different objects in the image data, and the depth features are used to represent the distance between the different objects and the vehicle; Based on the target calibration parameters, the image distribution features and the depth distribution features are fused to obtain the target image features; Based on the features of the target image, a road elevation map of the road where the vehicle is currently located is predicted, wherein the road elevation map is used to represent the height distribution of the road.

2. The method according to claim 1, characterized in that, Based on the target calibration parameters, feature fusion is performed on the image distribution features and the depth distribution features to obtain target image features, including: Based on the target calibration parameters, the image distribution features are spatially projected to obtain the first image features; Based on the target calibration parameters, the depth distribution features are subjected to multi-plane projection to obtain the second image features, wherein the multi-plane projection is used to represent the process of projection onto planes at different heights; The target image features are obtained based on the first image features and the second image features.

3. The method according to claim 2, characterized in that, Based on the target calibration parameters, the image distribution features are spatially projected to obtain the first image features, including: Based on the target calibration parameters, the image distribution features are spatially projected to obtain the first projection coordinate point; The first projection coordinate point is mapped to the first space and the second space respectively to obtain the first feature and the second feature, wherein the first space is composed of multiple rectangular units and the second space is composed of multiple sector units; The first image feature is obtained based on the first feature and the second feature.

4. The method according to claim 3, characterized in that, Based on the first feature and the second feature, the first image feature is obtained, including: The second feature is mapped to the first space to obtain the third feature; The first image feature is obtained by weighted fusion of the first feature and the third feature.

5. The method according to claim 2, characterized in that, Based on the target calibration parameters, the depth distribution features are projected onto multiple planes to obtain second image features, including: Based on the target calibration parameters, the depth distribution features are projected onto multiple planes to obtain multiple second projection coordinate points, wherein different second projection coordinate points are located on different height planes; The positional encoding of the plurality of second projection coordinate points is performed to obtain the second image feature.

6. The method according to claim 2, characterized in that, The target image features are obtained based on the first image features and the second image features, including: The first image feature and the second image feature are stitched together to obtain the first stitched image feature; Multi-scale feature enhancement is performed on the features of the first stitched image to obtain the enhanced image features of the target; Based on the first stitched image features and the target enhanced image features, the second stitched image features are obtained; The first stitched image features, the second stitched image features, the first image features, and the second image features are weighted and fused to obtain the target image features.

7. The method according to claim 6, characterized in that, Based on the first stitched image features and the target enhanced image features, a second stitched image feature is obtained, including: The features of the first stitched image are projected onto a preset grid cell to obtain multiple image pixels; A distance confidence map is determined based on the average variance of the plurality of image pixels, wherein the distance confidence map is used to represent the confidence level of the distance represented by different image pixels; The distance confidence map and the target enhanced image features are stitched together to obtain the second stitched image features.

8. The method according to claim 6, characterized in that, Multi-scale feature enhancement is performed on the features of the first stitched image to obtain the target enhanced image features, including: Multi-scale feature enhancement is performed on the features of the first stitched image to obtain multiple enhanced image features, wherein the number of features of different enhanced image features is different; Global average pooling is performed on the multiple enhanced image features to obtain the fusion weights of the multiple enhanced image features; The target enhanced image features are obtained by weighted fusion of the multiple enhanced image features based on the fusion weights.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Obtain the initial calibration parameters of the vehicle data acquisition device; Depth prediction is performed on the pixels in the image data to obtain the depth parameters of the pixels, wherein the depth parameters are used to represent the distance between the object corresponding to the pixel and the vehicle; The initial calibration parameters are corrected based on the depth parameters to obtain the target calibration parameters.

10. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 9.

Images