A method, system, device, and medium for multimodal fusion of fisheye images and point clouds.
By fusing fisheye images and point clouds in the feature domain and combining them with body posture information for adaptive distortion correction, the accuracy and real-time performance issues of fusing fisheye images and LiDAR point clouds in quadrupedal robot dogs are solved, thereby improving environmental perception and obstacle avoidance capabilities.
Patent Information
- Application Number
- CN202611255904.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-08-19
AI Technical Summary
In the application of quadruped robot dogs, existing technologies suffer from problems such as low alignment accuracy in near-field strong distortion regions, large loss of edge details, poor dynamic posture adaptability, and difficulty in balancing accuracy and real-time performance when fusing fisheye images with LiDAR point clouds.
By shifting distortion compensation from the pixel domain to the feature domain, introducing fisheye geometric priors, and combining real-time body posture information, multi-scale feature extraction and adaptive distortion correction are used to achieve synergistic optimization of distortion correction and cross-modal fusion.
While maintaining the integrity of the fisheye field of view information, the accuracy of environmental perception and obstacle avoidance of the quadruped robot dog in dynamic undulating terrain has been improved, the training cost and latency have been reduced, and the perception update rate requirement of more than 10Hz has been met.
Smart Images

Figure CN122820462B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of embodied intelligence, and in particular to a method, system, device, and medium for multimodal fusion of fisheye images and point clouds. Background Technology
[0002] Quadruped robot dogs have demonstrated great potential in recent years for tasks such as inspection in complex environments, disaster relief, and logistics delivery. One of their core capabilities is the ability to move stably and autonomously on unstructured terrain, which places extremely high demands on the environmental perception system: it needs to simultaneously acquire the fine geometry of the ground (for foothold selection) and the semantic information of obstacles ahead (for obstacle avoidance and path planning).
[0003] Multimodal perception fusion is a key technological approach to improving the environmental understanding capabilities of quadruped robots. Cameras provide rich textures and semantics (such as recognizing stairs, puddles, and grass), while LiDAR provides high-precision 3D geometry (such as terrain undulations and obstacle distances). Fusion of these two technologies can compensate for the shortcomings of a single sensor.
[0004] Chinese patent application CN119942290A discloses a method, system, and product for multi-sensor feature fusion. However, this solution is designed for autonomous vehicles and deals with the slight distortion of ordinary vehicle cameras, and is not applicable to the application field of quadruped robot dogs.
[0005] In practical applications of quadruped robot dogs, fisheye cameras have unique advantages due to their ultra-wide field of view (usually ≥180°): they can simultaneously cover the robot dog's lower front (near-field landing area), front (obstacles), and sides (passable space), making them particularly suitable for legged robots' need for integrated near / far-field perception of their surroundings in 360° or a large frontal area; compared to multiple ordinary cameras, fisheye cameras reduce system hardware costs and calibration complexity.
[0006] However, fisheye imaging suffers from significant barrel distortion, which is particularly severe in the image edge regions (corresponding to large incident angles, i.e., the near-body region). This leads to the following challenges unique to quadrupedal robot dogs when spatially aligning fisheye images with LiDAR point clouds: (1) Real-time changes in body posture: When the quadruped robot dog walks, its body pitches, tilts, and undulates, causing dynamic shifts in the combined external parameters of the camera and lidar, which are difficult to adapt to using traditional fixed calibration methods.
[0007] (2) Strong distortion in near-field perception: The robot dog needs to perceive the terrain within a range of 0.2~1.5m under its feet. This area corresponds to the edge region of the large incident angle of the fisheye image, where the distortion is most severe and the cross-modal correspondence is extremely unstable.
[0008] (3) High real-time requirements: Leg-based control requires a perception update rate of more than 10Hz. The fusion method must be lightweight and efficient, while most existing fusion methods have high computational load and high inference latency.
[0009] (4) Blurred terrain edges: Pixel domain distortion correction will cause the image edge area to be blurred by resampling, which will degrade important geometric features such as the edge of the steps and the boundary of the stones, affecting the accuracy of the landing point selection.
[0010] Based on this, existing technologies have formed two main routes similar to autonomous driving but with slightly different focuses.
[0011] Option 1: Image-level distortion correction followed by fusion (the most common engineering approach).
[0012] This approach first performs panoramic unfolding or perspective correction on fisheye images based on camera calibration parameters (intrinsic parameters and distortion coefficients), converting them into approximately distortion-free images. These images are then input into a conventional multimodal fusion network for alignment and fusion with the point cloud. The core idea is to first transform the fisheye image into a normal image, and then apply a mature fusion framework.
[0013] The main drawback of this type of solution in quadruped robot dog applications is: (1) Relying on high-precision calibration, the calibration error will be directly transmitted to the fusion stage, and the fixed calibration parameters cannot adapt to the pitch and roll changes of the robot dog during its movement; (2) Image-level resampling inevitably leads to blurring, artifacts and loss of detail, especially in the near field (image edge) area, making it difficult for the robot dog to distinguish the edge of a 5cm step or a small stone under its feet. (3) The effective field of view after correction is usually reduced (e.g., from 180° to below 120°), the lower side and foot area are lost, and the blind spot is increased; (4) The correction and fusion are decoupled, making it difficult to achieve end-to-end joint optimization around the final fusion task.
[0014] Option 2: Directly use the original fisheye image for end-to-end fusion (implicit adaptation route).
[0015] This approach avoids explicit image correction, directly inputting the original fisheye image and point cloud into the network, relying on the network to implicitly learn distortion adaptation capabilities during training. Its core idea is to let the network learn to handle fisheye distortion on its own.
[0016] While this type of solution avoids the traditional two-stage process, it still exists in the context of quadruped robot dogs: (1) When there is a lack of geometric priors, the network needs a large amount of real terrain interaction data to learn stable distortion invariance, which results in high training costs; (2) Implicit learning has poor interpretability and poor cross-modal alignment consistency in areas of strong distortion (near the fuselage), which can easily lead to the robot dog misjudging the passability (such as misjudging the inclined ground as an obstacle). (3) A large amount of network capacity is consumed in distortion compensation learning, which squeezes out the ability to model terrain semantics and limits the upper limit of fusion accuracy; (4) Even if an online calibration branch is introduced, if the correction is still performed in the pixel domain, it is still difficult to avoid resampling loss and field of view reduction, and additional training instability factors will be introduced.
[0017] Therefore, there is an urgent need for a fisheye image and point cloud fusion solution that can solve the problems of low alignment accuracy in near-field strong distortion regions, large loss of edge details, poor dynamic pose adaptability, and difficulty in balancing accuracy and real-time performance in existing technologies. Summary of the Invention
[0018] The purpose of this invention is to address the problems of existing methods for multimodal perception using fisheye cameras and LiDAR when quadrupedal robot dogs operate in complex terrains (such as stairs, gravel, and grass). These problems include the loss of near-field details and field of view due to pixel-domain distortion correction of fisheye images, sensitivity of cross-modal alignment to changes in robot posture, and instability in terrain edge perception due to the lack of geometric priors in implicit fusion. This invention provides a method, system, device, and medium for multimodal fusion of fisheye images and point clouds. It shifts distortion compensation from the pixel domain to the feature domain, introduces fisheye geometric priors, and combines them with the real-time posture information of the quadrupedal robot dog. This achieves coordinated optimization of correction and fusion under dynamic motion, thereby achieving deep coupling of distortion correction and cross-modal fusion while preserving the integrity of the large field of view of the fisheye (especially the near-field area around the robot body). This improves the environmental perception accuracy, terrain accessibility judgment ability, and obstacle avoidance robustness of the quadrupedal robot dog in dynamically undulating terrain.
[0019] The objective of this invention can be achieved through the following technical solutions: A multimodal fusion method for fisheye images and point clouds includes the following steps: Acquire fisheye images and point cloud data synchronously collected on the quadruped robot dog, as well as the current body posture information; Multi-scale feature extraction is performed on the fisheye image to obtain a multi-scale feature map, and a geometric coding map of the corresponding scale is constructed based on the incident angle and radial distance. Based on the geometric coding map and the current body posture information, adaptive feature-level distortion correction is performed for each scale of the multi-scale feature map to obtain multi-scale correction features; Point cloud features are extracted based on the point cloud data; Perform coordinate transformation, projection transformation, and geometric transformation on point cloud data, and establish cross-modal correspondences based on point cloud features; Based on the cross-modal correspondence, the multi-scale correction features and point cloud features are weighted and fused to obtain fused features; The fused features are decoded based on the shared decoding backbone, and the corresponding prediction head is selected according to the task requirements to output the corresponding prediction results.
[0020] The adaptive feature-level distortion correction specifically includes the following steps: The normalized current body posture information is combined into a global posture vector. The low-dimensional geometric code of the geometric coding map corresponding to each scale at each spatial location is concatenated with the global posture vector and mapped to a geometric condition vector with the same number of feature channels as the feature map of the corresponding scale. Using the geometric condition vector as the modulation condition, feature modulation is performed on the feature map at the corresponding scale to obtain the modulated features corresponding to each scale; Modulated features are used as input features for deformable convolution, and geometric condition vectors of the corresponding scale are used as offset prediction conditions. The offset of each sampling point of the deformable convolution sampling kernel is predicted through a lightweight offset prediction sub-network. Then, the intermediate features after scale correction are obtained by deformable convolution operation. At the same time, a regularization term based on the physical law of fisheye distortion is introduced to constrain offset learning. By using residual connections, the intermediate features corrected at each scale are added to the original feature maps of the corresponding scales to obtain multi-scale corrected features.
[0021] The method for constructing the geometric coding map is as follows: first, at the original resolution, calculate the incident angle and radial distance of each pixel in the original feature map to construct a baseline geometric coding map, and then generate the corresponding scale geometric coding map according to the downsampling ratio of each scale.
[0022] The establishment of cross-modal correspondence specifically includes: Coordinate transformation: The rigid body transformation extrinsic parameters between the fisheye camera and the lidar are obtained in advance through offline joint calibration. The rigid body transformation extrinsic parameters are used to transform each point cloud point in the point cloud data to the instantaneous camera coordinate system. Projection transformation: The point cloud points transformed to the instantaneous camera coordinate system are projected onto the image plane coordinates through a fisheye projection model; Geometric transformation: Based on the image plane coordinates, the points falling within the effective field of view of the camera are converted into the corresponding feature layer coordinates according to the downsampling ratio of the current feature layer. The geometric coding map of the corresponding scale is queried according to the feature layer coordinates to obtain the geometric conditions of the projection position. Establish cross-modal correspondence: Construct cross-modal correspondence by combining point cloud points, point cloud features, image plane coordinates, and geometric conditions of projection positions.
[0023] In the weighted fusion process, the weighting fusion weights are determined as follows: The geometric conditions of the projection positions in the cross-modal correspondence are input into a lightweight gated network, which outputs unnormalized weights corresponding to image features and point cloud features, respectively. After normalization, weighted fusion weights corresponding to image features and point cloud features are obtained. The weighted fusion weights are point-by-point weights generated for each valid point cloud projection position. Image pixels without corresponding point cloud projections do not participate in the weighted fusion.
[0024] The fusion feature is one of the following: point-by-point fusion feature, BEV fusion feature map, unified fusion feature, or multi-scale fusion feature set. The BEV fusion feature map is obtained by backfilling the point-by-point fusion feature to the BEV mesh based on the three-dimensional coordinates of the point cloud points. The unified fusion feature is obtained by aggregating the point-by-point fusion features of the same point cloud point at all scales, using the point cloud points as the classification dimension. The multi-scale fusion feature set is obtained by packaging the point-by-point fusion features of all point cloud points at each scale into an independent feature group, using scale as the classification dimension, and combining the feature groups corresponding to all scales.
[0025] The task prediction header includes one or more of the following prediction headers: Terrain accessibility prediction head: used to output the walkable probability of each grid in the bird's-eye view feature space or camera projection space; Landing point heatmap prediction head: It consists of convolutional layers and heatmap regression layers, and is used to output a heatmap of suitable landing locations and their confidence scores; Obstacle detection head: It consists of a classification branch and a regression branch. The classification branch determines the type of obstacle or whether an obstacle exists, while the regression branch outputs the obstacle boundary, contour points, or minimum bounding box parameters. Terrain type classification header: Used to output the terrain type classification results.
[0026] A multimodal fusion system for fisheye images and point clouds, the system comprising: Data acquisition module: used to acquire fisheye images and point cloud data synchronously collected on the quadruped robot dog, as well as the current body posture information; Fisheye Image Multi-Scale Feature Extraction and Geometric Coding Map Construction Module: Used to extract multi-scale features from the fisheye image to obtain a multi-scale feature map, and construct a geometric coding map of the corresponding scale based on the incident angle and radial distance; Adaptive feature-level distortion correction module: Based on the geometric coding map and the current body posture information, it performs adaptive feature-level distortion correction for each scale of the multi-scale feature map to obtain multi-scale corrected features; Point cloud feature extraction module: used to extract point cloud features based on the point cloud data; The fusion module is used to perform coordinate transformation, projection transformation, and geometric transformation on point cloud data, and to establish cross-modal correspondences based on point cloud features; based on the cross-modal correspondences, the multi-scale correction features and point cloud features are weighted and fused to obtain fused features; Output module: used to decode the fused features based on the shared decoding backbone, and select the corresponding task prediction head to output the corresponding prediction result according to the task requirements.
[0027] An electronic device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the fisheye image and point cloud multimodal fusion method.
[0028] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the fisheye image and point cloud multimodal fusion method.
[0029] Compared with the prior art, the present invention has the following beneficial effects: (1) Preserve the near-field large field of view information of fisheye and improve the accuracy of footing point perception: Based on the incident angle and radial distance, the present invention constructs a geometric coding map corresponding to each scale, performs adaptive distortion correction at the feature level, rather than pixel-level resampling to remove distortion, avoids edge blurring and artifacts caused by pixel domain resampling, and effectively preserves important geometric features such as step edges and stone boundaries, that is, it completely preserves the information of the side and foot area, eliminates the perception blind spot, and enables the robot dog to clearly distinguish the step edge, stone and terrain undulation 5cm away from the foot.
[0030] (2) High cross-modal alignment stability under dynamic motion: This invention combines the current body posture information with the geometric coding map, and performs adaptive feature-level distortion correction for each scale of the multi-scale feature map. It can compensate for the sensor pose shift caused by the pitch, tilt, undulation and other posture changes during the walking of the quadruped robot dog in real time. The correction is linked with the posture, which breaks through the limitation of fixed calibration extrinsic parameters and effectively suppresses the image and point cloud misalignment caused by the body shaking when the quadruped robot dog walks. It can maintain stable fusion even when running fast or climbing uphill.
[0031] (3) Significantly improved perception reliability in areas with strong distortion (foot area): This invention guides the network to rely more on point cloud geometric information in the near field area through geometric coding map and physical regular constraints, avoiding misleading by distorted images and reducing the risk of misjudgment.
[0032] (4) High training efficiency and strong interpretability: This invention directly models the essence of fisheye imaging with geometric prior (incident angle + radial distance), and the correction accuracy is higher than that of network implicit learning. Moreover, it does not require massive training data. The introduction of physical prior regularization reduces the dependence on massive real terrain interaction data and shortens the training cycle before the quadruped robot dog is deployed in the field.
[0033] (5) The present invention performs coordinate transformation and projection transformation on point cloud data, establishes explicit cross-modal correspondence based on point cloud features, and performs weighted fusion based on the correspondence. The pixel-point cloud level explicit correspondence provides accurate geometric constraints. The cross-modal alignment consistency of near-field strong distortion region is far better than the implicit learning scheme. The weighted fusion mechanism can suppress the erroneous fusion of low confidence region at fisheye edge and reduce the risk of terrain misjudgment.
[0034] (6) The shared decoding backbone of the present invention greatly improves the feature reuse rate between multiple tasks, avoids repeated calculations of each task, and has a lightweight and efficient overall architecture. It can decode and output terrain accessibility maps and landing point heat maps without intermediate conversion, reducing the total latency from perception to control. It can meet the requirements of a perception update rate of more than 10Hz for legged control and is suitable for deployment on the edge computing platform of quadruped robot dogs. Attached Figure Description
[0035] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the multi-scale feature extraction and geometric coding map construction of fisheye images according to the present invention; Figure 3 This is a schematic diagram of the adaptive feature-level distortion correction process of the present invention; Figure 4 This is a schematic diagram illustrating the process of constructing cross-modal correspondences and feature fusion in this invention; Figure 5 This is a schematic diagram illustrating the correspondence between fusion weights and distortion levels in this invention; Figure 6 This is a graph showing the changes in the losses of four auxiliary tasks as a function of the global training steps in one embodiment. Figure 7 This is a training variation curve of the absolute offset and regression offset of the DAFRM module in one embodiment; Figure 8 This is a graph showing the change in the success rate of landing point prediction under various complex terrains as a function of training rounds in one embodiment. Figure 9 This is a schematic diagram of the electronic device structure of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.
[0038] Figure 1 This is a flowchart illustrating a multimodal fusion method for fisheye images and point clouds according to an embodiment of the present invention. Figure 1 As shown, the fisheye image and point cloud multimodal fusion method in this embodiment of the invention may include the following steps: S1, Multimodal data acquisition and preprocessing.
[0039] S11, Multimodal Data Acquisition.
[0040] In this embodiment, a fisheye camera (horizontal FOV ≥ 180°, top-down installation, covering the area in front of and below the robot body) and a lidar (such as solid-state or mechanical type, used for terrain geometry acquisition) are fixedly mounted on the quadruped robot dog body. The fisheye camera and point cloud data are acquired simultaneously, and the onboard IMU (Inertial Measurement Unit) is connected to obtain the robot dog's current attitude information, including the current attitude angles (including pitch angle). and roll angle ) and fuselage height All sensors are synchronized via hardware timestamps.
[0041] S12, Preprocessing.
[0042] Point cloud preprocessing: processing the raw point cloud data acquired by the lidar. Voxel downsampling is performed, and motion compensation is applied to the point cloud using the current body posture information to eliminate projection errors caused by time synchronization errors and body sway, resulting in a preprocessed point cloud. .
[0043] Fisheye image preprocessing: The fisheye image is only normalized in size, without any pixel-domain distortion correction.
[0044] S2, Multi-scale feature extraction and geometric coding map construction of fisheye images.
[0045] like Figure 2 As shown, the specific process is as follows: S21, Multi-scale feature extraction.
[0046] Extract using image backbone network Feature maps at each scale ,in, Indicates the first Feature maps at various scales. The image backbone network can be a convolutional neural network (CNN) or a vision transformer (ViT) structure, such as ResNet, SwinTransformer, etc. This embodiment does not make specific limitations on the type of backbone network used.
[0047] Features at different scales are used to balance local details and global semantics, providing hierarchical information for subsequent cross-modal fusion.
[0048] S22, Geometric coding graph construction.
[0049] To address the nonlinear distortion characteristics of fisheye imaging, this invention constructs a geometric encoding map (GEM) at each pixel location, encoding two types of physical quantities: (1) Incident angle information , which corresponds to the angle between the pixel in the fisheye image and the direction of the real light ray, where, Represents the coordinates of a pixel's position; (2) Radial distance information .
[0050] In particular, in the application of quadruped robot dogs, the area with an incident angle > 60° corresponds to the near-field range within 0.5m around the robot body, which is the key area for landing point perception.
[0051] like Figure 2 As shown, let the coordinates of the image center be... Then the pixel The radial distance is Based on radial distance and fisheye projection model inverse function Calculate the angle of incidence The incident angle and radial distance calculated based on the entire feature map are then normalized to obtain the normalized incident angle. and normalized radial distance .
[0052] First, for the fisheye image at its original resolution, based on the normalized incident angle of each pixel... and normalized radial distance Construct a baseline geometric coding map Then, according to the multi-scale feature map, the first... Downsampling factor at each scale Generate the corresponding first Geometric coding diagram at each scale .
[0053] In this embodiment, downsampling is preferably performed using area interpolation to maintain physical consistency in the sense of geometric average.
[0054] Based on this step, the fisheye geometry prior is explicitly obtained before distortion correction, rather than relying entirely on data-driven implicit learning.
[0055] S3, Adaptive feature-level distortion correction.
[0056] This step constructs a Distortion-Aware Feature Rectification Module (DAFRM). This module does not process features at a single scale, but performs feature-level correction on each scale feature map output from step S2. Specifically, for the... At each scale, the input to the DAFRM module is image features. Geometric coding map consistent with the spatial resolution at this scale In addition to the real-time attitude information of the aircraft, the output is a correction feature at that scale. .right After performing the same processing using the DAFRM module, a multi-scale corrected feature set was obtained. This allows subsequent cross-modal projection sampling and fusion modules to call upon the data at different scales. For example... Figure 3 As shown, the DAFRM module performs the following steps to achieve feature correction: S31, Geometric Condition Embedding.
[0057] The normalized body posture ( Combined into a global attitude vector ,in, These are the normalized pitch angle, roll angle, and fuselage height, respectively. A multi-layer perceptron (MLP) is used to process the... Geometric coding diagram at each scale In each spatial location Low-dimensional geometrically encoded feature vectors and After concatenation, it is mapped to a geometric condition vector with the same number of feature channels as the corresponding scale feature map. This vector serves as a conditional input for subsequent modulation and offset prediction.
[0058] S32, Feature Modulation.
[0059] The FiLM (Feature-wise Linear Modulation) mechanism is used to modify the geometric condition vector. Perform position-dependent affine transformations: ; in, , These represent scaling and translation mapping functions, respectively. This represents element-wise multiplication. For the multi-scale feature map, the first Feature maps at each scale In spatial location The feature vector at that location, For the first Feature maps at each scale In spatial location The modulated feature vector at that location.
[0060] This step employs the FiLM mechanism, enabling the network to adopt differentiated channel response strategies in different distortion regions, and innovatively incorporates geometric condition vectors. As a modulation condition, spatial position-dependent local FiLM modulation is realized, so that the modulation parameters change synchronously with the spatial position and the body posture, thereby enabling dynamic adjustment of the modulation intensity according to the body posture, and adapting to the real-time correction processing during the dynamic movement of the quadruped robot dog.
[0061] S33, deformable convolution correction and physical regularization.
[0062] Obtain the modulated features in step S32 After that, with As input features for deformable convolution, geometric conditional vectors As a condition for offset prediction, feature-level spatial sampling correction is performed. Specifically, a lightweight offset prediction subnetwork is first used to predict deformable convolution offsets. Specifically, a lightweight offset prediction subnetwork is set up for each scale, consisting of two layers of 3×3 or 1×1 convolutions, with the geometric condition vector as input. Output the offset corresponding to the deformable convolution sampling kernel. : ; in, The first term in the deformable convolution sampling kernel represents the... One sampling point, Indicates the first A lightweight migration prediction subnetwork at each scale, Indicates the first Each scale in spatial location The corresponding number The offset of the nth sampling point. Scale, spatial location At this point, the corrected intermediate features are represented as follows: ; in, Indicates the first Each scale in spatial location The intermediate features after correction. Indicates the first A fixed sampling offset for a regular convolution kernel at each sampling point. , They represent the first Fixed horizontal sampling offset and fixed vertical sampling offset for each sampling point Indicates the first The convolution kernel weights for each sampling point and They represent the first Each scale in spatial location The corresponding number The horizontal and vertical learning offsets are calculated for each sampling point. In this embodiment, feature values are obtained for non-integer sampling positions through bilinear interpolation. Therefore, It is the direct input to deformable convolution. Used to predict sampling offset, both together determine the feature-level distortion correction result.
[0063] To avoid unconstrained divergence in offset learning, this embodiment introduces a physical prior regularization term. : ; in, For the first Each scale in spatial location The corresponding number The expected offset of each sampling point is calculated from the expected offset prior derived from the fisheye model. This is the regularization coefficient. The expected offset can be determined based on the back-projection relationship from the pixel to the incident angle and the feature layer downsampling rate. And the radial distortion law of the fisheye is obtained and scaled to the th Each scale corresponds to a coordinate unit in the feature layer.
[0064] This constraint aligns the offset learning direction with the physical laws of fisheye distortion: generally, the closer to the edge, the greater the expected compensation; the closer to the center, the smaller the expected compensation. Simultaneously, body posture information is transmitted through... Indirectly affects offset prediction (e.g., when looking down, the near-field region moves toward the image center, and the offset is adjusted accordingly).
[0065] S34, Residual Connections and Multilayer Outputs.
[0066] Perform the above correction process S31~S33 on each scale to obtain the corrected intermediate features. In this embodiment, residual connection is preferably used to add the intermediate feature and the original multi-scale feature map to obtain the multi-scale corrected feature output. each All correspond to the geometric coding diagram of the corresponding scale Maintain consistent spatial resolution. Specifically, for the ... Each scale, correction feature The calculation method is as follows: .
[0067] Subsequent steps S5 and S6 can be performed based on the point cloud projection location and the selected fusion scale, using the corresponding correction features. Mid-sampled image features; if only a single fusion scale is subsequently used, multiple features can be combined using a feature pyramid network or upsampling / downsampling. The features are then aggregated to obtain corrected image features at a uniform scale. .
[0068] S4, Point Cloud Feature Extraction and Spatial Coding.
[0069] This step uses backbones such as PointNet++, PointPillars, VoxelNet, or SparseConvNet (sparse convolutional network) to extract point features or bird's-eye view features as point cloud features.
[0070] S5, Cross-modal feature projection and alignment.
[0071] The purpose of this step is to provide directly usable cross-modal index relationships for subsequent S6 operations, rather than generating the final fused features independently. Figure 4 As shown, the specific process is as follows: S51 obtains camera-LiDAR extrinsic parameters through offline joint calibration. This extrinsic parameter can be calibrated using a checkerboard, calibration board, reflective markers, or point, line, and surface constraints in a natural scene. It represents the rigid body transformation relationship from the lidar coordinate system to the camera coordinate system, including the rotation matrix. Translation vector After calibration, the external parameters are saved as system parameters and can be directly called during runtime.
[0072] Fisheye projection model The intrinsic parameters are determined by the fisheye camera's intrinsic parameters and distortion model. The intrinsic parameters include focal length, principal point coordinates, and fisheye distortion parameters, which can be obtained through the fisheye camera calibration process; the fisheye projection model can be an equidistant projection, a fixed-angle projection, an orthographic projection, or a polynomial fisheye model.
[0073] For the Point cloud points First, perform the preprocessing in step S1 to compensate for the body's motion, then use external parameters. Transformation to obtain point coordinates in the instantaneous camera coordinate system Complete the coordinate transformation: ; in, This indicates that in step S1, the current... The body motion compensation transformation parameters are obtained from the IMU attitude and fuselage height at each moment. Then, the corresponding image plane coordinates are obtained through a fisheye projection model. Complete the projection transformation: ; For points falling within the effective field of view of the fisheye, according to the first... Downsampling factor at each scale Convert the original image coordinates to the corresponding feature layer coordinates: ; in, Indicates the first The corresponding scale is the first Feature layer coordinates of a point cloud.
[0074] Then according to Query the corresponding geometric code image The geometric conditions for obtaining the projection position are obtained. Perform geometric transformations, where, Indicates the first The corresponding scale is the first The angle of incidence of a point cloud point Indicates the first The corresponding scale is the first The radial distance of a point cloud point.
[0075] Therefore, cross-modal correspondences are established by combining point cloud features. ,in, Indicates the first The point cloud features of each point cloud point are the one-to-one correspondence between "point cloud point - point cloud feature - image sampling position - geometric conditions".
[0076] This cross-modal correspondence It will be used in S6 for three operations: First, according to From correction features Mid-sampled image features; second, based on The first step is to generate fusion weights for the image modality and the point cloud modality; the second step is to backfill the fused point-by-point features into the point cloud structure or BEV mesh. Therefore, S5 is the index and geometric condition generation step for performing geometrically perceptual fusion, which is performed by S6.
[0077] S6, distortion-aware cross-modal feature fusion.
[0078] This step constructs a Distortion-Aware Cross-Modal Fusion Module (DACFM) to perform cross-modal feature fusion, with the input being the corrected features. Point cloud features and cross-modal correspondence Output fused features. For example... Figure 4 As shown, DACFM specifically performs the following sub-steps to achieve feature fusion: S61, Image feature sampling and feature stitching.
[0079] For each valid point cloud point According to the cross-modal correspondence Projected coordinates in Corrected features at the corresponding scale are obtained through bilinear interpolation. Medium-sampled image features If the number of channels in the image features and the point cloud features are different, they are mapped to the same channel dimension through 1×1 convolutions or linear layers, respectively. d Obtain the aligned image features and point cloud features The dimension alignment in this step is to ensure that the subsequent weighted summation can be performed.
[0080] S62, Geometric Condition Gating and Weight Generation.
[0081] Cross-modal correspondence Geometric conditions of the projection position in Input the lightweight gated network GateNet. Optionally, aligned image features and point cloud features can be concatenated and then input into the gated network to enhance the context-awareness of weight prediction. The gated network outputs unnormalized weights corresponding to the image features and point cloud features, respectively. , The weighted fusion weights are obtained by using the softmax function: ; in, Representing the image features and point cloud features respectively for the first... The fusion weights of the point cloud points, and This weight is a point-by-point weight generated for each valid point cloud projection location, rather than a pixel-by-pixel summation over all pixels in the entire image. Image pixels without corresponding point cloud projections do not participate in this point-by-point fusion step; if a dense image spatial result is required, it can be achieved through BEV backfilling or upsampling in the subsequent decoding stage.
[0082] Figure 5 The weight distribution is illustrated: in areas with large incident angles and strong distortion (corresponding to the near field of the quadruped robot dog's feet), unstable image information weights can be automatically suppressed, while in areas with small incident angles and weak distortion, the contribution of image information can be increased, thereby achieving position-adaptive fusion.
[0083] S63, weighted fusion and output.
[0084] Perform point-by-point fusion on the dimension-aligned features from step S61 according to the point-by-point fusion weights determined in step S62, to obtain the feature corresponding to the first... The first point cloud point Point-by-point fusion features at each scale : ; Among them, the cross-modal correspondence obtained in step S5 This step is used simultaneously to determine the image feature sampling location, acquire the geometric gating input, and determine the point cloud point or BEV mesh location corresponding to the fused features. Point-by-point feature fusion. The point cloud points can be retained as point-by-point fusion features based on their indexes, or the 3D coordinates of the point cloud points can be used to backfill the BEV mesh to form a BEV fusion feature map. Alternatively, multi-scale fusion can be used, which involves fusing the point-by-point features obtained at different scales. By splicing, summing, or focusing attention, a unified and integrated feature is obtained. At the same time, it can also be measured by scale. To classify the features, the features of all laser points at each scale are fused point by point. Packed into a single feature group The multi-scale fusion feature set is composed of feature groups corresponding to all scales.
[0085] In one preferred embodiment, residual connections and layer normalization can be used to enhance optimization stability, and the final output can be a point-by-point fusion feature, a BEV fusion feature map, a unified fusion feature, or a multi-scale fusion feature set for decoding in step S7.
[0086] S7, task decoding and prediction output.
[0087] The pointwise fused features, BEV fused feature maps, unified fused features, or multi-scale fused feature sets obtained from S6 are input into the task decoding module. The task decoding module preferably employs a structure combining a shared decoding backbone and a multi-task prediction head, rather than generating all results directly and indiscriminately from a single feature map. The shared decoding backbone can consist of several convolutional layers, deconvolutional layers, upsampling layers, or Transformer decoding layers, used to convert the fused features into shared decoding features with a unified spatial resolution. .
[0088] In shared decoding features Based on this, set one or more task prediction headers according to application requirements: (1) Terrain accessibility prediction head: It consists of convolutional layers and sigmoid / softmax classification layers, and outputs the walkability probability (0~1) of each grid in BEV space or camera projection space. (2) Landing point heatmap prediction head: It consists of a convolutional layer and a heatmap regression layer, and outputs a heatmap of the suitable landing location and its confidence level; (3) Obstacle detection head: It consists of a classification branch and a regression branch. The classification branch determines the type of obstacle or whether there is an obstacle. The regression branch outputs the obstacle boundary, contour points or minimum bounding box parameters. (4) Terrain type classification header: It consists of convolutional layer, pooling layer and fully connected layer, and can output ground types such as stairs, gravel, grass, flat ground, etc.
[0089] When the point cloud branch outputs bird's-eye view features (BEV), the point-by-point fused features can be first backfilled into the BEV mesh according to the 3D coordinates of the point cloud to form... Then generated by the BEV decoding backbone. It is then connected to the aforementioned task prediction head. When using a multi-scale fusion feature set, a unified decoding feature can be formed first through feature pyramid fusion, upsampling and splicing, or attention convergence, before prediction is performed.
[0090] The aforementioned multiple task prediction heads can be enabled simultaneously, or one or more can be selected for activation based on actual deployment needs. Their output can be directly interfaced with the quadruped robot dog's motion planner (such as MPC or reinforcement learning strategies).
[0091] This embodiment constructs a joint loss during the training phase. : ; in, For balance coefficient, For physical prior regularization loss, This is a loss for the mission.
[0092] If only a single task prediction head is enabled, the task loss will be... This represents the loss corresponding to the task; if multiple task prediction heads are enabled simultaneously, the task loss will be... The weighted sum of losses from multiple tasks does not mean that only one result can be output at a time. Specifically, it can be represented as: ; in, This represents the segmentation or classification loss of the terrain accessibility map. This represents the regression loss of the landing point heatmap. This represents the classification and boundary regression loss for obstacle detection. This represents the terrain type classification loss; , , , The loss weights are assigned to each task. When a task's prediction head is not enabled, its corresponding weight can be set to 0 or removed from the loss function.
[0093] Through joint optimization, the DAFRM module, DACFM module, and task decoding module converge collaboratively around the final task objective, avoiding the problem of target fragmentation caused by first correcting and then fusing.
[0094] This invention has been validated in both a physical quadruped robot dog platform and a simulation environment. Physical platform: A quadruped robot dog equipped with a fisheye camera (190° FOV), a 16-line LiDAR, and an IMU was used to collect data for training and testing on real terrains such as stairs, gravel slopes, grasslands, and gullies. Simulation platform: Using a Gazebo+ quadruped robot model, complex terrains were randomly generated for large-scale strategy validation.
[0095] This embodiment primarily verifies four tasks: accessibility, landing point prediction, altitude perception, and terrain perception. Figure 6 The model training process demonstrates the convergence of the losses for the four auxiliary tasks. It can be seen that the losses for the four auxiliary tasks steadily decrease and tend to converge as the number of training steps increases, which can effectively achieve the learning of the corresponding tasks.
[0096] Figure 7 The offset evaluation results of the DAFRM module are presented, showing that both absolute offset and regression offset continuously decrease as training progresses. These two metrics characterize the error between the predicted offset and the true geometric offset. The continuous decrease in error indicates that the DAFRM module can gradually correct the spatial offset estimation, and its geometric inference ability continuously improves with training. Although noise jitter exists throughout the training process, the overall downward trend is clear, proving that this module can effectively complete the regression learning of spatial offset.
[0097] 1. Real-world terrain perception experiment This embodiment uses the success rate of landing point selection as the evaluation index, and the landing point prediction success rate results as a function of training rounds are shown in the figure below. Figure 8 As shown. By Figure 8 As can be seen, the success rate of model landing points significantly improved with each training iteration across all tested terrains. The overall success rate was best in scenarios involving ascending and descending stairs; while the initial success rate was lower for challenging terrains such as ravine crossings and scree slopes, significant performance gains were achieved after training. The synchronized performance trends across different terrains indicate that the model does not overfit to a single terrain but can generalize and adapt to various complex terrains. Furthermore, the results of the real-world terrain perception experiment, compared with existing solutions, are shown in Table 1. It is evident that the success rate of landing point selection in this invention is improved by an average of over 15%, particularly in areas with strong distortion in the near field (such as the first step of a staircase), where it significantly outperforms the comparative methods.
[0098] Table 1
[0099] 2. Ablation test (1) Module ablation (self-collected data, success rate) In this embodiment, a module ablation experiment was conducted on the proposed multimodal fusion method, and the success rate of landing point selection was used as the evaluation index. The results are shown in Table 2. The ablation results show that each module brings positive improvement, among which IMU attitude compensation contributes about 4% of the success rate improvement.
[0100] Table 2
[0101] (2) Regularity coefficient Sensitivity test This embodiment uses the landing success rate as the evaluation metric for regularization coefficients. The sensitivity experiment results are shown in Table 3. Optimal performance at that time.
[0102] Table 3
[0103] (3) Gating weight distribution In this embodiment, the gating weights for different incident angle intervals are shown in Table 4. This distribution indicates that image features in the small incident angle region (distant) are dominant, while point cloud features in the large incident angle region (near the fuselage and feet) are dominant, which is consistent with the fisheye distortion law and the quadruped robot dog's dependence on near-field geometry.
[0104] Table 4
[0105] 3. Real-time testing On the Jetson Orin edge computing platform, the method of this invention has a single-frame processing time of 35ms (image + point cloud), which meets the 20Hz real-time control requirements of a quadruped robot dog.
[0106] In summary, the feasibility of this invention has been proven through physical robot dog experiments, simulation verification, and system ablation analysis, and it has achieved better results than existing solutions in key indicators such as landing point selection and terrain passability judgment.
[0107] In this embodiment, the electronic device may be provided as a terminal, a server, or other form of device.
[0108] Figure 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 9 Electronic device 100 can be provided as a server or terminal device. (See reference...) Figure 9 The electronic device 100 includes a processing component 101, which further includes one or more processors, and memory resources represented by memory 103 for storing instructions, such as application programs, that can be executed by the processing component 101. The application programs stored in memory 103 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 101 is configured to execute instructions to perform the methods described above.
[0109] Electronic device 100 may further include a power supply component 102 configured to perform power management of electronic device 100, a wired or wireless network interface 104 configured to connect electronic device 100 to a network, and an input / output interface 105. Electronic device 100 can operate on an operating system stored in memory 103, such as a Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)
[0110] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 103 including computer program instructions that can be executed by a processing component 101 of an electronic device 100 to perform the above-described method.
[0111] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.
[0112] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0113] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0114] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.
[0115] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0116] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0117] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0119] The computer program product can be implemented in hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0120] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0121] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0122] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.
[0123] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A method for multimodal fusion of fisheye images and point clouds, characterized in that, Includes the following steps: Acquire fisheye images and point cloud data synchronously collected on the quadruped robot dog, as well as the current body posture information; Multi-scale feature extraction is performed on the fisheye image to obtain a multi-scale feature map, and a geometric coding map of the corresponding scale is constructed based on the incident angle and radial distance. Based on the geometric coding map and the current body posture information, adaptive feature-level distortion correction is performed for each scale of the multi-scale feature map to obtain multi-scale correction features; Point cloud features are extracted based on the point cloud data; Perform coordinate transformation, projection transformation, and geometric transformation on point cloud data, and establish cross-modal correspondences based on point cloud features; Based on the cross-modal correspondence, the multi-scale correction features and point cloud features are weighted and fused to obtain fused features; The fused features are decoded based on the shared decoding backbone, and the corresponding prediction head is selected according to the task requirements to output the corresponding prediction results. The adaptive feature-level distortion correction specifically includes the following steps: The normalized current body posture information is combined into a global posture vector. The low-dimensional geometric code of the geometric coding map corresponding to each scale at each spatial location is concatenated with the global posture vector and mapped to a geometric condition vector with the same number of feature channels as the feature map of the corresponding scale. Using the geometric condition vector as the modulation condition, feature modulation is performed on the feature map at the corresponding scale to obtain the modulated features corresponding to each scale; Modulated features are used as input features for deformable convolution, and geometric condition vectors of the corresponding scale are used as offset prediction conditions. The offset of each sampling point of the deformable convolution sampling kernel is predicted through a lightweight offset prediction sub-network. Then, the intermediate features after scale correction are obtained by deformable convolution operation. At the same time, a regularization term based on the physical law of fisheye distortion is introduced to constrain offset learning. By using residual connections, the intermediate features corrected at each scale are added to the original feature maps at the corresponding scales to obtain multi-scale corrected features; The regularization term based on the physical laws of fisheye distortion is expressed as follows: ; in, For regularization terms, Indicates the first Each scale in spatial location The corresponding number The offset of each sampling point For the first Each scale in spatial location The corresponding number The expected offset of each sampling point is calculated from the expected offset prior derived from the fisheye model. is the regularization coefficient.
2. The method for multimodal fusion of fisheye image and point cloud according to claim 1, characterized in that, The method for constructing the geometric coding map is as follows: first, at the original resolution, calculate the incident angle and radial distance of each pixel in the original feature map to construct a baseline geometric coding map, and then generate the corresponding scale geometric coding map according to the downsampling ratio of each scale.
3. The method for multimodal fusion of fisheye image and point cloud according to claim 1, characterized in that, The establishment of cross-modal correspondence specifically includes: Coordinate transformation: The rigid body transformation extrinsic parameters between the fisheye camera and the lidar are obtained in advance through offline joint calibration. The rigid body transformation extrinsic parameters are used to transform each point cloud point in the point cloud data to the instantaneous camera coordinate system. Projection transformation: The point cloud points transformed to the instantaneous camera coordinate system are projected onto the image plane coordinates through a fisheye projection model; Geometric transformation: Based on the image plane coordinates, the points falling within the effective field of view of the camera are converted into the corresponding feature layer coordinates according to the downsampling ratio of the current feature layer. The geometric coding map of the corresponding scale is queried according to the feature layer coordinates to obtain the geometric conditions of the projection position. Establish cross-modal correspondence: Construct cross-modal correspondence by combining point cloud points, point cloud features, image plane coordinates, and geometric conditions of projection positions.
4. The method for multimodal fusion of fisheye image and point cloud according to claim 3, characterized in that, In the weighted fusion process, the weighting fusion weights are determined as follows: The geometric conditions of the projection positions in the cross-modal correspondence are input into a lightweight gated network, which outputs unnormalized weights corresponding to image features and point cloud features, respectively. After normalization, weighted fusion weights corresponding to image features and point cloud features are obtained. The weighted fusion weights are point-by-point weights generated for each valid point cloud projection position. Image pixels without corresponding point cloud projections do not participate in the weighted fusion.
5. The method for multimodal fusion of fisheye image and point cloud according to claim 1, characterized in that, The fusion feature is one of the following: point-by-point fusion feature, BEV fusion feature map, unified fusion feature, or multi-scale fusion feature set. The BEV fusion feature map is obtained by backfilling the point-by-point fusion feature to the BEV mesh based on the three-dimensional coordinates of the point cloud points. The unified fusion feature is obtained by aggregating the point-by-point fusion features of the same point cloud point at all scales, using the point cloud points as the classification dimension. The multi-scale fusion feature set is obtained by packaging the point-by-point fusion features of all point cloud points at each scale into an independent feature group, using scale as the classification dimension, and combining the feature groups corresponding to all scales.
6. The method for multimodal fusion of fisheye image and point cloud according to claim 1, characterized in that, The task prediction header includes one or more of the following prediction headers: Terrain accessibility prediction head: used to output the walkable probability of each grid in the bird's-eye view feature space or camera projection space; Landing point heatmap prediction head: It consists of convolutional layers and heatmap regression layers, and is used to output a heatmap of suitable landing locations and their confidence scores; Obstacle detection head: It consists of a classification branch and a regression branch. The classification branch determines the type of obstacle or whether an obstacle exists, while the regression branch outputs the obstacle boundary, contour points, or minimum bounding box parameters. Terrain type classification header: Used to output the terrain type classification results.
7. A multimodal fusion system for fisheye images and point clouds, characterized in that, The system includes: Data acquisition module: used to acquire fisheye images and point cloud data synchronously collected on the quadruped robot dog, as well as the current body posture information; Fisheye Image Multi-Scale Feature Extraction and Geometric Coding Map Construction Module: Used to extract multi-scale features from the fisheye image to obtain a multi-scale feature map, and construct a geometric coding map of the corresponding scale based on the incident angle and radial distance; Adaptive feature-level distortion correction module: Based on the geometric coding map and the current body posture information, it performs adaptive feature-level distortion correction for each scale of the multi-scale feature map to obtain multi-scale corrected features; Point cloud feature extraction module: used to extract point cloud features based on the point cloud data; The fusion module is used to perform coordinate transformation, projection transformation, and geometric transformation on point cloud data, and to establish cross-modal correspondences based on point cloud features; based on the cross-modal correspondences, the multi-scale correction features and point cloud features are weighted and fused to obtain fused features; Output module: used to decode the fused features based on the shared decoding backbone, and select the corresponding task prediction head to output the corresponding prediction result according to the task requirements; The adaptive feature-level distortion correction specifically includes the following steps: The normalized current body posture information is combined into a global posture vector. The low-dimensional geometric code of the geometric coding map corresponding to each scale at each spatial location is concatenated with the global posture vector and mapped to a geometric condition vector with the same number of feature channels as the feature map of the corresponding scale. Using the geometric condition vector as the modulation condition, feature modulation is performed on the feature map at the corresponding scale to obtain the modulated features corresponding to each scale; Modulated features are used as input features for deformable convolution, and geometric condition vectors of the corresponding scale are used as offset prediction conditions. The offset of each sampling point of the deformable convolution sampling kernel is predicted through a lightweight offset prediction sub-network. Then, the intermediate features after scale correction are obtained by deformable convolution operation. At the same time, a regularization term based on the physical law of fisheye distortion is introduced to constrain offset learning. By using residual connections, the intermediate features corrected at each scale are added to the original feature maps at the corresponding scales to obtain multi-scale corrected features; The regularization term based on the physical laws of fisheye distortion is expressed as follows: ; in, For regularization terms, Indicates the first Each scale in spatial location The corresponding number The offset of each sampling point For the first Each scale in spatial location The corresponding number The expected offset of each sampling point is calculated from the expected offset prior derived from the fisheye model. is the regularization coefficient.
8. An electronic device, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the fisheye image and point cloud multimodal fusion method according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the multimodal fusion method of fisheye image and point cloud as described in any one of claims 1-6.
Citation Information
Patent Citations
Multi-sensor feature fusion method, system and product
CN119942290A
Autonomous navigation and intelligent obstacle avoidance method of bionic robot
CN121232834A
Multi-view fisheye camera real-time look-around image splicing method and device
CN121391595A
