Embodied intelligent multi-modal perception fusion method and system based on instance-scene collaborative enhancement

Abstract

The application provides a kind of embodiment-scene collaborative enhancement-based embodied intelligent multi-modal perception fusion method and system, the method obtains the heterogeneous sensor information of embodied intelligent agent, including ring vision camera image and laser radar three-dimensional point cloud, and carries out space coordinate registration;Obtain calibrated multi-modal data;Global scene feature map with hierarchical context relationship is generated;Candidate entity target is extracted from global scene feature map using instance-guided fusion mechanism, to obtain refined instance feature including target physical boundary, motion attribute and surface semantics;Get instance perception enhanced fusion feature map;Output the three-dimensional perception state of work target, form the environment understanding model considering global coherence and local fineness;The application can realize effective perception of micro object, can realize accurate positioning target, can effectively improve the spatiotemporal consistency and robustness of embodied intelligent agent in complex work environment.

Description

Technical Field

[0001] This invention relates to an embodied intelligence multimodal perception fusion method and system based on instance-scene collaborative enhancement, belonging to the field of embodied intelligence robot technology. Background Technology

[0002] In the field of embodied intelligence multimodal perception, the accuracy and real-time performance of embodied intelligent agents in three-dimensional cognition of dynamic and complex environments directly determine their operational safety and decision-making reliability. Typical embodied tasks such as power distribution network inspection and urban services have particularly stringent requirements for accurate identification of long-tail targets, spatiotemporal continuity, and multi-scale environmental modeling. The perception system of embodied intelligent agents needs to process heterogeneous data from surround-view cameras and LiDAR in real time, covering object semantic recognition, three-dimensional scale assessment, and omnidirectional space occupancy estimation. Moreover, the operating environment often includes complex conditions such as low light occlusion, motion blur, and cross-view drift. At the same time, the scale differences of the operating targets are huge, making it difficult to balance the accuracy of cross-modal data association with the stability of full-scene understanding.

[0003] Existing perception fusion technologies mostly employ bird's-eye view (BEV) fusion strategies based on a single spatial scale. While these strategies have achieved a preliminary unified representation of heterogeneous data, they often follow linear unidirectional mapping logic during feature extraction, lacking bidirectional feedback interaction between local key entities and global background features. On the one hand, the global gridding process easily leads to the small proportion of subtle target features being submerged by background noise. On the other hand, existing alignment mechanisms often use fixed granularity, making it difficult to simultaneously meet the sub-centimeter resolution requirements of near-field fine-grained operations and the large-scale semantic extraction requirements of far-field path planning. Furthermore, the asynchronous sampling time difference and spatial geometric mapping errors between sensors can generate severe perception holes and wireframe stretching in dynamic environments, causing the generated environmental model to be unable to support the embodied agent in performing high-precision obstacle avoidance and interaction actions. This has become a core technical bottleneck restricting the embodied perception system from "seeing clearly" to "understanding." Summary of the Invention

[0004] The purpose of this invention is to provide an embodied intelligent multimodal perception fusion method based on instance-scene collaborative enhancement to solve the problems existing in the prior art, such as perception distortion of small objects, missed detection due to data sparsity in far-field navigation, and the need to improve the spatiotemporal consistency and robustness of perception in complex operating environments.

[0005] The technical solution of this invention is: An embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement includes the following steps: S1. Obtain heterogeneous sensor information of the embodied intelligent agent, including surround-view camera images and LiDAR 3D point cloud, and perform spatial coordinate registration to obtain registered multimodal data. S2. For the registered multimodal data, perform time-series alignment preprocessing using master time base matching and motion compensation to obtain calibrated multimodal data; S3. For the calibrated multimodal data, a hierarchical scene fusion mechanism is used to construct a three-dimensional bird's-eye view space based on the preset operation detection range and voxel discretization parameters. Through cross-dimensional transformation of points to grids and cross-dimensional transformation of grids to regions, a global scene feature map with hierarchical context relationship is generated. The heterogeneous sensor information is uniformly mapped to the three-dimensional bird's-eye view space, and multi-granular top view features with global topological relationship are extracted. S4. The instance-guided fusion mechanism is used to extract candidate entity targets from the global scene feature map. The local high-precision texture and geometric details of the target are aggregated across modalities through the deformable attention mechanism to obtain refined instance features that include the target's physical boundary, motion attributes and surface semantics. S5. Utilize the reverse feedback path from instance to scene to dynamically feed the refined instance features back to the global feature field, thereby achieving background semantic enhancement and interference filtering, and obtaining the fused feature map after instance perception enhancement. S6. Input the fused feature map after instance perception enhancement into the target decoding network, and output the three-dimensional perception state of the target to form an environment understanding model that takes into account both global coherence and local fineness.

[0006] Further, in step S1, spatial coordinate registration is performed, specifically as follows: S11. Obtain the camera intrinsic parameter matrix, extrinsic parameter matrix and the transformation relationship from the lidar coordinate system to the camera coordinate system pre-calibrated by the embodied intelligent agent. Use perspective projection to transform each discrete geometric point in the 3D point cloud to the pixel coordinate plane corresponding to each camera and establish the initial mapping relationship between geometric points and pixels. S12. For multiple 3D geometric points projected to the same pixel coordinate, a depth buffer sorting mechanism is introduced. Based on the projection depth value of each point, visibility filtering is performed to remove occluded invalid projection points and retain the effective mapping relationship closest to the camera as the effective projection point. S13. For effective projection points, subpixel-level coordinate interpolation is used to calculate their precise subpixel coordinate positions on the corresponding pixel plane. S14. For projection points within the overlapping field of view of multiple cameras, perform dynamic weighted consistency aggregation based on the angle of view to determine the unique pixel coordinates of each 3D geometric point in each camera view, and form a point-to-point association mapping index between 3D geometric points and 2D pixel coordinates.

[0007] Furthermore, in step S2, timing alignment preprocessing is performed using master time base matching and motion compensation, specifically as follows: S21. Using the sampling time of the lidar as the main time base, perform dynamic time window matching on the asynchronous image frames of each surround-view camera, retrieve the image frame pair with the smallest time residual with the main time base, and establish the initial temporal domain association of the multimodal data. S22. For sampling gaps where the time residual exceeds the preset hard synchronization threshold, trigger image interpolation compensation based on flow prediction, and use the motion estimation operator between adjacent image frames to reconstruct the pseudo-synchronous semantic background map under the current main time base to eliminate the perception boundary artifacts caused by sensor shutter deviation. S23. The instantaneous quaternion attitude parameters reported by the airborne inertial measurement unit and wheel speedometer are synchronously fused. The self-evolution equation of posture is used to perform resampling calibration on each discrete sampling point in the lidar scanning cycle. The discrete sampling points collected during the motion are uniformly translated and rotated back to the standardized frame center position of the current scanning cycle to obtain the calibrated multimodal data.

[0008] Furthermore, in step S3, the cross-dimensional transformation of points to the mesh and the cross-dimensional transformation of the mesh to the region generate a global scene feature map with hierarchical contextual relationships. Specifically, S31. Discretize the 3D point cloud of the LiDAR into several voxel cylinders. Each discrete geometric point in the 3D point cloud of the LiDAR contains 3D spatial coordinates and laser reflection intensity information. Based on the point-to-point association mapping index between the 3D geometric points and 2D pixel coordinates in step S14, perform bilinear interpolation at the sub-pixel coordinate positions corresponding to each 3D geometric point. Sample from the high-dimensional feature map extracted by the visual backbone network from the surround-view camera image to extract the high-dimensional image semantic features corresponding to each 3D geometric point, and calculate the coordinate offset of the 3D geometric point relative to the center of its voxel cylinder. Concatenate the high-dimensional image semantic features with the 3D coordinates, laser reflection intensity information and coordinate offset of the 3D geometric point itself to obtain point-level multimodal features. S32. The multi-head self-attention mechanism is used to cross-compare point-level multimodal features within each voxel cylinder to align calibration residual noise across modalities, and pooling is used to aggregate multi-point features within the cylinder into a single grid-level feature vector.

[0009] S33. Divide a single grid-level feature vector into multiple non-overlapping local windows of preset size according to the bird's-eye view space, i.e., the BEV space plane. Construct a local region tensor for the grid feature vector contained in each local window. Calculate the two-dimensional spatial correlation weight between any two grid points in the local window using the spatial self-attention operator within the region. Perform weighted aggregation on the grid attributes based on the two-dimensional spatial correlation weight to generate an initial region semantic block that characterizes the environmental features of each local physical sub-region. S34. The local window division reference is synchronously translated along the horizontal and vertical axes of the bird's-eye view plane, i.e., the BEV plane, by a preset distance, so that the grid points at the original window boundary fall into the internal region of the new window. The translated grid array is divided into secondary regions and spatial self-attention operation is applied again to force the extraction of long-range dependency features between grids across the original window boundary. After reverse translation and reset, it is fused with the initial region semantic block by position-wise tensor. S35. The translation window-self-attention-reset fusion process of step S34 is iteratively executed on BEV feature maps at multiple spatial resolution levels, expanding the effective receptive field of each grid unit layer by layer, and finally generating a global scene feature map with hierarchical contextual relationships.

[0010] Furthermore, in step S4, an instance-guided fusion mechanism is used to extract candidate entity targets from the global scene feature map. A deformable attention mechanism is then used to aggregate the local high-precision texture and geometric details of the targets across modalities, resulting in refined instance features that include the target's physical boundaries, motion attributes, and surface semantics. Specifically, S41. Candidate Instance Extraction: On the global scene feature map generated in step S3, run the center point heatmap detection branch, and map the global scene feature map to the category discrimination space through multiple convolutions to generate a two-dimensional center point probability distribution map with the same scale as the BEV space. The value of each pixel in the two-dimensional center point probability distribution map represents the confidence that there is an interactive target in the corresponding geographical location. S42. Perform category-adaptive local maximum suppression on the two-dimensional center point probability distribution map to select the entity target with the highest confidence as the candidate entity target; S43. Use the high-dimensional feature vector of each candidate entity target at the corresponding position in the global scene feature map as the starting instance query vector; S44. Cross-modal context aggregation: Using the query vector of each initial instance as the query, a deformable attention mechanism is used to perform two-level feature detection in parallel to obtain refined instance features that include the target physical boundary, motion attributes and surface semantics.

[0011] Further, in step S42, category-adaptive local maximum suppression is performed on the two-dimensional center point probability distribution map. Specifically, spatial filtering is performed using a first-size pooling kernel for categories with a scale greater than a preset threshold, and spatial filtering is performed using a second-size pooling kernel for categories with a scale less than the preset threshold, wherein the first size is greater than the second size.

[0012] Further, in step S44, using the query vector of each initial instance as the query, a deformable attention mechanism is used to perform two-level feature detection in parallel to obtain refined instance features, including the target physical boundary, motion attributes, and surface semantics. Specifically, S441. The first level initiates local sampling of the multi-scale image feature map extracted from the surround-view camera image by the visual backbone network to extract high-precision visual texture of the candidate entity neighborhood. S442. The second level initiates local sampling to the spatial neighborhood grid corresponding to the candidate entities in the global scene feature map to extract the depth details of the sparse geometric structure. S443. Introduce a modal importance discrimination mechanism, and calculate the dynamic fusion weighting ratio of the high-precision visual texture and the depth details of the sparse geometric structure of the candidate entity neighborhood in real time based on the perception quality of each local sampling region, so as to eliminate the detection error introduced by the failure of a single modality. S444 Finally, the high-precision visual texture of the candidate entity's neighborhood and the depth details of the sparse geometric structure are summed by attention weighting to construct refined instance features that include the target's physical boundary, motion attributes, and surface semantics.

[0013] Furthermore, in step S5, the refined instance features are dynamically fed back to the global feature field using the instance-to-scene feedback path to achieve background semantic enhancement and interference filtering, resulting in a fused feature map with enhanced instance perception. Specifically, S51. The refined instance features obtained in step S4 are first processed by a multi-head self-attention mechanism to perform feature interaction between instances, eliminating feature redundancy and detection overlap between different candidate entities, and generating deduplicated refined instance features. S52. Using a cross-attention mechanism, the global scene feature map from step S3 is used as the query, and the deduplicated refined instance features are used as the key and value. By calculating the correlation weight between the scene grid and each instance, the target spatial location and semantic category information contained in the deduplicated refined instance features are dynamically injected into the corresponding grid position of the global scene feature map to obtain the injected and enhanced scene features. S53. Perform residual summation on the injected enhanced scene features and the global scene feature map. Under the premise of preserving the original global topology information, perform semantic feature completion and confidence attenuation of background noise on the perception blind zone grid caused by sensor occlusion or data sparsity to obtain the fused feature map after instance perception enhancement.

[0014] Furthermore, in step S6, the fused feature map enhanced with instance perception is input into the target decoding network, which outputs the 3D perception state of the target, forming an environment understanding model that balances global coherence and local refinement. Specifically, S61. Input the fusion feature map after instance perception enhancement into the target decoding network, perform classification confidence regression and 3D attribute calculation on the candidate entity targets corresponding to each spatial location in the fusion feature map after instance perception enhancement, filter low confidence background false detections based on classification scores and preset confidence thresholds, and output the 3D perception state of the target for the retained valid instances, including 3D bounding box, physical pose, motion vector and attribute label. S62. For valid instances whose motion velocity vector is lower than a preset static threshold, a three-dimensional spatial occupancy map based on BEV is obtained based on the fusion feature map after instance perception enhancement. The three-dimensional spatial occupancy map based on BEV is combined with the three-dimensional perception state of the task target to form an environment understanding model that takes into account both global coherence and local refinement.

[0015] A system for implementing any of the above-mentioned embodied intelligence multimodal perception fusion methods based on instance-scene collaborative enhancement includes a heterogeneous sensor spatial coordinate registration module, a heterogeneous sensor spatiotemporal alignment module, a hierarchical feature space construction module, a task-related instance mining module, an instance-scene collaborative feedback enhancement module, and a perception state fusion parsing and output module. Heterogeneous sensor spatial coordinate registration module: acquires heterogeneous sensor information of the embodied intelligent agent, including surround-view camera images and LiDAR 3D point cloud, and performs spatial coordinate registration to obtain registered multimodal data; Heterogeneous sensor spatiotemporal alignment module: After registration, the multimodal data is preprocessed for temporal alignment using master time base matching and motion compensation to obtain calibrated multimodal data; Hierarchical feature space construction module: For calibrated multimodal data, a hierarchical scene fusion mechanism is used to construct a three-dimensional bird's-eye view space based on the preset operation detection range and voxel discretization parameters. Through cross-dimensional transformation of points to grids and cross-dimensional transformation of grids to regions, a global scene feature map with hierarchical context relationship is generated, and heterogeneous sensor information is uniformly mapped to the three-dimensional bird's-eye view space to extract multi-granular top view features with global topological relationship. Task-related instance mining module: It uses an instance-guided fusion mechanism to extract candidate entity targets from the global scene feature map, and aggregates the local high-precision texture and geometric details of the target across modalities through a deformable attention mechanism to obtain refined instance features that include the target's physical boundary, motion attributes and surface semantics; Instance-scene collaborative feedback enhancement module: Utilizing the reverse feedback path from instance to scene, refined instance features are dynamically fed back to the global feature field to achieve background semantic enhancement and interference filtering, resulting in a fused feature map with enhanced instance perception. Perception State Fusion Analysis and Output Module: Input the fused feature map after instance perception enhancement into the target decoding network, and output the three-dimensional perception state of the task target to form an environment understanding model that takes into account both global coherence and local fineness.

[0016] The beneficial effects of this invention are: I. Compared with existing technologies, this embodied intelligence multimodal perception fusion method and system based on instance-scene collaborative enhancement can effectively perceive small objects, accurately locate targets, effectively avoid missed detections caused by data sparsity during far-field navigation, and effectively improve the spatiotemporal consistency and robustness of embodied intelligent agents in complex working environments.

[0017] II. This invention employs a triple calibration mechanism of "time-space-logic" for heterogeneous data to align heterogeneous sensors in time and space. By utilizing master time base matching and motion compensation, it solves the problem of perception offset caused by asynchronous sensor sampling during the movement of embodied robots. Through cross-dimensional transformations of points to grids and grids to regions, it effectively eliminates projection distortion across viewpoint regions, enabling perception and recognition to possess a high degree of physical geometric consistency.

[0018] Third, this embodied intelligent multimodal perception fusion method and system based on instance-scene collaborative enhancement has a unique “instance-scene bidirectional closure” feature completion logic: In response to the problem that small object features are easily masked in conventional fusion algorithms, this invention accurately locates the target through an instance-guided fusion mechanism and constructs a unique reverse feedback path through a reverse feedback path; by using refined instance features to deeply enhance the global scene feature map, it can dynamically repair damaged background semantics and filter out interference noise, significantly improving the continuity and robustness of recognition under occlusion and complex environmental conditions.

[0019] IV. This embodied intelligence multimodal perception fusion method and system based on instance-scene collaborative enhancement features a highly integrated embodied intelligence perception decision chain: through the target decoding network, it can output accurate 3D physical pose, and based on the background features after feedback enhancement, it innovatively generates a 3D spatial occupancy map based on BEV; this technical path directly combines high-level semantic recognition with low-level spatial security modeling, thereby providing a truly reliable and standardized physical perception base for the path planning and precise interactive operation of embodied intelligent agents. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram illustrating the spatial coordinate registration process in the embodiment.

[0022] Figure 3 This is an illustrative diagram illustrating the generation of a global scene feature map with hierarchical contextual relationships in the embodiment.

[0023] Figure 4 This is an illustrative diagram illustrating the extraction of refined instance features in the embodiment.

[0024] Figure 5 This is an illustrative diagram illustrating how, in this embodiment, the fused feature map after instance perception enhancement is input into the target decoding network, and the three-dimensional perception state of the target is output, forming an environment understanding model that balances global coherence and local refinement.

[0025] Figure 6 This is an illustrative diagram illustrating an embodied intelligent multimodal perception fusion system based on instance-scene collaborative enhancement. Detailed Implementation

[0026] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] The embodiment provides an embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement, such as Figure 1 This includes the following steps: S1. Obtain heterogeneous sensor information of the embodied intelligent agent, including surround-view camera images and LiDAR 3D point clouds, and perform spatial coordinate registration to obtain registered multimodal data.

[0028] In step S1, spatial coordinate registration is performed, specifically as follows: Figure 2 : S11. Obtain the camera intrinsic parameter matrix, extrinsic parameter matrix and the transformation relationship from the lidar coordinate system to the camera coordinate system pre-calibrated by the embodied intelligent agent. Use perspective projection to transform each discrete geometric point in the 3D point cloud to the pixel coordinate plane corresponding to each camera and establish the initial mapping relationship between geometric points and pixels. S12. For multiple 3D geometric points projected to the same pixel coordinate, a depth buffer sorting mechanism is introduced. Based on the projection depth value of each point, visibility filtering is performed to remove occluded invalid projection points and retain the effective mapping relationship closest to the camera as the effective projection point. S13. For effective projection points, subpixel-level coordinate interpolation is used to calculate their precise subpixel coordinate positions on the corresponding pixel plane. S14. For projection points within the overlapping field of view of multiple cameras, perform dynamic weighted consistency aggregation based on the angle of view to determine the unique pixel coordinates of each 3D geometric point in each camera view, and form a point-to-point association mapping index between 3D geometric points and 2D pixel coordinates.

[0029] S2. For the registered multimodal data, time alignment preprocessing is performed using master time base matching and motion compensation to obtain calibrated multimodal data.

[0030] In step S2, timing alignment preprocessing is performed using master time base matching and motion compensation, specifically as follows: S21. Using the sampling time of the lidar as the main time base, perform dynamic time window matching on the asynchronous image frames of each surround-view camera, retrieve the image frame pair with the smallest time residual with the main time base, and establish the initial temporal domain association of the multimodal data. S22. For sampling gaps where the time residual exceeds the preset hard synchronization threshold, trigger image interpolation compensation based on flow prediction, and use the motion estimation operator between adjacent image frames to reconstruct the pseudo-synchronous semantic background map under the current main time base to eliminate the perception boundary artifacts caused by sensor shutter deviation. S23. The instantaneous quaternion attitude parameters reported by the airborne inertial measurement unit and wheel speedometer are synchronously fused. The embodied pose self-evolution equation is used to perform resampling calibration on each discrete sampling point within the lidar scanning cycle. This uniformly translates and rotates each discrete sampling point acquired during motion back to the standardized frame center position of the current scanning cycle, obtaining calibrated multimodal data. This eliminates intra-frame drift distortion of the point cloud caused by the rapid movement of the embodied agent.

[0031] In step S2, considering that the timestamps generated by the various sensors of the embodied intelligent agent, including cameras and radars, are naturally asynchronous due to the different clock frequencies of the underlying sampling hardware (e.g., radar frequency is 10Hz, image frequency is 20-30Hz), strict time base alignment is performed. In this embodiment, the sampling cutoff time of the current frame of lidar data is selected as the global master time base t. Master A ±50ms sliding matching time window is opened, and minimum time residual matching is performed within the multi-channel asynchronous image stream buffer to retrieve the image frame TargetImg of the i-th camera that best matches the LiDAR sampling time. i TargetImg i =argmin j |timage i，j -t Master |, where t Master The time base is the time after the full point cloud scan is completed; j is the frame sequence index number in the image buffer to be retrieved; argmin j The optimization operator is used to search and extract the index parameter corresponding to the minimum time offset in the image sequence index j to be retrieved; timage i，jThe original system timestamp recorded by the i-th camera during sampling in frame j is used. The best-matching image frame pair is selected as the semantic background of the current environment. For sampling gaps exceeding the hard synchronization threshold, image interpolation compensation based on flow prediction is triggered. Motion estimation operators generated from images at adjacent times are used to reconstruct the pseudo-synchronous background map under the current time base, achieving high-fidelity "synchronous perception" at the absolute physical time base level and eliminating boundary artifacts caused by shutter deviation in the perceived image. Since the embodied agent generates significant body acceleration and turning movements during operation, to eliminate the point cloud drift distortion (i.e., "jelly effect") caused by the rapid movement of the agent, the instantaneous quaternion attitude parameters reported by the airborne inertial measurement unit (IMU) and wheel speedometer are fused in real time. The point cloud within the sampling period is resampled and calibrated using the embodied pose self-evolution equation to obtain a geometrically rigid 3D point cloud set P after self-motion compensation calibration. corr :P corr =T ego -1 ·T ego ·P raw , where P raw The raw point cloud data collected due to spatial distortion caused by the movement of the embodied intelligent agent; T ego The initial pose transformation matrix of the embodied intelligent agent relative to the global world coordinate system at the instant of laser sampling; T ego -1 This is the inverse matrix of the standardized pose transformation matrix for uniformly aligning the target within the current scanning cycle. Each point sampled during motion is uniformly translated or rotated back to the standardized frame center position of the current cycle. This compensation step ensures that the environment reconstruction under motion still possesses strict geometric rigidity, not only solving the ghosting phenomenon of object contours but also providing a high-accuracy 3D point cloud foundation for the refined instance feature mining (IGF) in the next embodiment. Through the series of calibration procedures in steps S1-S2, calibrated multimodal data with accurate spatiotemporal registration can be obtained.

[0032] S3. For the calibrated multimodal data, a hierarchical scene fusion mechanism is used to construct a three-dimensional bird's-eye view space based on the preset operation detection range and voxel discretization parameters. Through cross-dimensional transformation of points to grids and cross-dimensional transformation of grids to regions, a global scene feature map with hierarchical context relationship is generated. The heterogeneous sensor information is uniformly mapped to the three-dimensional bird's-eye view space, and multi-granular top view features with global topological relationship are extracted.

[0033] In step S3, point-to-mesh cross-dimensional transformation and mesh-to-region cross-dimensional transformation generate a global scene feature map with hierarchical contextual relationships, such as... Figure 3 Specifically: S31. Discretize the 3D point cloud of the LiDAR into several voxel cylinders. Each discrete geometric point in the 3D point cloud of the LiDAR contains 3D spatial coordinates and laser reflection intensity information. Based on the point-to-point association mapping index between the 3D geometric points and 2D pixel coordinates in step S14, perform bilinear interpolation at the sub-pixel coordinate positions corresponding to each 3D geometric point. Sample from the high-dimensional feature map extracted by the visual backbone network from the surround-view camera image to extract the high-dimensional image semantic features corresponding to each 3D geometric point, and calculate the coordinate offset of the 3D geometric point relative to the center of its voxel cylinder. Concatenate the high-dimensional image semantic features with the 3D coordinates, laser reflection intensity information and coordinate offset of the 3D geometric point itself to obtain point-level multimodal features. S32. The multi-head self-attention mechanism is used to cross-compare point-level multimodal features within each voxel cylinder to align calibration residual noise across modalities, and pooling is used to aggregate multi-point features within the cylinder into a single grid-level feature vector.

[0034] S33. Divide a single grid-level feature vector into multiple non-overlapping local windows of preset size according to the bird's-eye view space, i.e., the BEV space plane. Construct a local region tensor for the grid feature vector contained in each local window. Calculate the two-dimensional spatial correlation weight between any two grid points in the local window using the spatial self-attention operator within the region. Perform weighted aggregation on the grid attributes based on the two-dimensional spatial correlation weight to generate an initial region semantic block that characterizes the environmental features of each local physical sub-region. S34. The local window division reference is synchronously translated along the horizontal and vertical axes of the bird's-eye view plane, i.e., the BEV plane, by a preset distance, so that the grid points at the original window boundary fall into the internal region of the new window. The translated grid array is divided into secondary regions and spatial self-attention operation is applied again to force the extraction of long-range dependency features between grids across the original window boundary. After reverse translation and reset, it is fused with the initial region semantic block by position-wise tensor. S35. The translation window-self-attention-reset fusion process of step S34 is iteratively executed on BEV feature maps at multiple spatial resolution levels, expanding the effective receptive field of each grid unit layer by layer, and finally generating a global scene feature map with hierarchical contextual relationships.

[0035] In step S3, the calibrated multimodal feature stream is subjected to hierarchical scene fusion processing at three levels: point, grid, and region. By performing multi-scale cross-dimensional feature mapping and translation window attention calculation, the discrete point cloud geometric features and image texture semantics are uniformly aggregated into a bird's-eye view (BEV) scene feature field with long-range environmental dependencies.

[0036] S4. An instance-guided fusion mechanism is used to extract candidate entity targets from the global scene feature map. A deformable attention mechanism is then used to aggregate the local high-precision texture and geometric details of the targets across modalities, resulting in refined instance features that include the target's physical boundaries, motion attributes, and surface semantics. For example... Figure 4 : S41. On the global scene feature map generated in step S3, run the center point heat map detection branch, map the global scene feature map to the category discrimination space through multi-layer convolution, and generate a two-dimensional center point probability distribution map consistent with the BEV space scale. The value of each pixel in the two-dimensional center point probability distribution map represents the confidence that there is an interactive target (such as a power distribution cabinet, work support or live wire) in the corresponding geographical location. S42. Perform category-adaptive local maximum suppression on the two-dimensional center point probability distribution map to select the entity target with the highest confidence as the candidate entity target; In step S42, category-adaptive local maximum suppression is performed on the two-dimensional center point probability distribution map. Specifically, a first-size pooling kernel is used for spatial filtering of categories with a scale greater than a preset threshold, and a second-size pooling kernel is used for spatial filtering of categories with a scale less than the preset threshold, wherein the first size is greater than the second size.

[0037] S43. Use the high-dimensional feature vector of each candidate entity target at the corresponding position in the global scene feature map as the starting instance query vector; In step S43, the BEV space coordinates of the candidate entity targets selected in step S42 are first transformed into the instance feature space through a convolutional mapping layer. Then, the transformed scene feature map is flattened into a one-dimensional feature sequence along the BEV plane. Based on the BEV coordinate index of each candidate entity, an index-based aggregation operation is performed in the one-dimensional feature sequence to extract the high-dimensional feature vectors corresponding to each candidate position and encapsulate them into the starting instance query vector.

[0038] S44. Cross-modal context aggregation: Using the query vector of each initial instance as the query, a deformable attention mechanism is used to perform two-level feature detection in parallel to obtain refined instance features that include the target physical boundary, motion attributes and surface semantics.

[0039] In step S44, using the query vector of each initial instance as the query, a two-level feature detection is performed in parallel using a deformable attention mechanism to obtain refined instance features that include the target's physical boundary, motion attributes, and surface semantics. Specifically, S441. The first level initiates local sampling of the multi-scale image feature map extracted from the surround-view camera image by the visual backbone network to extract high-precision visual texture of the candidate entity neighborhood. S442. The second level initiates local sampling to the spatial neighborhood grid corresponding to the candidate entities in the global scene feature map to extract the depth details of the sparse geometric structure. S443. Introduce a modal importance discrimination mechanism, and calculate the dynamic fusion weighting ratio of the high-precision visual texture and the depth details of the sparse geometric structure of the candidate entity neighborhood in real time based on the perception quality of each local sampling region, so as to eliminate the detection error introduced by the failure of a single modality. S444 Finally, the high-precision visual texture of the candidate entity's neighborhood and the depth details of the sparse geometric structure are summed by attention weighting to construct refined instance features that include the target's physical boundary, motion attributes, and surface semantics.

[0040] S5. Utilize the reverse feedback path from instance to scene to dynamically feed refined instance features back to the global feature field, thereby achieving background semantic enhancement and interference filtering, and obtaining a fused feature map with enhanced instance perception.

[0041] S51. The refined instance features obtained in step S4 are first processed by a multi-head self-attention mechanism to perform feature interaction between instances, eliminating feature redundancy and detection overlap between different candidate entities, and generating deduplicated refined instance features. In step S51, feature interaction is performed between instances through a multi-head self-attention mechanism to eliminate feature redundancy and detection overlap between different candidate entities, generating deduplicated refined instance features. Specifically, the instance feature vectors in the refined instance features generated in step S4 are transformed in dimension by a multilayer perceptron (MLP) to form an instance feature matrix. This instance feature matrix is ​​used as the query, key, and value of the self-attention mechanism. After linear projection, the relevance weight between any two instance features is calculated. When two candidate entities point to the same task target, their feature vectors are highly similar. After self-attention interaction, the feature differences between the two are further compressed and converged. After Dropout regularization and residual connection are superimposed on the original instance features, the LayerNorm layer normalizes the data and outputs the deduplicated refined instance features.

[0042] S52. Using a cross-attention mechanism, the global scene feature map from step S3 is used as the query, and the deduplicated refined instance features are used as the key and value. By calculating the correlation weight between the scene grid and each instance, the deduplicated refined instance features are weighted and summed. The target spatial location and semantic category information contained in the deduplicated refined instance features are dynamically injected into the corresponding grid position of the global scene feature map to obtain the injected and enhanced scene features. In step S52, the global scene feature map from step S3 is flattened into a scene feature matrix as a query, and the deduplicated refined instance features are used as keys and values. The correlation score matrix between each grid position in the scene and each instance is calculated by matrix multiplication. After Softmax normalization, a correlation weight distribution is generated. The grid position with a higher weight value corresponds to the physical space that is strongly associated with the current instance in terms of geometric location or semantic attributes. The refined instance features are weighted and summed using this attention weight. The target space location and semantic category information contained in the deduplicated refined instance features are dynamically injected into the corresponding grid positions of the scene features to obtain the injected and enhanced scene features.

[0043] S53. Perform residual summation on the injected enhanced scene features and the global scene feature map. Under the premise of preserving the original global topology information, perform semantic feature completion and confidence attenuation of background noise on the perception blind zone grid caused by sensor occlusion or data sparsity to obtain the fused feature map after instance perception enhancement.

[0044] In step S53, the injected enhanced scene features provide semantic completion information from the instance level. The original scene features retain the untransformed global topology, and residual connections ensure that the completion operation does not cover or distort grid positions with reliable information in the original scene. Semantic feature completion is only performed on grids in perceptual blind spots caused by occlusion or data sparsity. At the same time, residual connections naturally attenuate the confidence level for potential noise introduced by low-confidence instances during the injection process. When the injected information at a certain grid position is inconsistent with the original information, the residual structure allows the original features to dominate the output at that position, thereby filtering out unreliable injected components and outputting a fused feature map after instance perceptual enhancement.

[0045] In step S5, global scene association retrieval and information anchoring are performed to establish an accurate spatial correspondence index for subsequent attribute injection. Then, dynamic semantic completion and background depth feedback operations are started, along with active filtering of perceptual noise and consistency self-checking, to output a fusion feature map with highly robust instance perception enhancement.

[0046] S6. Input the fused feature map enhanced with instance perception into the target decoding network, and output the 3D perception state of the target, forming an environment understanding model that balances global coherence and local refinement. For example... Figure 5 : S61. Input the fusion feature map after instance perception enhancement into the target decoding network. Perform classification confidence regression and 3D attribute calculation on the candidate entity targets corresponding to each spatial location in the fusion feature map after instance perception enhancement. Filter low-confidence background false detections based on classification scores and preset confidence thresholds. Output the 3D perception state of the target, including 3D bounding box, physical pose, motion vector and attribute label, for the retained valid instances.

[0047] In step S61, the instance-aware enhanced fusion feature map output from step S5 is input into the target decoding network. The target decoding network uses a 3D target detection head based on a Transformer decoder. The target decoding network performs mapping operations based on center point regression and multi-dimensional attribute calculation; it calculates the three-dimensional physical pose vector of each salient entity through decoding logic, including the target's spatial rectangular coordinates, three-dimensional geometric dimensions, yaw angle orientation, and real-time motion velocity components; simultaneously, it uses the trained classification weight matrix to perform semantic label mapping on the target, extracting effective instances with high-dimensional attributes.

[0048] S62. For valid instances whose motion velocity vector is lower than a preset static threshold, a three-dimensional spatial occupancy map based on BEV is obtained based on the fusion feature map after instance perception enhancement. The three-dimensional spatial occupancy map based on BEV is combined with the three-dimensional perception state of the task target to form an environment understanding model that takes into account both global coherence and local refinement.

[0049] In step S62, the fusion feature map enhanced by instance perception is used to generate a three-dimensional spatial occupancy map based on BEV around the agent in real time. The spatial distribution of the fusion feature map enhanced by instance perception is mapped to each grid of the BEV space through a thresholding decision function, and the existence form of valid instances is marked simultaneously. Semantic information (such as ground feasible domain and obstacle edge protection zone) obtained by reverse enhancement of instance-level features is integrated. This process converts the feature map into semantic map data containing obstacle geometry and operational safety constraints, providing a physical risk benchmark for obstacle avoidance planning.

[0050] In step S6, deep logic decoding is performed on the fused feature map enhanced by instance perception. The 3D bounding box, physical pose, motion vector, and attribute labels of the target are calculated in real time. Based on the environmental background features fed back from the instance, a 3D spatial occupancy map is simultaneously generated for the embodied agent to perform obstacle avoidance path planning and precise operation control, achieving a standardized mapping from perception data to embodied decision-making tasks. When the target decoding network decodes the fused feature map enhanced by instance perception, the sparse locations in the far field already carry the refined instance features extracted in step S4 and fed back in step S5. These refined instance features include the target's physical boundary, motion attributes, and surface semantics, thus enabling the effective detection of far-field targets that were previously missed due to point cloud sparsity. This effectively avoids missed detections caused by data sparsity during far-field navigation.

[0051] Compared with existing technologies, this embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement can effectively perceive small objects, accurately locate targets, effectively avoid missed detections caused by data sparsity during far-field navigation, and effectively improve the spatiotemporal consistency and robustness of embodied intelligent agents in complex working environments.

[0052] This invention employs a triple calibration mechanism of "time-space-logic" for heterogeneous data to align heterogeneous sensors in time and space. By utilizing master time base matching and motion compensation, it solves the problem of perception offset caused by asynchronous sensor sampling during the movement of embodied robots. Through cross-dimensional transformations of points to grids and grids to regions, it effectively eliminates projection distortion across viewpoints, enabling perception and recognition to possess a high degree of physical geometric consistency.

[0053] like Figure 6 The embodiments also provide a system for implementing the embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described above, including a heterogeneous sensor spatial coordinate registration module, a heterogeneous sensor spatiotemporal alignment module, a hierarchical feature space construction module, a task-related instance mining module, an instance-scene collaborative feedback enhancement module, and a perception state fusion parsing and output module. Heterogeneous sensor spatial coordinate registration module: acquires heterogeneous sensor information of the embodied intelligent agent, including surround-view camera images and LiDAR 3D point cloud, and performs spatial coordinate registration to obtain registered multimodal data; Heterogeneous sensor spatiotemporal alignment module: After registration, the multimodal data is preprocessed for temporal alignment using master time base matching and motion compensation to obtain calibrated multimodal data; Hierarchical feature space construction module: For calibrated multimodal data, a hierarchical scene fusion mechanism is used to construct a three-dimensional bird's-eye view space based on the preset operation detection range and voxel discretization parameters. Through cross-dimensional transformation of points to grids and cross-dimensional transformation of grids to regions, a global scene feature map with hierarchical context relationship is generated, and heterogeneous sensor information is uniformly mapped to the three-dimensional bird's-eye view space to extract multi-granular top view features with global topological relationship. Task-related instance mining module: It uses an instance-guided fusion mechanism to extract candidate entity targets from the global scene feature map, and aggregates the local high-precision texture and geometric details of the target across modalities through a deformable attention mechanism to obtain refined instance features that include the target's physical boundary, motion attributes and surface semantics; Instance-scene collaborative feedback enhancement module: Utilizing the reverse feedback path from instance to scene, refined instance features are dynamically fed back to the global feature field to achieve background semantic enhancement and interference filtering, resulting in a fused feature map with enhanced instance perception. Perception State Fusion Analysis and Output Module: Input the fused feature map after instance perception enhancement into the target decoding network, and output the three-dimensional perception state of the task target to form an environment understanding model that takes into account both global coherence and local fineness.

[0054] This embodied intelligent multimodal perception fusion method and system based on instance-scene collaborative enhancement features a unique “instance-scene bidirectional closure” feature completion logic: addressing the problem that small object features are easily masked in conventional fusion algorithms, this invention accurately locates the target through an instance-guided fusion mechanism and constructs a unique reverse feedback path through a reverse feedback path; by using refined instance features to deeply enhance the global scene feature map, it can dynamically repair damaged background semantics and filter out interference noise, significantly improving the continuity and robustness of recognition under occlusion and complex environmental conditions.

[0055] This embodied intelligence multimodal perception fusion method and system based on instance-scene collaborative enhancement features a highly integrated embodied intelligence perception decision chain: through the target decoding network, it can output accurate 3D physical pose, and based on the background features after feedback enhancement, it innovatively generates a 3D spatial occupancy map based on BEV; this technical approach directly combines high-level semantic recognition with low-level spatial security modeling, thereby providing a truly reliable and standardized physical perception foundation for the path planning and precise interaction of embodied intelligent agents.

[0056] This embodied intelligence multimodal perception fusion method and system based on instance-scene collaborative enhancement performs hierarchical scene fusion on information acquired by cameras and radar to characterize instance features with extremely high accuracy, and uses instances to guide the characterization of scene features to achieve instance-scene collaborative enhancement. This provides technical support for embodied intelligent agents to "understand" the environment when performing tasks and thus accurately make corresponding actions.

[0057] This embodied intelligent multimodal perception fusion method and system, based on instance-scene collaborative enhancement, acquires surround-view images and LiDAR point cloud data in the embodied intelligent agent's working environment. It performs spatial coordinate registration and absolute temporal alignment on the point cloud data and image data to generate synchronized heterogeneous multimodal feature streams. Utilizing a hierarchical scene fusion mechanism, it uniformly maps the heterogeneous multimodal feature streams to a 3D bird's-eye view space, capturing the long-range topological relationships of the working environment through multi-granularity spatial transformations, generating multi-scale hierarchical scene feature maps. Employing an instance-guided fusion mechanism, it extracts candidate entities of the task target from the hierarchical scene feature maps and aggregates the local high-definition textures and geometric physical features of the candidate entities across modalities to generate refined instance features. It constructs a reverse collaborative enhancement path from instance to scene, dynamically feeding the refined instance features back into the global scene features, performing background semantic feature completion and perceptual noise filtering to generate a unified feature representation for instance perception. The unified feature representation is input into the target parsing model, outputting an environmental model that balances global coherence and local refinement, as well as 3D perceptual state information of the task target. This invention solves the problems of distorted perception of small objects by embodied intelligent agents in the near field of operation and missed detection caused by data sparsity during far-field navigation, effectively improving the spatiotemporal consistency and robustness of embodied intelligent agents in complex working environments.

[0058] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for embodied intelligence multimodal perception fusion based on instance-scene collaborative enhancement, characterized in that: Includes the following steps, S1. Obtain heterogeneous sensor information of the embodied intelligent agent, including surround-view camera images and LiDAR 3D point cloud, and perform spatial coordinate registration to obtain registered multimodal data. S2. For the registered multimodal data, perform time-series alignment preprocessing using master time base matching and motion compensation to obtain calibrated multimodal data; S3. For the calibrated multimodal data, a hierarchical scene fusion mechanism is used to construct a three-dimensional bird's-eye view space based on the preset operation detection range and voxel discretization parameters. Through cross-dimensional transformation of points to grids and cross-dimensional transformation of grids to regions, a global scene feature map with hierarchical context relationship is generated. The heterogeneous sensor information is uniformly mapped to the three-dimensional bird's-eye view space, and multi-granular top view features with global topological relationship are extracted. S4. The instance-guided fusion mechanism is used to extract candidate entity targets from the global scene feature map. The local high-precision texture and geometric details of the target are aggregated across modalities through the deformable attention mechanism to obtain refined instance features that include the target's physical boundary, motion attributes and surface semantics. S5. Utilize the reverse feedback path from instance to scene to dynamically feed the refined instance features back to the global feature field, thereby achieving background semantic enhancement and interference filtering, and obtaining the fused feature map after instance perception enhancement. S6. Input the fused feature map after instance perception enhancement into the target decoding network, and output the three-dimensional perception state of the target to form an environment understanding model that takes into account both global coherence and local fineness.

2. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in claim 1, characterized in that: In step S1, spatial coordinate registration is performed, specifically as follows: S11. Obtain the camera intrinsic parameter matrix, extrinsic parameter matrix and the transformation relationship from the lidar coordinate system to the camera coordinate system pre-calibrated by the embodied intelligent agent. Use perspective projection to transform each discrete geometric point in the 3D point cloud to the pixel coordinate plane corresponding to each camera and establish the initial mapping relationship between geometric points and pixels. S12. For multiple 3D geometric points projected to the same pixel coordinate, a depth buffer sorting mechanism is introduced. Based on the projection depth value of each point, visibility filtering is performed to remove occluded invalid projection points and retain the effective mapping relationship closest to the camera as the effective projection point. S13. For effective projection points, subpixel-level coordinate interpolation is used to calculate their precise subpixel coordinate positions on the corresponding pixel plane. S14. For projection points within the overlapping field of view of multiple cameras, perform dynamic weighted consistency aggregation based on the angle of view to determine the unique pixel coordinates of each 3D geometric point in each camera view, and form a point-to-point association mapping index between 3D geometric points and 2D pixel coordinates.

3. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in claim 1, characterized in that: In step S2, timing alignment preprocessing is performed using master time base matching and motion compensation, specifically as follows: S21. Using the sampling time of the lidar as the main time base, perform dynamic time window matching on the asynchronous image frames of each surround-view camera, retrieve the image frame pair with the smallest time residual with the main time base, and establish the initial temporal domain association of the multimodal data. S22. For sampling gaps where the time residual exceeds the preset hard synchronization threshold, trigger image interpolation compensation based on flow prediction, and use the motion estimation operator between adjacent image frames to reconstruct the pseudo-synchronous semantic background map under the current main time base to eliminate the perception boundary artifacts caused by sensor shutter deviation. S23. The instantaneous quaternion attitude parameters reported by the airborne inertial measurement unit and wheel speedometer are synchronously fused. The self-evolution equation of posture is used to perform resampling calibration on each discrete sampling point in the lidar scanning cycle. The discrete sampling points collected during the motion are uniformly translated and rotated back to the standardized frame center position of the current scanning cycle to obtain the calibrated multimodal data.

4. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in claim 2, characterized in that: In step S3, point-to-mesh cross-dimensional transformation and mesh-to-region cross-dimensional transformation generate a global scene feature map with hierarchical contextual relationships. Specifically, S31. Discretize the 3D point cloud of the LiDAR into several voxel cylinders. Each discrete geometric point in the 3D point cloud of the LiDAR contains 3D spatial coordinates and laser reflection intensity information. Based on the point-to-point association mapping index between the 3D geometric points and 2D pixel coordinates in step S14, perform bilinear interpolation at the sub-pixel coordinate positions corresponding to each 3D geometric point. Sample from the high-dimensional feature map extracted by the visual backbone network from the surround-view camera image to extract the high-dimensional image semantic features corresponding to each 3D geometric point, and calculate the coordinate offset of the 3D geometric point relative to the center of its voxel cylinder. Concatenate the high-dimensional image semantic features with the 3D coordinates, laser reflection intensity information and coordinate offset of the 3D geometric point itself to obtain point-level multimodal features. S32. The multi-head self-attention mechanism is used to cross-compare point-level multimodal features within each voxel cylinder to align the calibration residual noise across modalities, and the pooling operation is used to aggregate the multi-point features within the cylinder into a single grid-level feature vector. S33. Divide a single grid-level feature vector into multiple non-overlapping local windows of preset size according to the bird's-eye view space, i.e., the BEV space plane. Construct a local region tensor for the grid feature vector contained in each local window. Calculate the two-dimensional spatial correlation weight between any two grid points in the local window using the spatial self-attention operator within the region. Perform weighted aggregation on the grid attributes based on the two-dimensional spatial correlation weight to generate an initial region semantic block that characterizes the environmental features of each local physical sub-region. S34. The local window division reference is synchronously translated along the horizontal and vertical axes of the bird's-eye view plane, i.e., the BEV plane, by a preset distance, so that the grid points at the original window boundary fall into the internal region of the new window. The translated grid array is divided into secondary regions and spatial self-attention operation is applied again to force the extraction of long-range dependency features between grids across the original window boundary. After reverse translation and reset, it is fused with the initial region semantic block by position-wise tensor. S35. The translation window-self-attention-reset fusion process of step S34 is iteratively executed on BEV feature maps at multiple spatial resolution levels, expanding the effective receptive field of each grid unit layer by layer, and finally generating a global scene feature map with hierarchical contextual relationships.

5. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in any one of claims 1-4, characterized in that: In step S4, an instance-guided fusion mechanism is used to extract candidate entity targets from the global scene feature map. A deformable attention mechanism is then used to aggregate the local high-precision texture and geometric details of the targets across modalities, resulting in refined instance features that include the target's physical boundaries, motion attributes, and surface semantics. Specifically, S41. Candidate Instance Extraction: On the global scene feature map generated in step S3, run the center point heatmap detection branch, and map the global scene feature map to the category discrimination space through multiple convolutions to generate a two-dimensional center point probability distribution map with the same scale as the BEV space. The value of each pixel in the two-dimensional center point probability distribution map represents the confidence that there is an interactive target in the corresponding geographical location. S42. Perform category-adaptive local maximum suppression on the two-dimensional center point probability distribution map to select the entity target with the highest confidence as the candidate entity target; S43. Use the high-dimensional feature vector of each candidate entity target at the corresponding position in the global scene feature map as the starting instance query vector; S44. Cross-modal context aggregation: Using the query vector of each initial instance as the query, a deformable attention mechanism is used to perform two-level feature detection in parallel to obtain refined instance features that include the target physical boundary, motion attributes and surface semantics.

6. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in claim 5, characterized in that: In step S42, category-adaptive local maximum suppression is performed on the two-dimensional center point probability distribution map. Specifically, spatial filtering is performed using a first-size pooling kernel for categories with a scale greater than a preset threshold, and spatial filtering is performed using a second-size pooling kernel for categories with a scale less than the preset threshold, wherein the first size is greater than the second size.

7. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in claim 6, characterized in that: In step S44, using the query vector of each initial instance as the query, a deformable attention mechanism is used to perform two-level feature detection in parallel to obtain refined instance features, which include the target physical boundary, motion attributes, and surface semantics. Specifically, S441. The first level initiates local sampling of the multi-scale image feature map extracted from the surround-view camera image by the visual backbone network to extract high-precision visual texture of the candidate entity neighborhood. S442. The second level initiates local sampling to the spatial neighborhood grid corresponding to the candidate entities in the global scene feature map to extract the depth details of the sparse geometric structure. S443. Introduce a modal importance discrimination mechanism, and calculate the dynamic fusion weighting ratio of the high-precision visual texture and the depth details of the sparse geometric structure of the candidate entity neighborhood in real time based on the perception quality of each local sampling region, so as to eliminate the detection error introduced by the failure of a single modality. S444 Finally, the high-precision visual texture of the candidate entity's neighborhood and the depth details of the sparse geometric structure are summed by attention weighting to construct refined instance features that include the target's physical boundary, motion attributes, and surface semantics.

8. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in any one of claims 1-4, characterized in that: In step S5, the refined instance features are dynamically fed back to the global feature field using the instance-to-scene feedback path, achieving background semantic enhancement and interference filtering, resulting in a fused feature map with enhanced instance perception. Specifically, S51. The refined instance features obtained in step S4 are first processed by a multi-head self-attention mechanism to perform feature interaction between instances, eliminating feature redundancy and detection overlap between different candidate entities, and generating deduplicated refined instance features. S52. Using a cross-attention mechanism, the global scene feature map from step S3 is used as the query, and the deduplicated refined instance features are used as the key and value. By calculating the correlation weight between the scene grid and each instance, the target spatial location and semantic category information contained in the deduplicated refined instance features are dynamically injected into the corresponding grid position of the global scene feature map to obtain the injected and enhanced scene features. S53. Perform residual summation on the injected enhanced scene features and the global scene feature map. Under the premise of preserving the original global topology information, perform semantic feature completion and confidence attenuation of background noise on the perception blind zone grid caused by sensor occlusion or data sparsity to obtain the fused feature map after instance perception enhancement.

9. The embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement as described in any one of claims 1-4, characterized in that: In step S6, the fused feature map enhanced with instance perception is input into the target decoding network, which outputs the 3D perception state of the target, forming an environment understanding model that balances global coherence and local refinement. Specifically, S61. Input the fusion feature map after instance perception enhancement into the target decoding network, perform classification confidence regression and 3D attribute calculation on the candidate entity targets corresponding to each spatial location in the fusion feature map after instance perception enhancement, filter low confidence background false detections based on classification scores and preset confidence thresholds, and output the 3D perception state of the target for the retained valid instances, including 3D bounding box, physical pose, motion vector and attribute label. S62. For valid instances whose motion velocity vector is lower than a preset static threshold, a three-dimensional space occupancy map based on BEV is obtained based on the fusion feature map after instance perception enhancement. The three-dimensional space occupancy map based on BEV is combined with the three-dimensional perception state of the task target to form an environment understanding model that takes into account both global coherence and local refinement.

10. A system for implementing the embodied intelligence multimodal perception fusion method based on instance-scene collaborative enhancement according to any one of claims 1-9, characterized in that: It includes a heterogeneous sensor spatial coordinate registration module, a heterogeneous sensor spatiotemporal alignment module, a hierarchical feature space construction module, a task-related instance mining module, an instance-scene collaborative feedback enhancement module, and a perception state fusion parsing and output module. Heterogeneous sensor spatial coordinate registration module: acquires heterogeneous sensor information of the embodied intelligent agent, including surround-view camera images and LiDAR 3D point cloud, and performs spatial coordinate registration to obtain registered multimodal data; Heterogeneous sensor spatiotemporal alignment module: After registration, the multimodal data is preprocessed for temporal alignment using master time base matching and motion compensation to obtain calibrated multimodal data; Hierarchical feature space construction module: For calibrated multimodal data, a hierarchical scene fusion mechanism is used to construct a three-dimensional bird's-eye view space based on the preset operation detection range and voxel discretization parameters. Through cross-dimensional transformation of points to grids and cross-dimensional transformation of grids to regions, a global scene feature map with hierarchical context relationship is generated, and heterogeneous sensor information is uniformly mapped to the three-dimensional bird's-eye view space to extract multi-granular top view features with global topological relationship. Task-related instance mining module: It uses an instance-guided fusion mechanism to extract candidate entity targets from the global scene feature map, and aggregates the local high-precision texture and geometric details of the target across modalities through a deformable attention mechanism to obtain refined instance features that include the target's physical boundary, motion attributes and surface semantics; Instance-scene collaborative feedback enhancement module: Utilizing the reverse feedback path from instance to scene, refined instance features are dynamically fed back to the global feature field to achieve background semantic enhancement and interference filtering, resulting in a fused feature map with enhanced instance perception. Perception State Fusion Analysis and Output Module: Input the fused feature map after instance perception enhancement into the target decoding network, and output the three-dimensional perception state of the task target to form an environment understanding model that takes into account both global coherence and local fineness.

Images