Environment perception method, system, computer device, and storage medium

Abstract

The embodiment of the application belongs to the field of unmanned aerial vehicles and robots, and relates to an environment perception method and system, a computer device and a storage medium. Multi-source sensor data is acquired, preprocessed, and preprocessed data is obtained. The preprocessed data is time-synchronized and spatial coordinate-unified, and the preprocessed data after time-space alignment is obtained. The preprocessed data after time-space alignment is projected to a preset four-dimensional feature space, feature extraction is performed, and a four-dimensional feature carrier is constructed. A fusion network is constructed, feature fusion of the four-dimensional feature carrier is performed through the fusion network, and a fusion feature body is obtained. Feature decoding is performed on the fusion feature body, and an environment perception result is output.

Description

Technical Field

[0001] This application relates to the field of drones and robots, and more particularly to an environmental perception method, system, computer device, and storage medium. Background Technology

[0002] In the fields of drones and robotics, the accuracy and reliability of environmental perception directly determine the device's autonomous decision-making capabilities. Currently, multi-sensor fusion is a core method for improving perception performance; however, existing multi-sensor systems exhibit significant perception shifts in high-speed motion scenarios, making it difficult to fully represent the environment and leading to missed or misjudged obstacles—in short, insufficient perception accuracy.

[0003] In view of the above, this application is hereby submitted. Summary of the Invention

[0004] The purpose of this application is to provide an environmental perception method, system, computer device, and storage medium, which has the advantages of improving the accuracy of multi-sensor data fusion, enhancing the adaptability to dynamic scenes, and optimizing computational efficiency.

[0005] This application provides an environmental perception method, the technical solution of which is as follows: An environmental perception method includes: acquiring multi-source sensor data; preprocessing the multi-source sensor data to obtain preprocessed data; performing time synchronization and spatial coordinate unification on the preprocessed data to obtain spatiotemporally aligned preprocessed data; projecting the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space, performing feature extraction, and constructing a four-dimensional feature carrier; constructing a fusion network, performing feature fusion on the four-dimensional feature carrier through the fusion network to obtain a fused feature body; performing feature decoding on the fused feature body, and outputting the environmental perception result.

[0006] Furthermore, this application proposes to synchronize the time and unify the spatial coordinates of the preprocessed data, including: using the high-frequency timestamp of the inertial measurement unit in the preprocessed data as a reference, predicting the time offset of other sensor data in the preprocessed data; interpolating and adjusting the preprocessed data according to the time offset to obtain the time-synchronized preprocessed data; and converting the time-synchronized preprocessed data to a unified coordinate system to unify the spatial coordinates and obtain the spatiotemporally aligned preprocessed data.

[0007] Furthermore, this application also proposes to project the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space for feature extraction and to construct a four-dimensional feature carrier, including: projecting the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space to obtain projected data; extracting semantic features, geometric features, and motion features from the projected data; and integrating the semantic features, geometric features, and motion features to obtain a four-dimensional feature carrier.

[0008] Furthermore, this application proposes to construct a fusion network to perform feature fusion on a four-dimensional feature carrier to obtain a fused feature body, including: constructing a fusion network that includes temporal feature association and cross-modal feature complementarity; determining the feature dependency relationship of the same sensor based on temporal feature association; determining the feature complementarity information of different sensors based on cross-modal feature complementarity; and dynamically weighting and fusing the multimodal features in the four-dimensional feature carrier based on feature dependency relationship and feature complementarity information to generate a fused feature body.

[0009] Furthermore, this application also proposes that, after obtaining the fused feature body, the following steps are taken: calculating the confidence level of the sensor based on the preprocessed data; and adjusting the weight of the sensor when the confidence level is lower than a preset threshold.

[0010] Furthermore, this application also proposes to perform feature decoding on the fused feature body and output environmental perception results, including: parsing the fused feature body to generate obstacle information; calculating pose parameters based on motion constraint information in the fused feature body; and integrating obstacle information and pose parameters to output environmental perception results.

[0011] Furthermore, this application also proposes that after performing feature decoding on the fused feature volume and outputting the environmental perception result, it further includes: evaluating the accuracy of the environmental perception result and calculating the perception error; and adjusting the parameters of the fusion network based on the perception error.

[0012] Furthermore, this application also proposes an environmental perception system, comprising: an acquisition module for acquiring multi-source sensor data and preprocessing the multi-source sensor data to obtain preprocessed data; a calibration module for synchronizing the preprocessed data in time and unifying its spatial coordinates; a construction module for projecting the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space, extracting features, and constructing a four-dimensional feature carrier; a fusion module for constructing a data fusion network and fusing features of the four-dimensional feature carrier through the data fusion network to obtain a fused feature body; and an output module for decoding the fused feature body and outputting the environmental perception result.

[0013] Furthermore, this application also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the above-described environmental perception method.

[0014] Furthermore, this application also proposes a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the above-described environment-awareness method.

[0015] As can be seen from the above, the environmental perception method, system, computer device and storage medium provided in this application effectively eliminate the temporal and spatial deviation of multi-source sensor data through spatiotemporal alignment, four-dimensional feature space projection and dynamic weighted fusion technology, realize the complementary optimization of cross-modal features, significantly improve the perception accuracy and real-time performance in complex dynamic scenes, and have the advantages of improving the accuracy of multi-sensor data fusion, enhancing the adaptability to dynamic scenes and optimizing computational efficiency. Attached Figure Description

[0016] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of the environmental perception method according to this application; Figure 3 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0021] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0022] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0023] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, e-book readers, MP3 players, MP4 players, laptops, and desktop computers.

[0024] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0025] It should be noted that the environmental perception method provided in this application embodiment is generally executed by a terminal device, and correspondingly, the environmental perception system is generally set in the terminal device.

[0026] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0027] Continue to refer to Figure 2 The diagram shows a flowchart of an embodiment of the environmental perception method according to this application.

[0028] In existing technologies, environmental perception technologies for drones and robots rely on multi-sensor data fusion, but traditional methods have significant drawbacks. Differences in sampling frequencies between different sensors lead to temporal misalignment; static calibration parameters cannot compensate for coordinate shifts during dynamic motion; and multimodal data fusion ignores the temporal dimension, making it difficult to handle fast-moving objects and occluded scenes. When a single sensor fails, the fixed weight allocation mechanism causes a sharp decline in system performance, and the computational resource limitations of embedded platforms further restrict the real-time operation of high-precision models.

[0029] To address the aforementioned issues, the inventors discovered that existing fusion frameworks lack dynamic spatiotemporal modeling capabilities and cannot effectively handle asynchronous sensor readings and scene changes. By analyzing the spatiotemporal correlation of multimodal data, they proposed introducing the time dimension into the feature space to construct a four-dimensional spatiotemporal coordinate system. To address sensor failure, a dynamic weight adjustment mechanism was designed, combining the cross-modal complementary characteristics of the feature space to achieve robust perception. Simultaneously, a lightweight network structure was employed to balance computational efficiency and ensure the feasibility of embedded deployment.

[0030] Therefore, this application proposes a technical solution to acquire multi-source sensor data and preprocess it, synchronize the preprocessed data in time and unify its spatial coordinates, project the spatiotemporally aligned data onto a four-dimensional feature space for feature extraction, construct a fusion network to achieve feature fusion, and finally decode and generate environmental perception results.

[0031] Multi-source sensor data refers to heterogeneous data sets from cameras, LiDAR, IMU, and millimeter-wave radar. This can be achieved through parallel acquisition using multi-threaded acquisition modules, addressing the issue of limited information dimensionality in traditional single-sensor data. Preprocessing includes data cleaning and format standardization, implemented using filtering algorithms and coordinate transformation toolchains to eliminate noise interference and provide a unified input for subsequent processing. Time synchronization aligns the sampling time axes of different sensors, achieved using a dynamic prediction model based on an LSTM network to address the cumulative errors of fixed interpolation methods in moving scenes. Spatial coordinate unification maps multi-sensor data to the same reference frame, using learnable transformation matrices to compensate for installation errors and motion deformation. A four-dimensional feature space adds a time dimension to a three-dimensional space, achieved using voxel projection and a temporal encoder to capture the spatiotemporal evolution of dynamic scenes. The fusion network is a neural network architecture for multimodal feature interaction, implemented using cross-modal attention mechanisms to enhance the complementarity of data from different sensors. Feature decoding refers to mapping high-dimensional features into executable perceptual results. Specifically, it can be achieved using lightweight convolutional layers and regression heads, balancing computational accuracy and speed.

[0032] Specifically, multi-source sensor data, after preprocessing, enters the dynamic alignment module, which eliminates spatiotemporal biases by predicting time offsets and spatial transformation parameters. The aligned data is projected into a four-dimensional feature space, and a temporal encoder extracts motion features in the temporal dimension. The fusion network integrates semantic, geometric, and motion information from different sensors through a cross-modal attention mechanism, generating robust feature representations. The decoding stage maps the fused features to obstacle positions, motion trajectories, and scene semantics, outputting complete perception results. These steps form a closed-loop processing flow. The construction of the four-dimensional feature space overcomes the static limitations of traditional three-dimensional models, the dynamic alignment mechanism improves data consistency under high-speed motion, and cross-modal attention enables adaptive compensation when sensors fail.

[0033] Compared to existing technologies, traditional methods employ fixed-time interpolation and rigid coordinate transformations, which cannot adapt to spatiotemporal changes in dynamic environments. This application achieves accurate spatiotemporal alignment through a dynamic prediction model, capturing the temporal evolution of the scene using a four-dimensional feature space and enhancing feature complementarity through a cross-modal attention mechanism. Existing technologies rely on fixed-weight fusion, resulting in a sharp performance drop when sensors fail. This application maintains system stability through dynamic weight adjustment and feature space redundancy design. Traditional 3D models ignore the time dimension, leading to missed detections of dynamic objects. The four-dimensional modeling in this application effectively improves the detection accuracy of moving targets.

[0034] Through the above technical solutions, this application achieves high-precision spatiotemporal alignment of multi-sensor data, eliminating accumulated errors in moving scenes. It enhances the system's perception capabilities for occluded and rapidly moving objects by compensating for missing information through temporal feature backtracking. It improves system robustness in the event of sensor failure by maintaining perception accuracy using a dynamic weighting mechanism. Furthermore, it optimizes computational resource utilization, and the lightweight network design meets the real-time requirements of embedded platforms.

[0035] This application further proposes to achieve time synchronization and spatial coordinate unification of multi-sensor data through a dynamic spatiotemporal alignment module. The dynamic spatiotemporal alignment module predicts the time offset through an LSTM network and uses a learnable transformation matrix to compensate for spatial extrinsic parameter errors.

[0036] The dynamic spatiotemporal alignment module refers to a neural network structure used to eliminate temporal and spatial deviations in multi-source sensor data. Specifically, it can be implemented using a network architecture containing LSTM layers and fully connected layers. It predicts the temporal deviation at the current moment by inputting historical time offset data from the sensors. The LSTM network's prediction of the time offset utilizes the temporal modeling capabilities of the Long Short-Term Memory (LSTM) network to learn the non-fixed time deviation patterns caused by differences in sampling frequencies among different sensors. Specifically, a two-layer LSTM structure can be used to handle the correlation between the IMU's high-frequency timestamps and the low-frequency sampling intervals of other sensors. The learnable transformation matrix refers to a spatial coordinate transformation matrix dynamically adjusted through neural network parameters. Specifically, it can use the initial extrinsic parameter calibration results as input, combined with a projection error loss function, for end-to-end training, achieving autonomous compensation for extrinsic parameter installation errors and environmental disturbances.

[0037] Specifically, in the time synchronization phase, the high-frequency time series of the inertial measurement unit (IMU) is used as a reference. An LSTM network is used to model the sampling interval variation of low-frequency sensors such as LiDAR and cameras, predicting their offset relative to the reference time axis. Cubic spline interpolation is then used to align the low-frequency data to a unified time axis. In the spatial alignment phase, a differentiable coordinate transformation layer is constructed based on the initial sensor calibration extrinsic parameters. Backpropagation is used to optimize the transformation matrix parameters, minimizing the projection error of multi-sensor data in the vehicle coordinate system, thereby eliminating spatial offsets caused by mechanical vibration or temperature deformation. This scheme achieves dual calibration of spatiotemporal dimensions in dynamic scenarios by jointly optimizing time offset prediction and spatial transformation parameters.

[0038] Compared to existing technologies, traditional methods use fixed interpolation step sizes for time synchronization, which cannot adapt to dynamic fluctuations in sensor sampling frequency. Furthermore, spatial alignment relies on pre-calibrated fixed extrinsic parameter matrices, making it difficult to eliminate coordinate offsets caused by environmental disturbances. This solution uses an LSTM network to capture the nonlinear variation of time offset and combines this with a learnable transformation matrix to dynamically correct spatial projection parameters, thus solving the problem of insufficient alignment accuracy in high-speed motion or complex vibration scenarios using traditional hard synchronization methods.

[0039] Through the above technical solution, this application achieves fine alignment of multi-sensor data in time and space, effectively eliminates perception errors in dynamic scenes, provides an accurate spatiotemporal consistency basis for subsequent feature fusion, and enhances the system's robustness to sensor installation errors and environmental disturbances.

[0040] This application further proposes to project the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space for feature extraction and to construct a four-dimensional feature carrier. This includes projecting the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space to obtain projected data; extracting semantic features, geometric features, and motion features from the projected data; and integrating the semantic features, geometric features, and motion features to obtain a four-dimensional feature carrier.

[0041] The pre-defined four-dimensional feature space refers to a feature space containing three-dimensional spatial coordinates and a time dimension. Specifically, inverse perspective transformation or voxelization methods can be used to map multimodal data to a unified coordinate system, and the introduction of the time dimension enables continuous representation of dynamic scenes. Semantic features describe the category attributes of objects. These can be extracted from image data using convolutional neural networks or semantic segmentation models to identify static or dynamic targets in the environment. Geometric features reflect spatial structure and positional relationships. These can be extracted from LiDAR data using point cloud processing algorithms or 3D convolutional networks to capture the shape, size, and relative position of obstacles. Motion features characterize the dynamic changes of targets. These can be extracted from IMU or radar data using optical flow estimation or time series analysis to infer the speed and trajectory of objects. Integration refers to fusing features from different modalities. This can be achieved through feature stitching, channel weighting, or attention mechanisms to complement multi-dimensional information and eliminate the perceptual blind spots of a single feature dimension.

[0042] Specifically, spatiotemporally aligned multi-source data is mapped to a four-dimensional feature space through projection operations, forming a unified feature representation framework. Semantic feature extraction is based on deep learning models to parse image data and identify semantic categories such as roads, vehicles, and pedestrians. Geometric feature extraction is achieved through point cloud voxelization to quantify obstacle distribution and terrain structure in three-dimensional space. Motion feature extraction combines time-series data to analyze the displacement changes of targets in consecutive frames. These three types of features are extended in four-dimensional space through channel dimension expansion and are associated with the time dimension, forming a feature carrier containing spatial structure, semantic attributes, and dynamic information. This carrier complements information from different modalities through feature fusion mechanisms, such as using geometric features to correct boundary errors in semantic segmentation or using motion features to enhance the detection stability of dynamic targets.

[0043] Compared to existing technologies, traditional methods typically employ two-dimensional or three-dimensional feature spaces, lacking dynamic modeling capabilities in the temporal dimension, and feature fusion is often limited to a single modality. Existing technologies often integrate semantic and geometric features through simple feature concatenation, neglecting spatiotemporal correlation, leading to significant errors in motion trajectory prediction in dynamic scenes. This solution constructs a four-dimensional feature space, achieving spatiotemporal alignment of multimodal features within a unified dimension. Combined with time-series motion feature extraction, it effectively improves the accuracy of trajectory inference for fast-moving objects.

[0044] Through the above technical solution, this application solves the problem of incomplete perception caused by missing feature dimensions in dynamic scenes. By co-analyzing semantic, geometric, and motion features, it enhances the detection capability of occluded areas and fast-moving targets. The construction of a four-dimensional feature space allows temporal information to effectively participate in the feature fusion process, reducing misjudgments caused by instantaneous changes in the object's state and improving the accuracy of obstacle position and motion state estimation in complex environments.

[0045] This application further proposes to construct a fusion network, which includes temporal feature association and cross-modal feature complementarity. The feature dependency relationship of the same sensor is determined based on the temporal feature association, and the feature complementarity information of different sensors is determined based on the cross-modal feature complementarity. Based on the feature dependency relationship and the feature complementarity information, the multimodal features in the four-dimensional feature carrier are dynamically weighted and fused to generate a fusion feature body.

[0046] Temporal feature association refers to establishing temporal coherence by analyzing the data dependencies of the same sensor at different time points. This can be achieved using long short-term memory networks or self-attention mechanisms to capture the dynamic changes in sensor data over time. Cross-modal feature complementarity refers to identifying complementary information by mining the feature correlations between different sensors. This can be achieved using cross-attention mechanisms or feature similarity metrics to establish synergistic effects between multimodal data. Dynamic weighted fusion refers to adjusting the weight coefficients of each modality feature based on feature dependencies and complementary information. This can be achieved using gating mechanisms or reinforcement learning strategies to optimize the sensor contribution in real time.

[0047] Specifically, the temporal feature association module establishes a time-series dependency model of the sensor's own state by analyzing the feature change trends of the same sensor in consecutive time frames. For example, when the inertial measurement unit detects abnormal acceleration fluctuations in multiple consecutive frames, this module can identify the trend of declining data reliability. The cross-modal feature complementarity module identifies complementary information regions between different modes by calculating the correlation between features of different sensors. For example, when the image quality of the camera deteriorates due to backlighting, this module can enhance the weight of the LiDAR in geometric feature extraction. Based on the above analysis results, the dynamic weighted fusion module uses a learnable weight allocation strategy to fuse multimodal features, automatically reducing the fusion weight of abnormal data when a sensor fails, while increasing the contribution ratio of reliable sensors.

[0048] Compared to existing technologies, traditional methods employ a fixed-weight fusion strategy, which cannot dynamically adjust the fusion ratio based on sensor status, leading to a significant decline in overall performance when a single sensor fails. This solution addresses the insufficient robustness caused by static weight allocation by introducing temporal feature correlation and cross-modal feature complementarity mechanisms to establish an adaptive fusion model based on data quality.

[0049] Through the above technical solution, this application can automatically reduce the fusion weight of abnormal sensors and enhance the information contribution of other reliable sensors when sensors experience temporary failure or data quality degradation, effectively maintaining the stability of the environmental perception system. This solution is particularly suitable for complex operating environments with strong electromagnetic interference, extreme lighting conditions, or mechanical vibration, ensuring that drones or robots can still maintain reliable obstacle detection and pose estimation capabilities even when some sensors malfunction.

[0050] This application further proposes that after obtaining the fused feature body, the confidence level of each sensor is calculated based on the preprocessed data, and the weight of the sensor is adjusted when the confidence level is lower than a preset threshold.

[0051] Confidence level refers to the quantitative indicator of the reliability of sensor data under the current environment. It can be achieved through data consistency verification or error threshold judgment, such as comparing the measurement results of different sensors on the same target at the same time and calculating their degree of deviation. This indicator is used to dynamically evaluate the sensor's working status and provide a basis for weight adjustment. The preset threshold refers to the critical condition that triggers weight adjustment. It can be set using empirical values ​​or adaptive algorithms, for example, 0.6 based on historical data statistics. This threshold serves as a benchmark for judging whether a sensor has failed, ensuring that the system can respond promptly to abnormal situations.

[0052] Specifically, after the sensor data undergoes spatiotemporal alignment and feature fusion, a confidence score is generated by calculating the deviation between the output of each sensor and the fusion result. For example, when a camera image is overexposed due to backlighting, the difference between its detection result and the geometric features of the LiDAR point cloud increases. In this case, the confidence calculation module recognizes that the depth error of the sensor exceeds a preset threshold, triggering a weight adjustment mechanism. The attention layer in the fusion network dynamically reduces the feature weight of the sensor based on the confidence score, while increasing the contribution ratio of reliable sensors such as millimeter-wave radar. This process is achieved through online monitoring and real-time feedback, without relying on pre-set fixed weight parameters.

[0053] Compared to existing technologies, traditional methods employ fixed weight allocation strategies, which cannot cope with sudden sensor failures or environmental interference. This solution, through online confidence assessment and dynamic adjustment mechanisms, proactively optimizes the fusion strategy when sensor data quality deteriorates, avoiding the impact of erroneous features on the perception results. For example, in scenarios where lidar is affected by rain and fog, traditional methods still retain its high weight, leading to perception bias. This solution, however, can reduce its weight based on confidence level, instead relying on millimeter-wave radar and IMU data for compensation.

[0054] Through the above technical solution, this application can maintain the stability of the perception system when some sensors are interfered with or fail. For example, when the confidence level of the camera falls below the threshold due to strong light interference, the system automatically reduces the fusion weight of its semantic features and instead relies on the geometric features of the LiDAR and the motion features of the millimeter-wave radar for obstacle detection, thereby avoiding the overall performance degradation caused by the failure of a single sensor. This mechanism effectively solves the problem of insufficient robustness caused by the fixed weight allocation of traditional fusion models.

[0055] This application further proposes to perform feature decoding on the fused feature body and output the environmental perception result, including parsing the fused feature body to generate obstacle information, calculating pose parameters based on the motion constraint information in the fused feature body, and integrating the obstacle information and pose parameters to output the environmental perception result.

[0056] The fused feature volume refers to a high-dimensional spatiotemporal feature representation generated through the fusion of multi-sensor data. Specifically, it can be achieved using a four-dimensional spatiotemporal perception network to perform cross-modal interaction and time-series modeling of multimodal data. It includes a joint representation of obstacle geometric distribution, motion trends, and sensor pose changes. Obstacle information refers to the spatial location, size, and category attributes of movable or static objects in the environment. This can be achieved by using a deep learning-based detection head to decode the fused feature volume and generate bounding boxes and semantic labels, improving detection robustness in complex scenes through the fusion of multi-sensor features. Motion constraint information refers to the correlation features reflecting the sensor's own motion state and the dynamic changes of the scene. This can be achieved by performing integration or geometric transformation on the temporal motion vectors in the fused feature volume, coupling sensor pose changes with the motion of environmental obstacles. Pose parameters refer to the sensor's position and orientation information in a unified coordinate system. This can be achieved by using kinematic model-based optimization algorithms to solve for motion constraints, reducing the cumulative error of pose estimation by fusing multi-sensor motion features.

[0057] Specifically, a multi-stage decoding mechanism is used to achieve accurate output of environmental perception results. First, the high-dimensional feature representation capabilities of multi-sensor fusion are utilized to analyze the fused feature body, generating structured information containing obstacle location, size, and category, overcoming detection failures caused by occlusion or noise in dynamic scenes due to single sensors. Second, motion constraint information reflecting the sensor's motion state and the dynamic changes of the scene is extracted from the fused feature body. Pose parameters are calculated through kinematic equations and geometric transformations, avoiding error propagation caused by the separation of pose estimation and environmental perception in traditional methods. Finally, obstacle information and pose parameters are spatially correlated and physically constrained to establish a closed-loop consistency check between the absolute coordinates of obstacles and the relative pose of sensors, eliminating perception inaccuracies caused by coordinate drift or motion distortion.

[0058] Compared to existing technologies, traditional methods typically treat obstacle detection and pose estimation as independent modules, leading to error accumulation in dynamic scenes due to a lack of coupling analysis. For example, when the sensor moves rapidly, the relative position change of the obstacle is not synchronized with the pose parameter update, easily causing detection box jitter or positioning offset. This solution embeds motion constraint information into the fused feature volume to achieve joint calculation of obstacle motion and sensor pose, and uses spatiotemporal consistency constraints to eliminate the decoupling error between the two.

[0059] Through the above technical solution, this application solves the problem of inaccuracy in environmental perception results caused by the separation of obstacle detection and pose parameters in dynamic scenes. By using a multi-stage decoding and closed-loop verification mechanism, the consistency between the spatial position of obstacles and their own motion state is improved, and stable perception output can still be maintained when the sensor is moving at high speed or the scene changes drastically.

[0060] This application further proposes to evaluate the accuracy of the output environment perception results and calculate the perception error after outputting the results, and to adjust the parameters of the fusion network according to the error.

[0061] Accuracy evaluation refers to generating error metrics by quantitatively analyzing the difference between the perceived environmental conditions and the actual environmental state. This can be achieved through cross-validation or residual calculation, and is used to judge the performance of the current fusion network. Perception error refers to the deviation between the perceived environmental conditions and the real environment, which can be achieved using mean squared error or intersection-over-union ratio (IoU), providing data for parameter adjustment. Adjusting the parameters of the fusion network involves dynamically optimizing the network's internal weights based on error feedback. This can be achieved using gradient descent or reinforcement learning strategies, enabling the network to adapt to environmental changes.

[0062] Specifically, after generating the environmental perception results, a pre-defined evaluation module compares the results with real-world environmental data to calculate the perception error. This error reflects the performance deviation of the fusion network in the current environment. Based on the error magnitude and distribution characteristics, backpropagation algorithms or online learning mechanisms are used to adjust the weight parameters in the network, such as modifying the weight coefficients of the attention mechanism or updating the convolutional kernel parameters. This process forms a closed-loop control system, enabling the network to automatically optimize the feature fusion strategy according to real-time environmental changes, avoiding a decrease in perception accuracy due to fixed parameters. In dynamic scenes, when sensor data characteristics change, the error calculation module can quickly identify performance degradation, triggering the parameter adjustment mechanism to restore or improve perception accuracy.

[0063] Compared to existing technologies, traditional methods typically employ fixed-parameter fusion networks, which cannot adaptively adjust to environmental changes or sensor status, leading to performance degradation in dynamic scenarios. This solution introduces a closed-loop feedback mechanism, using perception error as the optimization objective to achieve dynamic adjustment of network parameters. This overcomes the limitations of fixed-parameter models and enhances the system's adaptability to complex environments.

[0064] Through the above technical solution, this application achieves online optimization of fusion network parameters, solves the problem of perception accuracy attenuation caused by static parameters, and improves perception stability in dynamic environments. Through the error feedback mechanism, the system can automatically adapt to changes in sensor characteristics or fluctuations in environmental conditions, maintaining high-precision perception output and enhancing robustness in complex application scenarios.

[0065] This application further proposes an environmental perception system, including an acquisition module, a calibration module, a construction module, a fusion module, and an output module. The acquisition module acquires multi-source sensor data and performs preprocessing; the calibration module synchronizes the preprocessed data in time and unifies its spatial coordinates; the construction module projects the spatiotemporally aligned data onto a four-dimensional feature space and constructs a feature carrier; the fusion module performs feature fusion through a data fusion network; and the output module decodes the fused feature carrier and outputs the environmental perception result.

[0066] The system comprises the following modules: **Acquisition Module:** This module receives raw data from multiple sensors and performs standardization processing. This can be achieved using data filtering algorithms and format conversion interfaces to eliminate sensor noise and unify data formats. **Calibration Module:** This module aligns the multi-source data spatiotemporally. This can be achieved using timestamp interpolation algorithms and coordinate system transformation matrices to eliminate spatiotemporal offset errors between sensors. **Construction Module:** This module maps multimodal data to a unified feature space. This can be achieved using inverse perspective transformation and voxelization methods to generate structured feature representations containing spatiotemporal information. **Fusion Module:** This module is a neural network unit that performs cross-modal feature interactions. This can be achieved using multi-head attention mechanisms and dynamic weighting strategies to achieve complementary fusion of multi-source information. **Output Module:** This module transforms high-dimensional features into executable results. This can be achieved using convolutional decoders and regression head structures to generate obstacle detection boxes and semantic segmentation results.

[0067] Specifically, the system constructs a complete data processing chain through a modular architecture. The acquisition module denoises and unifies the format of the raw sensor data, providing standardized input for subsequent processing. The calibration module employs a dual alignment mechanism, eliminating sampling frequency differences through time interpolation and resolving spatial reference inconsistencies through coordinate transformation. The construction module projects multimodal data onto a four-dimensional feature space, forming a unified representation containing geometric, semantic, and motion information. The fusion module achieves cross-modal feature interaction through neural networks and captures spatiotemporal correlations using an attention mechanism. The output module decodes the fused high-dimensional features into executable environmental perception commands, forming a closed-loop system from data acquisition to decision output. The collaborative work of these modules effectively solves problems such as spatiotemporal misalignment of multi-source data and poor fault tolerance of sensor failures.

[0068] Compared to existing technologies, most environmental perception systems employ fixed-weight fusion strategies, failing to dynamically adjust sensor contributions and relying on manual parameter calibration in the spatiotemporal alignment stage. This system achieves adaptive parameter adjustment through a learnable spatiotemporal alignment module and a dynamic weighted fusion mechanism. Compared to traditional rule-based feature fusion methods, the four-dimensional feature space constructed by this system can more comprehensively represent the spatiotemporal evolution characteristics of dynamic scenes, and its modular design facilitates flexible configuration for different sensor combinations.

[0069] Through the above technical solutions, this application effectively improves the reliability of environmental perception in complex dynamic scenarios. The system can automatically compensate for sensor installation errors and clock skew, maintaining stable perception accuracy even when the device is moving at high speed. When some sensors experience temporary failures, the dynamic weight adjustment mechanism can maintain the overall system performance and avoid sudden changes in perception results. The construction of a four-dimensional feature space enhances the system's ability to predict the trajectory of moving targets, especially in scenarios with visual occlusion, enabling the inference of the current environmental state through historical frame information. The modular architecture design gives the system good scalability, adapting to different types of sensor combinations and computing platform deployment requirements.

[0070] This application further proposes a computer device, including a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of an environmental perception method. The method includes: acquiring multi-source sensor data and preprocessing it; synchronizing the preprocessed data in time and unifying its spatial coordinates; projecting the spatiotemporally aligned data onto a four-dimensional feature space to construct a feature carrier; constructing a fusion network to dynamically weight and fuse the feature carriers; dynamically adjusting the fusion weights based on sensor confidence; decoding the fused feature body and outputting the perception result; and realizing real-time inference on an embedded platform through a knowledge distillation compression model.

[0071] Time synchronization refers to the process of unifying sensor data from different sampling frequencies to the same time reference. Specifically, this can be achieved by using the high-frequency timestamp of the inertial measurement unit as the reference and aligning the time axis of data from other sensors using interpolation algorithms, eliminating temporal misalignment caused by differences in sampling intervals. Spatial coordinate unification refers to the process of transforming multi-source sensor data to the same coordinate system. Specifically, this can be achieved by using a learnable transformation matrix to compensate for installation errors and resolve spatial deviations caused by differences in sensor positions. Four-dimensional feature carrier refers to a data structure that integrates three-dimensional spatial coordinates and the time dimension. Specifically, this can be achieved by projecting image features onto a bird's-eye view plane using inverse perspective transformation and combining this with a voxelized network to extract point cloud geometric features, thus achieving a unified representation of spatiotemporal information. Dynamic weighted fusion refers to adjusting the contribution of sensor data in the fusion process based on its quality. Specifically, this can be achieved by using reinforcement learning algorithms to update attention weights in real time, automatically reducing the weight of a failed sensor and increasing the feature proportion of reliable sensors. Knowledge distillation refers to the process of transferring knowledge from complex models to lightweight models. Specifically, this can be achieved by using a teacher-student model architecture to compress network parameters, reducing computation while preserving accuracy, and adapting to the computing power limitations of embedded platforms.

[0072] Specifically, after preprocessing, multi-source sensor data undergoes a time synchronization module to eliminate timing errors caused by sampling frequency differences. A spatial alignment module then transforms data from different coordinate systems to a unified benchmark. Projected into a four-dimensional feature space, the temporal information and three-dimensional spatial features are integrated into a continuous spatiotemporal carrier, providing structured input for subsequent fusion. The fusion network evaluates the confidence level of each sensor through a dynamic weighting mechanism, achieving real-time weight allocation at the hardware level to ensure stable system operation under abnormal conditions. The feature decoding module maps the fused high-dimensional features into understandable perceptual results and optimizes the model structure through knowledge distillation technology, enabling it to perform real-time inference tasks on an embedded processor.

[0073] In some specific implementations, time synchronization can be achieved by resampling low-frequency sensor data using linear interpolation or spline interpolation methods; spatial alignment can be achieved by combining calibration parameters with neural network optimization of the extrinsic parameter matrix; the construction of a four-dimensional feature space can introduce an attention mechanism to screen key spatiotemporal features; dynamic weighted fusion can be achieved by statistically analyzing the consistency index of sensor data based on a sliding window; and knowledge distillation can be achieved by combining feature matching and output distillation to compress the model size.

[0074] Compared to existing technologies, current embedded sensing devices typically employ fixed-weight fusion strategies, which are unable to cope with sensor failures or environmental interference. Furthermore, spatiotemporal alignment relies on pre-calibrated static parameters, making it difficult to compensate for dynamic errors. This solution achieves adaptive feature fusion under hardware resource constraints through a learnable spatiotemporal alignment module and a dynamic weight allocation mechanism, solving the robustness problem caused by fixed parameters in traditional methods. Simultaneously, by combining a four-dimensional feature space and lightweight network design, the computational complexity is reduced to a level manageable by embedded platforms while maintaining sensing accuracy.

[0075] Through the above technical solutions, this application realizes efficient spatiotemporal alignment and dynamic fusion of multi-sensor data on embedded devices, improves the accuracy of obstacle detection and pose estimation in complex environments, solves the problem of insufficient real-time performance caused by computational resource limitations in traditional methods, and enhances the system's fault tolerance in the event of sensor anomalies through an adaptive weight adjustment mechanism, providing a reliable embedded environmental perception solution for mobile platforms such as drones and robots.

[0076] This application further proposes a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of an environment-aware method.

[0077] Computer-readable storage media refers to the physical carrier used to persistently store algorithm instructions. Specifically, it can be implemented using solid-state drives, flash memory chips, or embedded memory. Its function is to solidify the preprocessing, feature fusion, and decoding processes involved in the environmental perception method into a set of executable binary instruction sequences. The execution of these computer-readable instructions by the processor refers to the parsing and computation of the instructions in the storage medium by the central processing unit (CPU) or graphics processing unit (GPU). Specifically, it can be implemented using ARM architecture processors or GPU acceleration chips. Its function is to transform the algorithm logic into hardware-executable operations, completing end-to-end computation from data input to perception result output.

[0078] Specifically, this technical solution encodes the complete algorithm flow of the environmental perception method into computer-readable instructions and stores them in a physical medium, enabling the processor to directly call a pre-defined standardized instruction set. During embedded device operation, the processor reads instructions one by one from the storage medium and sequentially executes data preprocessing, spatiotemporal alignment, feature fusion, and decoding operations, avoiding the computational delays associated with dynamically loading model parameters in traditional methods. Simultaneously, the storage medium supports receiving updated instruction data via a wireless communication module, enabling dynamic adjustment of the fusion network parameters. For example, when sensor performance degrades, model optimization can be completed without re-flashing the firmware by updating the weight allocation strategy in the instructions online.

[0079] Compared to existing technologies, current embedded systems typically deeply couple perception algorithms with hardware firmware, leading to the need to redeploy the entire system for model updates and frequent access to external storage devices during real-time computation, causing processing delays. This solution separates the storage medium from the processor logic, encapsulating algorithm instructions independently in an erasable and rewritable storage unit. This reduces data transfer overhead during computation and enables dynamic updating of algorithm parameters, solving the computational efficiency and model update challenges of resource-constrained platforms.

[0080] Through the above technical solutions, this application realizes the efficient deployment of environmental perception algorithms on embedded devices, reduces the processor's computational load, and ensures real-time requirements; at the same time, it supports dynamic optimization of model parameters through instruction updates, improves the system's adaptability to scenarios with changing sensor performance, and avoids the problem of decreased perception performance caused by the difficulty of firmware updates in traditional methods.

[0081] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0083] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0084] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 3 , Figure 3This is a basic structural block diagram of the computer device in this embodiment.

[0085] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only computer device 4 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0086] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0087] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the # method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0088] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the # method.

[0089] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0090] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the environment-awareness method described above.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0092] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. An environmental perception method, characterized in that, include: Acquire multi-source sensor data, and preprocess the multi-source sensor data to obtain preprocessed data; The preprocessed data is synchronized in time and unified in spatial coordinates to obtain the preprocessed data after spatiotemporal alignment. The preprocessed data after spatiotemporal alignment is projected onto a preset four-dimensional feature space for feature extraction, and a four-dimensional feature carrier is constructed. A fusion network is constructed, and the four-dimensional feature carrier is fused through the fusion network to obtain a fused feature body; The fused feature body is decoded to output the environment perception result; The step of synchronizing the preprocessed data in time and unifying its spatial coordinates includes: Based on the high-frequency timestamp of the inertial measurement unit in the preprocessed data, predict the time offset of other sensor data in the preprocessed data; Based on the time offset, the preprocessed data is interpolated and adjusted to obtain the time-synchronized preprocessed data; The preprocessed data after time synchronization is unified in spatial coordinates to obtain the preprocessed data after spatiotemporal alignment. The step of projecting the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space for feature extraction and construction of a four-dimensional feature carrier includes: The preprocessed data, after spatiotemporal alignment, is projected onto a preset four-dimensional feature space to obtain the projected data; Semantic features, geometric features, and motion features are extracted from the projected data; The semantic features, geometric features, and motion features are integrated to obtain a four-dimensional feature carrier; The construction of the fusion network, which involves fusing features of the four-dimensional feature carrier to obtain a fused feature body, includes: Construct a fusion network, which includes temporal feature association and cross-modal feature complementarity; Based on the temporal feature association, the feature dependencies of the same sensor are determined; Based on the cross-modal feature complementarity, feature complementarity information of different sensors is determined; Based on the feature dependency relationship and the feature complementarity information, the multimodal features in the four-dimensional feature carrier are dynamically weighted and fused to generate a fused feature body.

2. The environmental perception method according to claim 1, characterized in that, After obtaining the fused feature body, the process includes: Based on the preprocessed data, calculate the confidence level of each sensor; When the confidence level is lower than a preset threshold, the weight of the sensor is adjusted.

3. The environmental perception method according to claim 1, characterized in that, The step of decoding the fused feature body and outputting the environment perception result includes: The fused feature body is parsed to generate obstacle information; Calculate pose parameters based on the motion constraint information in the fused feature body; The obstacle information and the pose parameters are integrated to output the environmental perception result.

4. The environmental perception method according to claim 1, characterized in that, After performing feature decoding on the fused feature body and outputting the environment perception result, the method further includes: The accuracy of the environmental perception results is evaluated to obtain the perception error; The parameters of the fusion network are adjusted based on the perceived error.

5. An environmental sensing system, characterized in that, include: The acquisition module is used to acquire multi-source sensor data and preprocess the multi-source sensor data to obtain preprocessed data. The calibration module is used to synchronize the time and unify the spatial coordinates of the preprocessed data; The construction module is used to project the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space, extract features, and construct a four-dimensional feature carrier. A fusion module is used to construct a data fusion network, and to perform feature fusion on the four-dimensional feature carrier through the data fusion network to obtain a fused feature body. The output module is used to perform feature decoding on the fused feature body and output the environment perception result; The step of synchronizing the preprocessed data in time and unifying its spatial coordinates includes: Based on the high-frequency timestamp of the inertial measurement unit in the preprocessed data, predict the time offset of other sensor data in the preprocessed data; Based on the time offset, the preprocessed data is interpolated and adjusted to obtain the time-synchronized preprocessed data; The preprocessed data after time synchronization is unified in spatial coordinates to obtain the preprocessed data after spatiotemporal alignment. The step of projecting the spatiotemporally aligned preprocessed data onto a preset four-dimensional feature space for feature extraction and construction of a four-dimensional feature carrier includes: The preprocessed data, after spatiotemporal alignment, is projected onto a preset four-dimensional feature space to obtain the projected data; Semantic features, geometric features, and motion features are extracted from the projected data; The semantic features, geometric features, and motion features are integrated to obtain a four-dimensional feature carrier; The construction of the fusion network, which involves fusing features of the four-dimensional feature carrier to obtain a fused feature body, includes: Construct a fusion network, which includes temporal feature association and cross-modal feature complementarity; Based on the temporal feature association, the feature dependencies of the same sensor are determined; Based on the cross-modal feature complementarity, feature complementarity information of different sensors is determined; Based on the feature dependency relationship and the feature complementarity information, the multimodal features in the four-dimensional feature carrier are dynamically weighted and fused to generate a fused feature body.

6. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the environmental perception method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the environmental perception method as described in any one of claims 1 to 4.

Images