Multi-modal environment perception system and method based on super data large model

Abstract

The application discloses a multi-modal environment perception system and method based on a super data large model, relates to the technical field of environment perception, and comprises a multi-modal sensor array module used for synchronously collecting multi-source perception data in an unstructured environment, a multi-modal fusion module used for performing weighted fusion processing on the multi-source perception data collected by the multi-modal sensor array module, a super data large model modeling module that adopts a graph structure modeling mode to combine time sequence perception features and node structure relationships to perform high-dimensional feature extraction and semantic modeling, an edge-cloud collaborative computing module that performs computing scheduling between edge computing nodes and cloud servers, and a modular interface and a low-cost integrated module that realize module-level low-cost deployment of the system. The application solves the core problems of the traditional system, such as insufficient perception dimension, difficult data fusion and weak intelligent processing in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology, specifically to a multimodal environmental perception system and method based on a super data large model. Background Technology

[0002] Environmental perception is a technology that involves sensing, analyzing, and understanding the surrounding environment through sensors, algorithms, and data processing techniques. Its main components include sensors, data processing and analysis, data fusion, and decision control. Sensors are devices used to acquire environmental information, such as cameras, radar, lidar, infrared sensors, and temperature and humidity sensors; different types of sensors capture different types of data. Data processing and analysis involves processing and analyzing the collected data using algorithms to identify and understand objects, locations, and states in the environment. Data fusion combines data from different sensors to obtain a more complete and accurate environmental perception. Decision control allows the system to make decisions and control actuators based on the results of environmental perception.

[0003] In highly dynamic and unstructured complex environments, such as urban roads, post-disaster ruins, or extreme climate zones, traditional environmental sensing systems face multiple challenges. First, they have limited sensing dimensions and often rely on a single type of sensor (such as a camera or radar), making it difficult to cover multi-dimensional environmental variables such as temperature, humidity, air pressure, and electromagnetic interference. Second, the inconsistent data formats and sampling frequencies among heterogeneous sensors lead to low data fusion efficiency and complex processing. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a multimodal environment perception system and method based on a super data large model.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] This invention provides a multimodal environment perception system based on a super data large model, comprising:

[0007] A multimodal sensor array module, comprising multiple heterogeneous sensing units arranged on the surface of a spherical shell, for synchronously acquiring multi-source sensing data in an unstructured environment;

[0008] A multimodal fusion module is used to perform weighted fusion processing on the multi-source sensing data collected by the multimodal sensor array module to construct a continuous spatiotemporal sensing map to represent the changes in environmental state with time and spatial dimensions.

[0009] The super data large model modeling module is based on the spatiotemporal perception map constructed by the multimodal fusion module. It adopts a graph structure modeling method to combine temporal perception features and node structure relationships to perform high-dimensional feature extraction and semantic modeling.

[0010] The edge-cloud collaborative computing module, based on a preset data importance judgment function, performs computing scheduling between edge computing nodes and cloud servers to achieve hierarchical collaboration and dynamic allocation of data processing;

[0011] Modular interfaces and low-cost integrated modules adopt a unified power and communication protocol, are compatible with multiple platform forms, and are combined with a cost control constraint model to achieve low-cost modular deployment of the system.

[0012] As a preferred embodiment of the present invention, the multimodal sensor array module is constructed as a compound eye spatial structure, and each of the heterogeneous sensing units is arranged on the outer surface of the spherical shell with a fixed field of view.

[0013] As a preferred technical solution of the present invention, the multimodal fusion module is configured to perform fusion processing based on the raw data collected by multiple heterogeneous sensing units at spatial locations and time points, combined with a preset weighting factor.

[0014] The weighting factors of each sensing unit are dynamically adjusted based on the historical sampling accuracy and reliability assessment of this type of sensor;

[0015] During the fusion process, the phase difference signal extracted between the radar channel and the ultrasonic channel is further combined to enhance the fusion effect of spatial perception data through phase interference relationship, and a fusion perception map corresponding to time and space is constructed to express the environmental state.

[0016] As a preferred technical solution of the present invention, the super data large model modeling module adopts a spatiotemporal graph neural network structure to construct a semantic graph using multimodal perception data;

[0017] This network generates high-dimensional feature representations by weighting and mapping the feature vectors between each node and its neighboring nodes in each layer.

[0018] During the feature mapping process, the input data is linearly transformed and a bias term is added by combining the adjacency relationship between nodes and the normalization factor. Then, a nonlinear mapping is performed through the activation function to realize the transfer and extraction of feature information between layers.

[0019] As a preferred embodiment of the present invention, the anomaly detection module processes the input multimodal data based on an autoencoder architecture and outputs reconstructed data samples through a reconstruction network;

[0020] The system compares the error value between the original input data and the reconstructed data, and makes a judgment based on the comparison between the error value and the preset error threshold;

[0021] When the error value exceeds the preset threshold, the system determines that there is an abnormal situation, triggers the module redundancy mechanism, and replaces the corresponding sensor node to maintain the integrity of the system's perception.

[0022] As a preferred technical solution of the present invention, the edge-cloud collaborative computing module is configured to perform dynamic path scheduling processing based on task event levels;

[0023] When an event to be processed meets the preset critical event level conditions, the system selects to execute the task through the cloud server; if the critical event level conditions are not met, the task is processed by the edge computing node.

[0024] The system calculates the edge processing latency and cloud processing latency based on the judgment condition, and determines the final overall computing path and task scheduling strategy accordingly.

[0025] As a preferred embodiment of the present invention, the modular interface and low-cost integration module are constructed according to cost control constraints.

[0026] The total system cost consists of the unit costs of the sensing module, power supply module, control module, and integration module, and the functional deployment is limited to a set cost threshold.

[0027] Each module adopts a unified power and communication protocol and supports hot-swapping and automatic identification of each functional module, ensuring the system's interconnectivity under different platform configurations.

[0028] As a preferred technical solution of the present invention, the weighting factors of various sensors in the fusion perception map model are updated based on the sensor reliability assessment score at the current time point and the historical weight values.

[0029] This update mechanism combines a smoothing factor to perform a weighted average of the current reliability and historical weights, and then normalizes the average of the weighting factors of all sensors. This allows for dynamic adjustment of the participation weights of each type of sensor, achieving temporal adaptation and dynamic response in weight allocation.

[0030] As a preferred embodiment of the present invention, the system further includes an energy supply and material structure module, which includes a flexible perovskite photovoltaic structure, a piezoelectric nanogenerator based on ZnO nanowires, and a UV-resistant and biodegradable composite encapsulation material, for providing electrical energy and adapting to various deployment environments.

[0031] This invention also provides a multimodal environment perception method based on a super data large model, comprising the following steps:

[0032] Multi-source sensing data is simultaneously acquired by a multi-modal sensor array module deployed on the surface of a spherical shell.

[0033] Multi-source sensing data is input into a multimodal fusion module, and a continuous spatiotemporal sensing map is constructed based on weighted and phase interference enhancement mechanisms to characterize the dynamic changes of environmental state with time and space.

[0034] Using the super data large model modeling module, based on the constructed spatiotemporal perception map, graph neural networks are used to perform high-dimensional semantic modeling of the temporal features and structural relationships of nodes, and an anomaly detection mechanism is integrated to evaluate the reconstruction error in order to identify abnormal data.

[0035] In the edge-cloud collaborative computing module, computing tasks are dynamically allocated between edge computing nodes and cloud servers based on the event level judgment function, thereby optimizing the overall system processing latency.

[0036] Finally, a unified communication and power interface for the system is achieved through modular interfaces and low-cost integration modules, and the total integration cost is controlled based on a cost constraint model to meet the needs of rapid deployment across multiple platforms.

[0037] The beneficial effects of this invention are:

[0038] This invention systematically solves the core problems of traditional systems, such as insufficient perception dimensions, difficulty in data fusion, and weak intelligent processing, through technical solutions such as full-dimensional perception, multimodal fusion, graph neural semantic modeling, intelligent computing scheduling, and modular deployment. Attached Figure Description

[0039] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0040] In the attached diagram:

[0041] Figure 1 This is a schematic diagram of the overall process of the multimodal environment perception system of the present invention;

[0042] Figure 2 This is a flowchart of the multimodal fusion process of the super data big model of the present invention. Detailed Implementation

[0043] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0044] Example 1

[0045] like Figure 1 As shown, the multimodal environment perception system based on a super data large model includes:

[0046] A multimodal sensor array module, comprising multiple heterogeneous sensing units (similar to a bionic compound eye structure) arranged on the surface of a spherical shell, is used to simultaneously acquire multi-source sensing data such as image data, radar data, ultrasonic data, temperature and humidity data, air pressure data, and electromagnetic field strength data in an unstructured environment.

[0047] The module is designed to perceive a wide range of environmental factors in complex, unstructured environments. The spherical shell layout maximizes coverage of multiple dimensions in the environment, improving the comprehensiveness and accuracy of data collection.

[0048] The selection of heterogeneous sensing units is based on the characteristics of different sensing data. For example, image data can provide visual information about the environment, radar and ultrasonic data can be used for distance and position perception, temperature, humidity and air pressure data can reflect the climate characteristics of the environment, and electromagnetic field strength data can help determine changes in the electromagnetic environment. When different sensors operate simultaneously, multi-dimensional environmental data can be obtained through synchronous acquisition, which provides a rich information foundation for subsequent data fusion and modeling.

[0049] A multimodal fusion module is used to perform weighted fusion processing on the multi-source sensing data collected by the multimodal sensor array module to construct a continuous spatiotemporal sensing map to represent the changes in environmental state with time and spatial dimensions.

[0050] The core idea of ​​weighted fusion is to assign different weights based on the accuracy, reliability and applicability of the data collected by each sensor in a specific environment. Specifically, image data and radar data may have larger weights because they can provide relatively intuitive and high-resolution environmental information, while data such as temperature, humidity and air pressure have smaller weights, but they can provide key environmental change information in specific scenarios.

[0051] For the construction of the spatiotemporal perception map, the map not only considers the time dimension of the data, but also combines the spatial dimension information. The time dimension reflects the dynamic changes of the environmental state over time, while the spatial dimension reflects the distribution of the perceived data through the spatial layout of the sensor array. Through this spatiotemporal fusion, a map that accurately reflects the changes of the environmental state over time and space can be constructed.

[0052] The super data large model modeling module is based on the spatiotemporal perception map constructed by the multimodal fusion module. It adopts a graph structure modeling method to combine temporal perception features and node structure relationships to perform high-dimensional feature extraction and semantic modeling.

[0053] In this module, based on time-series sensing features and node structure relationships, a deep learning model is used to extract high-dimensional features. Specifically, time-series sensing features mainly refer to the environmental change patterns extracted from time-series data, while node structure relationships refer to the network structure of interactions between different sensors. These features will be input into the super data model to achieve efficient modeling and analysis of environmental conditions.

[0054] For example, in a specific scenario, temperature and humidity data may affect the display of certain objects in image data, while radar data may provide information on the position and distance of objects in image data. By using graph structure modeling, the interrelationships of these data can be clearly expressed, and optimization and adjustment can be performed under various environmental conditions.

[0055] The edge-cloud collaborative computing module, based on a preset data importance judgment function, performs computing scheduling between edge computing nodes and cloud servers to achieve hierarchical collaboration and dynamic allocation of data processing;

[0056] This module uses a data importance judgment function to rationally allocate tasks between the edge and the cloud based on the real-time needs and computing load of each data point, thereby optimizing the utilization efficiency of computing resources and improving the overall performance of the system.

[0057] At edge computing nodes, preliminary data processing and rapid response can be performed, reducing reliance on the cloud and improving real-time performance and local processing capabilities. Meanwhile, on cloud servers, large-scale data processing and deep learning model training can be carried out, further improving the accuracy and predictive ability of the models. Through this layered collaboration and dynamic allocation, the use of computing resources can be flexibly adjusted according to the different characteristics of the data, avoiding waste of system resources.

[0058] Modular interfaces and low-cost integrated modules adopt a unified power and communication protocol, are compatible with multiple platform forms, and are combined with a cost control constraint model to achieve low-cost deployment of the system at the module level. This modular design can greatly reduce the system's production and maintenance costs, while improving the system's scalability and flexibility.

[0059] The multimodal environmental perception system of this invention collects environmental data through a multimodal sensor array module, and processes and models the data efficiently through a multimodal fusion module and a super data large model modeling module. Combined with edge-cloud collaborative computing and intelligent energy supply design, this invention can achieve low-latency, accurate and efficient environmental perception and analysis in complex environments, providing strong technical support for multiple fields such as intelligent environmental monitoring, autonomous driving, and drone navigation.

[0060] Furthermore, the multimodal sensor array module is constructed as a compound eye spatial structure, with each heterogeneous sensing unit arranged on the outer surface of the spherical shell with a fixed field of view, forming a full-space acquisition model with a sensing field coverage of 360°×180°. The main advantage of this structure is that it can achieve all-round perception without blind spots in three-dimensional space, ensuring that information acquisition in all directions is not missed.

[0061] In this invention, heterogeneous sensing units are arranged in a regular manner on the outer surface of a spherical shell. Each sensing unit has a fixed field of view, ensuring that the entire space is uniformly covered. This design not only solves the blind spot problem caused by the limited field of view of traditional sensors, but also enhances the ability to perceive environmental details through the compound eye structure.

[0062] The heterogeneous sensing unit is typically composed of different types of sensors, such as vision sensors, lidar, and ultrasonic sensors. Each sensing unit has an independent identification code for easy identification and data management, and maintains clock synchronization with the central fusion control unit to reduce data drift caused by asynchronous sampling during the acquisition process.

[0063] Furthermore, the multimodal fusion module performs weighted fusion of the multi-source sensing data based on the following mathematical model:

[0064]

[0065] Where M(x,y,t) represents the fused perception map at spatial location (x,y) and time t. This map integrates data from different sensors and is the core perception result output by the system, reflecting the all-round state of the environment.

[0066] S i (x,y,t) represents the raw data collected by the heterogeneous sensing unit of type i at this location and time point. This data may come from different types of sensors, such as image data, radar echoes, ultrasonic signals, etc.

[0067] w iThis represents the weighting factor corresponding to the i-th type of sensor. Its value is dynamically adjusted based on the historical sampling accuracy and reliability assessment of the sensor. For example, the accuracy of the sensor decreases after long-term use, or the reliability of the sensor changes under certain environments (such as strong light, rain, snow, etc.). These factors can be reflected by adjusting the weight. The basis for the dynamic adjustment of the weight is that the weight is set based on the historical performance data of the sensor (such as accuracy, stability, etc.) and the influence of environmental conditions. Specifically, the acquisition accuracy of the sensor in different time periods and under different environments will affect its weight. For example, in sunny weather, the weight of the visual sensor may be higher, while in smoggy weather, the weight of the lidar may need to be increased.

[0068] α is the phase interference enhancement factor parameter, used to correct phase interference effects during the fusion process. In multimodal sensors (such as radar and ultrasound), interference may occur due to signal phase differences, affecting the accuracy of the sensing results. This factor reduces the negative impact of interference on data fusion by optimizing the phase difference.

[0069] Δφ r With Δφ u These are the phase difference signals extracted from the radar channel and the ultrasonic channel, respectively. The phase difference between these two channels may have shifts or errors, cos(Δφ) r -Δφ u ).

[0070] This weighted fusion algorithm takes into account the acquisition quality and phase interference effect of different sensors, so that when multiple modal sensors work together, the system can optimally combine the output data of each sensor, reducing the perception distortion caused by noise or error of individual sensors.

[0071] Weighting factor w i The introduction of the phase interference enhancement factor α ensures that the system can still provide accurate perception information in dynamic environments, especially in complex application scenarios, such as intelligent driving systems traveling at high speeds and navigation and positioning of drones in changing environments.

[0072] w i The weighting range can be adjusted based on the sensor's historical performance. Generally, the weighting range is between 0 and 1, where 1 indicates that the sensor data has the highest reliability and accuracy.

[0073] The value of α is generally set between 0.1 and 1.0, and the specific value is adjusted according to the phase difference between radar and ultrasonic sensors in different application scenarios.

[0074] Furthermore, such as Figure 2As shown, in multimodal environmental perception systems, especially in typical application scenarios such as intelligent transportation, smart cities, and autonomous driving, the system faces the challenges of high-dimensional complexity of multi-source heterogeneous data and spatiotemporal dynamic changes. Traditional models based on single-modal perception or static neural networks are difficult to effectively extract spatiotemporal correlation features from the perceived data, resulting in insufficient semantic understanding capabilities and low accuracy of environmental cognition.

[0075] To address this, the present invention proposes a multimodal environment perception system based on a super data large model, which achieves unified modeling of the spatial topology and temporal dynamics between sensors. The super data large model modeling module uses a spatiotemporal graph neural network structure to construct a semantic graph of the multimodal perception data. The node feature mapping function of the modeling network between the l-th layer and the (l+1)-th layer is expressed as follows:

[0076]

[0077] in, Let W represent the feature vector of the i-th node in the (l+1)-th layer of the graph, where N(i) represents the set of nodes adjacent to node i. (l) and B (l) These are the weight matrix and bias vector of the l-th layer, respectively. (l) Used for linear transformation of features, initialized with standard Gaussian, with dimensions set according to input and output features, B (l) The initial value is set to 0. The number of network layers L is generally set to 2 to 4 to prevent over-smoothing, and the specific value is adjusted according to the system size. ij The normalization factor between node i and node j is used to prevent numerical instability during feature aggregation and is defined as follows:

[0078]

[0079] σ is the activation function, which can be either the ReLU function or the Tanh function, depending on the specific scenario.

[0080] This formula originates from the classic Graph Convolutional Network (GCN), which achieves spatial neighborhood information fusion by weighted feature aggregation between nodes. Considering that this system is for perceptual modeling in a dynamic environment, a temporal encoder is introduced, and temporal modeling is embedded in the graph network by fusing time step feature vectors in the graph structure.

[0081] In autonomous driving scenarios, such as multiple sensors perceiving the vehicle's surrounding environment, This formula can represent the observation features of sensor j at time t, such as visual features and object detection vectors. By fusing these features, a semantic association graph can be effectively constructed, thereby improving the robustness of perception.

[0082] Furthermore, in actual deployment, multimodal sensing systems may face situations such as sensor failure, occlusion, and false detection. Traditional methods often lack dynamic detection and fault tolerance mechanisms, leading to a decrease in system reliability. To improve the stability and adaptability of environmental sensing systems, this invention introduces an anomaly detection module to identify and process abnormal patterns in the output of sensing nodes.

[0083] The super data large model modeling module further includes an anomaly detection module. The anomaly detection module reconstructs the input multimodal data based on an autoencoder structure and determines anomalies based on the reconstruction error. The reconstruction error is expressed as follows:

[0084]

[0085] Where X represents the original input multimodal sensing data. The data represents the reconstructed data of the autoencoder structure. LossAE represents the squared Euclidean distance between the original and reconstructed data, which is used to measure information loss. When LossAE > δ, where δ is a preset error threshold, the module redundancy mechanism in the system is triggered to replace faulty or inefficient sensing nodes.

[0086] Self-encoder structure design:

[0087] The encoder consists of 2 to 3 fully connected layers, compressing the input dimension layer by layer to the latent representation Z;

[0088] Decoder: Reconstructs the latent representation, restoring the original data as much as possible;

[0089] Activation function: ReLU or Sigmoid, depending on the input distribution;

[0090] Training objective: Minimize Loss AE This improves the system's ability to fit normal data.

[0091] The threshold δ should be set based on the system's sensitivity to error; it is recommended to set it to the loss threshold on the training set. AE The mean μ plus k times the standard deviation σ, that is:

[0092] δ=μ+kσ,k∈[2,4];

[0093] In this invention, it is recommended that k=3, that is, to control the error using the 3σ principle (δ=μ+3σ) to ensure that more than 95% of normal samples are within this range.

[0094] When the detected reconstruction error exceeds the threshold δ, the system automatically triggers the module redundancy mechanism, which includes:

[0095] Mark abnormal nodes;

[0096] Activate backup sensing modules (such as auxiliary cameras or virtual sensors);

[0097] Dynamically adjust the structure of the graph neural network to remove or reduce the influence of abnormal nodes;

[0098] Update the adjacency graph weights to reflect the latest trust assessment.

[0099] Example of use case: In a smart factory, multiple sensors work together to monitor equipment status. If a temperature sensor generates abnormal data due to a hardware failure, the anomaly is identified through a self-encoder detection mechanism, and the system immediately activates redundant sensing modules to prevent abnormal data from affecting production decisions.

[0100] Furthermore, multimodal environmental perception systems are playing an increasingly important role in complex and dynamic scenarios such as intelligent transportation, unmanned systems, and disaster early warning. However, such systems often face two major technical bottlenecks:

[0101] Uneven distribution of computing resources and difficulty in latency control: When the sensing system is deployed at the edge, it needs to respond in real time, but edge devices have limited computing resources and it is difficult to handle highly complex multimodal fusion tasks. If cloud computing is frequently called, communication latency will be introduced, reducing the system response speed and stability.

[0102] Poor environmental adaptability of energy supply and system structure: Traditional energy solutions (such as lithium batteries) are inefficient in extreme environments (high temperature, low light, frequent vibration), the system materials are conventional and have insufficient environmental resistance, which is not conducive to long-term deployment and sustainable operation of the system.

[0103] In this invention, to address the problem of unreasonable resource allocation between the edge and cloud during complex task processing, an edge-cloud collaborative computing module is set up to determine the computing path based on the following delay scheduling model:

[0104] D total =D edge +I(E critical )·D cloud ;

[0105] Among them, D total D represents the total latency of the overall system computation and processing (the system's latency during the perception-decision-response process). edge This represents the latency of processing at edge computing nodes, primarily constrained by local computing power, sensor data preprocessing overhead, and lightweight model runtime. (D) cloud This indicates the latency in completing the processing on the cloud server, including the latency of data upload, cloud processing, and result delivery. I(E) criticalThis is the event level judgment function. When the event level meets the key judgment conditions, its value is 1, and the system triggers the cloud computing path; otherwise, its value is 0, and it is only processed on the edge side.

[0106] D edge The range is 20ms to 200ms, which varies depending on the configuration of the edge device;

[0107] D cloud (Unit: milliseconds) depends on the network environment and task complexity, and is usually 300ms to 2s.

[0108] Furthermore, in the field of intelligent sensing systems, especially multimodal environmental sensing, the high complexity of system component integration, the difficulty in cost control, and the lack of unified interface standards between modules are common problems.

[0109] To this end, the modular interface and low-cost integrated module include a unified interface structure using the Type-C protocol, supporting PD power supply and USB 3.1 high-speed data transmission; the physical connection structure of each functional module has hot-swappable attributes, so inserting or removing modules during operation will not affect the overall stability of the system, and the initial configuration is achieved through an automatic identification protocol. The system automatically completes module identification and configuration initialization through an embedded identification protocol (such as a module identification mechanism based on UUID), improving the dynamic scalability and maintenance efficiency of the system;

[0110] The following is one embodiment of unit cost, where the overall unit cost of the system is constrained by the following cost constraint model:

[0111] C unit =C sensor +C energy +C control +C integration ≤200;

[0112] Among them, C unit C represents the total cost of the module (unit: yuan). sensor C energy C control With C integration These represent the unit costs of the sensing, power supply, control, and integration components, respectively.

[0113] C sensor Including IMU, camera, sound sensor, etc., C energy Including a Type-C power supply interface, a power management chip, and C control Including STM32, ESP32 and other MCUs, integrating WiFi / BT, etc., C integration This includes physically integrated components such as housings, connectors, and PCB packaging.

[0114] Furthermore, the weights w of various sensors in the fused sensing map model i The following update mechanism will be used for dynamic adjustments:

[0115]

[0116] Among them, R i (t) represents the reliability assessment score of the i-th type of sensor at time t, γ is a smoothing factor used to balance the current weight and the latest reliability, and the recommended value range is γ∈(0.2,0.5), and N is the total number of the heterogeneous sensing units.

[0117] The larger γ is, the more "sensitive" the system is to changes in current reliability, making it suitable for highly dynamic scenarios; the smaller γ is, the more "stable" the system is, making it suitable for scenarios with high accuracy requirements but slow environmental changes. Generally, setting γ = 0.3 can achieve a good balance between robustness and adaptability.

[0118] Furthermore, to address the shortcomings of existing sensing systems that are difficult to deploy in extreme environments for extended periods, the system also includes an energy supply and material structure module. This energy supply and material structure module includes a flexible perovskite photovoltaic structure, a piezoelectric nano-power generation unit based on ZnO nanowires, and a UV-resistant and biodegradable composite encapsulation material, which is used to provide electrical energy and adapt to various deployment environments.

[0119] The energy supply and material structure module includes a flexible perovskite photovoltaic material module for continuously powering the system in low-light environments. This module uses flexible perovskite photovoltaic material, which has excellent photoelectric conversion efficiency (up to 21%), and also has the advantages of being lightweight, flexible, and low-cost, making it suitable for deployment on the surface of the system casing. It can maintain a high power output even in low-light environments (such as cloudy days, smog, and indoors).

[0120] A piezoelectric vibration energy harvesting structure based on a ZnO nanowire array is embedded in the surface of the system's support or shell. In scenarios such as transportation and industry, the system is often subjected to low-frequency vibration or periodic stress environments. Utilizing the piezoelectric properties of the zinc oxide (ZnO) nanowire array, mechanical vibration is converted into electrical energy. This structure, embedded in the support or shell surface, can be integrated with the system's micro-energy management module. The resonant frequency range is 10–200 Hz, the energy harvested per vibration is 0.1–1.2 microjoules (μJ), and the peak output voltage is >0.8 V.

[0121] The outer shell is made of a biodegradable composite material that has undergone polymer modification. This shell material possesses UV resistance and is suitable for deployment in extreme environments. It can be made of polylactic acid (PLA) and nano-TiO2 composite material, with polymer modification improving the material's mechanical strength and UV resistance, while also possessing biodegradability, meeting the requirements of sustainable development. This shell is suitable for extreme deployment scenarios such as deserts, high-radiation environments, and high-temperature and high-humidity environments. Its UV resistance rating is ≥4 (ISO4892-3 standard), tensile strength is >45MPa, and operating temperature range is -20℃ to +85℃.

[0122] Example 2

[0123] This invention also provides a multimodal environment perception method based on a super data large model, comprising the following steps:

[0124] Multi-source sensing data is simultaneously acquired by a multi-modal sensor array module deployed on the surface of a spherical shell.

[0125] Multi-source sensing data is input into a multimodal fusion module, and a continuous spatiotemporal sensing map is constructed based on weighted and phase interference enhancement mechanisms to characterize the dynamic changes of environmental state with time and space.

[0126] Using the super data large model modeling module, based on the constructed spatiotemporal perception map, graph neural networks are used to perform high-dimensional semantic modeling of the temporal features and structural relationships of nodes, and an anomaly detection mechanism is integrated to evaluate the reconstruction error in order to identify abnormal data.

[0127] In the edge-cloud collaborative computing module, computing tasks are dynamically allocated between edge computing nodes and cloud servers based on the event level judgment function, thereby optimizing the overall system processing latency.

[0128] Finally, a unified communication and power interface for the system is achieved through modular interfaces and low-cost integration modules, and the total integration cost is controlled based on a cost constraint model to meet the needs of rapid deployment across multiple platforms.

[0129] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal environment perception system based on a super data large model, characterized in that, include: A multimodal sensor array module, comprising multiple heterogeneous sensing units arranged on the surface of a spherical shell, for synchronously acquiring multi-source sensing data in an unstructured environment; A multimodal fusion module is used to perform weighted fusion processing on the multi-source sensing data collected by the multimodal sensor array module to construct a continuous spatiotemporal sensing map to represent the changes in environmental state with time and spatial dimensions. The super data large model modeling module is based on the spatiotemporal perception map constructed by the multimodal fusion module. It adopts a graph structure modeling method to combine temporal perception features and node structure relationships to perform high-dimensional feature extraction and semantic modeling. The edge-cloud collaborative computing module, based on a preset data importance judgment function, performs computing scheduling between edge computing nodes and cloud servers to achieve hierarchical collaboration and dynamic allocation of data processing; Modular interfaces and low-cost integrated modules adopt a unified power and communication protocol, are compatible with multiple platform forms, and are combined with a cost control constraint model to achieve low-cost modular deployment of the system.

2. The multimodal environment perception system based on a super data large model according to claim 1, characterized in that, The multimodal sensor array module is constructed as a compound eye spatial structure, with each heterogeneous sensing unit arranged on the outer surface of the spherical shell with a fixed field of view.

3. The multimodal environment perception system based on a super data large model according to claim 1, characterized in that, The multimodal fusion module is configured to perform fusion processing based on the raw data collected by multiple heterogeneous sensing units at spatial locations and time points, combined with a preset weighting factor. The weighting factors of each sensing unit are dynamically adjusted based on the historical sampling accuracy and reliability assessment of this type of sensor; During the fusion process, the phase difference signal extracted between the radar channel and the ultrasonic channel is further combined to enhance the fusion effect of spatial perception data through phase interference relationship, and a fusion perception map corresponding to time and space is constructed to express the environmental state.

4. The multimodal environment perception system based on a super data large model according to claim 1, characterized in that, The super data large model modeling module adopts a spatiotemporal graph neural network structure to construct a semantic graph from multimodal perception data; This network generates high-dimensional feature representations by weighting and mapping the feature vectors between each node and its neighboring nodes in each layer. During the feature mapping process, the input data is linearly transformed and a bias term is added by combining the adjacency relationship between nodes and the normalization factor. Then, a nonlinear mapping is performed through the activation function to realize the transfer and extraction of feature information between layers.

5. The multimodal environment perception system based on a super data large model according to claim 1, characterized in that, The anomaly detection module processes the input multimodal data based on an autoencoder architecture and outputs reconstructed data samples through a reconstruction network. The system compares the error value between the original input data and the reconstructed data, and makes a judgment based on the comparison between the error value and the preset error threshold; When the error value exceeds the preset threshold, the system determines that there is an abnormal situation and triggers the preset module redundancy mechanism to replace the corresponding sensor node in order to maintain the integrity of the system's perception.

6. The multimodal environment perception system based on a super data large model according to claim 1, characterized in that, The edge-cloud collaborative computing module is configured to perform dynamic path scheduling based on task event levels; When an event to be processed meets the preset critical event level conditions, the system selects to execute the task through the cloud server; if the critical event level conditions are not met, the task is processed by the edge computing node. The system calculates the edge processing latency and cloud processing latency based on the judgment condition, and determines the final overall computing path and task scheduling strategy accordingly.

7. The multimodal environment perception system based on a super data large model according to claim 1, characterized in that, The modular interface and low-cost integration module are constructed based on cost control constraints; The total system cost consists of the unit costs of the sensing module, power supply module, control module, and integration module, and the functional deployment is limited to a set cost threshold. Each module adopts a unified power and communication protocol and supports hot-swapping and automatic identification of each functional module, ensuring the system's interconnectivity under different platform configurations.

8. The fusion sensing method according to claim 3 and the multimodal environment sensing system based on a super data large model, characterized in that, The weighting factors of various sensors in the fusion sensing map model are updated based on the sensor reliability assessment scores at the current time point and historical weight values; This update mechanism combines a smoothing factor to perform a weighted average of the current reliability and historical weights, and then normalizes the result under the sum of the weighting factors of all sensors. This allows for dynamic adjustment of the participation weights of each type of sensor, achieving temporal adaptation and dynamic response in weight allocation.

9. The multimodal environment perception system based on a super data large model according to claim 1, characterized in that, The system also includes an energy supply and material structure module, which includes a flexible perovskite photovoltaic structure, a piezoelectric nanogenerator based on ZnO nanowires, and a UV-resistant and biodegradable composite encapsulation material, used to provide power and adapt to various deployment environments.

10. A multimodal environment perception method based on a super-large data model, applied to the multimodal environment perception system based on a super-large data model as described in any one of claims 1-9, characterized in that, Includes the following steps: Multi-source sensing data is simultaneously acquired by a multi-modal sensor array module deployed on the surface of a spherical shell. Multi-source sensing data is input into a multimodal fusion module, and a continuous spatiotemporal sensing map is constructed based on weighted and phase interference enhancement mechanisms to characterize the dynamic changes of environmental state with time and space. Using the super data large model modeling module, based on the constructed spatiotemporal perception map, graph neural networks are used to perform high-dimensional semantic modeling of the temporal features and structural relationships of nodes, and an anomaly detection mechanism is integrated to evaluate the reconstruction error in order to identify abnormal data. In the edge-cloud collaborative computing module, computing tasks are dynamically allocated between edge computing nodes and cloud servers based on the event level judgment function, thereby optimizing the overall system processing latency. Finally, a unified communication and power interface for the system is achieved through modular interfaces and low-cost integration modules, and the total integration cost is controlled based on a cost constraint model to meet the needs of rapid deployment across multiple platforms.

Images