Real-time operating system level management method with intelligent multi-modal perception data fusion

Abstract

The application discloses a real-time operating system level management method with intelligent multi-modal perception data fusion, and relates to the technical field of embodied intelligent robot operating systems. The method comprises the following steps: a multi-modal data acquisition module accesses various sensors through a standardized interface, acquires original data and evaluates the quality; a data preprocessing module adopts a hybrid synchronization mechanism of hardware time stamping and software compensation and a multi-scale time window algorithm to realize high-precision time synchronization and format standardization; and a feedback optimization module continuously optimizes the fusion strategy through online reinforcement learning, experience replay and multi-objective Pareto optimization. The application solves the problems of insufficient multi-modal data synchronization accuracy, heterogeneous fusion difficulty, real-time and resource conflicts and the like, reduces the synchronization error to within 0.5 ms, improves the target detection accuracy by more than 15%, and makes the system availability reach 99.3%, thereby being suitable for embodied intelligent scenarios such as service robots, industrial robots, medical robots and the like.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligent robot operating system technology, and specifically relates to a real-time operating system-level management method with intelligent multimodal perception data fusion. Background Technology

[0002] Embodied intelligence, as one of the core development directions of artificial intelligence, faces the key challenge of enabling robots to perceive dynamic environments through multi-sensory collaboration. The human perceptual system is a highly integrated multimodal system, with visual, auditory, and tactile channels working together to achieve a comprehensive understanding of the environment. Endowing robots with similar capabilities is crucial for the practical application of embodied intelligence. While multimodal perception fusion technology has made some progress, the following bottlenecks still exist: First, the spatiotemporal synchronization accuracy of multimodal data is insufficient. The sampling frequency, data format, and latency characteristics of different types of sensing devices vary significantly. For example, visual sensors are typically 30-60Hz, while force sensors can reach over 1000Hz. Traditional hardware triggering or timestamp alignment methods can have errors of tens of milliseconds in high dynamic environments, which cannot meet the millisecond-level synchronization requirements of scenarios such as precision assembly and surgical assistance.

[0003] Second, data heterogeneity and semantic gaps are difficult to bridge. Visual data consists of dense two-dimensional / three-dimensional image sequences, auditory data consists of one-dimensional temporal signals, and tactile data consists of sparsely distributed force values. Their physical characteristics and semantic levels differ greatly, and traditional feature extraction methods are prone to losing details of the original data. Furthermore, cross-modal semantic mapping lacks a unified standard.

[0004] Third, there is a significant conflict between real-time performance and computational resources. High-precision fusion algorithms (such as deep learning models) have high computational complexity, making it difficult to meet the end-to-end latency requirements of hundreds of milliseconds in embedded robot platforms. Simplifying the algorithm, on the other hand, can lead to a decrease in fusion accuracy. For example, decision-level fusion that ignores intermodal interactions may reduce the target detection accuracy by more than 20%.

[0005] Fourth, poor adaptability to environmental dynamics. In real-world environments, factors such as changes in lighting, noise interference, and target movement can cause fluctuations in the quality of perceived data. Traditional offline optimization fusion strategies cannot be adjusted in real time. For example, when a visual sensor fails in a strong light environment, the system cannot automatically switch to a fusion mode dominated by touch / force.

[0006] Fifth, there is a lack of cross-modal conflict resolution mechanisms. When different modal data provide conflicting information (such as visual detection of an obstacle but radar failure to detect it), traditional methods lack reliable conflict detection and resolution strategies, which can easily lead to decision-making errors.

[0007] Feature-level fusion-based methods require converting heterogeneous data into a unified feature representation. While they can handle data heterogeneity, the feature extraction process is computationally expensive (e.g., ResNet-50 feature extraction takes about 20ms / frame) and may lose detailed information in the original data (e.g., pressure distribution texture of tactile sensors).

[0008] The decision-level fusion method, which processes each modality independently and then fuses the decision results, improves parallelism but ignores the complementary relationship between modalities (such as the inability to link the spatial information provided by vision with the contact force information provided by force perception), which may lead to decision conflicts (such as vision judging that it can be grasped but force perception detecting that the resistance is too great).

[0009] Fusion methods based on attention mechanisms: Although they have been applied in recent years, existing attention mechanisms are mostly designed for specific modalities (such as visual Transformers), lack cross-modal interaction modeling capabilities, and their computational complexity increases exponentially with the number of modalities, making them difficult to deploy in resource-constrained embedded systems.

[0010] Lack of management solutions at the operating system level: Existing multimodal fusion is mostly implemented as an application layer task and is not deeply integrated with the operating system kernel scheduling and resource management, resulting in low overall system efficiency (e.g., when CPU resources are preempted by non-critical tasks, the latency of fusion tasks increases by more than 50%). Summary of the Invention

[0011] In view of this, the present invention provides a real-time operating system-level management method with intelligent multimodal sensing data fusion to solve or alleviate one of the technical problems existing in the prior art, and at least provides a beneficial option.

[0012] The technical solution of this invention is implemented as follows: a real-time operating system-level management method with intelligent multimodal sensing data fusion, comprising the following steps: Step S1: Obtain raw sensory data from visual sensors, auditory sensors, tactile sensors, force sensors and inertial measurement units through the multimodal data acquisition module. The acquisition module adapts to heterogeneous hardware through a standardized device abstraction interface, configures a hardware timestamp unit (accuracy ±5ns) for each sensor, and monitors data integrity, noise level and outlier ratio in real time. When a decline in data quality is detected, the acquisition parameters are automatically adjusted or the redundant sensor is switched. Step S2: The raw sensing data is time-synchronized, format-standardized, and noise-filtered through the data preprocessing module. The time synchronization adopts a hybrid synchronization mechanism based on hardware timestamps (nanosecond-level reference) and software compensation (dynamic correction based on historical synchronization error statistics). Combined with a multi-scale time window algorithm (a coarse-scale window of 100ms ensures basic synchronization, and a fine-scale window of 10ms improves synchronization resolution through interpolation and prediction algorithms), a standardized sensing data stream with unified format (Tensor format, scale normalized to [-1,1]) and time alignment is generated. Step S3: The standardized perceptual data stream is fused hierarchically through a multi-level fusion engine, including: raw data layer fusion, which performs spatial alignment (error ≤ 2mm) and temporal correlation extraction on raw data from different sensors; feature layer fusion, which uses a Transformer-based cross-modal attention mechanism to model modal interaction relationships and generate joint features with attention weights; and semantic layer fusion, which combines task semantics to generate high-level perceptual representations that meet the requirements. The fusion engine also implements a dynamic modality selection mechanism (dynamically adjusting modality weights based on data quality) and a cross-modal conflict detection and resolution mechanism (triggering conflict resolution based on multi-modal consistency metrics). Step S4: The real-time scheduling management module dynamically allocates computing resources, memory bandwidth, and communication bandwidth based on task priority and resource status. The scheduling module is deeply integrated with the operating system kernel and adopts a multi-priority queue scheduling algorithm (setting 5 priority levels, with P0 being the highest level). Combined with a resource pre-allocation mechanism based on the LSTM load prediction model (prediction error ≤8%), a dynamic load balancing mechanism (task migration when single-core load >80%), and a performance isolation mechanism (allocating dedicated CPU cores and memory bandwidth to P0 tasks), the real-time performance (end-to-end latency ≤100ms) and stability of the fusion process are ensured. Step S5: The feedback optimization module continuously optimizes the fusion strategy parameters based on the deviation between the fusion result and the actual execution effect. The optimization module adopts online reinforcement learning (modeling the fusion strategy adjustment as a Markov decision process, with reward function = 0.4 × fusion accuracy + 0.3 × response latency + 0.3 × resource efficiency, and DQN network updating parameters every 100ms), experience replay mechanism (maintaining 100,000 experience buffers, randomly sampling and batch updating), and multi-objective Pareto front tracking (simultaneously optimizing four objectives: fusion accuracy, latency, efficiency, and robustness) to achieve adaptive improvement of the fusion system.

[0013] Preferably, the multimodal data acquisition module supports unified access to more than 20 types of sensing devices, such as visual cameras, depth cameras, microphone arrays, tactile sensors, torque sensors, and inertial measurement units. It shields hardware differences through standardized device abstraction interfaces and has a built-in data quality assessment algorithm to monitor the integrity (packet loss rate ≤0.1%), noise level (signal-to-noise ratio ≥20dB), and outlier ratio (≤5%) of each sensor data in real time. When a decline in data quality is detected, it automatically adjusts the acquisition parameters (such as visual exposure time and microphone gain) or switches to redundant sensors (switching delay ≤10ms) to ensure the continuity and reliability of data acquisition. Preferably, in the hybrid synchronization mechanism of the data preprocessing module, the hardware timestamp is generated by a high-precision clock source (frequency ≥ 100MHz) built into the sensor, and the software compensation algorithm is based on a sliding window (window length 1000 frames) to count historical synchronization errors and correct synchronization deviations in real time through Kalman filtering, so that the multimodal data synchronization error is ≤ 0.5ms; the multi-scale time window synchronization algorithm dynamically adjusts the window size according to the time requirements of different tasks. For example, a coarse window of 50ms and a fine window of 5ms are used in autonomous driving scenarios, and a coarse window of 20ms and a fine window of 2ms are used in medical surgery scenarios, taking into account both synchronization accuracy and computational efficiency.

[0014] Preferably, in the three-layer fusion architecture of the multi-level fusion engine, the raw data layer fusion extracts cross-modal spatial correlation information (such as the position of the tactile pressure center in the visual image) and temporal correlation information (such as the time difference between visual detection of a collision and the peak of force perception) by aligning the coordinate systems of different sensors (e.g., the visual coordinate system and the force perception coordinate system are aligned by a calibration plate, with an error ≤0.02mm); the feature layer fusion adopts a 12-layer cross-modal Transformer network, including an 8-head attention mechanism, to model the interaction weights between visual, auditory, tactile, and force perception modalities (e.g., when grasping a fragile object, the tactile feature weight increases to 0.6, while the visual feature weight decreases to 0.3); the semantic layer fusion combines a task semantic library (containing 1000+ task descriptions) to generate a high-level perceptual representation that includes environmental status, risk level, and operational suggestions (e.g., "There is a moving obstacle 1m ahead, the risk level is high, it is recommended to slow down to 0.5m / s"). Preferably, in the multi-priority queue scheduling algorithm of the real-time scheduling management module, the task priority is determined by a combination of the task urgency (e.g., emergency braking tasks have higher priority than path planning tasks), data timeliness (e.g., visual data validity period ≤ 100ms), and task benefits (e.g., increased success rate of task completion). P0-level tasks can preempt all resources of P1-P4 level tasks. The performance isolation mechanism reserves dedicated computing resources (e.g., 2 CPU cores, 4GB memory, 2GB / s memory bandwidth) for P0 tasks and restricts other tasks from accessing these resources through cgroups technology, ensuring that P0 tasks can still be completed on time even in the worst case. Preferably, the dynamic modality selection mechanism of the multi-level fusion engine evaluates the quality indicators of each modality data in real time (such as visual signal-to-noise ratio, force data integrity, and tactile data outlier ratio). When the quality of a certain modality data is lower than the threshold (such as visual signal-to-noise ratio < 20dB), the fusion weight of that modality is automatically reduced (reduction ≥ 30%) or switched to an alternative modality (such as switching to LiDAR + tactile fusion when vision fails). The cross-modal conflict detection and resolution mechanism judges consistency by calculating the KL divergence of different modal output results (threshold set to 0.3). When a conflict is detected (such as vision detecting an obstacle but radar not detecting it), weighted fusion is performed based on modality confidence (such as visual confidence 0.8, radar confidence 0.6) (weight = confidence / total confidence), or high reliability modality is preferentially adopted based on historical consistency records (the accuracy rate of a certain modality in the past 100 interactions ≥ 95%). Preferably, the real-time scheduling and management module's prediction-based resource pre-allocation mechanism adopts an LSTM load prediction model. The input features include the number of tasks, CPU utilization, memory utilization, and data throughput in the past 10 seconds, and the output is a prediction of resource demand in the next 500ms (prediction error ≤8%). This pre-allocates CPU cores, GPU memory, and bus bandwidth in advance for upcoming high-load tasks (such as simultaneous triggering of multiple sensors), avoiding latency fluctuations caused by resource contention. The dynamic load balancing mechanism monitors the load status of the 8-core CPU in real time (sampling interval 10ms). When the load of a single core is >80% and the load of other cores is <50%, it automatically migrates non-P0 tasks on that core to idle cores, achieving dynamic balancing of computing resources.

[0015] Preferably, in the online reinforcement learning of the feedback optimization module, the state space includes current environmental parameters (light intensity, noise level), task parameters (task type, urgency), and fusion strategy parameters (weights of each modality, synchronization window size). The action space is the adjustment amount of the fusion strategy parameters (e.g., visual weight ±0.1, synchronization window ±5ms). The reward function is calculated based on the matching degree between the fusion result and the actual execution effect (e.g., in an assembly task, the smaller the deviation between the actual assembly accuracy and the target accuracy, the higher the reward). The experience replay buffer stores the state, action, reward, and next state of each experience. The Priority Experience Replay (PER) algorithm is used to assign a higher sampling probability to high-reward samples, thereby improving learning efficiency. Preferably, the multi-objective Pareto front tracking of the feedback optimization module adopts the NSGA-II multi-objective evolutionary algorithm, with a population size of 100 and an iteration count of 500. It simultaneously optimizes four objectives: fusion accuracy (target value ≥ 95%), response latency (target value ≤ 100ms), computational efficiency (CPU utilization ≤ 80%), and robustness (performance fluctuation ≤ 15% under environmental changes). It generates a solution set containing 50 Pareto optimal solutions. Users can select the optimal solution according to the application scenario requirements through the human-computer interaction interface (e.g., selecting the low-latency solution for autonomous driving scenarios and the high-precision solution for medical robot scenarios). Preferred options include: The multimodal data acquisition module is used to connect to visual sensors, auditory sensors, tactile sensors, force sensors and inertial measurement units through a standardized device abstraction interface, acquire raw sensing data and evaluate data quality (integrity, noise level, outlier ratio) in real time, and supports adaptive acquisition rate control and device redundancy switching; The data preprocessing module is used to synchronize the raw sensing data in time through a hybrid synchronization mechanism (hardware timestamp + software compensation) and a multi-scale time window algorithm, and to generate a standardized sensing data stream in a unified format through adaptive noise filtering and format standardization. A multi-level fusion engine is used to generate unified perceptual representations through raw data layer fusion, feature layer fusion (based on the Transformer cross-modal attention mechanism) and semantic layer fusion, supporting dynamic modality selection and cross-modal conflict resolution; The real-time scheduling and management module is used to dynamically allocate computing resources, memory bandwidth, and communication bandwidth through multi-priority queue scheduling, predictive resource pre-allocation, dynamic load balancing, and performance isolation mechanisms that are deeply integrated with the operating system, ensuring that the converged tasks are completed in real time. The feedback optimization module is used to continuously optimize the fusion strategy parameters based on the deviation between the fusion result and the actual performance through online reinforcement learning, experience playback and multi-objective Pareto optimization, so as to achieve adaptive improvement of the system.

[0016] The embodiments of the present invention have the following advantages due to the adoption of the above technical solutions: (1) Through a hybrid synchronization mechanism of hardware timestamp + software compensation, the multimodal data synchronization error is ≤0.5ms, which is 67% lower than the traditional software synchronization method (the traditional method has an error of about 1.5ms), meeting the microsecond-level synchronization requirements of precision assembly (such as chip mounting).

[0017] (2) The three-layer fusion architecture combined with the cross-modal Transformer attention mechanism achieves a target detection accuracy of 96.8% (81.5% for a single visual modality) and improves the completeness of environmental understanding by 23.7%. For example, service robots can simultaneously recognize three modal information: “user waving” (visual), “saying hello” (auditory), and “approaching” (tactile), and accurately judge the interaction intention.

[0018] (3) Operating system-level scheduling optimization reduces the end-to-end fusion latency to ≤85ms (approximately 150ms in traditional methods), a reduction of 43.3%, significantly improving the robot's response speed in dynamic environments. For example, in the case of sudden obstacles, the perception-decision latency of autonomous vehicles is reduced to less than 70ms.

[0019] (4) Online reinforcement learning optimization makes the performance fluctuation of the system ≤12% in different environments (the fluctuation of traditional static methods is >30%). For example, the assembly accuracy of industrial robots remains above 98.5% in both strong light (light intensity >100000 lux) and weak light (<100 lux) environments.

[0020] (5) Equipment redundancy, conflict resolution and performance isolation mechanisms enable the system to achieve an availability of 99.3% (compared to about 95% with traditional methods), and reduce the task failure rate by 62% under sensor failure or strong interference.

[0021] (6) Dynamic load balancing and resource pre-allocation mechanism increases CPU utilization to 82% (about 65% with traditional methods), and reduces the average completion time of multi-task concurrency by 22%, making it suitable for embedded platform deployment.

[0022] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the invention will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 This is a detailed flowchart of the data preprocessing process of the present invention; Figure 3 This is a detailed flowchart of the multi-level fusion engine of the present invention. Detailed Implementation

[0025] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0026] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] Implementation Method 1: Service Robot Home Environment Perception Scenario Scene Description For home companion robots, it is necessary to perceive user commands, environmental status and interaction needs in real time to complete tasks such as delivering objects, avoiding obstacles and emotional interaction.

[0028] Implementation details Sensor configuration: RGB-D camera (30Hz), microphone array (16kHz), tactile sensor (100Hz, distributed on the surface of the robotic arm), infrared human body sensor (10Hz).

[0029] Data Acquisition: The multimodal data acquisition module connects to all sensors through a standardized interface, and each data packet is timestamped by hardware (error ±5ns) to monitor data quality in real time (e.g., automatic gain when microphone signal-to-noise ratio <15dB).

[0030] Preprocessing: The data preprocessing module uses a 100ms coarse window + 10ms fine window synchronization. Visual data is denoised using bilateral filtering, auditory data is denoised using Wiener filtering, and tactile data is smoothed using Kalman filtering. The format is unified as [1, 3,224, 224] (visual) and [1, 1, 16000] (auditory) Tensor formats.

[0031] Fusion process: Raw data layer: Project the tactile pressure distribution onto the 3D coordinate system of the visual image to establish the "user touch position - object" association; Feature layer: A cross-modal Transformer network fuses visual features (extracted by ResNet-18), auditory features (Mel spectrogram), and tactile features (pressure gradient) to generate a 128-dimensional joint feature vector; Semantic layer: Combining task semantics ("the user needs a water cup"), outputting a perceptual representation that "the water cup is located on the left side of the coffee table, 0.8m away, and can be safely grabbed".

[0032] Scheduling Management: The real-time scheduling management module sets "User Command Response" to P1 priority and "Environmental Obstacle Avoidance" to P0 priority. P0 tasks occupy 2 dedicated CPU cores and reserve 1GB / s of memory bandwidth.

[0033] Feedback optimization: The feedback optimization module adjusts the fusion weights based on user feedback ("incorrectly delivered items") (such as increasing the weight of color features of objects in vision), and updates the DQN network parameters once a week.

[0034] Implementation effect Actual operational data shows that the accuracy rate of user command recognition reached 97.2% (compared to 74.5% for traditional methods), the task success rate increased by 31.8%, and the user experience satisfaction score improved from 72 to 90.

[0035] Implementation Method 2: Precision Assembly Scenario with Industrial Robots Scene Description When applied to automotive parts assembly robots, it is necessary to accurately sense information such as workpiece position and contact force to complete the precision assembly of engine pistons and cylinder blocks (tolerance ±0.05mm).

[0036] Implementation details Sensor configuration: Industrial camera (200Hz, resolution 1280×1024), six-dimensional force sensor (1000Hz, accuracy 0.1N), laser displacement sensor (500Hz, accuracy 1μm).

[0037] Data acquisition: The multimodal data acquisition module uses hardware interrupts to trigger data acquisition from the force sensor to ensure real-time data transmission; the vision sensor transmits data through the GigE Vision interface, with data integrity verified (CRC32).

[0038] Preprocessing: The data preprocessing module uses hardware timestamp synchronization (time alignment error between visual and force data ≤ 0.3ms), performs low-pass filtering (cutoff frequency 50Hz) on the force data to remove high-frequency noise, and performs sub-pixel edge detection on the visual data to improve pose estimation accuracy.

[0039] Fusion process: Raw data layer: Align the force sensor coordinate system with the visual coordinate system using a calibration plate (error ≤ 0.02mm). Feature layer: Extract visual workpiece pose features (x,y,z,rx,ry,rz) and force-sensing contact force features (Fx,Fy,Fz,Mx,My,Mz), and fuse them through a cross-attention mechanism; Semantic layer: Generates an assembly state representation of "piston has entered 80% of the cylinder and axial force of 50N±5N needs to be applied".

[0040] Scheduling Management: The real-time scheduling management module sets the assembly task to P0 priority, and the predictive resource pre-allocation reserves GPU resources 500ms in advance based on the historical assembly cycle (average 2s) (for visual pose calculation).

[0041] Feedback optimization: The feedback optimization module optimizes the fusion parameters based on the assembly quality inspection results (such as leakage test data). For example, when the assembly interference is greater than 0.03mm, the fusion weight of the force perception feature is increased by 15%.

[0042] Implementation effect Actual operation data shows that the first-pass yield rate reached 99.2% (compared to 96.1% with the traditional method), the assembly efficiency increased by 18.3%, and the assembly time per piece was shortened from 2.5s to 2.04s.

[0043] Implementation Method 3: Medical Robot Surgical Assistance Scenarios Scene Description When applied to laparoscopic surgical robots, visual, force, and positional information must be integrated to assist surgeons in performing operations such as tissue cutting and suturing, ensuring surgical safety and precision.

[0044] Implementation details Sensor configuration: Endoscopic camera (60Hz, 1080P), six-dimensional force feedback sensor (500Hz, accuracy 0.01N), optical position tracking system (1000Hz, accuracy 0.1mm).

[0045] Data acquisition: The multimodal data acquisition module uses DMA to reduce latency for force data, performs distortion correction on endoscopic images, and transmits position data via optical fiber to ensure anti-interference.

[0046] Preprocessing: The data preprocessing module uses multi-scale time windows (coarse window 50ms, fine window 5ms) to synchronize, perform gravity compensation on force data (remove the influence of the instrument's own weight), and perform real-time enhancement on visual data (contrast stretching, noise suppression).

[0047] Fusion process: Raw data layer: Mapping force sensor data onto the organ surface of visual images to establish an "instrument-tissue" contact model; Feature layer: Integrates visual tissue texture features (CNN extraction) and force-feel hardness features (force-displacement curve slope) to identify tissue types (such as blood vessels, nerves, and fat). Semantic layer: Generates safe operation instructions such as "Currently contacted tissue is a blood vessel wall, 0.5mm thick, with a maximum tensile force of 0.3N".

[0048] Scheduling Management: The real-time scheduling management module sets the surgical perception task to the highest P0 priority, exclusively using 4 CPU cores and 50% of GPU resources, and performance isolation ensures that it is not interfered with by other system tasks.

[0049] Feedback optimization: The feedback optimization module adjusts the fusion strategy in real time based on the surgeon's operational feedback (such as "too much force"). For example, it increases the weight of force features by 20% and decreases the weight of visual features by 10%.

[0050] Implementation effect Actual clinical test data shows that the response time of surgical assistance is ≤45ms, the sensing accuracy reaches 0.2mm, which meets the requirements of delicate surgery, and the incidence of safety events is reduced by 65.7% (0.3 times / unit for traditional methods, 0.103 times / unit for this invention).

[0051] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time operating system-level management method with intelligent multimodal sensing data fusion, characterized in that: Includes the following steps: Step S1: Obtain raw sensory data from visual sensors, auditory sensors, tactile sensors, force sensors and inertial measurement units through the multimodal data acquisition module. The acquisition module adapts to heterogeneous hardware through a standardized device abstraction interface, configures a hardware timestamp unit (accuracy ±5ns) for each sensor, and monitors data integrity, noise level and outlier ratio in real time. When a decline in data quality is detected, the acquisition parameters are automatically adjusted or the redundant sensor is switched. Step S2: The raw sensing data is time-synchronized, format-standardized, and noise-filtered through the data preprocessing module. The time synchronization adopts a hybrid synchronization mechanism based on hardware timestamps (nanosecond-level reference) and software compensation (dynamic correction based on historical synchronization error statistics). Combined with a multi-scale time window algorithm (a coarse-scale window of 100ms ensures basic synchronization, and a fine-scale window of 10ms improves synchronization resolution through interpolation and prediction algorithms), a standardized sensing data stream with unified format (Tensor format, scale normalized to [-1,1]) and time alignment is generated. Step S3: The standardized perceptual data stream is fused hierarchically through a multi-level fusion engine, including: raw data layer fusion, which performs spatial alignment (error ≤ 2mm) and temporal correlation extraction on raw data from different sensors; feature layer fusion, which uses a Transformer-based cross-modal attention mechanism to model modal interaction relationships and generate joint features with attention weights; and semantic layer fusion, which combines task semantics to generate high-level perceptual representations that meet the requirements. The fusion engine also implements a dynamic modality selection mechanism (dynamically adjusting modality weights based on data quality) and a cross-modal conflict detection and resolution mechanism (triggering conflict resolution based on multi-modal consistency metrics). Step S4: The real-time scheduling management module dynamically allocates computing resources, memory bandwidth, and communication bandwidth based on task priority and resource status. The scheduling module is deeply integrated with the operating system kernel and adopts a multi-priority queue scheduling algorithm (setting 5 priority levels, with P0 being the highest level). Combined with a resource pre-allocation mechanism based on the LSTM load prediction model (prediction error ≤8%), a dynamic load balancing mechanism (task migration when single-core load >80%), and a performance isolation mechanism (allocating dedicated CPU cores and memory bandwidth to P0 tasks), the real-time performance (end-to-end latency ≤100ms) and stability of the fusion process are ensured. Step S5: The feedback optimization module continuously optimizes the fusion strategy parameters based on the deviation between the fusion result and the actual execution effect. The optimization module adopts online reinforcement learning (modeling the fusion strategy adjustment as a Markov decision process, with reward function = 0.4 × fusion accuracy + 0.3 × response latency + 0.3 × resource efficiency, and DQN network updating parameters every 100ms), experience replay mechanism (maintaining 100,000 experience buffers, randomly sampling and batch updating), and multi-objective Pareto front tracking (simultaneously optimizing four objectives: fusion accuracy, latency, efficiency, and robustness) to achieve adaptive improvement of the fusion system.

2. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: The multimodal data acquisition module supports unified access to more than 20 sensing devices, including visual cameras, depth cameras, microphone arrays, tactile sensors, torque sensors, and inertial measurement units. It shields hardware differences through standardized device abstraction interfaces and has a built-in data quality assessment algorithm to monitor the integrity (packet loss rate ≤0.1%), noise level (signal-to-noise ratio ≥20dB), and outlier ratio (≤5%) of each sensor data in real time. When a decline in data quality is detected, it automatically adjusts the acquisition parameters (such as visual exposure time and microphone gain) or switches to redundant sensors (switching delay ≤10ms) to ensure the continuity and reliability of data acquisition.

3. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: In the hybrid synchronization mechanism of the data preprocessing module, the hardware timestamp is generated by a high-precision clock source (frequency ≥ 100MHz) built into the sensor. The software compensation algorithm is based on a sliding window (window length 1000 frames) to count historical synchronization errors and uses Kalman filtering to correct synchronization deviations in real time, so that the multimodal data synchronization error is ≤ 0.5ms. The multi-scale time window synchronization algorithm dynamically adjusts the window size according to the time requirements of different tasks. For example, a coarse window of 50ms and a fine window of 5ms are used in autonomous driving scenarios, and a coarse window of 20ms and a fine window of 2ms are used in medical surgery scenarios, balancing synchronization accuracy and computational efficiency.

4. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: In the three-layer fusion architecture of the multi-level fusion engine, the raw data layer fusion extracts cross-modal spatial correlation information (such as the position of the tactile pressure center in the visual image) and temporal correlation information (such as the time difference between visual detection of a collision and the peak of force) by aligning the coordinate systems of different sensors (e.g., the visual coordinate system and the force coordinate system are aligned by a calibration plate with an error ≤ 0.02mm). The feature layer fusion adopts a 12-layer cross-modal Transformer network, including an 8-head attention mechanism, to model the interaction weights between visual, auditory, tactile, and force modalities (e.g., when grasping a fragile object, the tactile feature weight increases to 0.6, while the visual feature weight decreases to 0.3). The semantic layer fusion combines a task semantic library (containing 1000+ task descriptions) to generate a high-level perceptual representation that includes environmental status, risk level, and operational suggestions (e.g., "There is a moving obstacle 1m ahead, the risk level is high, it is recommended to slow down to 0.5m / s").

5. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: In the multi-priority queue scheduling algorithm of the real-time scheduling management module, the task priority is determined by a combination of the task urgency (e.g., emergency braking tasks have higher priority than path planning tasks), data timeliness (e.g., visual data validity period ≤ 100ms), and task benefits (e.g., increased success rate of task completion). P0-level tasks can preempt all resources of P1-P4 level tasks. The performance isolation mechanism reserves dedicated computing resources (e.g., 2 CPU cores, 4GB memory, 2GB / s memory bandwidth) for P0 tasks and restricts other tasks from accessing these resources through cgroups technology, ensuring that P0 tasks can still be completed on time even in the worst case.

6. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: The dynamic modality selection mechanism of the multi-level fusion engine evaluates the quality indicators of each modality data in real time (such as visual signal-to-noise ratio, force data integrity, and tactile data outlier ratio). When the quality of a certain modality data is lower than the threshold (such as visual signal-to-noise ratio <20dB), the fusion weight of that modality is automatically reduced (reduction ≥30%) or switched to an alternative modality (such as switching to LiDAR + tactile fusion when vision fails). The cross-modal conflict detection and resolution mechanism judges consistency by calculating the KL divergence of different modal output results (threshold set to 0.3). When a conflict is detected (such as visual detection of an obstacle but radar not detecting it), weighted fusion is performed based on modality confidence (such as visual confidence 0.8, radar confidence 0.6) (weight = confidence / total confidence), or high reliability modality is prioritized based on historical consistency records (the accuracy rate of a certain modality in the past 100 interactions ≥95%).

7. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: The real-time scheduling and management module's prediction-based resource pre-allocation mechanism employs an LSTM load prediction model. Input features include the number of tasks, CPU utilization, memory usage, and data throughput over the past 10 seconds. The output is a prediction of resource demand for the next 500ms (prediction error ≤ 8%), pre-allocating CPU cores, GPU memory, and bus bandwidth for upcoming high-load tasks (such as simultaneous triggering of multiple sensors) to avoid latency fluctuations caused by resource contention. The dynamic load balancing mechanism monitors the load status of the 8-core CPU in real time (sampling interval 10ms). When the load of a single core is >80% and the load of other cores is <50%, it automatically migrates non-P0 tasks on that core to idle cores, achieving dynamic balancing of computing resources.

8. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: In the online reinforcement learning of the feedback optimization module, the state space includes current environmental parameters (light intensity, noise level), task parameters (task type, urgency), and fusion strategy parameters (weights of each modality, synchronization window size). The action space is the adjustment amount of the fusion strategy parameters (e.g., visual weight ±0.1, synchronization window ±5ms). The reward function is calculated based on the matching degree between the fusion result and the actual execution effect (e.g., in an assembly task, the smaller the deviation between the actual assembly accuracy and the target accuracy, the higher the reward). The experience replay buffer stores the state, action, reward, and next state of each experience. The Priority Experience Replay (PER) algorithm is used to assign higher sampling probabilities to high-reward samples, thereby improving learning efficiency.

9. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to claim 1, characterized in that: The multi-objective Pareto front tracking of the feedback optimization module adopts the NSGA-II multi-objective evolutionary algorithm, with a population size of 100 and an iteration count of 500. It simultaneously optimizes four objectives: fusion accuracy (target value ≥ 95%), response latency (target value ≤ 100ms), computational efficiency (CPU utilization ≤ 80%), and robustness (performance fluctuation ≤ 15% under environmental changes). It generates a solution set containing 50 Pareto optimal solutions. Users can select the optimal solution according to the application scenario requirements through the human-computer interaction interface (e.g., selecting the low-latency solution for autonomous driving scenarios and the high-precision solution for medical robot scenarios).

10. The real-time operating system-level management method with intelligent multimodal sensing data fusion according to any one of claims 1 to 9, characterized in that, include: The multimodal data acquisition module is used to connect to visual sensors, auditory sensors, tactile sensors, force sensors and inertial measurement units through a standardized device abstraction interface, acquire raw sensing data and evaluate data quality (integrity, noise level, outlier ratio) in real time, and supports adaptive acquisition rate control and device redundancy switching; The data preprocessing module is used to synchronize the raw sensing data in time through a hybrid synchronization mechanism (hardware timestamp + software compensation) and a multi-scale time window algorithm, and to generate a standardized sensing data stream in a unified format through adaptive noise filtering and format standardization. A multi-level fusion engine is used to generate unified perceptual representations through raw data layer fusion, feature layer fusion (based on the Transformer cross-modal attention mechanism) and semantic layer fusion, supporting dynamic modality selection and cross-modal conflict resolution; The real-time scheduling and management module is used to dynamically allocate computing resources, memory bandwidth, and communication bandwidth through multi-priority queue scheduling, predictive resource pre-allocation, dynamic load balancing, and performance isolation mechanisms that are deeply integrated with the operating system, ensuring that the converged tasks are completed in real time. The feedback optimization module is used to continuously optimize the fusion strategy parameters based on the deviation between the fusion result and the actual performance through online reinforcement learning, experience playback and multi-objective Pareto optimization, so as to achieve adaptive improvement of the system.

Images