Multi-modal missing robustness fusion processing method suitable for edge device

Abstract

The invention belongs to the technical field of artificial intelligence, particularly relates to a multi-modal missing robustness fusion processing method suitable for edge equipment, and aims to solve the problems of high calculation overhead and unstable output caused by dynamic weight adjustment when multi-modal data of the edge equipment is partially missing. The method comprises the following steps: constructing an end-to-end architecture comprising an independent trunk feature extraction module, a cross-modal semantic mapping module, a fixed weight fusion module and a task decoding module, and presetting compensation feature vectors corresponding to various modal missing modes; during reasoning, according to a modal effectiveness detection result, real-time features of an effective modal and curing compensation features of a corresponding missing mode are fused according to a preset fixed weight, and online optimization or dynamic adjustment is not needed. Through adoption of the technical scheme, low-delay, high-certainty and resource-limited and friendly robust fusion reasoning can be realized under the condition of any mode missing.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and more specifically, relates to a robust fusion processing method for multimodal missing features applicable to edge devices. Background Technology

[0002] With the deep integration of artificial intelligence and the Internet of Things (IoT) technologies, edge intelligent computing is gradually becoming a key infrastructure supporting real-time perception, autonomous decision-making, and low-latency response. In typical application scenarios such as autonomous driving, industrial inspection, wearable health monitoring, and unmanned systems, multimodal perception architectures are widely adopted due to their ability to comprehensively utilize heterogeneous sensor information such as vision, sound, inertia, and environmental data, aiming to achieve a more comprehensive and robust understanding of the physical world. Against this backdrop, how to efficiently achieve the fusion and processing of multimodal information on resource-constrained edge devices, while ensuring the continuous and stable operation of the system in the event of partial sensor channel failure or data loss, has become one of the core technological bottlenecks driving edge intelligence from the laboratory to large-scale deployment.

[0003] Current mainstream multimodal fusion methods generally employ early-stage, mid-stage, or late-stage fusion strategies, with feature-level fusion mechanisms based on deep neural networks being the most widely used. These methods typically configure dedicated feature extraction subnetworks for each input modality, then integrate the extracted high-dimensional feature vectors through concatenation, weighted summation, or attention mechanisms, finally feeding them into a shared high-level semantic decoder to complete tasks such as classification, detection, or regression. Under ideal conditions, i.e., when all modal data is complete and synchronously available, this architecture can effectively exploit the complementarity and redundancy between modalities, significantly improving the overall performance of the model. Especially after introducing modality missing data simulation strategies during the training phase, some existing solutions have also demonstrated a certain degree of fault tolerance, allowing them to maintain basic functionality even in the face of occasional data interruptions.

[0004] However, with the continuous development of related technologies and the increasingly stringent performance requirements of application scenarios, some inherent characteristics of the aforementioned technical solutions at the principle level have gradually revealed their limitations in addressing new challenges. Specifically, existing methods often rely on dynamically adjusting fusion weights or introducing complex gating mechanisms to suppress the influence of invalid modalities when dealing with the problem of missing modalities. While such strategies improve the adaptability of the model to some extent, they inevitably introduce additional computational overhead and parameter uncertainty. For edge devices with highly limited computing resources, memory bandwidth, and energy consumption, this dynamic weighting mechanism is not only difficult to deploy efficiently, but may also lead to drastic fluctuations in the fusion results due to weight estimation bias, thereby weakening the stability of the system output. Furthermore, if missing modes not fully covered during training are encountered during the inference phase (such as the long-term failure of a specific modal combination), the model is highly susceptible to performance degradation or even functional paralysis due to the lack of effective feature compensation paths. The reason for this is that the existing architecture still treats each modality as an independent source of information and fails to build an intrinsic connection and substitutability channel across modalities at the feature representation level. When a certain modality is missing, the system can only passively "ignore" that branch instead of actively "filling in" its semantic contribution, which leads to a structural degradation of the overall representation capability.

[0005] Correspondingly, in edge computing scenarios, modal loss is not a rare anomaly, but a normal occurrence caused by factors such as sensor failure, communication interruption, power management, or environmental interference. Under these circumstances, if the fusion mechanism cannot quickly switch to a pre-defined, deterministic feature combination mode when a loss occurs, it will be difficult to meet the stringent reliability and deterministic response requirements of industrial applications. Therefore, there is an urgent need for a fusion architecture that can maximize performance with all modal inputs and seamlessly switch to a stable operating state when any modality is lost. This architecture should avoid relying on complex online weight optimization and instead embed robustness through structured design, enabling the features of the remaining modalities to work together in a fixed and predictable manner. This ensures the consistency and reliability of the output results without increasing the real-time computing burden on edge devices.

[0006] Therefore, how to construct a multimodal fusion processing mechanism suitable for edge devices, so that it can maintain the stability and accuracy of model output through preset backup feature channels and fixed weight strategies even when any modality is missing, has become a key challenge and a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0007] To achieve the aforementioned objectives, this invention provides a robust fusion processing method for multimodal data loss in edge devices. This method constructs a multi-branch feature extraction and fusion architecture with structured redundancy and deterministic switching mechanisms. Even with missing data for any modality, it maintains the stability and accuracy of the system output without dynamically adjusting fusion weights or performing online optimization.

[0008] A robust fusion processing method for multimodal missing features in edge devices, the method comprising: During the model training phase, an end-to-end neural network architecture is constructed, which includes a backbone feature extraction module, a cross-modal semantic mapping module, a fixed weight fusion module, and a task decoding module. Following the cross-modal semantic mapping module, a set of compensation feature vectors corresponding to various modal missing modes is preset; During the inference phase, the validity of data from each sensor channel is monitored in real time through the modal state detection unit. When a mode is detected to be missing, the system loads the compensation feature vector corresponding to the current missing mode and inputs it together with the real-time feature vector of the effective mode into the fixed weight fusion module. The fixed-weight fusion module linearly fuses the input real-time feature vector and the compensation feature vector according to a preset fixed weighting coefficient to generate a deterministic fused feature vector. The task decoding module decodes the fused feature vector to obtain the final task output.

[0009] Furthermore, the method for generating the compensation feature vector in this invention includes the following steps: During model training, for each predefined modality missing pattern Its compensation feature vector Online updates are performed using a moving average algorithm.

[0010] in: This indicates the missing patterns at the t-th training iteration. The compensated feature vector; It is a smoothing factor; This indicates that in the current training batch, when in missing mode... At that time, the mean of the feature vectors of all effective modalities in the unified semantic embedding space; After training is complete, a global fixation operation is performed to recalculate on the validation set. As the final compensation feature vector And then solidify and store it.

[0011] Furthermore, in this invention, the loss function of the training method... Defined as:

[0012] in: The task loss (such as cross-entropy loss) is valid when all modalities are valid.

[0013] and These represent the sets of all possible single-modal and bimodal missing modes, respectively.

[0014] Indicates missing mode The task loss is calculated by replacing the missing modalities with the corresponding compensation feature vectors.

[0015] and These are preset weighting coefficients used to balance the importance of different missing cases.

[0016] In each training iteration, a portion of samples are randomly selected to simulate modality loss using forward propagation, and the total loss is calculated accordingly for backpropagation. This is done to jointly optimize the parameters of the backbone feature extraction module, cross-modal semantic mapping module, fixed-weight fusion module, and task decoding module, while also updating the compensation feature vector.

[0017] Furthermore, in this invention, the fixed-weight fusion module receives four 256-dimensional vectors, including one real-time feature and three compensation features, and weights them according to a preset quaternary weighting coefficient. Perform linear combination to output fused features All weighting coefficients All are 8-bit fixed-point numbers and satisfy the following conditions: Fusion computing is performed using integer multiplication and addition instructions, without floating-point operations.

[0018] Furthermore, in this invention, the backbone feature extraction module is composed of several independent sub-networks, each of which is dedicated to processing raw input data of a specific modality, and the sub-networks are completely isolated from each other at the parameter level.

[0019] Furthermore, in this invention, the cross-modal semantic mapping module is implemented by a linear fully connected layer, the function of which is to project the original high-dimensional feature vectors extracted from each modality onto a unified semantic embedding space with a fixed dimension.

[0020] Furthermore, in this invention, the modal state detection unit is directly connected to the data ready pin of each sensor via a hardware signal line. When the data ready signal of a certain sensor is not set within multiple consecutive sampling periods, it is determined that the mode is missing.

[0021] This invention also discloses a multimodal missing robust fusion processing system suitable for edge devices, used to implement the above method, the system comprising: The modal state detection unit is used to monitor the validity of data from each sensing channel in real time. The storage unit is divided into three physical storage areas: the first area stores the quantization parameters of the backbone feature extraction module and the cross-modal semantic mapping module; the second area stores the compensation feature vectors corresponding to all missing modes; and the third area stores the weighting coefficients of the fixed weight fusion module and the parameters of the task decoding module. The processing unit is used to execute the computational logic of the fixed-weight fusion module and the task decoding module.

[0022] As a further preferred embodiment of the system of the present invention, the cross-modal semantic mapping module, the fixed weight fusion module and the task decoding module are all implemented in a fixed-point quantization format, and their weights, activation values ​​and weighting coefficients are all represented by 8-bit integers and are calculated by integer multiplication and addition instructions.

[0023] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the above-described method.

[0024] The beneficial effects of this invention are as follows: By employing a fixed-weight fusion strategy and pre-defined compensation features, dynamic weight calculations or online optimizations during the inference phase are completely avoided. All calculations are deterministic operations, without branch predictions or complex logic, which not only reduces computational latency but also completely eliminates output jitter, meeting the stringent requirements for real-time performance and determinism in scenarios such as industrial control.

[0025] The entire converged architecture is based on fixed-point integer arithmetic, eliminating the need for a floating-point unit (FPU), enabling efficient deployment on low-cost, resource-constrained microcontrollers (MCUs). Fixed parameters and compensation vectors are easily stored in ROM, reducing runtime memory usage and power consumption.

[0026] Compared to traditional dynamic weighting methods that passively "ignore" missing modalities, this invention actively "compensates" for the semantic information of missing modalities. This allows the system to maintain stable performance output even when faced with any combination of missing modalities, effectively preventing a precipitous drop in performance. Experimental data shows that the performance advantage is particularly pronounced under extreme conditions such as single-modal operation.

[0027] Model parameters, fusion weights, and compensation vectors are all fixed after training, eliminating the need for complex runtime adaptations during deployment. Modal state detection is directly triggered by hardware signals, resulting in rapid and reliable responses. The entire system exhibits predictable behavior, is easy to test and verify, and meets the development standards for high-reliability embedded systems. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall architecture of the multimodal missing robust fusion processing method for edge devices described in this invention.

[0029] Figure 2 This is a schematic diagram of the structure of the backbone feature extraction module and the cross-modal semantic mapping module in this invention.

[0030] Figure 3 This is a schematic diagram illustrating the workflow of the present invention, which switches to the corresponding compensation feature channel based on the modality missing mode during the inference phase.

[0031] Figure 4 This is a schematic diagram of the data flow and connection relationship between the fixed-weight fusion module and the task decoding module of the present invention.

[0032] Figure 5 This is a flowchart illustrating the multi-objective joint optimization and compensation feature solidification process during the training phase of this invention. Detailed Implementation

[0033] This invention provides a robust fusion processing method for multimodal data loss in edge devices. By constructing a multi-branch feature extraction and fusion architecture with structured redundancy and deterministic switching mechanisms, this method maintains the stability and accuracy of the system output even when any modality data is missing, without requiring dynamic adjustment of fusion weights or online optimization.

[0034] S1: During the model training phase, an end-to-end neural network architecture is first constructed. This architecture includes a backbone feature extraction module, a cross-modal semantic mapping module, a fixed-weight fusion module, and a task decoding module. The backbone feature extraction module consists of several independent sub-networks, each dedicated to processing the raw input data of a specific modality. These sub-networks are completely isolated at the parameter level, with no shared layers or cross-connections. The cross-modal semantic mapping module is placed after each backbone feature extraction sub-network. Its function is to project the raw high-dimensional feature vectors extracted from that modality onto a unified semantic embedding space, making the features of different modalities comparable and composable in terms of dimensionality and distribution characteristics. The dimension of the unified semantic embedding space is a preset constant, and all mapping operations for all modalities are implemented using linear fully connected layers, whose weight matrices are fixed after training.

[0035] Furthermore, a set of pre-defined backup feature channels is introduced after the cross-modal semantic mapping module. Each backup feature channel corresponds to a specific modality missing mode and a set of compensation feature vectors generated during the training phase are pre-stored. These compensation feature vectors do not originate from real sensor data but are representative prototype vectors obtained by clustering the statistical centers of each modality feature in a unified semantic embedding space under complete modality input conditions. The set of compensation feature vectors for each missing mode is solidified and stored in non-volatile memory after training, serving as a substitute input for the missing modality under that mode.

[0036] In a preferred embodiment of the present invention, the fixed-weight fusion module receives real-time feature vectors from the effective modalities and compensated feature vectors from the backup feature channels, and linearly combines them according to preset, immutable weighting coefficients. The weighting coefficients are jointly optimized during model training by minimizing the sum of the task loss function under full-modal input conditions and the task loss functions under all single-modal missing conditions, and are fixed in read-only storage during deployment. The fixed-weight fusion module does not contain any learnable parameters or dynamic adjustment mechanisms; its output is a deterministic fusion feature vector, which is directly fed into the subsequent task decoding module.

[0037] The task decoding module is a shared, fully connected network whose structure and parameters are determined once during training. It is used to map the fused feature vectors to the final task output space, including but not limited to classification labels, regression values, or detection box parameters. This module remains static throughout the inference process and its structure or parameters are not adjusted due to changes in the combination of input modalities.

[0038] S2: During the inference phase, a lightweight modal state detection unit monitors the data validity of each sensor channel in real time. Integrated into the operating system kernel or dedicated coprocessor of the edge device, the modal state detection unit determines whether each modality is in a valid state based on a preset signal integrity threshold. When a modality is determined to be missing, the system immediately loads the compensation feature vector corresponding to the current missing mode from non-volatile memory and sends it, along with the real-time feature vectors of the remaining valid modes, to the fixed-weight fusion module. The entire switching process is triggered by a hardware interrupt, with a response latency below a preset microsecond threshold, ensuring that the system completes the state transition within the first inference cycle after a modality is missing.

[0039] Furthermore, this invention defines a complete set of modality missing modes. This set includes all possible single-modal missing modes, bimodal missing modes, and even the extreme case where only a single modality remains valid. For each missing mode, there exists a unique set of compensation feature vectors and a unique set of fixed fusion weight vectors. After model training, the compensation feature vector sets and fusion weight vectors for all missing modes are serialized and stored in the flash memory chip of the edge device, occupying no more than 15% of the total model volume after compression.

[0040] In another preferred embodiment of the present invention, the linear fully connected layer in the cross-modal semantic mapping module is implemented using a fixed-point quantization format, with its weights and activation values ​​represented by 8-bit integers. The weighting coefficients in the fixed-weight fusion module are also stored in an 8-bit fixed-point format, and fusion calculations are performed through shift and accumulation operations, thereby avoiding the use of floating-point arithmetic units. The task decoding module also employs the same quantization strategy, ensuring that the entire inference process can be executed efficiently on a microcontroller without a floating-point coprocessor.

[0041] Each subnetwork in the backbone feature extraction module must meet the resource constraints of the edge device. Specifically, the visual modality subnetwork is constructed using depthwise separable convolutions, with no more than twelve layers and a total number of parameters controlled within 500,000; the audio modality subnetwork uses a hybrid structure of one-dimensional convolutions and gated recurrent units, with a maximum time step of 128 and no more than 256 hidden units; the inertial modality subnetwork uses a three-layer fully connected structure with six input dimensions (three-axis acceleration and three-axis angular velocity) and a preset dimension of the unified semantic embedding space for output; the environmental modality subnetwork uses a two-layer fully connected structure with four input dimensions (temperature, humidity, air pressure, and light intensity) and the same output dimension as above. The activation functions of all subnetworks are limited to modified linear units and do not include batch normalization layers.

[0042] During the training phase, a multi-objective joint optimization strategy is employed. First, under the condition of complete modal input, the parameters of the entire network are optimized through standard supervised learning. Second, in each training iteration, some modal inputs are randomly masked, and the corresponding compensation feature vectors are forcibly used to replace the missing modalities, while simultaneously calculating the task loss at this time. Finally, the weighted sum of the complete modal loss and the losses of all sampled missing modes is used as the total loss function for backpropagation to update the network parameters. The compensation feature vector is initialized to a zero vector at the beginning of training and is gradually updated to the conditional expectation value of the effective modal features under the corresponding missing modes through a moving average method as training progresses.

[0043] Furthermore, a global compensation feature solidification operation is performed after training is complete. This operation iterates through all predefined missing modes, clusters the feature distribution of the effective modes under each mode, and selects the cluster centers as the final compensation feature vector for the missing modes under that mode. The clustering algorithm uses the K-means algorithm, and the number of clusters is fixed at one, meaning that only one compensation feature vector is generated for each missing mode. This vector is then written into the device firmware as the sole alternative source during the inference phase.

[0044] The method described in this invention divides the entire model into three physical storage areas during deployment: the first area stores the quantization parameters of the backbone feature extraction module and the cross-modal semantic mapping module; the second area stores the compensation feature vectors corresponding to all missing modes; and the third area stores the weighting coefficients of the fixed-weight fusion module and the parameters of the task decoding module. All three areas are implemented using read-only memory to ensure they cannot be modified during runtime, thereby preventing fusion logic anomalies caused by memory errors.

[0045] In another embodiment of the present invention, the modal state detection unit is directly connected to the data ready pins of each sensor via hardware signal lines. If the data ready signal of a sensor is not set within three consecutive sampling periods, that mode is marked as missing. This mechanism does not rely on software polling or operating system scheduling, thus ensuring the real-time and deterministic nature of missing state detection.

[0046] The unified semantic embedding space has a dimension of 256. This dimension value was determined after comprehensively considering the feature representation capability and the memory bandwidth limitations of edge devices. It is sufficient to carry multimodal semantic information while keeping the data transfer volume of a single fusion operation within 8,000 bytes.

[0047] When any mode is missing, the method described in this invention ensures that the generation process of the fused feature vector does not introduce any randomness or uncertainty. All operations are deterministic computations, and their results depend only on the currently valid set of modes and the pre-stored compensation data. Therefore, under the same input conditions, regardless of how the missing mode changes, the system output remains consistent, meeting the requirements of deterministic response in industrial control scenarios.

[0048] In one specific embodiment, consider an edge intelligent terminal for industrial predictive maintenance, which integrates four sensing modalities: a visible light camera, a microphone array, a six-axis inertial measurement unit (IMU), and a four-in-one environmental sensor (temperature, humidity, air pressure, and light intensity). This terminal needs to determine the equipment's operating status locally in real time and output three categories of tags: normal, warning, or fault. According to the present invention, four independent backbone feature extraction subnetworks are first constructed. The visual subnetwork uses a lightweight variant of MobileNetV2, consisting of eleven depthwise separable convolutional layers. The output feature map is then subjected to global average pooling to obtain a 1280-dimensional vector, which is further reduced to 256 dimensions by a fully connected layer. The audio subnetwork takes 128 frames of 40-dimensional Mel-ray spectrograms as input, passes through two 1D convolutional layers (kernel size 5, channels 64 and 128 respectively), and then connects to two GRU layers (256 hidden units per layer). The output at the last time step is taken and mapped to 256 dimensions through a fully connected layer. The IMU subnetwork receives 6-dimensional raw data (3-axis acceleration + 3-axis angular velocity), which is processed by three fully connected layers (128, 64, and 256 nodes respectively). The environment subnetwork receives 4-dimensional sensor readings, which are processed by two fully connected layers (32 and 256 nodes respectively). The activation function for all subnetworks is ReLU.

[0049] The 256-dimensional original feature vectors output by each sub-network are input into their respective cross-modal semantic mapping modules. Each mapping module is a 256×256 linear fully connected layer, whose weights are quantized into 8-bit integers and fixed after training. The mapped features are then located in a unified 256-dimensional semantic embedding space.

[0050] For this four-modal system, the total number of predefined missing modes is 2. 4 -1 = 15 possibilities (excluding invalid cases with complete missing features). Each missing mode corresponds to a 256-dimensional compensation feature vector. For example, in the "visual modality only" missing mode, audio, IMU, and environmental modalities are all missing. In this case, the system loads three pre-stored compensation vectors for this mode from flash memory to replace the three missing modalities respectively. The fixed-weight fusion module receives four 256-dimensional vectors (one real-time feature + three compensation features) and weights them according to preset quaternary weighting coefficients. Perform linear combination to output fused features All weighting coefficients All are 8-bit fixed-point numbers and satisfy the following conditions: (This facilitates normalized division by right shifting by 8 bits). The fused calculation is performed using integer multiplication and addition instructions, without floating-point operations.

[0051] The task decoding module is a three-layer fully connected network (256→128→64→3), and the output is processed by Softmax to obtain three types of probabilities. This module also uses 8-bit quantization, and all parameters are stored in ROM.

[0052] The modal state detection unit monitors the data readiness signals of each sensor via GPIO pins. If a sensor fails to raise its readiness signal within three consecutive 10ms sampling cycles, that mode is marked as missing. This determination is implemented by a hardware state machine with a delay of less than 5μs.

[0053] During the training phase, a dataset containing 100,000 labeled samples was used. Each sample contained synchronously collected four-modal data and device status labels. During training, in addition to the standard full-modal forward propagation, any modality was randomly masked with a probability of 0.3, and any two modalities were masked with a probability of 0.1. When masking, the features of the missing modality were replaced with the compensation vector (initially a zero vector) corresponding to the missing modality in the current iteration. The total loss function is:

[0054] in For full-modal cross-entropy loss, , These are sets of single-deletion and double-deletion patterns, respectively. =0.7, =0.5. The compensation vector is updated using a moving average:

[0055] in As a smoothing factor, The preferred value is 0.99. For the current batch in the missing pattern The mean of the effective modal features.

[0056] After training, feature consolidation is performed: for each missing pattern The mapping results of all valid modal features are collected on the validation set, and their mean in 256-dimensional space is calculated. This mean is then used as the final compensation vector. The vector is written to the second storage area of ​​the device firmware.

[0057] To verify the effectiveness of the present invention, the following embodiments and comparative examples were designed.

[0058] Example: The above-described quadmodal system was deployed on an edge device based on an ARM Cortex-M7 core (480MHz clock speed, no FPU) using the method described in this invention. The total model size was 1.8MB, of which the compensation features and fusion weights occupied 260KB (approximately 14.4%).

[0059] Comparative Example: The traditional dynamic weight fusion method is adopted, that is, equal weights are dynamically allocated according to the number of effective modes during inference (e.g., if three modes are effective, each accounts for 1 / 3), without compensation mechanism. The structure of the remaining modules is the same as the example.

[0060] In a simulated field environment, the classification accuracy of the two schemes under different missing patterns was tested, and the results are shown in the table below:

[0061] As can be seen from the table, the two methods perform similarly under full-modal conditions; however, in cases where any modality is missing, the accuracy of the embodiments is significantly higher than that of the comparative embodiments, especially in single-modal scenarios. This indicates that the present invention effectively mitigates the performance degradation caused by modality loss through preset compensation features and fixed fusion weights.

[0062] In addition, inference latency was measured. On the Cortex-M7 platform, the single inference time of the embodiment was 18.7ms (including modality detection and feature switching), while that of the comparative embodiment was 18.5ms, a negligible difference. Both meet the industrial real-time requirement of 20ms. However, the comparative embodiment exhibits microsecond-level nondeterministic jitter during modality switching due to the need to recalculate the weight allocation logic, while the embodiment, using fixed logic throughout, has jitter of less than 0.1μs, meeting the deterministic control requirements.

[0063] In summary, this invention fundamentally avoids the stability issues caused by computational overhead and weight drift on edge devices in existing technologies by transforming the modality loss response mechanism from dynamic weight adjustment to structured pre-setting. The method achieves maximum performance when all modalities are available and seamlessly switches to a predefined stable operating state when any modality is missing. Furthermore, the entire switching process requires no additional computational resources, fully meeting the core requirements of edge intelligence for low latency, high reliability, and strong robustness.

[0064] All contents not described in detail in the specification are existing technologies known to those skilled in the art, and the model parameters of each sensor and processor are not specifically limited; conventional equipment can be used. Electrical control components not mentioned in this technical solution are existing technologies and are therefore not shown in the figures, nor will they be described here.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robust fusion processing method for multimodal missing features applicable to edge devices, characterized in that, The method includes: During the model training phase, an end-to-end neural network architecture is constructed, which includes a backbone feature extraction module, a cross-modal semantic mapping module, a fixed weight fusion module, and a task decoding module. Following the cross-modal semantic mapping module, a set of compensation feature vectors corresponding to various modal missing modes is preset; During the inference phase, the validity of data from each sensor channel is monitored in real time through the modal state detection unit. When a mode is detected to be missing, the system loads the compensation feature vector corresponding to the current missing mode and inputs it together with the real-time feature vector of the effective mode into the fixed weight fusion module. The fixed-weight fusion module linearly fuses the input real-time feature vector and the compensation feature vector according to a preset fixed weighting coefficient to generate a deterministic fused feature vector. The task decoding module decodes the fused feature vector to obtain the final task output.

2. The method according to claim 1, characterized in that, The method for generating the compensation feature vector includes the following steps: During model training, for each predefined modality missing mode m, its compensation feature vector cm is updated online using a moving average algorithm: ； in: This indicates the missing patterns at the t-th training iteration. The compensated feature vector; It is a smoothing factor; This indicates that in the current training batch, when in missing mode... At that time, the mean of the feature vectors of all effective modalities in the unified semantic embedding space; After training is complete, a global fixation operation is performed to recalculate on the validation set. As the final compensation feature vector And then solidify and store it.

3. The method according to claim 1, characterized in that, The loss function of the model training method Defined as: ； in: The task loss (such as cross-entropy loss) is valid when all modalities are effective. and These represent the sets of all possible single-modal and bimodal missing modes, respectively. Indicates missing mode The task loss is calculated by replacing the missing mode with the corresponding compensation feature vector. and These are the preset weighting coefficients; In each training iteration, a portion of samples are randomly selected to simulate modality loss using forward propagation, and the total loss is calculated accordingly for backpropagation. This is done to jointly optimize the parameters of the backbone feature extraction module, cross-modal semantic mapping module, fixed-weight fusion module, and task decoding module, while also updating the compensation feature vector.

4. The method according to claim 1, characterized in that, The fixed-weight fusion module receives four 256-dimensional vectors, including one real-time feature and three compensation features, and weights them according to a preset quaternary weighting coefficient. Perform linear combination to output fused features All weighting coefficients All are 8-bit fixed-point numbers and satisfy the following conditions: Fusion computing is performed using integer multiplication and addition instructions, without floating-point operations.

5. The method according to claim 1, characterized in that, The backbone feature extraction module consists of several independent sub-networks, each of which is dedicated to processing raw input data of a specific modality, and the sub-networks are completely isolated from each other at the parameter level.

6. The method according to claim 1, characterized in that, The cross-modal semantic mapping module is implemented by a linear fully connected layer, which projects the original high-dimensional feature vectors extracted from each modality onto a unified semantic embedding space with a fixed dimension.

7. The method according to claim 1, characterized in that, The modal state detection unit is directly connected to the data ready pin of each sensor through a hardware signal line. When the data ready signal of a certain sensor is not set within multiple consecutive sampling periods, it is determined that the mode is missing.

8. A multimodal missing robust fusion processing system suitable for edge devices, for implementing the method according to any one of claims 1-7, characterized in that, The system includes: The modal state detection unit is used to monitor the validity of data from each sensing channel in real time. The storage unit is divided into three physical storage areas: the first area stores the quantization parameters of the backbone feature extraction module and the cross-modal semantic mapping module; the second area stores the compensation feature vectors corresponding to all missing modes; and the third area stores the weighting coefficients of the fixed weight fusion module and the parameters of the task decoding module. The processing unit is used to execute the computational logic of the fixed-weight fusion module and the task decoding module.

9. The system according to claim 8, characterized in that, The cross-modal semantic mapping module, fixed-weight fusion module, and task decoding module are all implemented using fixed-point quantization format. Their weights, activation values, and weighting coefficients are all represented by 8-bit integers and are calculated using integer multiplication and addition instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

Images