Dynamic multi-mode signal fusion algorithm based on FPGA and adaptive noise suppression system

Abstract

The invention relates to the field of signal processing, and discloses an FPGA-based dynamic multi-modal signal fusion algorithm and an adaptive noise suppression system, and the algorithm comprises a signal collection module which is used for collecting signals of different modals through a multi-modal sensor, adding a timestamp and a sensor identifier to each modal signal, and obtaining an original signal; the space-time calibration module is used for performing time synchronization and space external parameter calibration on the original signal and outputting a space-time aligned single-mode signal; and the noise suppression module is used for mapping high-dimensional signals to low-dimensional Riemannian manifolds through a manifold embedding algorithm for each single-mode signal aligned in time and space, and suppressing non-Gaussian noise by using a manifold thermonuclear filtering algorithm. Space-time labels are given to multi-mode signals by means of a signal acquisition module, time deviation and space position difference between sensors are eliminated through a space-time calibration module, and non-Gaussian noise is filtered out through a noise suppression module based on manifold learning and intrinsic characteristics of the signals are reserved.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a dynamic multimodal signal fusion algorithm and an adaptive noise suppression system based on FPGA. Background Art

[0002] Multimodal signal fusion technology integrates heterogeneous signals such as vision, inertial measurement, and audio to support environmental perception and state estimation in scenarios such as autonomous driving and industrial inspection. As application scenarios become more complex, multimodal systems must meet the core requirements of spatiotemporal consistency, noise robustness, and real-time processing.

[0003] Traditional systems often employ static weight allocation strategies (such as fixed-ratio weighting) or quasi-static optimization algorithms (such as Kalman filtering), adjusting the contribution of each modality through preset weights or a slowly varying covariance matrix based on historical data, thereby relying on the ideal assumption of "stable modal performance." However, visual signals may fail suddenly due to occlusion, IMUs may experience a sudden increase in drift due to vibration, and audio signals may be contaminated by impulse noise. These sudden changes in modal performance cannot be perceived and responded to in real time by static or quasi-static strategies, causing fusion decisions to be out of sync with actual scenario requirements and leading to "fusion distortion" in multimodal signals. Signals with noisy or failed modes are incorrectly weighted, contaminating the fusion result. The fusion system is unable to reduce their weights in a timely manner, leading to misjudgment of device status. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides an FPGA-based dynamic multimodal signal fusion algorithm and an adaptive noise suppression system to solve the problem in traditional systems where fusion decisions are out of touch with actual scenario requirements, leading to misjudgment of device status.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a dynamic multimodal signal fusion algorithm and adaptive noise suppression system based on FPGA, comprising:

[0006] Signal acquisition module: used to collect signals of different modes using multimodal sensors, and add timestamps and sensor identifiers to each modal signal to obtain the original signal;

[0007] Time and space calibration module: used to synchronize the original signal in time and calibrate the spatial external parameters, and output a single-mode signal with time and space alignment;

[0008] Noise suppression module: This module is used to map high-dimensional signals to low-dimensional Riemannian manifolds using a manifold embedding algorithm for each spatiotemporally aligned single-modal signal, and suppress non-Gaussian noise using a manifold heat kernel filtering algorithm, outputting the denoised modal signal.

[0009] Fusion decision module: It is used to solve the dynamic fusion weights of each modal signal by constructing a dynamic system model including state equations and performance indicator functions, and perform weighted fusion on the denoised modal signals based on the dynamic fusion weights to output a multimodal fusion signal.

[0010] By adopting the above technical solution, the signal acquisition module can be used to assign spatiotemporal labels to multimodal signals. The spatiotemporal calibration module eliminates the time deviation and spatial position difference between sensors. The noise suppression module uses manifold learning to filter out non-Gaussian noise and retain the essential characteristics of the signal. The dynamic system model of the fusion decision module is used to solve the fusion weight of each modality in real time. When the performance of a certain modality is reduced by environmental interference, the system automatically reduces its weight ratio and enhances the contribution of reliable modalities, so that the fusion process can closely follow the scene changes and adjust dynamically. This realizes the full process adaptive processing of multimodal signals from acquisition, calibration, noise reduction to fusion, ensuring that the fusion decision always fits the real-time performance status of each modality, solving the problem in traditional systems where the fusion decision is out of touch with the actual scene requirements, leading to misjudgment of equipment status.

[0011] Preferably, the signal acquisition module includes a signal acquisition unit and a preprocessing unit. The signal acquisition unit is used to use a multimodal sensor to collect signals of different modes. The multimodal sensor includes an industrial camera, a microphone, and an inertial measurement unit. The signal includes a visual image, audio data, and an inertial measurement unit signal. The preprocessing unit is used to use a hard-core timer inside the FPGA to generate a nanosecond timestamp, generate a unique sensor identifier based on the sensor type encoding rules, and attach the timestamp and sensor identifier to the frame header / packet header of the corresponding modal signal through a bit splicing operation to form an original signal with a time and space label.

[0012] Preferably, the space-time calibration module includes a time synchronization unit and a space extrinsic parameter calibration unit. The time synchronization unit is used to perform bidirectional timestamp interaction on the original signal through a two-way PTP protocol, calculate clock deviation and transmission delay, and output a single-modal signal after time synchronization. The space extrinsic parameter calibration unit is used to solve the rotation matrix and translation vector between sensors for the single-modal signal after time synchronization based on a manifold alignment algorithm, convert each modal signal into a unified spatial coordinate system, and output a single-modal signal aligned in space and time.

[0013] Preferably, the two-way PTP protocol calculation method satisfies where t master1 , t master2 Send / receive timestamp for the master clock, t slave1 , t slave2 is the timestamp received / sent from the clock, δ is the time deviation, and τ is the transmission delay;

[0014] The manifold alignment algorithm optimizes the objective function Solve the rotation matrix R∈SO(3) and the translation vector Among them, f j (·) is the feature extraction function of the i-th mode, is the manifold distance metric, x j is the coordinate of the jth feature point.

[0015] Preferably, the noise suppression module includes a manifold embedding unit and a heat kernel filtering unit. The manifold embedding unit is used to map the high-dimensional signal to a low-dimensional Riemann manifold through an isometric mapping algorithm for a single-modal signal, and output a manifold feature vector. The heat kernel filtering unit is used to suppress non-Gaussian noise by solving the manifold heat equation for the manifold feature vector, and output the denoised modal signal.

[0016] Preferably, the isometric mapping algorithm includes constructing a k-nearest neighbor graph and calculating the local Euclidean distance matrix D local , solve the manifold shortest path matrix D by Floyd-Warshall algorithm shortest , solving the eigenvalue problem YLY T =ΛY, extract the first d-dimensional feature vectors as manifold embedding features, where L=DW is the manifold Laplace matrix, D is the degree matrix, W is the adjacency weight matrix, Y is the embedding feature matrix, and Λ is the eigenvalue matrix;

[0017] The manifold heat equation is in, is the Laplace-Beltrami operator, u i is the signal after noise reduction, y i is the input signal, and β is the noise suppression parameter, which is dynamically adjusted according to the noise type.

[0018] Preferably, the fusion decision module includes a dynamic system modeling unit and a weight optimization unit. The dynamic system modeling unit is used to construct a dynamic system model including a state equation and a performance index function based on the modal signal after noise reduction. The weight optimization unit is used to solve the dynamic fusion weights of each modal signal based on the dynamic system model, perform weighted fusion on the modal signal after noise reduction, and output a multimodal fusion signal.

[0019] Preferably, the state equation is The performance indicator function is Among them, X(t) is the fusion state vector, X * (t) is the ideal state vector, A i is the state transfer matrix of the i-th mode, C i is the noise input matrix, Q / S is the state error weight matrix, R i is the weight regularization coefficient.

[0020] Preferably, the dynamic fusion weight is solved by gradient descent method And through the projection operation to ensure that w i (t)∈[0,1] and Where η is the adaptive learning rate.

[0021] The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression method are applied to the above-mentioned FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system, including the following steps:

[0022] Signal acquisition: Use multimodal sensors to collect signals of different modes, and add timestamps and sensor identifiers to each modal signal to obtain the original signal;

[0023] Time and space calibration: perform time synchronization and spatial extrinsic calibration on the original signal to output a single-mode signal with time and space alignment;

[0024] Noise suppression: For each spatiotemporally aligned single-modal signal, the high-dimensional signal is mapped to a low-dimensional Riemannian manifold using a manifold embedding algorithm. The manifold heat kernel filtering algorithm is then used to suppress non-Gaussian noise, and the denoised modal signal is output.

[0025] Fusion decision: By constructing a dynamic system model containing state equations and performance indicator functions, the dynamic fusion weights of each modal signal are solved, and the denoised modal signals are weightedly fused based on the dynamic fusion weights to output a multimodal fusion signal.

[0026] The present invention provides a dynamic multimodal signal fusion algorithm and an adaptive noise suppression system based on FPGA. It has the following beneficial effects:

[0027] 1. The present invention assigns spatiotemporal labels to multimodal signals by means of a signal acquisition module, eliminates time deviations and spatial position differences between sensors through a spatiotemporal calibration module, filters non-Gaussian noise based on manifold learning and retains the essential characteristics of the signal through a noise suppression module, and solves the fusion weights of each modality in real time through a dynamic system model of a fusion decision module. When a certain modality is affected by environmental interference and its performance degrades, the system automatically reduces its weight and enhances the contribution of reliable modalities, allowing the fusion process to closely follow the scene changes and dynamically adjust. This achieves full-process adaptive processing of multimodal signals from acquisition, calibration, noise reduction to fusion, ensuring that the fusion decision always fits the real-time performance status of each modality, solving the problem in traditional systems where fusion decisions are out of touch with actual scene requirements, leading to misjudgment of device status.

[0028] 2. The present invention uses an FPGA hardware timer to add nanosecond-level timestamps and unique sensor identifiers to each modal signal, giving the signal spatiotemporal attributes from the source. It eliminates time deviations through the bidirectional PTP protocol and unifies the spatial coordinate system with the help of a manifold alignment algorithm. This allows the synchronization errors of visual, inertial, audio and other signals in the time dimension to be controlled at the nanosecond level, and achieves millimeter-level position alignment in the spatial dimension, thereby providing physically consistent input for subsequent fusion processing, avoiding the problem of fusion logic breaks caused by spatiotemporal misalignment, ensuring that the time phase of the multimodal signal strictly corresponds to the spatial coordinates, and laying the foundation for accurate perception in complex scenarios.

[0029] 3. The present invention uses an isometric mapping algorithm to embed high-dimensional signals into low-dimensional Riemannian manifolds for single-modal signals after time-space alignment, stripping off redundant dimensions and retaining the nonlinear geometric structure of the signal; then uses the manifold heat kernel filtering algorithm to simulate the diffusion process of the signal on the manifold, and implements "structure-aware suppression" for non-Gaussian noise such as pulses and salt and pepper. The noise is attenuated due to its discrete distribution on the manifold, while the continuous manifold characteristics of the signal are retained, providing a pure and feature-complete single-modal signal for the fusion decision module, reducing the noise pollution on the weight calculation.

[0030] 4. The present invention constructs a dynamic system model that includes state equations and performance indicator functions to capture the state evolution laws and noise characteristics of each modal signal in real time, enabling the multimodal fusion system to "actively perceive" modal performance changes and "respond instantly", ensuring that the fusion results in complex scenarios are always dominated by high-reliability modalities, thereby improving the robustness of environmental perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a system architecture diagram of the FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system proposed in the present invention;

[0032] Figure 2 This is a flow chart of the method of the FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression method proposed by the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] Please see the attached Figure 1 The embodiment of the present invention provides a dynamic multimodal signal fusion algorithm and an adaptive noise suppression system based on FPGA, including:

[0035] Signal acquisition module: used to use multimodal sensors to collect signals of different modes, and add timestamps and sensor identifiers to each modal signal to obtain the original signal; the signal acquisition module includes a signal acquisition unit and a preprocessing unit. The signal acquisition unit is used to use multimodal sensors to collect signals of different modes. Multimodal sensors include industrial cameras, microphones, and inertial measurement units. The signals include visual images, audio data, and inertial measurement unit signals. The preprocessing unit is used to use the hard-core timer inside the FPGA to generate nanosecond-level timestamps, generate a unique sensor identifier based on the sensor type encoding rules, and attach the timestamp and sensor identifier to the frame header / packet header of the corresponding modal signal through bit splicing operations to form the original signal with time and space labels.

[0036] Specifically, the signal acquisition unit of the signal acquisition module deploys industrial cameras, microphones, and inertial measurement units (IMUs) to acquire visual images, audio data, and inertial measurement signals, respectively. It uses the spatial scene perception capabilities of visual signals, the environmental acoustic perception capabilities of audio signals, and the motion state perception capabilities of IMU signals to construct the original information input of the multimodal perception dimension; the preprocessing unit relies on the FPGA internal timestamp counter (TTC) hard-core architecture to generate nanosecond timestamps. The parallel execution characteristics of its hardware circuit avoid the instruction delay loss of the software timer, ensuring that the timestamp accuracy matches the time-space calibration module's accuracy requirements for multimodal signal time synchronization; at the same time, the preprocessing unit The element generates a unique sensor identifier according to the preset coding rules of the sensor type (such as 0x01 for industrial cameras, 0x02 for microphones, and 0x03 for IMUs), implements the bit splicing operation of the timestamp and the sensor identifier through the FPGA logic gate array, and encapsulates the two into the frame header or packet header field of each modal signal, so that the time-space calibration module can directly parse the time-space label from the signal frame structure, providing basic data for the time deviation calculation of the two-way PTP protocol and the modal feature differentiation of the manifold alignment algorithm, ensuring that the multimodal signal carries the associated identifier of the unified time-space reference before entering the subsequent processing flow, supporting the time-space consistency of the system multimodal fusion from the signal acquisition source.

[0037] Time-space calibration module: used to perform time synchronization and spatial extrinsic parameter calibration on the original signal, and output a time-space aligned single-modal signal; the time-space calibration module includes a time synchronization unit and a spatial extrinsic parameter calibration unit. The time synchronization unit is used to perform bidirectional timestamp interaction on the original signal through a two-way PTP protocol, calculate clock deviation and transmission delay, and output a time-synchronized single-modal signal. The spatial extrinsic parameter calibration unit is used to solve the rotation matrix and translation vector between sensors for the time-synchronized single-modal signal based on the manifold alignment algorithm, convert each modal signal into a unified spatial coordinate system, and output a time-space aligned single-modal signal.

[0038] The calculation method of the two-way PTP protocol meets the requirements where t master1 , t master2 Send / receive timestamp for the master clock, t slave1 , t slave2 is the timestamp received / sent from the clock, δ is the time deviation, and τ is the transmission delay;

[0039] The manifold alignment algorithm optimizes the objective function Solve the rotation matrix R∈SO(3) and the translation vector Among them, f j (·) is the feature extraction function of the i-th mode, is the manifold distance metric, x j is the coordinate of the jth feature point.

[0040] Specifically, the time-space calibration module receives the original signal with time-space labels output by the signal acquisition module, and relies on the cascaded collaboration of the time synchronization unit and the spatial external parameter calibration unit to achieve time-space consistency calibration of the multimodal signal. The time synchronization unit is based on the two-way PTP protocol, using the nanosecond timestamp of the original signal frame header (generated by the FPGA hard-core timer of the signal acquisition module) through the two-way timestamp interaction between the master and slave clocks - the master clock (such as the clock source of the industrial camera) sends the timestamp t master1 , receiving timestamp t master2 , receives the timestamp t from the clock (such as the clock source of IMU or microphone) slave1 , send timestamp t slave2 , substitute the formula to calculate the clock deviation δ and transmission delay τ, correct the timestamp of each modal signal, and output the single-modal signal after time synchronization; this process uses the FPGA's hardware parallel logic to accelerate the timestamp calculation, ensure the alignment accuracy of multi-modal signals in the time dimension, and provide the same-frequency signal basis for subsequent spatial calibration.

[0041] The spatial extrinsic calibration unit extracts the features of each modality (such as the scene structure features of the visual signal and the motion trajectory features of the IMU) for the time-synchronized single-modal signal to construct a manifold structure, and then optimizes the objective function through the manifold alignment algorithm. where f j (·) Adapting modal characteristics to extract differentiated features (e.g., SIFT features for visual signals and motion trajectory features for IMU signals), The nonlinear distance between feature points is measured using the manifold geodesic distance to solve the rotation matrix R∈SO(3) and translation vector that satisfy the orthogonal constraint. Different modal signals are mapped to a unified spatial coordinate system. This operation distinguishes the modal type based on the sensor identification of the signal acquisition module, ensuring the modal specificity of the external parameter solution. The final output is a spatiotemporal aligned single-modal signal, which provides a unified spatiotemporal reference for the manifold embedding processing of the noise suppression module, avoiding fusion distortion of multimodal signals caused by spatiotemporal deviation.

[0042] Noise suppression module: used for mapping high-dimensional signals to low-dimensional Riemann manifolds for each spatiotemporally aligned single-modal signal through a manifold embedding algorithm, and suppressing non-Gaussian noise using a manifold heat kernel filtering algorithm, and outputting the denoised modal signal; the noise suppression module includes a manifold embedding unit and a heat kernel filtering unit. The manifold embedding unit is used to map high-dimensional signals to low-dimensional Riemann manifolds for single-modal signals through an isometric mapping algorithm, and outputs the manifold eigenvector. The heat kernel filtering unit is used to suppress non-Gaussian noise for the manifold eigenvector by solving the manifold heat equation, and outputs the denoised modal signal.

[0043] The isometric mapping algorithm includes constructing a k-nearest neighbor graph and calculating the local Euclidean distance matrix D local , solve the manifold shortest path matrix D by Floyd-Warshall algorithm shortest , solving the eigenvalue problem YLY T =ΛY, extract the first d-dimensional feature vectors as manifold embedding features, where L=DW is the manifold Laplace matrix, D is the degree matrix, W is the adjacency weight matrix, Y is the embedding feature matrix, and Λ is the eigenvalue matrix;

[0044] The heat equation for the manifold is in, is the Laplace-Beltrami operator, u i is the signal after noise reduction, y i is the input signal, and β is the noise suppression parameter, which is dynamically adjusted according to the noise type.

[0045] Specifically, the noise suppression module receives the spatiotemporal aligned unimodal signal output by the spatiotemporal calibration module, and relies on the cascade processing of the manifold embedding unit and the hot kernel filter unit to achieve non-Gaussian noise suppression. The manifold embedding unit first captures the local structural association of the signal by constructing a k-nearest neighbor graph for unimodal signals (such as the pixel sequence of a visual image and the motion data sequence of an IMU). The neighbor relationship is based on the distribution of the signal in the high-dimensional feature space (such as the grayscale gradient feature of the visual signal and the acceleration change feature of the IMU signal), and calculates the local Euclidean distance matrix D. local , convert the local approximate Euclidean space characteristics in high-dimensional space into matrix form; then iteratively solve the manifold shortest path matrix D through the Floyd-Warshall algorithm shortest, expand the local distance to the manifold geodesic distance, restore the nonlinear manifold structure of the signal; then construct the manifold Laplace matrix L = DW (where D is the degree matrix, which counts the node connection weights and W is the adjacency weight matrix, which quantifies the neighbor association strength) to solve the eigenvalue problem YLY T =ΛY, extract the first d-dimensional feature vectors as the embedding features of the low-dimensional Riemann manifold, map the high-dimensional signal to the low-dimensional manifold, strip off the high-dimensional redundancy and retain the essential structure of the signal, laying the foundation for noise suppression.

[0046] The heat kernel filter unit receives the manifold embedded feature vector, based on the manifold heat equation Processing: Among them is the Laplace-Beltrami operator, which describes the signal diffusion characteristics on the manifold u i is the signal after noise reduction, y i The embedded feature input is β, which is dynamically adjusted based on the noise type (e.g., increasing for impulse noise and adapting for Gaussian noise). This simulates the heat diffusion process to suppress non-Gaussian noise. The discrete distribution of noise on the manifold attenuates with "heat diffusion," while preserving the signal's continuous manifold structure. This processing results in a de-noised modal signal, providing a cleaner input for the fusion decision module, reducing noise interference in the calculation of dynamic fusion weights and indirectly improving the accuracy of the multimodal fusion signal.

[0047] Fusion decision module: This module is used to solve the dynamic fusion weights of each modal signal by constructing a dynamic system model that includes state equations and performance indicator functions. It then performs weighted fusion on the denoised modal signals based on the dynamic fusion weights to output a multimodal fusion signal. The fusion decision module includes a dynamic system modeling unit and a weight optimization unit. The dynamic system modeling unit is used to construct a dynamic system model that includes state equations and performance indicator functions based on the denoised modal signals. The weight optimization unit is used to solve the dynamic fusion weights of each modal signal based on the dynamic system model, perform weighted fusion on the denoised modal signals, and output a multimodal fusion signal.

[0048] The state equation is The performance indicator function is Among them, X(t) is the fusion state vector, X * (t) is the ideal state vector, A i is the state transfer matrix of the i-th mode, C i is the noise input matrix, Q / S is the state error weight matrix, R i is the weight regularization coefficient.

[0049] Dynamic fusion weights are solved by gradient descent method And through the projection operation to ensure that w i (t)∈[0,1] and Where η is the adaptive learning rate.

[0050] Specifically, the fusion decision module receives the denoised modal signal output by the noise suppression module, and relies on the collaborative operation of the dynamic system modeling unit and the weight optimization unit to achieve dynamic weighted fusion of multimodal signals. The dynamic system modeling unit first analyzes the single-modal dynamic characteristics of each modal denoised signal (such as the visual scene feature flow and the IMU motion state sequence): based on the visual frame-to-frame change law and the IMU kinematic model, it constructs the state transfer matrix A. i , characterize the state evolution within the modal; extract the modal noise characteristics (such as visual residual noise, IMU drift noise) to construct the noise input matrix C i , quantize noise interference. On this basis, integrate the multimodal signals to form a fusion state vector X * (t), combined with the ideal state X expected by the system * (t), establish the state equation where w i (t) is the dynamic weight to be determined, reflecting the modal contribution, and Bu(t) describes the impact of external control on the fusion state.

[0051] Synchronously define performance indicator functions Matrices Q and S weight the real-time and terminal errors respectively, reflecting the system accuracy requirements; R i Regularizing the weights prevents excessive dominance of a single modality and ensures fusion robustness. This function is then used to integrate fusion accuracy and weight rationality into a unified optimization objective, providing a criterion for weight solution.

[0052] The weight optimization unit is based on the above model and iteratively solves the weight through the gradient descent method: calculate the gradient of the performance index with respect to the weight Among them, η adaptively adjusts the step size to balance the convergence speed and stability; synchronously constrains w through the projection operation i (t)∈[0,1] and Ensure the physical rationality of weights (e.g., automatic weight reduction in the event of visual distortion, and improved adaptation of modal weights such as IMU). After optimization, the noise-reduced multimodal signal is weighted and fused. The output multimodal fusion signal integrates the advantages of each modality, maintains stable state estimation accuracy in complex scenarios, and provides reliable support for system decision-making.

[0053] The signal acquisition module first acquires heterogeneous signals such as vision, inertial measurement, and audio through multimodal sensors. It then uses the FPGA's internal hardware timer to add a high-precision timestamp to each signal. It also generates a unique identifier based on the sensor type code, ensuring that the original signal carries clear spatiotemporal attributes and modal identity. These tagged signals then enter the spatiotemporal calibration module, where they exchange timestamps between master and slave clocks using the bidirectional Precision Time Protocol (PTP). This accurately calculates and compensates for the time offsets and transmission delays of the different sensors. The manifold alignment algorithm then solves for the spatial extrinsic parameters between the sensors, unifying the modal signals to the same time reference and spatial coordinate system, providing a temporally and spatially consistent single-modal signal foundation for subsequent processing.

[0054] The noise suppression module applies an isometric mapping algorithm to the spatiotemporally aligned unimodal signal, constructing local neighborhood relationships for the high-dimensional signal. It then restores the signal's nonlinear geometric structure through manifold shortest path calculations and maps it onto a low-dimensional Riemannian manifold that preserves its essential characteristics. Furthermore, it utilizes a manifold heat kernel filtering algorithm to simulate the signal's diffusion along the manifold, targeting the discrete characteristics of non-Gaussian noise. While filtering out pulses, salt-and-pepper noise, and other noise, it also maximizes the preservation of the signal's continuous manifold structure, preventing noise interference with modal reliability assessment.

[0055] The fusion decision module constructs a dynamic system model that includes state equations and performance indicator functions, and incorporates the state evolution laws and noise characteristics of each modal signal into a unified framework: the state equation describes the dynamic coupling relationship and noise influence of multimodal signals, and the performance indicator function balances the fusion accuracy and weight rationality. The dynamic fusion weights of each modality are solved in real time by the gradient descent method. This process not only considers the real-time error of the current signal, but also avoids excessive dominance of a single modality through regularization constraints. When the performance of a certain modality degrades due to environmental interference (such as visual occlusion, IMU drift), the model automatically adjusts the corresponding weight to reduce its contribution and increase the proportion of reliable modalities, so that the fusion result is always dominated by the most credible signal in the current scenario.

[0056] Relying on the FPGA's parallel computing architecture, the core algorithms of the above modules (such as manifold distance calculation and dynamic weight iteration) can be executed efficiently, ensuring that the entire process from signal acquisition to fusion output is completed within the real-time requirements of the embedded system, thus forming a closed-loop adaptive mechanism from the signal source to the final decision. It can perceive subtle changes in modal performance in real time, accurately evaluate the reliability of each modality through the pure signal after noise reduction, and then use dynamic weights to achieve instant response to scene changes. This fundamentally solves the problem of disconnection between fusion decisions and actual needs, avoids misjudgment of device status due to failure or incorrect weighting of noisy modes, and provides stable and reliable multimodal perception capabilities for complex scenarios such as autonomous driving and industrial inspection.

[0057] Please see the attached Figure 2 , a dynamic multimodal signal fusion algorithm and adaptive noise suppression method based on FPGA, applied to the above-mentioned dynamic multimodal signal fusion algorithm and adaptive noise suppression system based on FPGA, includes the following steps:

[0058] Signal acquisition: Use multimodal sensors to collect signals of different modes, and add timestamps and sensor identifiers to each modal signal to obtain the original signal;

[0059] Time and space calibration: perform time synchronization and spatial extrinsic calibration on the original signal to output a single-mode signal with time and space alignment;

[0060] Noise suppression: For each spatiotemporally aligned single-modal signal, the high-dimensional signal is mapped to a low-dimensional Riemannian manifold using a manifold embedding algorithm. The manifold heat kernel filtering algorithm is then used to suppress non-Gaussian noise, and the denoised modal signal is output.

[0061] Fusion decision: By constructing a dynamic system model containing state equations and performance indicator functions, the dynamic fusion weights of each modal signal are solved, and the denoised modal signals are weightedly fused based on the dynamic fusion weights to output a multimodal fusion signal.

[0062] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system, characterized by: include: Signal acquisition module: used to collect signals of different modes using multimodal sensors, and add timestamps and sensor identifiers to each modal signal to obtain the original signal; Time and space calibration module: used to synchronize the original signal in time and calibrate the spatial external parameters, and output a single-mode signal with time and space alignment; Noise suppression module: This module is used to map high-dimensional signals to low-dimensional Riemannian manifolds using a manifold embedding algorithm for each spatiotemporally aligned single-modal signal, and suppress non-Gaussian noise using a manifold heat kernel filtering algorithm, outputting the denoised modal signal. Fusion decision module: It is used to solve the dynamic fusion weights of each modal signal by constructing a dynamic system model including state equations and performance indicator functions, and perform weighted fusion on the denoised modal signals based on the dynamic fusion weights to output a multimodal fusion signal.

2. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 1, characterized in that: The signal acquisition module includes a signal acquisition unit and a preprocessing unit. The signal acquisition unit is used to use a multimodal sensor to collect signals of different modes. The multimodal sensor includes an industrial camera, a microphone, and an inertial measurement unit. The signals include visual images, audio data, and inertial measurement unit signals. The preprocessing unit is used to use a hard-core timer inside the FPGA to generate a nanosecond-level timestamp, generate a unique sensor identifier based on the sensor type encoding rules, and append the timestamp and sensor identifier to the frame header / packet header of the corresponding modal signal through a bit splicing operation to form an original signal with a time and space label.

3. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 1, characterized in that: The spatiotemporal calibration module includes a time synchronization unit and a spatial extrinsic parameter calibration unit. The time synchronization unit is used to perform bidirectional timestamp interaction on the original signal through a two-way PTP protocol, calculate clock deviation and transmission delay, and output a single-modal signal after time synchronization. The spatial extrinsic parameter calibration unit is used to solve the rotation matrix and translation vector between sensors for the single-modal signal after time synchronization based on a manifold alignment algorithm, convert each modal signal into a unified spatial coordinate system, and output a single-modal signal aligned in time and space.

4. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 3, characterized in that: The two-way PTP protocol calculation method satisfies where t master1 , t master2 Send / receive timestamp for the master clock, t slave1 , t slave2 is the timestamp received / sent from the clock, δ is the time deviation, and τ is the transmission delay; The manifold alignment algorithm optimizes the objective function Solve the rotation matrix R∈SO(3) and the translation vector Among them, f j (·) is the feature extraction function of the i-th mode, is the manifold distance metric, x j is the coordinate of the jth feature point.

5. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 1, characterized in that: The noise suppression module includes a manifold embedding unit and a heat kernel filtering unit. The manifold embedding unit is used to map the high-dimensional signal to a low-dimensional Riemann manifold through an isometric mapping algorithm for a single-modal signal, and output a manifold feature vector. The heat kernel filtering unit is used to suppress non-Gaussian noise by solving the manifold heat equation for the manifold feature vector, and output a denoised modal signal.

6. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 5, characterized in that: The isometric mapping algorithm includes constructing a k-nearest neighbor graph and calculating the local Euclidean distance matrix D local , solve the manifold shortest path matrix D by Floyd-Warshall algorithm shortest , solving the eigenvalue problem YLY T =ΛY, extract the first d-dimensional feature vectors as manifold embedding features, where L=DW is the manifold Laplace matrix, D is the degree matrix, W is the adjacency weight matrix, Y is the embedding feature matrix, and Λ is the eigenvalue matrix; The manifold heat equation is in, is the Laplace-Beltrami operator, u i is the signal after noise reduction, y i is the input signal, and β is the noise suppression parameter, which is dynamically adjusted according to the noise type.

7. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 1, characterized in that: The fusion decision module includes a dynamic system modeling unit and a weight optimization unit. The dynamic system modeling unit is used to construct a dynamic system model including a state equation and a performance index function based on the modal signal after noise reduction. The weight optimization unit is used to solve the dynamic fusion weights of each modal signal based on the dynamic system model, perform weighted fusion on the modal signal after noise reduction, and output a multimodal fusion signal.

8. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 7, characterized in that: The state equation is The performance indicator function is Among them, X(t) is the fusion state vector, X * (t) is the ideal state vector, A i is the state transfer matrix of the i-th mode, C i is the noise input matrix, Q / S is the state error weight matrix, R i is the weight regularization coefficient.

9. The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system according to claim 7, characterized in that: The dynamic fusion weight is solved by gradient descent method And through the projection operation to ensure that w i (t)∈[0,1] and Where η is the adaptive learning rate.

10. FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression method, characterized by: The FPGA-based dynamic multimodal signal fusion algorithm and adaptive noise suppression system applied to any one of claims 1 to 9 comprises the following steps: Signal acquisition: Use multimodal sensors to collect signals of different modes, and add timestamps and sensor identifiers to each modal signal to obtain the original signal; Time and space calibration: perform time synchronization and spatial extrinsic calibration on the original signal to output a single-mode signal with time and space alignment; Noise suppression: For each spatiotemporally aligned single-modal signal, the high-dimensional signal is mapped to a low-dimensional Riemannian manifold using a manifold embedding algorithm. The manifold heat kernel filtering algorithm is then used to suppress non-Gaussian noise, and the denoised modal signal is output. Fusion decision: By constructing a dynamic system model containing state equations and performance indicator functions, the dynamic fusion weights of each modal signal are solved, and the denoised modal signals are weightedly fused based on the dynamic fusion weights to output a multimodal fusion signal.

Images