Distributed streaming multi-mode fusion adaptive gradient compression optimization method and system

Abstract

The invention belongs to the field of multi-modal RGB-T data fusion, and discloses a distributed streaming multi-modal fusion-oriented adaptive gradient compression optimization method and system, and the method comprises the steps: carrying out the personalized compression of data features of different modals through employing a modal-sensitive gradient compression strategy, and enabling the modals to comprise an RGB modal and an infrared modal; introducing physical constraint time sequence alignment loss, and performing time sequence alignment on the multi-modal data subjected to personalized compression by utilizing a physical model based on a thermal diffusion equation; the dynamic batch processing strategy is sensed through the video memory, the use condition of the video memory is monitored in real time, and the batch processing size is dynamically adjusted according to the residual capacity of the video memory. According to the method, a brand new solution is provided for efficient training of distributed streaming multi-modal data, and the precision of a multi-modal data fusion task and the resource utilization rate are improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal RGB-T data fusion, and specifically relates to an adaptive gradient compression optimization method and system for distributed streaming multimodal fusion. Background Art

[0002] With the rapid development of distributed computing and multimodal perception technologies, real-time fusion training of streaming multimodal data (such as RGB-T video streams and multi-sensor time series data) has become a core technical requirement in fields such as autonomous driving and industrial inspection. By collaboratively processing multimodal data such as visible light (RGB) and thermal infrared (Thermal), it is possible to break through the environmental adaptability limitations of a single modality (such as lighting changes and extreme temperatures) and significantly improve the robustness of target detection. However, problems such as the transmission efficiency of high-dimensional data in distributed multimodal training, cross-modal time series consistency, and dynamic resource utilization have seriously restricted the efficiency and accuracy of practical application deployment. Existing technologies generally fail to effectively solve these problems, resulting in low efficiency in multimodal data processing.

[0003] Current distributed multimodal training optimization technologies have the following shortcomings: First, there is a mismatch between modal characteristics and compression strategies. Traditional unified gradient compression methods ignore the physical differences between gradients of different modalities. For example, RGB gradients usually exhibit strong spatial sparsity due to the concentration of edge features in the image, while infrared gradients are affected by heat diffusion characteristics and exhibit a continuous distribution of low amplitude. Therefore, a single gradient compression strategy cannot efficiently process data of different modalities, resulting in a waste of communication bandwidth and computational efficiency during training. Second, the lack of physical laws leads to the accumulation of temporal errors. Traditional sliding window alignment methods mainly rely on local feature matching and do not introduce physical constraints such as heat diffusion equations. This causes the spatiotemporal offset error to grow exponentially as the training iterations proceed in cross-modal training, seriously affecting the training stability and accuracy of the model.

[0004] Third, static resource allocation is inefficient. The fixed batch strategy used in existing methods cannot adapt to the dynamic changes in streaming data resolution and memory requirements, resulting in drastic fluctuations in memory utilization during training. In actual training, especially when facing high-resolution data streams, fixed batch sizes can lead to memory overflow or underutilization of computing resources when sufficient memory is available. Inefficient memory management not only limits the maximum utilization of hardware resources but also affects the stability and efficiency of training. Summary of the Invention

[0005] In order to solve the problems of communication overhead, timing alignment and video memory management in the prior art, the present invention provides an adaptive gradient compression optimization method and system for distributed streaming multimodal fusion. In view of the difference in gradient sparsity between RGB and infrared modes, a dynamic compression mechanism is designed, which significantly reduces the communication burden and maintains the model accuracy. In order to optimize the timing alignment of multimodal data, physical constraints based on the heat diffusion equation and an optical noise injection model are introduced to improve the timing consistency of streaming data and ensure the precise alignment of data of different modalities. In addition, combined with the dynamic batching mechanism of video memory awareness and the zero-copy data pipeline, the present invention can adjust the batch size in real time and optimize GPU utilization and computing efficiency. The present invention provides a new solution for the efficient training of distributed streaming multimodal data, improving the accuracy and resource utilization of multimodal data fusion tasks.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] An adaptive gradient compression optimization method for distributed streaming multimodal fusion, the method comprising:

[0008] A modality-sensitive gradient compression strategy is used to perform personalized compression on data features of different modalities, including RGB and infrared modalities.

[0009] A physical constraint timing alignment loss is introduced, and a physical model based on the heat diffusion equation is used to perform timing alignment on personalized compressed multimodal data.

[0010] Through the memory-aware dynamic batch processing strategy, the memory usage is monitored in real time and the batch size is dynamically adjusted according to the remaining memory capacity.

[0011] Preferably, the RGB modality adopts a Top-K sparse compression strategy, wherein the K value is dynamically adjusted according to the gradient sparsity of the RGB modality, and the K value is the number of maximum gradients retained, and the calculation formula is: K = max(1, round(r·N)) where r is the gradient sparsity ratio and N is the total length of the current gradient vector;

[0012] The infrared mode uses an 8-bit quantization compression strategy, with a quantization level of 2 8 -1, and adaptively adjust the precision of quantization compression according to the gradient change and feature distribution. The calculation formula for dynamic adjustment of the quantization compression precision is: Among them, S is the scaling factor, G is the current gradient value, and L is the quantization level (2 8 -1).

[0013] Preferably, a physical constraint timing alignment loss is introduced, and a physical model based on the heat diffusion equation is used to perform timing alignment on multimodal data, including:

[0014] A heat diffusion equation constraint is introduced to optimize the temporal consistency between different modal data streams. The heat diffusion equation is: Where T is the temperature, α is the thermal diffusion coefficient, β is the optical noise coefficient, and I(x,y,t) is the image noise term. is the vector differential operator, It is the Laplace operator, which represents the second-order derivative of temperature T in space in the heat diffusion equation and describes the diffusion process of heat or data;

[0015] Image features are modified through an optical noise injection model to achieve temporal alignment of cross-modal data;

[0016] By calculating the feature similarity of different modal data, the cosine similarity formula is used: Where a and b represent the aligned feature vectors of RGB and infrared modalities, respectively, to optimize the temporal alignment of cross-modal data.

[0017] The heat diffusion equation and the optical noise injection model are further combined to introduce physical constraints in the timing alignment of multimodal data streams to enhance the accuracy of data alignment.

[0018] Preferably, the method of introducing physical constraint timing alignment includes:

[0019] The spatial Laplace difference calculation is performed on the cross-modal data to detect the temporal differences between the modes in the time series. The spatial Laplace operator is: Wherein, T(x,y,t) is the feature of the modal data, is the Laplace operator in the spatial dimension;

[0020] The delay compensation amount of the modal data is dynamically adjusted based on the results of Laplace operator calculation, and data synchronization is performed based on the delay difference between the modes.

[0021] An optical noise injection model is further adopted to simulate the noise effect in the image by adding the image noise term I(x, y, t).

[0022] Preferably, a memory-aware dynamic batch processing strategy is used to monitor the usage of the video memory in real time and dynamically adjust the batch size according to the remaining capacity of the video memory, including:

[0023] By monitoring the memory usage on each computing node in real time, the current batch size is dynamically adjusted to adapt to the computing requirements under different memory conditions. The memory-aware algorithm calculates the batch size B based on the memory usage U and the memory threshold T. The calculation formula is: Among them, S available is the currently available video memory, E per_batch For each batch of video memory requirements, Bmax is the maximum batch size;

[0024] The batch size is dynamically adjusted based on the video memory usage. If the video memory usage is close to the maximum threshold, the batch size is reduced; if the video memory usage is lower than the preset threshold, the batch size is increased. The video memory adjustment formula is: Among them, S used is the video memory currently used, S total is the total capacity of video memory, and U is the usage rate of video memory;

[0025] Further combined with the memory monitoring results, an adaptive algorithm is used to optimize the batch size at different training stages. The adaptive adjustment formula is: adjusted =B current ×(1-α·U), where B adjusted is the adjusted batch size, B current is the current batch size, α is the adjustment coefficient, and U is the memory usage.

[0026] The present invention also provides an adaptive gradient compression optimization system for distributed streaming multimodal fusion, which is used to implement the above method. The system includes: a modality-sensitive gradient compression module, a physical constraint timing alignment module, and a memory-aware dynamic batch processing module;

[0027] The modality-sensitive gradient compression module is used to adopt a modality-sensitive gradient compression strategy to perform personalized compression on data features of different modalities, wherein the modalities include RGB modality and infrared modality;

[0028] The physical constraint timing alignment module is used to introduce physical constraint timing alignment loss and perform timing alignment on the personalized compressed multimodal data using a physical model based on the heat diffusion equation;

[0029] The memory-aware dynamic batch processing module is used to monitor the usage of the video memory in real time through the memory-aware dynamic batch processing strategy, and dynamically adjust the batch size according to the remaining capacity of the video memory.

[0030] Preferably, the RGB modality adopts a Top-K sparse compression strategy, wherein the K value is dynamically adjusted according to the gradient sparsity of the RGB modality, and the K value is the number of maximum gradients retained, and the calculation formula is: K = max(1, round(r·N)) where r is the gradient sparsity ratio and N is the total length of the current gradient vector;

[0031] The infrared mode uses an 8-bit quantization compression strategy, with a quantization level of 2 8 -1, and adaptively adjust the precision of quantization compression according to the gradient change and feature distribution. The calculation formula for dynamic adjustment of the quantization compression precision is: Among them, S is the scaling factor, G is the current gradient value, and L is the quantization level (2 8 -1).

[0032] Preferably, the physical constraint timing alignment module includes: a first introduction unit, a correction unit, a calculation unit, and a second introduction unit;

[0033] The first introduction unit is used to introduce a heat diffusion equation constraint to optimize the timing consistency between different modal data streams. The heat diffusion equation is: Where T is the temperature, α is the thermal diffusion coefficient, β is the optical noise coefficient, and I(x,y,t) is the image noise term. is the vector differential operator, It is the Laplace operator, which represents the second-order derivative of temperature T in space in the heat diffusion equation and describes the diffusion process of heat or data;

[0034] The correction unit is used to correct the image features through the optical noise injection model to achieve temporal alignment of cross-modal data;

[0035] The calculation unit is used to calculate the feature similarity of different modal data using the cosine similarity formula: Where a and b represent the aligned feature vectors of RGB and infrared modalities, respectively, to optimize the temporal alignment of cross-modal data.

[0036] The second introducing unit is used to further combine the heat diffusion equation and the optical noise injection model to introduce physical constraints in the timing alignment of the multimodal data stream to enhance the accuracy of data alignment.

[0037] Preferably, the process of introducing physical constraint timing alignment includes:

[0038] The spatial Laplace difference calculation is performed on the cross-modal data to detect the temporal differences between the modes in the time series. The spatial Laplace operator is: Wherein, T(x,y,t) is the feature of the modal data, is the Laplace operator in the spatial dimension;

[0039] The delay compensation amount of the modal data is dynamically adjusted based on the results of Laplace operator calculation, and data synchronization is performed based on the delay difference between the modes.

[0040] An optical noise injection model is further adopted to simulate the noise effect in the image by adding an image noise term I(x, y, t).

[0041] Preferably, the memory-aware dynamic batch processing module includes: a first adjustment unit, a judgment unit, and a second adjustment unit;

[0042] The first adjustment unit is used to dynamically adjust the current batch size by monitoring the video memory usage on each computing node in real time to adapt to computing requirements under different video memory conditions. The video memory perception algorithm calculates the batch size B based on the video memory usage U and the video memory threshold T. The calculation formula is: Among them, S available is the currently available video memory, E per_batch For each batch of video memory requirements, B max is the maximum batch size;

[0043] The judgment unit is used to dynamically adjust the batch size according to the video memory usage. If the video memory usage is close to the maximum threshold, the batch size is reduced; if the video memory usage is lower than the preset threshold, the batch size is increased. The video memory adjustment formula is: Among them, S used is the video memory currently used, S total is the total capacity of video memory, and U is the usage rate of video memory;

[0044] The second adjustment unit is used to further combine the video memory monitoring results and adopt an adaptive algorithm to optimize and adjust the batch size at different training stages. The determination formula for adaptive adjustment is: B adjusted =B current ×(1-α·U), where B adjusted is the adjusted batch size,

[0045] B current is the current batch size, α is the adjustment coefficient, and U is the memory usage.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. Reduced communication overhead: A modality-sensitive gradient compression strategy reduces the communication bandwidth requirements during distributed training. In particular, when processing multimodal data, appropriate compression strategies are used for data of different modalities, further improving training efficiency.

[0048] 2. Improve timing alignment accuracy: By physically constraining the timing alignment loss, the timing alignment between different modal data is optimized, reducing training errors caused by inconsistent timestamps and improving the accuracy of multimodal data training.

[0049] 3. Optimize video memory usage: Through the memory-aware dynamic batching strategy, the batch size can be dynamically adjusted to optimize video memory utilization, avoid video memory overflow and resource waste, and ensure training stability and efficiency.

[0050] Among them, the combination of modality-sensitive gradient compression, physical constraint timing alignment loss and memory-aware dynamic batch processing enables the present invention to effectively improve the efficiency of multimodal data processing in distributed training, and provides a new streaming multimodal data training optimization method for the field of deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A schematic diagram of the composition of the adaptive gradient compression optimization method for distributed streaming multimodal fusion provided by an embodiment of the present invention;

[0053] Figure 2 This is a diagram of the overall architecture of the adaptive gradient compression optimization framework for distributed streaming multimodal fusion provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the working principle of the modal-sensitive gradient compression strategy provided by an embodiment of the present invention;

[0055] Figure 4 A diagram of the physical constraint timing modeling process provided by an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of the memory-aware dynamic batching mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Example 1

[0060] An embodiment of the present invention provides an adaptive gradient compression optimization method for distributed streaming multimodal fusion, the method comprising:

[0061] A modality-sensitive gradient compression strategy is used to perform personalized compression on data features of different modalities, including RGB and infrared, to reduce communication overhead and improve training efficiency.

[0062] Merge the compressed gradient data of different modalities and perform cross-modal gradient updates in a specific way, thereby effectively optimizing the communication efficiency during model training.

[0063] A physical constraint timing alignment loss is introduced to address the timing misalignment problem in multimodal data caused by differences in acquisition time. A physical model based on the heat diffusion equation is used to align data timing, ensuring that data from different modalities remain consistent in time during training.

[0064] Through the memory-aware dynamic batching strategy, video memory usage is monitored in real time, and the batch size is dynamically adjusted according to the remaining memory capacity, thereby avoiding video memory overflow and maximizing video memory utilization, ensuring stability and efficiency during training.

[0065] In this embodiment, the present invention combines a modality-sensitive gradient compression strategy, physically constrained timing alignment, and memory-aware dynamic batching technology to improve the training efficiency and accuracy of multimodal data streams;

[0066] Modality-sensitive gradient compression strategy: Based on the difference in gradient sparsity between different modalities (e.g., RGB and infrared), the adaptive gradient compression method is automatically selected to reduce the amount of data in cross-node communication, significantly reducing communication overhead while ensuring that training accuracy is not affected.

[0067] Physically constrained timing alignment: This method further optimizes the timing alignment process of cross-modal data streams by combining physical constrained timing alignment technology with the heat diffusion equation and optical noise injection model. This effectively resolves the timing mismatch problem in traditional methods, improves the timing consistency of data streams in different modalities, and enhances the training stability and accuracy of the model.

[0068] Memory-aware dynamic batching mechanism; further, through the memory-aware dynamic batching mechanism, the training batch size can be dynamically adjusted according to the real-time memory usage and GPU resource load, avoiding memory overflow or resource waste, and ensuring maximum utilization of hardware resources, thereby reducing training latency and improving overall training efficiency.

[0069] In this embodiment, by analyzing the difference in gradient sparsity of multimodal data, a Top-K sparse compression method is adopted for the RGB modality, wherein the K value is dynamically adjusted according to the gradient sparsity of the RGB modality. The K value is the number of maximum gradients retained. The specific calculation formula is: K = max(1, round(r·N)) where r is the gradient sparsity ratio and N is the total length of the current gradient vector, so as to minimize the amount of communication data and ensure the transmission of key gradient information; wherein the gradient sparsity ratio r is the following ρ

[0070]

[0071] Among them: significant gradients: refers to the gradient elements whose absolute value is greater than a certain threshold 0. Total number of gradients: the total length of the gradient vector, usually denoted by N.

[0072] Calculation method To make ρ adaptive, the threshold θ can be dynamically determined according to the statistical characteristics of the gradient. The specific calculation steps are as follows: Statistical gradient distribution: For the gradient vector g in the current training step = [g1, g2, ..., g N ], calculate its mean μ and standard deviation σ:

[0073] Determine the threshold θ: Set the threshold θ = μ + kσ, where k is a scaling factor (e.g., k = 1 or k = 2) that controls the sensitivity of the sparsity ratio. The specific value of k can be optimized experimentally and is typically between 1 and 2.

[0074] Calculate the number of important gradients: Count the number of elements in the gradient vector whose absolute value is greater than e: Where II(·) is an indicator function, which takes the value 1 when the condition is met and 0 otherwise.

[0075] Calculate ρ: By definition, calculate the sparsity ratio:

[0076] Applied to K value: In your patent, K value is calculated by the formula k = max(1, \round(p·N)), ensuring that K is at least 1 and dynamically adjusting the number of retained gradients according to p.

[0077] The infrared mode uses an 8-bit quantization compression method, where the quantization level is 2 8 -1, and adaptively adjust the precision of quantization compression according to the gradient change and feature distribution. The quantization compression precision is dynamically adjusted by the following formula: Among them, S is the scaling factor, G is the current gradient value, and L is the quantization level (28 -1) to reduce the accuracy loss in data transmission while maintaining the accuracy of model training;

[0078] The adaptive compression strategy is based on a dynamic gradient sparsity detection module, which calculates the sparsity and importance of the gradients in each training step in real time. By setting the compression rate and compression method, it ensures that the gradients retained after each gradient update can effectively guide model training. The sparsity calculation formula is: Among them, Sparse_rate is the sparse ratio, G i For each gradient element, is the compressed gradient element; wherein, the adaptive compression strategy optimizes communication efficiency by analyzing the gradient data of the RGB-T modality. Among them, the dynamic gradient sparsity detection module can calculate the gradient sparsity and importance in real time, and dynamically adjust the compression rate and method to meet the needs of multimodal training. In some embodiments, the module can selectively retain key gradient components according to the characteristics of the RGB and T modalities, thereby maintaining model performance while reducing communication overhead. The specific structure of the dynamic gradient sparsity detection module includes an input interface, a sparsity calculation unit, an importance evaluation unit, a decision unit and an output interface. These components work together, through real-time analysis and dynamic adjustment, to enable the adaptive compression strategy to achieve a balance between efficient communication and accurate training in RGB-T multimodal training. Because the subsequent compression strategy for RGB-T is basically determined, and since the compression strategy has been determined, the module can be mentioned as an optional component.

[0079] By combining Top-K sparse compression and 8-bit quantization, this embodiment can compare the gradient changes of different modalities in each training cycle and automatically select the most suitable compression strategy, thereby reducing communication overhead and maintaining high-precision training results of the model (RGB-T model). Automatic selection of the compression strategy is achieved through the following formula: Among them, Compression_efficiency is the compression efficiency, is the gradient after compression;

[0080] During training, the gradient compression rate and update frequency are automatically adjusted based on the gradient characteristics after each compression and the communication bandwidth limit to ensure the best balance between data transmission efficiency and training performance between different computing nodes. The update frequency is dynamically adjusted according to the following formula: Where B is the available bandwidth and T is the total amount of training data;

[0081] Based on the obtained gradient compression results, the gradient update rule of the model is further optimized, the compressed gradient is transmitted to each computing node, and the impact of the compressed gradient on the back propagation algorithm is minimized, thereby improving the global convergence speed and accuracy of the training. The optimization goal is achieved by the following formula:final =L initial +λ·Error(G compressed ), where L final is the optimized loss function, L initial is the initial loss function, λ is the adjustment coefficient, G compressed is the compressed gradient, Error(G compressed ) is the compression error.

[0082] The modality-sensitive gradient compression strategy further includes:

[0083] In each training cycle, the K value of Top-K sparse compression is dynamically adjusted to adapt to the changes of each gradient in the data stream and ensure that the amount of data transmitted after each gradient update is minimized while ensuring the effectiveness of the transmitted gradient information;

[0084] By monitoring the communication bandwidth and memory usage of each computing node, the parameters of Top-K sparse compression and 8-bit quantization compression are dynamically optimized to balance communication overhead and training accuracy.

[0085] When training multimodal data, the compression ratio of each modality is adjusted based on the characteristic differences of different modalities to ensure that the compression method of each modal data during training is highly matched with its characteristics, avoiding accuracy loss and over-compression;

[0086] Taking into account the communication bandwidth and latency between distributed nodes, and targeting the differences in gradient synchronization time between different computing nodes, the frequency and compression ratio of gradient transmission are dynamically adjusted to improve overall training efficiency and reduce latency in distributed training.

[0087] In this embodiment, merging the gradient compressed data of different modalities includes:

[0088] The specific steps for merging the compressed gradient data of the RGB modality and the infrared modality include: first, using Top-K sparse compression for the RGB modality gradient and 8-bit quantization compression for the infrared modality gradient on each worker node; then, transmitting the compressed data to the parameter server and decompressing and restoring it; finally, calculating the aggregated gradient g through the weighted aggregation strategy. agg =w RGB ·g RGB +w T ·g T , the weights are dynamically adjusted based on the gradient norm for cross-modal gradient updates.

[0089] In this embodiment, a physical constraint timing alignment loss is introduced, and a physical model based on the heat diffusion equation is used to perform timing alignment on multimodal data, including:

[0090] A heat diffusion equation constraint is introduced to optimize the temporal consistency between different modal data streams. The heat diffusion equation is: Where T is the temperature, α is the thermal diffusion coefficient, β is the optical noise coefficient, and I(x,y,t) is the image noise term. is the vector differential operator, The Laplace operator represents the second-order spatial derivative of temperature T in the heat diffusion equation, describing the diffusion of heat or data. α can be defined as a smoothing coefficient for intermodal time series differences, estimated based on the data sampling frequency or delay. β represents the weight of noise influence and can be determined experimentally. I(x,y,t) is defined as the specific form of sensor noise or environmental interference (e.g., Gaussian noise).

[0091] The optical noise injection model is used to correct image features, enhance the synchronization between different modal data, and ensure the temporal alignment of cross-modal data. The optical noise injection model introduces simulated optical noise to correct image features, thereby enhancing the synchronization between different modal data (such as RGB and thermal imaging) and ensuring the temporal alignment of cross-modal data.

[0092] By calculating the feature similarity of different modal data, the cosine similarity formula is used: Here, a and b represent the alignment feature vectors of the RGB and infrared modalities, respectively, optimizing the temporal alignment of cross-modal data. Furthermore, combining the heat diffusion equation and the optical noise injection model, physical constraints are introduced into the temporal alignment of multimodal data streams to enhance data alignment accuracy and mitigate the temporal misalignment issues encountered in traditional methods. The physical constraints are based on the physical laws of the heat diffusion equation and, in this embodiment, are specifically implemented as the spatial Laplacian operator.

[0093] In this embodiment, the method of introducing physical constraint timing alignment includes:

[0094] The spatial Laplace difference calculation is performed on the cross-modal data to accurately detect the temporal differences between the modes in the time series. The spatial Laplace operator is: Wherein, T(x,y,t) is the feature of the modal data, It is the Laplace operator in the spatial dimension, which can effectively capture the timing error between modes and correct it;

[0095] The delay compensation amount of the modal data is dynamically adjusted based on the results of the Laplace operator calculation, and data synchronization is performed based on the delay difference between the modalities to optimize the accuracy of timing alignment.

[0096] We further adopt an optical noise injection model to simulate the noise effect in the image by adding the image noise term I(x,y,t). This enhances the robustness of the timing alignment process, ensures that the noise impact is effectively handled, and reduces the inconsistency in timing alignment.

[0097] In this embodiment, a memory-aware dynamic batch processing strategy is used to monitor memory usage in real time and dynamically adjust the batch size based on the remaining memory capacity, including:

[0098] By monitoring the memory usage on each computing node in real time, the current batch size is dynamically adjusted to adapt to the computing requirements under different memory conditions. The memory-aware algorithm calculates the batch size B based on the memory usage U and the memory threshold T. The calculation formula is: Among them, S available is the currently available video memory, E per_batch For each batch of video memory requirements, B max is the maximum batch size;

[0099] The batch size is dynamically adjusted based on the video memory usage. If the video memory usage is close to the maximum threshold, the batch size is reduced; if the video memory usage is lower than the preset threshold, the batch size is increased. The video memory adjustment formula is: Among them, S used is the video memory currently used, S total is the total capacity of video memory, U is the usage rate of video memory;

[0100] Further combined with the memory monitoring results, an adaptive algorithm is used to optimize the batch size at different training stages. The adaptive adjustment formula is: B adjusted =B current ×(1-α·U), where B adjusted is the adjusted batch size, B current is the current batch size, α is the adjustment coefficient, and U is the memory usage.

[0101] The memory-aware dynamic batching mechanism supports cross-node collaborative optimization and performs global optimization based on the memory status of each computing node. This ensures that the memory of each node is reasonably allocated during distributed training, avoiding memory overload or inefficient utilization of a certain node.

[0102] Example 2

[0103] like Figure 1-Figure 5As shown, this embodiment details how to use the method of Example 1 and how to combine it with other networks. The method is not just a standalone optimization algorithm, but can be integrated with existing deep learning networks (such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs)) to jointly improve the efficiency and accuracy of multimodal data training.

[0104] The core technology of this invention lies in the use of technical modules such as modality-sensitive gradient compression, physically constrained timing alignment, and memory-aware dynamic batching. These modules are specifically designed to effectively improve the overall efficiency of streaming multimodal data training in a distributed training environment. The following describes the specific steps and framework module operations of this invention when used in conjunction with other networks.

[0105] The system requires two sensors: an RGB camera and a thermal imaging sensor (or an existing RGB-T dataset that meets the requirements of this article). The RGB camera captures conventional visible light images, while the thermal imaging sensor collects temperature-related information. Data from both sensors is synchronized to ensure that data acquired at the same time corresponds.

[0106] The system synchronizes the two data streams to ensure that data from both is processed at the same point in time. To achieve this, hardware synchronization devices or software mechanisms can be used to match timestamps.

[0107] Efficient bandwidth management is required during data transmission to prevent data asynchrony caused by packet loss or delay.

[0108] During the data collection process, the synchronization of timestamps is crucial for accurate matching of multimodal data. To ensure time alignment, this embodiment defines the timestamp of each data point as T RGB and T Thermal , the formula is as follows: T RGB =T Thermal That is, the timestamps of RGB images and thermal imaging images should be consistent during data acquisition to ensure that data at the same moment are processed synchronously.

[0109] Since the sensor sampling frequencies of RGB images and thermal imaging images may be different, this will lead to differences in the timestamps of the two modal data. Therefore, this embodiment requires time alignment. The core idea of the time alignment process is to align the data of the two modalities by calculating the timestamp difference between RGB and thermal imaging data. In this method, a heat diffusion model is used to deal with the timing alignment problem. This model corrects the time difference between the two data streams through physical equations to ensure that the data of different modalities are aligned in time to the greatest extent. For each pair of RGB and thermal imaging images, calculate their timestamp difference: ΔT = |T RGB-T Thermal |Where ΔT is the time difference between the RGB image and the thermal image.

[0110] If the time difference between the two exceeds a certain threshold, the timestamp needs to be corrected through the heat diffusion process. The heat diffusion model calculates a correction value based on the current time difference to ensure time alignment.

[0111] If the time difference is small, the images are synchronized directly to the same timestamp.

[0112] Then, the timing difference is corrected by using physical constraints (such as the heat diffusion equation) in the heat diffusion model. The model is calculated using the following formula: Among them, T i and T j is the timestamp of the RGB and thermal images, α is the thermal diffusion coefficient, ||T i -T j || represents the difference between timestamps. This formula calculates the square of the time difference and penalizes larger time differences to achieve the effect of optimizing timing alignment.

[0113] After time series alignment is complete, some basic data preprocessing is required, including denoising and normalization, to ensure that the input data meets the requirements of subsequent network training. For example, image filtering techniques can be used to remove noise, or thermal images can be linearly normalized to uniform their numerical range.

[0114] Through the above steps, RGB images and thermal imaging images can be accurately aligned in time, allowing the data of the two modalities to work together in subsequent processing, reducing the impact of timing errors on model training and target detection accuracy.

[0115] When training large network models, video memory management is crucial for deep learning training. This is especially true when processing multimodal data, where RGB and thermal images are large in size and computationally demanding. To avoid video memory overflow and improve training efficiency, the batch size must be dynamically adjusted based on video memory usage. The system monitors video memory usage in real time during training. The video memory monitoring module regularly checks the remaining and currently used video memory. This can be accomplished by accessing the graphics card's memory status through API calls, such as the cudaMemGetInfo() method in CUDA. Based on video memory usage, the system dynamically adjusts the batch size. When video memory usage approaches the upper limit, the system reduces the batch size to ensure training stability. When more video memory is available, the system increases the batch size to improve training efficiency.

[0116] The formula for adjusting the batch size is as follows:

[0117]

[0118] Among them, B old is the current batch size, B new is the adjusted batch size, MemoryFree is the remaining video memory, and MemoryUsage is the used video memory.

[0119] During the training process, by adjusting the batch size, the system can flexibly respond to computing requirements under different video memory states, maximize the utilization of computing resources, and avoid training interruptions or delays caused by insufficient video memory.

[0120] In addition to batch adjustments, the system also dynamically adjusts other network parameters based on memory fluctuations during training to adapt to changes in memory usage during training. For example, when memory is insufficient, the system may temporarily reduce computational precision (such as using 16-bit floating precision) to reduce memory usage. Through the memory management module, the system can adaptively adjust training parameters based on current memory usage, ensuring efficient use of memory during training, avoiding memory overflow issues, and improving the efficiency of computing resources.

[0121] Before inputting data into the object detection network, the system adjusts the batch size based on memory usage. Using a memory-aware dynamic batching strategy, the system dynamically adjusts the training batch size based on the remaining memory capacity, thus avoiding memory overflow or resource waste. During each training cycle, the system calculates the current memory usage (MemoryUsage) and the remaining memory (MemoryFree). The batch size is then adjusted according to the following formula: Among them, B old is the current batch size, B new is the adjusted batch size. This step ensures optimal use of video memory and efficient training.

[0122] Furthermore, the memory-aware dynamic batching mechanism:

[0123] 1. Video memory monitoring:

[0124] Function: Real-time monitoring of the video memory usage of each computing node.

[0125] Implementation: Use GPU management API (such as CUDA's cudaMemGetInfo()) to obtain the current video memory status, including the total capacity (M total ), used video memory (M used ) and remaining video memory (M free ).

[0126] Calculate video memory usage:

[0127] Frequency: Monitor once per training iteration or at a fixed time interval (such as every second).

[0128] Connection with RGB-T modality: RGB and infrared modalities have different data resolution and feature extraction requirements. Video memory monitoring needs to record the independent impact of each modality's data on the video memory.

[0129] 2. Historical record update, record the historical data of video memory usage, and provide a basis for subsequent predictions. Maintain a video memory usage history queue,

[0130] Stores the memory usage, batch size, and data load of the past several iterations. Data structure example: History = [(t1, R1, B1, D1), (t2, R2, B2, D2), ..., (tn, Rn, Bn, Dn)],

[0131] Among them, t i is the timestamp, R i is the video memory usage, B i is the batch size, D i The amount of data loaded.

[0132] Update frequency: New records are added after each iteration, and the queue length is fixed (such as the last 100 rounds).

[0133] 3. LSTM prediction model, based on historical video memory usage data, predicts video memory requirements in future training iterations.

[0134] Implementation: Input: Time series of the history queue (memory usage R, batch size B, data load D). Model structure: Single-layer or multi-layer LSTM network, input dimension is the feature vector ([R, B, D]) of each round, output is the predicted memory demand M for the next round pred Training objective: Minimize the mean square error between the predicted video memory demand and the actual demand. Training: During system initialization or pre-training, use historical data to train the LSTM model, and then update it online. Output: Predict the video memory demand M for the next iteration pred .

[0135] 4. Dynamic batch adjustment strategy, combining real-time monitoring and LSTM prediction results to dynamically adjust the batch size.

[0136] accomplish:

[0137] Rule adjustment (real-time feedback): If the current memory usage R is close to the maximum threshold (such as 90%), reduce the batch size: B new=B current ×(1-k×(RR target )).

[0138] If R is below a preset threshold (e.g. 50%), increase the batch size:

[0139] B new =min(B current ×(1+k×(R target -R)),B max ). k is the adjustment coefficient (such as 0.1), R target is the target utilization rate (e.g., 70%).

[0140] Prediction adjustment (forward-looking optimization): Based on the LSTM predicted memory demand M pred , pre-adjust batch size: Combined real-time and predicted results: B final =α×B new +(1-α)×B pred , where α (such as 0.7) is the weight adjusted in real time.

[0141] Cross-node collaboration: In a distributed environment, all nodes share memory state, and the central controller optimizes batch allocation based on global memory usage. Modality-specific: RGB and infrared modalities have different memory requirements, and adjustments can be made to independently calculate batch sizes for each modality.

[0142] 5. Load training data. Load training data according to the adjusted batch size to ensure efficient use of video memory. Based on the timing alignment results, prioritize loading aligned RGB and infrared data pairs.

[0143] During training, the system performs gradient compression based on the data characteristics of different modalities to reduce communication bandwidth requirements. For RGB image data, a Top-K sparse compression strategy is used to retain only the most important gradient parts. Specifically, the system calculates the number of retained gradients K according to the following formula: K = max(1, round(r·N)) where r is the gradient sparsity ratio and N is the length of the current gradient vector. 8-bit quantization compression is used for infrared data. This method quantizes the gradient value to 8 bits to reduce the amount of data transmission. The quantization formula is: Here, S is the compression accuracy, G is the current gradient value, and L is the quantization level. These compression methods are implemented by the Gradientcompressor module, which selects an appropriate compression strategy based on the gradient modality type, reducing the amount of gradient data that needs to be transmitted and improving communication efficiency during distributed training.

[0144] The processed RGB and thermal imaging data are input into the object detection network. Within the network, the RGB and thermal imaging data undergo feature extraction and sensitivity compression, respectively. Each branch performs gradient compression based on a specific compression strategy to ensure efficient data transmission and processing during the feature extraction process. During this step, the object detection network uses a convolutional neural network (CNN) or other similar deep learning models for feature extraction, obtaining mid- and deep-level features from the multimodal data.

[0145] Finally, the target detection results output by the network will be adjusted by the timing alignment correction module to ensure the timing consistency between multimodal data. Specifically, the output results are corrected using the physical constraint timing alignment loss to reduce the detection error caused by the timing deviation between the modalities. Through the formula: L final =L initial +λ·L align Among them, L final is the final detection loss, L initial is the initial loss, λ is the weight coefficient, L align is the physical constraint timing alignment loss.

[0146] When using the training optimization method of the present invention, modules such as the modality-sensitive gradient compression module, the physical constraint timing alignment module, and the memory-aware dynamic batch processing module are not used independently, but are integrated with existing deep learning networks (such as convolutional neural networks (CNN) or recurrent neural networks (RNN), etc.).

[0147] During integration, these optimization modules are first embedded into the original network training process through the NetworkIntegrator module. The NetworkIntegrator module is responsible for receiving the parameters from each optimization module and applying them to the network training process.

[0148] Training Process: The training process begins with a standard network forward propagation to calculate the loss, followed by backpropagation to update the gradient. At this point, the modality-sensitive gradient compression module compresses the calculated gradients to reduce data transmission. Next, the physically constrained timing alignment module temporally aligns cross-modal data to ensure data consistency. Finally, the memory-aware dynamic batching module adjusts the batch size based on memory usage to ensure training stability.

[0149] These modules work together with other layers of the deep learning network to ensure efficient and stable data processing during training.

[0150] In this invention, dynamic calculations within each module's formulas are used to optimize the training process. For example, in modality-sensitive gradient compression, the selection of Top-K compression strategies and 8-bit quantization compression strategies are based on the dynamic calculation results of gradients. In physically constrained timing alignment, the application of the heat diffusion equation ensures timing consistency. The memory-aware dynamic batching module dynamically optimizes batch size by monitoring memory usage.

[0151] When these optimization techniques provided by the present invention are integrated with existing network models, they can not only significantly improve training efficiency, but also enhance the accuracy and robustness of target detection models in complex environments (such as low light or extreme temperatures).

[0152] Example 3

[0153] The present invention also provides an adaptive gradient compression optimization system for distributed streaming multimodal fusion, which is used to implement the above method. The system includes: a modality-sensitive gradient compression module, a physical constraint timing alignment module, and a memory-aware dynamic batch processing module;

[0154] The modality-sensitive gradient compression module is used to perform personalized compression on data features of different modalities using a modality-sensitive gradient compression strategy. The modalities include RGB and infrared modalities.

[0155] The physical constraint timing alignment module is used to introduce physical constraint timing alignment loss and use a physical model based on the heat diffusion equation to perform timing alignment on personalized compressed multimodal data;

[0156] The memory-aware dynamic batch processing module is used to monitor the memory usage in real time through the memory-aware dynamic batch processing strategy and dynamically adjust the batch size according to the remaining memory capacity.

[0157] In this embodiment, the implementation process of the modality-sensitive gradient compression strategy is as follows: different compression algorithms are used for the data of each modality, where the RGB data adopts the Top-K sparse compression strategy, and the infrared data adopts the 8-bit quantization compression strategy. By retaining the part with larger gradient values, unnecessary data transmission is reduced, thereby optimizing training efficiency.

[0158] The RGB modality adopts the Top-K sparse compression strategy, where the K value is dynamically adjusted according to the gradient sparsity of the RGB modality. The K value is the number of maximum gradients retained, and the calculation formula is: K = max(1, round(r·N)) where r is the gradient sparsity ratio and N is the total length of the current gradient vector;

[0159] The infrared mode uses an 8-bit quantization compression strategy, with a quantization level of 2 8-1, and adaptively adjust the precision of quantization compression according to the gradient change and feature distribution. The calculation formula for dynamic adjustment of the quantization compression precision is: Among them, S is the scaling factor, G is the current gradient value, and L is the quantization level (2 8 -1).

[0160] In this embodiment, the physical constraint timing alignment loss is implemented as follows: a physical model is established based on the heat diffusion equation, data timestamps are smoothly adjusted through physical constraints, the timing alignment of cross-modal data is optimized, and the timing consistency during training is improved.

[0161] A physical constraint timing alignment module includes: a first introduction unit, a correction unit, a calculation unit, and a second introduction unit;

[0162] The first introduction unit is used to introduce a heat diffusion equation constraint to optimize the timing consistency between different modal data streams. The heat diffusion equation is: Where T is the temperature, α is the thermal diffusion coefficient, β is the optical noise coefficient, and I(x,y,t) is the image noise term. is the vector differential operator, It is the Laplace operator, which represents the second-order derivative of temperature T in space in the heat diffusion equation and describes the diffusion process of heat or data;

[0163] A correction unit, used to correct image features through an optical noise injection model to achieve temporal alignment of cross-modal data;

[0164] The calculation unit is used to calculate the feature similarity of different modal data using the cosine similarity formula: Where a and b represent the aligned feature vectors of RGB and infrared modalities, respectively, to optimize the temporal alignment of cross-modal data.

[0165] The second introduction unit is used to further combine the heat diffusion equation and the optical noise injection model to introduce physical constraints in the timing alignment of the multimodal data streams to enhance the accuracy of the data alignment.

[0166] In this embodiment, the process of introducing physical constraint timing alignment includes:

[0167] The spatial Laplace difference calculation is performed on the cross-modal data to detect the temporal differences between the modes in the time series. The spatial Laplace operator is: Wherein, T(x,y,t) is the feature of the modal data, is the Laplace operator in the spatial dimension;

[0168] The delay compensation amount of the modal data is dynamically adjusted based on the results of Laplace operator calculation, and data synchronization is performed based on the delay difference between the modes.

[0169] An optical noise injection model is further adopted to simulate the noise effect in the image by adding the image noise term I(x, y, t).

[0170] In this embodiment, the memory-aware dynamic batching strategy is implemented by dynamically adjusting the batch size by real-time monitoring of memory usage to ensure that the batch size is increased as much as possible without overflowing the memory, thereby improving computing efficiency and resource utilization.

[0171] A memory-aware dynamic batch processing module includes: a first adjustment unit, a judgment unit, and a second adjustment unit;

[0172] The first adjustment unit is used to dynamically adjust the current batch size by monitoring the memory usage on each computing node in real time to adapt to the computing requirements under different memory conditions. The memory perception algorithm calculates the batch size B based on the memory usage U and the memory threshold T. The calculation formula is: Among them, S available is the currently available video memory, E per_batch For each batch of video memory requirements, B max is the maximum batch size;

[0173] The judgment unit is used to dynamically adjust the batch size according to the video memory usage. If the video memory usage is close to the maximum threshold, the batch size is reduced; if the video memory usage is lower than the preset threshold, the batch size is increased. The video memory adjustment formula is: Among them, S used is the video memory currently used, S total is the total capacity of video memory, and U is the usage rate of video memory;

[0174] The second adjustment unit is used to further combine the video memory monitoring results and use an adaptive algorithm to optimize the batch size at different training stages. The adaptive adjustment determination formula is: B adjusted =B current ×(1-α·U), where B adjusted is the adjusted batch size, B current is the current batch size, α is the adjustment coefficient, and U is the memory usage.

[0175] The present invention combines physical constraints with adaptive gradient compression strategies to solve the problems of high communication overhead and cross-modal timing misalignment in traditional distributed training methods. In view of the difference in gradient sparsity between RGB and infrared modalities, a dynamic compression mechanism is designed to significantly reduce the communication burden and maintain model accuracy. In order to optimize the timing alignment of multimodal data, physical constraints based on the heat diffusion equation and an optical noise injection model are introduced to improve the timing consistency of streaming data and ensure the precise alignment of data of different modalities. In addition, combined with the dynamic batching mechanism of video memory awareness and the zero-copy data pipeline, the present invention can adjust the batch size in real time and optimize GPU utilization and computational efficiency. The present invention provides a new solution for the efficient training of distributed streaming multimodal data, improving the accuracy and resource utilization of multimodal data fusion tasks.

[0176] This invention improves the efficiency and accuracy of real-time multimodal data processing. It is primarily used in fields such as autonomous driving and industrial inspection, processing multimodal data such as RGB (visible light) and infrared data streams.

[0177] These technologies are particularly suitable for scenarios requiring real-time fusion of multimodal data or optimizing the efficiency of existing fusion networks for RGB-T modalities, particularly in environments with varying lighting conditions or extreme temperatures, where high accuracy and robustness of object detection models are required. This system addresses challenges commonly encountered in distributed training, such as communication overhead, timing alignment issues, and inefficient static resource allocation.

[0178] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. An adaptive gradient compression optimization method for distributed streaming multimodal fusion, characterized by: The method comprises: A modality-sensitive gradient compression strategy is used to perform personalized compression on data features of different modalities, including RGB and infrared modalities. A physical constraint timing alignment loss is introduced, and a physical model based on the heat diffusion equation is used to perform timing alignment on personalized compressed multimodal data. Through the memory-aware dynamic batch processing strategy, the memory usage is monitored in real time and the batch size is dynamically adjusted according to the remaining memory capacity.

2. The method according to claim 1, characterized in that The RGB modality adopts the Top-K sparse compression strategy, where the K value is dynamically adjusted according to the gradient sparsity of the RGB modality. The K value is the number of maximum gradients retained, and the calculation formula is: K = max(1, round(r·N)) where r is the gradient sparsity ratio and N is the total length of the current gradient vector; The infrared mode uses an 8-bit quantization compression strategy, with a quantization level of 2 8 -1, and adaptively adjust the precision of quantization compression according to the gradient change and feature distribution. The calculation formula for dynamic adjustment of the quantization compression precision is: Among them, S is the scaling factor, G is the current gradient value, and L is the quantization level (2 8 -1).

3. The method according to claim 1, characterized in that A physical constraint timing alignment loss is introduced, and a physical model based on the heat diffusion equation is used to perform timing alignment on multimodal data, including: A heat diffusion equation constraint is introduced to optimize the temporal consistency between different modal data streams. The heat diffusion equation is: Where T is the temperature, α is the thermal diffusion coefficient, β is the optical noise coefficient, and I(x,y,t) is the image noise term. is the vector differential operator, It is the Laplace operator, which represents the second-order derivative of temperature T in space in the heat diffusion equation and describes the diffusion process of heat or data; Image features are modified through an optical noise injection model to achieve temporal alignment of cross-modal data; By calculating the feature similarity of different modal data, the cosine similarity formula is used: Where a and b represent the aligned feature vectors of RGB and infrared modalities, respectively, to optimize the temporal alignment of cross-modal data. The heat diffusion equation and the optical noise injection model are further combined to introduce physical constraints in the timing alignment of multimodal data streams to enhance the accuracy of data alignment.

4. The method according to claim 3, characterized in that Methods for introducing physical constraint timing alignment include: The spatial Laplace difference calculation is performed on the cross-modal data to detect the temporal differences between the modes in the time series. The spatial Laplace operator is: Wherein, T(x,y,t) is the feature of the modal data, is the Laplace operator in the spatial dimension; The delay compensation amount of the modal data is dynamically adjusted based on the results of Laplace operator calculation, and data synchronization is performed based on the delay difference between the modes. An optical noise injection model is further adopted to simulate the noise effect in the image by adding the image noise term I(x, y, t).

5. The method according to claim 1, wherein Through the memory-aware dynamic batching strategy, real-time monitoring of memory usage is carried out, and the batch size is dynamically adjusted according to the remaining memory capacity, including: By monitoring the memory usage on each computing node in real time, the current batch size is dynamically adjusted to adapt to the computing requirements under different memory conditions. The memory-aware algorithm calculates the batch size B based on the memory usage U and the memory threshold T. The calculation formula is: Among them, S available is the currently available video memory, E per_batch For each batch of video memory requirements, B max is the maximum batch size; The batch size is dynamically adjusted based on the video memory usage. If the video memory usage is close to the maximum threshold, the batch size is reduced; if the video memory usage is lower than the preset threshold, the batch size is increased. The video memory adjustment formula is: Among them, S used is the video memory currently used, S total is the total capacity of video memory, U is the usage rate of video memory; Further combined with the memory monitoring results, an adaptive algorithm is used to optimize the batch size at different training stages. The adaptive adjustment formula is: adjusted =B current ×(1-α·U), where B adjusted is the adjusted batch size, B current is the current batch size, α is the adjustment coefficient, and U is the memory usage.

6. An adaptive gradient compression optimization system for distributed streaming multimodal fusion, the system being used to implement the method according to any one of claims 1 to 5, characterized in that: The system includes: a modality-sensitive gradient compression module, a physical constraint timing alignment module, and a memory-aware dynamic batch processing module; The modality-sensitive gradient compression module is used to adopt a modality-sensitive gradient compression strategy to perform personalized compression on data features of different modalities, wherein the modalities include RGB modality and infrared modality; The physical constraint timing alignment module is used to introduce physical constraint timing alignment loss and perform timing alignment on the personalized compressed multimodal data using a physical model based on the heat diffusion equation; The memory-aware dynamic batch processing module is used to monitor the usage of the video memory in real time through the memory-aware dynamic batch processing strategy, and dynamically adjust the batch size according to the remaining capacity of the video memory.

7. The system according to claim 6, characterized in that The RGB modality adopts the Top-K sparse compression strategy, where the K value is dynamically adjusted according to the gradient sparsity of the RGB modality. The K value is the number of maximum gradients retained, and the calculation formula is: K = max(1, round(r·N)) where r is the gradient sparsity ratio and N is the total length of the current gradient vector; The infrared mode uses an 8-bit quantization compression strategy, with a quantization level of 2 8 -1, and adaptively adjust the precision of quantization compression according to the gradient change and feature distribution. The calculation formula for dynamic adjustment of the quantization compression precision is: Among them, S is the scaling factor, G is the current gradient value, and L is the quantization level (2 8 -1).

8. The system according to claim 6, wherein: The physical constraint timing alignment module includes: a first introduction unit, a correction unit, a calculation unit, and a second introduction unit; The first introduction unit is used to introduce a heat diffusion equation constraint to optimize the timing consistency between different modal data streams. The heat diffusion equation is: Where T is the temperature, α is the thermal diffusion coefficient, β is the optical noise coefficient, and I(x,y,t) is the image noise term. is the vector differential operator, It is the Laplace operator, which represents the second-order derivative of temperature T in space in the heat diffusion equation and describes the diffusion process of heat or data; The correction unit is used to correct the image features through the optical noise injection model to achieve temporal alignment of cross-modal data; The calculation unit is used to calculate the feature similarity of different modal data using the cosine similarity formula: Where a and b represent the aligned feature vectors of RGB and infrared modalities, respectively, to optimize the temporal alignment of cross-modal data. The second introducing unit is used to further combine the heat diffusion equation and the optical noise injection model to introduce physical constraints in the timing alignment of the multimodal data stream to enhance the accuracy of data alignment.

9. The system according to claim 8, characterized in that The process of introducing physical constraint timing alignment includes: The spatial Laplace difference calculation is performed on the cross-modal data to detect the temporal differences between the modes in the time series. The spatial Laplace operator is: Wherein, T(x,y,t) is the feature of the modal data, is the Laplace operator in the spatial dimension; The delay compensation amount of the modal data is dynamically adjusted based on the results of Laplace operator calculation, and data synchronization is performed based on the delay difference between the modes. An optical noise injection model is further adopted to simulate the noise effect in the image by adding an image noise term I(x, y, t).

10. The system according to claim 6, wherein: The memory-aware dynamic batch processing module includes: a first adjustment unit, a judgment unit, and a second adjustment unit; The first adjustment unit is used to dynamically adjust the current batch size by monitoring the video memory usage on each computing node in real time to adapt to computing requirements under different video memory conditions. The video memory awareness algorithm calculates the batch size B based on the video memory usage U and the video memory threshold T. The calculation formula is: Among them, S available is the currently available video memory, E per_batch For each batch of video memory requirements, B max is the maximum batch size; The judgment unit is used to dynamically adjust the batch size according to the video memory usage. If the video memory usage is close to the maximum threshold, the batch size is reduced; if the video memory usage is lower than the preset threshold, the batch size is increased. The video memory adjustment formula is: Among them, S used is the video memory currently used, S total is the total capacity of video memory, U is the usage rate of video memory; The second adjustment unit is used to further combine the video memory monitoring results and adopt an adaptive algorithm to optimize and adjust the batch size at different training stages. The determination formula for adaptive adjustment is: B adjusted =B current ×(1-α·U), where B adjusted is the adjusted batch size, B current is the current batch size, α is the adjustment coefficient, and U is the memory usage.

Images