High-speed data recording method and system based on multi-bus heterogeneous fusion

Abstract

The invention relates to the technical field of data recording, and discloses a high-speed data recording method and system based on multi-bus heterogeneous fusion, and the high-speed data recording method based on multi-bus heterogeneous fusion comprises the steps: collecting real-time data characteristic parameters of each heterogeneous bus, and constructing a characteristic vector; extracting a time sequence mode of each bus feature vector; dynamically adjusting a cache resource allocation proportion of each bus based on a reinforcement learning technology; constructing a bus data feature knowledge graph by pre-training a basic strategy model under a plurality of working conditions; recognizing system working condition changes and triggering strategy switching; a two-stage cache architecture is realized, and the cache size of each bus is dynamically adjusted according to the cache resource allocation proportion; according to the method and the device, the technical problems that cache resource distribution is fixed in heterogeneous bus data recording, the adaptability to burst data streams is insufficient, and the performance is temporarily reduced during working condition switching are solved, and efficient cache resource utilization and stable data recording are realized.

Description

Technical Field

[0001] This invention relates to the field of data recording technology, and more specifically, to a method and system for high-speed data recording based on multi-bus heterogeneous fusion. Background Technology

[0002] In scenarios such as aerospace and testing of complex mechanical equipment, it is necessary to simultaneously record data from multiple different bus protocols (such as PCIe, USB, CAN, Ethernet, etc.) to comprehensively monitor system status and performance parameters. Various bus data differ significantly in terms of rate, format, and protocol, and their characteristics change markedly at different operational stages (such as takeoff, cruise, and landing).

[0003] Currently, common data logging systems typically employ a static cache allocation strategy, pre-allocating a fixed amount of cache resources to each bus interface. This method works correctly under stable data traffic conditions, but in practical applications, especially when high-speed buses such as PCIe and USB coexist with low-speed buses such as CAN and ARINC429, the following technical problems arise: Static cache allocation strategies cannot effectively handle sudden data flows. When a large amount of data is suddenly generated on a high-speed bus, insufficient cache resources can lead to overflow, while the cache resources of low-speed buses may be idle, resulting in unbalanced system resource utilization.

[0004] Existing systems lack the ability to effectively identify and adapt to the characteristics of heterogeneous bus data flows, and cannot dynamically adjust resource allocation strategies according to the real-time data characteristics of different buses. This results in fluctuations in system performance when data flow characteristics change, creating local performance bottlenecks.

[0005] In applications such as aerospace, systems typically undergo multiple distinct operational phases, each with significantly different data flow patterns. Existing systems lack sensitivity and adaptability to these operational condition transitions, failing to promptly identify changes and adjust caching strategies accordingly. This results in temporary performance degradation during operational condition transitions, impacting the continuity and integrity of data recording.

[0006] Furthermore, due to the lack of intelligent cache management strategies, the utilization rate of cache resources in existing systems is generally low. This not only wastes hardware resources but also indirectly limits the overall performance and data processing capabilities of the system.

[0007] Therefore, there is a need for a high-speed data recording method and system that can sense the characteristics of heterogeneous buses, adaptively adjust cache allocation strategies, and quickly adapt to changes in operating conditions, in order to meet the high reliability data recording requirements of modern complex environments. Summary of the Invention

[0008] This invention provides a method and system for high-speed data recording based on multi-bus heterogeneous fusion, which solves the technical problems of fixed buffer allocation, insufficient adaptability to data flow characteristics, and performance degradation when switching operating conditions in related technologies.

[0009] This invention provides a method for high-speed data recording based on multi-bus heterogeneous fusion, comprising the following steps: Collect real-time data feature parameters from each heterogeneous bus, construct feature vectors, and calculate the buffer overflow risk index; A temporal convolutional network model is used to extract the temporal patterns of the feature vectors of each bus. The temporal dependencies are captured by combining causal convolution and dilated convolution, and the feature representation is output. The cache allocation is modeled as a Markov decision process, and a deep reinforcement learning algorithm is used to train the intelligent agent model. The optimal cache allocation ratio is dynamically calculated based on a combined reward function of weighted throughput and overflow penalty. A basic cache allocation strategy model is trained for different working conditions. A modular structure is adopted to decompose the strategy into feature extraction and decision modules. The meta-learning algorithm is used to achieve rapid policy adaptation. Based on bus data flow features, operating condition feature vectors are extracted, unsupervised clustering is applied to identify changes in operating conditions, a sliding window is used for continuous monitoring, a smooth transition is achieved through strategy mixing, and the two-level cache architecture is dynamically adjusted.

[0010] In a preferred embodiment, the real-time data characteristic parameters collected from each heterogeneous bus include: data arrival rate, burst index, data priority, and historical cache utilization rate, wherein the burst index is obtained by calculating the ratio of short-term data traffic to long-term average traffic.

[0011] In a preferred embodiment, constructing the feature vector includes calculating a cache overflow risk index, which is obtained by the relationship between data arrival rate, burst index and current available cache capacity.

[0012] In a preferred embodiment, the step of extracting the temporal pattern of each bus feature vector using a temporal convolutional network model includes: capturing long-term and short-term temporal dependencies through a combination of causal convolution and dilated convolution, and outputting feature representations.

[0013] In a preferred embodiment, modeling cache allocation as a Markov decision process includes: The state space is defined as the characteristic representation of all buses, and the action space is defined as the buffer allocation ratio vector of each bus. The reward function is defined as a combination of weighted throughput and overflow penalty; A deep reinforcement learning algorithm is used to train an intelligent agent model, and the optimal cache allocation ratio vector is calculated based on the trained model.

[0014] In a preferred embodiment, the reward function includes two parts: bus throughput and overflow rate. The bus throughput is weighted and summed according to the weight of each bus, and the overflow rate is penalized by a penalty coefficient. The final reward value is the weighted throughput minus the overflow penalty.

[0015] In a preferred embodiment, training a basic cache allocation strategy model for different operating conditions includes: Train corresponding basic cache allocation strategy models for different operating conditions; The strategy is decomposed into a feature extraction module and a decision module using a modular structure, wherein the feature extraction module shares parameters across operating conditions; A model-independent meta-learning algorithm is used to train the meta-policy model, enabling the meta-policy model to quickly adapt to new working conditions with a small number of samples.

[0016] In a preferred embodiment, the extraction of the operating condition feature vector based on bus data stream features includes: Based on the statistical characteristics and timing patterns of the bus data stream, extract the operating condition feature vector; An unsupervised clustering algorithm is applied to cluster the operating condition feature vectors to identify the current operating condition category of the system; A sliding time window method is used to continuously monitor changes in the operating condition feature vector. When a change in the operating condition is detected, a strategy hybrid method is used to achieve a smooth transition.

[0017] In a preferred embodiment, the strategy mixing method includes: During the switching of operating conditions, the mixing ratio of the old and new strategies is dynamically adjusted according to the time progress, so that the weight of the old and new strategies transitions smoothly over time. The weight of the old strategy gradually decreases from 1 to 0, while the weight of the new strategy gradually increases from 0 to 1, thus achieving a smooth switching of strategies.

[0018] In a preferred embodiment, a system based on multi-bus heterogeneous fusion high-speed data recording is used to perform a method for multi-bus heterogeneous fusion high-speed data recording, including: The bus interface and feature acquisition module are used to connect with various heterogeneous buses and receive data streams, monitor the data characteristics of each bus in real time, collect data arrival rate, burst index, data priority and historical cache usage, construct feature vectors and calculate the cache overflow risk index. The temporal feature extraction module is used to extract the temporal patterns of each bus feature vector using a temporal convolutional network model. It captures temporal dependencies by combining causal convolution and dilated convolution and outputs feature representations. The reinforcement learning cache allocation module is used to model cache allocation as a Markov decision process, train the intelligent agent model with deep reinforcement learning algorithm, and dynamically calculate the optimal cache allocation ratio based on the combined reward function of weighted throughput and overflow penalty. The strategy training and transfer module is used to train the basic cache allocation strategy model for different working conditions. It adopts a modular structure to decompose the strategy into feature extraction and decision modules, and realizes rapid policy adaptation through meta-learning algorithm. The operating condition identification and cache management module extracts operating condition feature vectors based on bus data flow features, applies unsupervised clustering to identify operating condition changes, uses a sliding window for continuous monitoring, achieves smooth transitions through strategy mixing, and dynamically adjusts the two-level cache architecture.

[0019] The beneficial effects of this invention are as follows: High-efficiency resource utilization: Through an adaptive cache allocation algorithm, the system achieves dynamic optimization of cache resource allocation, significantly improving system cache utilization and greatly reducing resource waste.

[0020] Reduced overflow risk: The cache allocation strategy based on bus characteristic self-awareness and reinforcement learning greatly reduces the probability of high-speed bus overflow, significantly improves data integrity, and ensures that critical data is not lost.

[0021] System performance improvement: Through optimized cache resource allocation, the overall data throughput of the system is significantly improved, meeting the needs of high-speed data recording and enabling the system to process multiple heterogeneous bus data simultaneously without bottlenecks.

[0022] Rapid environmental adaptation: By utilizing knowledge transfer and rapid adaptation technologies, the strategy adaptation time during operating condition switching is significantly shortened, ensuring stable recording throughout the entire process and avoiding performance degradation during operating condition switching.

[0023] Fully automated operation: Through operating condition identification and strategy switching technology, the system can automatically detect changes in operating conditions and optimize caching strategies without manual intervention, reducing operational complexity and improving the system's autonomy and stability. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method for high-speed data recording based on multi-bus heterogeneous fusion according to the present invention; Figure 2 This is a bar chart comparing the data load characteristics of different buses according to the present invention; Figure 3 This is a line graph illustrating the reinforcement learning-based cache allocation training process of this invention. Figure 4 This is a line graph comparing the speed of strategy adaptation when the operating conditions change according to the present invention; Figure 5This is a bar chart comparing cache utilization under different operating conditions of the present invention; Figure 6 This is an area graph comparing overflow probabilities under the hierarchical caching architecture of this invention; Figure 7 This is a radar chart comparing the knowledge transfer effects of bus data features in this invention. Detailed Implementation

[0025] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0026] At least one embodiment of the present invention discloses a method for high-speed data recording based on multi-bus heterogeneous fusion, such as... Figure 1 As shown, it includes the following steps: Step 1: Collect real-time data feature parameters of each heterogeneous bus, construct feature vectors and calculate the buffer overflow risk index; Specifically, it includes the following sub-steps: Step 1.1, Bus data feature acquisition; Collect real-time data characteristic parameters from each heterogeneous bus, including data arrival rate. Suddenness Index Data priority and historical cache usage rate .in, Indicates the bus number. Indicates the current time point.

[0027] Data arrival rate The calculation is as follows: ; in, Indicates bus At the point of time Data arrival rate; Indicates a point in time Time bus Total amount of data received; Indicates a point in time Time bus Total amount of data received; The sampling time interval; This is the bus number, used to identify different bus interfaces.

[0028] Suddenness Index This is obtained by calculating the ratio of short-term data flow to long-term average flow: ; in, Indicates bus At the point of time The burst index is used to measure the degree of fluctuation in data traffic; Indicates bus The average data arrival rate within a short time window (e.g., 100ms) is used to reflect the instantaneous data flow. Indicates bus The average data arrival rate within a long time window (e.g., 10 seconds) is used to reflect steady-state data flow; when When, it indicates that the current data traffic is higher than the average level and there is a sudden surge; when When the current data traffic is below average, it indicates that the current data traffic is below average.

[0029] Step 1.2, Feature vector construction; According to embodiments of this application, a bus is constructed based on the acquired original feature parameters. At the point of time eigenvectors : ; in, Indicates bus At the point of time eigenvectors; For bus At the point of time Data arrival rate, representing the amount of data received per unit of time; For bus At the point of time The burst index is used to measure the degree of fluctuation in data traffic; For bus The data priority is used to indicate the importance of the data on the bus. For bus The historical cache utilization rate reflects the historical usage of cache resources by the bus. The buffer overflow risk index is calculated using the following formula: ; in, Indicates bus At the point of time The cache overflow risk index is used to assess the likelihood of a cache overflow. Indicates bus At the point of time Data arrival rate; Indicates bus At the point of time The burst index is used to measure the degree of fluctuation in data traffic; Indicates bus At the point of time Total allocated cache capacity; Indicates bus At the point of time The amount of cache already used; Indicates bus At the point of time The remaining available cache capacity.

[0030] This formula shows that the risk of cache overflow is directly proportional to the data arrival rate and the burstiness index, and inversely proportional to the remaining cache capacity. The risk of overflow is higher when data traffic is large and highly bursty, while the remaining cache is small.

[0031] Step 1.3, Feature temporal pattern extraction; A Temporal Convolutional Network (TCN) model is applied to extract the temporal patterns of feature vectors from each bus. It should be noted that the TCN model, through a combination of causal convolution and dilated convolution, can effectively capture long- and short-term temporal dependencies and extract the time-varying patterns of the bus data stream. The input to the TCN is a time window. The sequence of feature vectors within the input is used as the output to represent the feature representation of the bus data timing pattern. .

[0032] The convolutional layer of a TCN is defined as follows: ; in, Indicates bus At the point of time The Output feature representation of a layered TCN network; Indicates bus At the point of time The The output feature representation of the layer TCN network is used as the input of the current layer; Indicates the TCN network's... The convolution kernel parameter matrix of the layer is used to extract temporal features; This indicates a causal convolution operation, ensuring that the output at the current time step depends only on historical information; Indicates the TCN network's... The layer's bias parameter vector is used to adjust the feature mapping; This represents a non-linear activation function (such as ReLU) used to enhance the expressive power of the network.

[0033] like Figure 2 As shown, the differences between four main buses (PCIe, Ethernet, CAN, and ARINC429) in terms of data rate, burstiness, and periodicity are illustrated to visually demonstrate the challenges faced by heterogeneous bus data logging.

[0034] Step 2: Use a temporal convolutional network model to extract the temporal patterns of the feature vectors of each bus, capture the temporal dependencies by combining causal convolution and dilated convolution, and output the feature representation; Specifically, it includes the following sub-steps: Step 2.1, Reinforcement learning environment modeling; The cache allocation problem is modeled as a Markov Decision Process (MDP), defined as follows: state space The current state of the system is represented by the characteristics of all buses. Composition, in which, , , They represent the first , , Each bus at a given time point Temporal characteristics representation; This indicates the total number of buses in the system.

[0035] Action space Actions are defined as the cache allocation ratio vectors for each bus. ,in, , , They respectively represent the assignments to the first , , The cache ratio of each bus; This indicates the total number of buses in the system.

[0036] reward function Defined as a combination of weighted throughput and overflow penalty: ; in, Indicates at a point in time The system reward value; This indicates the total number of buses in the system; Indicates bus At the point of time Throughput, which is the amount of data successfully transmitted per unit of time; Indicates bus At the point of time The overflow rate is the ratio of the amount of data lost due to insufficient cache to the total amount of data. Indicates bus The weighting coefficients are determined based on data priority; the higher the priority, the greater the weight, which is used to balance the importance of different buses. This represents the overflow penalty coefficient, which is used to adjust the system's sensitivity to data overflow. The larger the coefficient, the more the system values ​​avoiding data overflow.

[0037] Step 2.2, Deep reinforcement learning model training; The intelligent agent model is trained using the Deep Deterministic Policy Gradient (DDPG) algorithm. DDPG is a reinforcement learning algorithm suitable for continuous action spaces, combining the advantages of deep Q-networks and policy gradient methods.

[0038] DDPG includes Actor networks and Critic networks: Actor Network: Directly outputs action values ​​(buffer allocation ratio), the network structure is a multilayer perceptron, and the input is the state. The output is an action. .

[0039] Critic Network: Evaluates the value of state-action pairs, with state as input. and actions The output is the Q value. .

[0040] The network parameters are optimized using the following loss function: Critic loss: ; ; in, The loss function of the Critic network is represented by . These represent the parameters of the Critic network; Indicates in the parameter Below, regarding the state and actions Q-value estimation; Indicates the target Q value; Indicates an immediate reward; This represents a discount factor used to balance the importance of immediate rewards and future rewards; Represents the target Critic network; Indicates the next state; Represents the target Actor network; Represents the parameters of the target Actor network; These represent the parameters of the target Critic network; This indicates the expected operation.

[0041] Actor loss: ; in, Represents the Actor network parameters For the objective function The gradient; This represents the expectation operation; The Q-value function represents the action. The gradient; Indicates the current state; Indicates the state of the Actor network. The action to be output; This represents the gradient of the Actor network output with respect to its parameters; These represent the parameters of the Critic network; Represents the parameters of the Actor network; These represent the parameters of the target Critic network; Represents the parameters of the target Actor network; This represents the discount factor, used to balance the importance of immediate rewards and future rewards, and its value ranges from [0, 1].

[0042] In some implementations, in addition to the DDPG algorithm, the Proximal Policy Optimization (PPO) algorithm or the Soft Actor-Critic (SAC) algorithm can also be used to implement reinforcement learning training.

[0043] The PPO algorithm improves training stability by introducing a trust region constraint to limit the magnitude of each policy update. The SAC algorithm introduces an entropy regularization term to encourage exploration and makes it more robust in strategy selection.

[0044] Different algorithms are suitable for system environments of different complexity and scale.

[0045] To accelerate the training process, the embodiments of this application also employ a priority experience playback technique, which samples the experience based on the absolute value of the time-series difference error, focusing on learning those experiences with larger time-series difference errors, thereby improving learning efficiency.

[0046] Step 2.3: Dynamically allocate cache resources; Based on the trained DDPG model, at each decision time point Calculate the optimal cache allocation ratio vector based on the current system state, and then allocate the total system cache... Distribute them to each bus according to this ratio: ; in, Indicates at a point in time Assigned to the bus Cache capacity; Indicates at a point in time Assigned to the bus The cache ratio, with a value range of [0, 1]; This indicates the total cache capacity of the system.

[0047] Cache allocation operations are performed through the cache manager, ensuring a smooth adjustment of the size of each bus cache without interrupting the data flow.

[0048] In some implementations, an allocation threshold mechanism can be introduced to avoid increased system overhead caused by frequent cache adjustments.

[0049] Specifically, only when the difference between the newly calculated cache allocation ratio and the current allocation ratio exceeds a preset threshold... The actual cache adjustment operation will only be performed at that time. ; in, Indicates the current time point Assigned to the bus The cache ratio; Indicates the previous time point Assigned to the bus The cache ratio; This indicates the preset cache allocation ratio adjustment threshold, used to control the frequency of cache adjustments; This represents the absolute value of the difference between the cache allocation ratio at the current time point and the previous time point.

[0050] like Figure 3 As shown, the trends of reward value and system throughput during the training of the reinforcement learning intelligent agent model with the number of training rounds are demonstrated, proving the convergence and effectiveness of the adaptive cache allocation algorithm of this patent.

[0051] Step 3: Model cache allocation as a Markov decision process, train the intelligent agent model using a deep reinforcement learning algorithm, and dynamically calculate the optimal cache allocation ratio based on a combined reward function of weighted throughput and overflow penalty. Specifically, it includes the following sub-steps: Step 3.1, Multi-condition basic strategy training; For different operating conditions (such as takeoff, cruise, and landing), corresponding basic buffer allocation strategy models are trained separately. It should be understood that the input of each operating condition model is the bus characteristic representation under that operating condition, and the output is the optimal buffer allocation strategy.

[0052] The basic strategy model adopts a modular structure, including a feature extraction module. and decision module : ; in, Indicates working condition Below, input status The output of the strategy model at that time; Indicates the input system status; Represents the function mapping of the feature extraction module; This represents the parameters of the feature extraction module, which are shared across different operating conditions; Represents the function mapping of the decision module; Indicates working conditions The corresponding decision module parameters; An index representing the operating condition.

[0053] Step 3.2, Construction of bus data feature knowledge graph; According to embodiments of this application, a bus data feature knowledge graph is constructed based on bus data features under multiple operating conditions. Among them, vertex set Feature representations of different bus data modes, edge sets This represents the transformation relationship between different data patterns, and the weight of the edge represents the transformation probability.

[0054] Knowledge graphs are constructed through the following steps: Clustering of bus feature representations under various operating conditions yields a set of representative data pattern feature vectors, which serve as vertices of the knowledge graph. Analyze the transformation sequences between data patterns in historical data, count the transformation frequency, and establish weighted directed edges; The PageRank algorithm is applied to calculate the importance score of each vertex, which is used for model selection in the subsequent rapid adaptation process.

[0055] Step 3.3, training the meta-learning model; In the embodiments provided in this application, a model-independent meta-learning (MAML) algorithm is used to train a meta-policy model, enabling it to quickly adapt to new working conditions using a small number of samples. Furthermore, the goal of MAML is to find a set of initial parameters. This makes it possible to handle any new working condition. Good working condition-specific parameters can be obtained with only a small number of gradient updates. .

[0056] The optimization objective of MAML is defined as: ; in, Indicates the initial parameters of the model; An index representing operating conditions; Indicates working condition On the data of parameters The parameters after one or more gradient updates; Indicates working conditions The loss function is used to measure the model's performance under this operating condition; Indicates the parameter Perform optimization to minimize the total loss; Indicates all operating conditions The losses are summed.

[0057] Step 3.4: The strategy adapts quickly to changes in operating conditions; When a change in operating conditions is detected, the system performs the following steps to achieve rapid policy adaptation: Retrieve the set of vertices in the knowledge graph that are most similar to the current bus data pattern; Based on the importance score of the vertex, the most relevant pre-trained base policy model is selected as the initial policy; Using a small amount of sample data collected under current operating conditions, and based on a meta-learning model, only the parameters of the decision module are analyzed. Make minor adjustments: ; in, This represents the updated model parameters; Indicates the initial parameters of the model; This represents the learning rate, which controls the step size for parameter updates. Indicates the parameter Find the gradient; Represents the loss function; This represents the sample dataset collected under the current operating conditions.

[0058] The updated strategy model will be applied to cache allocation decisions for the current operating conditions.

[0059] In this way, the system provided by this application can quickly adapt to new data flow characteristics when operating conditions change, reducing the strategy adaptation time from minutes to seconds, thus ensuring stable recording throughout the entire process.

[0060] like Figure 4 As shown, the comparison of the knowledge transfer and rapid adaptation mechanism of this patent with traditional methods demonstrates the speed of strategy adaptation when working conditions change, showcasing the advantages of this patented technology in rapidly adapting to environmental changes.

[0061] Step 4: Train a basic cache allocation strategy model for different working conditions. Use a modular structure to decompose the strategy into feature extraction and decision modules, and use a meta-learning algorithm to achieve rapid strategy adaptation. Specifically, it includes the following sub-steps: Step 4.1, Extraction and Recognition of Operating Condition Features; Based on the statistical characteristics and timing patterns of the bus data flow, extract the operating condition feature vector. : ; in, Represents the characteristic vector of the working condition; This represents the average data arrival rate at the system level, used to measure the amount of data arriving at the system per unit of time. This represents the mean burst index, used to quantify the burstiness of a data stream; a larger value indicates that the data arrival is more uneven. This indicates the priority distribution, describing the proportion of data with different priorities in the total data stream; This represents the overflow risk index, reflecting the likelihood of system cache overflow, and is related to the amount of data and processing capacity. The correlation matrix represents the data flow between buses, used to describe the degree of correlation between different bus data flows. Each element in the matrix... Indicates bus and bus The correlation coefficient between data traffic.

[0062] Unsupervised clustering algorithms (such as DBSCAN) are applied to cluster the feature vectors of operating conditions to identify the current operating condition category of the system. It should be noted that the parameters of the DBSCAN algorithm are set as follows: the radius of the neighborhood of the core point. The minimum number of samples in the neighborhood of the core point is adaptively determined based on the distribution of historical data. Set to 5% of the total sample size.

[0063] In some implementations, besides the DBSCAN algorithm, other clustering methods can be used to identify operating conditions, such as k-means clustering, hierarchical clustering, or Gaussian mixture models (GMM). Among these, the k-means algorithm is computationally simple but requires pre-specifying the number of categories; hierarchical clustering can construct a hierarchical structure of operating conditions and is suitable for multi-level operating condition scenarios; GMM can handle overlapping probability distributions of operating condition features, providing the probability that each data point belongs to different operating conditions. Different methods can be selected based on the characteristics of the actual application scenario.

[0064] In addition, in some other implementations, supervised learning methods can be combined for work condition identification. A classifier (such as a support vector machine, random forest, or deep neural network) can be trained using pre-labeled work condition data to achieve more accurate work condition identification. Once the system has accumulated enough labeled data, it can smoothly transition from unsupervised to supervised methods, improving identification accuracy.

[0065] Step 4.2, Detection of changes in operating conditions; A sliding time window method is used to continuously monitor the operating condition feature vector. The change in operating condition is determined when the following conditions are detected, according to an embodiment of this application: continuous Each time point ( The feature vectors of the working conditions (default value 5) are clustered into new categories; The distance between the working condition feature vector and the center of the current working condition category exceeds a preset threshold. : ; in, This represents the working condition feature vector at the current time t, which includes multiple feature dimensions such as data arrival rate and suddenness index. The center vector representing the current working condition category is calculated from the mean of the feature vectors of all samples within that category. This represents the vector norm, used to calculate the distance between two vectors; This represents the judgment threshold, which is adaptively set based on the distribution of historical data and is used to determine whether the feature vector deviates too far from the current working condition category.

[0066] In some implementations, to improve the robustness of operating condition change detection, the system also employs change point detection algorithms (such as CUSUM, PELT, or Bayesian change point detection). By analyzing the statistical characteristics of changes in the time series of operating condition features, these algorithms can more reliably detect operating condition transition points. These algorithms can effectively distinguish between real operating condition changes and random fluctuations, reducing the false alarm rate.

[0067] Step 4.3, smooth strategy switching; To avoid performance fluctuations caused by policy switching, the method provided in this application employs a policy hybrid approach to achieve a smooth transition: ; in, Indicates the state and time The hybrid strategy is used to achieve a smooth transition between the old and new strategies; Indicates the state under the old operating conditions. The cache allocation strategy, that is, the strategy originally used by the system; Indicates the state under the new operating condition. The cache allocation strategy, i.e. the strategy that the system will switch to; This represents the time-dependent mixing coefficient, which controls the mixing ratio of the old and new strategies. It decays linearly from 1 to 0 over time, with a default decay time of 2 seconds. It indicates the system status, including information such as the current data flow characteristics of each bus and cache usage; This indicates the time elapsed since the start of the policy switch. In some implementations, the mixing coefficient Nonlinear decay functions, such as exponential decay or S-shaped decay, can be used to better adapt to the characteristics of different operating conditions.

[0068] Specifically, for operating conditions requiring rapid switching, an exponential function with a faster initial decay can be used; for operating conditions requiring gradual switching, an S-shaped function with a faster change in the middle stage can be used.

[0069] In addition, in other implementations, policy switching can also be adaptively adjusted based on the reliability of operating condition transitions.

[0070] When the confidence level of the operating condition change detection is high, a shorter strategy switching time is adopted; When confidence is low, extend the switching time to reduce unnecessary policy oscillations caused by misjudgments.

[0071] Confidence can be calculated based on a combination of factors, such as the distance between the feature vector of the working condition and the center of the new working condition category, and the number of time points in which the new working condition is continuously classified.

[0072] like Figure 5 As shown, the buffer utilization rates of the traditional fixed allocation method and the adaptive allocation method of this patent are compared under different operating conditions (takeoff, cruise, acceleration, and landing), verifying the technical effect of this patent in improving resource utilization efficiency.

[0073] Step 5: Extract operating condition feature vectors based on bus data flow features, apply unsupervised clustering to identify changes in operating conditions, use a sliding window for continuous monitoring, achieve smooth transition through strategy mixing, and dynamically adjust the two-level cache architecture. Specifically, it includes the following sub-steps: Step 5.1, cache hierarchy design; The two-level caching architecture includes: L1 level: Allocate an independent dedicated buffer for each bus interface, with dynamically variable size to meet short-term burst requirements; L2 level: The system shares a total cache pool, which is dynamically allocated to each bus according to the allocation strategy in step 2.

[0074] In addition, cache management uses virtual memory mapping technology to support dynamic adjustment of cache size while maintaining the efficiency of physical memory access.

[0075] Step 5.2, High-speed data channel configuration; According to embodiments of this application, a high-speed data channel is configured between the bus interface and the cache, employing zero-copy technology to reduce data transfer overhead. For high-speed buses such as PCIe, a direct memory access (DMA) channel is configured to support direct data writing to the system cache, bypassing CPU processing.

[0076] Data channel configuration parameters are automatically optimized based on bus type, including DMA transfer block size, interrupt threshold, and batch processing strategy, to maximize data transfer efficiency.

[0077] Step 5.3, Cache management unit implementation; A cache management unit is implemented, responsible for executing cache allocation strategies and monitoring cache usage. According to embodiments of this application, the cache management unit includes the following functional modules: Cache allocation executor: dynamically adjusts the cache size of each bus according to the allocation decision in step 2; Cache usage monitor: Real-time monitoring of cache usage and overflow risk on each bus; Emergency Handler: When an impending cache overflow is detected, an emergency handling mechanism is triggered, including measures such as temporarily expanding the cache and reducing data priority.

[0078] like Figure 6 As shown, the data overflow probability of traditional single-level cache and the hierarchical cache architecture of this patent is compared under different bus data load conditions, demonstrating the advantages of this patented technology in improving data integrity.

[0079] Through the above steps, the implementation method provided in this application realizes a high-speed data recording method based on multi-bus heterogeneous fusion, which can sense the data load characteristics of different buses, adaptively adjust the cache allocation strategy, quickly adapt to changes in working conditions, and ensure stable recording throughout the process. Therefore, it effectively solves the technical problems existing in the prior art.

[0080] like Figure 7 As shown, the performance of three different methods (meta-learning, fine-tuning the base model, and retraining the model) in five dimensions—rapid adaptability, stability, adaptation speed, model generalization ability, and resource utilization—is compared, intuitively demonstrating the advantages of the knowledge transfer and rapid adaptation module of this patent.

[0081] Implementation example of this system: This application also provides a system for high-speed data recording based on multi-bus heterogeneous fusion, the system comprising: Bus interface module: Connects to various heterogeneous buses (such as PCIe, USB, CAN, Ethernet, etc.), receives data streams from each bus, and transmits the data to the cache system.

[0082] Bus characteristic self-sensing module: Real-time monitoring of the data characteristics of each bus, collection of parameters such as data arrival rate and burst index, construction of feature vectors, extraction of timing patterns, and output of feature representations characterizing the timing characteristics of bus data.

[0083] Adaptive cache allocation module: Based on a reinforcement learning model, it dynamically calculates the optimal cache allocation ratio according to the feature representation output by the bus characteristic self-aware module, and controls the cache management unit to perform resource allocation.

[0084] Knowledge transfer and rapid adaptation module: Pre-train basic policy models under multiple operating conditions, construct a bus data feature knowledge graph, and achieve rapid policy adaptation through meta-learning methods.

[0085] Operating condition identification and strategy switching module: Based on the characteristics of bus data flow, it automatically identifies changes in system operating conditions, triggers strategy switching, and uses a strategy hybrid method to achieve a smooth transition.

[0086] Tiered caching architecture module: Provides a two-level caching structure, including a bus-dedicated L1 cache and a system-shared L2 cache, to achieve efficient cache resource management.

[0087] Data processing and storage module: Receives data from the cache, performs necessary format conversions, compression, and other processing, and stores the processed data to a persistent storage device.

[0088] It should be understood that the connection relationships and data flows between the modules are as follows: The bus interface module receives external bus data and introduces the data stream into the system.

[0089] The bus characteristic self-sensing module monitors the data flow characteristics of the bus interface module in real time and outputs the characteristic representation to the adaptive buffer allocation module and the operating condition identification and strategy switching module.

[0090] The operating condition identification and strategy switching module, based on bus characteristics, identifies changes in operating conditions and controls the strategy selection of the adaptive cache allocation module and the knowledge transfer and rapid adaptation module.

[0091] The adaptive cache allocation module calculates the cache allocation ratio based on the current strategy and controls the resource allocation of the hierarchical cache architecture module.

[0092] The knowledge transfer and rapid adaptation module provides rapid adaptation strategy parameter updates when operating conditions change.

[0093] The hierarchical caching architecture module manages cache resources according to the allocation strategy to ensure that the data stream is stably transmitted to the data processing and storage module.

[0094] The data processing and storage module receives cached data, performs post-processing, and persists the data.

[0095] It should be noted that, through the collaborative work of the above modules, the system provided in this application can achieve stable recording of high-speed data from multi-bus heterogeneous fusion, adapt to changes in data flow characteristics under different working conditions, improve system resource utilization, and reduce the risk of data overflow.

[0096] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments based on the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for high-speed data recording based on multi-bus heterogeneous fusion, characterized in that, Includes the following steps: Collect real-time data feature parameters from each heterogeneous bus, construct feature vectors, and calculate the buffer overflow risk index; A temporal convolutional network model is used to extract the temporal patterns of the feature vectors of each bus. The temporal dependencies are captured by combining causal convolution and dilated convolution, and the feature representation is output. The cache allocation is modeled as a Markov decision process, and a deep reinforcement learning algorithm is used to train the intelligent agent model. The optimal cache allocation ratio is dynamically calculated based on a combined reward function of weighted throughput and overflow penalty. A basic cache allocation strategy model is trained for different working conditions. A modular structure is adopted to decompose the strategy into feature extraction and decision modules. The meta-learning algorithm is used to achieve rapid policy adaptation. Based on bus data flow features, operating condition feature vectors are extracted, unsupervised clustering is applied to identify changes in operating conditions, a sliding window is used for continuous monitoring, a smooth transition is achieved through strategy mixing, and the two-level cache architecture is dynamically adjusted.

2. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 1, characterized in that, The real-time data characteristic parameters collected from each heterogeneous bus include: data arrival rate, burst index, data priority, and historical cache utilization. The burst index is obtained by calculating the ratio of short-term data traffic to long-term average traffic.

3. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 1, characterized in that, Constructing the feature vector includes calculating the cache overflow risk index, which is calculated by the relationship between data arrival rate, burst index and current available cache capacity.

4. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 1, characterized in that, The temporal pattern of extracting feature vectors of each bus using a temporal convolutional network model includes: capturing long-term and short-term temporal dependencies through a combination of causal convolution and dilated convolution, and outputting feature representations.

5. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 1, characterized in that, The step of modeling cache allocation as a Markov decision process includes: The state space is defined as the characteristic representation of all buses, and the action space is defined as the buffer allocation ratio vector of each bus. The reward function is defined as a combination of weighted throughput and overflow penalty; A deep reinforcement learning algorithm is used to train an intelligent agent model, and the optimal cache allocation ratio vector is calculated based on the trained model.

6. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 5, characterized in that, The reward function consists of two parts: bus throughput and overflow rate. The bus throughput is a weighted sum based on the weight of each bus, and the overflow rate is penalized by a penalty coefficient. The final reward value is the weighted throughput minus the overflow penalty.

7. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 1, characterized in that, Training basic cache allocation strategy models for different operating conditions includes: Train corresponding basic cache allocation strategy models for different operating conditions; The strategy is decomposed into a feature extraction module and a decision module using a modular structure, wherein the feature extraction module shares parameters across operating conditions; A model-independent meta-learning algorithm is used to train the meta-policy model, enabling the meta-policy model to quickly adapt to new working conditions with a small number of samples.

8. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 1, characterized in that, The operating condition feature vector extracted based on bus data stream features includes: Based on the statistical characteristics and timing patterns of the bus data stream, extract the operating condition feature vector; An unsupervised clustering algorithm is applied to cluster the operating condition feature vectors to identify the current operating condition category of the system; A sliding time window method is used to continuously monitor changes in the operating condition feature vector. When a change in the operating condition is detected, a strategy hybrid method is used to achieve a smooth transition.

9. The method for high-speed data recording based on multi-bus heterogeneous fusion according to claim 8, characterized in that, Strategy hybrid approaches include: During the switching of operating conditions, the mixing ratio of the old and new strategies is dynamically adjusted according to the time progress, so that the weight of the old and new strategies transitions smoothly over time. The weight of the old strategy gradually decreases from 1 to 0, while the weight of the new strategy gradually increases from 0 to 1, thus achieving a smooth switching of strategies.

10. A system for high-speed data recording based on multi-bus heterogeneous fusion, used to execute the method for high-speed data recording based on multi-bus heterogeneous fusion as described in any one of claims 1-9, characterized in that, include: The bus interface and feature acquisition module are used to connect with various heterogeneous buses and receive data streams, monitor the data characteristics of each bus in real time, and collect data arrival rate, burst index, data priority, and historical buffer usage. Construct feature vectors and calculate the buffer overflow risk index; The temporal feature extraction module is used to extract the temporal patterns of each bus feature vector using a temporal convolutional network model. It captures temporal dependencies by combining causal convolution and dilated convolution and outputs feature representations. The reinforcement learning cache allocation module is used to model cache allocation as a Markov decision process, train the intelligent agent model with deep reinforcement learning algorithm, and dynamically calculate the optimal cache allocation ratio based on the combined reward function of weighted throughput and overflow penalty. The strategy training and transfer module is used to train the basic cache allocation strategy model for different working conditions. It adopts a modular structure to decompose the strategy into feature extraction and decision modules, and realizes rapid policy adaptation through meta-learning algorithm. The operating condition identification and cache management module extracts operating condition feature vectors based on bus data flow features, applies unsupervised clustering to identify operating condition changes, uses a sliding window for continuous monitoring, achieves smooth transitions through strategy mixing, and dynamically adjusts the two-level cache architecture.

Images