Acquired data stream transmission method and system based on memory

Abstract

The invention discloses an acquired data stream transmission method and system based on a memory, and relates to the field of data processing. Real-time data of a sensor are acquired and converted into digital signals, and original data are obtained; constructing a data transmission problem, and solving the data transmission problem by using a distributed collaborative gradient method to obtain an optimal data transmission method; transmitting the original data to an internal memory by using an optimal data transmission method; the memory calculates the storage priority of the original data, distributes the storage space of the original data according to the storage priority and temporarily stores the original data; obtaining a resource use state of the target system, and inputting a regression model constructed based on deep learning to calculate an optimal packaging parameter of the data packet; packaging based on the optimal packaging parameter to obtain a data packet; transmitting the data packet to a target system through a high-speed network transmission technology; the method has the advantages that the data transmission speed is increased, the delay is reduced, and the reliability and the processing performance of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and mainly to a method and system for transmitting collected data streams based on memory. Background Art

[0002] With the rapid development of the Internet of Things, industrial automation, and real-time monitoring, the real-time data streams collected by sensors are characterized by high frequency, massive volume, and low latency requirements. Traditional data transmission and storage solutions are gradually exposing significant limitations. Traditional hard drive-based storage methods store collected data on mechanical hard drives or solid-state drives, and then transmit and process the data by reading the data from the hard drives.

[0003] In existing technologies, the storage phase before data transmission often relies on hard drives (mechanical hard drives or solid-state drives). Although this approach offers large storage capacity and long data retention, it also has some significant drawbacks: First, there is a transmission efficiency bottleneck. Hard drive read and write speeds are limited by their physical structure (such as head addressing for mechanical hard drives and flash memory erase and write delays for solid-state drives). This results in a response cycle from raw data storage to transmission of milliseconds or even seconds, which cannot meet the microsecond response requirements of scenarios such as industrial control and real-time monitoring. Second, there is resource usage conflict. When high-frequency acquired data (such as sensor data sampled at 10,000 samples per second) is continuously written to the hard drive, the frequent I / O operations consume a large amount of system resources, creating resource competition with data transmission and processing processes, further exacerbating latency. Third, there are reliability risks. Hard drives are prone to failures such as bad sectors and read and write errors under high loads. Real-time data streams require extremely high continuity, and single-point storage failures may lead to data loss. In addition, existing data transmission packaging strategies are mostly static configurations (such as fixed packet size and fixed sending interval), which cannot adapt to the dynamic resource status of the target system (such as CPU load fluctuations and network bandwidth changes). When the target system resources are tight, static packaging can easily lead to data packet congestion or transmission interruption, further reducing the overall transmission efficiency. Summary of the Invention

[0004] In order to solve the above-mentioned problems existing in the prior art, the present application provides a memory-based acquisition data stream transmission method and system.

[0005] The technical solution of this application is as follows: In one aspect, the present invention provides a method for transmitting a collected data stream based on a memory, the method comprising: Acquire real-time sensor data, and convert the real-time sensor data from analog signals into digital signals as raw data; Construct a data transmission problem, including minimizing the transmission rate as the objective function and setting constraints based on the original data; solve the data transmission problem using the distributed collaborative gradient method to obtain the optimal data transmission method; The original data is transmitted to the memory using the optimal data transmission method; the memory calculates the storage priority of the original data, allocates storage space for the original data according to the storage priority, and temporarily stores the original data; Obtain the resource usage status of the target system and input the regression model built based on deep learning to calculate the optimal packaging parameters of the data packet; based on the optimal packaging parameters, the original data required by the target system is packaged to obtain a data packet; The data packets are transmitted to the target system via high-speed network transmission technology.

[0006] Preferably, the data transmission problem is constructed as follows: The objective function is expressed as follows: ; Where, represents the objective function; Indicates the The transmission rate of raw data; Indicates the The index value of the original data; Indicates the amount of original data; The constraints include transmission data integrity constraint, channel capacity constraint and transmission rate non-negative constraint, which can be expressed as follows: ; Where, Indicates constraints; Indicates the maximum allowed transmission time; Indicates the The amount of raw data; Indicates the The original data is used channels; Indicates the The index value of the channel; Indicates the The maximum capacity of a channel.

[0007] Preferably, the distributed collaborative gradient method is used to solve the data transmission problem, specifically: The objective function is split into sub-problems to solve the minimization of each original data transmission rate; the transmission data integrity constraint in the constraints is defined as a local constraint, and the channel capacity constraint and the transmission rate non-negativity constraint are defined as global constraints; Calculate each raw data transmission rate and determine whether it meets the local constraint conditions; if so, the data transmission rate gradient of the current raw data is 1; otherwise, the data transmission rate gradient of the current raw data is the preset rate suppression constant; Each channel is used as a collaborative node to collect the transmission rates of all raw data on the current channel and calculate the total rate of the current channel. It is determined whether the total rate of the current channel meets the global constraint conditions. If so, no correction is made; otherwise, the transmission rate of each raw data in the current channel is corrected. The above solution process is iterated until the maximum number of iterations is reached or the objective function converges, then the iteration is stopped to obtain the optimal data transmission method.

[0008] Preferably, the specific calculation process of the data value factor is: the data value factor is calculated based on the business importance and timeliness of the original data; The specific calculation process of the memory load factor is: the memory load factor is calculated based on the memory real-time performance index.

[0009] Preferably, the storage space of the original data is allocated and temporarily stored according to the storage priority, specifically as follows: Dividing the memory cache area into three levels of cache areas, namely, high, medium and low levels, based on the storage priority, and allocating an initial capacity to each cache area according to a preset ratio; Determine the relationship between the current storage priority of the original data and the preset storage threshold, and use the corresponding memory storage technology to temporarily store the original data.

[0010] Preferably, the optimal packaging parameters of the data packet are calculated by inputting a regression model built based on deep learning, specifically: The regression model is specifically a dual-channel attention regression model, wherein the dual channels include a resource feature extraction channel and a data feature extraction channel; Define the data packet feature set, including the resource features and raw data features of the target system; Calculating a resource feature vector of a target system's resource features using a resource feature extraction channel; Calculate the data feature vector of the original data features using the data feature extraction channel; Use the cross-attention mechanism to fuse the resource feature vector and the data feature vector to obtain a joint feature vector; The joint feature vector is linearly transformed to obtain the optimal packing parameter.

[0011] Preferably, the method further includes a target system receiving the data packet and performing data analysis; the data analysis specifically comprises using a big data analysis tool to mine association rules between data in the data packet to obtain a mining result; the big data analysis tool is an Apache Hadoop platform, and the CHARM algorithm is selected according to the requirements of the mining task to mine association rules between data, specifically: Preprocessing the received data packets, including format conversion and data partitioning strategy selection; The format conversion is specifically to convert the data packet into a universal code if it is in text format, and to perform serialization conversion if it is in binary format; The data partitioning strategy is specifically selected based on the characteristics and data volume of the data packet, and partitioned storage is performed according to the timestamp, data packet source and data type to determine the data partitioning strategy; Upload the preprocessed data packets to the distributed file system HDFS and write an analysis program using the MapReduce framework, which includes the Map phase and the Reduce phase. The Map phase is used to process data in parallel and convert each data item into an itemset. The Reduce phase mines frequent itemsets based on the CHARM algorithm. After the mining is completed, the mining results are output from the MapReduce program and stored in the specified location of HDFS; the mining results are evaluated to check whether they meet the expected business needs; The Reduce function in the Reduce phase receives the item sets from the Map phase, merges the item sets with the same key, performs statistics on the collected item sets, calculates the frequency of occurrence of each item, and forms a new item set set; Traversing the new item set set to obtain all 1-item sets, specifically itemsets containing only one element; screening 1-item sets with an occurrence frequency greater than or equal to a preset minimum support, and storing them in a separate equivalence class; Perform a connection operation on the item sets in the same equivalence class, specifically connecting the item sets with the same prefix as the item set to generate a new item set; calculate the occurrence frequency of the newly generated item set and compare it with the preset minimum support; if the occurrence frequency of the newly generated item set is lower than the preset minimum support, the current newly generated item set is discarded; if the occurrence frequency of the newly generated item set is higher than the preset minimum support, it is retained; Determining whether the retained newly generated itemset is a closed frequent itemset is specifically to calculate the frequency of occurrence of the superset of the retained newly generated itemset, and determine whether the frequency of occurrence of the superset is equal to the frequency of occurrence of the retained newly generated itemset; if they are equal, the retained newly generated itemset is not a closed frequent itemset and is discarded; if they are not equal, the retained newly generated itemset is stored in a new equivalence class; Traversing all newly generated item sets to perform a merging operation, specifically merging newly generated item sets with the same closure and occurrence frequency to generate a merged item set, and updating the equivalence class of the merged item set; Iterate through the new item set set, connection operation and merging operation to filter out closed frequent item sets until no new closed frequent item sets can be generated, and obtain all closed frequent item sets as mining results.

[0012] Preferably, the method further includes performing decision support on the mining results based on a decision tree algorithm, specifically: The mining results are read from the distributed file system HDFS; the mining results are cleaned, including processing missing values, outliers and duplicate data, and features useful for decision-making are extracted from the cleaned mining results to form a data set; based on the feature data set, a training set and a validation set are divided according to a preset ratio; Build a decision tree and set its parameters, including maximum depth, minimum number of samples, and information gain, based on data characteristics and business requirements. Starting from the root node of the decision tree, calculate the information gain of all features in the dataset. The characteristics of the dataset are Different values, according to the value of the feature, the data set is divided into subsets; calculate the information entropy of each subset and the information gain of the feature to the data set; The feature with the largest information gain is selected as the partition feature of the current decision tree node. A subtree is recursively constructed on each subset of the data set until the information gain is less than the preset threshold. The recursion is stopped and a decision tree is generated. Each non-leaf node of the subtree represents a feature judgment, and each leaf node represents a predicted value, that is, the decision result of the operating status. The trained decision tree is evaluated using the validation set, and the model is optimized based on the evaluation results to obtain an optimized decision model.

[0013] On the other hand, the present invention also proposes a memory-based data collection stream transmission system, which includes a data acquisition module, a data transmission module, a memory processing module and a data packet processing module, specifically: The data acquisition module is used to acquire real-time sensor data and convert the real-time sensor data from analog signals into digital signals as raw data; The data transmission module is used to construct a data transmission problem, including minimizing the transmission rate as the objective function and constraints based on the original data; using the distributed collaborative gradient method to solve the data transmission problem and obtain the optimal data transmission method; The memory processing module transmits the original data to the memory using the optimal data transmission method; the memory calculates the storage priority of the original data, allocates storage space for the original data according to the storage priority, and temporarily stores the original data; The data packet processing module obtains the resource usage status of the target system and inputs the regression model built based on deep learning to calculate the optimal packaging parameters of the data packet; based on the optimal packaging parameters, the original data required by the target system is packaged to obtain a data packet; The data packets are transmitted to the target system via high-speed network transmission technology.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention provides a memory-based method and system for transmitting collected data streams. This method utilizes high-speed memory caching, eliminating the traditional hard drive storage transit model. Raw sensor data is directly transmitted to memory, and verification data is temporarily stored using memory caching technology. Memory read and write speeds (microseconds) are much higher than hard drives (milliseconds), eliminating resource competition for hard drive I / O operations and resolving transmission delay issues caused by traditional hard drive storage. This system meets the low-latency requirements of real-time interactive and critical transaction data. 2) This invention provides a memory-based method and system for transmitting collected data streams. It implements priority-based dynamic memory management, calculating storage priority through a data value factor (integrating business importance and timeliness) and a memory load factor (reflecting memory usage and latency in real time). This method allocates data to high, medium, and low cache areas in a hierarchical manner and matches differentiated storage technologies (such as direct memory access for high-frequency data). This method dynamically adapts to data access requirements, improves memory cache space utilization, and prevents invalid data from occupying resources. 3) The present invention provides a memory-based method and system for transmitting collected data streams. Based on the intelligent packaging of the dual-channel attention regression model, it extracts the target system resource characteristics and original data characteristics, combines the cross-attention mechanism to fuse the characteristics and output the optimal packaging parameters, thereby improving the success rate of data packet transmission and enhancing the system's adaptability to dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0017] The present invention provides the following technical solution: a method and system for transmitting collected data streams based on memory.

[0018] Example 1: See Figure 1 This embodiment provides a method for transmitting a collected data stream based on memory, and the specific steps include: S1. Acquire real-time sensor data and convert the real-time sensor data from analog signals into digital signals as raw data; The method further includes performing data cleaning on the raw data before transmission, wherein the data cleaning specifically comprises using filtering technology to remove high-frequency noise signals in the raw data, performing missing value and outlier detection on the denoised raw data, filling missing values ​​using an interpolation method, and identifying and eliminating outliers through a statistical analysis method; S2. Construct a data transmission problem, including minimizing the transmission rate as the objective function and setting constraints based on the original data; S21. Construct a data transmission problem, specifically: The objective function is expressed as follows: ; Where, represents the objective function; Indicates the The transmission rate of raw data; Indicates the The index value of the original data; Indicates the amount of original data; The constraints include transmission data integrity constraint, channel capacity constraint and transmission rate non-negative constraint, which can be expressed as follows: ; Where, Indicates constraints; Indicates the maximum allowed transmission time; Indicates the The amount of raw data; Indicates the The original data is used channels; Indicates the The index value of the channel; Indicates the The maximum capacity of a channel; S3. Use the distributed collaborative gradient method to solve the data transmission problem and obtain the optimal data transmission method; The objective function is split into sub-problems to solve the minimization of each original data transmission rate; the transmission data integrity constraint in the constraints is defined as a local constraint, and the channel capacity constraint and the transmission rate non-negativity constraint are defined as global constraints; Calculate each raw data transmission rate and determine whether it meets the local constraint conditions; if so, the data transmission rate gradient of the current raw data is 1; otherwise, the data transmission rate gradient of the current raw data is the preset rate suppression constant; Take each channel as a cooperative node, collect the transmission rates of all original data on the current channel, and calculate the total rate of the current channel as: ; Determine whether the total rate of the current channel meets the global constraint conditions; if so, no correction is made; otherwise, the transmission rate of each original data in the current channel is corrected. , ,in Indicates the corrected transmission rate of each original data. Indicates the Correction factor for each channel; Iterate the above solution process until the maximum number of iterations is reached or the objective function converges, stop the iteration, and obtain the optimal data transmission method; S4, using the optimal data transmission method to transfer the original data into the memory; S5. The memory calculates the storage priority of the original data; the storage priority is calculated based on the data value factor and the memory load factor, specifically: The data value factor is calculated based on the business importance and timeliness of the original data, and is expressed as follows: ; Where, Indicates the data value factor; Indicates business importance; Indicates the weight of business importance; represents the timeliness weight; represents the time attenuation coefficient; Indicates the current moment; Indicates the time when the original data is generated; The memory load factor is calculated based on the memory real-time performance index and is expressed as follows: ; Where, Indicates the memory load factor; Represents the memory load factor weight; Indicates the memory used space; Indicates the total memory capacity; Indicates the memory capacity weight; Indicates the current memory access latency; represents the memory access latency benchmark; The storage priority is expressed as follows: ; Where, Indicates storage priority; S6. Allocate storage space for the original data according to the storage priority and temporarily store it; Dividing the memory cache area into three levels of cache areas, namely, high, medium and low levels, based on the storage priority, and allocating an initial capacity to each cache area according to a preset ratio; Determine the relationship between the current storage priority of the original data and the preset storage threshold, and use the corresponding memory storage technology to temporarily store the original data; If the storage priority of the current raw data is greater than the preset high storage threshold, the current raw data is temporarily stored in the high cache area using direct memory access technology; If the storage priority of the current raw data is between the preset medium storage threshold and the high storage threshold, the asynchronous memory copy technology is used to temporarily store the current raw data in the medium cache area; If the storage priority of the current raw data is less than the preset low storage threshold, the asynchronous memory mapped file technology is used to temporarily store the current raw data in the low cache area; S7. Obtain the resource usage status of the target system and input the regression model built based on deep learning to calculate the optimal packaging parameters of the data packet; The optimal packaging parameters include the optimal size of the data packet and the duration of the packaging interval; The regression model is specifically a dual-channel attention regression model, wherein the dual channels include a resource feature extraction channel and a data feature extraction channel; S71. Define a data packet feature set, including resource features and original data features of the target system; The resource characteristics of the target system include CPU utilization, remaining memory ratio, network bandwidth utilization, historical transmission delay, etc. The original data characteristics include data priority, data size, data type identifier (such as text / binary), data update frequency, etc.; S72, using the resource feature extraction channel to calculate the resource feature vector of the target system's resource features; In this embodiment, a temporal convolutional network is used as a resource feature extraction channel. The resource feature vector is expressed as follows: ; Where, Represents resource feature vector; Represents a temporal convolutional network; Indicates CPU utilization; Indicates the remaining memory ratio; Indicates network bandwidth utilization; Indicates historical transmission delay; S73, using the data feature extraction channel to calculate the data feature vector of the original data feature; In this embodiment, a multi-layer perceptron is used as a data feature extraction channel, which is expressed as follows: ; Where, represents the data feature vector; represents a multilayer perceptron; Indicates the amount of data; Indicates the data type identifier; Indicates the data update frequency; S74. Use the cross-attention mechanism to fuse the resource feature vector and the data feature vector to obtain a joint feature vector, which can be expressed as: ; Where, represents the joint eigenvector; represents the cross attention mechanism; S75. Perform a linear transformation on the joint feature vector to obtain an optimal packing parameter, which is expressed as: ; Where, Indicates the optimal size of the data packet; Indicates the packaging interval length; The weight matrix representing the linear transformation; represents the bias of the linear transformation; S8. Packing the original data required by the target system based on the optimal packaging parameters to obtain a data packet; S9, transmitting the data packet to the target system via high-speed network transmission technology; S10. The method further includes a target system receiving the data packet and performing data analysis; the data analysis specifically comprises using a big data analysis tool to mine association rules between data in the data packet to obtain a mining result; the big data analysis tool is an Apache Hadoop platform, and the CHARM algorithm is selected according to the requirements of the mining task to mine association rules between data, specifically: Preprocessing the received data packets, including format conversion and data partitioning strategy selection; The format conversion is specifically to convert the data packet into a universal code if it is in text format, and to perform serialization conversion if it is in binary format; The data partitioning strategy is specifically selected based on the characteristics and data volume of the data packet, and partitioned storage is performed according to the timestamp, data packet source and data type to determine the data partitioning strategy; Upload the preprocessed data packets to the distributed file system HDFS and write an analysis program using the MapReduce framework, which includes the Map phase and the Reduce phase. The Map phase is used to process data in parallel and convert each data item into an itemset. The Reduce phase mines frequent itemsets based on the CHARM algorithm. After the mining is completed, the mining results are output from the MapReduce program and stored in the specified location of HDFS; the mining results are evaluated to check whether they meet the expected business needs; The Reduce function in the Reduce phase receives the item sets from the Map phase, merges the item sets with the same key, performs statistics on the collected item sets, calculates the frequency of occurrence of each item, and forms a new item set set; Traversing the new item set set to obtain all 1-item sets, specifically itemsets containing only one element; screening 1-item sets with an occurrence frequency greater than or equal to a preset minimum support, and storing them in a separate equivalence class; Perform a connection operation on the item sets in the same equivalence class, specifically connecting the item sets with the same prefix as the item set to generate a new item set; calculate the occurrence frequency of the newly generated item set and compare it with the preset minimum support; if the occurrence frequency of the newly generated item set is lower than the preset minimum support, the current newly generated item set is discarded; if the occurrence frequency of the newly generated item set is higher than the preset minimum support, it is retained; Determining whether the retained newly generated itemset is a closed frequent itemset is specifically to calculate the frequency of occurrence of the superset of the retained newly generated itemset, and determine whether the frequency of occurrence of the superset is equal to the frequency of occurrence of the retained newly generated itemset; if they are equal, the retained newly generated itemset is not a closed frequent itemset and is discarded; if they are not equal, the retained newly generated itemset is stored in a new equivalence class; Traversing all newly generated item sets to perform a merging operation, specifically merging newly generated item sets with the same closure and occurrence frequency to generate a merged item set, and updating the equivalence class of the merged item set; Iterate through the new item set set, connection operation and merging operation to filter out closed frequent item sets until no new closed frequent item sets can be generated, and obtain all closed frequent item sets as mining results; The target system receives the data packet and performs data analysis; the data analysis specifically comprises using a big data analysis tool to mine association rules between data in the received data packet to obtain mining results; S11. The method further includes performing decision support on the mining results based on a decision tree algorithm, specifically: The mining results are read from the distributed file system HDFS; the mining results are cleaned, including processing missing values, outliers and duplicate data, and features useful for decision-making are extracted from the cleaned mining results to form a data set; based on the feature data set, a training set and a validation set are divided according to a preset ratio; Build a decision tree and set its parameters, including maximum depth, minimum number of samples, and information gain, based on data characteristics and business requirements. Starting from the root node of the decision tree, calculate the information gain of all features in the dataset, expressed as: ; Where, represents information entropy; Represents a dataset; Indicates the total number of categories preset based on the mining results; Representation dataset Middle The proportion of each category; Indicates the The index value of each category; The characteristics of the dataset are Different values, according to the value of the feature, the data set is divided into The subset is represented as ; Calculate the information entropy of each subset , ; Calculate the information gain of the feature pair dataset , expressed as: ; Where, Represents the characteristics of the dataset; Represents a subset of the dataset The number of data sets the proportion of quantity; The feature with the largest information gain is selected as the partition feature of the current decision tree node. A subtree is recursively constructed on each subset of the data set until the information gain is less than the preset threshold. The recursion is stopped and a decision tree is generated. Each non-leaf node of the subtree represents a feature judgment, and each leaf node represents a predicted value, that is, the decision result of the operating status. The trained decision tree is evaluated using the validation set, and the model is optimized according to the evaluation results to obtain an optimized decision model; S12. The method further includes monitoring the operating status of the target system, defining key indicators of the operating status of the target system, configuring and deploying monitoring tools, and monitoring various components of the system; Use monitoring tools to set thresholds, detect abnormal situations and potential failures, configure alarm mechanisms, and send timely notifications when monitoring indicators exceed set thresholds. Based on the alarm information, quickly locate the problem and take appropriate measures; identify performance bottlenecks in the system through monitoring data and log analysis.

[0019] Example 2: This embodiment provides a memory-based data collection and transmission system. The system includes a data acquisition module, a data transmission module, a memory processing module, and a data packet processing module. Specifically, The data acquisition module is used to acquire real-time sensor data and convert the real-time sensor data from analog signals into digital signals as raw data; The data transmission module is used to construct a data transmission problem, including minimizing the transmission rate as the objective function and constraints based on the original data; using the distributed collaborative gradient method to solve the data transmission problem and obtain the optimal data transmission method; The memory processing module transmits the original data to the memory using the optimal data transmission method; the memory calculates the storage priority of the original data, allocates storage space for the original data according to the storage priority, and temporarily stores the original data; The data packet processing module obtains the resource usage status of the target system and inputs the regression model built based on deep learning to calculate the optimal packaging parameters of the data packet; based on the optimal packaging parameters, the original data required by the target system is packaged to obtain a data packet; The data packets are transmitted to the target system via high-speed network transmission technology.

[0020] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for transmitting collected data streams based on memory, characterized in that: The method comprises: Acquire real-time sensor data, and convert the real-time sensor data from analog signals into digital signals as raw data; Construct a data transmission problem, including minimizing the transmission rate as the objective function and setting constraints based on the original data; solve the data transmission problem using the distributed collaborative gradient method to obtain the optimal data transmission method; The original data is transmitted to the memory using the optimal data transmission method; the memory calculates the storage priority of the original data, allocates storage space for the original data according to the storage priority, and temporarily stores the original data; Obtain the resource usage status of the target system and input the regression model built based on deep learning to calculate the optimal packaging parameters of the data packet; based on the optimal packaging parameters, the original data required by the target system is packaged to obtain a data packet; The data packets are transmitted to the target system via high-speed network transmission technology.

2. The memory-based data collection stream transmission method according to claim 1, characterized in that: Construct a data transfer problem, specifically: The objective function is expressed as follows: ； Where, represents the objective function; Indicates the The transmission rate of raw data; Indicates the The index value of the original data; Indicates the amount of original data; The constraints include transmission data integrity constraint, channel capacity constraint and transmission rate non-negative constraint, which can be expressed as follows: ； Where, Indicates constraints; Indicates the maximum allowed transmission time; Indicates the The amount of raw data; Indicates the The original data is used channels; Indicates the The index value of the channel; Indicates the The maximum capacity of a channel.

3. The memory-based data collection stream transmission method according to claim 1, characterized in that: The distributed collaborative gradient method is used to solve the data transmission problem, specifically: The objective function is split into sub-problems to solve the minimization of each original data transmission rate; the transmission data integrity constraint in the constraints is defined as a local constraint, and the channel capacity constraint and the transmission rate non-negativity constraint are defined as global constraints; Calculate each raw data transmission rate and determine whether it meets the local constraints; If satisfied, the data transmission rate gradient of the current original data is 1; otherwise, the data transmission rate gradient of the current original data is the preset rate suppression constant; Take each channel as a collaborative node, collect the transmission rates of all original data on the current channel, calculate the total rate of the current channel; and determine whether the total rate of the current channel meets the global constraint conditions; If it is satisfied, no correction is made; otherwise, the transmission rate of each original data in the current channel is corrected; The above solution process is iterated until the maximum number of iterations is reached or the objective function converges, then the iteration is stopped to obtain the optimal data transmission method.

4. The memory-based data collection stream transmission method according to claim 1, characterized in that: The specific calculation process of the data value factor is as follows: the data value factor is calculated based on the business importance and timeliness of the original data; The specific calculation process of the memory load factor is: the memory load factor is calculated based on the memory real-time performance index.

5. The memory-based data collection stream transmission method according to claim 4, characterized in that: Allocate storage space for the original data according to the storage priority and temporarily store it, specifically: Dividing the memory cache area into three levels of cache areas, namely, high, medium and low levels, based on the storage priority, and allocating an initial capacity to each cache area according to a preset ratio; Determine the relationship between the current storage priority of the original data and the preset storage threshold, and use the corresponding memory storage technology to temporarily store the original data.

6. The memory-based data collection stream transmission method according to claim 1, characterized in that: And input the regression model built based on deep learning to calculate the optimal packaging parameters of the data packet, specifically: The regression model is specifically a dual-channel attention regression model, wherein the dual channels include a resource feature extraction channel and a data feature extraction channel; Define the data packet feature set, including the resource features and raw data features of the target system; Calculating a resource feature vector of a target system's resource features using a resource feature extraction channel; Calculate the data feature vector of the original data features using the data feature extraction channel; Use the cross-attention mechanism to fuse the resource feature vector and the data feature vector to obtain a joint feature vector; The joint feature vector is linearly transformed to obtain the optimal packing parameter.

7. The memory-based data collection stream transmission method according to claim 1, characterized in that: The method further includes a target system receiving the data packet and performing data analysis; the data analysis specifically comprises using a big data analysis tool to mine association rules between data in the data packet to obtain a mining result; the big data analysis tool is an Apache Hadoop platform, and the CHARM algorithm is selected according to the requirements of the mining task to mine association rules between data, specifically: Preprocessing the received data packets, including format conversion and data partitioning strategy selection; The format conversion is specifically to convert the data packet into a universal code if it is in text format, and to perform serialization conversion if it is in binary format; The data partitioning strategy is specifically selected based on the characteristics and data volume of the data packet, and partitioned storage is performed according to the timestamp, data packet source and data type to determine the data partitioning strategy; Upload the preprocessed data packets to the distributed file system HDFS and use the MapReduce framework to write the analysis program, which includes the Map phase and the Reduce phase; The Map phase is used to process data in parallel and convert each piece of data into an item set; the Reduce phase mines frequent item sets based on the CHARM algorithm; After the mining is completed, the mining results are output from the MapReduce program and stored in the specified location of HDFS; the mining results are evaluated to check whether they meet the expected business needs; The Reduce function in the Reduce phase receives the item sets from the Map phase, merges the item sets with the same key, performs statistics on the collected item sets, calculates the frequency of occurrence of each item, and forms a new item set set; Traversing the new item set set to obtain all 1-item sets, specifically itemsets containing only one element; screening 1-item sets with an occurrence frequency greater than or equal to a preset minimum support, and storing them in a separate equivalence class; Perform a connection operation on the item sets in the same equivalence class, specifically connecting the item sets with the same prefix as the item set to generate a new item set; calculate the occurrence frequency of the newly generated item set and compare it with the preset minimum support; if the occurrence frequency of the newly generated item set is lower than the preset minimum support, the current newly generated item set is discarded; if the occurrence frequency of the newly generated item set is higher than the preset minimum support, it is retained; Determining whether the retained newly generated itemset is a closed frequent itemset is specifically to calculate the frequency of occurrence of the superset of the retained newly generated itemset, and determine whether the frequency of occurrence of the superset is equal to the frequency of occurrence of the retained newly generated itemset; if they are equal, the retained newly generated itemset is not a closed frequent itemset and is discarded; if they are not equal, the retained newly generated itemset is stored in a new equivalence class; Traversing all newly generated item sets to perform a merging operation, specifically merging newly generated item sets with the same closure and occurrence frequency to generate a merged item set, and updating the equivalence class of the merged item set; Iterate through the new item set set, connection operation and merging operation to filter out closed frequent item sets until no new closed frequent item sets can be generated, and obtain all closed frequent item sets as mining results.

8. The memory-based data collection stream transmission method according to claim 7, characterized in that: The method further includes performing decision support on the mining results based on a decision tree algorithm, specifically: The mining results are read from the distributed file system HDFS; the mining results are cleaned, including processing missing values, outliers and duplicate data, and features useful for decision-making are extracted from the cleaned mining results to form a data set; based on the feature data set, a training set and a validation set are divided according to a preset ratio; Build a decision tree and set its parameters, including maximum depth, minimum number of samples, and information gain, based on data characteristics and business requirements. Starting from the root node of the decision tree, calculate the information gain of all features in the dataset. The characteristics of the dataset are Different values, according to the value of the feature, the data set is divided into subsets; calculate the information entropy of each subset and the information gain of the feature to the data set; The feature with the largest information gain is selected as the partition feature of the current decision tree node. A subtree is recursively constructed on each subset of the data set until the information gain is less than the preset threshold. The recursion is stopped and a decision tree is generated. Each non-leaf node of the subtree represents a feature judgment, and each leaf node represents a predicted value, that is, the decision result of the operating status. The trained decision tree is evaluated using the validation set, and the model is optimized based on the evaluation results to obtain an optimized decision model.

9. The memory-based data collection stream transmission method according to claim 1, characterized in that: The method further includes monitoring the operating status of the target system, defining key indicators of the operating status of the target system, configuring and deploying monitoring tools, and monitoring various components of the system; Use monitoring tools to set thresholds, detect abnormal situations and potential failures, configure alarm mechanisms, and send timely notifications when monitoring indicators exceed set thresholds. Based on the alarm information, quickly locate the problem and take appropriate measures; identify performance bottlenecks in the system through monitoring data and log analysis.

10. A memory-based data collection stream transmission system, characterized in that: The system includes a data acquisition module, a data transmission module, a memory processing module and a data packet processing module, specifically: The data acquisition module is used to acquire real-time sensor data and convert the real-time sensor data from analog signals into digital signals as raw data; The data transmission module is used to construct a data transmission problem, including minimizing the transmission rate as the objective function and constraints based on the original data; using the distributed collaborative gradient method to solve the data transmission problem and obtain the optimal data transmission method; The memory processing module transfers the original data to the memory using the optimal data transmission method; The memory calculates the storage priority of the original data, allocates storage space for the original data according to the storage priority, and temporarily stores the original data; The data packet processing module obtains the resource usage status of the target system and inputs the regression model built based on deep learning to calculate the optimal packaging parameters of the data packet; based on the optimal packaging parameters, the original data required by the target system is packaged to obtain a data packet; The data packets are transmitted to the target system via high-speed network transmission technology.

Images