Large-scale energy storage power station high-frequency data acquisition and multi-channel parallel transmission method and system

Abstract

The invention discloses a large-scale energy storage power station high-frequency data acquisition and multi-channel parallel transmission method and system, and the system comprises a data acquisition module which is responsible for the acquisition, reading and data subpackage; the data compression module is used for implementing diversified compression strategies according to data characteristics; the multi-channel parallel transmission module significantly improves the data transmission rate and the system throughput through a parallel transmission mechanism. According to the method, different data types and application requirements are combined, and a flexible compression strategy is provided. And through a multi-channel parallel mechanism, the speed of data acquisition, reading, compression and transmission is greatly improved, so that the data management challenge caused by scale expansion of the energy storage power station is met. According to the invention, high-efficiency storage, compression and transmission of high-frequency data of the battery can be realized, and a solid technical support is provided for stable operation and data analysis of an energy storage power station.

Description

Technical Field

[0001] The present invention belongs to the technical field of battery data acquisition, and in particular relates to a high-frequency data acquisition and multi-channel parallel transmission method for a large energy storage power station. Background Art

[0002] With the development of various forms of energy such as hydropower, wind power, and photovoltaic power generation, the installed capacity of these energy sources has taken the lead globally, with the total installed capacity of power generation in the country reaching about 2.92 billion kWh. As the installed capacity increases, it also brings new challenges to the operation and management of the power grid. In this context, energy storage technology has become a key technology to solve the problem of large-scale storage and dispatching of electricity, and large-scale energy storage power stations have played an increasingly important role in balancing the supply and demand of the power grid and enhancing the stability and flexibility of the power grid. However, large-scale energy storage power stations generate a huge amount of high-frequency data during operation. How to efficiently and comprehensively collect this data and reduce the cost of data transmission has become an important technical challenge facing the current field of energy storage technology.

[0003] Traditional data collection and transmission methods are implemented through technologies such as Huffman coding or discarding data. These methods often have problems such as low data collection efficiency, slow transmission rate, poor real-time performance, and high transmission and storage costs, making it difficult to meet the needs of large-scale energy storage power stations for high-frequency data collection and transmission. In addition, with the development of energy storage technology and the increase in market demand, the scale of energy storage power stations has expanded significantly. From the 50MW photovoltaic energy storage project in Shigatse, Tibet Autonomous Region in 2020 to the 61MW / 123MWh of Nanjing Nangang Energy Storage Power Station in 2024. As the capacity of the power station increases, the number of battery cells also increases significantly, and with it a significant increase in the amount of high-frequency data. Traditional data collection and transmission methods can no longer meet the expansion needs of large-scale energy storage power stations, and a new technical solution is urgently needed to solve these problems. Summary of the invention

[0004] In view of the above problems, the present invention provides a method for high-frequency data collection and multi-channel parallel transmission of a large-scale energy storage power station, which can quickly and comprehensively collect and read the high-frequency data of battery cells in the energy storage power station, and improve the data transmission rate and real-time performance through multi-channel parallel transmission technology, thereby meeting the needs of daily monitoring, management and optimized operation of the energy storage power station.

[0005] The technical solution is as follows:

[0006] The present invention provides a method for high-frequency data collection and multi-channel parallel transmission of a large energy storage power station, comprising the following steps:

[0007] Step S1, data collection and reading: Use power station equipment to monitor battery status and upload data to the local storage system through the collection terminal. Create a producer thread and use numpy technology to efficiently read source files. This technology supports both large file reading and custom data parsing. After data cleaning, the read data is read and transmitted externally based on the producer thread. Data cleaning includes but is not limited to missing value processing and data type conversion.

[0008] Step S2, reading the data outputted in step S1, selecting a corresponding compression algorithm according to the data type, and obtaining a data compression package based on differential compression.

[0009] Data compression: Select the appropriate compression algorithm based on the data characteristics, such as time and date sequence, numerical sequence, and specific requirements for accuracy, and apply the corresponding compression algorithm to each data block. After compression, save the sparse matrix and compression information of the compressed data as .npz files and .json files respectively, encapsulate them into data packets and convert them into a transmission format suitable for the network environment.

[0010] Step S3, create producer-consumer threads to perform multi-channel parallel transmission of data compression packets: create a thread-safe task queue to store the data packets to be transmitted. The producer thread is responsible for putting the data packets into the task queue and dynamically adjusting the frequency of putting them into the queue according to the data volume and compression progress. Each consumer thread takes the data packet from the task queue and prepares for transmission. Set the connection parameters for each transmission channel, such as the receiving end address, etc. At the receiving end, reassemble the received data packets to restore the original data structure.

[0011] Step S4, monitoring and logging: Calculate the compression time, file size before and after compression and other indicators to evaluate the algorithm performance and compression effect. Record key events in the transmission process, including the start, end, error and retry information of the transmission, etc., to facilitate later analysis and debugging.

[0012] Step S5: Data upload: upload the compressed data to the cloud platform server.

[0013] Preferably, S101: monitor the battery status through the power station equipment, and upload the battery data to the local storage system using the acquisition terminal; S102: use python's threading.Thread class to create and manage the producer thread, start the thread by calling the start() method of the thread object, and read the data packet from the acquisition terminal; S103: in each producer thread, use the loadtxt() or genfromtxt() function in the numpy library to customize the high-frequency data parsing method to adapt to different data types; S104: during the data reading process, check whether there is a data reading anomaly based on the anomaly detection mechanism, so as to timely discover and handle errors; during the execution of each producer thread, clean and preprocess the read data, including missing value processing and data type conversion.

[0014] Preferably, in S104, for missing value processing, the dropna() method of the pandas library is used; for data type conversion, the astype() method is used to convert string type data into corresponding data types.

[0015] Step 2 includes: S201: obtaining the header information of the data, and filtering the data according to the pre-set compression features; further optimizing the data storage space through precise screening; S202: selecting a suitable compression algorithm according to the data type and accuracy requirements, and realizing data compression based on differential calculation; after compression, saving the sparse matrix and compression information of the compressed data as .npz files and .json files respectively, encapsulating them into data packets and converting them into a transmission format suitable for the network environment.

[0016] Preferably, the data types include date data, total voltage of battery cluster, total current of battery cluster, insulation resistance of positive and negative electrodes, main control environment temperature, total SOC value of cluster, total SOH value of cluster, cumulative discharge capacity, battery cell voltage and battery cell temperature. When the data type is date data, the timestamp is converted into integer form D′ i Then perform differential compression:

[0017] D′ i =f(D i )

[0018] D i represents the i-th data point in the original timestamp data sequence, and f is the data type conversion function;

[0019] When the data type is the total voltage of the battery cluster, only the integer part V′ is retained. i Then perform differential compression:

[0020] V′ i =round(V i )

[0021] V i The ith data point in the original data sequence representing the total voltage of the battery cluster;

[0022] When the data type is the total current of the battery cluster, each data point I i Separate into integer part I′ int,i and the decimal part I′ float,i Then perform differential compression separately:

[0023] I′ int,i =(I i ×10 d ) / 10 d

[0024] I′ float,i =(I i ×10 d )%10 d

[0025] I i The ith data point in the original data sequence representing the total flow of the battery cluster, d is the number of decimal places to be retained, and the integer part I′ int,i The fractional part I' is obtained by integer division. float,i Obtained by the remainder operation;

[0026] When the data type is positive and negative insulation resistance, the storage space is further reduced based on dictionary mapping before differential compression:

[0027] When the data types are main control environment temperature, cluster total SOC value, cluster total SOH value, battery cell voltage and battery cell temperature, the original data is scaled and then differentially compressed;

[0028] When the data type is cumulative discharge capacity, the difference calculation is performed directly, and then a sparse matrix M is created and compressed by columns.

[0029] As a preferred method, the dictionary mapping method is:

[0030] First, create a mapping dictionary D, which maps the resistance values ​​in order of frequency from high to low and value from large to small to {0, 1, 2, ...}:

[0031] D={Value 1 :0,Value 2 :1,…,Value n :n-1}

[0032] Where Value 1 Frequency of occurrence > Value 2,…n ,Value 1 >Value2,…n ;

[0033] According to the mapping dictionary D, the original data sequence is mapped into a new data sequence R, R i Represents the i-th data point, followed by differential calculation and compression.

[0034] Preferably, the differential compression method is: firstly, the difference between adjacent data points is calculated, and then the mode of the difference sequence is subtracted from the difference to obtain a sparse matrix for achieving compression.

[0035] Preferably, in step 3, multi-channel parallel transmission includes: S301: using queue.Queue() method to initialize the task queue and set the maximum capacity of the queue to avoid memory overflow; S302: using thread lock threading.Lock() to ensure thread safety when the producer thread puts the data packet into the queue; S303: the consumer thread takes the data packet from the queue and prepares for transmission; S304: after the data transmission is completed, using spicy.sparse.vstack() method to reorganize the scattered data packets, restore the original data structure, and maintain the consistency of the data structure.

[0036] Preferably, step S4 is to perform monitoring and log recording: calculate compression time, file size indicators before and after compression, evaluate algorithm performance and compression effect; record key events in the transmission process, including transmission start, end, error and retry information, to facilitate later analysis and debugging.

[0037] The present invention also discloses a high-frequency data acquisition and multi-channel parallel transmission system for a large energy storage power station, which includes three modules:

[0038] The data acquisition module is responsible for acquiring data from power station equipment, such as power storage converters (PCS), battery management systems (BMS), and energy management systems (EMS). The data is then stored in the local system using the acquisition terminal. A producer thread is created to efficiently read the original data of the source file, perform data cleaning and data preprocessing, and ensure the uniformity and accuracy of the data.

[0039] The data compression module adopts a variety of compression strategies to adapt to different types of data characteristics. The present invention implements diversified compression processing according to the accuracy requirements, distribution characteristics, and data types of the data, such as date data, continuous data, etc. The compression is completed through differential calculation and sparse matrix, and the compressed data and compression information are saved and encapsulated into a data packet. Compared with the traditional compression algorithm, the data storage requirements in the battery field are taken into consideration, and flexible and appropriate compression is provided according to the type of data. This compression can significantly reduce the data volume, improve compression efficiency, and reduce transmission costs.

[0040] The multi-channel parallel transmission module uses multiple producer-consumer thread pairs to simultaneously put or take out data packets from the task queue, achieving fast data distribution, ensuring data transmission efficiency, and meeting the needs of high-frequency data processing.

[0041] Beneficial effects:

[0042] (1) The high-frequency data acquisition and multi-channel parallel transmission method of the present invention uses a differential sparse matrix compression algorithm to significantly reduce the space required for data storage and reduce storage costs. The compression rate of the algorithm is about 15%. In addition, compared with existing compression methods, the present invention selectively retains and blurs decimal places based on data characteristics and actual application requirements, which not only ensures data availability but also optimizes data storage. For example, when processing key parameters such as the total voltage of the battery cluster, the temperature of the main control unit, and the total SOC of the cluster, the adjustable decimal places are retained.

[0043] (2) Compared with the prior art, the high-frequency data acquisition and multi-channel parallel transmission method involved in the present invention introduces a producer-consumer model to realize multi-channel parallel acquisition, compression and transmission of data. This method can significantly improve the compression rate of data, so that it can ensure the efficient compression of large-scale data streams. For example, the compression time of a 510MB file can be shortened to 71 seconds. In addition, through parallel transmission, the system can make better use of network bandwidth, reduce the transmission bottleneck caused by bandwidth limitations, and thus improve the overall data transmission efficiency.

[0044] (3) The high-frequency data acquisition and multi-channel parallel transmission method involved in the present invention can obtain: 1. More flexible and effective data compression: through dictionary coding, differential calculation, sparse matrix compression and other technologies, the present invention can effectively compress different types of data according to data characteristics, significantly reducing the amount of data storage; 2. More efficient data compression: The multi-channel parallel mechanism divides the collected data and reads, compresses and transmits it through multiple channels at the same time, thereby improving the data processing rate; 3. Lower network bandwidth occupancy: The multi-channel transmission design disperses the high load of single-channel data processing and improves bandwidth utilization; 4. More flexible scalability: The modular design allows the system to flexibly expand new transmission channels according to demand to adapt to the growing scale and data volume of energy storage power stations; 5. More powerful system stability: Through exception handling and log mechanisms, it is ensured that the system can still operate stably when facing abnormal data or transmission errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below.

[0046] Figure 1 It is a structural diagram of a method for high-frequency data acquisition and multi-channel parallel transmission according to an embodiment of the present invention;

[0047] Figure 2 is a flowchart of the steps of the high-frequency data acquisition and multi-channel parallel transmission method provided by an embodiment of the present invention;

[0048] Figure 3 The diagram is a schematic diagram of the data block segmentation and reassembly process in multi-channel transmission according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical scheme in the embodiment of the present invention will be described in more detail below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] Example 1

[0051] like Figure 1 As shown, an embodiment of the present invention provides a high-frequency data acquisition and multi-channel parallel transmission system, including:

[0052] The data acquisition module collects data in the battery pack at a frequency of seconds, including but not limited to: the total voltage, total current, battery cell voltage, positive and negative electrode insulation resistance, cumulative discharge capacity, total charge state (SOC) and health state (SOH) of the battery cluster. In addition, the module also takes into account external environmental factors that affect battery performance, such as the ambient temperature of the main control unit and the temperature of the battery cell. Collecting these key data is conducive to monitoring the status of the battery pack, conducting subsequent predictions and analysis, and controlling potential aging and thermal runaway risks. Ensure high quality and reliability of data. After collecting the data, create a producer thread to sub-packetize the high-frequency data. The present invention adopts a producer-consumer model to realize high-frequency data acquisition and multi-channel parallel transmission of large-scale energy storage power stations. The producer can divide the data into multiple data packets, each of which can be regarded as a task and run independently in each producer thread. In the present invention, the producer is mainly responsible for completing data acquisition and data compression, and the consumer is responsible for completing data parallel transmission. The model uses a thread pool to manage the producer-consumer thread pair so that the transmission channels run in different threads respectively to realize multi-channel parallel acquisition, compression and transmission. The thread pool can also control the number of parallel threads to avoid excessive consumption of system resources caused by too many threads. At the same time, the data is initially cleaned in this module, including but not limited to processing missing data and converting data types.

[0053] The data compression module selects appropriate compression algorithms to compress the collected high-frequency data according to the data characteristics. In order to better illustrate the core idea of ​​the present invention, a single producer thread is used as an example. In producer thread 1, for date and time data, it is converted into a timestamp, and a sparse matrix is ​​generated by differential calculation, thereby significantly reducing the storage space. The continuity of time series data leads to a small difference between adjacent time points, which can greatly reduce the storage space. For key parameters such as the temperature of the main control unit, the total SOC of the cluster, and the voltage of the battery cell, this type of data can directly reflect the safety and performance of the battery system, so the accuracy requirements are relatively high. This module uses differential processing to achieve sparse matrix compression, while retaining an appropriate number of decimal places, to ensure that while reducing the data storage space, the storage accuracy of the data is taken into account. For the total current of the battery cluster, this type of data has higher accuracy requirements. This module separates integers and decimal places, and performs differential processing and sparse matrix compression respectively, which can reduce storage requirements and ensure storage accuracy. For data such as positive and negative electrode insulation resistance and cluster total SOH, since the data values ​​are large and repeated, dictionary encoding technology is used to map the data into a smaller set of integers, and then differential compression is performed, thereby greatly reducing the storage space. For the cumulative discharge capacity, the difference is directly calculated and sparse matrix compression is performed. The algorithm fully considers factors such as the characteristics of the data, the purpose of the data, the compression rate, and the compression speed to ensure that the data can be compressed and decompressed quickly without distortion, meeting the needs of high-frequency data transmission.

[0054] The multi-channel parallel transmission module transmits compressed data in parallel through multiple communication channels to improve data transmission speed and system throughput. After the producer thread completes the compression, it puts the data packet into the security queue, and the consumer (i.e., the transmission channel) is responsible for taking the data packet out of the queue and transmitting it to the destination.

[0055] Example 2

[0056] like Figure 2 As shown, the present invention provides a high-frequency data acquisition and multi-channel parallel transmission method. The method steps of the present invention are specifically described below in conjunction with specific embodiments:

[0057] Step S1: Data collection and reading. The specific implementation process is as follows:

[0058] S101: Monitor the battery status through the power station equipment and upload the data to the local storage system using the acquisition terminal.

[0059] S102: Use Python's threading.Thread class to create and manage producer threads. By calling the start() method of the thread object, the threads can be started, and they will be responsible for reading data packets from the acquisition terminal.

[0060] S103: In each producer thread, use the loadtxt() or genfromtxt() function in the numpy library to customize the high-frequency data parsing method to adapt to different data types. Custom high-frequency data parsing is to convert the data into timestamps, retain decimal places, and use dictionary mapping and other methods to parse according to different data characteristics, and then complete the difference and construct a sparse matrix (implemented in step S3). Custom high-frequency data parsing and conversion data corresponds to the data preprocessing in step S3 according to different compression processing methods for different data types.

[0061] S104: During the data reading process, an exception detection mechanism is implemented to check whether there are data reading exceptions, such as reading failure, data format error, etc., and try-except is used to capture exceptions to promptly discover and handle errors. During the execution of each producer thread, the read data is cleaned and preprocessed, mainly including missing value processing and data type conversion. Among them, for missing value processing, the dropna() method of the pandas library is used; for data type conversion, the astype() method is used to convert string type data to the corresponding data type, for example, converting string type date data to date type.

[0062] Data source in this embodiment: The data mainly comes from the equipment in the energy storage power station of Nanjing Iron and Steel Group, including PCS, BMS and EMS. Between May 27, 2024 and May 31, 2024, a total of 21,578 data were collected. The interval of these data is 18 seconds, and the cumulative data volume is about 77MB. The header of the data file lists in detail the device id, recording time, total voltage of the battery cluster, total current of the battery cluster, positive and negative insulation resistance, maximum single cell voltage value and device id, minimum single cell voltage value and device id, single cell voltage mean and range value, maximum single cell temperature value and device id, minimum single cell temperature value and device id, main control ambient temperature, cluster total SOC value, cluster total SOH value, cumulative discharge capacity, single cluster battery remaining capacity, charging single capacity, charging cumulative capacity, discharge single capacity, file path, and key parameters such as battery cell voltage and temperature, totaling 665 features.

[0063] S105: Get the header information of the data and filter the data according to the pre-set compression features. Through accurate filtering, the data storage space is further optimized. Before data compression, the feature columns in the data set are first filtered based on business needs to filter out data features that are critical to the business, and identify elements that can be simplified or ignored. The feature classes that need to be compressed are pre-set to facilitate flexible compression for different data types in the future.

[0064] Step S2: The specific implementation process of data compression is as follows:

[0065] S201: Select a suitable compression algorithm according to data type, accuracy requirements, etc., and sparse different types of data according to data characteristics, so as to achieve the purpose of data compression. Data types include date data, battery cluster total voltage, battery cluster total current, positive and negative insulation resistance, main control environment temperature, cluster total SOC value, cluster total SOH value, cumulative discharge capacity, battery cell voltage, battery cell temperature, etc.

[0066] S202: Data compression is achieved by differential calculation, which is described below according to the data type:

[0067] ①Record time, which is a date type data.

[0068] Assume that D is the original timestamp data sequence, D i represents the i-th data point. First, convert the timestamp into integer form D′ i :

[0069] D′ i =f(D i )

[0070] Where f is a data type conversion function.

[0071] Next, calculate the adjacent timestamp D′ i-1 and D′ i The difference between ΔD′ i :

[0072] ΔD′ i =D′ i -D′ i-1

[0073] In order to generate more zero values ​​to facilitate sparse matrix construction, each timestamp T′ i Subtract the mode of the difference (ΔD′):

[0074] D″ i =D′ i -Mode(ΔD′)

[0075] Among them, D i It can be used to generate a sparse matrix M. Mode(ΔD′) is the mode of the difference sequence ΔD′, that is, the difference with the most occurrences.

[0076] Finally, use the sparse matrix M to compress the timestamp D″ by column i , to save storage space. The scipy.sparse.csc_matrix() method is used. Since sparse matrices only record the positions and values ​​of non-zero elements, they are particularly efficient when storing a large amount of zero-valued data.

[0077] ②Total voltage of battery cluster, no need to retain decimal places.

[0078] Assume that V is the original data sequence of the total voltage of the battery cluster, V i represents the i-th data point. Since the data accuracy requirement is not high, only the integer part is retained:

[0079] V′ i =round(V i )

[0080] Among them, round() represents the rounding function, V′ i Yes V i The integer part of .

[0081] Then, the integer data is differentially calculated to calculate the total voltage V′ of the adjacent battery clusters. i-1 and V′ i The difference between ΔV′ i :

[0082] ΔV′ i =V′ i -V′ i-1

[0083] Similarly, each data point V′ i Subtract the mode of the difference Mode(ΔV′):

[0084] V″ i =V′ i -Mode(ΔV′)

[0085] Finally, the total voltage V″ of the battery cluster after column compression of the sparse matrix M is generated i .

[0086] ③ The total current of the battery cluster needs to retain multiple decimal places.

[0087] Assume that I is the original data sequence of the total flow of the battery cluster, I i represents the i-th data point.

[0088] Since the data accuracy of the total flow of the battery cluster is high, it is necessary to compress and save the integers and decimals separately. First, separate the integers and decimals. i Separate into integer part I′ int,i and the decimal part I′ float,i :

[0089] I′ int,i =(I i ×10 d ) / 10 d

[0090] I′float,i =(I i ×10 d )%10 d

[0091] Where d is the number of decimal places to be retained. The total current of the battery cluster needs to retain 1 decimal place, so d = 1. Integer part I' int,i The fractional part I′ is obtained by integer division. float,i Obtained through the remainder operation.

[0092] Then, for the integer part I′ int,i and the decimal part I′ float,i Perform difference calculations separately.

[0093] ΔI′ int,i =I′ int,i -I′ int,i-1

[0094] ΔI′ float,i =I′ float,i -I′ float,i-1

[0095] Similarly, the integer part I′ of each data point int,i and the decimal part I′ float,i Subtract the mode of the difference Mode(ΔI′ int ) and Mode(ΔI′ float ):

[0096] I″ int,i =I′ int,i -Mode(ΔI′ int )

[0097] I″ float,i =I′ float,i -Mode(ΔI′ float )

[0098] Finally, the sparse matrix M will be generated int and M float Merge by column and then compress by column.

[0099] ④ The insulation resistance of the positive and negative electrodes has many repeated values ​​and the values ​​are large.

[0100] Since there are many repeated values ​​of the positive and negative insulation resistances and the values ​​are large, even if differential calculation is used, the large difference is not conducive to storage. Therefore, dictionary mapping can be used to further reduce the storage space.

[0101] First, a mapping dictionary D is created, and the resistance values ​​are mapped to {0, 1, 2, ...} in order of frequency from high to low and value from large to small.

[0102] D={Value 1 :0,Value 2 :1,…,Value n :n-1}

[0103] Where Value 1 Frequency of occurrence > Value 2,…n ,Value 1 >Value 2,…n .

[0104] According to the mapping dictionary D, the original data sequence is mapped into a new data sequence R, R i represents the ith data point. Then the difference calculation is performed for compression.

[0105] ΔR′ i =R′ i -R′ i-1

[0106] R″ i =R′ i -Mode(ΔR′)

[0107] Among them, R′ i and R′ i-1 Indicates the insulation resistance of the positive and negative electrodes of adjacent data points, ΔR′ i represents their difference. Mode(ΔR′) represents the mode of the difference sequence, R″ i Used to create a sparse matrix M.

[0108] ⑤ Main control environment temperature, retain 1 decimal place.

[0109] Assume that T is the original data sequence of the main control environment temperature, T i Represents the i-th data point. Since the difference of temperature data is small, in order to simplify floating-point operations, the original data sequence T is multiplied by a constant 10 to obtain the scaled data sequence T′:

[0110] T′ i =T i ×10

[0111] Then the scaled data sequence T′ is differentially calculated and compressed:

[0112] ΔT′ i =T′ i -T′ i-1

[0113] T″ i =T′ i -Mode(ΔT′)

[0114] Among them, T′i and T′ i-1 Indicates the main control environment temperature of the adjacent data point, ΔT′ i represents their difference. Mode(ΔT′) represents the mode of the difference sequence, T″ i Used to create a sparse matrix M.

[0115] ⑥ The differential calculation process of the total cluster SOC value is the same as that of the main control ambient temperature.

[0116] ⑦ The differential calculation process of the total SOH value of the cluster is the same as that of the positive and negative insulation resistance.

[0117] ⑧Accumulate the discharge capacity, directly perform differential calculation, and then create a sparse matrix M and compress it by column.

[0118] ΔAccq′ i =Accq′ i -Accq′ i-1

[0119] Accq″ i =Accq′ i -Mode(ΔAccq′)

[0120] Among them, Accq′ i and Accq′ i-1 Represents the cumulative discharge capacity of adjacent data points, ΔAccq′ i represents their difference. Mode(ΔAccq′) represents the mode of the difference sequence, Accq″ i Used to create a sparse matrix M.

[0121] ⑨ The differential calculation process of the battery cell voltage is similar to that of the main control ambient temperature. The difference is that 3 decimal places need to be retained, and d is set to 3 for differential calculation.

[0122] ⑩ The differential calculation process of the battery cell temperature is the same as that of the main control ambient temperature.

[0123] S203: Encapsulate the compressed data into a packet.

[0124] Step S3: Multi-channel parallel transmission. The specific implementation process is as follows:

[0125] S301: Use the queue.Queue() method to initialize the task queue and set the maximum capacity of the queue to avoid memory overflow.

[0126] S302: In order to ensure the security of the task queue, a thread lock threading.Lock() is used to ensure thread safety when the producer thread puts the data packet into the queue.

[0127] S303: The consumer thread takes the data packet from the queue and prepares for transmission.

[0128] S304: After the data transmission is completed, the scattered data packets are reassembled using the spicy.sparse.vstack() method to restore the original data structure while maintaining the consistency of the data structure.

[0129] Step S4: Monitoring and logging. The specific implementation process is as follows:

[0130] S401: Calculate and report key indicators such as compression time T comp , evaluate algorithm performance and compression effect.

[0131] Compression time refers to the total time from the start of compression to the completion of compression. Use the time.time() method to collect the start time t s and end time t e , compression time T comp :

[0132] T comp =t e -t s

[0133] S402: Save key nodes in the transmission process into a log, such as compression or transmission start, end, compression or transmission error, etc.

[0134] Step S5: Data upload: uploading the processed data to the cloud platform server to achieve remote storage and access of data.

[0135] In order to better illustrate the producer-consumer model adopted by the present invention, Figure 3 A schematic diagram of the data segmentation and reorganization process in multi-channel transmission is given in FIG.

[0136] like Figure 3 As shown in the figure, the producer thread is responsible for reading data, packetizing, data cleaning and data compression, and is responsible for adding the processed data packets to the task queue in order. The consumer thread is responsible for taking the data packets from the queue and performing parallel transmission.

[0137] The present invention has carried out a comparative experiment, using the present invention and a general compression tool to compress multiple files. The experimental results are shown in Table 1.

[0138] Table 1 Comparison of compression tool performance

[0139]

[0140] The embodiments described above are only examples of specific applications of the present invention and are not intended to limit the present invention. Those skilled in the art may modify, supplement or replace some of the technologies in the technical solutions of the specific embodiments. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for high-frequency data collection and multi-channel parallel transmission of a large energy storage power station, characterized in that: The steps include: Step S1, based on the collection terminal, the battery data is collected to the local storage, a producer thread is created to read the data, and the read data is cleaned and then transmitted externally based on the producer thread; Step S2, reading the data outputted in step S1 and performing data compression, selecting a corresponding compression algorithm according to the data type, and obtaining a data compression package based on differential compression; Step S3, create producer-consumer threads to perform multi-channel parallel transmission of data compression packets. Specifically, create a thread-safe task queue to store data packets to be transmitted; each producer thread puts the data packet into the task queue, and dynamically adjusts the frequency of putting it into the queue according to the data volume and compression progress; each consumer thread takes the data packet from the task queue and prepares for transmission; set connection parameters for each transmission channel, reorganize the received data packets at the receiving end, and restore the original data structure.

2. The method according to claim 1, characterized in that Step 1 includes: S101: Monitor the battery status through the power station equipment and upload the battery data to the local storage system using the acquisition terminal; S102: Use Python's threading.Thread class to create and manage the producer thread, start the thread by calling the start() method of the thread object, and read the data packet from the acquisition terminal; S103: In each producer thread, use the loadtxt() or genfromtxt() function in the numpy library to customize the high-frequency data parsing method to adapt to different data types; S104: During the data reading process, whether there is a data reading anomaly is checked based on the anomaly detection mechanism. During the execution of each producer thread, the read data is cleaned and preprocessed, including missing value processing and data type conversion.

3. The method according to claim 2, characterized in that In S104, for missing value processing, the dropna() method of the pandas library is used; for data type conversion, the astype() method is used to convert string type data into the corresponding data type.

4. The method according to claim 1, characterized in that Step 2 includes: S201: obtaining header information of the data, and filtering the data according to pre-set compression features; further optimizing the data storage space through accurate filtering; S202: Select a suitable compression algorithm according to the data type and accuracy requirements, and implement data compression based on differential calculation; after compression, save the sparse matrix and compression information of the compressed data as .npz files and .json files respectively, encapsulate them into data packets and convert them into a transmission format suitable for the network environment.

5. The method according to any one of claims 1 to 4, characterized in that: The data types include date data, total battery cluster voltage, total battery cluster current, positive and negative insulation resistance, main control environment temperature, total cluster SOC value, total cluster SOH value, cumulative discharge capacity, battery cell voltage and battery cell temperature. When the data type is date data, the timestamp is converted to integer form D′ i Then perform differential compression: D′ i =f(D i ) D i represents the i-th data point in the original timestamp data sequence, and f is the data type conversion function; When the data type is the total voltage of the battery cluster, only the integer part V′ is retained. i Then perform differential compression: V′ i =round(V i ) V i The ith data point in the original data sequence representing the total voltage of the battery cluster; When the data type is the total current of the battery cluster, each data point I i Separate into integer part I′ int，i and the decimal part I′ float，i Then perform differential compression separately: I′ int，i =(I i ×10 d ) / 10 d I′ float，i =(I i ×10 d )%10 d I i The ith data point in the original data sequence representing the total flow of the battery cluster, d is the number of decimal places to be retained, and the integer part I′ int，i The fractional part I' is obtained by integer division. float，i Obtained by the remainder operation; When the data type is positive and negative insulation resistance, the storage space is further reduced based on dictionary mapping before differential compression: When the data types are main control environment temperature, cluster total SOC value, cluster total SOH value, battery cell voltage and battery cell temperature, the original data is scaled and then differentially compressed; When the data type is cumulative discharge capacity, the difference calculation is performed directly, and then a sparse matrix M is created and compressed by columns.

6. The method according to claim 5, characterized in that The dictionary mapping method is: First, create a mapping dictionary D, which maps the resistance values ​​in order of frequency from high to low and value from large to small to {0, 1, 2, ...}: D={Value1:0,Value2:1,…,Value n :n-1} The frequency of Value1 appears>Value2,…n, Value1>Value2,...n; According to the mapping dictionary D, the original data sequence is mapped into a new data sequence R, R i Represents the i-th data point, followed by differential calculation and compression.

7. The method according to claim 5, characterized in that The differential compression method is: firstly calculate the difference between adjacent data points, then subtract the mode of the difference sequence from the difference, and obtain a sparse matrix for achieving compression.

8. The method according to claim 1, characterized in that In step S3, multi-channel parallel transmission includes: S301: Use the queue.Queue() method to initialize the task queue and set the maximum capacity of the queue to avoid memory overflow; S302: Use thread lock threading.Lock() to ensure thread safety when the producer thread puts the data packet into the queue; S303: The consumer thread takes the data packet from the queue and prepares for transmission; S304: After the data transmission is completed, the scattered data packets are reassembled using the spicy.sparse.vstack() method to restore the original data structure while maintaining the consistency of the data structure.

9. The method according to claim 1, characterized in that: The method further comprises step S4, monitoring and logging: calculating compression time, file size index before and after compression, and evaluating algorithm performance and compression effect; Record key events during the transmission process, including transmission start, end, error, and retry information, to facilitate later analysis and debugging.

10. A high-frequency data acquisition and multi-channel parallel transmission system for a large energy storage power station, characterized in that: The method according to claim 1, comprising: The data acquisition module acquires the data of the power station equipment and stores the data in the local system through the acquisition terminal; performs preliminary cleaning of the data; creates a producer thread, sub-packets the high-frequency data and outputs it to the data compression module; The data compression module, based on the input data, completes compression and encapsulation through differential calculation and sparse matrix, obtains data packets and outputs them to the multi-channel parallel transmission module; The multi-channel parallel transmission module puts or takes out data packets from the task queue at the same time based on multiple producer-consumer thread pairs, realizing fast data distribution.

Images