Data storage method and device

Abstract

The invention is suitable for the technical field of big data storage, and provides a data storage method and device. The data storage method is applied to a data storage management system, and the data storage management system comprises a data acquisition server, a data processing server, a data storage service cluster and a time sequence database. The data acquisition server sends the received initial data to the data processing server, and the data processing server performs data extraction processing on the received initial data to obtain necessary data and sends the obtained necessary data to the data storage service cluster; and the data storage service cluster stores the necessary data to the time sequence database. By adopting the distributed storage technology, the processing speed is remarkably improved, the load of the whole system is balanced, single-point overload of the time sequence database is avoided, and the storage pressure of a single node of the time sequence database can be greatly reduced.

Description

Technical Field

[0001] The present application belongs to the field of big data storage technology, and in particular, to a data storage method and device. Background Art

[0002] With the rapid increase in installed capacity of renewable energy sources such as solar photovoltaic (PV) and wind power, the intermittent and unpredictable nature of these energy sources has brought challenges to traditional power grids. In order to better integrate renewable energy, efficient energy storage systems are needed to smooth output fluctuations, reduce peak loads and fill valleys, thereby improving the stability and reliability of the power grid. With the opening up and intensified competition in the power market, power companies need to manage energy resources more flexibly. Energy storage power stations can not only help power companies reduce costs, but also participate in frequency regulation services, spare capacity supply, etc. in the power market to create additional income. The advancement of information technology and the development of smart grid technology have promoted the intelligence of power systems. As the core control system of energy storage power stations, the energy management system (EMS) of energy storage power stations can realize intelligent scheduling and optimization of energy storage equipment and improve energy utilization efficiency. As the capacity of energy storage power stations increases and the amount of equipment increases, the amount of data generated also increases. EMS needs to process a large amount of real-time data and optimize the operation strategy of energy storage systems through technologies such as data analysis and machine learning.

[0003] As the control core of the energy storage system, EMS bears the responsibility of charging and discharging management of all energy storage system units and is applicable to various energy storage application scenarios, such as the grid side, power generation side, user side, etc. In large-capacity energy storage systems on the grid side, such as 500MW / 1GWH, energy storage battery data is particularly important. The second-level storage of battery data can accurately reproduce the operating status of the battery, providing a data basis for the stable and safe operation of the power station.

[0004] In existing storage technologies, relational databases, such as open source MySQL databases and domestic databases, are usually used to store historical data such as telemetry, telesignaling, and electricity consumption. With the continuous increase in the capacity of energy storage power stations and the significant increase in the number of equipment, the amount of data has also increased dramatically, which has posed new challenges to existing data storage technologies. Traditional storage solutions often fail to meet the growing demand for big data processing and storage, especially with the increasing demand for data sophistication. The traditional technical solution of writing to relational databases through scheduled tasks can no longer meet storage needs. In order to effectively address this bottleneck, more advanced storage technologies and architectures need to be adopted. Summary of the invention

[0005] The embodiments of the present application provide a data storage method and device, which can solve the technical problem in the prior art that it is difficult to meet the large data storage requirements by writing into a relational database through scheduled tasks.

[0006] In a first aspect, an embodiment of the present application provides a data storage method, which is applied to a data storage management system, wherein the data storage management system includes a data acquisition server, a data processing server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes:

[0007] The data acquisition server sends the received initial data to the data processing server, wherein the initial data is sent to the data acquisition server by one or more energy storage devices;

[0008] The data processing server performs data extraction processing on the received initial data to obtain necessary data, and sends the obtained necessary data to the data storage service cluster;

[0009] The data storage service cluster stores the necessary data in a time series database.

[0010] In a possible implementation manner of the first aspect, the data processing server performs data extraction processing on the received initial data to obtain necessary data, including:

[0011] The data processing server performs data extraction processing on the received initial data according to a predefined data structure to obtain the necessary data.

[0012] In a possible implementation manner of the first aspect, after sending the obtained necessary data to the data storage service cluster and before the data storage service cluster stores the necessary data in the time series database, the method further includes:

[0013] The data storage service cluster writes the necessary data to one or more cache files;

[0014] The data storage service cluster reads the required data from the one or more cache files.

[0015] In a possible implementation manner of the first aspect, the data storage service cluster writes the required data to one or more cache files, including:

[0016] The data storage service cluster determines whether the sum of the data to be stored and the data stored in the current cache file is less than the size of the cache file;

[0017] If it is smaller, write the current cache file;

[0018] If it is greater, write to the next cache file.

[0019] In a possible implementation manner of the first aspect, the data storage service cluster writes the required data to one or more cache files, further comprising:

[0020] Record the written sequence number of the current cache file;

[0021] Before the data storage service cluster reads the necessary data from the one or more cache files, the data storage service cluster further includes:

[0022] Determine whether the sequence number of the current cache file to be read is less than or equal to the written sequence number. If it is less than or equal to the written sequence number, read the current cache file.

[0023] In a possible implementation manner of the first aspect, the method further includes:

[0024] When a data query request is received, the data storage server corresponding to the data to be queried is determined according to the correspondence between the identification information of the stored data and the data storage server, wherein the data query request includes the identification information of the data to be queried.

[0025] In a second aspect, an embodiment of the present application provides a data storage method, which is applied to a data processing server in a data storage management system, wherein the data storage management system further includes a data acquisition server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes:

[0026] The received initial data is subjected to data extraction processing to obtain necessary data, and the obtained necessary data is sent to the data storage service cluster.

[0027] In a third aspect, an embodiment of the present application provides a data storage method, which is applied to a data storage service cluster in a data storage management system, wherein the data storage management system further includes a data acquisition server, a data processing server, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes:

[0028] Store the required data in a time series database.

[0029] In a fourth aspect, an embodiment of the present application provides a data storage device, wherein the data storage device is included in a data processing server in a data storage management system, wherein the data storage management system further includes a data acquisition server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage device includes:

[0030] The data extraction module is used to perform data extraction processing on the received initial data to obtain necessary data, and send the obtained necessary data to the data storage service cluster.

[0031] In a fifth aspect, an embodiment of the present application provides a data storage device, which is applied to a data storage service cluster in a data storage management system, wherein the data storage management system further includes a data acquisition server, a data processing server, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage device includes:

[0032] The data storage module is used to store the necessary data in the time series database.

[0033] In a sixth aspect, an embodiment of the present application provides a network device, comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein when the processor executes the computer program, it implements the data storage method as described in the second or third aspect above.

[0034] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the data storage method as described in the second or third aspect above.

[0035] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when executed on a mobile terminal, enables the mobile terminal to implement the data storage method of the second or third aspect described above.

[0036] It can be understood that the beneficial effects of the second to eighth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0037] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: In the present application, the data storage management system includes a data acquisition server, a data processing server, a data storage service cluster, and a time series database. Through a new storage architecture solution, the data acquisition server sends the received initial data to the data processing server, and the data processing server extracts and processes the received initial data to obtain the necessary data, and sends the obtained necessary data to the data storage service cluster, and the data storage service cluster stores the necessary data in the time series database. By adopting distributed storage technology, the processes are deployed in three types of functional node servers, namely, data acquisition server, data processing server, and data storage service cluster, respectively, for parallel processing, which significantly improves the processing speed, balances the load of the entire system, avoids single-point overload of the time series database, and can greatly reduce the storage pressure of a single node of the time series database. By extracting and processing the received initial data by the data processing server, useless data information can be eliminated, thereby reducing the amount of necessary data obtained, thereby meeting the necessary information granularity requirements, retaining key necessary data, reducing data storage pressure, and improving data storage efficiency. The data storage service cluster can improve processing speed and throughput, share the processing pressure of each data processing service, greatly shorten the processing time of huge data sets, and the data storage service cluster nodes can be easily expanded to multiple nodes, which improves the system's robustness, processing efficiency, query efficiency and scalability. The data storage management system in this application can, for example, be an energy management system (EMS) for energy storage power stations, or be applied to an energy management system (EMS) for energy storage power stations. In order to meet the challenges brought by energy transformation, this application can improve the efficiency and reliability of the energy storage system, with more complete functions and a wider range of applications, providing strong support for the full data storage of ultra-large energy storage power stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a structural schematic diagram of a data storage management system according to an embodiment of the present application;

[0040] Figure 2 It is a flowchart of a data storage method according to an embodiment of the present application;

[0041] Figure 3A schematic diagram of a data storage method according to an application embodiment of the present application;

[0042] Figure 4 This is a schematic diagram of experimental results obtained by conducting an experiment according to an application example of the present application;

[0043] Figure 5 It is a structural diagram of a network device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0044] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0045] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0046] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0047] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0048] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0049] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0050] This application is mainly to solve the storage pain points of large-capacity energy storage platforms, as follows:

[0051] (1) The data volume of ultra-large energy storage power stations is huge, and the number of measurement points can reach millions. For sites with high time precision requirements, the pressure of storing daily historical data is huge. A single node of a time series database cannot support the storage of such a large amount of data.

[0052] (2) During the peak charging and discharging period of ultra-large energy storage power stations, the impact of changing telemetry is huge. For this scenario, it is very difficult for the time series database to record holographic data (that is, all changing telemetry data).

[0053] (3) For historical data queries, for a single-node time series database, if you choose to query multiple measurement points or query data with a longer time span, the query efficiency of large amounts of data is low.

[0054] This application proposes a data storage method, which is applied to a data storage management system. This application belongs to the field of big data storage and can be applied to application scenarios such as large energy storage stations and large merchant management platforms. For simplicity, this application is mostly explained by taking the application scenario of large energy storage stations as an example.

[0055] Figure 1 The figure is a schematic diagram of the structure of a data storage management system according to an embodiment of the present application.

[0056] like Figure 1 As shown, the data storage management system 12 includes a data acquisition server 121, a data processing server 122, a data storage service cluster 123, and a time series database 124. The data storage service cluster includes one or more data storage servers.

[0057] The present invention adopts distributed storage technology and utilizes the different functionalities of processes to deploy the processes on three types of functional node servers: data acquisition server, data processing server, and data storage service cluster.

[0058] Figure 2 The figure is a flowchart of a data storage method according to an embodiment of the present application.

[0059] like Figure 2 As shown, S21, the data acquisition server 121 sends the received initial data to the data processing server 122, wherein the initial data is sent to the data acquisition server by one or more energy storage devices.

[0060] Here, the data acquisition server 121 may receive initial data respectively sent by one or more energy storage devices. For example, the data acquisition server 121 may receive initial data respectively sent by the first energy storage device 111, the second energy storage device 112, ..., and the Nth energy storage device 11N.

[0061] The data acquisition server 121 can establish a data connection channel with the energy storage device through protocols such as 61850mms or 104. Each energy storage sub-unit in the energy storage device generally adopts AB network dual channels to ensure the standby state of the channel. The energy storage sub-unit includes but is not limited to the energy storage converter (Power Conversion System, referred to as PCS) and the battery management system (Battery Management System, referred to as BMS).

[0062] The data acquisition server 121 can start the front-end communication process. When the process is initialized, the data upload method can be set. The data upload method includes but is not limited to change upload, periodic upload, general call upload, etc. After the communication connection is established, each energy storage subunit (BMS or PCS) uploads initial data to the EMS through a pre-defined data upload method.

[0063] The initial data may be in the form of message data. The front receiving module in the data acquisition server 121 receives the message data sent by one or more energy storage devices. Then, the message data of each channel is packaged and sent to the data processing server 122.

[0064] S22 , the data processing server 122 performs data extraction processing on the received initial data to obtain necessary data, and sends the obtained necessary data to the data storage service cluster 123 .

[0065] Here, the data processing server 122 can reduce the amount of necessary data obtained by performing data extraction processing on the initial data and eliminating useless data information, thereby meeting the data information granularity requirements, retaining key necessary data, reducing data storage pressure, and improving data storage efficiency.

[0066] In one embodiment, the data processing server performs data extraction processing on the received initial data according to a predefined data structure to obtain the necessary data.

[0067] Here, the present application defines the necessary information in the initial data by defining the data storage information structure. The data processing server 122 performs data analysis on the received initial data, and packages the parsed initial data into a defined data storage information structure data packet (i.e., the necessary data), and sends the data packet to the data storage service cluster 123.

[0068] The data processing server 122 can create each channel instance, and each channel instance will package the initial data parsed by its own channel according to the defined data structure. The data processing server 122 sends the packaged data structure package (ie, necessary data) in the queue to the data storage service cluster 123.

[0069] Taking the definition of telemetry data field structure as an example, the predefined data structure includes at least one of the following information: string identifier, time information, signal point type, engineering data, and raw data. The predefined data structure is shown in the following Table 1:

[0070]

[0071] Table 1

[0072] "name" indicates the string identifier of the signal point, which is used as the primary key in the database. "sdate" and "time" jointly indicate the specific time information of the signal point (for example, it can be the timestamp information of the device data change). "flag" indicates whether the signal point is an identification bit (indicating the data type), for example, whether the signal point is a calculated value. "data" indicates the engineering data after processing. "rawdata" indicates the raw data sent by the device.

[0073] In the present application, the data processing server performs data extraction processing on the received initial data according to a predefined data structure to obtain the required data. Since the predefined data structure defines the necessary information of the historical data, useless data information can be eliminated, the data information granularity requirements can be met, key required data can be retained, the data storage pressure can be reduced, and the efficiency of data storage can be improved.

[0074] S23 , the data storage service cluster 123 stores the necessary data in the time series database 124 .

[0075] Here, the data storage service cluster 123 may include one or more data storage servers. For the data storage service cluster 123 and the downstream device, the data storage server and the device use the ntp clock synchronization method to ensure that the time of each data storage server is consistent with the device time. For device data, the data packet structure saves the device data change timestamp information, and uses the device change timestamp as the change data storage time, which can ensure that the data of each data storage server in the data storage service cluster 123 is stored in the data storage service cluster 123 according to the device change data order.

[0076] In this application, the data storage management system includes a data acquisition server, a data processing server, a data storage service cluster, and a time series database. Through a new storage architecture solution, the data acquisition server sends the received initial data to the data processing server, and the data processing server extracts and processes the received initial data to obtain the necessary data, and sends the obtained necessary data to the data storage service cluster, and the data storage service cluster stores the necessary data in the time series database. By adopting distributed storage technology, the processes are deployed in three types of functional node servers, namely, data acquisition server, data processing server, and data storage service cluster, for parallel processing, which significantly improves the processing speed, balances the load of the entire system, avoids single-point overload of the time series database, and can greatly reduce the storage pressure of a single node of the time series database. By extracting and processing the received initial data through the data processing server, useless data information can be eliminated, thereby reducing the amount of data obtained by the necessary data, thereby meeting the necessary information granularity requirements, retaining key necessary data, reducing data storage pressure, and improving data storage efficiency. The data storage service cluster can improve processing speed and throughput, share the processing pressure of each data processing service, greatly shorten the processing time of huge data sets, and the data storage service cluster nodes can be easily expanded to multiple nodes, which improves the system's robustness, processing efficiency, query efficiency and scalability. The data storage management system in this application can, for example, be an energy management system (EMS) for energy storage power stations, or be applied to an energy management system (EMS) for energy storage power stations. In order to meet the challenges brought by energy transformation, this application can improve the efficiency and reliability of the energy storage system, with more complete functions and a wider range of applications, providing strong support for the full data storage of ultra-large energy storage power stations.

[0077] Under the same conditions, this application scheme greatly improves the efficiency of data insertion and data query. The new technology solution for historical data storage can insert 50,000 historical data per second, which is 10 times higher than the existing technology that can only insert 5,000 data using traditional relational databases. The query efficiency can be increased from about 30s to about 1-2s, greatly improving the overall query efficiency.

[0078] In one embodiment, after sending the acquired necessary data to the data storage service cluster and before the data storage service cluster stores the necessary data in the time series database, the data storage method further includes:

[0079] The data storage service cluster writes the necessary data to one or more cache files;

[0080] The data storage service cluster reads the required data from one or more cache files.

[0081] The number of cache files that can be enabled can be configured on the EMS data storage service cluster. The number ranges from 1,000 to 100,000, and can generally be set to 10,000.

[0082] Here, the cache file can improve the data storage and processing capabilities during data peaks. For a single-node time series database, the efficiency of a single storage is limited. For the storage of changing data during the peak period of charging and discharging, a single node of the time series database cannot meet the requirements of full holographic data storage. The present application uses caching technology to store the peak change data in a cache temporary file, and then uses a thread pool to write the data packets of the necessary data in the cache file into the time series database table in a multi-threaded manner. In this way, the peak data can be stored in a temporary cache file, which can avoid data loss in the time series database due to storage performance not keeping up. The caching method can effectively avoid the huge impact of peak data storage, and when the process is abnormal, it can be retained in the data storage service cluster in the form of a file, and after it returns to normal, data catch-up storage can be performed to effectively avoid data loss.

[0083] In one embodiment, the data storage service cluster writes the necessary data to one or more cache files, including:

[0084] The data storage service cluster determines whether the sum of the data to be stored and the data stored in the current cache file is less than the size of the cache file;

[0085] If it is smaller, write the current cache file;

[0086] If it is greater, write to the next cache file.

[0087] The maximum number of bytes of a single cache file can be configured on the EMS data storage service cluster. The maximum number of bytes ranges from 100MB to 1GB, and can generally be set to 500MB.

[0088] For example, the data storage service cluster 123 may record the total number of bytes stored in the current cache file. Before writing the necessary data into the current cache file, the sum of the number of bytes stored and the necessary data (raw data) read is calculated. If it is less than the size of the cache file (e.g., 500MB), the necessary data is written into the current cache file. If the sum of the number of bytes stored and the necessary data (raw data) read is greater than the size of the cache file (e.g., 500MB), the next cache file is written.

[0089] In this application, the data storage service cluster compares the sum of the data to be stored and the data already stored in the current cache file with the size of the cache file, which can avoid write errors and ensure the effectiveness and reliability of writing the necessary data into the cache file.

[0090] In one embodiment, the data storage service cluster writes the required data to one or more cache files, further comprising:

[0091] Record the written sequence number of the current cache file;

[0092] Before the data storage service cluster reads the necessary data from the one or more cache files, the data storage service cluster further includes:

[0093] Determine whether the sequence number of the current cache file to be read is less than or equal to the written sequence number. If it is less than or equal to the written sequence number, read the current cache file.

[0094] Here, the prefix of the cache file name is configured on the data storage service cluster, for example, it can be set to cacheData, the actual cache file name = cacheData + cache file write sequence number i, the write sequence number i starts from 0 and increases sequentially. For example, if the sum of the data to be stored and the data stored in the current cache file is greater than the size of the cache file, then the next cache file is written, and i++. When the data storage service cluster writes data to the first cache file cacheData i (where the write sequence number i=0), the total number of bytes stored in the cache file at this time can be recorded as M.

[0095] In the data storage service cluster, starting from the first cache file, the cache file contents are read in sequence. The cache file read sequence number is j, and the initial value j = 0. In the data storage service cluster, the thread pool in the data storage process obtains the necessary data (data packets) from the cache file in parallel and inserts them into the time series database in sequence. Each subsequent read of the cache file must ensure that the current read sequence number j <= i.

[0096] This application ensures that data is not lost, data can be efficiently processed and written, and information integrity can be guaranteed by having a data storage service cluster write cache files and using multi-threading technology to simultaneously obtain necessary data in the cache files and write them into a time series database.

[0097] In one embodiment, the data storage method further includes:

[0098] When a data query request is received, the data storage server corresponding to the data to be queried is determined according to the correspondence between the identification information of the stored data and the data storage server, wherein the data query request includes the identification information of the data to be queried.

[0099] Here, the corresponding relationship between the identification information of the stored data (such as the type information of the stored data) and the data storage server can be saved in a certain server in the data storage management system, such as a certain data storage server in the data storage service cluster, or an operator station server in the data storage management system. The corresponding relationship between the identification information of the stored data and the data storage server.

[0100] In terms of query methods, by adopting the above data routing method, the stored data can be directed to the corresponding data storage server, and the stored data that matches the query conditions of the data to be queried can be obtained from the data storage server in the data storage server cluster. For a single-node time series database, the data storage service cluster reduces the database pressure when querying data and can improve query efficiency. Under the same conditions, the present application scheme greatly improves the data query efficiency. The query efficiency can be improved from the original 30s to about 1 to 2S, greatly improving the overall query efficiency.

[0101] Figure 3 The figure is a flow chart of a data storage method according to an application embodiment of the present application.

[0102] Before you start, you can do some pre-configuration work. Pre-configuration work includes:

[0103] (1) Configure the number of cache files to be enabled on the EMS data storage service cluster, which is generally set to 10,000.

[0104] (2) Configure the maximum size of a single cache file on the EMS data storage service cluster, which is generally set to 500 MB.

[0105] (3) Configure the prefix of the cache file name on the EMS data storage service cluster. It is generally set to cacheData. The actual cache file name = cacheData + cache file write sequence number i. The write sequence number i starts from 0 and increases sequentially.

[0106] (4) On the EMS data storage service cluster, create a new cache file as specified in the above steps.

[0107] (5) On the EMS front-end server, a data connection channel is established with the energy storage device through protocols such as 61850mms or 104. Each energy storage sub-unit (PCS or BMS) generally adopts AB network dual channels to ensure the standby status of the channel.

[0108] After the start, S301, the time series database storage mode is set to compressed storage.

[0109] S302, start the time series database service.

[0110] S303, determining whether the time series database service is started successfully.

[0111] S3031: If it fails, the prompt "Time Series Database Startup Failed" will be displayed.

[0112] S304, if successful, start training all channels in turn and open all data channels.

[0113] S305, determining whether the data channel is opened successfully.

[0114] S3051, if unsuccessful, the prompt is "Failed to open the channel".

[0115] S306, if successful, start the front-end communication process on the EMS front-end server (i.e., data acquisition server). When the process is initialized, the data upload method will be set, mainly including change upload, periodic upload, general call upload, etc.

[0116] S307, determining whether the setting of the data sending mode is successful.

[0117] S3071, prompt "Failed to set data upload mode".

[0118] S308, the EMS front-end data receiving module receives the data sent by the energy storage unit. After the communication connection is established, each energy storage subunit (BMS or PCS) sends data to the EMS through a pre-defined data sending method. On the EMS front-end service module, the front-end receiving module receives the data message.

[0119] S309, the front-end server packages the message data of each channel and sends it to the data processing server.

[0120] S310, each channel instance created by the data processing server will package the data parsed by its own channel according to the defined data structure.

[0121] S311, the data processing service module sends the packed data structure package in the queue to the data cache receiving module.

[0122] S312, the EMS data storage service cluster takes out the received data cache package and puts it into the cache file.

[0123] S313, directly write data to the first cache file cacheData i (wherein the write sequence number i=0), and record the total number of bytes stored in the cache file as M at this time.

[0124] S314, each time the cache file is written, the sum of the number of bytes stored and the number of bytes of raw data read needs to be calculated. If it is greater than 500MB, the next cache file needs to be written, and i++.

[0125] S315, repeat the previous step.

[0126] S316, on the EMS data storage service cluster, starting from the first cache file, read the cache file contents in sequence, the cache file reading sequence number is j, and the initial value j=0.

[0127] S317, determining whether the message pulling is successful.

[0128] If S3171 fails, the message "Data storage service cluster failed to pull message" is displayed.

[0129] S318, in the EMS data storage service cluster, the thread pool in the data storage process obtains data packets from the cache file in parallel and inserts them into the time series database in sequence. Each subsequent read of the cache file must ensure that the current read sequence number j <= i until the end.

[0130] Figure 4 The figure is a schematic diagram of experimental results obtained by conducting an experiment according to an application example of the present application.

[0131] Figure 4 The figure shows the summary data of the experimental results. The horizontal axis represents the data sending time under the same conditions, and the vertical axis represents the hard disk size occupied.

[0132] Before the experiment, the compression storage algorithm of the time series database is set to TSZ compression algorithm. TSZ compression algorithm is a more abundant compression function provided by TDengine for floating point data types, which can realize the full state compression of floating point numbers from lossy to lossless. Compared with the original compression algorithm in TDengine, TSZ compression algorithm has richer compression options and higher compression rate. Even if floating point numbers are compressed in lossless state, the compression rate will be twice as high as the original compression algorithm. Set the data file directory to / home / taos / data, all historical data are stored in this directory, and all historical data are cleared. The simulation tool runs 30 PCS and 30 BMS devices. All PCS share one icd (IED Configuration Description, the Chinese name is Intelligent Electronic Device Configuration Description File, where IED is the full name of Intelligent Electronic Device, that is, Intelligent Electronic Device), all BMS share one icd, and all telemetry data are changed and uploaded at the same time. Each PCS icd includes 81 telemetry points; each BMS includes 1002 telemetry points. EMS communicates with PCS and BMS using the 61850mms protocol, and at the beginning of the channel, the telemetry data upload mode is set to change upload, and the telemeter signal is not uploaded. In the simulation tool, the PCS and BMS data change upload cycle is set to 1s, and the running time is 10min. After all channels of EMS communicate normally, capture packets in turn to confirm that all channel data upload is normal and the data storage process is normal.

[0133] The telemetry data field structure is defined as shown in Table 2 below:

[0134]

[0135] Table 2

[0136] "name" indicates the string identifier of the signal point, which is used as the primary key in the database. "sdate" and "time" jointly indicate the specific time information of the signal point (for example, it can be the timestamp information of the device data change). "flag" indicates whether the signal point is an identification bit (indicating the data type), for example, whether the signal point is a calculated value. "data" indicates the engineering data after processing. "rawdata" indicates the raw data sent by the device.

[0137] After the simulation tool has finished sending data and the data has been stored, log in to the EMS server. You can use the command du-sh / home / taos / vnode--exclude='wal' to check the specific historical data file size, and the query result is 72MB. Under the same circumstances, when using MySQL database to store data, use the command select concat(round(sum(DATA_LENGTH / 1024 / 1024),2),'MB')as data from TABLES to check the specific historical data file size, which is 1072MB.

[0138] An embodiment of the present application provides a data storage method, which is applied to a data acquisition server in a data storage management system, wherein the data storage management system includes a data acquisition server, a data processing server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes:

[0139] The received initial data is sent to a data processing server, wherein the initial data is sent to the data acquisition server by one or more energy storage devices.

[0140] An embodiment of the present application provides a data storage method, which is applied to a data processing server in a data storage management system, wherein the data storage management system includes a data acquisition server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes:

[0141] The received initial data is subjected to data extraction processing to obtain necessary data, and the obtained necessary data is sent to the data storage service cluster.

[0142] In one embodiment, performing data extraction processing on the received initial data includes:

[0143] The received initial data is subjected to data extraction processing according to a predefined data structure to obtain necessary data.

[0144] An embodiment of the present application provides a data storage method, which is applied to a data storage service cluster in a data storage management system, wherein the data storage management system includes a data acquisition server, a data processing server, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes:

[0145] Store the required data in a time series database.

[0146] In one embodiment, before storing the necessary data in the time series database, the method further includes:

[0147] Writing the necessary data to one or more cache files;

[0148] The required data is read from one or more cache files.

[0149] In one embodiment, writing the necessary data to one or more cache files includes:

[0150] Determine whether the sum of the data to be stored and the data already stored in the current cache file is less than the size of the cache file;

[0151] If it is smaller, write the current cache file;

[0152] If it is greater, write to the next cache file.

[0153] In one embodiment, writing the necessary data to one or more cache files further includes:

[0154] Record the written sequence number of the current cache file;

[0155] Before reading the necessary data from one or more cache files, the method further includes:

[0156] Determine whether the sequence number of the current cache file to be read is less than or equal to the written sequence number. If it is less than or equal to the written sequence number, read the current cache file.

[0157] In one embodiment, the method further comprises:

[0158] When a data query request is received, the data storage server corresponding to the data to be queried is determined according to the correspondence between the identification information of the stored data and the data storage server, wherein the data query request includes the identification information of the data to be queried.

[0159] An embodiment of the present application provides a data storage method, which is applied to an operator station server in a data storage management system, wherein the data storage management system includes a data acquisition server, a data storage service cluster, a time series database, and an operator station server, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes:

[0160] When a data query request is received, the data storage server corresponding to the data to be queried is determined according to the correspondence between the identification information of the stored data and the data storage server, wherein the data query request includes the identification information of the data to be queried.

[0161] An embodiment of the present application provides a data storage device, which is applied to a data acquisition server in a data storage management system, wherein the data storage management system includes a data acquisition server, a data processing server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage device includes:

[0162] The data forwarding module is used to send the received initial data to the data processing server, wherein the initial data is sent to the data acquisition server by one or more energy storage devices.

[0163] An embodiment of the present application provides a data storage device, which is applied to a data processing server in a data storage management system, wherein the data storage management system includes a data acquisition server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage device includes:

[0164] The data extraction module is used to perform data extraction processing on the received initial data to obtain necessary data, and send the obtained necessary data to the data storage service cluster.

[0165] In one embodiment, the data extraction module is used to:

[0166] The received initial data is subjected to data extraction processing according to a predefined data structure to obtain necessary data.

[0167] An embodiment of the present application provides a data storage device, which is applied to a data storage service cluster in a data storage management system, wherein the data storage management system includes a data acquisition server, a data processing server, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage device includes:

[0168] The data storage module is used to store the necessary data in the time series database.

[0169] In one embodiment, the apparatus further comprises:

[0170] The cache read-write module is used to write the necessary data into one or more cache files; and read the necessary data from one or more cache files.

[0171] In one embodiment, the cache read-write module is used to:

[0172] Determine whether the sum of the data to be stored and the data already stored in the current cache file is less than the size of the cache file;

[0173] If it is smaller, write the current cache file;

[0174] If it is greater, write to the next cache file.

[0175] In one embodiment, the cache read-write module is further used for:

[0176] Record the written sequence number of the current cache file;

[0177] Determine whether the sequence number of the current cache file to be read is less than or equal to the written sequence number. If it is less than or equal to the written sequence number, read the current cache file.

[0178] In one embodiment, the apparatus further comprises:

[0179] The data query module is used to determine the data storage server corresponding to the data to be queried according to the corresponding relationship between the identification information of the stored data and the data storage server when receiving a data query request, wherein the data query request includes the identification information of the data to be queried.

[0180] An embodiment of the present application provides a data storage device, which is applied to an operator station server in a data storage management system. The data storage management system includes a data acquisition server, a data storage service cluster, a time series database, and an operator station server. The data storage service cluster includes one or more data storage servers. The data storage device includes:

[0181] The data query module is used to determine the data storage server corresponding to the data to be queried according to the corresponding relationship between the identification information of the stored data and the data storage server when receiving a data query request, wherein the data query request includes the identification information of the data to be queried.

[0182] An embodiment of the present application provides a data storage management system, the data storage management system includes a data acquisition server, a data processing server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage system includes:

[0183] A data acquisition server, used for sending the received initial data to a data processing server, wherein the initial data is sent to the data acquisition server by one or more energy storage devices;

[0184] The data processing server is used to perform data extraction processing on the received initial data to obtain necessary data, and send the obtained necessary data to the data storage service cluster;

[0185] The data storage service cluster is used to store the required data in a time series database.

[0186] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0187] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0188] The present application also provides a network device, such as Figure 5 As shown, the network device 5 includes: at least one processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, and when the processor 50 executes the computer program 52, the steps in any of the above-mentioned method embodiments are implemented.

[0189] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0190] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0192] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0193] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0194] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data storage method, characterized in that: The data storage method is applied to a data storage management system, which includes a data acquisition server, a data processing server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes: The data acquisition server sends the received initial data to the data processing server, wherein the initial data is sent to the data acquisition server by one or more energy storage devices; The data processing server performs data extraction processing on the received initial data to obtain necessary data, and sends the obtained necessary data to the data storage service cluster; The data storage service cluster stores the necessary data in a time series database.

2. The data storage method according to claim 1, characterized in that: The data processing server performs data extraction processing on the received initial data to obtain necessary data, including: The data processing server performs data extraction processing on the received initial data according to a predefined data structure to obtain the necessary data.

3. The data storage method according to claim 1, characterized in that: After sending the acquired necessary data to the data storage service cluster and before the data storage service cluster stores the necessary data in the time series database, the method further includes: The data storage service cluster writes the necessary data to one or more cache files; The data storage service cluster reads the required data from the one or more cache files.

4. The data storage method according to claim 3, characterized in that: The data storage service cluster writes the required data to one or more cache files, including: The data storage service cluster determines whether the sum of the data to be stored and the data stored in the current cache file is less than the size of the cache file; If it is smaller, write the current cache file; If it is greater, write to the next cache file.

5. The data storage method according to claim 3 or 4, characterized in that: The data storage service cluster writes the necessary data into one or more cache files, further comprising: Record the written sequence number of the current cache file; Before the data storage service cluster reads the necessary data from the one or more cache files, the data storage service cluster further includes: Determine whether the sequence number of the current cache file to be read is less than or equal to the written sequence number. If it is less than or equal to the written sequence number, read the current cache file.

6. The data storage method according to claim 1, characterized in that: The method further includes: When a data query request is received, the data storage server corresponding to the data to be queried is determined according to the correspondence between the identification information of the stored data and the data storage server, wherein the data query request includes the identification information of the data to be queried.

7. A data storage method, characterized in that: The data storage method is applied to a data processing server in a data storage management system, wherein the data storage management system further comprises a data acquisition server, a data storage service cluster, and a time series database, wherein the data storage service cluster comprises one or more data storage servers, and the data storage method comprises: The received initial data is subjected to data extraction processing to obtain necessary data, and the obtained necessary data is sent to the data storage service cluster.

8. A data storage method, characterized in that: The data storage method is applied to a data storage service cluster in a data storage management system, wherein the data storage management system further includes a data acquisition server, a data processing server, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage method includes: Store the required data in a time series database.

9. A data storage device, characterized in that: The data storage device is included in a data processing server in a data storage management system, and the data storage management system also includes a data acquisition server, a data storage service cluster, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage device includes: The data extraction module is used to perform data extraction processing on the received initial data to obtain necessary data, and send the obtained necessary data to the data storage service cluster.

10. A data storage device, characterized in that: The data storage device is applied to a data storage service cluster in a data storage management system, and the data storage management system further includes a data acquisition server, a data processing server, and a time series database, wherein the data storage service cluster includes one or more data storage servers, and the data storage device includes: The data storage module is used to store the necessary data in the time series database.

Images