A method, apparatus, and related product for industrial data storage caching

Abstract

The application provides a method, device and related product for industrial data storage cache. The method comprises: calculating the historical data amount of a current measuring point and the data amount written per second according to the continuously written historical data. Then, the historical data amount of the current measuring point is compared with the current buffer space size or the configured refresh time. If the current buffer space size is exceeded or the configured refresh time is reached, the currently buffered data is written to the disk, and the current buffer space is updated according to the data amount written per second and the data type of the current measuring point. In this way, the buffer space size is dynamically adjusted according to the data type and data flow of the measuring point, thereby avoiding the problem of waste of buffer space caused by the measuring point with unchanged data constantly occupying the buffer space, and improving the resource utilization rate of the buffer space.

Description

Technical Field

[0001] This application relates to the field of large-scale real-time historical databases for industry, and in particular to a method, apparatus and related products for industrial data storage caching. Background Technology

[0002] In large-scale industrial real-time historical database applications, it is necessary to continuously collect real-time data from large industrial sites and store it in the historical database. Historical data buffering technology has become an important component of data storage due to its ability to improve write performance and reduce disk I / O. Historical data buffering refers to temporarily storing measurement point data in a buffer space; when the buffered data reaches a preset threshold, the measurement point data is then integrated and written to disk.

[0003] Existing technologies often allocate a fixed-size buffer space based on the number of data measurement points. However, in practical applications, not all measurement point data changes in real time. Therefore, allocating a fixed-size buffer space for each measurement point will cause measurement points with unchanging data to continuously occupy the buffer space, resulting in wasted cache space. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus and related products for industrial data storage caching, which aims to dynamically adjust the size of the buffer space according to the data type and data flow of the measurement points, thereby avoiding measurement points with unchanged data from occupying the buffer space continuously and improving the resource utilization of the buffer space.

[0005] In a first aspect, this application provides a method for industrial data storage caching, applied to a real-time historical database, the method comprising:

[0006] In response to the continuously written historical data, calculate the amount of historical data buffered at the current measurement point and the amount of data written per second at the current measurement point;

[0007] In response to the historical data volume of the current measurement point buffer exceeding the size of the current buffer space, and / or reaching the configured refresh time, the data in the current buffer is written to the disk, and the current buffer space is updated according to the data volume written per second and data type of the current measurement point; the current buffer space is used to buffer the data of the current measurement point.

[0008] Optionally, updating the buffer space based on the current data write volume per second and data type at the current measurement point includes:

[0009] The number of bytes written per second is determined based on the amount of data written per second and the data type of the current measurement point;

[0010] The buffer space is updated based on the number of bytes written per second and the preset mapping relationship between the buffer space and the number of bytes written per second.

[0011] Optionally, the preset mapping relationship between the buffer space and the number of bytes written per second specifically includes:

[0012] Buffer space = cache switching time * bytes written per second * first random number coefficient.

[0013] Optionally, the method further includes, prior to responding to the continuously written historical data:

[0014] In response to the first launch of the historical archive service, a fixed-size buffer space is created based on the data type of the measurement point and the preset mapping relationship between the buffer space and the data type.

[0015] The fixed-size buffer space created will be used to determine the current buffer space.

[0016] Optionally, the preset mapping relationship between the buffer space and the data type specifically includes:

[0017] Buffer space = data type * second random number coefficient.

[0018] Secondly, this application provides an industrial data storage caching device for use in a real-time historical database, the device comprising:

[0019] The calculation unit is used to calculate the amount of historical data buffered at the current measurement point and the amount of data written per second at the current measurement point in response to the continuously written historical data.

[0020] The write unit is used to write the data in the current buffer to the disk in response to the historical data volume of the previous test point buffer exceeding the size of the current buffer space and / or reaching the configured refresh time, and to update the current buffer space according to the data volume written per second and data type of the current test point; the current buffer space is used to buffer the data of the current test point.

[0021] Optionally, the writing unit includes:

[0022] The byte count determination module is used to determine the number of bytes written per second based on the amount of data written per second and the data type of the current measurement point;

[0023] The update module is used to update the buffer space based on the number of bytes written per second and the preset mapping relationship between the buffer space and the number of bytes written per second.

[0024] Optionally, the preset mapping relationship between the buffer space and the number of bytes written per second specifically includes:

[0025] Buffer space = Cache switching time * Bytes written per second * First random number coefficient

[0026] Thirdly, embodiments of this application provide an electronic device. The device includes a memory and a processor.

[0027] The memory is used to store program code and transfer program code to the processor.

[0028] Processor: for executing the method steps of the industrial data storage cache described in any one of the first aspects according to instructions in the program code.

[0029] Fourthly, embodiments of this application provide a computer-readable storage medium storing code that, when executed by a processor, implements the steps of the industrial data storage caching method as described in any of the first aspects.

[0030] This application provides a method, apparatus, and related products for industrial data storage caching. When executing the method, firstly, based on continuously written historical data, the historical data volume and data write rate per second of the current measuring point are calculated. Then, it is compared whether the historical data volume of the current measuring point exceeds the current buffer space size or reaches the configured refresh time. If it exceeds the current buffer space size or reaches the configured refresh time, the currently buffered data is written to disk, and the current buffer space is updated according to the data write rate and data type of the current measuring point. In this way, the buffer space size is dynamically adjusted based on the data type and data flow of the measuring point, thereby avoiding the waste of buffer space caused by measuring points with unchanging data continuously occupying buffer space, and thus improving the resource utilization rate of the buffer space. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A schematic diagram illustrating the creation of a fixed-size buffer space based on the number of data measurement points;

[0033] Figure 2 A flowchart illustrating an industrial data storage buffering method provided in this application embodiment;

[0034] Figure 3 A flowchart illustrating another industrial data storage caching method provided in this application embodiment;

[0035] Figure 4 A schematic diagram of cache management provided for an embodiment of this application;

[0036] Figure 5 A flowchart illustrating a third industrial data storage caching method provided in this application embodiment;

[0037] Figure 6 This is a schematic diagram of an industrial data storage cache device provided in an embodiment of this application. Detailed Implementation

[0038] The terms “first,” “second,” “third,” and “fourth,” etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0039] In the embodiments of this application, the terms "as an example" or "for example" are used to indicate that they are examples, illustrations, or explanations. Any embodiment or design that is described as "as an example" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Specifically, the use of terms such as "as an example" or "for example" is intended to present the relevant concepts in a specific manner.

[0040] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0041] As mentioned earlier, existing technologies often allocate a fixed-size buffer space based on the number of data measurement points. See also Figure 1 This diagram illustrates the allocation of a fixed-size buffer space based on the number of data measurement points. Data pushed from the real-time data service to the historical storage service is first archived and buffered. When the buffer reaches a certain threshold, the buffered data is written to the disk to an archive file for storage. The archive buffer space is a fixed-size buffer space allocated based on the number of pushed data points. Clearly, this buffer space includes a large number of empty locations used to store measurement points whose data remains unchanged for a long time. This leads to a waste of cache space resources during actual storage.

[0042] Based on this, this application proposes to dynamically adjust the size of the buffer space by adjusting the data type and data flow of the measurement points, thereby avoiding the waste of buffer space caused by measurement points with unchanged data occupying the buffer space, and thus improving the rational and practical use of buffer space memory resources.

[0043] To enable those skilled in the art to better understand the present application, some technical terms are explained below.

[0044] Data buffering: Data that needs to be written to disk each time is temporarily cached in memory and written to disk in batches and blocks, reducing random disk I / O and improving disk write performance.

[0045] Disk I / O: Generally refers to disk read / write I / O. Read I / O is issuing instructions to read the contents of a certain sector of the disk, and write I / O is issuing instructions to write data into a certain sector of the disk.

[0046] Measurement data: such as temperature, humidity, and pressure inside a chemical reactor. Each data point is recorded as a tag in the industrial real-time historical database. The measurement data includes three parameters: current value, upload time, and data quality code.

[0047] Disk: Computer disk has a larger capacity than computer memory, but its operation speed is slower. However, it can store data for a long time and the data will not be erased after power failure. A typical computer has 500GB or 1TB of disk space.

[0048] Memory: refers to computer memory. Compared to computer disk, it has less space but faster speed. It cannot store data for a long time and the data will be cleared after power failure. Generally, computer memory is 16G or 32G.

[0049] Data type: refers to the numerical type of the data measurement point, such as floating point, integer, string, switch, etc.

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] See Figure 2 This is a flowchart illustrating a method for industrial data storage buffering provided in an embodiment of this application, applied to a large-scale real-time historical industrial database. The method includes at least:

[0052] S201: In response to continuously written historical data, calculate the amount of historical data buffered at the current measurement point and the amount of data written per second at the current measurement point.

[0053] In this embodiment, the historical database needs to continuously collect real-time data from the industrial site, and the collected real-time data is continuously written into the real-time database as historical data. The historical database first needs to calculate the amount of historical data buffered at the current measuring point based on the continuously written historical data. This buffered historical data amount includes the buffer space required for the archive cache and the computer memory occupied by disk I / O. Based on the amount of historical data buffered at the current measuring point, such as the amount of temperature and humidity data inside the buffered chemical reactor, the buffer space occupied by the archive cache is determined.

[0054] In this embodiment, based on continuous historical data, the historical database can calculate the amount of data written per second at the current measurement point. For example, the data write volume at the current measurement point over 1 minute can be obtained, and dividing that data write volume by 60 gives the data write volume per second. Based on the data write volume per second at the current measurement point, combined with the data type, the buffer space can be dynamically adjusted.

[0055] S202: In response to the data write volume per second exceeding the current buffer space size, or reaching the configured refresh time, write the current buffer data to the disk, and update the current buffer space according to the data flow and data type of the current measurement point.

[0056] In this embodiment, after the historical database obtains the buffered data volume and write volume per second for the current measurement point, it first determines whether the buffered data volume for the current measurement point exceeds the size of the current buffer space. The buffer space is used to buffer the data for the current measurement point. If it exceeds the size of the current buffer space, the currently buffered data is written to disk. The current buffer space is dynamically adjusted based on the write volume per second and data type of the current measurement point.

[0057] In this embodiment, a mapping relationship between the buffer space and the write data rate per second and the data type can be pre-established. Based on this mapping relationship, the size of the buffer space can be obtained. For example: Assume the buffer space size is f(x, y), where x represents the write data rate per second and y represents the data type length. Then f(x, y) = axy + c. Here, a is a fixed coefficient, and c is a fixed constant. Thus, determining x and the data type determines x and y, and substituting them into the above formula yields the buffer space size.

[0058] In this embodiment, the current buffer space can be dynamically updated in the following way:

[0059] Based on the amount of data written per second and the data type at the current measurement point, determine the number of bytes written per second. Example: Assuming the amount of data written per second is 60, and the data is floating-point type with a data type length of 4 bytes, then the number of bytes written per second is 60 * 4 = 240 bytes. Update the buffer space based on the above number of bytes written per second and the preset mapping relationship between the buffer space and the number of bytes written per second.

[0060] In this embodiment, considering that most measurement points change at the same frequency, to avoid the problem of large short-term memory usage and impact on computer speed caused by simultaneous writing of measurement point data to the archive file, the preset mapping relationship between buffer space and bytes written per second can be:

[0061] Buffer space = cache switching time * bytes written per second * first random number coefficient.

[0062] The buffer switching time refers to the time required from the start of buffering to the writing of cached data to disk. In this embodiment, the buffer switching time can be a pre-configured refresh time for the computer. For the computer, due to functions such as pre-write logging and power-off recovery, the historical database buffer is periodically written to disk according to the configured time. For example, if the configured time is 30 minutes, the cached data needs to be refreshed and written to disk every 30 minutes. This buffer switching time is 30 minutes. Alternatively, the buffer switching time can also be the buffer time used when memory space is insufficient. The first random coefficient is an arbitrary random number assigned to the current data of each measurement point. Its purpose is to avoid the problem of large short-term memory usage and impact on computer speed caused by measurement point data being written to the archive file at the same time when most measurement point data changes at the same frequency.

[0063] Example Explanation: Assuming a configured refresh time of 30 minutes and consistent data frequency (i.e., the same number of bytes written per second), all data will be written to the archive file simultaneously. However, assigning a random number, such as 0.9, to the current data point will reduce the buffer memory space. The continuous writing of historical data will quickly reach the memory limit, even before the 30-minute refresh time is reached. Therefore, the current data point will be written to disk before the configured refresh time is reached, instead of being written to the document all at once.

[0064] This application provides a method for industrial data storage caching. Specifically, it first calculates the amount of data written per second (bps) at the current measurement point based on continuously written historical data. Then, it compares whether the bps write volume exceeds the current buffer space size or reaches the configured refresh time. If it does, the currently buffered data is written to disk, and the current buffer space is updated based on the bps write volume and data type of the current measurement point. In this way, the buffer space size is dynamically adjusted based on the data type and data flow of the measurement point, thus avoiding the waste of buffer space caused by measurement points with unchanging data continuously occupying it, thereby improving the resource utilization of the buffer space.

[0065] See Figure 3 This is a flowchart illustrating another industrial data storage caching method provided in this application embodiment, applied to a large-scale industrial real-time historical database. This method switches caches when memory space is insufficient. The method includes at least:

[0066] S301: Pre-configure refresh time.

[0067] In this embodiment, refresh time refers to the time it takes to periodically write data to the disk.

[0068] S302: Determine if the history archive service is being started for the first time. If yes, proceed to S303; otherwise, proceed to S305.

[0069] S303: Load the measurement points in the configuration, and create a fixed-size buffer space based on the measurement point data type and the preset mapping relationship between the buffer space and the data type.

[0070] In this embodiment of the application, considering that most measurement point data change at the same frequency, the mapping relationship between the preset buffer space and data type is constructed as follows:

[0071] Buffer space = data type * second random number coefficient.

[0072] Example explanation: If the data type is switch type, when the second random number coefficient is 1, the buffer space is determined to be 128 bytes. If the second random number coefficient changes, the buffer space will also change accordingly. If the data type is floating point type, when the second random number coefficient is 1, the buffer space is 4096 bytes, etc.

[0073] S304: The above-mentioned fixed-size buffer space is determined as the current buffer space.

[0074] S305: In response to continuously written historical data, calculate the amount of historical data buffered at the current measurement point and the amount of data written per second at the current measurement point.

[0075] S306: Determine whether the amount of historical data in the current measurement point buffer exceeds the size of the current buffer space. If yes, proceed to S307; otherwise, proceed to S305.

[0076] S307: Write the current buffered data to disk and update the current buffer space according to the amount of data written per second and the data type of the current measurement point.

[0077] In this embodiment, the number of bytes written per second is determined based on the amount and type of data written per second at the current measurement point. The buffer space is then updated according to the formula: buffer switching time * number of bytes written per second * first random number coefficient.

[0078] See Figure 4 This is a schematic diagram of cache management provided in an embodiment of this application. (Comparison) Figure 4 and Figure 1 It is quite obvious that, compared with the prior art, the embodiments provided in this application do not need to allocate a fixed size of cache space for each measurement point data, thus avoiding the waste of cache space.

[0079] This application provides an industrial data buffering method that uses random number coefficients to create phase deviation, thereby reducing the probability of batch disk writes at the same time, reducing disk I / O, and improving storage performance.

[0080] See Figure 5 The flowchart below illustrates a third method for industrial data storage caching provided in this application embodiment. Applied to a large-scale industrial real-time historical database, this method performs cache switching, taking into account not only insufficient memory space but also configuration refresh time. The method includes at least the following steps:

[0081] S501: Pre-configure refresh time.

[0082] S502: Determine if the history archive service is being started for the first time. If yes, proceed to S503; otherwise, proceed to S505.

[0083] S503: Load the measurement points in the configuration, and create a fixed-size buffer space based on the measurement point data type and the preset mapping relationship between the buffer space and the data type.

[0084] S504: The above-mentioned fixed-size buffer space is determined as the current buffer space.

[0085] S505: In response to continuously written historical data, calculate the amount of historical data in the current measurement point buffer and the amount of data written per second at the current measurement point.

[0086] S506: Determine if the configuration refresh time has been reached. If yes, proceed to S508; otherwise, proceed to S507.

[0087] S507: Determine whether the amount of historical data in the current measurement point buffer exceeds the size of the current buffer space. If yes, proceed to S508; otherwise, proceed to S505.

[0088] S508: Write the current buffered data to disk and update the current buffer space according to the amount of data written per second and the data type of the current measurement point.

[0089] Furthermore, embodiments of this application also provide corresponding apparatus. See [link to relevant documentation]. Figure 6 This is a schematic diagram of an industrial data storage and caching device 600 provided in an embodiment of this application. It is applied to a real-time historical database. The device 600 includes:

[0090] The calculation unit 601, in response to the continuously written historical data, calculates the amount of historical data buffered at the current measurement point and the amount of data written per second at the current measurement point;

[0091] The writing unit 602, in response to the buffered historical data exceeding the size of the current buffer space, or reaching the configured refresh time, writes the currently buffered data to the disk, and updates the current buffer space according to the data write per second and data type of the current measurement point; the current buffer space is used to buffer the data of the current measurement point.

[0092] Optionally, the writing unit 602 includes:

[0093] The byte count determination module is used to determine the number of bytes written per second based on the amount of data written per second and the data type of the current measurement point;

[0094] The update module is used to update the buffer space based on the number of bytes written per second and the preset mapping relationship between the buffer space and the number of bytes written per second.

[0095] Optionally, the preset mapping relationship between the buffer space and the number of bytes written per second specifically includes:

[0096] Buffer space = cache switching time * bytes written per second * first random number coefficient.

[0097] Optionally, the device 600 further includes:

[0098] The initial creation unit is used to respond to the first startup of the historical archive service and create a fixed-size buffer space based on the data type of the measurement point and the preset mapping relationship between the buffer space and the data type.

[0099] The fixed-size buffer space created will be used to determine the current buffer space.

[0100] Optionally, the preset mapping relationship between the buffer space and the data type specifically includes:

[0101] Buffer space = data type * second random number coefficient.

[0102] This application provides an industrial data storage caching device. The calculation unit 601 first calculates the amount of historical data and the data write rate per second for the current measurement point based on continuously written historical data. The writing unit 602 compares the amount of historical data for the current measurement point with the current buffer space size, or the configured refresh time. If the data exceeds the current buffer space size or the configured refresh time is reached, the currently buffered data is written to the disk, and the current buffer space is updated according to the data write rate and data type of the current measurement point. In this way, the buffer space size is dynamically adjusted based on the data type and data flow of the measurement point, thereby avoiding the waste of buffer space caused by measurement points with unchanging data continuously occupying buffer space, and thus improving the resource utilization rate of the buffer space.

[0103] This application also provides an electronic device. The device includes a memory and a processor.

[0104] The memory is used to store program code and transfer program code to the processor.

[0105] Processor: Used to execute the steps of the above-described method for controlling device access according to instructions in the program code.

[0106] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an industrial data storage caching method according to this application.

[0107] In practical applications, the computer-readable storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0108] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0109] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0110] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0112] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A method for industrial data storage caching, characterized in that, Applied to a real-time historical database, the method includes: In response to the continuously written historical data, calculate the amount of historical data buffered at the current measurement point and the amount of data written per second at the current measurement point; In response to the buffered historical data exceeding the current buffer space size and / or reaching the configured refresh time, the currently buffered data is written to disk, the current buffer space being used to buffer the data of the current measurement point; The number of bytes written per second is determined based on the amount and type of data written per second at the current measurement point; Update the buffer space based on the number of bytes written per second and the preset mapping relationship between the buffer space and the number of bytes written per second; The preset mapping relationship between the buffer space and the number of bytes written per second specifically includes: Buffer space = Cache switching time * Number of bytes written per second * First random number coefficient; The preset mapping relationship between buffer space and data type specifically includes: buffer space = data type * second random number coefficient.

2. The method according to claim 1, characterized in that, Prior to responding to continuously written historical data, the method further includes: In response to the first launch of the historical archive service, a fixed-size buffer space is created based on the data type of the measurement point and the preset mapping relationship between the buffer space and the data type. The fixed-size buffer space created will be used to determine the current buffer space.

3. An industrial data storage cache device, characterized in that, The device, applied to a real-time historical database, includes: The computing unit, in response to the continuously written historical data, calculates the amount of historical data buffered at the current measurement point and the amount of data written per second at the current measurement point; The write unit, in response to the buffered historical data exceeding the current buffer space size or reaching the configured refresh time, writes the currently buffered data to the disk, and determines the number of bytes written per second based on the current measurement point's write data volume and data type; it then updates the buffer space based on the number of bytes written per second and a preset mapping relationship between the buffer space and the number of bytes written per second; the current buffer space is used to buffer the data of the current measurement point; the preset mapping relationship between the buffer space and the number of bytes written per second specifically includes: buffer space = cache switching time. bytes written per second The first random number coefficient; the preset mapping relationship between the buffer space and the data type specifically includes: buffer space = data type Second random number coefficient.

4. An electronic device comprising: Memory and processor; The memory is used to store program code and transfer the program code to the processor; Processor: configured to execute the method steps of the industrial data storage cache as described in any one of claims 1-2, according to instructions in the program code.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores code that, when executed by a processor, implements the method steps of the industrial data storage cache as described in any one of claims 1-2.

Images