Data storage method, system and equipment and computer readable storage medium
By allocating different storage intervals and reading compressed data between storage spaces, the problem of insufficient data storage capacity in storage space is solved, and efficient data storage and flexible storage management are achieved.
Patent Information
- Application Number
- CN202411118891.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-15
AI Technical Summary
Existing storage technologies are difficult to effectively improve the data storage capacity of storage space, resulting in increased data storage pressure and difficulty in effectively saving statistical data.
By allocating different saving intervals between the first storage space and the second storage space, the target value is determined, the data is read and compressed, thereby storing data in the second storage space, and improving storage with the time period difference between the two Flexibility and performance savings.
It realizes more efficient data storage in limited storage space, improves the data storage ability of storage space, and flexibly processes data for different time periods through compression and interval management, avoids data storage pressure.
Smart Images

Figure CN118963671A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of storage technology, and more specifically, to a data storage method, system, device and computer-readable storage medium. Background Art
[0002] As storage is applied to all walks of life, there is a deeper pursuit of transparency in the use of various objects within the storage. Performance statistics display charts can better display business loads, identify key businesses in advance, and change the business deployment scope or business migration in a timely manner, effectively ensuring the smooth progress of business and avoiding the risk of business downtime due to unclear identification.
[0003] However, as storage services are running, more and more statistical data are generated, but storage space is limited, resulting in high pressure on data storage and difficulty in effectively saving data.
[0004] In summary, how to improve the ability of storage space to store data is currently a problem that needs to be solved urgently by those skilled in the art. Summary of the invention
[0005] The purpose of the present application is to provide a data storage method, which can solve the technical problem of how to improve the ability of storage space to store data to a certain extent. The present application also provides a data storage system, an electronic device and a computer-readable storage medium.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] A data storage method, comprising:
[0008] determining a target value according to the first saving interval and the second saving interval;
[0009] In the first storage space, the first group of data stored last is read, wherein the number of data in the first group of data is the target value;
[0010] compressing the first set of data to obtain first compressed data;
[0011] storing the first compressed data in a second storage space;
[0012] The first saving interval includes the interval for saving data in the first storage space; the second saving interval includes the interval for saving data in the second storage space; and the second saving interval is one level larger than the first saving interval.
[0013] In an exemplary embodiment, determining the target value according to the first saving interval and the second saving interval includes:
[0014] The target value is obtained by dividing the second storage interval by the first storage interval.
[0015] In an exemplary embodiment, compressing the first set of data includes:
[0016] Check whether the waiting queue is full;
[0017] In response to the waiting queue being not full, placing the first set of data at the end of the waiting queue;
[0018] Check whether the execution queue is empty;
[0019] In response to the execution queue being empty, a second group of data is determined at the head of the waiting queue, and the second group of data is transferred to the execution queue for compression.
[0020] In an exemplary embodiment, the detecting whether the execution queue is empty includes:
[0021] Counting the compression duration of the execution queue;
[0022] Detecting whether the compression duration exceeds the set duration of the second storage space;
[0023] In response to the compression duration exceeding the set duration, the execution queue is cleared.
[0024] In an exemplary embodiment, storing the first compressed data in the second storage space includes:
[0025] Encapsulating the first compressed data into a target IO;
[0026] Call a write process to write the target IO into the volume corresponding to the second storage space.
[0027] In an exemplary embodiment, it further includes:
[0028] In response to the first saving interval being the smallest, acquiring a third set of data according to the first saving interval;
[0029] compressing the third set of data to obtain second compressed data;
[0030] The second compressed data is stored in the first storage space.
[0031] In an exemplary embodiment, in the first storage space, the first group of data stored last is read, wherein the number of data in the first group of data is the target value, including:
[0032] Subtract the second storage interval from the current time to obtain the sampling start time;
[0033] generating a ratio of the sampling start time to the first saving interval;
[0034] Determine the total value of the data stored in the first storage space;
[0035] generating a first remainder value of the ratio and the total number of values;
[0036] Determining a unit size value of each block device in the storage space, the block device being used to store data;
[0037] Generate a product value of the first remainder value and the unit size value as a sampling point starting address;
[0038] In the first storage space, read the data corresponding to the starting address of the sampling point as one of the first group of data;
[0039] Updating the real-time quantity value of the data in the first set of data;
[0040] Detecting whether the real-time quantity value is less than the target value;
[0041] In response to the real-time quantity value being less than the target value, a sum value of the starting address of the last read data and the unit length value is generated, and a second remainder value of the sum value and the total length value of the first storage space is generated as the starting address of the next read data, and in the first storage space, data corresponding to the starting address of the next read data is read as data in the first group of data, and the step of updating the real-time quantity value of the data in the first group of data is returned to be executed; the unit length value includes the length value of a single data stored in the first storage space;
[0042] In response to the real-time quantity value being equal to the target value, the process ends.
[0043] A data storage system, comprising:
[0044] A first determining module, configured to determine a target value according to a first saving interval and a second saving interval;
[0045] A first reading module is used to read the first group of data stored last in the first storage space, where the number of data in the first group of data is the target value;
[0046] A first compression module, used for compressing the first group of data to obtain first compressed data;
[0047] A first storage module, used for storing the first compressed data in a second storage space;
[0048] The first saving interval includes the interval for saving data in the first storage space; the second saving interval includes the interval for saving data in the second storage space; and the second saving interval is one level larger than the first saving interval.
[0049] An electronic device, comprising:
[0050] Memory for storing computer programs;
[0051] A processor is used to implement the steps of any of the above data storage methods when executing the computer program.
[0052] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above data storage methods are implemented.
[0053] The present application provides a data storage method, which determines a target value according to a first storage interval and a second storage interval; in the first storage space, reads the last stored first group of data, the number of data in the first group of data is the target value; compresses the first group of data to obtain first compressed data; stores the first compressed data in the second storage space; wherein the first storage interval includes the interval for storing data in the first storage space; the second storage interval includes the interval for storing data in the second storage space; and the second storage interval is one level larger than the first storage interval. In the present application, since the second storage interval is one level larger than the first storage interval, the data stored in the first storage space is newer than the data stored in the second storage space, but the time span of the data stored in the second storage space is larger than that of the first storage space, so data of different time periods can be stored with the help of the first storage space and the second storage space, which improves the flexibility of the storage space to store data; and in the data storage process, the target value data is read from the first storage space as the first group of data, and the first group of data is compressed and stored in the second storage space, which can save the performance of the second storage space and improve the number of data stored in the storage space. The data storage system, electronic device and computer-readable storage medium provided by the present application also solve the corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0055] Figure 1 A flowchart of a data storage method provided in an embodiment of the present application;
[0056] Figure 2 A schematic diagram of compressing data with the help of queues and IO;
[0057] Figure 3 A schematic diagram of compressing data using a single timer;
[0058] Figure 4 A schematic diagram of the structure of a data storage system provided in an embodiment of the present application;
[0059] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0060] Figure 6 Another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0062] See also Figure 1 , Figure 1 A flowchart of a data storage method provided in an embodiment of the present application.
[0063] A data storage method provided in an embodiment of the present application may include the following steps:
[0064] Step S101: Determine a target value according to a first saving interval and a second saving interval.
[0065] In practical applications, the first storage interval for storing data in the first storage space and the second storage interval for storing data in the second storage space can be determined first. In other words, for the first storage space, the data collected in the first storage interval needs to be stored in the first storage space, and for the second storage space, the data collected in the second storage interval needs to be stored in the second storage space. And the second storage interval is one level larger than the first storage interval, that is, the present application has multiple storage spaces, each storage space corresponds to a storage interval, and the storage intervals of each storage space are different, then different storage intervals will have a size relationship, and the second storage interval is only one level larger than the first storage interval. For example, there are 4 storage intervals, namely 5s, 300s, 900s, 3600s, etc., that is, the data is stored once every 5s, 300s, 900s, and 3600s, respectively. If the first storage interval is 5s, the second storage interval is 300s, if the first storage interval is 300s, the second storage interval is 3600s, etc.
[0066] It should be noted that since the second storage interval is one level larger than the first storage interval, after the data is stored in the first storage space, it will be stored in the second storage space after a period of time. Therefore, the data required for the second storage space can be obtained from the first storage space. Assuming that the first storage interval is 5s and the second storage interval is 300s, the first storage space stores data every 5s. Then, after the first storage space stores 60 data, the storage interval of 60 data is exactly one second storage interval, that is, the 60 data in the first storage space are the data that need to be stored in the second storage space at one time. Therefore, the target value can be determined first according to the first storage interval and the second storage interval, so that the data to be stored in the second storage space can be determined in the first storage space according to the target data later.
[0067] In a specific application scenario, in the process of determining the target value according to the first storage interval and the second storage interval, the second storage interval can be divided by the first storage interval to obtain the target value. It should be noted that when the value obtained by dividing the second storage interval by the first storage interval is not an integer, the value can be rounded down or rounded up to obtain the target value. It should also be noted that the specific values of the saving intervals at each level of storage space can be flexibly determined according to the application scenario, and the total amount of data that the storage space can store can be determined according to the storage requirements. For example, storage space 0 is used to save the data of the last 7 days, and the saving interval of storage space 0 is 5s, then storage space 0 can store 120,960 data, storage space 1 is used to save the data of the last 30 days, and the saving interval of storage space 1 is 300s, then storage space 1 can store 8,640 data, storage space 2 is used to save the data of the last 90 days, and the saving interval of storage space 2 is 900s, then storage space 2 can store 8,640 data, storage space 3 is used to save the data of the last 360 days, and the saving interval of storage space 3 is 3,600s, then storage space 0 can store 26,280 data, and so on.
[0068] Step S102: In the first storage space, read the first group of data stored last, the number of data in the first group of data being the target value.
[0069] Step S103: compress the first group of data to obtain first compressed data.
[0070] Step S104: storing the first compressed data in the second storage space; wherein the first storage interval includes the interval for storing data in the first storage space; the second storage interval includes the interval for storing data in the second storage space; and the second storage interval is one level larger than the first storage interval.
[0071] In practical applications, after determining the target value, the last stored target value data group can be read as the first group of data in the first storage space; the first group of data is compressed to obtain the first compressed data, and the compression method can be flexibly determined according to actual needs. For example, if the data is performance statistical data, the compression method can be addition and averaging, etc.; the first compressed data is stored in the second storage space. Still taking the above storage spaces 0, 1, 2, and 3 as an example, storage space 1 needs to obtain 300 / 5=60 data from storage space 0 and compress them before storing them in itself; storage space 2 needs to obtain 900 / 300=3 data from storage space 1 and compress them before storing them in itself; storage space 3 needs to obtain 3600 / 900=4 data from storage space 2 and compress them before storing them in itself.
[0072] In practical applications, in the first storage space, the first group of data stored last is read. When the number of data in the first group of data is the target value, the second storage interval can be subtracted from the current time to obtain the sampling start time; the ratio of the sampling start time to the first storage interval is generated; the total number of data stored in the first storage space is determined, and this total number of data is used to record the number of data stored in the first storage space; a first remainder value of the ratio and the total number of data is generated; the unit size value of each block device in the storage space is determined, and the block device is used to store data, and the unit size value can be bytes, etc.; the product value of the first remainder value and the unit size value is generated as the starting address of the sampling point; in the first storage space, the data corresponding to the starting address of the sampling point is read. as one of the data in the first group of data; updating the real-time quantity value of the data in the first group of data; detecting whether the real-time quantity value is less than the target value; in response to the real-time quantity value being less than the target value, generating the sum of the starting address of the previous read data and the unit length value, generating the second remainder value of the sum and the total length value of the first storage space as the starting address of the next read data, this address can be an LBA address, etc., in the first storage space, reading the data corresponding to the starting address of the next read data as the data in the first group of data, and returning to execute the step of updating the real-time quantity value of the data in the first group of data; the unit length value includes the length value of a single data stored in the first storage space; in response to the real-time quantity value being equal to the target value, ending.
[0073] From this implementation process, it can be seen that the present application takes into account that when reading data from the storage space, it is necessary to determine the data address, and the data in the second storage interval are all stored in the first storage space, so the current time can be subtracted from the second storage interval to obtain the sampling start time, generate the ratio of the sampling start time to the first storage interval, determine the total number of data stored in the first storage space, and generate a first remainder value of the ratio and the total number of data. Considering that the storage space is limited, the data is stored in the storage space in an overwrite manner, so taking the remainder can ensure that the first storage space is recycled; determine the unit size value of each block device in the storage space, the block device is used to store data, and generate the product value of the first remainder value and the unit size value as the starting address of the sampling point, so as to comprehensively determine the sampling point based on the second storage interval and the first storage interval, the total number of data in the first storage space, and the unit size value of the block device. Determine the starting address of the data that needs to be stored in the second storage space at present; then, in the first storage space, read the data corresponding to the starting address of the sampling point as one of the data in the first group of data; update the real-time quantity value of the data in the first group of data; detect whether the real-time quantity value is less than the target value; in response to the real-time quantity value being less than the target value, generate the sum of the starting address of the last read data and the unit length value, generate the second remainder value of the sum and the total length value of the first storage space as the starting address of the next read data, read the data corresponding to the starting address of the next read data in the first storage space as the data in the first group of data, and return to execute the step of updating the real-time quantity value of the data in the first group of data; the unit length value includes the length value of the single data stored in the first storage space; in response to the real-time quantity value being equal to the target value, end. Accurately read the target value of data from the first storage space as the first group of data in a loop.
[0074] In a specific application scenario, considering that reading a single piece of data also takes time, a two-layer loop can be set to read the data in the first storage space. The outer loop can be used to read a single piece of data, and the memory loop can be used to read data according to the address.
[0075] In actual applications, in the process of compressing the first group of data, in order not to affect the storage business, the task control can be performed by setting the compression task queue. For example, an execution queue can be set to execute the compression task, and a waiting queue can be set to store the data to be compressed. Therefore, in the data compression process, it can be detected whether the waiting queue is full; in response to the waiting queue being not full, the first group of data is placed at the end of the waiting queue, and correspondingly, in response to the waiting queue being full, the first group of data can be discarded, etc.; detect whether the execution queue is empty; in response to the execution queue being empty, the second group of data is determined at the head of the waiting queue, and the second group of data is transferred to the execution queue for compression. It should be noted that the depth of the execution queue and the waiting queue can be determined according to actual needs. For example, the depth of the execution queue can be 1, and the depth of the waiting queue can be 16, etc.
[0076] In specific application scenarios, since there are certain limitations on the compression queue, a single task cannot wait indefinitely in the queue, so it is necessary to set a certain timeout control strategy to limit the compression progress, that is, in the process of detecting whether the execution queue is empty, the compression duration of the execution queue can be counted; detect whether the compression duration exceeds the set duration of the second storage space; in response to the compression duration exceeding the set duration, the execution queue will be cleared. In this process, considering that the storage space needs to store new data after each save interval, data compression is required at this time, so the set duration of each storage space can be set to the save interval of the storage space.
[0077] In actual applications, when storing the first compressed data in the second storage space, the first compressed data can be encapsulated into a target IO, and the write process is called to write the target IO into the volume corresponding to the second storage space, so as to avoid using a database to manage data and reduce the complexity of data management.
[0078] In specific application scenarios, the process of compressing data with the help of queues and IO can be as follows: Figure 2 As shown, the following process is included:
[0079] The user sends the storage unit compression task to the compression module;
[0080] The compression module detects whether the execution queue is empty; if not, the task is placed in the waiting queue and a corresponding message is returned to the user; if so, the task is placed in the submission queue and a corresponding message is returned to the user;
[0081] The compression module processes the tasks in the submission queue; queries the index of the upper-level storage unit and sends it to the management module;
[0082] The compression module determines whether the data has been read. If so, it calculates the timestamp of the query start sampling point, queries the management module for the start lba, submits the IO for reading data to the IO stack based on the lba, and saves the cache returned by the IO stack locally; if not, it aggregates the data in all the caches read according to the average value, generates a new sampling point, queries the management module for the index of the storage unit, calculates the lba for this write, and submits the IO for writing data to the IO stack;
[0083] The compression module updates the index of this storage unit. This task is completed and the waiting queue obtains new tasks and executes them.
[0084] In actual applications, when the storage space is the first space to store data, this storage space cannot read data from other storage spaces and can only obtain the latest data from the memory. However, the data in this storage space can also be compressed before storage, that is, in response to the first saving interval being the smallest, the third set of data can be obtained according to the first saving interval, for example, the third set of data is obtained from the memory according to the first saving interval; the third set of data is compressed to obtain second compressed data; and the second compressed data is stored in the first storage space.
[0085] In practical applications, in the process of timing the saving interval of each storage unit, the minimum saving interval can be used as the timing duration of the timer, the corresponding timer timing times are set for each storage unit, and the ratio of the saving interval of the storage unit to the minimum saving interval is determined as the maximum timing value of the storage unit; thereafter, for each storage unit, each time the timer completes a timing, the real-time timing times of the current round is increased by 1, and it is detected whether the real-time timing times are equal to the corresponding maximum timing value. If the real-time timing times are less than the maximum timing value, the real-time timing times continue to be accumulated. If the real-time timing times are equal to the maximum timing value, it indicates that the saving interval of the storage unit has expired, and the data storage operation can be performed, the real-time timing times are cleared, and the accumulation of the next round of real-time timing times begins.
[0086] From this implementation process, it can be seen that the present application uses the minimum saving interval as the timing duration of the timer. In this way, after the timer completes a timing, it has passed a minimum saving interval, which means that the storage unit can save the latest data once; and the present application sets the timing times of the corresponding timer for each storage unit, and determines the ratio of the storage interval of the storage unit to the minimum saving interval as the maximum timing value of the storage unit, so that it can be detected by counting the number of timer timings to see whether the storage unit corresponding to the saving interval is reached; then for each storage unit, every time the timer completes a timing, the real-time timing times of the current round are added by 1, and the real-time timing times are detected to be equal to the corresponding maximum timing value. If the real-time timing times are less than the maximum timing value, the real-time timing times continue to be accumulated. If the real-time timing times are equal to the maximum timing value, the real-time timing times are cleared to zero, and the next round of real-time timing times accumulation is started. After this operation, the present application can count the saving intervals of each storage unit by only using one timer, which can reduce the number of storage timers required and improve the applicability of the method.
[0087] For ease of understanding, still taking the above storage units 0, 1, 2, and 3 as an example, assuming that the timing times of storage units 1, 2, and 3 are represented by group1.round, group2.round, and group3.round respectively, the data compression conditions of storage units 1, 2, and 3 are as follows Figure 3 shown.
[0088] It should be noted that the number of storage spaces in the present application can be flexibly determined according to the application scenario. When the storage intervals of any two storage spaces differ by one level, the data of the two storage spaces can be processed according to the solution of the present application.
[0089] The present application provides a data storage method, which determines a target value according to a first storage interval and a second storage interval; in the first storage space, reads the last stored first group of data, the number of data in the first group of data is the target value; compresses the first group of data to obtain first compressed data; stores the first compressed data in the second storage space; wherein the first storage interval includes the interval in which the first storage space stores data; the second storage interval includes the interval in which the second storage space stores data; and the second storage interval is one level larger than the first storage interval. In the present application, since the second storage interval is one level larger than the first storage interval, the data stored in the first storage space is newer than the data stored in the second storage space, but the time span of the data stored in the second storage space is larger than that of the first storage space, so data of different time periods can be stored with the help of the first storage space and the second storage space, thereby improving the flexibility of the storage space to store data; and in the data storage process, the target value of data is read from the first storage space as the first group of data, and the first group of data is compressed and stored in the second storage space, which can save the performance of the second storage space and improve the number of data stored in the storage space.
[0090] See also Figure 4 , Figure 4 A structural schematic diagram of a data storage system provided in an embodiment of the present application.
[0091] A data storage system provided in an embodiment of the present application may include:
[0092] A first determination module 101 is used to determine a target value according to a first saving interval and a second saving interval;
[0093] A first reading module 102 is used to read the first group of data stored last in the first storage space, where the number of data in the first group of data is a target value;
[0094] A first compression module 103, used to compress the first group of data to obtain first compressed data;
[0095] A first storage module 104, used for storing the first compressed data in a second storage space;
[0096] The first saving interval includes the interval for saving data in the first storage space; the second saving interval includes the interval for saving data in the second storage space; and the second saving interval is one level larger than the first saving interval.
[0097] An embodiment of the present application provides a data storage system, in which a first reading module may include:
[0098] A first generating unit, configured to subtract the second saving interval from the current time to obtain a sampling start time;
[0099] A second generating unit, used for generating a ratio of the sampling start time to the first saving interval;
[0100] A first determining unit, used to determine a total value of data stored in the first storage space;
[0101] A third generating unit is used to generate a first remainder value between the ratio and the total number of values;
[0102] A second determining unit, used to determine a unit size value of each block device in the storage space, the block device being used to store data;
[0103] A fourth generating unit, used to generate a product value of the first remainder value and the unit size value as a sampling point starting address;
[0104] A first reading unit, used for reading data corresponding to a starting address of a sampling point in a first storage space as one of the data in a first group of data;
[0105] A first updating unit, used for updating the real-time quantity value of the data in the first set of data;
[0106] The first detection unit is used to detect whether the real-time quantity value is less than the target value; in response to the real-time quantity value being less than the target value, the sum of the starting address of the previous read data and the unit length value is generated, and the second remainder value of the sum and the total length value of the first storage space is generated as the starting address of the next read data. In the first storage space, the data corresponding to the starting address of the next read data is read as the data in the first group of data, and the step of updating the real-time quantity value of the data in the first group of data is returned; the unit length value includes the length value of a single data stored in the first storage space; in response to the real-time quantity value being equal to the target value, the step ends.
[0107] In a data storage system provided in an embodiment of the present application, a first determination module may include:
[0108] The third determining unit is configured to divide the second storage interval by the first storage interval to obtain a target value.
[0109] In a data storage system provided by an embodiment of the present application, a first compression module may include:
[0110] The second detection unit is used to detect whether the waiting queue is full; in response to the waiting queue being not full, the first group of data is placed at the end of the waiting queue; detect whether the execution queue is empty; in response to the execution queue being empty, the second group of data is determined at the head of the waiting queue, and the second group of data is transferred to the execution queue for compression.
[0111] In a data storage system provided by an embodiment of the present application, the second detection unit can be used to: count the compression time of the execution queue; detect whether the compression time exceeds the set time of the second storage space; and clear the execution queue in response to the compression time exceeding the set time.
[0112] In a data storage system provided in an embodiment of the present application, a first storage module may include:
[0113] The first storage unit is used to encapsulate the first compressed data into a target IO; and call a writing process to write the target IO into a volume corresponding to the second storage space.
[0114] A data storage system provided in an embodiment of the present application may further include:
[0115] A second compression module is configured to obtain a third set of data according to the first storage interval in response to the first storage interval being the smallest; and compress the third set of data to obtain second compressed data;
[0116] The second storage module is used to store the second compressed data in the first storage space.
[0117] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of a data storage method provided in an embodiment of the present application. Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0118] An electronic device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the steps of the data storage method described in any of the above embodiments are implemented.
[0119] See also Figure 6, another electronic device provided in the embodiment of the present application may also include: an input port 203 connected to the processor 202, for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202, for displaying the processing results of the processor 202 to the outside; a communication module 205 connected to the processor 202, for realizing communication between the electronic device and the outside. The display unit 204 can be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes but is not limited to mobile high-definition link technology (Mobile High-Definition Link, MHL), Universal Serial Bus (Universal Serial Bus, USB), High-Definition Multimedia Interface (High-DefinitionMultimedia Interface, HDMI), wireless connection: wireless fidelity technology (WIreless Fidelity, WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.
[0120] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the data storage method described in any of the above embodiments are implemented.
[0121] The computer-readable storage medium involved in this application includes random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.
[0122] A computer program product provided in an embodiment of the present application includes a computer program / instruction. When the computer program / instruction is executed by a processor, the following steps are implemented:
[0123] determining a target value according to the first saving interval and the second saving interval;
[0124] In the first storage space, the first group of data stored last is read, and the number of data in the first group of data is the target value;
[0125] Compressing the first group of data to obtain first compressed data;
[0126] Storing the first compressed data in the second storage space;
[0127] The first saving interval includes the interval for saving data in the first storage space; the second saving interval includes the interval for saving data in the second storage space; and the second saving interval is one level larger than the first saving interval.
[0128] A computer program product provided by an embodiment of the present application includes a computer program / instruction. When the computer program / instruction is executed by a processor, the following steps are implemented: subtracting a second storage interval from a current time to obtain a sampling start time; generating a ratio of the sampling start time to the first storage interval; determining a total number of data stored in a first storage space; generating a first remainder value of the ratio and the total number of data; determining a unit size value of each block device in the storage space, the block device being used to store data; generating a product value of the first remainder value and the unit size value as a sampling point start address; and reading data corresponding to the sampling point start address in the first storage space as one of the first group of data. Data; update the real-time quantity value of the data in the first group of data; detect whether the real-time quantity value is less than the target value; in response to the real-time quantity value being less than the target value, generate the sum of the starting address of the last read data and the unit length value, generate the second remainder value of the sum and the total length value of the first storage space as the starting address of the next read data, read the data corresponding to the starting address of the next read data in the first storage space as the data in the first group of data, and return to execute the step of updating the real-time quantity value of the data in the first group of data; the unit length value includes the length value of a single data stored in the first storage space; in response to the real-time quantity value being equal to the target value, end.
[0129] A computer program product provided by an embodiment of the present application includes a computer program / instruction. When the computer program / instruction is executed by a processor, the following steps are implemented: dividing the second storage interval by the first storage interval to obtain a target value.
[0130] A computer program product provided by an embodiment of the present application includes a computer program / instruction, which implements the following steps when executed by a processor: detecting whether a waiting queue is full; in response to the waiting queue being not full, placing a first group of data at the end of the waiting queue; detecting whether an execution queue is empty; in response to the execution queue being empty, determining a second group of data at the head of the waiting queue, and transferring the second group of data to the execution queue for compression.
[0131] A computer program product provided in an embodiment of the present application includes a computer program / instruction, which implements the following steps when executed by a processor: counting the compression time of the execution queue; detecting whether the compression time exceeds the set time of the second storage space; and clearing the execution queue in response to the compression time exceeding the set time.
[0132] A computer program product provided in an embodiment of the present application includes a computer program / instruction, which implements the following steps when executed by a processor: encapsulating first compressed data into a target IO; calling a write process to write the target IO into a volume corresponding to a second storage space.
[0133] A computer program product provided in an embodiment of the present application includes a computer program / instruction, which implements the following steps when executed by a processor: in response to the first saving interval being the smallest, obtaining a third group of data according to the first saving interval; compressing the third group of data to obtain second compressed data; and storing the second compressed data in a first storage space.
[0134] For the description of the relevant parts of a data storage system, an electronic device, and a computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding parts in a data storage method provided in the embodiments of the present application, which will not be repeated here. In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.
[0135] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0136] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data storage method, characterized in that: include: determining a target value according to the first saving interval and the second saving interval; In the first storage space, the first group of data stored last is read, wherein the number of data in the first group of data is the target value; compressing the first set of data to obtain first compressed data; storing the first compressed data in a second storage space; The first saving interval includes the interval for saving data in the first storage space; the second saving interval includes the interval for saving data in the second storage space; and the second saving interval is one level larger than the first saving interval.
2. The method according to claim 1, characterized in that The step of determining the target value according to the first storage interval and the second storage interval includes: The target value is obtained by dividing the second storage interval by the first storage interval.
3. The method according to claim 1, characterized in that The compressing the first set of data includes: Check whether the waiting queue is full; In response to the waiting queue being not full, placing the first set of data at the end of the waiting queue; Check whether the execution queue is empty; In response to the execution queue being empty, a second group of data is determined at the head of the waiting queue, and the second group of data is transferred to the execution queue for compression.
4. The method according to claim 3, characterized in that The step of detecting whether the execution queue is empty includes: Counting the compression duration of the execution queue; Detecting whether the compression duration exceeds the set duration of the second storage space; In response to the compression duration exceeding the set duration, the execution queue is cleared.
5. The method according to claim 1, characterized in that The storing the first compressed data into the second storage space includes: Encapsulating the first compressed data into a target IO; Call a write process to write the target IO into the volume corresponding to the second storage space.
6. The method according to claim 1, characterized in that Also includes: In response to the first saving interval being the smallest, acquiring a third set of data according to the first saving interval; compressing the third set of data to obtain second compressed data; The second compressed data is stored in the first storage space.
7. The method according to claim 1, characterized in that In the first storage space, the first group of data stored last is read, wherein the number of data in the first group of data is the target value, including: Subtract the second storage interval from the current time to obtain the sampling start time; generating a ratio of the sampling start time to the first saving interval; Determine the total value of the data stored in the first storage space; generating a first remainder value of the ratio and the total number of values; Determining a unit size value of each block device in the storage space, the block device being used to store data; Generate a product value of the first remainder value and the unit size value as a sampling point starting address; In the first storage space, read the data corresponding to the starting address of the sampling point as one of the first group of data; Updating the real-time quantity value of the data in the first set of data; Detecting whether the real-time quantity value is less than the target value; In response to the real-time quantity value being less than the target value, a sum value of the starting address of the last read data and the unit length value is generated, and a second remainder value of the sum value and the total length value of the first storage space is generated as the starting address of the next read data, and in the first storage space, data corresponding to the starting address of the next read data is read as data in the first group of data, and the step of updating the real-time quantity value of the data in the first group of data is returned to be executed; the unit length value includes the length value of a single data stored in the first storage space; In response to the real-time quantity value being equal to the target value, the process ends.
8. A data storage system, characterized in that: include: A first determining module, configured to determine a target value according to a first saving interval and a second saving interval; A first reading module is used to read the first group of data stored last in the first storage space, where the number of data in the first group of data is the target value; A first compression module, used for compressing the first group of data to obtain first compressed data; A first storage module, used for storing the first compressed data in a second storage space; The first saving interval includes the interval for saving data in the first storage space; the second saving interval includes the interval for saving data in the second storage space; and the second saving interval is one level larger than the first saving interval.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data storage method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Intelligent management method and system for enterprise operation data
CN119988379A
