Monitoring data storage method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a monitoring data storage method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a monitoring data set in a target time period; the monitoring data set comprises monitoring data of a plurality of user identifiers; according to the target weight of each storage engine, scheduling each piece of monitoring data, and determining a target storage engine corresponding to each piece of monitoring data; the target weight represents the weight of the data volume of the monitoring data stored in the storage engine within the target time period; and storing each piece of monitoring data to the corresponding target storage engine. By adopting the method, the storage efficiency can be improved, the storage pressure of the storage engine is reduced, and the performance of the storage engine is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a monitoring data storage method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of 5G and the huge demand for video surveillance, the amount of data stored in video surveillance has exploded, which has also led to an increase in energy consumption. This has put forward higher requirements and challenges for the storage devices, systems and methods for storing these massive videos.

[0003] Currently, these massive video surveillance videos are generally stored through a single storage engine.

[0004] Then, the current monitoring data storage method stores the monitoring data in a single storage engine, which greatly affects the storage efficiency when there is a massive amount of monitoring data. Summary of the invention

[0005] Based on this, it is necessary to provide a monitoring data storage method, device, computer equipment, computer-readable storage medium and computer program product that can improve storage efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a monitoring data storage method, the method comprising:

[0007] Acquire a monitoring data set within a target time period; the monitoring data set includes monitoring data of multiple user identifiers;

[0008] According to the target weight of each storage engine, each monitoring data is scheduled to determine the target storage engine corresponding to each monitoring data; the target weight represents the weight of the amount of data stored in the storage engine by the monitoring data within the target time period;

[0009] Each of the monitoring data is stored in a corresponding target storage engine.

[0010] In one embodiment, storing each of the monitoring data into a corresponding target storage engine includes:

[0011] For each of the monitoring data of the user identification, the monitoring data is segmented according to the segmentation rule to obtain the first block metadata and the last block metadata;

[0012] The first block metadata and the last block metadata are written into the target storage engine through the interface of the target storage engine.

[0013] In one embodiment, the method for determining the target weight of each storage engine includes:

[0014] Acquire historical monitoring data of multiple user identifiers within a historical time period and historical energy data consumed to maintain the operation of the storage engine; wherein the historical energy data includes historical new energy data;

[0015] Inputting the historical monitoring data and the historical new energy data into a prediction model, and obtaining a prediction result of the prediction model, wherein the prediction result includes the predicted monitoring data and the predicted new energy data within the target time period;

[0016] The target weight of each of the storage engines within the target time period is determined according to the predicted monitoring data, the predicted new energy data and the constraints of the storage engines.

[0017] In one embodiment, the constraint condition includes a load condition for storing monitoring data of the storage engine and an energy data condition for running the storage engine; determining a target weight of each storage engine within the target time period according to the predicted monitoring data, the predicted new energy data and the constraint condition of each storage engine includes:

[0018] Acquire traditional energy data and energy consumption per unit time of each storage engine;

[0019] Generating an initial weight of each of the storage engines that meets the energy data condition according to the predicted new energy data, the traditional energy data and the energy consumption per unit time of each of the storage engines;

[0020] If the initial weight meets the load condition, the initial weight is used as the target weight of each storage engine in the target time period.

[0021] In one embodiment, generating the initial weight of each storage engine that meets the energy data condition according to the predicted new energy data, the traditional energy data and the energy consumption per unit time of each storage engine includes:

[0022] Determining total energy data based on the traditional energy data and the predicted new energy data;

[0023] Substitute the total energy data and the energy consumption per unit time of each storage engine into a first relational formula to determine the initial weight of each storage engine that meets the energy data conditions, wherein the first relational formula represents the relationship between the product of the energy consumption per unit time of each storage engine and the corresponding initial weight of the storage engine and the total energy data.

[0024] In one embodiment, if the initial weight satisfies the load condition, using the initial weight as the target weight of each storage engine within the target time period includes:

[0025] Obtaining load data of each storage engine storing monitoring data;

[0026] Determining total load data of the storage engine according to the initial weight and the load data;

[0027] If the total load data of the storage engine is greater than the predicted monitoring data, it indicates that the initial weight meets the load condition of the storage monitoring data, and the initial weight is used as the target weight of each storage engine in the target time period.

[0028] In a second aspect, the present application further provides a monitoring data storage device, the device comprising:

[0029] A first acquisition module is used to acquire a monitoring data set within a target time period; the monitoring data set includes monitoring data of multiple user identifiers;

[0030] A scheduling module, configured to schedule each of the monitoring data according to a target weight of each storage engine, and determine a target storage engine corresponding to each of the monitoring data; the target weight represents the weight of the amount of data stored in the storage engine by the monitoring data within the target time period;

[0031] The storage module is used to store each of the monitoring data in the corresponding target storage engine.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0034] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0035] The monitoring data storage method, device, computer equipment, computer-readable storage medium and computer program product are as follows: first, a monitoring data set is obtained, and the monitoring data set includes monitoring data of multiple user identifiers; second, each monitoring data is scheduled according to the target weight of each storage engine, and the target storage engine corresponding to each monitoring data is determined; finally, each monitoring data is stored in the corresponding target storage engine. By setting up multiple storage engines, the acquired massive monitoring data can be stored in the corresponding target storage engine respectively, which reduces the queuing waiting time for the monitoring data to be stored in the storage engine, improves the storage efficiency, and also reduces the storage pressure of the storage engine, thereby improving the performance of the storage engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A diagram of an application environment of a monitoring data storage method in one embodiment;

[0038] Figure 2 A schematic diagram of a flow chart of a monitoring data storage method in one embodiment;

[0039] Figure 3 A schematic diagram of a process of storing each monitoring data into a corresponding target storage engine according to an embodiment;

[0040] Figure 4 A flowchart of a method for determining target weights of various storage engines in one embodiment;

[0041] Figure 5 A schematic diagram of a process for determining a target weight within a target time period in one embodiment;

[0042] Figure 6 A schematic diagram of a process flow for generating initial weights that meet energy data conditions in one embodiment;

[0043] Figure 7 A schematic diagram of a process for generating a target weight that satisfies both energy conditions and load conditions in one embodiment;

[0044] Figure 8 is a structural block diagram of a monitoring data storage device in one embodiment;

[0045] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The monitoring data storage method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the monitoring terminal 102 communicates with the storage gateway 106 through the network. The monitoring data is stored in the storage engine 108 through the storage gateway 106. Among them, the storage gateway 106 can be integrated on the scheduling server 104, or it can be placed on the cloud or other network servers. The storage engine 108 can also be integrated on the scheduling server 104, or it can be placed on the cloud or other network servers. 110 is the power supply, which supplies power to the scheduling server 104, storage gateway 106, storage engine 108 of the system, or the scheduling server 104 integrated with the storage gateway 106 and storage engine 108, so that each device maintains operation. The storage gateway 106 or the scheduling server 104 integrated with the storage gateway 106 obtains a monitoring data set within a target time period; the monitoring data set includes monitoring data of multiple user identifiers; according to the target weight of each storage engine, each monitoring data is scheduled to determine the target storage engine corresponding to each monitoring data; the target weight represents the weight of the amount of data stored in the storage engine of the monitoring data within the target time period; each monitoring data is stored in the corresponding target storage engine. The scheduling server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0048] In an exemplary embodiment, Figure 2 As shown, a monitoring data storage method is provided, which is applied to Figure 1 The scheduling server 104 in the example is used to illustrate, including the following steps S202 to S206. Among them:

[0049] Step S202, obtaining a monitoring data set within a target time period; the monitoring data set includes monitoring data of multiple user identifiers.

[0050] The monitoring data set includes monitoring data of multiple user identifiers, for example, monitoring data of a monitoring device with a user identifier of 1 and monitoring data of a monitoring device with a user identifier of n.

[0051] Optionally, it is assumed that the target time period is T. The scheduling server obtains monitoring data of N monitoring devices within the time period T, where N represents the Nth monitoring device.

[0052] Optionally, the scheduling server may also obtain monitoring data of N monitoring devices within a time period T through a storage gateway, where N represents the Nth monitoring device.

[0053] Step S204: Schedule each monitoring data according to the target weight of each storage engine, and determine the target storage engine corresponding to each monitoring data.

[0054] The target weight represents the weight of the amount of monitoring data stored in the storage engine within the target time period.

[0055] Optionally, the scheduling server pre-calculates the target weight of each storage engine. The target weight is determined by the scheduling server based on the historical load data of the storage engine and the historical energy data of the storage engine. If the target time period is T, the historical load data and historical energy data should come from data no earlier than the time period (T-1), such as historical load data and historical energy data from the time period (T-2). If four storage engines are stored, numbered 1, 2, 3, and 4, the weights corresponding to each storage engine are determined by calculation to be 0.1, 0.2, 0.3, and 0.4 respectively.

[0056] Optionally, the scheduling server schedules N monitoring data according to the target weight of each storage engine, determines the target storage engine corresponding to each monitoring data, 10% of the data volume Date1 of the N monitoring data should be stored in the storage engine numbered 1, that is, the target storage engine corresponding to these monitoring data is storage engine 1; 20% of the data volume Date2 of the N monitoring data should be stored in the storage engine numbered 2, that is, the target storage engine corresponding to these monitoring data is storage engine 2, and so on, until all N monitoring data are stored in the storage engine.

[0057] Step S206: store each monitoring data into the corresponding target storage engine.

[0058] Optionally, the scheduling server stores the monitoring data under each user ID in the corresponding target storage engine. For example, the scheduling server stores date1 in storage engine 1 and stores date2 in storage engine 2.

[0059] Optionally, the scheduling server stores each monitoring data to a corresponding target storage engine through a storage gateway.

[0060] In the above monitoring data storage method, a monitoring data set is first obtained, and the monitoring data set includes monitoring data of multiple user identifiers; secondly, each monitoring data is scheduled according to the target weight of each storage engine, and the target storage engine corresponding to each monitoring data is determined; finally, each monitoring data is stored in the corresponding target storage engine. By setting multiple storage engines, the acquired massive monitoring data can be stored in the corresponding target storage engine respectively, which reduces the queuing waiting time for the monitoring data to be stored in the storage engine, improves the storage efficiency, and also reduces the storage pressure of the storage engine, thereby improving the performance of the storage engine.

[0061] In an exemplary embodiment, Figure 3 As shown, each monitoring data is stored in the corresponding target storage engine, including steps S302 to S304. Among them:

[0062] Step S302 : for each user identified monitoring data, the monitoring data is segmented according to a segmentation rule to obtain a first block of metadata and a last block of metadata.

[0063] Optionally, the scheduling server divides the monitoring data for each user ID according to the division rule through the storage gateway to obtain metadata, wherein the metadata includes the first block metadata and the last block metadata.

[0064] Optionally, the scheduling server divides the monitoring data of each user ID into fixed blocks according to the segmentation rule through the storage gateway, and obtains the first block metadata and the last block metadata of the monitoring data of each user ID.

[0065] Step S304: write the first block metadata and the last block metadata into the target storage engine through the interface of the target storage engine.

[0066] Optionally, the scheduling server converts the write request of the first block metadata and the last block metadata into the corresponding storage engine interface through the storage gateway, and writes it into the corresponding storage engine through the storage engine interface. Writing only the first block metadata and the last block metadata can greatly save metadata storage overhead. Since these metadata are unstructured, choose to store these metadata in NoSQL mode to improve the efficiency and performance of metadata management. It should be noted that the storage engine includes but is not limited to high-density storage, high-performance storage, cache layer, etc.

[0067] Optionally, when the storage engine receives a data read instruction, the intermediate metadata is calculated according to the first block metadata, the last block metadata and the segmentation rule stored in the storage engine, so that the monitoring data can be read completely.

[0068] In this embodiment, by recording the first block metadata and the last block metadata of each monitoring data, the monitoring video data storage overhead can be greatly saved and the performance of the storage engine can be improved.

[0069] In an exemplary embodiment, Figure 4 As shown, the method for determining the target weight of each storage engine includes steps S402 to S406. Among them:

[0070] Step S402: Acquire historical monitoring data of multiple user identifiers within a historical time period and historical energy data consumed to maintain the operation of the storage engine.

[0071] Wherein, when the target time period is T, the historical time period is not earlier than (T-1). The historical energy data includes historical new energy data and historical traditional energy data, wherein the energy data may be power.

[0072] Optionally, the scheduling server obtains the initial historical monitoring data of multiple user identifiers within the historical time period and the initial historical energy data consumed to maintain the operation of the storage engine. For example, the initial historical monitoring data of multiple user identifiers in the time period (T-1) is obtained, and the initial historical monitoring data of each 5 minutes is obtained within the time period (T-1). The corresponding initial historical energy data consumed to maintain the operation of the storage engine is also divided into 5-minute granularity to obtain each 5-minute initial historical energy data. The initial historical energy data is assumed to be wind energy, wherein the wind power of 3 consecutive 5-minute periods can be calculated to obtain 15 minutes of new energy short-term data; and then 8 consecutive 15-minute wind energy data are processed to obtain 2 hours of long-term wind energy data.

[0073] Optionally, the dispatch server preprocesses the initial historical monitoring data and the initial historical energy data. The invalid data is cleaned from the acquired initial historical monitoring data to obtain the historical monitoring data. The load conditions of each time period are plotted with time as the horizontal axis and the initial historical monitoring data as the vertical axis, and arranged in chronological order for subsequent model prediction.

[0074] Similarly, the dispatch server obtains the initial historical energy data, cleans up the invalid data, and obtains the historical energy data. The historical new energy data is selected from the historical energy data. The new energy data of each time period is plotted with time as the horizontal axis and new energy data, such as wind power, as the vertical axis, and arranged in chronological order for subsequent model prediction.

[0075] Step S404: input the historical monitoring data and the historical new energy data into the prediction model, and obtain the prediction result of the prediction model, where the prediction result includes the predicted monitoring data and the predicted new energy data within the target time period.

[0076] Optionally, the scheduling server uses part of the historical monitoring data and historical new energy data in the time period (T-1) as a sample set and part as a training set, and uses the training set to train the recurrent neural network GRU (Gated Recurrent Unit) to obtain a trained prediction model. The historical monitoring data and historical new energy data as sample sets are input into the prediction model, and the prediction model outputs the predicted monitoring data in the time period T, which is recorded as and predicted new energy data, denoted as Among them, GRU is a type of RNN, and RNN is very effective in processing time sequence information. Each input is associated with the previous input. This ability to process sequence information enables RNN to perceive temporal correlation well.

[0077] Step S406, determining the target weight of each storage engine within the target time period according to the predicted monitoring data, the predicted new energy data and the constraints of the storage engine.

[0078] Optionally, the scheduling server predicts monitoring data, predicts new energy data and storage engine constraints, and calculates the target weight of each storage engine within the target time period, so that the traditional energy consumption is as low as possible under the premise of satisfying the monitoring data, thereby achieving the goal of energy saving.

[0079] This embodiment calculates the target weight of each storage engine in time period T through the data of time period (T-1). When there is insufficient data in the future, it is very likely that the scheduling will be delayed, resulting in increased energy consumption of traditional energy. Therefore, this embodiment obtains historical information of load and new energy and adopts a recurrent neural network to predict information at future moments, thereby increasing the supply of new energy while ensuring the storage prediction monitoring data requirements.

[0080] In another embodiment, new energy data within a target time period is obtained; if the target time period is T, the scheduling server uses the monitoring data within the target time period as historical monitoring data, and uses the new energy data within the target time period as historical new energy data; the historical monitoring data and the historical new energy data are input into the prediction model, and the predicted monitoring data and predicted new energy data of the prediction model within the next target time period (T+1) are obtained; according to the predicted monitoring data, the predicted new energy data and the constraints of the energy consumption of each storage engine, the target weight of each storage engine within the next target time period is determined. The scheduling server obtains the monitoring data set within the next target time period (T+1) and the target weight within the next target time period (T+1); the monitoring data set includes monitoring data of multiple user identifiers; the next target time period is, for example, (T+1); re-execute the steps of scheduling the monitoring data set according to the target weight of each storage engine, determining the target storage engine corresponding to each monitoring data; and storing each monitoring data into the corresponding target storage engine.

[0081] In this embodiment, by calculating the target weight for each time period and adjusting it at any time according to the timing, the accuracy of the target weight can be improved, making it closer to the actual situation. It also allows for subsequent scheduling while taking into account the storage load of the monitoring data volume itself and the energy consumption of maintaining the running storage engine, thereby reducing the use of traditional energy.

[0082] In an exemplary embodiment, Figure 5 As shown, the constraint conditions include the load conditions of the storage engine storing monitoring data and the energy data conditions of the running storage engine; according to the predicted monitoring data, the predicted new energy data and the constraint conditions of each storage engine, the target weight of each storage engine within the target time period is determined, including steps S502 to S506. Among them:

[0083] Step S502, obtaining traditional energy data and energy consumption per unit time of each storage engine.

[0084] Optionally, the dispatch server obtains traditional energy data through the storage gateway, denoted as ; and obtain the energy consumption per unit time of each storage engine, recorded as Among them, the energy data may be power.

[0085] Step S504, generating initial weights of each storage engine that meets the energy data condition based on the predicted new energy data, traditional energy data and the energy consumption per unit time of each storage engine.

[0086] Among them, the predicted new energy data is recorded as .

[0087] Optionally, the dispatch server stores the predicted new energy data through the storage gateway. , Traditional energy data And the energy consumption per unit time of each storage engine , generate the initial weights of each storage engine that meets the energy data conditions ( ).

[0088] Step S506: If the initial weight meets the load condition, the initial weight is used as the target weight of each storage engine in the target time period.

[0089] Optionally, the scheduling server determines the generated initial weight ( ) Whether the load condition is met, if so, the initial weight is used as the target weight of each storage engine in the target time period. If not, the initial weight is readjusted to form a new initial weight.

[0090] In this embodiment, the target weight that satisfies both the load condition and the energy data condition is generated by the load condition of the storage engine storing the monitoring data and the energy data condition of the storage engine running. When scheduling each monitoring data according to the weight, the energy depletion may be low while satisfying the user load, which is conducive to reducing carbon emissions.

[0091] In an exemplary embodiment, Figure 6 As shown, according to the predicted new energy data, traditional energy data and the energy consumption per unit time of each storage engine, the initial weight of each storage engine that meets the energy data condition is generated, including steps S602 to S604. Among them:

[0092] Step S602, determining total energy data based on traditional energy data and predicted new energy data.

[0093] Optionally, the dispatch server uses traditional energy data according to formula (1) and forecast new energy data , determine the total energy data Total_Energy.

[0094] + =Total_Energy formula (1)

[0095] In the formula, Indicates traditional energy data; Represents predicted new energy data; Total_Energy represents total energy data.

[0096] Step S604: Substitute the total energy data and the energy consumption per unit time of each storage engine into the first relational expression to determine the initial weight of each storage engine that meets the energy data condition.

[0097] The first relational expression represents the relationship between the product of the energy consumption per unit time of each storage engine and the initial weight of the corresponding storage engine and the total energy data.

[0098] Optionally, the energy consumption per unit time of each storage engine ( ); the initial weight of the corresponding storage engine is recorded as ( ). The scheduling server substitutes the total energy data and the energy consumption per unit time of each storage engine into the first relational expression to determine the initial weight of each storage engine that meets the energy data condition. The first relational expression is shown in formula (2).

[0099] Total_Energy= * + * + * +...+ * Formula (2)

[0100] In the formula, ( ) represents the energy consumption per unit time of each storage engine; ( ) represents the initial weight of each storage engine; Total_Energy represents the total energy data.

[0101] In this embodiment, the initial weight of each storage engine that meets the energy data condition can be accurately calculated through the energy data and the first relational expression.

[0102] It should be noted that if the predicted new energy data is data for time period T, the traditional energy data for time period T is obtained accordingly; then the calculated initial weights of each storage engine also conform to time period T.

[0103] In an exemplary embodiment, Figure 7 As shown, if the initial weight meets the load condition, the initial weight is used as the target weight of each storage engine in the target time period, including steps S702 to S706.

[0104] Step S702: Obtain the load data of each storage engine storing monitoring data.

[0105] The load data includes memory load, network load, and read / write request load.

[0106] Optionally, the scheduling server obtains the load data of each storage engine storing monitoring data within the time period T. The load data of each storage engine storing monitoring data is calculated by formula (3).

[0107] Formula (3)

[0108] in, Indicates the load of the i-th storage engine; Represents the memory of the i-th storage engine; Indicates the read and write request load of the i-th storage engine; Indicates the network load of the i-th storage engine; Respectively represent the coefficients corresponding to the memory, read and write request load, and network load of the i-th storage engine.

[0109] Step S704, determining the total load data of the storage engine according to the initial weight and the load data.

[0110] Optionally, the scheduling server calculates the initial weight of each storage engine ( ) and the calculated load data for each storage engine Substitute them into formula (4) to get the total load of the storage engine: .

[0111] Formula (4)

[0112] In the formula, Indicates the total load of the storage engine; Indicates the initial weight of each storage engine; Indicates the load of the i-th storage engine.

[0113] Step S706: If the total load data of the storage engine is greater than the predicted monitoring data, it indicates that the initial weight meets the load condition of the storage monitoring data, and the initial weight is used as the target weight of each storage engine in the target time period.

[0114] Optionally, the scheduling server uses formula (5) to determine the size relationship between the total load data of the storage engine and the predicted monitoring data. If the total load data of the storage engine is greater than the predicted monitoring data, it means that the initial weight meets the load condition of the storage monitoring data, and the initial weight is used as the target weight of each storage engine in the target time period. If the total load data of the storage engine is greater than the predicted monitoring data, it means that the initial weight does not meet the load conditions for storing the monitoring data. The initial weight needs to be readjusted and the above judgment steps need to be re-executed.

[0115] Formula (5)

[0116] In the formula, Indicates the total load of the storage engine; Represents forecast monitoring data.

[0117] In this embodiment, by judging whether the load conditions are met through load data, the initial weights of each storage engine that meets the energy data conditions can be accurately calculated, and by continuing to judge whether the initial weights meet the load conditions, both the storage load of the monitored data volume itself and the energy consumption target weight for maintaining the operation of the storage engine can be obtained.

[0118] In one embodiment, there are two stages: one is the stage of generating target weights, and the other is the stage of scheduling using target weights.

[0119] The stage of generating target weights: The scheduling server obtains the initial historical monitoring data of multiple user identifiers within the historical time period and the initial historical energy data consumed to maintain the operation of the storage engine. For example, the initial historical monitoring data of multiple user identifiers in the time period (T-1) is obtained. In the time period (T-1), the initial historical monitoring data of each 5-minute period is divided into 5-minute granularity to obtain the initial historical energy data of each 5-minute period. The corresponding initial historical energy data consumed to maintain the operation of the storage engine is also divided into 5-minute granularity to obtain the initial historical energy data of each 5-minute period. The initial historical energy data is assumed to be wind energy, where the wind power of 3 consecutive 5-minute periods can be calculated to obtain 15-minute short-term data of new energy; and then the continuous 8 15-minute wind energy data can be processed to obtain 2-hour long-term data of wind energy. The scheduling server preprocesses the initial historical monitoring data and the initial historical energy data. The invalid data is cleared from the obtained initial historical monitoring data to obtain the historical monitoring data. Then, the load conditions of each time period are plotted with time as the horizontal axis and the initial historical monitoring data as the vertical axis, and arranged in time series for subsequent model prediction. Similarly, the dispatch server obtains the initial historical energy data, cleans up the invalid data, and obtains the historical energy data. Filter out the historical new energy data from the historical energy data. Use time as the horizontal axis and new energy data, such as wind power, as the vertical axis to plot the new energy data for each time period and arrange them in chronological order for subsequent model prediction. The dispatch server uses part of the historical monitoring data and historical new energy data for the time period (T-1) as a sample set and part as a training set. It uses the training set to train the recurrent neural network GRU (Gated Recurrent Unit) to obtain a trained prediction model. The historical monitoring data and historical new energy data as sample sets are input into the prediction model, and the prediction model outputs the predicted monitoring data within the time period T, recorded as and predicted new energy data, denoted as .

[0120] The dispatch server obtains traditional energy data through the storage gateway, which is recorded as ; and obtain the energy consumption per unit time of each storage engine, recorded as Among them, the energy data may be power.

[0121] The dispatch server uses traditional energy data according to formula (1) and forecast new energy data , determine the total energy data Total_Energy.

[0122] + =Total_Energy formula (1)

[0123] In the formula, Indicates traditional energy data; Represents predicted new energy data; Total_Energy represents total energy data.

[0124] The scheduling server substitutes the total energy data and the energy consumption per unit time of each storage engine into the first relational expression to determine the initial weight of each storage engine that meets the energy data condition. The first relational expression is shown in formula (2).

[0125] Total_Energy= * + * + * +...+ * Formula (2)

[0126] In the formula, ( ) represents the energy consumption per unit time of each storage engine; ( ) represents the initial weight of each storage engine; Total_Energy represents the total energy data.

[0127] The scheduling server obtains the load data of each storage engine storing monitoring data within the time period T. The load data of each storage engine storing monitoring data is calculated by formula (3).

[0128] Formula (3)

[0129] in, Indicates the load of the i-th storage engine; Represents the memory of the i-th storage engine; Indicates the read and write request load of the i-th storage engine; Indicates the network load of the i-th storage engine; Respectively represent the coefficients corresponding to the memory, read and write request load, and network load of the i-th storage engine.

[0130] The scheduling server calculates the initial weight of each storage engine ( ) and the calculated load data for each storage engine Substitute them into formula (4) to get the total load of the storage engine: .

[0131] Formula (4)

[0132] In the formula, Indicates the total load of the storage engine; Indicates the initial weight of each storage engine; Indicates the load of the i-th storage engine.

[0133] The scheduling server uses formula (5) to determine the relationship between the total load data of the storage engine and the predicted monitoring data. If the total load data of the storage engine is greater than the predicted monitoring data, it means that the initial weight meets the load condition of the storage monitoring data, and the initial weight is used as the target weight of each storage engine in the target time period T. If the total load data of the storage engine is greater than the predicted monitoring data, it means that the initial weight does not meet the load conditions for storing the monitoring data. The initial weight needs to be readjusted and the above judgment steps need to be re-executed.

[0134] Formula (5)

[0135] In the formula, Indicates the total load of the storage engine; Represents forecast monitoring data.

[0136] Use target weights to schedule phases:

[0137] Assume that in a large-scale video surveillance scenario, high-concurrency, large-capacity, high-reliability, and low-energy storage is required to store these video surveillance data.

[0138] The scheduling server obtains the monitoring data of N monitoring devices within the time period T, where N represents the Nth monitoring device. The scheduling server schedules the N monitoring data according to the target weight of each storage engine, and determines the target storage engine corresponding to each monitoring data. 10% of the data volume Date1 of the N monitoring data should be stored in the storage engine numbered 1, that is, the target storage engine corresponding to these monitoring data is storage engine 1; 20% of the data volume Date2 of the N monitoring data should be stored in the storage engine numbered 2, that is, the target storage engine corresponding to these monitoring data is storage engine 2, and so on, until all N monitoring data are stored in the storage engine. The scheduling server divides the monitoring data of each user ID into fixed block sizes according to the segmentation rules through the storage gateway; and obtains the first block metadata and the last block metadata of the monitoring data of each user ID. The scheduling server converts the write request of the first block metadata and the last block metadata into the corresponding storage engine interface through the storage gateway, and writes them into the corresponding storage engine through the storage engine interface. Writing only the first block metadata and the last block metadata can greatly save metadata storage overhead. Since these metadata are unstructured, we choose to store them in NoSQL to improve the efficiency and performance of metadata management. It should be noted that the storage engine includes but is not limited to high-density storage, high-performance storage, cache layer, etc.

[0139] When the storage engine receives a data read instruction, it calculates the intermediate metadata based on the first block metadata, the last block metadata, and the segmentation rules stored in the storage engine, so that the monitoring data can be read completely.

[0140] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0141] Based on the same inventive concept, the embodiment of the present application also provides a monitoring data storage device for implementing the monitoring data storage method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more monitoring data storage device embodiments provided below can refer to the limitations on the monitoring data storage method above, and will not be repeated here.

[0142] In an exemplary embodiment, Figure 8 As shown, a monitoring data storage device is provided, including: a first acquisition module 801, a scheduling module 802 and a storage module 803, wherein:

[0143] The first acquisition module 801 is used to acquire a monitoring data set within a target time period; the monitoring data set includes monitoring data of multiple user identifiers.

[0144] The scheduling module 802 is used to schedule each monitoring data according to the target weight of each storage engine and determine the target storage engine corresponding to each monitoring data; the target weight represents the weight of the amount of monitoring data stored in the storage engine within the target time period.

[0145] The storage module 803 is used to store each monitoring data into the corresponding target storage engine.

[0146] In an exemplary embodiment, the storage module 803 is also used to segment the monitoring data of each user identifier according to the segmentation rules to obtain the first block metadata and the last block metadata; and write the first block metadata and the last block metadata into the target storage engine through the interface of the target storage engine.

[0147] In an exemplary embodiment, a monitoring data storage device is provided, further comprising:

[0148] The second acquisition module is used to acquire historical monitoring data of multiple user identifiers within a historical time period and historical energy data consumed to maintain the operation of the storage engine; wherein the historical energy data includes historical new energy data.

[0149] The prediction module is used to input historical monitoring data and historical new energy data into the prediction model and obtain the prediction results of the prediction model, which include the predicted monitoring data and predicted new energy data within the target time period.

[0150] The determination module is used to determine the target weight of each storage engine within the target time period according to the predicted monitoring data, the predicted new energy data and the constraints of the storage engine.

[0151] In an exemplary embodiment, the constraint conditions include a load condition for the storage engine to store monitoring data and an energy data condition for running the storage engine; the determination module includes:

[0152] The acquisition unit is used to acquire traditional energy data and the energy consumption per unit time of each storage engine.

[0153] The initial weight generating unit is used to generate the initial weights of each storage engine that meets the energy data conditions according to the predicted new energy data, traditional energy data and the energy consumption per unit time of each storage engine.

[0154] The target weight generating unit is used to use the initial weight as the target weight of each storage engine within the target time period if the initial weight meets the load condition.

[0155] In an exemplary embodiment, the initial weight unit includes:

[0156] The total energy determination subunit is used to determine the total energy data based on the traditional energy data and the predicted new energy data.

[0157] The initial weight generation subunit is used to substitute the total energy data and the energy consumption per unit time of each storage engine into the first relational expression to determine the initial weight of each storage engine that meets the energy data conditions, wherein the first relational expression represents the relationship between the product of the energy consumption per unit time of each storage engine and the initial weight of the corresponding storage engine and the total energy data.

[0158] In an exemplary embodiment, the target weight generating unit includes:

[0159] The acquisition subunit is used to obtain the load data of the storage monitoring data of each storage engine.

[0160] The total load data determination subunit is used to determine the total load data of the storage engine according to the initial weight and the load data.

[0161] The target weight generating subunit is used to indicate that the initial weight meets the load condition of the storage monitoring data if the total load data of the storage engine is greater than the predicted monitoring data, and the initial weight is used as the target weight of each storage engine in the target time period.

[0162] Each module in the monitoring data storage device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0163] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store monitoring data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a monitoring data storage method is implemented.

[0164] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0167] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0168] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0169] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0170] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A monitoring data storage method, characterized in that: The method comprises: Acquire a monitoring data set within a target time period; the monitoring data set includes monitoring data of multiple user identifiers; According to the target weight of each storage engine, each monitoring data is scheduled to determine the target storage engine corresponding to each monitoring data; the target weight represents the weight of the amount of data stored in the storage engine by the monitoring data within the target time period; Each of the monitoring data is stored in a corresponding target storage engine.

2. The method according to claim 1, characterized in that The storing of each monitoring data into a corresponding target storage engine includes: For each of the monitoring data of the user identification, the monitoring data is segmented according to the segmentation rule to obtain the first block metadata and the last block metadata; The first block metadata and the last block metadata are written into the target storage engine through the interface of the target storage engine.

3. The method according to claim 1, characterized in that The method for determining the target weight of each storage engine includes: Acquire historical monitoring data of multiple user identifiers within a historical time period and historical energy data consumed to maintain the operation of the storage engine; wherein the historical energy data includes historical new energy data; Inputting the historical monitoring data and the historical new energy data into a prediction model, and obtaining a prediction result of the prediction model, wherein the prediction result includes the predicted monitoring data and the predicted new energy data within the target time period; The target weight of each of the storage engines within the target time period is determined according to the predicted monitoring data, the predicted new energy data and the constraints of the storage engines.

4. The method according to claim 3, characterized in that The constraint conditions include a load condition for storing monitoring data of the storage engine and an energy data condition for running the storage engine; determining a target weight of each storage engine within the target time period according to the predicted monitoring data, the predicted new energy data and the constraint conditions of each storage engine includes: Acquire traditional energy data and energy consumption per unit time of each storage engine; The predicted new energy data, the traditional energy data and the energy consumption per unit time of each storage engine are used to generate an initial weight of each storage engine that meets the energy data condition; If the initial weight meets the load condition, the initial weight is used as the target weight of each storage engine in the target time period.

5. The method according to claim 4, characterized in that The generating, based on the predicted new energy data, the traditional energy data and the energy consumption per unit time of each storage engine, the initial weight of each storage engine that meets the energy data condition comprises: Determining total energy data based on the traditional energy data and the predicted new energy data; According to the total energy data and the energy consumption per unit time of each storage engine, the first relational formula is substituted into the first relational formula to determine the initial weight of each storage engine that meets the energy data condition, wherein the first relational formula represents the relationship between the product of the energy consumption per unit time of each storage engine and the corresponding initial weight of the storage engine and the total energy data.

6. The method according to claim 4, characterized in that If the initial weight satisfies the load condition, taking the initial weight as the target weight of each storage engine within the target time period includes: Obtaining load data of each storage engine storing monitoring data; Determining total load data of the storage engine according to the initial weight and the load data; If the total load data of the storage engine is greater than the predicted monitoring data, it indicates that the initial weight meets the load condition of the storage monitoring data, and the initial weight is used as the target weight of each storage engine in the target time period.

7. A monitoring data storage device, characterized in that: The device comprises: A first acquisition module is used to acquire a monitoring data set within a target time period; the monitoring data set includes monitoring data of multiple user identifiers; A scheduling module, configured to schedule each of the monitoring data according to a target weight of each storage engine, and determine a target storage engine corresponding to each of the monitoring data; the target weight represents the weight of the amount of data stored in the storage engine by the monitoring data within the target time period; The storage module is used to store each of the monitoring data in the corresponding target storage engine.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images