Low-load time sequence data extraction method and related device

Abstract

The invention discloses a low-load time sequence data extraction method and a related device, and belongs to the technical field of DCS. The method comprises the following steps: receiving related information of a request point measurement task sent by a client through a server; estimating the total amount of required data according to the related information of the request measuring point task; calculating the fragment size according to the current CPU utilization rate; calculating the number of the fragments according to the total amount of the required data and the sizes of the fragments; the method comprises the following steps: decomposing and sorting a request measuring point task, and extracting data of measuring points; according to the method, the CPU idle value of the current system, the data extraction quantity and the CPU utilization rate can be combined, the size of the container applied for single data extraction can be dynamically adjusted, and a large demand is adjusted into a plurality of small demands. And sorting is carried out through service demand priorities, so that the overhead of query tasks in busy hours is remarkably reduced, and the lagging condition is relieved.

Description

Technical Field

[0001] The present invention belongs to the field of DCS technology and relates to a low-load time series data extraction method and related devices. Background Art

[0002] The rapid development of machine learning and artificial intelligence (AI) technologies is ushering in unprecedented changes and opportunities across all industries. Driven by these technologies, plug-in systems, as auxiliary tools within distributed control systems (DCSs), are increasingly relying on and demanding data, becoming key factors in improving system performance and optimizing decision-making. In DCS environments, the real-time and accuracy of data are crucial. To obtain sufficient data to support machine learning and AI algorithms, plug-in systems often need to frequently extract large amounts of data from the system. When this data extraction demand exceeds reasonable limits, it can significantly impact system resource allocation. In particular, CPU utilization, a key indicator of system processing power, is directly related to system efficiency and stability.

[0003] In practice, when an external system extracts large amounts of data, CPU utilization can skyrocket. This is because data processing and analysis consume significant computing resources, which are otherwise used to support the system's normal operations. When data extraction consumes excessive CPU resources, it crowds out normal operations, slowing system response times and potentially even causing service interruptions.

[0004] However, machine learning or AI training is not very time-sensitive. If data extraction for machine learning and AI training is given the same priority as regular business, the system may experience performance degradation during peak data extraction periods due to uneven resource allocation, which will have a significant impact on the normal operation of the system. Therefore, it is necessary to develop a new low-load time-series data extraction method. For AI plug-in systems that do not have high latency requirements, this method can appropriately lower the priority of data extraction and reduce the impact of large amounts of data extraction on the system. Summary of the Invention

[0005] The purpose of the present invention is to provide a low-load time series data extraction method and related devices to solve the technical problem that the data extraction requirements in the existing technology are too large, thereby squeezing out normal business and causing lag.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a low-load time series data extraction method, comprising the following steps: The server receives the relevant information of the requested measurement point task sent by the client; Estimate the total amount of data required based on the relevant information of the requested measurement point task; Calculate the shard size based on the current CPU utilization; Calculate the number of shards based on the total amount of data required and the shard size; Decompose and sort the requested measurement point tasks, and then extract the measurement point data.

[0007] Furthermore, the relevant information of the requested measurement point task includes: the total number of requested measurement points, a unique identification list ptid representing the measurement points, the required data start time, the required data end time, the sampling interval, the requested data type and the requested industrial control type.

[0008] Furthermore, the industrial control types of the requests include: manual real-time trend requests, internal program operation requests, external program operation requests and AI large-scale data operation requests; among them, manual real-time trend requests and internal program operation requests have the highest priority and are inserted at the head of the queue; external program operation requests have a medium priority and are inserted at the tail of the queue; AI large-scale data operation requests are fragmented and then inserted at the tail of the queue.

[0009] Furthermore, the step of estimating the total amount of required data based on the relevant information of the requested measurement point task specifically includes: The total amount of data required is estimated based on the total number of measurement points, the required data start time, the required data end time, and the sampling interval. The specific calculation formula is:

[0010]

[0011] Where n is the total number of measurement points; Ts is the start time of the required data; Te is the end time of the required data; S is the sampling interval; M1 is the total amount of data required for analog measurement points; M2 is the total amount of data required for switch measurement points.

[0012] Furthermore, the step of calculating the shard size based on the current CPU utilization specifically includes: The shard size is calculated based on the current CPU utilization and updated every 30 seconds. When the CPU utilization is above 80%, the shard size is 1 MB; when the CPU utilization is between 60% and 80%, the shard size is 3 MB; and when the CPU utilization is between 40% and 60%, the shard size is 5 MB. When the CPU utilization is below 40%, no shard size limit is set.

[0013] Furthermore, the step of calculating the number of shards based on the total amount of data required and the shard size specifically includes: The number of split tasks is calculated based on the total amount of required data and the shard size to obtain the number of shards, which is then fed back to the client. The client starts asynchronous listening based on the number of shards and the size of the dynamic data extraction container.

[0014] Furthermore, the steps of decomposing and sorting the requested measurement point tasks and then extracting the measurement point data specifically include: After determining the number of shards, the request measurement point task is decomposed into multiple query tasks according to the number of shards according to the unique identification list ptid of the measurement point; The priority of the number of shards is set to the lowest, and shard query requests are only executed when the query request buffer is empty; If the number of requests exceeds the request queue length, the request is discarded and the request failure is reported.

[0015] In a second aspect, the present invention provides a low-load time series data extraction system, comprising: The information receiving module is used for the server to receive the relevant information of the requested measurement point task sent by the client; The data total amount calculation module is used to estimate the required total amount of data based on the relevant information of the requested measurement point task; The fragment size calculation module is used to calculate the fragment size based on the current CPU utilization; The shard quantity calculation module is used to calculate the number of shards based on the total amount of required data and the shard size; The decomposition and sorting module is used to decompose and sort the requested measurement point tasks, and then extract the data of the measurement points.

[0016] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a low-load time series data extraction method, which dynamically adjusts the container size of a single data extraction application in combination with the CPU idle value, the number of data extractions and the CPU utilization of the current system, and adjusts large demands into multiple small demands. When the server receives a large number of data requests, it needs to sort them according to the priority of business needs. The newly arrived requests are inserted into the request queue according to the priority calculation rules, and the priority calculation method is adjusted according to the business situation of the industrial control system. If the queue exceeds the cache upper limit, it enters the flow control state. The present invention can significantly reduce the busy-hour query task overhead. After using this method, the conventional time series data query jamming situation is alleviated, the overall time consumption of AI calculation is high, and it is not sensitive to changes in the duration of a small amount of extracted data. In addition, the present invention can also reduce busy-hour network overhead, and the optimized transmission overhead distribution is more dispersed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a flow chart of the method of the present invention; Figure 2 is a schematic diagram of the system of the present invention; Figure 3 A technical principle diagram of an embodiment of the present invention; Figure 4 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0022] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise indicated, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present invention.

[0023] See also Figure 1 and Figure 3 , an embodiment of the present invention discloses a low-load time series data extraction method, specifically comprising: 1. Configuration and device type Memory queue configuration: Based on configuration files and comprehensive cost considerations in engineering practice, the default setting is 2G cache, which stores data to be sent.

[0024] 2. Receive industrial control timing data request: 1) The client packages and sends to the server the total number of requested measurement points, the unique identifier list ptid representing the measurement points, the required data start time, the required data end time, the sampling interval, the requested data type, and the requested industrial control type.

[0025] Among them, industrial control types are mainly divided into the following four types: manual real-time trend requests, internal program operation requests, external program operation requests, and AI large-scale data operation requests.

[0026] Manual real-time trend requests and internal program operation requests have the highest priority and are inserted at the head of the queue. External program operation requests have medium priority and are inserted at the tail of the queue. AI large-scale data operation requests are fragmented and then inserted at the tail.

[0027] 2) After receiving the request, the server estimates the total amount of data required based on the total number of measurement points, the start time of the required data, the end time of the required data, and the sampling interval. The specific calculation formula is:

[0028]

[0029] Where n is the total number of measurement points; Ts is the start time of the required data; Te is the end time of the required data; S is the sampling interval; M1 is the total amount of data required for analog measurement points; M2 is the total amount of data required for switch measurement points.

[0030] 3) The shard size is calculated based on the current CPU utilization and updated every 30 seconds. If the CPU utilization is above 80%, the shard size is 1MB; if the CPU utilization is between 60% and 80%, the shard size is 3MB; and if the CPU utilization is between 40% and 60%, the shard size is 5MB. If the CPU utilization is below 40%, no shard size limit is set.

[0031] 4) The number of split tasks is calculated based on the total amount of data required and the shard size, and the number of shards is obtained. The number of shards is fed back to the client, and the client starts asynchronous listening based on the number of shards and the size of the dynamic data extraction container.

[0032] 3. The server sorts the service requests 1) After determining the number of shards, the request measurement point task is decomposed into multiple query tasks according to the number of shards according to the unique identification list ptid of the measurement point; 2) The priority of the number of shards is set to the lowest, that is, the shard query request is only executed when the query request buffer is empty; 3) If the number of requests exceeds the request queue length, the request is discarded and the request is reported as failed.

[0033] The embodiment of the present invention combines the CPU idle value of the current system, the number of data extractions, and the CPU utilization rate to dynamically adjust the container size of a single data extraction request, and adjusts large demands into multiple small demands. When the server receives a large number of data requests, it needs to sort them according to the priority of business needs. The newly arrived requests are inserted into the request queue according to the priority calculation rules. The priority calculation method is adjusted according to the business situation of the industrial control system. If the queue exceeds the cache upper limit, it enters the flow control state. It can significantly reduce the busy-hour query task overhead: in actual tests, when there are multiple measurement points and time series data query requests for more than 2 months, the server compliance increases significantly, and some regular time series data queries are stuck. After using this method, the stuck situation of regular time series data queries is alleviated, the overall time consumption of AI calculations is high, and it is not sensitive to changes in the time length of a small amount of extracted data. It can also reduce busy-hour network overhead: the original solution will significantly and instantaneously increase the network port and switch load during the query process, and the optimized transmission overhead distribution is more dispersed.

[0034] See also Figure 2 An embodiment of the present invention discloses a low-load time series data extraction system, including an information receiving module, a data total amount calculation module, a shard size calculation module, a shard number calculation module and a decomposition and sorting module.

[0035] The information receiving module is used for the server to receive the relevant information of the request measurement point task sent by the client; In this module, the relevant information of the requested measurement point task includes: the total number of requested measurement points, the unique identification list ptid representing the measurement points, the required data start time, the required data end time, the sampling interval, the requested data type and the requested industrial control type.

[0036] The industrial control types of the requests include: manual real-time trend requests, internal program operation requests, external program operation requests and AI large-scale data operation requests; among them, manual real-time trend requests and internal program operation requests have the highest priority and are inserted at the head of the queue; external program operation requests have a medium priority and are inserted at the tail of the queue; AI large-scale data operation requests are fragmented and then inserted at the tail of the queue.

[0037] The data total amount calculation module is used to estimate the required total amount of data based on the relevant information of the requested measurement point task; In this module, the total amount of required data is estimated based on the total number of measurement points, the required data start time, the required data end time, and the sampling interval. The specific calculation formula is:

[0038]

[0039] Where n is the total number of measurement points; Ts is the start time of the required data; Te is the end time of the required data; S is the sampling interval; M1 is the total amount of data required for analog measurement points; M2 is the total amount of data required for switch measurement points.

[0040] The fragment size calculation module is used to calculate the fragment size based on the current CPU utilization; In this module, the shard size is calculated based on the current CPU utilization and updated every 30 seconds. If the CPU utilization is above 80%, it is 1MB; if it is between 60% and 80%, it is 3MB; and if it is between 40% and 60%, it is 5MB. If it is below 40%, no shard limit is set.

[0041] The shard quantity calculation module is used to calculate the number of shards based on the total amount of required data and the shard size; In this module, the number of split tasks is calculated based on the total amount of data required and the shard size to obtain the number of shards, which is then fed back to the client. The client starts asynchronous listening based on the number of shards and the size of the dynamic data extraction container.

[0042] The decomposition and sorting module is used to decompose and sort the requested measurement point tasks, and then extract the data of the measurement points.

[0043] In this module, after determining the number of shards, the request measurement point task is decomposed into multiple query tasks according to the number of shards according to the unique identification list ptid of the measurement point; The priority of the number of shards is set to the lowest, and shard query requests are only executed when the query request buffer is empty; If the number of requests exceeds the request queue length, the request is discarded and the request failure is reported.

[0044] The embodiment of the present invention dynamically adjusts the frequency of extracting time series data based on the CPU load changes during busy and idle periods, reducing the impact of batch data extraction on the system load. This is mainly reflected in the following three aspects: 1. Dynamic data extraction container adjustment: Based on the current system's CPU idle value and the relationship curve between the number of data extractions and CPU utilization, the container size of a single data extraction request is dynamically adjusted to adjust large requests into multiple small requests.

[0045] 2. Dynamically adjust the priority of data extraction requests: When the server receives a large number of data requests, it needs to sort them according to the priority of business needs. New requests are inserted into the request queue according to the priority calculation rules. The priority calculation method is adjusted according to the business situation of the industrial control system.

[0046] 3. Server flow control measures: If the queue exceeds the cache limit, it enters the flow control state.

[0047] In one embodiment of the present invention, a computer device is provided. Figure 4 The computer device includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the low-load time series data extraction method.

[0048] The present invention also provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device, used to store programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The processor may load and execute the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the low-load time series data extraction method described in the above-mentioned embodiment.

[0049] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A low-load time series data extraction method, characterized in that: The following steps are involved: The server receives the relevant information of the requested measurement point task sent by the client; Estimate the total amount of data required based on the relevant information of the requested measurement point task; Calculate the shard size based on the current CPU utilization; Calculate the number of shards based on the total amount of data required and the shard size; Decompose and sort the requested measurement point tasks, and then extract the measurement point data.

2. The low-load time series data extraction method according to claim 1, characterized in that: The relevant information of the requested measurement point task includes: the total number of requested measurement points, a unique identification list ptid representing the measurement points, the required data start time, the required data end time, the sampling interval, the requested data type and the requested industrial control type.

3. The low-load time series data extraction method according to claim 2, characterized in that: The industrial control types of the requests include: manual real-time trend requests, internal program operation requests, external program operation requests and AI large-scale data operation requests; among them, manual real-time trend requests and internal program operation requests have the highest priority and are inserted at the head of the queue; external program operation requests have a medium priority and are inserted at the tail of the queue; AI large-scale data operation requests are fragmented and then inserted at the tail of the queue.

4. The low-load time series data extraction method according to claim 2, characterized in that: The step of estimating the total amount of required data based on the relevant information of the requested measurement point task specifically includes: The total amount of data required is estimated based on the total number of measurement points, the required data start time, the required data end time, and the sampling interval. The specific calculation formula is: Where n is the total number of measurement points; Ts is the start time of the required data; Te is the end time of the required data; S is the sampling interval; M1 is the total amount of data required for analog measurement points; M2 is the total amount of data required for switch measurement points.

5. The low-load time series data extraction method according to claim 2, characterized in that: The step of calculating the shard size based on the current CPU utilization specifically includes: The shard size is calculated based on the current CPU utilization and updated every 30 seconds. When the CPU utilization is above 80%, the shard size is 1 MB; when the CPU utilization is between 60% and 80%, the shard size is 3 MB; and when the CPU utilization is between 40% and 60%, the shard size is 5 MB. When the CPU utilization is below 40%, no shard size limit is set.

6. The low-load time series data extraction method according to claim 2, characterized in that: The step of calculating the number of shards based on the total amount of data required and the shard size specifically includes: The number of split tasks is calculated based on the total amount of required data and the shard size to obtain the number of shards, which is then fed back to the client. The client starts asynchronous listening based on the number of shards and the size of the dynamic data extraction container.

7. The low-load time series data extraction method according to claim 2, characterized in that: The steps of decomposing and sorting the requested measurement point tasks and then extracting the measurement point data specifically include: After determining the number of shards, the request measurement point task is decomposed into multiple query tasks according to the number of shards according to the unique identification list ptid of the measurement point; The priority of the number of shards is set to the lowest, and shard query requests are only executed when the query request buffer is empty; If the number of requests exceeds the request queue length, the request is discarded and the request failure is reported.

8. A low-load time series data extraction system, characterized in that: include: The information receiving module is used for the server to receive the relevant information of the requested measurement point task sent by the client; The data total amount calculation module is used to estimate the required total amount of data based on the relevant information of the requested measurement point task; The fragment size calculation module is used to calculate the fragment size based on the current CPU utilization; The shard quantity calculation module is used to calculate the number of shards based on the total amount of required data and the shard size; The decomposition and sorting module is used to decompose and sort the requested measurement point tasks, and then extract the data of the measurement points.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images