A method, device and medium for synchronous execution of data batch processing and range conversion

Abstract

This invention discloses a method, apparatus, and medium for synchronously executing batch data processing and range transformation. The method includes: batch acquiring transmitted raw time-series data and parsing the raw time-series data into a first tree structure model according to a hierarchical structure; comparing the first tree structure model with a preset tree structure model to determine a target tree structure model that matches the first tree structure model within the preset tree structure model; performing range transformation on the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain a second tree structure model; and synchronously storing the first tree structure model and the second tree structure model in a time-series database. By reconstructing the data structure of time-series data, batch data storage and range transformation can be executed synchronously, reducing latency during batch data processing and improving hardware resource utilization and data write throughput.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus and medium for synchronously executing data batch processing and range conversion. Background Technology

[0002] With the development of computer technology, data is growing explosively. In some application scenarios, such as IoT systems, to reduce the frequency of data writing and thus reduce input / output interface (I / O) overhead, the received raw data is usually cached in a local queue or intermediate storage. When the amount of cached data reaches a preset batch size or the data accumulation time reaches a time threshold, a batch data storage operation is triggered. However, in existing technologies, after the batch data storage is completed, a separate range transformation process needs to be performed on the batch data. That is, the data is read again from the data storage and converted into actual physical quantities according to preset rules. After the range transformation is completed, the transformed data is then written back to the data storage.

[0003] In other words, in existing technologies, batch data storage and range transformation are performed step-by-step. After batch processing is complete, the entire batch of data must undergo range transformation before it can be written to the database, resulting in excessive end-to-end latency from data reception to database entry. In high-concurrency scenarios, the step-by-step execution of batch processing and range transformation requires frequent switching of the computing context, and the two independent data traversals (one for batch processing and one for range transformation) lead to high CPU utilization. Step-by-step processing means that a single data write must go through four stages: "caching - batch processing - range transformation - writing," resulting in a long process chain and a limited number of batches that can be processed per unit time. In high-concurrency scenarios, data accumulation is likely to occur, leading to low data write throughput. Summary of the Invention

[0004] This invention provides a method, device, and medium for synchronously executing data batch processing and range switching, thereby reducing latency during data batch processing and improving hardware resource utilization and data write throughput.

[0005] According to one aspect of the present invention, a method for synchronously performing data batch processing and range switching is provided, the method comprising: Batch acquire the transmitted raw time series data, and parse the raw time series data into a first tree structure model according to the hierarchical structure; The first tree structure model is compared with the preset tree structure model, and a target tree structure model that matches the first tree structure model is determined in the preset tree structure model; The range transformation of the data in the first tree structure model is performed according to the range transformation formula in the target tree structure model to obtain the second tree structure model. The first tree structure model and the second tree structure model are stored synchronously in the time-series database.

[0006] According to another aspect of the present invention, an apparatus for synchronously performing data batch processing and range switching is provided, the apparatus comprising: The data structure parsing module is used to acquire the transmitted raw time series data in batches and parse the raw time series data into a first tree structure model according to the hierarchical structure. The tree structure matching module is used to compare the first tree structure model with a preset tree structure model and determine the target tree structure model that matches the first tree structure model in the preset tree structure model. The range transformation module is used to transform the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain the second tree structure model. The synchronous storage module is used to synchronously store the first tree structure model and the second tree structure model in the time-series database.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the method for synchronous execution of data batch processing and range conversion as described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute the method for synchronously performing data batch processing and range conversion as described in any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method for synchronous execution of data batch processing and range conversion as described in any embodiment of the present invention.

[0010] The technical solution of this invention involves batch acquiring the transmitted raw time series data and parsing it into a first tree structure model according to a hierarchical structure. The first tree structure model is then compared with a preset tree structure model to determine a target tree structure model that matches the first tree structure model. A range transformation is performed on the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain a second tree structure model. The first and second tree structure models are then synchronously stored in a time series database. This solves the problem of separate execution of batch storage and range transformation in existing technologies. By reconstructing the data structure of the time series data, batch storage and range transformation can be executed synchronously, reducing latency during batch data processing and improving hardware resource utilization and data write throughput.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a method for synchronously executing data batch processing and range conversion according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram illustrating a range conversion interface provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram illustrating another range conversion interface provided in Embodiment 1 of the present invention; Figure 4 This is a flowchart of a method for synchronously executing data batch processing and range conversion according to Embodiment 2 of the present invention; Figure 5 This is a flowchart of another method for synchronously executing data batch processing and range conversion according to Embodiment 2 of the present invention; Figure 6 This is a schematic diagram of a device for synchronously executing data batch processing and range conversion according to Embodiment 3 of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device that implements the method for synchronous execution of data batch processing and range conversion according to embodiments of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a method for synchronously executing data batch processing and range transformation according to Embodiment 1 of the present invention. This embodiment is applicable to IoT data processing, where batch data storage and range transformation are performed. The method can be executed by a device for synchronously executing data batch processing and range transformation. This device can be implemented in hardware and / or software and can be configured in electronic devices such as computers, servers, or controllers.

[0017] For example, one application scenario of the method for synchronous execution of data batch processing and range transformation provided in this embodiment of the invention can be: in an Internet of Things (IoT) system, data transmission between devices is performed using the lightweight, low-bandwidth Message Queuing Telemetry Transport (MQTT) protocol. For example, data transmission between sensors, terminal devices, and servers. In an IoT system, different connectors can be subscribed to, raw time-series data can be obtained from MQTT, and cached in a message buffer queue. In the MQTT data subscription, the broker address, client ID, username, and password can be configured. Fault tolerance strategies for MQTT reconnection and automatic recovery of historical topic subscriptions can be formulated, and MQTT callback functions can also be configured. Based on the specified connector, the required MQTT topic can be constructed, and the information of the subscribed connector can be saved in a relational database for use by the fault tolerance strategy. After an electronic device subscribes to a topic, it can write the raw data of that topic into the message buffer queue. The message buffer queue can adopt a first-in, first-out (FIFO) mechanism and can be dynamically expanded to avoid data loss in high-concurrency scenarios. Subsequently, the method for synchronously executing data batch processing and range transformation according to the embodiments of the present invention performs range transformation synchronously while batch storing the original time series data, and writes it into the time series database. The time series database is used to store and manage time series data with timestamps and is a core component of IoT data storage.

[0018] like Figure 1 As shown, the method for synchronously executing data batch processing and range switching includes: Step 110: Batch acquire the transmitted raw time series data, and parse the raw time series data into a first tree structure model according to the hierarchical structure.

[0019] MQTT can directly connect to IoT databases. For example, by listening to messages published by MQTT clients, data can be immediately written to storage. However, this method of directly connecting MQTT to an IoT database requires specifying the complete time series to which the inserted data belongs in the sent message, making the message construction process complex. In this embodiment of the invention, by parsing the original time series data into a first tree structure model according to a hierarchical structure, the time series data can maintain a clear data relationship. When parsing the original time series data, it can be parsed layer by layer according to the data structure process to obtain the tree structure of the data, thereby generating the first tree structure model.

[0020] Optionally, the original time series data is parsed into a first-tree structure model according to the hierarchical structure, including: parsing the original time series data into a first-tree structure model according to the hierarchical structure of connector-connector instance-connector instance measurement point.

[0021] For example, according to root.connector.<connector_name> .<connections.display_name> .<dataPoints.name> The `(value| qc| localtimestamp)` method generates the first tree structure model. Here, "root.connector" represents the connector root node, "connector_name" represents the connector name, "connections.display_name" represents the connector instance name, "dataPoints.name" represents the connector instance measurement point name, "value" stores the data value, "qc" stores the data quality, and "localtimestamp" stores the timestamp.

[0022] Step 120: Compare the first tree structure model with the preset tree structure model, and determine the target tree structure model that matches the first tree structure model in the preset tree structure model.

[0023] The preset tree structure model shares the same core structure as the first tree structure model; both are tree structures generated according to the hierarchical structure of the data. The preset tree structure model can also include corresponding range transformation formulas. The preset tree structure model can be stored in a time-series database to ensure that configuration information is not lost. Alternatively, to accelerate data storage and range transformation, the preset tree structure model can be pre-loaded into the local cache of the electronic device before batch data processing. When performing batch storage and range transformation of the original time-series data, the preset tree structure model in the local cache is directly loaded to improve the speed of range transformation. For example, the preset tree structure model in the time-series database can be loaded into the local cache using a "key-value pair" format to quickly determine whether the time-series data needs range transformation.

[0024] For example, the default tree structure model is root.rc.<connector_name> .<connections.display_name> .<dataPoints.name> . (rcform| rcdecimalplaces). Where, "root.rc" represents the root node for range transformation configuration, "rcform" stores the range transformation formula, and "rcdecimalplaces" stores the number of decimal places to retain in the range transformation result.

[0025] By parsing the original time series data into a tree structure and storing the range transformation formula in the tree structure, a target tree structure model that matches the first tree structure model can be determined through rapid tree structure comparison, so as to perform range transformation on the original time series data. Since the change in data structure can accelerate the range transformation speed, batch storage and range transformation can be executed synchronously.

[0026] Optionally, before comparing the first tree structure model with the preset tree structure model, the method further includes: constructing a preset tree structure model from each time series data in the time series database according to the hierarchical structure, and displaying the preset tree structure model on the range transformation interface; in response to the user's configuration operation on the preset tree structure model in the range transformation interface, generating a range transformation formula, and displaying the range transformation formula in the preset tree structure model.

[0027] For example, Figure 2 This is a schematic diagram illustrating a range conversion interface provided according to Embodiment 1 of the present invention. Figure 2 As shown, connector A is a preset tree structure model containing single-level measurement points. Connector Mt is a preset tree structure model containing multiple levels of measurement points. In... Figure 2 In the range transformation interface shown, users can configure the preset tree structure model. For example, users can select the time series for which range transformation is required, and set the range transformation formula and the number of decimal places in the calculation results.

[0028] After the range transformation settings are completed, the range transformation formula can be displayed in the preset tree structure model. Figure 3 This is a schematic diagram illustrating another range switching interface provided in Embodiment 1 of the present invention. Figure 3 As shown, in connector A, channels 0 and 1 have corresponding range conversion formulas, while channel 2 is non-numerical data and cannot have its range conversion set. In connector B, channel 0 has a corresponding range conversion formula, while channel 1 and channel 2 are both non-numerical data and cannot have their range conversion set.

[0029] By using a pre-defined tree structure model and a range transformation interface, a modular design is formed to encapsulate the range transformation formula, enabling fine-grained range transformation settings and dynamic updates. This avoids the problems caused by the range transformation formula being fixed in the code in existing technologies. In existing technologies, because the range transformation formula is fixed in the code, the core code needs to be modified when adapting to different types of sensors or switching time series databases, resulting in high adaptation costs. In contrast, this invention, through a pre-defined tree structure model and a range transformation interface, adopts a modular design for the range transformation formula and the database writing interface, reducing the coupling between range changes and code. This allows users to modify the range transformation formula for different sensors in real time, improving compatibility and flexibility in data processing.

[0030] Step 130: Perform range transformation on the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain the second tree structure model.

[0031] The range transformation formula in the target tree structure model is used to transform the data in the first tree structure model, that is, to transform the value into rcvalue, to generate the second tree structure model. For example, the second tree structure model is root.connector.<connector_name> .<connections.display_name> .<dataPoints.name> rcvalue. “rcvalue” is the value after the range change.

[0032] Step 140: Store the first tree structure model and the second tree structure model synchronously in the time series database.

[0033] The `value` and `rcvalue` can be batch-synchronized and stored in the time-series database, improving data storage efficiency. By batch data storage and synchronous execution of range changes, data traversal and configuration database queries can be reduced, end-to-end write latency is lowered, the time difference of data sources processed in stages is eliminated, and CPU context switching and repeated data traversal are also reduced, thus improving throughput.

[0034] The technical solution of this embodiment acquires the transmitted raw time series data in batches and parses the raw time series data into a first tree structure model according to the hierarchical structure; compares the first tree structure model with a preset tree structure model, and determines a target tree structure model that matches the first tree structure model in the preset tree structure model; performs range transformation on the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain a second tree structure model; and stores the first tree structure model and the second tree structure model synchronously in the time series database. This solves the problem of batch storage and range transformation being executed step by step in the prior art. By reconstructing the data structure of the time series data, batch storage and range transformation can be executed synchronously, reducing the latency during data batch processing and improving hardware resource utilization and data write throughput.

[0035] Example 2 Figure 4 This is a flowchart of a method for synchronously executing data batch processing and range switching according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 4 As shown, the method includes: Step 410: Obtain the hardware resource operating status during batch data transmission, and determine the comprehensive resource load value based on the hardware resource operating status.

[0036] Hardware resource operating status includes, but is not limited to: CPU utilization, memory utilization, and disk I / O load. CPU utilization reflects the level of computing resource strain, memory utilization reflects the level of storage resource strain, and disk I / O load can be the percentage of peak hours, reflecting the level of write resource strain. For example, the overall resource load value is... .in, CPU utilization, For memory usage, Disk I / O load, , , These are the weights for CPU utilization, memory utilization, and disk I / O load, respectively. For example, , , .

[0037] Step 420: Obtain the real-time data reception frequency, the data reception reference frequency, and the batch processing reference time interval in the batch data transmission.

[0038] Real-time frequency of data reception This could be the number of new messages added to the MQTT buffer queue divided by the sampling period, reflecting the message inflow rate. Data reception baseline frequency. This can be determined based on the peak number of business messages, for example, a data reception baseline frequency of 300 messages per second (msg / s). Batch processing baseline time interval. It can be set according to the real-time requirements of the business processing scenario. For example, the batch processing baseline time interval is 1 second (s), and it can be set to 0.5s in business scenarios with high real-time requirements.

[0039] Step 430: Determine the actual batch processing time interval based on the comprehensive resource load value, real-time data reception frequency, data reception reference frequency, and batch processing reference time interval.

[0040] The batch processing baseline time interval is positively correlated with the overall resource load and negatively correlated with the real-time data reception frequency. That is, the higher the overall resource load or the lower the real-time data reception frequency, the larger the batch processing baseline time interval; conversely, the lower the overall resource load or the higher the real-time data reception frequency, the smaller the batch processing baseline time interval. For example, the actual batch processing time interval is... The actual time interval for batch processing can be retained to 3 decimal places and converted to milliseconds for configuring scheduled tasks.

[0041] Step 440: Based on the actual time interval of batch processing, acquire the transmitted raw time series data in batches.

[0042] The original time series data can be acquired in batches at intervals of T.

[0043] Optionally, based on the actual time interval of batch processing, the original time-series data to be transmitted is acquired in batches, including: acquiring the real-time backlog of the buffer queue, the baseline backlog of the buffer queue, the baseline value of a single batch, and the range of single batch sizes in the batch data transmission; determining the single batch size value based on the comprehensive resource load value, the real-time data reception frequency, the baseline data reception frequency, the real-time backlog of the buffer queue, the baseline backlog of the buffer queue, the baseline value of a single batch, and the range of single batch sizes; and acquiring the original time-series data to be transmitted in each batch according to the single batch size value based on the actual time interval of batch processing.

[0044] Among them, the real-time backlog of the buffer queue This could be the total number of messages currently pending in the buffer queue, reflecting the degree of message backlog. Buffer queue baseline backlog. Up to 900 records. Single batch baseline. This can be adjusted according to actual business needs, such as 300 items. The single batch size range can be the single batch size value. The scope of constraints, such as Between and between, It can be 10. It can be 10000.

[0045] For example, the single batch size value is . To extract functions, constraints Between and The batch size can be rounded to the nearest integer. The batch size of messages processed per transaction scales linearly with the availability of hardware resources and is smoothly amplified using a logarithmic function based on the input message pressure. This allows for automatic increase in the batch size when hardware is idle and there is a message backlog, and adaptive convergence when the load increases or the message backlog pressure decreases.

[0046] Optionally, based on the actual batch processing time interval, the original time-series data to be transmitted in each batch is obtained according to the single batch size value, including: obtaining the baseline number of threads and the range of the number of threads in the batch data transmission; determining the actual number of threads based on the baseline number of threads, the range of the number of threads, the comprehensive resource load value, the baseline value of the single batch, and the single batch size value; and obtaining the original time-series data to be transmitted in each batch according to the single batch size value and using the actual number of parallel processing threads based on the actual batch processing time interval.

[0047] Among them, the number of threads is the baseline. This can be the number of CPU cores. The number of threads can range from [value missing]. Between and Between. For example =1, This is twice the number of CPU cores. For example, the actual number of threads is... The baseline number of threads can be rounded to the nearest integer. The number of parallel processing threads is dynamically adjusted based on the availability of electronic device hardware resources, and combined with the current batch size, it is smoothly amplified using a logarithmic function. This fully utilizes the CPU's computing power when hardware resources are idle, while automatically suppressing thread growth when the load increases, thereby avoiding excessive context switching overhead and ensuring stable device operation.

[0048] Optionally, based on the actual time interval of batch processing, the original time series data to be transmitted is obtained in each batch according to the single batch size value and the actual number of parallel processing threads, including: determining the number of messages processed by each thread based on the single batch size value and the actual number of threads; and obtaining the original time series data to be transmitted according to the corresponding allocated number of messages by the actual number of parallel processing threads used in each batch according to the actual time interval of batch processing.

[0049] To ensure an even distribution of the number of messages that each thread needs to process, the formula is used. Determine the thread The number of messages processed. Among them, , This is for rounding down. (By...) It can automatically pre-load the remainder, and the maximum difference in the number of messages allocated to each thread is 1.

[0050] When acquiring raw time series data in batches, based on calculated parameters, B messages can be retrieved from the data buffer queue every T milliseconds, and these B messages can be distributed to N threads. For the assigned Each message, i.e., the original time series data, undergoes the following structural analysis and range transformation processing, and the processed data is stored in the time series database.

[0051] By determining the dynamic parameters for data transmission based on the operating status of hardware resources and the indicator values ​​at the time of message reception, the instability of experience-based adjustments can be avoided, ensuring that batch data processing can run smoothly.

[0052] Step 450: Parse the original time series data into a first tree structure model according to the hierarchical structure.

[0053] Optionally, the original time series data is parsed into a first-tree structure model according to the hierarchical structure, including: parsing the original time series data into a first-tree structure model according to the hierarchical structure of connector-connector instance-connector instance measurement point.

[0054] Step 460: Compare the first tree structure model with the preset tree structure model, and determine the target tree structure model that matches the first tree structure model in the preset tree structure model.

[0055] Optionally, before comparing the first tree structure model with the preset tree structure model, the method further includes: constructing a preset tree structure model from each time series data in the time series database according to the hierarchical structure, and displaying the preset tree structure model on the range transformation interface; in response to the user's configuration operation on the preset tree structure model in the range transformation interface, generating a range transformation formula, and displaying the range transformation formula in the preset tree structure model.

[0056] Step 470: Perform range transformation on the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain the second tree structure model.

[0057] Step 480: Store the first tree structure model and the second tree structure model synchronously in the time series database.

[0058] The technical solution of this invention determines batch data acquisition parameters, such as the actual batch processing time interval, single batch size, actual number of threads, and number of messages processed by each thread, based on the hardware resource operating status and message queue indicators. It then acquires the transmitted raw time-series data in batches according to these parameters and parses the raw time-series data into a first tree structure model according to a hierarchical structure. The first tree structure model is compared with a preset tree structure model, and a target tree structure model matching the first tree structure model is determined within the preset tree structure model. The data in the first tree structure model undergoes range transformation according to the range transformation formula in the target tree structure model to obtain a second tree structure model. Finally, the first tree model and the second tree model are synchronously stored in a time-series database, solving the problem of separate execution of batch storage and range transformation in existing technologies. Specifically, this technology addresses the high data write latency caused by the step-by-step execution of MQTT data batch processing and range transformation in existing technologies; the high CPU and memory resource utilization caused by step-by-step processing in high-concurrency scenarios; the insufficient compatibility and flexibility caused by the high coupling between the range transformation formula and the database interface in existing technologies; and the low write throughput caused by the long step-by-step process chain.

[0059] By employing MQTT data reception, efficient time-series data storage and synchronous execution of range transformation, and flexible real-time range transformation, multi-threaded batch data acquisition can be adaptively adopted. Combined with a synchronous scheduling mechanism, batch storage and range transformation are executed synchronously, reducing data write latency and minimizing data traversal and intermediate caching. Simultaneously, a modular design is used to encapsulate the range transformation formula library, enabling fine-grained rule settings and dynamic rule updates. Ultimately, this achieves the goals of reducing data latency, improving consistency, optimizing resource consumption, and increasing throughput.

[0060] Figure 5 This is a flowchart of another method for synchronously executing data batch processing and range conversion according to Embodiment 2 of the present invention. Figure 5 As shown, the processing flow of the method for synchronously executing batch data processing and range transformation can be as follows: Subscribe to the connector via the MQTT protocol to store the raw time series data in a message buffer queue; determine the batch data acquisition parameters based on the hardware resource operating status and message queue indicators; execute batch raw time series data acquisition according to the batch data acquisition parameters; perform structural parsing on the raw time series data to obtain a first tree structure model, and obtain a preset tree structure model, determining a target tree structure model that matches the first tree structure model; perform range transformation on the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain a second tree structure model; and synchronously store the first tree structure model and the second tree structure model in the time series database.

[0061] By performing concurrent MQTT data writing at 120,000 test points per second in the same hardware environment, such as a 24-core CPU and 32GB of memory, it can be confirmed that the write latency is significantly reduced, the data consistency is greatly improved, the resource consumption is significantly optimized, the write throughput is increased by more than 3 times, and the compatibility and flexibility are extremely strong.

[0062] Specifically, parameter calculations in batch processing ensure reasonable dynamic parameters, synchronous processing reduces process redundancy, and the dual storage logic of database storage and local caching for range transformation configuration reduces configuration query time. In experiments, batch processing and range transformation are executed synchronously, reducing one data traversal and configuration database lookup, and the end-to-end write latency is reduced from approximately 1000ms in existing technologies to less than 150ms. Synchronous execution of batch storage and range transformation eliminates the time difference in data sources for step-by-step processing. Batch processing and range transformation of the same data block are executed synchronously based on the same data source, improving data accuracy from 95% in existing technologies to 99.9%, without issues such as timestamp errors or physical quantity calculation errors. Hardware parameter calculations ensure reasonable dynamic parameter settings based on hardware status and message indicators, preventing excessive consumption of computing resources; even with extremely large data volumes, it will not cause system crashes; furthermore, synchronous processing reduces CPU context switching and repeated data traversal. The integrated workflow of synchronous processing and batch writing shortens the processing chain, increasing the number of batches that can be processed per unit time from approximately 100 batches / second in the existing technology to over 300 batches / second, supporting high-concurrency data writing of over 120,000 records / second. The integrated workflow of synchronous processing and batch writing also shortens the processing chain. The range transformation formula can be dynamically updated in real time through the front-end interface (without restarting the service). The modular design decouples the range transformation formula from the time-series database interface, achieving flexible adaptation and reducing costs by 90% to accommodate different sensor types.

[0063] Example 3 Figure 6 This is a schematic diagram of a device for synchronously executing data batch processing and range switching according to Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a data structure parsing module 610, a tree structure matching module 620, a range conversion module 630, and a synchronization storage module 640. Wherein: The data structure parsing module 610 is used to acquire the transmitted raw time series data in batches and parse the raw time series data into a first tree structure model according to the hierarchical structure. The tree structure matching module 620 is used to compare the first tree structure model with the preset tree structure model and determine the target tree structure model that matches the first tree structure model in the preset tree structure model. The range transformation module 630 is used to transform the data in the first tree structure model according to the range transformation formula in the target tree structure model to obtain the second tree structure model. The synchronous storage module 640 is used to synchronously store the first tree structure model and the second tree structure model in the time-series database.

[0064] Optionally, the device may also include: The preset tree structure model display module is used to construct a preset tree structure model according to the hierarchical structure of each time series data in the time series database before comparing the first tree structure model with the preset tree structure model, and to display the preset tree structure model on the range transformation interface. The range transformation formula display module is used to respond to the user's configuration operation of the preset tree structure model in the range transformation interface, generate the range transformation formula, and display the range transformation formula in the preset tree structure model.

[0065] Optionally, the data structure parsing module 610 includes: The data structure parsing unit is used to parse the original time series data into a first tree structure model according to the hierarchical structure of connector-connector instance-connector instance measurement point.

[0066] Optionally, the data structure parsing module 610 includes: The resource load comprehensive value determination unit is used to obtain the hardware resource operating status during batch data transmission and determine the resource load comprehensive value based on the hardware resource operating status. The parameter acquisition unit is used to acquire the real-time frequency of data reception, the reference frequency of data reception, and the reference time interval of batch data transmission. The batch processing actual time interval determination unit is used to determine the batch processing actual time interval based on the comprehensive resource load value, the real-time data reception frequency, the data reception reference frequency, and the batch processing reference time interval. The batch data acquisition unit is used to acquire the transmitted raw time series data in batches according to the actual time interval of batch processing.

[0067] Optional, batch data acquisition unit, including: The parameter acquisition subunit is used to acquire the real-time backlog of the buffer queue, the baseline backlog of the buffer queue, the baseline value of a single batch, and the size range of a single batch during batch data transmission. The single batch size value determination subunit is used to determine the single batch size value based on the comprehensive resource load value, real-time data reception frequency, data reception reference frequency, real-time backlog of buffer queue, reference backlog of buffer queue, single batch reference value, and single batch size range. The batch data acquisition subunit is used to acquire the original time-series data transmitted in each batch according to the single batch size value based on the actual time interval of batch processing.

[0068] Optional, batch data acquisition subunit, including: The parameter acquisition sub-unit is used to obtain the baseline number of threads and the range of thread counts in batch data transmission. The sub-unit for determining the actual number of threads is used to determine the actual number of threads based on the baseline number of threads, the range of the number of threads, the comprehensive value of resource load, the baseline value of a single batch, and the value of a single batch size. The batch data acquisition sub-unit is used to acquire the original time series data in each batch according to the single batch size value and the actual number of parallel processing threads, based on the actual time interval of batch processing.

[0069] Optional, batch data acquisition sub-unit, specifically used for: The number of messages processed by each thread is determined based on the single batch size value and the actual number of threads. Based on the actual time interval of batch processing, the actual number of parallel processing threads used in each batch, according to the corresponding allocated number of messages, obtain the original time series data to be transmitted.

[0070] The device for synchronous execution of data batch processing and range transformation provided in the embodiments of the present invention can execute the method for synchronous execution of data batch processing and range transformation provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0071] Example 4 Figure 7 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0072] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) or random access memory (RAM), communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. Input / output (I / O) interfaces are also connected to the bus 14.

[0073] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for simultaneous execution of data batch processing and range switching.

[0075] In some embodiments, the method for synchronously executing data batch processing and range switching can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for synchronously executing data batch processing and range switching described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the method for synchronously executing data batch processing and range switching by any other suitable means (e.g., by means of firmware).

[0076] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0080] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0081] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0082] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0083] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for synchronously executing data batch processing and range switching, characterized in that, include: Batch acquire the transmitted raw time series data, and parse the raw time series data into a first tree structure model according to the hierarchical structure; The first tree structure model is compared with the preset tree structure model, and a target tree structure model that matches the first tree structure model is determined in the preset tree structure model; The range transformation of the data in the first tree structure model is performed according to the range transformation formula in the target tree structure model to obtain the second tree structure model. The first tree structure model and the second tree structure model are stored synchronously in the time-series database.

2. The method according to claim 1, characterized in that, Before comparing the first tree structure model with the preset tree structure model, the process also includes: A preset tree structure model is constructed by constructing each time series data in the time series database according to the hierarchical structure, and the preset tree structure model is displayed on the range transformation interface; In response to the user's configuration operation of the preset tree structure model in the range transformation interface, a range transformation formula is generated and displayed in the preset tree structure model.

3. The method according to claim 1, characterized in that, The original time series data is parsed into a first tree structure model according to the hierarchical structure, including: The original time series data is parsed into a first tree structure model according to the hierarchical structure of connector-connector instance-connector instance measurement point.

4. The method according to claim 1, characterized in that, Batch acquisition of transmitted raw time series data, including: Obtain the hardware resource operating status during batch data transmission, and determine the comprehensive resource load value based on the hardware resource operating status; Obtain the real-time data reception frequency, data reception reference frequency, and batch processing reference time interval during batch data transmission; The actual batch processing time interval is determined based on the comprehensive resource load value, the real-time data reception frequency, the data reception reference frequency, and the batch processing reference time interval. Based on the actual time interval of the batch processing, the original time series data to be transmitted is acquired in batches.

5. The method according to claim 4, characterized in that, Based on the actual time interval of the batch processing, the original time-series data to be transmitted is acquired in batches, including: Obtain the real-time backlog of the buffer queue, the baseline backlog of the buffer queue, the baseline value of a single batch, and the size range of a single batch during batch data transfer; The single batch size value is determined based on the overall resource load value, the real-time data reception frequency, the data reception reference frequency, the real-time backlog of the buffer queue, the reference backlog of the buffer queue, the single batch reference value, and the single batch size range. Based on the actual time interval of the batch processing, the original time-series data transmitted in each batch is obtained according to the single batch size value.

6. The method according to claim 5, characterized in that, Based on the actual time interval of the batch processing, the original time-series data transmitted in each batch is obtained according to the single batch size value, including: Obtain the baseline number of threads and the range of threads in batch data transfer; The actual number of threads is determined based on the baseline number of threads, the range of thread numbers, the comprehensive resource load value, the baseline value for a single batch, and the single batch size value. Based on the actual time interval of the batch processing, the original time series data is obtained in each batch according to the single batch size value and using the actual number of parallel processing threads.

7. The method according to claim 6, characterized in that, Based on the actual time interval of the batch processing, the original time-series data to be transmitted is obtained in each batch according to the single batch size value and using the actual number of parallel processing threads, including: The number of messages processed by each thread is determined based on the single batch size value and the actual number of threads. Based on the actual time interval of the batch processing, the actual number of parallel processing threads used in each batch acquire the original time-series data to be transmitted according to the corresponding allocated number of messages.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method of synchronously executing data batch processing and range conversion as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for synchronously executing data batch processing and range switching as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method for synchronously executing data batch processing and range conversion according to any one of claims 1-7.

Images