Data processing method and device, equipment, storage medium and program product

Abstract

Embodiments of the present application provide a kind of data processing method, device, equipment, storage medium and program product, wherein the method comprises: obtaining the equipment information reported by multiple metering instruments according to predefined time interval, and each equipment information is transformed, and the structured information flow corresponding to each metering instrument is obtained;According to the structured information flow corresponding to each metering instrument, perform time sequence verification operation;According to time sequence verification result and each structured information flow, generate time sequence data block, and send the time sequence data block to database, to make the database store the time sequence data in the time sequence data block;Wherein, the time sequence data block contains the ordered data set of the structured information with time sequence of each metering instrument.The embodiments of the present application can reduce the high cost caused by using high-end hardware materials, and improve data storage efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Intelligent metering instruments have the ability to communicate with remote business systems. They can report some data collected or generated by the metering instruments to the business systems, which will store the data reported by the metering instruments for settlement, statistics, etc.

[0003] In metering instrument communication scenarios, high-concurrency time-series data storage is generated. Business systems (or servers) often need to frequently perform I / O (i.e., storage and retrieval) operations with the database. Due to frequent interactions, some data storage may become invalid, which greatly affects data storage efficiency. Furthermore, under limited hardware resources, low-end hardware materials cannot meet the requirements for storing and retrieving high-concurrency time-series data; only high-end hardware materials can meet the requirements, but at a very high cost.

[0004] Therefore, under limited hardware resources, existing technologies cannot meet the storage and retrieval requirements of high-concurrency time-series data using low-end hardware materials, and thus cannot improve data storage efficiency while reducing the high costs associated with using high-end hardware materials. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, storage medium, and program product to overcome the problem that, under limited hardware resources, the use of low-end hardware materials cannot meet the storage and retrieval requirements of high-concurrency time-series data, thus failing to reduce the high costs associated with using high-end hardware materials while improving data storage efficiency.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] According to a predefined time interval, the equipment information reported by multiple metering instruments is obtained, and the equipment information is transformed to obtain the structured information flow corresponding to each metering instrument.

[0008] Perform timing verification operations based on the structured information flow corresponding to each metering instrument;

[0009] Based on the timing verification results and each of the structured information streams, a timing data block is generated and sent to the database so that the database can store the timing data in the timing data block; wherein, the timing data block contains an ordered dataset of the timing structured information of each of the metering instruments.

[0010] In one possible design, the device information includes at least one of the following: device power, device signal, device reading, and device status; the device information is a binary information stream.

[0011] The process of converting the information of each device to obtain the structured information stream corresponding to each metering instrument includes:

[0012] For each piece of device information, according to a predefined protocol, the information in the device information is transformed to obtain the feature description information and corresponding values ​​of each piece of information;

[0013] A structured information stream is generated based on the feature descriptions and corresponding values ​​of each item in the information of each device.

[0014] In one possible design, the step of performing timing verification based on the structured information flow corresponding to each metering instrument includes:

[0015] Based on the structured information flow corresponding to each metering instrument, determine whether to perform a timing verification operation;

[0016] If it is determined that a timing verification operation will be performed, then the timing verification will be performed on the structured information stream corresponding to the same metering instrument.

[0017] In one possible design, determining whether to perform a timing verification operation based on the structured information flow corresponding to each metering instrument includes:

[0018] If the number of structured information streams corresponding to the equipment information reported by multiple metering instruments reaches a predetermined value within a predefined time period, then a timing verification operation is determined to be performed, and the timing is restarted; and / or,

[0019] If the duration corresponding to the equipment information reported by multiple metering instruments reaches the predefined duration, then the timing verification operation will be performed and the timing will be restarted.

[0020] If the number of structured information streams corresponding to the equipment information reported by multiple metering instruments does not reach the predetermined value within the predefined time period, and if the time period corresponding to the equipment information reported by multiple metering instruments reaches the predefined time period, then it is determined to perform a timing verification operation.

[0021] In one possible design, the timing verification of the structured information stream corresponding to the same metering instrument includes:

[0022] For the structured information stream corresponding to the same metering instrument, if there are multiple structured information streams, and if the reporting time and reporting order of each device information corresponding to each structured information stream match, then it is determined that the sorting of each structured information stream is correct.

[0023] For each of the structured information streams, or if there is only one structured information stream, perform the following steps for that structured information stream:

[0024] The structured information stream is time-series verified based on the instruction status time and / or the communication time corresponding to the recorded device information.

[0025] The instruction status time includes the instruction pending transmission status generation time and the instruction success status generation time. The pending transmission status generation time indicates the time when the instruction is in the pending transmission status when it is sent to the metering instrument. The success status generation time indicates the time when the instruction response is received. The instruction is used to instruct the metering instrument to report device information. The communication time includes the first communication time when the metering instrument triggers the device information and the second communication time when the metering instrument executes the instruction response.

[0026] In one possible design, the timing verification of the structured information stream based on the instruction status time and / or the communication time corresponding to the recorded device information includes:

[0027] If the generation time of the pending state corresponding to the structured information stream is earlier than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream, then the timing is determined to be correct.

[0028] If the generation time of the pending state corresponding to the structured information stream is later than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is later than the first communication time corresponding to the structured information stream, then a timing error is determined.

[0029] In one possible design, generating time-series data blocks based on the timing verification results and each of the structured information streams includes:

[0030] If the timing verification result is that the timing is correct, then the data is packaged according to the timestamp order corresponding to each of the structured information streams to generate a timing data block;

[0031] If the timing verification result is a timing error, the success state generation time and / or the first communication time are time-corrected to determine that the generation time of the pending state corresponding to the structured information stream is earlier than the generation time of the success state corresponding to the structured information stream, and / or that the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream. Data is then packaged according to the corrected timestamp order corresponding to each structured information stream to generate a timing data block.

[0032] Secondly, embodiments of this application provide a data processing apparatus, including:

[0033] The first processing module is used to acquire equipment information reported by multiple metering instruments according to a predefined time interval, and to transform each piece of equipment information to obtain a structured information stream corresponding to each metering instrument.

[0034] The second processing module is used to perform timing verification operations based on the structured information flow corresponding to each metering instrument.

[0035] The third processing module is used to generate a time-series data block based on the time-series verification results and each of the structured information streams, and send the time-series data block to the database so that the database stores the time-series data in the time-series data block; wherein, the time-series data block contains an ordered dataset of structured information with time-series characteristics of each of the metering instruments.

[0036] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory;

[0037] The memory stores computer-executed instructions;

[0038] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any of the first aspects.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the first aspects.

[0040] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0041] The data processing method, apparatus, device, storage medium, and program product provided in this embodiment first acquire equipment information reported by multiple metering instruments at predefined time intervals, and transform each piece of equipment information to obtain a structured information stream corresponding to each metering instrument. Then, based on the structured information streams corresponding to each metering instrument, a timing verification operation is performed. Next, based on the timing verification results and each of the structured information streams, a timing data block is generated, and the timing data block is sent to a database so that the database stores the timing data in the timing data block. The timing data block contains an ordered dataset of structured information with timing characteristics from each metering instrument. By transforming the acquired equipment information to obtain a structured information stream, in order to process metering instrument reported information with high concurrency under limited hardware resources, each structured information stream can be packaged. During the packaging process, timing verification is performed on the structured information stream to ensure accurate and efficient storage of timing data from the same device. Simultaneously, the packaged timing data block is sent to the database so that the database stores the timing data of the timing data block, reducing the frequency of timing data storage I / O operations and thus improving performance. Therefore, under limited hardware resources, using low-end hardware materials to meet the storage and retrieval of high-concurrency time-series data reduces the high costs associated with using high-end hardware materials while improving data storage efficiency. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic diagram of a scenario for the data processing method provided in the embodiments of this application;

[0044] Figure 2 Signaling interaction diagram of the data processing method provided in the embodiments of this application;

[0045] Figure 3 A schematic diagram of the business system provided in the embodiments of this application;

[0046] Figure 4 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0048] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0051] Currently, there are two ways to implement communication between metering instruments, such as in a leak alarm scheme based on a smart home metering instrument. One is to connect a gas leak detector to the metering instrument. When the metering instrument receives a leak alarm signal from the detector, it immediately and automatically shuts off the built-in valve and sends an alarm to the remote system. The other is to integrate abnormally small flow rates and continuous constant flow rate detection functions into the smart metering instrument. When such anomalies are detected, the metering instrument determines that a gas leak may exist and immediately sends an alarm. The communication in the leak alarm state is merely an example and is not specifically limited here.

[0052] However, during communication with metering instruments, high-concurrency time-series data storage scenarios arise. Business systems (or servers) often need to frequently perform I / O (i.e., storage and retrieval) operations with the database. Due to frequent interactions, some stored data may become invalid, significantly impacting data storage efficiency. Furthermore, under limited hardware resources, low-end hardware cannot meet the requirements for storing and retrieving high-concurrency time-series data; only high-end hardware can, but at a high cost. Therefore, current technologies, under limited hardware resources, cannot meet the requirements for storing and retrieving high-concurrency time-series data using low-end hardware, thus failing to improve data storage efficiency while reducing the high costs associated with using high-end hardware.

[0053] Therefore, to address the aforementioned issues, the technical concept of this application involves transforming the acquired device information into a structured information stream, and then performing time-series verification on the structured information stream to ensure accurate and efficient storage of time-series data from the same device. Simultaneously, the packaged time-series data blocks are sent to the database, enabling the database to store the time-series data within these blocks. This reduces the frequency of I / O operations for time-series data storage. By reducing the frequency of I / O operations between the server (or cloud) and the database, high-efficiency real-time storage of high-concurrency time-series data is achieved, ensuring high reliability and high concurrency of the overall system under limited hardware resources. Furthermore, by employing a time-series storage strategy when packaging data from the same device into data blocks, accurate and efficient storage of time-series data from the same device is guaranteed.

[0054] In practical applications, see Figure 1 As shown, Figure 1 This is a schematic diagram illustrating a scenario of the data processing method provided in this application embodiment. Multiple metering instruments (here referring to smart metering instruments capable of communicating with a remote business system, such as water meters, gas meters, oil meters, etc., with gas meters being used as an example below) are connected to the business system. This business system is deployed in the cloud or on a server and communicates with a large number of registered smart gas meters. This business system can remotely control the smart gas meters, such as for billing, settlement, and control of gas meter information.

[0055] For example, in combination Figure 2 As shown, Figure 2This is a signaling interaction diagram of the data processing method provided in this application embodiment. At a fixed time each day, a metering instrument (e.g., a gas meter) establishes a communication connection with the business system, the session channel is opened, and the metering instrument reports the first frame of information (here referring to the metering instrument's device information; the device here refers to a smart metering instrument) to the business system. This device information may include device power, device signal, device reading (e.g., gas volume), device status (whether the valve is abnormal, whether communication is abnormal, etc.), etc., where the device information is a binary information stream. After receiving a large amount of device information reported by the metering instruments, the business system converts the binary information stream into a structured information stream and performs time-series verification on the large amount of structured information stream. Based on the time-series verification results and the large amount of structured information stream, a time-series data block is generated and sent to a database (e.g., a time-series database), where the time-series database stores the time-series data.

[0056] Specifically, in combination Figure 3 As shown, Figure 3 This is a schematic diagram of a business system provided in an embodiment of this application. The business system includes an editing code module and a high-concurrency data module; specifically, at the same time, the business system receives a large amount of equipment information reported by gas meters, and needs to convert the binary information stream into a general structured information stream through the editing code module configured in the business system.

[0057] The transformed structured information stream is sent to the high-concurrency data processing module configured in the business system via a message middleware. This module smooths out peaks and valleys in the message stream (the structured information stream) to improve the system's concurrency and reliability. Furthermore, the high-concurrency data module uses a data packaging strategy to package the data into data blocks or time-series data blocks (i.e., ordered datasets containing time-series data from several devices). Each data block involves a single database I / O operation, such as a time-series database (i.e., a distributed time-series database that efficiently stores time-series data and has horizontal scalability), thereby improving the service's concurrent time-series data storage and processing capabilities.

[0058] Therefore, under limited hardware resources, using low-end hardware materials to meet the storage and retrieval of high-concurrency time-series data reduces the high costs associated with using high-end hardware materials while improving data storage efficiency.

[0059] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0060] Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method may include:

[0061] S101. Obtain equipment information reported by multiple metering instruments according to a predefined time interval, and transform each piece of equipment information to obtain a structured information stream corresponding to each metering instrument.

[0062] In this embodiment, the executing entity can be a data processing device, such as a business system, or the data processing device can be configured within the business system, which is deployed in the cloud or on a server. The following detailed explanation uses a data processing device deployed on a server and a gas meter as an example to illustrate the data processing method.

[0063] Specifically, when a gas meter actively reports equipment information, it periodically establishes a communication connection with the server, opens a session channel, and then reports its own equipment information to the server. The server can receive a large number of equipment information reports from gas meters simultaneously. Since the equipment information is binary, to convert technical variables into easily understandable fields, making it easier for users or maintenance personnel to comprehend the content, the equipment information can be transformed into a structured information stream. Here, "structured" means uniformly expressing a characteristic and describing attributes for the same type of feature. For example, a feature like signal quantity can be uniformly parsed as excellent / poor, meaning its corresponding values ​​are categorized into different levels such as excellent / poor.

[0064] S102. Perform timing verification operations based on the structured information flow corresponding to each metering instrument.

[0065] In this embodiment, in order to reduce the frequency of time-series data storage I / O operations, the structured information streams corresponding to a large number of gas meters can be packaged and processed. Each data block calls the time-series database storage I / O operation once, thereby improving the service's concurrent time-series data storage processing capability.

[0066] Specifically, firstly, for the structured information streams corresponding to a large number of gas meters, it is determined whether they meet the packaging strategy or packaging conditions. If they do, the large number of structured information streams are packaged. Since the data of the same device needs to be packaged into data blocks according to the time-series storage strategy to ensure the accurate and efficient storage of time-series data of the same device (here referring to structured information streams with time sequence), the time sequence of the structured information streams of the same device is verified during the packaging process.

[0067] S103. Based on the timing verification results and each of the structured information streams, generate a timing data block and send the timing data block to the database so that the database stores the timing data in the timing data block.

[0068] The time-series data block contains an ordered dataset of structured information with time sequence for each of the metering instruments.

[0069] In this embodiment, if the timing verification result is that the timing is correct, the structured information flow with timing is directly packaged to generate a timing data block; if the timing verification result is that the timing is incorrect, the structured information with timing errors is corrected, and then packaged to generate a timing data block.

[0070] The data processing method provided in this embodiment first acquires equipment information reported by multiple metering instruments at predefined time intervals, and transforms each piece of equipment information to obtain a structured information stream corresponding to each metering instrument. Then, based on the structured information streams corresponding to each metering instrument, a timing verification operation is performed. Next, based on the timing verification results and each of the structured information streams, a timing data block is generated, and the timing data block is sent to a database so that the database stores the timing data in the timing data block. The timing data block contains an ordered dataset of structured information with timing characteristics from each metering instrument. By transforming the acquired equipment information to obtain a structured information stream, in order to process metering instrument reported information with high concurrency under limited hardware resources, each structured information stream can be packaged. During the packaging process, timing verification is performed on the structured information stream to ensure accurate and efficient storage of timing data from the same device. Simultaneously, the packaged timing data block is sent to the database so that the database stores the timing data of the timing data block, reducing the frequency of timing data storage I / O operations and thus improving performance. Therefore, under limited hardware resources, using low-end hardware materials to meet the storage and retrieval of high-concurrency time-series data reduces the high costs associated with using high-end hardware materials while improving data storage efficiency.

[0071] In one possible design, the device information includes at least one of the following: device power, device signal, device reading, and device status; the device information is a binary information stream; the conversion of each piece of device information to obtain the structured information stream corresponding to each metering instrument can be achieved through the following steps:

[0072] Step a1: For each piece of device information, according to a predefined protocol, transform each piece of information in the device information to obtain the feature description information and corresponding values ​​of each piece of information;

[0073] Step a2: Generate a structured information stream based on the feature descriptions and corresponding values ​​of each item in the device information.

[0074] In this embodiment, the predefined protocol can define specific binary positions corresponding to device information to represent specific characteristics, such as the first two bits representing the device signal and the next three adjacent bits representing the device power. Then, the binary value of the corresponding position is calculated, and a structured information stream is generated based on the characteristics corresponding to the position (here, it can refer to feature description information, such as describing the first two bits as a signal) and the corresponding binary value.

[0075] Each of the following—device power, device signal, device reading, and device status—corresponds to characteristic description information. Furthermore, the binary values ​​corresponding to these characteristic descriptions can be parsed into levels for the respective characteristics. For example, signal strength is categorized by value as excellent, good, average, poor, and extremely poor. Other...

[0076] In one possible design, the step of performing timing verification based on the structured information flow corresponding to each metering instrument includes:

[0077] Step b1: Based on the structured information flow corresponding to each metering instrument, determine whether to perform a timing verification operation;

[0078] Step b2: If it is determined that a timing verification operation will be performed, then the timing verification will be performed on the structured information flow corresponding to the same metering instrument.

[0079] In this embodiment, since the data of the same device needs to be packaged into data blocks according to the time-series storage strategy, so as to ensure the accurate and efficient storage of the time-series data of the same device (here referring to the structured information flow with time sequence), the time sequence of the structured information flow of the same device is verified during the packaging process.

[0080] Specifically, based on the device information obtained from the large amount of structured information flow that has been transformed, the packaging strategy or packaging conditions are determined by the duration of the acquisition process or the amount of data reached. If the conditions are met, it is determined that the timing verification operation can be performed.

[0081] In one possible design, determining whether to perform a timing verification operation based on the structured information flow corresponding to each metering instrument can be achieved through at least one of the following steps:

[0082] Step b11: If the number of structured information streams corresponding to the equipment information reported by multiple metering instruments reaches a predetermined value within a predefined time period, then determine to perform a timing verification operation and restart the timing.

[0083] Step b12: If the duration corresponding to the equipment information reported by multiple metering instruments reaches the predefined duration, then determine to perform the timing verification operation and reset the timing.

[0084] Step b13: If the number of structured information streams corresponding to the equipment information reported by multiple metering instruments does not reach the predetermined value within the predefined time period, and if the time period corresponding to the equipment information reported by multiple metering instruments reaches the predefined time period, then determine to perform the timing verification operation.

[0085] In this embodiment, the data packaging strategy can be implemented in any of the following ways:

[0086] Method 1: Within a certain time period (i.e., within a predefined duration), the concurrent device information data (referring to structured information flow) that first reaches a certain amount of data (i.e., a predetermined value, such as 1000 pieces of device information data) is packaged into data blocks and the timer is reset.

[0087] Method 2: Pack concurrent device information data within a certain time period (e.g., 1 second) into data blocks.

[0088] Method 3: Within a certain time period (e.g., 1 second), first determine whether the conditions of Method 1 are met before packaging. If the conditions of Method 1 are met within the time period (e.g., 1 second), then package the current concurrent device information data into data blocks, perform data storage I / O operations, and reset the time counter. If the conditions of Method 1 are not met within the time period (e.g., 1 second), then use the logic of Method 2 to perform data storage I / O operations and reset the time counter.

[0089] In one possible design, the timing verification of the structured information stream corresponding to the same metering instrument can be achieved through the following steps:

[0090] Step c1: For the structured information stream corresponding to the same metering instrument, if there are multiple structured information streams, and if the reporting time and reporting order of each device information corresponding to each structured information stream match, then it is determined that the sorting of each structured information stream is correct.

[0091] Step c2: For each of the structured information streams, or, if there is only one structured information stream, perform the following steps for that structured information stream:

[0092] The structured information stream is time-series verified based on the instruction status time and / or the communication time corresponding to the recorded device information.

[0093] The instruction status time includes the instruction pending transmission status generation time and the instruction success status generation time. The pending transmission status generation time indicates the time when the instruction is in the pending transmission status when it is sent to the metering instrument. The success status generation time indicates the time when the instruction response is received. The instruction is used to instruct the metering instrument to report device information. The communication time includes the first communication time when the metering instrument triggers the device information and the second communication time when the metering instrument executes the instruction response.

[0094] In this embodiment, when the structured information flow is packaged into data blocks, the timing of data from the same device (i.e., the smart gas meter) must be ensured to be correct. For example, if a device reports information three times consecutively within the data packaging period (the time within which the packaging conditions are met, i.e., within the predefined duration), then the data from that device must be correctly ordered according to the timing when the data is packaged.

[0095] Because time differences may exist between services, the timestamp information of the data may not be generated based on the same standard time. Therefore, it is necessary to control the accuracy of data timing. Thus, after confirming correct sorting, the timing of the structured information flow from the same device needs further verification. If a device reports information only once within the data packaging cycle, the timing of that structured information flow also needs to be verified.

[0096] Specifically, the data timing accuracy verification step is completed during the packaging process. For example, 1,000 data entries are accumulated first, and then the data is verified during the packaging process. After verifying that the timing is correct, the data is sent in the form of data blocks.

[0097] The process of further verifying the timing of the structured information flow of the same device after confirming that the sorting is correct, or the process of verifying the timing of the structured information flow if a device reports information once within the data packaging cycle, can be implemented in the following ways: Method 11: Based on the instruction status time corresponding to the recorded device information; Method 21: Based on the communication time corresponding to the recorded device information; Method 31: Based on the instruction status time corresponding to the recorded device information combined with the communication time corresponding to the recorded device information.

[0098] In one possible design, the timing verification of the structured information stream based on the instruction status time and / or the communication time corresponding to the recorded device information can be achieved through the following steps:

[0099] Step d1: If the generation time of the pending state corresponding to the structured information stream is earlier than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream, then the timing is determined to be correct.

[0100] Step d2: If the generation time of the pending state corresponding to the structured information stream is later than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is later than the first communication time corresponding to the structured information stream, then a timing error is determined.

[0101] In this embodiment, timing verification is achieved based on instruction status records and / or device communication records:

[0102] For method 11, instruction status recording—timing data. For example, if the same device triggers two instruction status records within the same time period, one in the pending state and one in the successful state, logically it must be ensured that the successful state instruction record is recorded after the pending state.

[0103] For method 21, device communication records—timing data. For example, when the same device triggers the first frame and command response communication information within the same time period, logically it must be ensured that the first frame communication information follows the command response communication information.

[0104] For method 31, refer to the combination of method 11 and method 21, and will not be elaborated here.

[0105] In one possible design, generating time-series data blocks based on the timing verification results and each of the structured information streams can be achieved through the following steps:

[0106] Step e1: If the timing verification result is that the timing is correct, then the data is packaged according to the timestamp order corresponding to each of the structured information streams to generate a timing data block;

[0107] Step e2: If the timing verification result is a timing error, then the success state generation time and / or the first communication time are time-corrected to determine that the generation time of the state to be sent corresponding to the structured information stream is earlier than the generation time of the success state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream. Data is then packaged according to the corrected timestamp order corresponding to each structured information stream to generate a timing data block.

[0108] In this embodiment, if 1000 data entries are accumulated first, verification is performed during the data packaging process. After verifying that the timing is correct, the data is sent in the form of data blocks. If the timing verification structure is incorrect, i.e., the timing verification result is a timing error, the logically earliest one is used as the benchmark, and subsequent ones are corrected at the millisecond level. For example, if the time for generating the pending transmission of an instruction (or the time for generating the pending transmission status of an instruction) is later than the end time (or the time for generating the success status of an instruction), this time is incorrect. By correcting the end time, it is corrected to the time for generating the pending transmission + 1 millisecond.

[0109] Therefore, this application solves the problem of high-concurrency processing of metering instrument reporting information in a business system under limited hardware resources through a series of operations including data transformation, packaging condition judgment, timing verification, and packaging. It utilizes a data packaging strategy to reduce the frequency of I / O operations for time-series data storage, thereby improving system performance and effectively reducing the I / O load pressure on the time-series database. Distributed storage of metering instrument time-series data facilitates the management of time-series information for large-scale equipment. Simultaneously, the data packaging strategy ensures efficient storage of time-series data.

[0110] To implement the aforementioned data processing method, this embodiment provides a data processing apparatus. See also... Figure 5 , Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; the data processing device includes: a first processing module 501, a second processing module 502, and a third processing module 503.

[0111] The first processing module 501 is used to acquire equipment information reported by multiple metering instruments according to a predefined time interval, and to convert each piece of equipment information to obtain a structured information stream corresponding to each metering instrument.

[0112] The second processing module 502 is used to perform timing verification operations based on the structured information flow corresponding to each metering instrument.

[0113] The third processing module 503 is used to generate a time-series data block based on the time-series verification result and each of the structured information streams, and send the time-series data block to the database so that the database stores the time-series data in the time-series data block; wherein, the time-series data block contains an ordered dataset of structured information with time-series characteristics of each of the metering instruments.

[0114] In this embodiment, the first processing module 501, the second processing module 502, and the third processing module 503 are used to acquire equipment information reported by multiple metering instruments according to a predefined time interval, and to transform each piece of equipment information to obtain a structured information stream corresponding to each metering instrument. Then, based on the structured information stream corresponding to each metering instrument, a timing verification operation is performed. Then, based on the timing verification result and each of the structured information streams, a timing data block is generated, and the timing data block is sent to the database so that the database stores the timing data in the timing data block. The timing data block contains an ordered dataset of structured information with timing characteristics from each metering instrument. By transforming the acquired device information into a structured information stream, and to process metering instrument reports with high concurrency under limited hardware resources, each structured information stream can be packaged. During the packaging process, the structured information streams undergo time-series verification to ensure accurate and efficient storage of time-series data from the same device. Simultaneously, the packaged time-series data blocks are sent to the database, enabling the database to store the time-series data, reducing the frequency of I / O operations for time-series data storage and thus improving performance. Therefore, under limited hardware resources, using low-end hardware materials satisfies the storage and retrieval of high-concurrency time-series data, thereby reducing the high costs associated with using high-end hardware materials while improving data storage efficiency.

[0115] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0116] In one possible design, the device information includes at least one of the following: device power, device signal, device reading, and device status; the device information is a binary information stream; the first processing module is specifically used for:

[0117] For each piece of device information, according to a predefined protocol, the information in the device information is transformed to obtain the feature description information and corresponding values ​​of each piece of information;

[0118] A structured information stream is generated based on the feature descriptions and corresponding values ​​of each item in the information of each device.

[0119] In one possible design, the second processing module includes: a first processing unit and a second processing unit;

[0120] The first processing unit is used to determine whether to perform a timing verification operation based on the structured information flow corresponding to each metering instrument.

[0121] The second processing unit is used to perform time-series verification on the structured information stream corresponding to the same metering instrument if it is determined that a time-series verification operation will be performed.

[0122] In one possible design, the second processing unit is specifically used for:

[0123] For the structured information stream corresponding to the same metering instrument, if there are multiple structured information streams, and if the reporting time and reporting order of each device information corresponding to each structured information stream match, then it is determined that the sorting of each structured information stream is correct.

[0124] For each of the structured information streams, or if there is only one structured information stream, perform the following steps for that structured information stream:

[0125] The structured information stream is time-series verified based on the instruction status time and / or the communication time corresponding to the recorded device information.

[0126] The instruction status time includes the instruction pending transmission status generation time and the instruction success status generation time. The pending transmission status generation time indicates the time when the instruction is in the pending transmission status when it is sent to the metering instrument. The success status generation time indicates the time when the instruction response is received. The instruction is used to instruct the metering instrument to report device information. The communication time includes the first communication time when the metering instrument triggers the device information and the second communication time when the metering instrument executes the instruction response.

[0127] In one possible design, the second processing unit is specifically used for:

[0128] If the generation time of the pending state corresponding to the structured information stream is earlier than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream, then the timing is determined to be correct.

[0129] If the generation time of the pending state corresponding to the structured information stream is later than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is later than the first communication time corresponding to the structured information stream, then a timing error is determined.

[0130] In one possible design, the third processing module is specifically used for:

[0131] When the timing verification result is that the timing is correct, the data is packaged according to the timestamp order corresponding to each of the structured information streams to generate a timing data block;

[0132] When the timing verification result is a timing error, the success state generation time and / or the first communication time are time-corrected to determine that the generation time of the pending state corresponding to the structured information stream is earlier than the generation time of the success state corresponding to the structured information stream, and / or that the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream. Data is then packaged according to the corrected timestamp order corresponding to each structured information stream to generate a timing data block.

[0133] In one possible design, the first processing unit is specifically used for:

[0134] When the number of structured information streams corresponding to the equipment information reported by multiple metering instruments reaches a predetermined value within a predefined time period, a timing verification operation is determined to be performed and the timing is restarted.

[0135] When the duration corresponding to the equipment information reported by multiple metering instruments reaches the predefined duration, a timing verification operation is performed and the timing is restarted.

[0136] If the number of structured information streams corresponding to the equipment information reported by multiple metering instruments does not reach the predetermined value within a predefined time period, and the time period corresponding to the equipment information reported by multiple metering instruments reaches the predefined time period, then the timing verification operation is determined to be executed.

[0137] To implement the methods of the above embodiments, this embodiment provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 60 of this embodiment includes a processor 601 and a memory 602; wherein, the memory 602 is used to store computer execution instructions; and the processor 601 is used to execute the computer execution instructions stored in the memory to implement the various steps performed in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0138] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0139] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms. Additionally, the functional modules in the various embodiments of this application may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned modular units can be implemented in hardware or in the form of hardware plus software functional units.

[0141] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application. It should be understood that the processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0142] The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk drive, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses in the accompanying drawings are not limited to a single bus or a single type of bus. The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk, or optical disc. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0143] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0144] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: According to a predefined time interval, the equipment information reported by multiple metering instruments is obtained, and the equipment information is transformed to obtain the structured information flow corresponding to each metering instrument. Based on the structured information flow corresponding to each metering instrument, determine whether to perform a timing verification operation; If it is determined that a timing verification operation will be performed, then the timing verification will be performed on the structured information stream corresponding to the same metering instrument. Based on the time-series verification results and each of the structured information streams, a time-series data block is generated and sent to the database so that the database can store the time-series data in the time-series data block; wherein, the time-series data block contains an ordered dataset of structured information with time-series characteristics of each of the metering instruments. The step of determining whether to perform a timing verification operation based on the structured information flow corresponding to each metering instrument includes: If the number of structured information streams corresponding to the equipment information reported by multiple metering instruments reaches a predetermined value within a predefined time period, then a timing verification operation will be performed and the timing will be restarted. If the duration corresponding to the equipment information reported by multiple metering instruments reaches the predefined duration, then the timing verification operation will be performed and the timing will be restarted. If the number of structured information streams corresponding to the equipment information reported by multiple metering instruments does not reach the predetermined value within the predefined time period, and if the time period corresponding to the equipment information reported by multiple metering instruments reaches the predefined time period, then it is determined to perform a timing verification operation.

2. The method according to claim 1, characterized in that, The device information includes at least one of the following: device power, device signal, device reading, and device status; the device information is a binary information stream. The process of converting the information of each device to obtain the structured information stream corresponding to each metering instrument includes: For each piece of device information, according to a predefined protocol, the information in the device information is transformed to obtain the feature description information and corresponding values ​​of each piece of information; A structured information stream is generated based on the feature descriptions and corresponding values ​​of each item in the information of each device.

3. The method according to claim 1, characterized in that, The timing verification of the structured information stream corresponding to the same metering instrument includes: For the structured information stream corresponding to the same metering instrument, if there are multiple structured information streams, and if the reporting time and reporting order of each device information corresponding to each structured information stream match, then it is determined that the sorting of each structured information stream is correct. For each of the structured information streams, or if there is only one structured information stream, perform the following steps for that structured information stream: The structured information stream is time-series verified based on the instruction status time and / or the communication time corresponding to the recorded device information. The instruction status time includes the instruction pending transmission status generation time and the instruction success status generation time. The pending transmission status generation time indicates the time when the instruction is in the pending transmission status when it is sent to the metering instrument. The success status generation time indicates the time when the instruction response is received. The instruction is used to instruct the metering instrument to report device information. The communication time includes the first communication time when the metering instrument triggers the device information and the second communication time when the metering instrument executes the instruction response.

4. The method according to claim 3, characterized in that, The step of performing time-series verification on the structured information stream based on the instruction status time and / or the communication time corresponding to the recorded device information includes: If the generation time of the pending state corresponding to the structured information stream is earlier than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream, then the timing is determined to be correct. If the generation time of the pending state corresponding to the structured information stream is later than the generation time of the successful state corresponding to the structured information stream, and / or the second communication time corresponding to the structured information stream is later than the first communication time corresponding to the structured information stream, then a timing error is determined.

5. The method according to claim 4, characterized in that, The step of generating time-series data blocks based on the time-series verification results and each of the structured information streams includes: If the timing verification result is that the timing is correct, then the data is packaged according to the timestamp order corresponding to each of the structured information streams to generate a timing data block; If the timing verification result is a timing error, the success state generation time and / or the first communication time are time-corrected to determine that the generation time of the pending state corresponding to the structured information stream is earlier than the generation time of the success state corresponding to the structured information stream, and / or that the second communication time corresponding to the structured information stream is earlier than the first communication time corresponding to the structured information stream. Data is then packaged according to the corrected timestamp order corresponding to each structured information stream to generate a timing data block.

6. A data processing apparatus, characterized in that, include: The first processing module is used to acquire equipment information reported by multiple metering instruments according to a predefined time interval, and to transform each piece of equipment information to obtain a structured information stream corresponding to each metering instrument. The second processing module is used to determine whether to perform a timing verification operation based on the structured information flow corresponding to each metering instrument. If it is determined that a timing verification operation will be performed, then the timing verification will be performed on the structured information stream corresponding to the same metering instrument. The step of determining whether to perform a timing verification operation based on the structured information streams corresponding to each metering instrument includes: if the number of structured information streams corresponding to the equipment information reported by multiple metering instruments reaches a predetermined value within a predefined time period, then it is determined to perform a timing verification operation and the timing is reset; if the time period corresponding to the equipment information reported by multiple metering instruments reaches a predefined time period, then it is determined to perform a timing verification operation and the timing is reset; if the number of structured information streams corresponding to the equipment information reported by multiple metering instruments does not reach a predetermined value within a predefined time period, and if the time period corresponding to the equipment information reported by multiple metering instruments reaches a predefined time period, then it is determined to perform a timing verification operation. The third processing module is used to generate a time-series data block based on the time-series verification results and each of the structured information streams, and send the time-series data block to the database so that the database stores the time-series data in the time-series data block; wherein, the time-series data block contains an ordered dataset of structured information with time-series characteristics of each of the metering instruments.

7. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1-5.

Images