Data processing method and device, electronic equipment and medium
By deduplication and fault completion of monitoring data in IOT devices, the problem of low data processing efficiency under high-frequency data acquisition is solved, the efficiency of data storage and resource optimization is achieved, and the load of edge computing gateways is reduced .
Patent Information
- Application Number
- CN202411258339.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2025-01-03
AI Technical Summary
In the high-frequency data acquisition scenario, the data processing efficiency of IOT devices in the prior art is low, resulting in servers with limited resources facing pressure and excessive CPU and disk load.
By obtaining the monitoring data of power equipment and storing it in the monitoring data table, deduplication and storage, obtaining offline interval data for fault completion, using the InfluxDB database to achieve efficient data storage and query, reducing data writing frequency, and improving data processing efficiency .
It realizes data deduplication and filtering, reduces data writing frequency, reduces the pressure on edge computing gateways, improves data processing efficiency and storage efficiency, optimizes resource usage, and reduces CPU and disk usage to below 8%.
Smart Images
Figure CN119248772A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of edge computing technology for the Internet of Things, and in particular to a data processing method, device, electronic device and medium. Background Art
[0002] InfluxDB is one of the most commonly used open source time series databases on the market. InfluxDB is an open source time series data developed by InfluxData, written in GO, focusing on high-performance query and storage of time series data. InfluxDB is widely used in monitoring data of storage systems, such as real-time data in operation and maintenance and IOT (Internet of Things) industries. The most stable version of InfluxDB is V2.0. InfluxDB uses a data model for data organization to organize time series data into buckets and measurements, that is, to manage compressed data by block and organization storage.
[0003] In the existing technology, when faced with high-frequency data collection work in industries such as industry and electricity, the millisecond and second collection frequencies often put pressure on program reading and writing, causing the device gateway to have CPU and disk overflow problems. Most IOT (Internet of Things) data cleansing are configured with regular cleanup strategies.
[0004] Although there are database management strategies in place, the high-frequency data collection and storage still puts pressure on servers with limited resources, resulting in low data processing efficiency. Summary of the invention
[0005] In view of this, it is necessary to provide a data processing method, device, electronic device and medium to solve the problem of low data processing efficiency in the prior art.
[0006] In order to solve the above problems, the present application provides a data processing method, including: Acquire monitoring data of the electric power equipment, and store the monitoring data in a monitoring data table, wherein the monitoring data includes a timestamp and a corresponding monitoring value, and the monitoring data table is used to store monitoring data different from monitoring values in adjacent times in the monitoring data table; Acquire offline interval data of the power equipment, and store the offline interval data in an offline data table; The query time is obtained, and based on the offline interval data within the query time in the offline data table, the monitoring data within the query time in the monitoring data table is fault-completed to obtain time series data.
[0007] In some possible implementations, the monitoring data is collected through the modbus TCP\RTU protocol.
[0008] In some possible implementations, the monitoring data is reported by direct acquisition or MQTT forwarding.
[0009] In some possible implementations, obtaining offline interval data of the power equipment includes: Acquire network on / off status data of the power equipment, wherein the network on / off status data includes a timestamp and a corresponding network on / off status value; Based on the network on / off status data and the monitoring data table, offline interval data of the power equipment is determined.
[0010] In some possible implementations, the network connection status data is obtained by checking through a timed ping / telnet method.
[0011] In some possible implementations, the time series data is displayed in the form of a line graph.
[0012] In some possible implementations, the line chart is drawn by the go-echarts tool.
[0013] The present application also provides a data processing device, comprising: A data acquisition unit, used to acquire monitoring data and offline interval data of the power equipment, wherein the monitoring data includes a timestamp and a corresponding monitoring value; A data storage unit, used to store the monitoring data in a monitoring data table, and store the offline interval data in an offline data table, wherein the monitoring data table is used to store monitoring data different from monitoring values in adjacent time periods in the monitoring data table; The data query unit is used to obtain the query time, and based on the offline interval data within the query time in the offline data table, perform fault completion on the monitoring data within the query time in the monitoring data table to obtain time series data.
[0014] The present application also provides an electronic device, including a memory and a processor; The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the data processing method described above.
[0015] The present application also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above-mentioned data processing method are implemented.
[0016] The beneficial effects of the present application are as follows: the data processing method provided by the present application, by discarding the monitoring data in the monitoring data that has the same monitoring value as the monitoring data in the adjacent time in the monitoring data table, and storing them differently, achieves data deduplication filtering, greatly reduces the frequency of data writing, improves data processing efficiency, and reduces the pressure on high-frequency data storage and edge computing gateways in the power and industrial industries; further, considering that there will be gaps in the monitoring data during the offline period of the power equipment, the faulty monitoring data can be supplemented by the offline data table, so that the status of the power equipment on the application side can be clearly understood. There is no need to worry about not being able to see the data at a certain point in time. At the same time, data storage is more efficient, and the data read is reduced, the reading efficiency is improved, and the data compression ratio is improved; finally, through the above method, in terms of resource utilization, the CPU and disk usage are greatly optimized. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A schematic diagram of an embodiment of the data processing method provided in this application; Figure 2 For this application Figure 1 A schematic diagram of an embodiment of a process flow for obtaining offline interval data of an electric power device; Figure 3 A schematic diagram of the structure of an embodiment of a data processing device provided by the present application; Figure 4 A schematic diagram of the structure of an embodiment of an electronic device provided in this application. DETAILED DESCRIPTION
[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0019] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts may be implemented out of order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowcharts, or may remove one or more operations from the flowcharts, under the guidance of the content of this application. Some of the block diagrams shown in the accompanying drawings are functional entities, and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor systems and / or microcontroller systems.
[0020] The descriptions of "first", "second", etc. involved in the embodiments of the present application are only used to describe the implicit purpose, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" and "second" may explicitly or implicitly include at least one of the features. "And / or" describes the association relationship of the associated objects, indicating that three relationships may exist, for example: A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone.
[0021] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0022] The present application provides a data processing method, which is described below.
[0023] Figure 1 A flow chart of an embodiment of the data processing method provided in this application is as follows: Figure 1 As shown, the data processing method includes: S101, acquiring monitoring data of the power equipment, and storing the monitoring data in a monitoring data table, wherein the monitoring data includes a timestamp and a corresponding monitoring value, and the monitoring data table is used to store monitoring data different from monitoring values in adjacent times in the monitoring data table; It should be noted that the power equipment on the application side is an electric meter or a battery management system BMS, etc., which uses the modbusTCP\RTU protocol to collect data (including configuration address, port, baud rate, data bit and other power equipment information, connection to read monitoring data, millisecond-level collection frequency), and uses direct collection or mqtt forwarding to report data.
[0024] It should also be noted that the monitoring data table stores monitoring data that is different from the monitoring values at adjacent times in the monitoring data table. The specific steps are: cache the timestamps of the monitoring data collected and reported from the power equipment, compare the monitoring values of the detection data with those of the monitoring data at adjacent times in the monitoring data table to see if they are consistent, if they are consistent, discard the monitoring data; if they are inconsistent, insert the monitoring data into the monitoring data table of the database for storage to complete data deduplication. The above data deduplication algorithm is written in Golang.
[0025] S102, obtaining offline interval data of the power equipment, and storing the offline interval data in an offline data table; It should be noted that the offline interval data refers to the last startup time and the latest startup time of the power equipment. This time interval is the period when the power equipment is offline, and there will be gaps in the monitoring data during this period.
[0026] It should also be noted that both the monitoring data table and the offline data table are created in the InfluxDB database.
[0027] S103, obtaining the query time, and based on the offline interval data in the query time in the offline data table, performing fault completion on the monitoring data in the query time in the monitoring data table to obtain time series data; It should be noted that fault completion specifically involves inserting the default value of the data type into the monitoring data based on the offline interval data, completing the fault combination, using the go-echarts tool to draw the time series data into a line chart, and displaying it through httpServer.
[0028] Compared with the prior art, the data processing method disclosed in the present application achieves data deduplication and filtering by discarding monitoring data with the same monitoring values at adjacent times in the monitoring data table and storing them differently, which greatly reduces the frequency of data writing, improves data processing efficiency, and reduces the pressure on high-frequency data storage and edge computing gateways in the power and industrial sectors; further, considering that there will be gaps in the monitoring data during the offline period of the power equipment, the broken monitoring data can be supplemented by the offline data table, so that the status of the power equipment on the application side can be clearly understood. There is no need to worry about not being able to see the data at a certain point in time. At the same time, data storage is more efficient, the data is reduced during reading, the reading efficiency is improved, and the data compression ratio is improved; finally, through the above method, in terms of resource utilization, the CPU and disk usage are greatly optimized, and the program CPU usage does not exceed 8%.
[0029] In order to better obtain the offline interval data of the power equipment, in some embodiments, such as Figure 2 As shown, the offline interval data of the power equipment is obtained in step S102, including: S201, obtaining network on / off status data of the power equipment, where the network on / off status data includes a timestamp and a corresponding network on / off status value; S202: Determine the offline interval data of the power equipment based on the network on / off status data and the monitoring data table.
[0030] It should be noted that the network connectivity status is checked by periodically pinging / telnetting the power equipment. If the ping succeeds, the power equipment network is considered to be connected and communicating normally. If the ping fails, the power equipment network is considered to be disconnected and unable to communicate.
[0031] It should also be noted that the network on / off status data is compared with the monitoring data table to obtain accurate offline interval data. The specific operation is: record the time points when ping is unsuccessful and when ping is successful, compare the time point when ping is unsuccessful with the time point in the monitoring data table, take the time point that is less than or equal to the time point, and then take the time point when ping is successful and compare it with the time point in the monitoring data table. Similarly, this process can obtain the accurate offline interval of the device and store it in the offline data table.
[0032] In order to better implement a data processing method in an embodiment of the present application, based on a data processing method, correspondingly, Figure 3 As shown, the embodiment of the present application further provides a data processing device 300, including: The data acquisition unit 301 is used to acquire monitoring data and offline interval data of the power equipment, where the monitoring data includes a timestamp and a corresponding monitoring value; The data storage unit 302 is used to store the monitoring data in a monitoring data table and store the offline interval data in an offline data table, wherein the monitoring data table is used to store monitoring data different from the monitoring values of adjacent time in the monitoring data table; The data query unit 303 is used to obtain the query time, and based on the offline interval data within the query time in the offline data table, perform fault completion on the monitoring data within the query time in the monitoring data table to obtain time series data.
[0033] The data processing device 300 provided in the above embodiment can implement the technical solution described in the above data processing method embodiment. The specific implementation principles of the above units can refer to the corresponding contents in the above data processing method embodiment, which will not be repeated here.
[0034] like Figure 4 As shown, the present application also provides an electronic device 400. The electronic device 400 includes a processor 401, a memory 402 and a display 404. Figure 4 Only some components of the electronic device 400 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0035] In some embodiments, the memory 402 may be an internal storage unit of the electronic device 400, such as a hard disk or memory of the electronic device 400. In other embodiments, the memory 402 may also be an external storage device of the electronic device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 400.
[0036] Furthermore, the memory 402 may include both an internal storage unit of the electronic device 400 and an external storage device. The memory 402 is used to store application software installed in the electronic device 400 and various data.
[0037] In some embodiments, the processor 401 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 402 or process data, such as the data processing method in the present application.
[0038] In some embodiments, the display 404 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 404 is used to display information on the electronic device 400 and to display a visual user interface. The components 401-404 of the electronic device 400 communicate with each other via a system bus.
[0039] In some embodiments of the present application, when the processor 401 executes the data processing program in the memory 402, the following steps may be implemented: Acquire monitoring data of the power equipment and store the monitoring data in a monitoring data table, wherein the monitoring data includes a timestamp and a corresponding monitoring value, and the monitoring data table is used to store monitoring data different from monitoring values in adjacent times in the monitoring data table; Obtain offline interval data of the power equipment, and store the offline interval data in an offline data table; The query time is obtained, and based on the offline interval data within the query time in the offline data table, the monitoring data within the query time in the monitoring data table is gap-filled to obtain time series data.
[0040] It should be understood that: when the processor 401 executes the data processing program in the memory 402, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.
[0041] Furthermore, the embodiment of the present application does not specifically limit the type of the electronic device 400 mentioned, and the electronic device 400 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable power device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present application, the electronic device 400 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).
[0042] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions of the data processing method provided in the above-mentioned method embodiments can be implemented.
[0043] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0044] The above is a detailed introduction to a data processing method provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
[0045] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.
Claims
1. A data processing method, characterized in that: include: Acquire monitoring data of the electric power equipment, and store the monitoring data in a monitoring data table, wherein the monitoring data includes a timestamp and a corresponding monitoring value, and the monitoring data table is used to store monitoring data different from monitoring values in adjacent times in the monitoring data table; Acquire offline interval data of the power equipment, and store the offline interval data in an offline data table; The query time is obtained, and based on the offline interval data within the query time in the offline data table, the monitoring data within the query time in the monitoring data table is fault-completed to obtain time series data.
2. The data processing method according to claim 1, characterized in that: The monitoring data is collected via the modbus TCP\RTU protocol.
3. The data processing method according to claim 1, characterized in that: The monitoring data is reported by direct acquisition or MQTT forwarding.
4. The data processing method according to claim 1, characterized in that: Obtain offline interval data of power equipment, including: Acquire network on / off status data of the power equipment, wherein the network on / off status data includes a timestamp and a corresponding network on / off status value; Based on the network on / off status data and the monitoring data table, offline interval data of the power equipment is determined.
5. The data processing method according to claim 4, characterized in that: The network connection and disconnection status data is obtained by checking through a regular ping / telnet method.
6. The data processing method according to claim 1, characterized in that: The time series data is displayed in the form of a line graph.
7. The data processing method according to claim 6, characterized in that: The line chart is drawn by the go-echarts tool.
8. A data processing device, characterized in that: include: A data acquisition unit, used to acquire monitoring data and offline interval data of the power equipment, wherein the monitoring data includes a timestamp and a corresponding monitoring value; A data storage unit, used to store the monitoring data in a monitoring data table, and store the offline interval data in an offline data table, wherein the monitoring data table is used to store monitoring data different from monitoring values in adjacent time periods in the monitoring data table; The data query unit is used to obtain the query time, and based on the offline interval data in the query time in the offline data table, perform fault completion on the monitoring data in the query time in the monitoring data table to obtain time series data.
9. An electronic device, characterized in that: including memory and processor; The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps of the data processing method described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the data processing method described in any one of claims 1 to 7 are implemented.