Electricity monitoring data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an electrical monitoring data processing method and device, electronic equipment and storage medium, which can be applied to the technical field of smart power utilization. The electrical monitoring data processing method comprises: in the case that a plurality of first electrical monitoring data are read from a message queue, calling a data reduction service to perform reduction processing on the plurality of first electrical monitoring data to obtain a plurality of second electrical monitoring data; calling a data packaging service to add the plurality of second electrical monitoring data as incremental data to a first data packet to obtain a second data packet; in the case that the data amount of the second data packet is greater than a first preset quantity threshold, calling a batch data warehousing service to batch write the data in the second data packet into a database.

Description

Technical Field

[0001] This disclosure relates to the field of smart electricity technology, specifically to an electrical monitoring data processing method, apparatus, equipment, medium, and program product. Background Technology

[0002] In IoT applications for preventing electrical fires, monitoring points are set up on electrical circuits within buildings, and edge-end electrical fire sensors are installed. These sensors monitor the voltage, current, and temperature data at these points in real time, and then transmit, store, and analyze this data to prevent electrical fires.

[0003] In the process of realizing the concept disclosed herein, it was found that due to the large number of monitoring points for electrical circuits at the edge, the amount of data collected is also very large. If this data is directly entered into the database in real time, the database cannot respond in time to insert the large number of data entries one by one during the data transmission and database entry process, resulting in a concurrency performance bottleneck. Summary of the Invention

[0004] In view of the above problems, this disclosure provides an electrical monitoring data processing method, apparatus, equipment, medium and program product.

[0005] One aspect of this disclosure provides an electrical monitoring data processing method, comprising:

[0006] When multiple first electrical monitoring data are read from the message queue, the data reduction service is called to reduce the amount of the multiple first electrical monitoring data to obtain multiple second electrical monitoring data.

[0007] The data packaging service is invoked to add multiple second electrical monitoring data as incremental data to the first data packet to obtain the second data packet;

[0008] If the amount of data in the second data packet exceeds the first preset quantity threshold, the batch data import service is invoked to write the data in the second data packet into the database in batches.

[0009] According to embodiments of this disclosure, obtaining multiple second electrical monitoring data by reducing multiple first electrical monitoring data includes:

[0010] After grouping multiple first electrical monitoring data, at least one array is obtained, wherein multiple first electrical monitoring data in the same array belong to the same electrical data type;

[0011] Multiple first electrical monitoring data in the array are dehomogenized to obtain multiple second electrical monitoring data.

[0012] According to embodiments of this disclosure, obtaining multiple second electrical monitoring data after dehomogenizing multiple first electrical monitoring data in an array includes:

[0013] Calculate the average value of multiple first electrical monitoring data in the array to obtain the target mean value;

[0014] Calculate the difference between each of the first electrical monitoring data points in the array and the target mean.

[0015] Based on the difference between multiple first electrical monitoring data and the target mean, target homogeneous data among the multiple first electrical monitoring data are determined, wherein the target homogeneous data is: among the multiple first electrical monitoring data, the first electrical monitoring data whose difference from the target mean is less than a preset difference threshold;

[0016] Multiple sets of second electrical monitoring data are obtained by deleting homogeneous data from multiple sets of first electrical monitoring data.

[0017] According to embodiments of this disclosure, the process of grouping multiple first electrical monitoring data to obtain at least one array includes:

[0018] Based on the different types of electrical data, multiple first electrical monitoring data are grouped and processed to obtain voltage arrays, current arrays, and temperature arrays.

[0019] According to embodiments of this disclosure, invoking the batch data import service to write data from the second data packet into the database in batches includes:

[0020] Assemble data of the same data type from the second data packet into the same target database operation statement;

[0021] Perform batch data write operations using the target database operation statements.

[0022] According to embodiments of this disclosure, the above method further includes:

[0023] If the first electrical monitoring data is not read from the message queue, the batch data entry service is invoked to write the data in the first data packet into the database in batches.

[0024] According to embodiments of this disclosure, reading multiple first electrical monitoring data from a message queue includes:

[0025] If the amount of data in the message queue is greater than or equal to the second preset quantity threshold, a preset quantity of data is read from the message queue as multiple first electrical monitoring data, wherein the preset quantity is equal to the second preset quantity threshold.

[0026] If the amount of data in the message queue is less than the second preset threshold, all data is read from the message queue as multiple first electrical monitoring data.

[0027] Another aspect of this disclosure provides an electrical monitoring data processing device, including a reduction module, a packaging module, and a first batch warehousing module.

[0028] Among them, the reduction module is used to call the data reduction service to reduce the multiple first electrical monitoring data after reading multiple first electrical monitoring data from the message queue, so as to obtain multiple second electrical monitoring data.

[0029] The packaging module is used to call the data packaging service to add multiple second electrical monitoring data as incremental data to the first data package to obtain the second data package;

[0030] The first batch data entry module is used to call the batch data entry service to write the data in the second data package into the database in batches when the data volume of the second data package exceeds the first preset quantity threshold.

[0031] According to embodiments of this disclosure, the reduction module includes a grouping unit and a dehomogenization unit.

[0032] The grouping unit is used to group multiple first electrical monitoring data to obtain at least one array, wherein multiple first electrical monitoring data in the same array belong to the same electrical data type; the dehomogenization unit is used to dehomogenize multiple first electrical monitoring data in the array to obtain multiple second electrical monitoring data.

[0033] According to embodiments of this disclosure, the dehomogenization unit includes a first calculation subunit, a second calculation subunit, a determination subunit, and a deletion subunit.

[0034] The system includes a first calculation subunit for calculating the average value of multiple first electrical monitoring data in an array to obtain a target mean; a second calculation subunit for calculating the difference between the multiple first electrical monitoring data and the target mean; a determination subunit for determining target homogeneous data among the multiple first electrical monitoring data based on the difference between the multiple first electrical monitoring data and the target mean, wherein the target homogeneous data is: the first electrical monitoring data whose difference from the target mean is less than a preset difference threshold; and a deletion subunit for deleting the target homogeneous data from the multiple first electrical monitoring data to obtain multiple second electrical monitoring data.

[0035] According to an embodiment of this disclosure, the grouping unit includes a grouping subunit, which is used to group multiple first electrical monitoring data according to different electrical data types to obtain voltage array, current array, and temperature array.

[0036] According to embodiments of this disclosure, the first batch warehousing module includes an assembly unit and a batch warehousing unit.

[0037] The assembly unit is used to assemble data of the same data type from the second data packet into the same target database operation statement; the batch data entry unit is used to perform batch data writing operations using the target database operation statement.

[0038] According to embodiments of this disclosure, the above-described apparatus further includes a second batch data entry module, used to call a batch data entry service to write the data in the first data packet into the database in batches when the first electrical monitoring data is not read from the message queue.

[0039] According to embodiments of this disclosure, the above-described apparatus further includes a reading module for reading multiple first electrical monitoring data from a message queue, wherein the reading module includes a first reading unit and a second reading unit.

[0040] The first retrieval unit is used to read a preset number of data from the message queue as multiple first electrical monitoring data when the amount of data in the message queue is greater than or equal to a second preset quantity threshold, wherein the preset number is equal to the second preset quantity threshold; the second reading unit is used to read all data from the message queue as multiple first electrical monitoring data when the amount of data in the message queue is less than the second preset quantity threshold.

[0041] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described electrical monitoring data processing method.

[0042] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described electrical monitoring data processing method.

[0043] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the above-described electrical monitoring data processing method.

[0044] According to the electrical monitoring data processing method, apparatus, equipment, medium, and program products provided in this disclosure, the electrical monitoring data is dynamically reduced by calling a data reduction service, thereby reducing the amount of data inserted into the database. Furthermore, by packaging the reduced monitoring data and assembling the packaged data for batch insertion into the database, the resource consumption from frequent database connection operations is reduced. By reducing the data volume and packaging it for batch insertion into the database on the server side, the amount of data is reduced, lowering the database storage requirements. This solves the technical problem of concurrency performance bottlenecks when electrical monitoring data is inserted into the database in real time on the server side, improving database insertion efficiency and eliminating the database as a performance bottleneck. In the implementation of large-scale IoT applications for electrical fire prevention, the massive amounts of data generated per minute by edge sensors (e.g., millions of real-time voltage, current, and temperature monitoring data points per minute) can be processed using the above data processing method to meet the requirements for large-scale data processing. Attached Figure Description

[0045] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0046] Figure 1 The illustration schematically depicts application scenarios of electrical monitoring data processing methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure;

[0047] Figure 2 A flowchart illustrating an electrical monitoring data processing method according to an embodiment of the present disclosure is shown schematically.

[0048] Figure 3 A schematic diagram illustrating the system architecture of an electrical monitoring data processing method according to an embodiment of the present disclosure is shown.

[0049] Figure 4 This schematically illustrates a flowchart of a method for dehomogenizing first electrical monitoring data according to an embodiment of the present disclosure;

[0050] Figure 5 A schematic block diagram of an electrical monitoring data processing apparatus according to an embodiment of the present disclosure is shown; and

[0051] Figure 6 A block diagram of an electronic device suitable for implementing an electrical monitoring data processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0052] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0053] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0054] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0055] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0056] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0057] In IoT applications for preventing electrical fires, monitoring points are set up on electrical circuits within buildings, and edge-end electrical fire sensors are installed. These sensors monitor the voltage, current, and temperature data at these points in real time and transmit the data to a server for storage and subsequent analysis.

[0058] One method for transmitting monitoring data into a database for processing in related technologies is as follows: Electrical fire sensors at several electrical circuit monitoring points within a building transmit real-time monitoring data to an edge-end electrical fire safety gateway, which then transmits the data to the server. The safety gateway itself has limited performance and lacks data processing capabilities. After receiving the sensor data, the safety gateway transmits the raw monitoring data to the server via the MQTT protocol. An MQ message queue is deployed on the server to receive and cache the monitoring data sent by each edge-end safety gateway in real time. An automatic data processing service is also deployed on the server, periodically reading monitoring data from the MQ message queue and inserting it into the database for storage in real time, for subsequent electrical fire data analysis.

[0059] The above method can operate normally during small-scale pilot testing. However, in the implementation of large-scale IoT applications for preventing electrical fires, there are more than 1 million electrical fire sensors and more than 100,000 electrical fire safety gateways installed at the edge electrical circuit monitoring points. Approximately 4 million real-time monitoring data points of voltage, current, and temperature are generated every minute. When these real-time monitoring data are transmitted to the server and the server's automatic data processing service directly inserts them into the database storage one by one in real time, the database cannot respond in time to the insertion of millions of data points per minute, resulting in a concurrency performance bottleneck.

[0060] In general, during the implementation of IoT applications for large-scale electrical fire prevention, there are technical challenges related to concurrency performance bottlenecks when monitoring data is inserted into the database on the server side in real time, due to the large number of monitoring points and the large amount of real-time monitoring data.

[0061] In view of the above, embodiments of this disclosure provide an electrical monitoring data processing method to at least partially solve the above-mentioned technical problems. The electrical monitoring data processing method includes: when multiple first electrical monitoring data are read from a message queue, calling a data reduction service to reduce the multiple first electrical monitoring data to obtain multiple second electrical monitoring data; calling a data packaging service to add the multiple second electrical monitoring data as incremental data to a first data packet to obtain a second data packet; and when the data volume of the second data packet is greater than a first preset quantity threshold, calling a batch data entry service to batch write the data in the second data packet into a database.

[0062] Figure 1 The illustration schematically depicts an application scenario of an electrical monitoring data processing method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0063] like Figure 1As shown, application scenario 100 according to this embodiment may include an electrical sensor 101, a gateway 102, middleware 103, a data processing server 104, and a database 105. The electrical sensor 101, gateway 102, middleware 103, data processing server 104, and database 105 can communicate with each other via a network, which may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0064] Electrical sensor 101 is a sensor installed on an electrical circuit circuit, which can detect data such as voltage, current, and temperature of the electrical circuit in real time.

[0065] Gateway 102 has simple remote communication capabilities, which are used to access data detected by electrical fire sensors and send it to middleware 103 via network transmission protocols.

[0066] Middleware 103 is a container that stores messages during message transmission and is used for communication between distributed systems. It is used to receive electrical line monitoring data sent by gateway 102.

[0067] The data processing server 104 can be used to execute the electrical monitoring data processing method of this disclosure embodiment, including four sub-services: data reading service, data reduction service, data packaging service, and batch database entry service, which work together to coordinate the dynamic reduction of electrical fire monitoring data and batch database insertion. For example, after reading electrical circuit monitoring data from the middleware 103, the data is dynamically reduced, the reduced data is packaged, and when the packaged data volume reaches a preset threshold, batch database entry is performed to store the packaged data in batches into the database 105.

[0068] It should be noted that the electrical monitoring data processing method provided in this embodiment can generally be executed by the data processing server 104. Correspondingly, the electrical monitoring data processing device provided in this embodiment can generally be located within the data processing server 104. The electrical monitoring data processing method provided in this embodiment can also be executed by a server or server cluster that is different from the data processing server 104 but capable of communicating with it. Correspondingly, the electrical monitoring data processing device provided in this embodiment can also be located within a server or server cluster that is different from the data processing server 104 but capable of communicating with it.

[0069] It should be understood that Figure 1 The number of electrical sensors 101, gateway 102, middleware 103, data processing server 104, and database 105 shown is merely illustrative. The number can be arbitrary depending on implementation requirements.

[0070] The following will be based on Figure 1The described scene, through Figures 2-6 The electrical monitoring data processing method of the disclosed embodiments will be described in detail.

[0071] Figure 2 A flowchart illustrating an electrical monitoring data processing method according to an embodiment of the present disclosure is shown. Figure 3 A schematic diagram illustrating a system architecture of an electrical monitoring data processing method according to an embodiment of this disclosure is provided below. Figure 2 , Figure 3 The present disclosure describes an embodiment of an electrical monitoring data processing method.

[0072] like Figure 2 As shown, the electrical monitoring data processing method of this embodiment includes operations S201 to S203.

[0073] In operation S201, when multiple first electrical monitoring data are read from the message queue, the data reduction service is called to reduce the multiple first electrical monitoring data to obtain multiple second electrical monitoring data.

[0074] In operation S202, the data packaging service is called to add multiple second electrical monitoring data as incremental data to the first data packet to obtain the second data packet;

[0075] In operation S203, if the amount of data in the second data packet exceeds the first preset quantity threshold, the batch data entry service is invoked to write the data in the second data packet into the database in batches.

[0076] like Figure 3 As shown, the system architecture of the electrical monitoring data processing method applicable to the embodiments of this disclosure includes an edge terminal, a server terminal, and an intermediate transmission architecture between the edge terminal and the server terminal.

[0077] Among them, the edge end includes an electrical fire sensor, which is a sensor installed on an electrical circuit to detect data such as voltage, current, and temperature of the electrical circuit in real time.

[0078] Data is transmitted between the edge and the server through an electrical fire safety gateway and middleware.

[0079] The electrical fire safety gateway has basic remote communication capabilities, which are used to access data detected by electrical fire sensors and send it to the server via network transmission protocols.

[0080] Middleware can be implemented using message queues (MQ), which are containers that store messages during transmission and are used for communication between distributed systems. The server-side MQ message queue is implemented by deploying RabbitMQ message middleware in a cluster. It receives monitoring data from edge security gateways in real time via the MQTT (Message Queuing Telemetry Transport) protocol and caches the data. MQTT is a client-server message publish / subscribe protocol; it is lightweight, simple, open, and easy to implement, and is widely used in performance-constrained IoT devices. RabbitMQ is an open-source message queuing service software (message-oriented middleware), an open-source implementation of the Advanced Message Queuing Protocol (AMQP) provided by LShift, written in Erlang, known for its high performance, robustness, and scalability.

[0081] The electrical monitoring data processing method of this disclosure can be applied to Internet of Things (IoT) applications for preventing electrical fires. The edge-end electrical fire sensors and electrical fire safety gateways are provided by relevant electrical fire safety hardware equipment manufacturers and installed at electrical circuit monitoring points. The electrical fire sensors monitor the voltage, current, and temperature data of the points in real time and transmit them to the electrical fire safety gateway. After receiving the sensor data, the safety gateway transmits the raw monitoring data to the server through an MQ message queue for storage and subsequent analysis.

[0082] According to embodiments of this disclosure, the server includes an automatic data processing service (cluster) for executing the electrical monitoring data processing method of this disclosure. It includes one main service (automatic data processing framework service) and four sub-services (data reading service, data reduction service, data packaging service, and batch database insertion service) to coordinate the dynamic reduction of electrical fire monitoring data and batch insertion into the database.

[0083] The main service (automatic data processing framework service) acts as a scheduling framework. It can be an independent process implemented using Spring Boot, automatically triggering calls to data reading, data reduction, data packaging, and batch data import services at regular intervals to perform data processing. Spring Boot is a Java framework designed to simplify the creation, running, debugging, and deployment of Spring applications, allowing developers to focus on Spring application development. The main service calls the various sub-services via RESTful APIs, reading primary electrical monitoring data from a message queue (MQ). RESTful is a web application design style and development approach based on HTTP, and can be defined using XML or JSON formats.

[0084] Specifically, in the above operation S201, the main service (automatic data processing framework service) calls the data reading service to perform the data reading operation, reads multiple first electrical monitoring data from the message queue, and, in the case of reading multiple first electrical monitoring data, calls the data reduction service to perform the data reduction operation, and obtains multiple second electrical monitoring data after reducing the multiple first electrical monitoring data.

[0085] Reading data could be done by calling a data reading service interface to read the first electrical monitoring data from an MQ message queue.

[0086] The first electrical monitoring data can be the raw voltage, current, and temperature data collected in real time by fire sensors. Multiple first electrical monitoring data points are processed to obtain multiple second electrical monitoring data points. This can be achieved by calling a data reduction service interface to reduce the data volume obtained in the previous step; for example, it can be achieved by removing homogeneous data from multiple first electrical monitoring data points to obtain the second electrical monitoring data, thereby reducing the data volume.

[0087] In operation S202, the main service (automatic data processing framework service) calls the data packaging service to perform data packaging. Multiple second electrical monitoring data points are added as incremental data to the first data package to obtain the second data package. The first data package is the old data package before the addition of the second electrical monitoring data points, and the second data package is the new data package obtained after the addition of the second electrical monitoring data points to the first data package. The data packaging service is an independent process implemented based on Spring Boot. For the reduced data, it can group the data by data type and add it to the corresponding packaged data array in append form (parameters passed by reference). Simultaneously, based on the processed data count (the amount of second electrical monitoring data), it adjusts the packaged data volume and returns the packaged data volume.

[0088] In operation S203, the main service (automatic data processing framework service) calls the batch data entry service. If the data volume of the second data packet is greater than the first preset quantity threshold (batch parameter p), the data in the second data packet is written into the database in batches.

[0089] For example, the electrical monitoring data processing method described above in this disclosure embodiment may be:

[0090] Before calling the data service, the initial data size of the first data packet can be set to c1 = 0, and the data packet array (empty array) can be initialized.

[0091] During the data processing, operation S201 calls the data reading service to read r first electrical monitoring data from the message queue, and determines the data volume r (number of data entries) and the data volume c1 of the first data packet.

[0092] Scenario 1: If r≠0, firstly, the data reduction service is called to perform the data reduction operation, and multiple first electrical monitoring data are obtained after the data reduction processing is performed on multiple second electrical monitoring data.

[0093] Next, the data packaging service interface is called to package the reduced second electrical monitoring data into a data package. Multiple second electrical monitoring data are added as incremental data to the first data package to obtain the second data package. The first data package is the old data package before the addition of multiple second electrical monitoring data, and its data volume (number of data entries) is set as c1. The second data package is the new data package obtained after the addition of multiple second electrical monitoring data in the first data package, and its data volume is c2 = c1 + k, where k is the data volume (number of data entries) of multiple second electrical monitoring data. The packaged data volume c2 is then returned.

[0094] Then, if the data volume c2 of the second data packet is greater than the first preset quantity threshold (batch parameter p), the data batch entry service interface is called to write the data in the second data packet into the database in batches.

[0095] Scenario 2: If r = 0 and c1 ≠ 0, that is, if the first electrical monitoring data is not read from the message queue, the batch data entry service can be directly called to write the data in the first data packet into the database in batches.

[0096] Scenario 3: If r = 0 and c1 = 0, then the process ends directly.

[0097] According to embodiments of this disclosure, the above-described electrical monitoring data processing method provides a data processing method for dynamic data reduction and batch data insertion into the database. By calling a data reduction service to dynamically reduce the amount of electrical monitoring data, the amount of data inserted into the database is reduced. By packaging the reduced monitoring data and assembling the packaged data for batch insertion into the database, the resource consumption of frequent database connection operations is reduced. By reducing the data volume and assembling it for batch insertion into the database on the server side, the amount of data is reduced, and the database storage usage is lowered. This solves the technical problem of concurrency performance bottlenecks when electrical monitoring data is inserted into the database in real time on the server side, improves database insertion efficiency, and eliminates the database as a performance bottleneck. In the implementation of large-scale IoT applications for preventing electrical fires, the above-described data processing method can meet the requirements for large-scale data processing for the massive amounts of data generated per minute by edge sensors (e.g., millions of real-time voltage, current, and temperature monitoring data per minute).

[0098] According to embodiments of this disclosure, after calling the data reading service to perform a data reading operation from the message queue, it is necessary to determine the data volume r (number of data entries) of the first electrical monitoring data to be read. If r ≠ 0, i.e., data has been read, the data reduction, data packaging, and batch storage operations described in the above embodiments of this disclosure are performed. If r = 0, i.e., no data has been read, a data anomaly has occurred. Embodiments of this disclosure also provide a method for handling data anomalies.

[0099] According to the embodiments of this disclosure, a comprehensive judgment needs to be made by combining the data volume c1 (number of data entries) of the first data packet and the data volume r (number of data entries) of the first electrical monitoring data.

[0100] Specifically, if r = 0 and c1 ≠ 0, that is, if the first electrical monitoring data is not read from the message queue, but the current packaged data volume is not zero, the batch data entry service can be directly called to write the data in the first data packet into the database in batches.

[0101] If r = 0 and c1 = 0, that is, if the first electrical monitoring data is not read from the message queue and the current packaged data volume is zero, the processing program will end directly.

[0102] According to the embodiments of this disclosure, by means of the above method, in the event of a data reading anomaly, there is no need to wait for the data volume of the current data packet to meet the batch entry conditions before performing batch entry. Instead, by directly entering the current data packet or terminating the current process, the next process of data processing can be carried out in a timely manner. Through this data anomaly handling mechanism, the technical problem of interruption of the data processing process due to data reading anomalies is solved, and the situation of subsequent processing processes being affected by the failure of the current process is avoided, further improving the timeliness of data processing.

[0103] According to embodiments of this disclosure, the data reduction service can be an independent process implemented based on Spring Boot. It can group data according to device number and data type, and eliminate homogeneous data to reduce the total data volume, taking into account the characteristics of large data volume, concentrated data distribution, and many identical data in a short period of time collected by electrical fire IoT sensors.

[0104] Specifically: After reducing the amount of multiple first electrical monitoring data, multiple second electrical monitoring data are obtained, including:

[0105] First, multiple first electrical monitoring data are grouped to obtain at least one array, wherein multiple first electrical monitoring data in the same array belong to the same electrical data type. In the application scenario of this disclosure embodiment, multiple first electrical monitoring data can be grouped according to different electrical data types to obtain voltage array, current array, and temperature array. Specifically, multiple first electrical monitoring data in the data list read from the message queue are traversed, and the data are grouped and recorded into a two-dimensional array arr[data type][device number] (data types include voltage, current, and temperature) according to the device number and data type of the data.

[0106] Then, multiple sets of first electrical monitoring data in the array are dehomogenized to obtain multiple sets of second electrical monitoring data.

[0107] According to embodiments of this disclosure, the electrical monitoring data collected by the sensors includes various types of data, such as current data, voltage data, temperature data, etc. Since the electrical monitoring data collected by the sensors is uploaded to the message queue, the data sent to the server in real time by the message queue does not distinguish between data types. Because different types of data have significantly different performance characteristics, subsequent de-homogenization processing cannot be performed directly using the same processing method. Therefore, it is necessary to pre-group the data according to different data types to form the data foundation for subsequent homogenization processing. During subsequent de-homogenization processing, each data type can be processed separately to avoid errors in subsequent processing results due to confusion between different data types.

[0108] According to the embodiments of this disclosure, considering the characteristics of large data volume, concentrated data distribution, and many identical data in a short period of time in the data collected by the IoT sensor for electrical fires, the above method can be used to group the data according to data type and remove homogeneous data, which can reduce the total data volume, improve the efficiency of data entry, reduce storage pressure, and improve data quality.

[0109] Figure 4 A flowchart illustrating a method for dehomogenizing first electrical monitoring data according to an embodiment of this disclosure is shown schematically. The following is in conjunction with... Figure 4 The method for dehomogenization processing according to embodiments of this disclosure will be described.

[0110] like Figure 4 As shown, the method for dehomogenizing the first electrical monitoring data in this embodiment includes operations S401 to S404.

[0111] In operation S401, the average value of multiple first electrical monitoring data in the array is calculated to obtain the target average value. Specifically, after grouping and recording the data into a two-dimensional array arr[data type][device number] according to the data type (data type includes voltage, current, and temperature) of the first electrical monitoring data, the sum s of the monitoring data values ​​recorded in the voltage array, current array, and temperature array, as well as the data count n in each array, are calculated respectively. Based on s and n, the average value a of each of the voltage array, current array, and temperature array is calculated.

[0112] In operation S402, the differences between multiple first electrical monitoring data in the array and the target mean are calculated respectively; specifically, each two-dimensional array arr[data type][device number] is traversed to calculate the deviation e of each monitoring data value from the average value a.

[0113] In operation S403, based on the differences between multiple first electrical monitoring data and the target mean, target homogeneous data is determined among the multiple first electrical monitoring data. Target homogeneous data refers to the first electrical monitoring data whose difference from the target mean is less than a preset difference threshold. Specifically, after calculating the deviation *e* between each monitoring data value and the average value *a*, it is compared with the maximum error parameter *me* (the maximum error parameter *me* is configured according to the data type; for example, the maximum error parameter *me* for voltage is configured as 1). If the deviation *e* < the maximum error *me*, then this data is considered target homogeneous data.

[0114] In operation S404, multiple homogeneous target data are deleted from multiple first electrical monitoring data to obtain multiple second electrical monitoring data. That is, if the deviation e < the maximum error me, the monitoring data value is discarded. After the traversal is completed, if the two-dimensional array arr is empty, a record with a data value of average value a is added (the time is also averaged).

[0115] According to embodiments of this disclosure, because the data collected by the IoT sensors for electrical fires is concentrated, there are many data points with the same or similar values ​​collected within a short period of time. For example, voltage, current, and temperature values ​​collected within a short time interval are quite similar, resulting in significant data homogenization. This high degree of data homogenization increases the pressure on data transmission and storage, but offers no practical reference value for electrical safety analysis. Therefore, embodiments of this disclosure provide a simple and easy-to-implement data reduction algorithm. By periodically calculating the average value of the collected data within a time segment and comparing the deviation of each monitoring data point from the average value, similar values ​​are eliminated using the maximum error parameter, thereby achieving data reduction. This significantly improves data quality, reduces the pressure on data transmission and storage, and improves data processing efficiency by reducing the amount of data.

[0116] According to embodiments of this disclosure, in the electrical monitoring data processing method of this disclosure, when the data volume of the packaged second data packet exceeds a first preset quantity threshold, a batch data entry service is invoked to batch write the data in the second data packet into the database. Specifically, batch writing the data in the second data packet into the database includes the following operations:

[0117] First, data of the same data type from the second data packet are assembled into the same target database operation statement; then, the target database operation statement is used to perform a batch data write operation.

[0118] For example, the batch data insertion service can be a standalone process implemented based on Spring Boot. This service takes the passed packaged data array arrp[data type], traverses the data according to data type (corresponding to different data table types), and assembles the batch insertion SQL. The assembled SQL is shown in the following example (using MySQL as an example):

[0119] insert into table name(key1, key2,…)

[0120] values(value1, value2,…), values(value3, value4,…),….

[0121] After the SQL is assembled, data can be batch-inserted into the database using middleware (such as MyBatis middleware).

[0122] According to embodiments of this disclosure, by packaging the reduced monitoring data and assembling the packaged data for batch insertion into the database, a large amount of data can be inserted by performing only a few database connection and transaction commit operations. Compared with the method of inserting data one record at a time, this reduces the resource consumption of frequent database connection operations and improves the overall data insertion efficiency.

[0123] According to embodiments of this disclosure, the automatic data processing framework service enhances the operation of calling the data reading service to perform data reading, reading multiple first electrical monitoring data from the message queue. Specifically, reading multiple first electrical monitoring data from the message queue can be done in a batch reading manner, which may include the following scenarios:

[0124] In scenario one, if the amount of data in the message queue is greater than or equal to the second preset quantity threshold, a preset number of data is read from the message queue as multiple first electrical monitoring data, wherein the preset number is equal to the second preset quantity threshold.

[0125] In scenario two, if the amount of data in the message queue is less than the second preset threshold, all data is read from the message queue as multiple first electrical monitoring data.

[0126] For example, the data reading service can be a standalone process implemented based on Spring Boot, embedding a RabbitMQ Java client, connecting to an MQ message queue cluster service, and batch reading messages from the MQ message queue. The number of data items read in each batch is set to the parameter 'm', and the value of 'm' is configured through a configuration file. For example, configuring 'm=2000' means that a maximum of 2000 data items will be retrieved at a time. The data reading service provides an interface to the outside world. When the interface is called, it triggers a batch reading of messages and returns a list of the read messages.

[0127] According to the embodiments of this disclosure, since the electrical fire sensors installed at the edge electrical circuit monitoring points generate a large amount of real-time monitoring data of voltage, current and temperature per minute (for example, about 4 million data points per minute), reading all the data at once would cause the data reading time to be too long, affecting the subsequent data processing process. By reading in batches, the performance bottleneck problem in the data reading process can be solved. By reading data in batches and in small amounts multiple times, the data can be effectively distributed, and the overall data processing process can be accelerated.

[0128] Based on the above-described electrical monitoring data processing method, this disclosure also provides an electrical monitoring data processing device. The following will be combined with... Figure 5 The device is described in detail.

[0129] Figure 5A schematic block diagram of an electrical monitoring data processing apparatus according to an embodiment of the present disclosure is shown.

[0130] like Figure 5 As shown, the electrical monitoring data processing device 500 of this embodiment includes a reduction module 501, a packaging module 502, and a first batch warehousing module 503.

[0131] Among them, the reduction module 501 is used to call the data reduction service to reduce the multiple first electrical monitoring data after reading multiple first electrical monitoring data from the message queue, so as to obtain multiple second electrical monitoring data.

[0132] The packaging module 502 is used to call the data packaging service to add multiple second electrical monitoring data as incremental data to the first data packet to obtain the second data packet;

[0133] The first batch data entry module 503 is used to call the batch data entry service to write the data in the second data package into the database in batches when the data volume of the second data package exceeds the first preset quantity threshold.

[0134] According to embodiments of this disclosure, the reduction module 501 calls a data reduction service to dynamically reduce the amount of electrical monitoring data, thereby reducing the amount of data inserted into the database. The packaging module 502 and the first batch insertion module 503 package the reduced monitoring data and then batch insert the packaged data into the database, reducing the resource consumption of frequent database connection operations. By reducing the amount of data and packaging it for batch insertion into the database on the server side, the amount of data is reduced, the database storage is lowered, and the technical problem of concurrency performance bottleneck when electrical monitoring data is inserted into the database in real time on the server side is solved. This improves the database insertion efficiency, making the database no longer a performance bottleneck. In the implementation of large-scale IoT applications for preventing electrical fires, this method can meet the requirements for large-scale data processing for the massive amounts of data generated per minute by edge sensors (e.g., millions of real-time voltage, current, and temperature monitoring data per minute).

[0135] According to embodiments of this disclosure, the reduction module includes a grouping unit and a dehomogenization unit.

[0136] The grouping unit is used to group multiple first electrical monitoring data to obtain at least one array, wherein multiple first electrical monitoring data in the same array belong to the same electrical data type; the dehomogenization unit is used to dehomogenize multiple first electrical monitoring data in the array to obtain multiple second electrical monitoring data.

[0137] According to embodiments of this disclosure, the dehomogenization unit includes a first calculation subunit, a second calculation subunit, a determination subunit, and a deletion subunit.

[0138] The system includes a first calculation subunit for calculating the average value of multiple first electrical monitoring data in an array to obtain a target mean; a second calculation subunit for calculating the difference between the multiple first electrical monitoring data and the target mean; a determination subunit for determining target homogeneous data among the multiple first electrical monitoring data based on the difference between the multiple first electrical monitoring data and the target mean, wherein the target homogeneous data is: the first electrical monitoring data whose difference from the target mean is less than a preset difference threshold; and a deletion subunit for deleting the target homogeneous data from the multiple first electrical monitoring data to obtain multiple second electrical monitoring data.

[0139] According to an embodiment of this disclosure, the grouping unit includes a grouping subunit, which is used to group multiple first electrical monitoring data according to different electrical data types to obtain voltage array, current array, and temperature array.

[0140] According to embodiments of this disclosure, the first batch warehousing module includes an assembly unit and a batch warehousing unit.

[0141] The assembly unit is used to assemble data of the same data type from the second data packet into the same target database operation statement; the batch data entry unit is used to perform batch data writing operations using the target database operation statement.

[0142] According to embodiments of this disclosure, the above-described apparatus further includes a second batch data entry module, used to call a batch data entry service to write the data in the first data packet into the database in batches when the first electrical monitoring data is not read from the message queue.

[0143] According to embodiments of this disclosure, the above-described apparatus further includes a reading module for reading multiple first electrical monitoring data from a message queue, wherein the reading module includes a first reading unit and a second reading unit.

[0144] The first reading unit is used to read a preset number of data from the message queue as multiple first electrical monitoring data when the amount of data in the message queue is greater than or equal to the second preset number threshold, wherein the preset number is equal to the second preset number threshold; the second reading unit is used to read all data from the message queue as multiple first electrical monitoring data when the amount of data in the message queue is less than the second preset number threshold.

[0145] According to embodiments of this disclosure, any multiple modules among the reduction module 501, packaging module 502, and first batch warehousing module 503 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the reduction module 501, packaging module 502, and first batch warehousing module 503 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the reduction module 501, packaging module 502, and first batch warehousing module 503 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0146] Figure 6 A block diagram of an electronic device suitable for implementing an electrical monitoring data processing method according to an embodiment of the present disclosure is shown schematically.

[0147] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0148] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0149] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0150] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0151] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0152] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the electrical monitoring data processing method provided in the embodiments of this disclosure.

[0153] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0155] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0156] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0158] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0159] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for processing electrical monitoring data, comprising: When multiple first electrical monitoring data are read from a message queue, a data reduction service is invoked to reduce the amount of first electrical monitoring data to obtain multiple second electrical monitoring data. This includes: grouping the multiple first electrical monitoring data to obtain at least one array, where multiple first electrical monitoring data in the same array belong to the same electrical data type; traversing each array to calculate the average value of the multiple first electrical monitoring data in the array to obtain a target average; calculating the difference between the multiple first electrical monitoring data in the array and the target average; determining target homogeneous data among the multiple first electrical monitoring data based on the difference between the multiple first electrical monitoring data in the array and the target average, wherein the target homogeneous data is: the first electrical monitoring data whose difference from the target average is less than a preset difference threshold; deleting the target homogeneous data from the multiple first electrical monitoring data to obtain the multiple second electrical monitoring data; after traversing the array, if the data in the array is empty, adding one data entry as the average value of the array; and invoking a data packaging service to add the multiple second electrical monitoring data as incremental data to a first data packet to obtain a second data packet. If the amount of data in the second data packet exceeds the first preset quantity threshold, the batch data entry service is invoked to write the data in the second data packet into the database in batches.

2. The method according to claim 1, wherein, After grouping the plurality of first electrical monitoring data, at least one array is obtained, including: According to the different types of electrical data, the multiple first electrical monitoring data are grouped and processed to obtain voltage array, current array, and temperature array.

3. The method according to claim 1, wherein, Calling the batch data import service to write the data in the second data packet into the database in batches includes: Assemble data of the same data type from the second data packet into the same target database operation statement; Batch data writing operations are performed using the target database operation statements.

4. The method according to any one of claims 1-3, further comprising: If the first electrical monitoring data is not read from the message queue, the batch data entry service is invoked to write the data in the first data packet into the database in batches.

5. The method according to claim 4, wherein, Reading multiple first electrical monitoring data from the message queue includes: If the amount of data in the message queue is greater than or equal to a second preset quantity threshold, a preset quantity of data is read from the message queue as the plurality of first electrical monitoring data, wherein the preset quantity is equal to the second preset quantity threshold; If the amount of data in the message queue is less than the second preset quantity threshold, all data is read from the message queue as the plurality of first electrical monitoring data.

6. An electrical monitoring data processing device, comprising: The reduction module is used to call the data reduction service to reduce the multiple first electrical monitoring data after reading multiple first electrical monitoring data from the message queue, so as to obtain multiple second electrical monitoring data. The reduction module includes a grouping unit and a dehomogenization unit. The grouping unit is used to group the multiple first electrical monitoring data to obtain at least one array, wherein multiple first electrical monitoring data in the same array belong to the same electrical data type. The dehomogenization unit is used to: traverse each array to calculate the average value of multiple first electrical monitoring data in the array to obtain a target average value; calculate the difference between the multiple first electrical monitoring data in the array and the target average value respectively; determine the target homogeneous data among the multiple first electrical monitoring data based on the difference between the multiple first electrical monitoring data in the array and the target average value, wherein the target homogeneous data is: the first electrical monitoring data among the multiple first electrical monitoring data whose difference from the target average value is less than a preset difference threshold; delete the target homogeneous data among the multiple first electrical monitoring data to obtain the multiple second electrical monitoring data; after traversing the array, if the data in the array is empty, add a data entry as the average value of the array; The packaging module is used to call the data packaging service to add the multiple second electrical monitoring data as incremental data to the first data packet to obtain the second data packet; The first batch data entry module is used to call the batch data entry service to write the data in the second data packet into the database in batches when the data volume of the second data packet exceeds the first preset quantity threshold.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images