A method, system and medium for processing edge data
By receiving and processing edge data on the central server, using sliding windows and modal decomposition technology, the problem of low data processing efficiency at the edge is solved, real-time efficient data processing and feature extraction are realized, and data accumulation is avoided.
Patent Information
- Application Number
- CN202411161427.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-23
AI Technical Summary
The data processing efficiency at the edge end is low, resulting in data accumulation problems. Especially when the number of measurement points increases, the data processing speed cannot keep up with the data reception speed.
The central server process uses the process to receive edge data and store it in the temporary queue. When it is judged that modal decomposition is needed, the data is merged into an array matrix, and the sliding window is used to identify the intrinsic modal function and iterate the parameter until the threshold is reached, and the modal is generated. state data sequence.
It improves the data reception and processing speed, avoids data accumulation, realizes real-time and efficient processing of edge-side data, simplifies the configuration process, and extracts local features of the data.
Smart Images

Figure CN118689654B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data processing methods, and more specifically, to a method, system and medium for processing edge data. Background Art
[0002] With the development of the Internet of Things, edge computing has gradually been valued by more and more industries. At present, in practical applications, the edge is mainly faced with two problems: first, the edge generates a large amount of data every day, and large-scale data transmission will incur more costs, so it is particularly important to filter and process the data at the edge and synchronize the processed data to the cloud; second, the number of measurement points at the edge continues to increase during use. For the increasing number of measurement point data, an IO (input and output) problem processing is particularly important. As the frequency of data collection at the measurement point increases, the data processing speed needs to be improved accordingly. If efficient data processing cannot be performed on limited machines, data accumulation may occur. Summary of the invention
[0003] The present application is provided to solve the above defects in the prior art. A method, system and medium for processing edge data are needed, which can improve the data receiving speed and data processing speed and avoid data accumulation when the number of edge measurement points increases and the measurement point data is large.
[0004] According to a first scheme of the present application, a method for processing edge data is provided, wherein the edge data is real-time data from at least one monitoring device at the edge, and the processing method comprises the following steps: a central server receives edge data from multiple edge monitoring devices using a processing process, and stores the received edge data in a temporary storage queue in chronological order; when it is determined that the edge data in the temporary storage queue needs to be modally decomposed, the edge data is stored in a modal analysis data queue; a modal decomposition unit deployed in the central server sequentially performs modal decomposition on the edge data in the modal analysis data queue, wherein the first edge monitoring device is subjected to a modal decomposition. The processing of edge data includes executing the following steps: merging the first edge data of the first edge monitoring device within the first time range in the modal analysis data queue into a first array matrix; in the first array matrix, using a sliding window to extract the first edge data within the second matrix range, identifying the intrinsic modal function in the extracted first edge data, and iteratively updating the parameters of the identified intrinsic modal function after the sliding window moves according to a preset step size until the end of the first array matrix is reached and / or the difference between two adjacent parameter iterations is less than a preset threshold; merging the data belonging to the same intrinsic modal function in the first array matrix into a corresponding modal data sequence.
[0005] According to the second scheme of the present application, a system for processing edge data is provided, the processing system comprising at least one edge monitoring device and a central server, the edge monitoring device sends real-time edge data to the central server; the central server executes the edge data processing method described in any embodiment of the present application.
[0006] According to a third solution of the present application, a non-temporary computer-readable medium is provided, on which instructions are stored. When executed by a processor, the instructions execute the steps of the edge data processing method as described in any embodiment of the present application.
[0007] The processing method, processing system and medium of edge data provided by each embodiment of the present application use a processing process to only receive edge data and store it in a temporary storage queue, without waiting for data processing before continuing to receive data, so that even when the data flow rate is large, data can still be received in real time and correctly. In the case of judging that the edge data in the temporary storage queue needs to be modally decomposed, the data is distinguished, and the data that needs to be modally decomposed is stored in the modal analysis data queue. When performing modal decomposition, the first edge data within the first time range is first merged into a first array matrix, and then the first edge data within the second matrix range is extracted using a sliding window, so that the data can be locally processed and the processing speed can be improved. At the end of the first array matrix and / or the difference between two adjacent parameter iterations is less than a preset threshold, the corresponding modal data sequence is obtained. It is convenient for the data processing speed to keep up with the data receiving speed, realize real-time and efficient data processing, avoid the accumulation of edge data in the central server, and can also be conducive to extracting local features of the data. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. The same reference numerals with letter suffixes or different letter suffixes may represent different instances of similar parts. The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the specification and claims, are used to illustrate the embodiments applied for. When appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.
[0009] Figure 1 A flowchart showing a method for processing edge data according to an embodiment of the present application is shown;
[0010] Figure 2 A flowchart showing a method for identifying an intrinsic modal function according to an embodiment of the present application;
[0011] Figure 3 A schematic diagram of the structure of an edge data processing system according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0012] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation examples. The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation examples, but are not intended to limit the present application.
[0013] The words "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish. The words "include" or "comprise" and similar words mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of including other elements.
[0014] According to an embodiment of the present application, a method for processing edge data is provided, where the edge data is real-time data from at least one monitoring device at the edge. Figure 1 A flow chart of a method for processing edge data according to an embodiment of the present application is shown. The processing method comprises the following steps. The edge monitoring device may be a displacement sensor of a bridge, or an inclinometer sensor of a wind turbine, etc. Generally, a large number of monitoring devices are set, so a large amount of data is generated at the edge every day.
[0015] In step 101, the central server receives edge data from multiple edge monitoring devices using a processing process, and stores the received edge data in a temporary storage queue in chronological order. The processing process stores the received data in the temporary storage queue, and then receives the next data, rather than waiting until the processing of one data is completed before receiving the next data. In this way, a large amount of data generated by multiple edge monitoring devices can be received in real time, which is convenient for obtaining the conditions of corresponding equipment such as bridges, and can also improve the data receiving performance when the data collection frequency of the edge monitoring device increases. The data stored in the temporary storage queue is arranged in chronological order so that the data can be processed in chronological order later.
[0016] In step 102, when it is determined that the edge data in the temporary storage queue needs to be modally decomposed, the edge data is stored in the modal analysis data queue. The edge data in the temporary storage queue needs to determine the type of edge data, and the edge data that needs to be modally decomposed is stored in the modal analysis data queue, which is convenient for distinguishing edge data that needs to be modally decomposed from edge data that does not need to be modally decomposed. Only the edge data that needs to be modally decomposed is processed, and the edge data that does not need to be modally decomposed does not need to perform the next modal decomposition process, thereby improving the data processing speed.
[0017] In step 103, the modal decomposition unit deployed in the central server sequentially performs modal decomposition on the edge data in the modal analysis data queue, wherein processing the edge data of the first edge monitoring device includes executing the following steps. After receiving the edge data, the central server directly processes the data without transmitting it to other devices for processing.
[0018] In step 1031, the first edge data of the first edge monitoring device within the first time range in the modal analysis data queue are merged into a first array matrix. When data is stored in the modal analysis data queue, or the amount of stored data reaches a threshold, the first array matrices corresponding to each first time range are processed in chronological order, and the first time range can be changed. The data in the modal analysis data queue can be processed in real time.
[0019] In some embodiments, each row in the first array matrix can represent edge data at a time point. For example, the edge data corresponding to the time point of 1:10:01 is used as a row in the first array matrix. Each column represents a feature or a variable, which facilitates the arrangement of data to form an ordered first array matrix.
[0020] In step 1032, within the first array matrix, a sliding window is used to extract first edge data within the range of the second matrix, an intrinsic mode function is identified in the extracted first edge data, and the parameters of the identified intrinsic mode function are iteratively updated after the sliding window moves according to a preset step size until the end of the first array matrix is reached and / or the difference between two adjacent parameter iterations is less than a preset threshold.
[0021] For example, the first array matrix includes 20*20, and the size of the sliding window is 5*5. The first edge end data extracted each time by the sliding window is the first edge end data within the range of 5*5, and then the intrinsic mode function is identified.
[0022] The sliding window is used to select local data in the first array matrix, and then the corresponding data extracted by the sliding window is subjected to intrinsic mode function recognition, so that the local features of the long signal can also be extracted and calculated. The first edge end data within the second matrix range is extracted by the sliding window, which can realize real-time analysis of the data stream and obtain the rules, trends and anomalies of the first edge end data corresponding to the sliding window. By continuously sliding the window and performing statistical analysis on the first edge end data corresponding to the sliding window, changes and anomalies in the data can be discovered in time, and corresponding measures can be taken to process or adjust them.
[0023] Using sliding windows to extract edge data can optimize resource consumption to a certain extent, because only a fixed-size window needs to be maintained, without loading the entire data stream into memory for processing. This can save memory and computing resources and enable the system to run more efficiently.
[0024] When the end of the first array matrix is reached, it indicates that the modal function of the data in the first data matrix has been identified. Alternatively, when the difference between two adjacent parameter iterations is less than a preset threshold, the intrinsic modal function is obtained.
[0025] In step 1033, the data belonging to the same intrinsic modal function in the first array matrix are combined into a corresponding modal data sequence, so as to obtain the modal data corresponding to the first array matrix.
[0026] The edge data processing method of the present application receives data through a processing process and stores it in a temporary queue, which can improve the data reception performance. The algorithm can be calculated in parallel by sliding windows, and the reading and writing speed of data can be accelerated by using array matrices. The improved iteration stop condition is used to meet the adaptability of the algorithm, and efficient data processing can be achieved. The data receiving speed and data processing speed have been improved accordingly, and the data processing speed keeps up with the data receiving speed. For example, 1s of data is received and the data can be processed within 1 second, avoiding the accumulation of edge data on the central server, which is conducive to the efficient processing of edge data.
[0027] In some embodiments, the method for identifying the data type of the first edge data is: parsing the measurement point type information contained in the data packet. The measurement point type information includes strain data, displacement data, etc. In addition, the data packet also includes information such as the measurement point ID number, frequency, and timestamp. The measurement point ID indicates which monitoring device of which device it corresponds to, such as a bridge or a wind turbine, and the monitoring device at which position on the wind turbine, so as to identify the source of the data packet and distinguish data packets of different monitoring devices. The first array matrix corresponds to the first edge data of the first edge monitoring device, so as to obtain the intrinsic modal function corresponding to the data of a monitoring device, so as to perform corresponding data analysis.
[0028] In some embodiments, the received data packet is parsed through a json (JavaScript Object Notation) program to obtain formatted data, and then the content of the data packet is parsed to obtain the data type corresponding to the data packet, thereby determining whether the data packet needs to be modally decomposed.
[0029] In some embodiments, when there is an updated edge monitoring device at the edge, the central server sends the URL corresponding to the central server to the updated edge monitoring device; after receiving the URL corresponding to the central server, the updated edge monitoring device uses the http protocol to send the edge data to the URL corresponding to the central server.
[0030] For example, when a new sensor is added to a wind turbine, it is equivalent to having an updated edge monitoring device. The central server sends the URL corresponding to the central server to the updated edge monitoring device, so that when the new edge monitoring device generates edge data, it directly sends the edge data to the central server of the corresponding URL without having to reconfigure. The central server opens the port to the entire network segment, and the newly connected edge monitoring device automatically receives edge data in real time and processes the data. This can simplify the configuration process and improve the efficiency of edge monitoring devices sending data to the central server.
[0031] In some embodiments, performing intrinsic modal function identification in the extracted first edge data specifically includes the following steps. Figure 2 A flow chart of a method for identifying an intrinsic mode function according to an embodiment of the present application is shown. In step 201, the first extreme point of the first edge end data is extracted, and the first extreme point includes a maximum point and a minimum point. In step 202, when the difference between adjacent maximum points or minimum points is greater than a first threshold, the pad_extrema_rilling function is used to reflectively fill the extreme points of the first edge end data to obtain the filled second extreme point corresponding to the first edge end data. When the difference between two adjacent extreme points in the signal is large, the use of the EMD (Empirical Mode Decomposition) algorithm may generate additional noise or pseudo components in these areas, that is, an edge effect. By performing reflective filling between two adjacent extreme points whose difference is greater than the first threshold, the EMD algorithm will not identify such extreme points as important peaks during the calculation process. After filling, the array dimension has not changed, but the data in the middle has changed, and the importance of the peak can be better analyzed.
[0032] In step 203, the first extreme point and the second extreme point are interpolated using a cubic spline difference function to obtain all extreme points corresponding to the first edge data. Other interpolation functions can also be used for interpolation. In step 204, all the maximum points in all the extreme points corresponding to the first edge data are sequentially connected to form an upper envelope, and all the minimum points in all the extreme points corresponding to the first edge data are sequentially connected to form a lower envelope. In step 205, the mean envelope is calculated based on the upper envelope and the lower envelope. In step 206, the intrinsic mode function corresponding to the first edge data is obtained based on the first edge data and the mean envelope.
[0033] In some embodiments, extracting the extreme point of the first edge data includes: using the argrelextrema function to extract each local extreme point corresponding to each window data; using the peak_prominences function to determine the significance value of each local extreme point, and taking the local extreme point whose significance value is greater than the first threshold as the corresponding extreme point of the first edge data. The significance value represents the vertical distance between the extreme point and the lowest valley bottom adjacent to it, and can be used to measure the prominence of the extreme point. The peak_prominences function accepts the first array matrix and an extreme point as input, and then outputs the significance value of the extreme point.
[0034] In the step of picking extreme points, the argrelextrema function and the peak_prominences function in the scientific computing library numpy scipy are used for data processing instead of additional abstraction layers or checks. The code is refactored to be more efficient and avoid unnecessary loops and conditional checks that may significantly affect performance.
[0035] In some embodiments, the processing method further includes: the modal decomposition unit publishes different modal data sequences to the MQTT (Message Queuing Telemetry Transport) deployed on the central server, and forwards them to the subscribers of the edge data of the corresponding modality through MQTT. Since there are many edge monitoring devices and each edge monitoring device generates a large amount of data every day, the data is forwarded to the subscribers of the edge data through the MQTT located on the central server to avoid the accumulation of a large amount of data.
[0036] In some embodiments, the processing method further includes: when the central server receives edge data sent by the edge and determines that the edge data does not need to be modally decomposed, directly publishing the edge data to the MQTT deployed on the central server, and forwarding it to the subscriber of the edge data through MQTT. The subscriber here can be a receiving end that needs to correspond to the edge monitoring device data, so that the receiving end can receive the data in real time and perform corresponding analysis.
[0037] In this way, edge data that does not require modal decomposition is published to MQTT and then directly forwarded to subscribers. Edge data that requires modal analysis is published to MQTT after an asynchronous modal decomposition process. This improves the speed of data processing and the data processing performance at the edge.
[0038] In some embodiments, the first time range is set in association with the transmission frequency of the edge data of the first edge monitoring device. The different transmission frequencies of the first edge monitoring devices result in different amounts of data received by the central server per unit time. When the data flow rate is faster, that is, the transmission frequency is higher, the window size can be appropriately increased to reduce the frequency of processing, for example, it can be 5 hours, etc. When the data flow rate slows down, that is, the transmission frequency is lower, the window size can be reduced, for example, it can be 5 seconds, etc. The first time range can be set according to the frequency of edge data generation, so that the first array matrix includes an appropriate amount of data, and the processing granularity is flexibly adjusted according to the changes in the data. Through dynamic adjustment, the real-time performance of data acquisition is improved.
[0039] In some embodiments, the range of the sliding window can be set in association with the transmission frequency of the edge data of the first edge monitoring device. When the transmission frequency is high, the range of the sliding window can be increased, and when the transmission frequency is low, the range of the sliding window can be reduced. The size of the sliding window can be 5*5, 3*3, 7*7, etc. The range of the sliding window can also be set according to the requirements of the local features to be extracted. In order to extract a suitable amount of data from the first array matrix, the local features of the data can also be extracted, which is conducive to extracting the intrinsic modal function of the edge data.
[0040] In some embodiments, based on the first array matrix, the preset step length of the sliding window movement is one data point in the first array matrix. The moving step length of the sliding window cannot be too large, otherwise some data may be omitted, which is not conducive to extracting local features. The sliding window moves one data point each time to avoid omitting data, which is conducive to extracting local features.
[0041] According to an embodiment of the present application, a system for processing edge data is also provided. Figure 3A structural schematic diagram of the edge data processing system according to an embodiment of the present application is shown. The processing system 300 includes at least one edge monitoring device 301 and a central server 302, and the edge monitoring device 301 sends real-time edge data to the central server 302; the central server 302 executes the edge data processing method described in any embodiment of the present application. The edge data processing system of the present application receives data through a processing process and stores it in a temporary queue, which can improve the data reception performance. The algorithm can be calculated in parallel by sliding windows, and the reading and writing speed of data can be accelerated by using array matrices. The improved iteration stop condition is used to meet the adaptability of the algorithm, and efficient data processing can be achieved. The data receiving speed and the data processing speed have been improved accordingly. The data processing speed keeps up with the data receiving speed, avoiding the accumulation of edge data in the central server, which is conducive to the efficient processing of edge data.
[0042] There is no specific limit to the number of edge monitoring devices 301, which may be single or multiple.
[0043] According to an embodiment of the present application, a non-temporary computer-readable medium is also provided, on which instructions are stored. When executed by a processor, the instructions execute the steps of the edge data processing method as described in any embodiment of the present application. The edge data processing method of the present application receives data through a processing process and stores it in a temporary queue, which can improve the data reception performance. The algorithm can be calculated in parallel by sliding windows, and the reading and writing speed of data can be accelerated by using array matrices. The use of improved iterative stop conditions can meet the adaptability of the algorithm and achieve efficient data processing. The data receiving speed and the data processing speed have been improved accordingly. The data processing speed keeps up with the data receiving speed, avoiding the accumulation of edge data on the central server, which is conducive to the efficient processing of edge data.
[0044] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. The elements in the claims are to be interpreted broadly based on the language adopted in the claims, and are not limited to the examples described in this specification or during the practice of this application, and the examples are to be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.
[0045] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more schemes thereof) may be used in combination with each other. For example, a person of ordinary skill in the art may use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features may be grouped together to simplify the present application. This should not be interpreted as an intention that a feature of an application that does not require protection is necessary for any claim. On the contrary, the subject matter of the present application may be less than all the features of an embodiment of a particular application. Thus, the following claims are incorporated herein in the specific embodiments as examples or embodiments, wherein each claim is independently a separate embodiment, and it is considered that these embodiments may be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the appended claims and the full scope of equivalent forms granted by these claims.
[0046] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.
Claims
1. A method for processing edge data, characterized in that: The edge data is real-time data from at least one monitoring device at the edge, and the processing method includes: The central server receives edge data from multiple edge monitoring devices using a processing process, and stores the received edge data in a temporary storage queue in chronological order; When it is determined that the edge data in the temporary storage queue needs to be modally decomposed, the edge data is stored in a modal analysis data queue; The modal decomposition unit deployed in the central server sequentially performs modal decomposition on the edge data in the modal analysis data queue, wherein processing the edge data of the first edge monitoring device includes executing the following steps: Merging first edge data of a first edge monitoring device within a first time range in the modal analysis data queue into a first array matrix; In the first array matrix, first edge data within the range of the second matrix is extracted using a sliding window, intrinsic mode function is identified in the extracted first edge data, and parameters of the identified intrinsic mode function are iteratively updated after the sliding window is moved according to a preset step size until the end of the first array matrix is reached and / or the difference between two adjacent parameter iterations is less than a preset threshold; Merging the data belonging to the same intrinsic modal function in the first array matrix into a corresponding modal data sequence; Identifying the intrinsic mode function in the extracted first edge data specifically includes: Extracting first extreme value points of the first edge data, wherein the extreme value points include maximum value points and minimum value points; When the difference between adjacent maximum value points or minimum value points is greater than a first threshold, the extreme value points of the first edge end data are reflectively filled using the pad_extrema_rilling function to obtain the filled second extreme value points corresponding to the first edge end data; Using a cubic spline difference function to interpolate the first extreme point and the second extreme point to obtain all extreme points corresponding to the first edge end data; All the maximum value points among all the extreme value points corresponding to the first edge end data are sequentially connected to form an upper envelope, and all the minimum value points among all the extreme value points corresponding to the first edge end data are sequentially connected to form a lower envelope; Calculate the mean envelope based on the upper envelope and the lower envelope; An intrinsic mode function corresponding to the first edge data is obtained based on the first edge data and the mean envelope.
2. The processing method according to claim 1, characterized in that: When there is an updated edge monitoring device at the edge, the central server sends the URL corresponding to the central server to the updated edge monitoring device; after receiving the URL corresponding to the central server, the updated edge monitoring device uses the http protocol to send the edge data to the URL corresponding to the central server.
3. The processing method according to claim 1, characterized in that: Extracting the extreme points of the first edge data includes: Use the argrelextrema function to extract the local extreme points corresponding to each window data; The peak_prominences function is used to determine the significance value of each local extreme point, and the local extreme point whose significance value is greater than the first threshold is used as the extreme point of the corresponding first edge end data.
4. The processing method according to claim 1, characterized in that: The processing method further comprises: The modality decomposition unit publishes different modality data sequences to the MQTT deployed on the central server respectively, and forwards them to subscribers of the edge data of the corresponding modality through MQTT.
5. The processing method according to claim 1, characterized in that: The processing method further comprises: When the central server receives edge data sent by the edge monitoring device and determines that the edge data does not need to be modally decomposed, the edge data is directly published to the MQTT deployed on the central server and forwarded to the subscribers of the edge data through MQTT.
6. The processing method according to claim 1, characterized in that: The first time range is set in association with a frequency of sending edge data of the first edge monitoring device.
7. The processing method according to claim 1, characterized in that: Based on the first array matrix, the preset step size of the sliding window movement is a data point in the first array matrix.
8. A system for processing edge data, characterized in that: The processing system includes at least one edge monitoring device and a central server, the edge monitoring device sends real-time edge data to the central server; the central server executes the edge data processing method described in any one of claims 1-7.
9. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, execute the edge data processing method as described in any one of claims 1-7.
