Real-time data processing method and system based on frequency layering

Abstract

The invention discloses a real-time data processing method and system based on frequency layering, and belongs to the technical field of computers, and the method comprises the steps: obtaining a layering configuration rule and a preset processing period of a data table according to attribute frequency; acquiring a corresponding change attribute according to the change of the real-time data, and acquiring a change plane and change time to which the change attribute belongs according to a hierarchical configuration rule; if the change attribute belongs to a frequent change plane in the hierarchical configuration rule, entering a frequent change increment queue and processing the change attribute and the change time according to the change time and a first period in a preset processing period, and if the change attribute belongs to a non-frequent change plane in the hierarchical configuration rule, entering a non-frequent change increment queue and processing the change attribute and the change time according to the non-frequent change plane in the hierarchical configuration rule; and processing the change attribute and the change time according to the change time and a second period in the preset processing period. The technical problem of low data processing efficiency caused by the fact that high-frequency data items and low-frequency data are not layered and distinguished in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a real-time data processing method and system based on frequency stratification. Background Technology

[0002] To ensure data reliability and consistency, existing SCADA real-time databases generally use a uniform granularity of persisting and synchronizing entire tables or rows during data processing. However, this approach does not differentiate data items according to their update frequency, meaning it fails to separate high-frequency real-time items from low-frequency parameter items. This indiscriminate processing leads to a chain of problems: when high-frequency items change, the system writes the entire row of data, including unchanged low-frequency parameter items, to the database, resulting in storage I / O being occupied by a large amount of redundant data. This not only increases the write pressure on the storage medium but also causes data redundancy due to repeated storage of static parameters. During cross-regional synchronization, transmitting entire rows causes low-frequency parameter items to crowd out the limited industrial network bandwidth. When combined with data from a large number of devices, this can easily lead to bandwidth saturation, affecting the transmission efficiency of core real-time data and failing to meet the demands of modern industrial scenarios for refined and efficient data processing. Summary of the Invention

[0003] To address the technical problem of low data processing efficiency caused by the lack of stratification between high-frequency and low-frequency data in existing technologies, this invention provides a real-time data processing method and system based on frequency stratification. The method obtains the stratification configuration rules of the data table based on the attribute frequency, acquires the corresponding changing attributes based on the changes in real-time data, obtains the change plane and change time of the changing attributes according to the stratification configuration rules, and performs targeted data processing on the changing attributes and their change times based on the plane to which the changing attributes belong. This solves the technical problem of low data processing efficiency caused by the lack of stratification between high-frequency and low-frequency data in existing technologies, significantly improving data processing efficiency.

[0004] To address the aforementioned technical problems, this invention provides a real-time data processing method based on frequency stratification, comprising the following steps: The hierarchical configuration rules and preset processing cycles of the data table are obtained based on the frequency of attributes in the data table. Obtain the corresponding change attributes based on the changes in real-time data, and obtain the change plane and change time of the change attributes according to the hierarchical configuration rules; If the changed attribute belongs to the frequent change plane in the hierarchical configuration rules, it enters the frequent change increment queue and is processed according to the change time of the changed attribute and the first cycle in the preset processing cycle. If the changed attribute belongs to the infrequent change plane in the hierarchical configuration rules, it enters the infrequent change increment queue and is processed according to the change time of the changed attribute and the second cycle in the preset processing cycle.

[0005] Preferably, the hierarchical configuration rule includes: if the attribute frequency is greater than the preset frequency, the attribute is assigned to the frequent change plane; otherwise, the attribute is assigned to the infrequent change plane. The preset processing cycle includes: if the attribute frequency is greater than the preset frequency, the preset processing cycle is a millisecond-level processing cycle; otherwise, the preset processing cycle is a second-level processing cycle.

[0006] Preferably, obtaining the change plane and change time of the change attribute according to the hierarchical configuration rules includes: The change attribute is compared with the allocation configuration rules to obtain the change plane to which the change attribute belongs. If the change plane to which it belongs is frequent... If the plane to which the change occurs consists of a change time in seconds and a change time in milliseconds, then the change time is in seconds if the plane to which the change occurs is a plane with infrequent changes.

[0007] Preferably, the step of obtaining the change plane and change time of the change attribute according to the hierarchical configuration rules further includes: If the change attribute has millisecond precision, the change time consists of a second-level change time and a millisecond-level change time; otherwise, the change time is a second-level change time.

[0008] Preferably, the step of entering the frequent change increment queue if the change attribute belongs to the frequent change plane in the hierarchical configuration rules further includes: using the change time of historical data as the enqueue threshold; if the change time of the change attribute is greater than the enqueue threshold, then it enters the frequent change increment queue. The statement that if a changed attribute belongs to the infrequent change plane in the hierarchical configuration rules, it will enter the infrequent change increment queue also includes: if the change time of the changed attribute is greater than the enqueue threshold, it will enter the infrequent change increment queue.

[0009] Preferably, the step of processing the change attribute and the change time of the change attribute according to the change time of the change attribute and the first period in the preset processing period includes: The change time of the changed attribute is compared with the first period to determine whether the changed attribute should be processed. If the comparison with the local high-frequency storage period in the first period is successful, the changed attribute and the change time of the changed attribute are stored in the local high-frequency database according to the local high-frequency storage period. If the comparison with the remote synchronization period in the first period is successful, the changed attribute and the change time of the changed attribute are synchronized to the remote node database according to the remote synchronization period. Otherwise, the changed attribute is not processed.

[0010] Preferably, the step of processing the change attribute and the change time of the change attribute according to the change time of the change attribute and the second period in the preset processing period includes: The change time of the changed attribute is compared with the second period to determine whether the changed attribute should be processed. If the comparison with the second period is successful, the attribute change and the change time of the attribute change are stored in the local high-frequency database according to the second period and synchronized to the remote node database. Otherwise, the changed attribute is not processed.

[0011] By adopting the above technical solution, the present invention has the following advantages: The hierarchical configuration rules of the data table are obtained based on the frequency of attributes in the data table. The corresponding changing attributes are obtained based on the changes in real-time data. The change plane and change time of the changing attributes are obtained based on the hierarchical configuration rules. The changing attributes and change time of the changing attributes are processed in a targeted manner based on the plane to which the changing attributes belong. This solves the technical problem of low data processing efficiency caused by the failure of existing technologies to distinguish between high-frequency data items and low-frequency data, and significantly improves the efficiency of data processing. By increasing the time precision of attributes requiring millisecond-level accuracy from seconds to milliseconds, the system accurately marks the moment of change for a single attribute. Furthermore, from change capture and hierarchical judgment to entry into the incremental queue and periodic processing, each step carries time precision to ensure that time information is not lost. This allows the system to reconstruct the order of changes of different attributes within the same second through time precision, providing accurate data for fault tracing in various scenarios.

[0012] This invention also provides a frequency-stratified real-time data processing system, applicable to the aforementioned frequency-stratified real-time data processing method, comprising: The configuration module is used to obtain the hierarchical configuration rules and preset processing cycles of the data table based on the frequency of attributes in the data table; The change capture module is used to obtain the corresponding change attributes based on changes in real-time data, and to obtain the change plane and change time of the change attributes according to the hierarchical configuration rules. The processing module is used to determine the affiliation of the changing attribute. When the changing attribute is determined to belong to the frequent change plane in the hierarchical configuration rules, it enters the frequent change increment queue and processes the changing attribute and its change time according to the change time of the changing attribute and the first cycle in the preset processing cycle. When the changing attribute is determined to belong to the infrequent change plane in the hierarchical configuration rules, it enters the infrequent change increment queue and processes the changing attribute and its change time according to the change time of the changing attribute and the second cycle in the preset processing cycle.

[0013] By adopting the above technical solution, the present invention has the following advantages: The hierarchical configuration rules of the data table are obtained based on the frequency of attributes in the data table. The corresponding changing attributes are obtained based on the changes in real-time data. The change plane and change time of the changing attributes are obtained according to the hierarchical configuration rules. The changing attributes and change time of the changing attributes are processed in a targeted manner according to the plane to which the changing attributes belong. This solves the technical problem of low data processing efficiency caused by the failure of existing technologies to distinguish between high-frequency data items and low-frequency data, and significantly improves the efficiency of data processing.

[0014] The present invention also provides a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the frequency-stratified real-time data processing method.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the frequency-stratified real-time data processing method. Attached Figure Description

[0016] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0017] Figure 1 This is a flowchart illustrating the frequency-stratified real-time data processing method of the present invention. Figure 2 This is a timing diagram of the Par and Frq planes being written to the HDB / RTDB in the frequency-stratified real-time data processing method of this invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only one preferred embodiment of this invention and are only used to explain this invention. They do not limit the scope of protection of this invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0019] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations (or steps) can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures; the process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0020] Example 1: like Figure 1 As shown, the real-time data processing method based on frequency stratification includes the following steps: S1: Obtain the hierarchical configuration rules and preset processing cycle of the data table based on the attribute frequency in the data table.

[0021] Specifically, the hierarchical configuration rules include: if the frequency of an attribute is greater than a preset frequency, the attribute is assigned to the frequent change plane; otherwise, the attribute is assigned to the infrequent change plane. The preset processing cycle includes: if the attribute frequency is greater than the preset frequency, the preset processing cycle is a millisecond-level processing cycle; otherwise, the preset processing cycle is a second-level processing cycle.

[0022] S2: Obtain the corresponding change attributes based on the changes in real-time data, and obtain the change plane and change time of the change attributes according to the hierarchical configuration rules.

[0023] In some embodiments, obtaining the change plane and change time of the change attribute according to the hierarchical configuration rules includes: The change attribute is compared with the allocation configuration rules to obtain the change plane to which the change attribute belongs. If the change plane to which it belongs is frequent... If the plane to which the change occurs consists of a change time in seconds and a change time in milliseconds, then the change time is in seconds if the plane to which the change occurs is a plane with infrequent changes.

[0024] A list of high-frequency attributes is obtained through hierarchical configuration rules. This list is loaded from `b_frqproplist` to generate `FrqProps`. Attributes not included in the frequent change plane `Frq` are assigned to the infrequent change plane `Par`. In this embodiment, a `Time_MS` column (0~999) with the same name (+_MS) is added to millisecond-precision time attributes to record millisecond values. Millisecond-level time changes are the recorded millisecond values, and second-level time changes are the recorded second values. Second-level and millisecond-level changes together describe the same point in time. When writing or refreshing the second-level column corresponding to a second-level change, the corresponding millisecond-level column `*_MS` is maintained synchronously to ensure strict alignment between second-level and millisecond-level changes. Users can flexibly set and adjust the preset frequency according to their needs.

[0025] In another embodiment, obtaining the change plane and change time of the change attribute according to the hierarchical configuration rules further includes: If the change attribute has millisecond precision, the change time consists of a second-level change time and a millisecond-level change time; otherwise, the change time is a second-level change time.

[0026] S3: If the changed attribute belongs to the frequent change plane in the hierarchical configuration rules, it enters the frequent change increment queue and is processed according to the change time of the changed attribute and the first cycle in the preset processing cycle. If the changed attribute belongs to the infrequent change plane in the hierarchical configuration rules, it enters the infrequent change increment queue and is processed according to the change time of the changed attribute and the second cycle in the preset processing cycle.

[0027] In some embodiments, the step of entering the frequent change increment queue if the changed attribute belongs to the frequent change plane in the hierarchical configuration rules further includes: using the change time of historical data as an enqueue threshold, and if the change time of the changed attribute is greater than the enqueue threshold, then entering the frequent change increment queue. The statement that if a changed attribute belongs to the infrequent change plane in the hierarchical configuration rules, it will enter the infrequent change increment queue also includes: if the change time of the changed attribute is greater than the enqueue threshold, it will enter the infrequent change increment queue.

[0028] By introducing historical data change times as the enqueue threshold, data is only enqueued when the current change time is later than the previous processing time. This avoids redundant processing of the same data multiple times, and prevents invalid enqueueing and duplicate calculations caused by repeated changes within a short period. It allows computing resources to focus on genuine incremental changes, thereby improving data processing efficiency.

[0029] The process of processing the change attribute and its change time based on the change time of the change attribute and the first period in the preset processing period includes: The change time of the changed attribute is compared with the first period to determine whether the changed attribute should be processed. If the comparison with the local high-frequency storage period in the first period is successful, the changed attribute and the change time of the changed attribute are stored in the local high-frequency database according to the local high-frequency storage period. If the comparison with the remote synchronization period in the first period is successful, the changed attribute and the change time of the changed attribute are synchronized to the remote node database according to the remote synchronization period. Otherwise, the changed attribute is not processed.

[0030] The process of processing the change attribute and its change time based on the change time of the change attribute and the second period in the preset processing cycle includes: The change time of the changed attribute is compared with the second period to determine whether the changed attribute should be processed. If the comparison with the second period is successful, the attribute change and the change time of the attribute change are stored in the local high-frequency database according to the second period and synchronized to the remote node database. Otherwise, the changed attribute is not processed.

[0031] In this embodiment, the second period is longer than the first period. The change time of the changing attribute is compared with the period to determine whether the changing attribute should be processed. When the change time is transmitted in pairs in seconds + milliseconds, the consumer side can reconstruct a unified 1ms precision time axis to ensure millisecond co-tracking across links. For attributes belonging to the frequently changing plane, their change frequency is extremely high, possibly occurring multiple times per second. If each change triggers an independent process, such as immediately writing to the database or synchronizing to a remote node, it will generate a large number of IO operations and network requests, instantly exhausting system resources (such as database connection pools and bandwidth), leading to processing delays or even service unavailability. Therefore, by introducing the first and second periods respectively, all changes within this period can be accumulated and processed in batches, significantly reducing the number of processing operations and resource call frequency.

[0032] In typical projects, frequently changing attributes account for about 20% but contribute more than 90% of the total. This invention persists / synchronizes frequently changing attributes separately, reducing network and storage write amplification by 50% to 90%. It also avoids attribute-by-attribute comparisons, reducing CPU overhead. In addition, accompanying columns achieve millisecond alignment, improving event / history tracing accuracy to 1ms.

[0033] In CRdbTable (the core relational database table), FrqProps (a set of frequently changing attributes) and ParProps (a set of infrequently changing attributes) are maintained. UpdateTimeProp (a row-level unified change time field) is used to unify row-level change times. The set of frequently changing attributes is declared by b_frqproplist. When an attribute changes, the row-level UpdateTime and the necessary *_MS accompanying columns are updated. Based on the table configuration HDBTableName (the name of the high-frequency database table) / HDBFrqTableName (the name of the infrequently changing database table) and HDBFrqMS (the millisecond-level synchronization parameters for high-frequency storage) / RDBFrqMS (the high-frequency synchronization parameters for relational databases), the Par and Frq planes are respectively inserted into and synchronized. The client / replica node pulls incremental data according to the plane. The time sequence diagram of Par and Frq planes being written to HDB / RTDB is as follows. Figure 2 As shown, the incremental synchronization steps for the frequently changing plane (Frq) are as follows: The client / replica initiates a request for incremental data (Frq) to the local service, specifying that the latest data for frequently changing attributes needs to be retrieved. The local service calls the interface GetSectionData(Frq, sections) to request the specified segment data of the frequently changing plane (Frq) from the original / historical database. The original / historical database returns the Frq incremental data to the local service, which then returns the Frq incremental data to the client / replica. The client / replica executes the application of the Frq increment to complete the update of the frequently changing attributes. The incremental synchronization steps for the infrequently changing plane (Par) are basically the same as those for the frequently changing plane (Frq). In this embodiment, a persistence module is used to write the incremental data into the HDB parameter history table (storing the historical change records of attributes) and the HDB real-time item history table (storing the real-time update status of attributes), and a synchronization module is used to synchronize the incremental data to the remote / replica nodes.

[0034] Example 2: This embodiment also provides a frequency-stratified real-time data processing system, applicable to the frequency-stratified real-time data processing method, including: The configuration module is used to obtain the hierarchical configuration rules and preset processing cycles of the data table based on the frequency of attributes in the data table; The change capture module is used to obtain the corresponding change attributes based on changes in real-time data, and to obtain the change plane and change time of the change attributes according to the hierarchical configuration rules. The processing module is used to determine the affiliation of the changing attribute. When the changing attribute is determined to belong to the frequent change plane in the hierarchical configuration rules, it enters the frequent change increment queue and processes the changing attribute and its change time according to the change time of the changing attribute and the first cycle in the preset processing cycle. When the changing attribute is determined to belong to the infrequent change plane in the hierarchical configuration rules, it enters the infrequent change increment queue and processes the changing attribute and its change time according to the change time of the changing attribute and the second cycle in the preset processing cycle.

[0035] The changes in the attribute and their change time are processed according to the first cycle of the preset processing cycle. This includes: comparing the change time of the attribute with the first cycle to determine whether to process the attribute; if the comparison is successful with the local high-frequency storage cycle in the first cycle, the attribute and its change time are stored in the local high-frequency database according to the local high-frequency storage cycle; if the comparison is successful with the remote synchronization cycle in the first cycle, the attribute and its change time are synchronized to the remote node database according to the remote synchronization cycle; otherwise, the attribute is not processed. The changes in the attribute and their change time are also processed according to the second cycle of the preset processing cycle. This includes: comparing the change time of the attribute with the second cycle to determine whether to process the attribute; if the comparison is successful with the second cycle, the attribute change and its change time are stored in the local high-frequency database and synchronized to the remote node database according to the second cycle; otherwise, the attribute is not processed.

[0036] Example 3: This embodiment also provides a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the frequency-stratified real-time data processing method.

[0037] Example 4: This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the frequency-stratified real-time data processing method.

[0038] The specific embodiments described above are preferred embodiments of the real-time data processing method and system based on frequency stratification of the present invention, and are not intended to limit the specific scope of the present invention. The scope of the present invention includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. A method for real-time data processing based on frequency stratification, characterized in that, The method comprises the following steps: According to the attribute frequency in the data table, the hierarchical configuration rule of the data table and the preset processing period are obtained; According to the change of real-time data, the corresponding change attribute is obtained, and the change plane and change time of the change attribute are obtained according to the hierarchical configuration rule; If the change attribute belongs to the frequent change plane in the hierarchical configuration rule, the change attribute enters the frequent change incremental queue, and the change attribute and the change time of the change attribute are processed according to the change time of the change attribute and the first period in the preset processing period; if the change attribute belongs to the non-frequent change plane in the hierarchical configuration rule, the change attribute enters the non-frequent change incremental queue, and the change attribute and the change time of the change attribute are processed according to the change time of the change attribute and the second period in the preset processing period.

2. The frequency hierarchy-based real-time data processing method of claim 1, wherein, The hierarchical configuration rule comprises: if the attribute frequency is greater than the preset frequency, the attribute belongs to the frequent change plane, otherwise the attribute belongs to the non-frequent change plane; The preset processing period comprises: if the attribute frequency is greater than the preset frequency, the preset processing period is a millisecond-level processing period, otherwise the preset processing period is a second-level processing period.

3. The frequency hierarchy-based real-time data processing method of claim 1, wherein, According to the hierarchical configuration rule, the change plane and the change time of the change attribute are obtained, comprising: The change attribute is compared with the allocation configuration rule to obtain the change plane of the change attribute, if the change plane is the frequent change plane, the change time is composed of the second-level change time and the millisecond-level change time, if the change plane is the non-frequent change plane, the change time is the second-level change time. According to the hierarchical configuration rule, the change plane and the change time of the change attribute are obtained, further comprising:

4. The frequency hierarchy-based real-time data processing method of claim 1, wherein, If the change attribute is of millisecond-level precision, the change time is composed of the second-level change time and the millisecond-level change time, otherwise the change time is the second-level change time. If the change attribute belongs to the frequent change plane in the hierarchical configuration rule, the change attribute enters the frequent change incremental queue, further comprising: the change time of the historical data is used as the entry threshold, if the change time of the change attribute is greater than the entry threshold, the change attribute enters the frequent change incremental queue; 5. The frequency hierarchy-based real-time data processing method of claim 1, wherein, If the change attribute belongs to the non-frequent change plane in the hierarchical configuration rule, the change attribute enters the non-frequent change incremental queue, further comprising: if the change time of the change attribute is greater than the entry threshold, the change attribute enters the non-frequent change incremental queue. According to the change time of the change attribute and the first period in the preset processing period, the change attribute and the change time of the change attribute are processed, comprising:

6. The frequency hierarchy-based real-time data processing method of claim 1, wherein, The change time of the change attribute is compared with the first period to determine whether the change attribute is processed, if the comparison with the local high-frequency storage period in the first period is successful, the change attribute and the change time of the change attribute are stored in the local high-frequency database according to the local high-frequency storage period, if the comparison with the remote synchronization period in the first period is successful, the change attribute and the change time of the change attribute are synchronized to the remote node database according to the remote synchronization period, otherwise the change attribute is not processed. According to the change time of the change attribute and the second period in the preset processing period, the change attribute and the change time of the change attribute are processed, comprising:

7. The frequency hierarchy-based real-time data processing method of claim 1, wherein, ​ The change time of the change attribute is compared with the second period to determine whether to process the change attribute, if the comparison with the second period is successful, the attribute change and the change time of the attribute change are stored in the local high-frequency database and synchronized to the remote node database according to the second period, otherwise, the change attribute is not processed.

8. A real-time data processing system based on frequency stratification, suitable for the real-time data processing method based on frequency stratification according to any one of claims 1 to 7, characterized in that, The method comprises the steps of: The configuration module is configured to obtain the hierarchical configuration rule and the preset processing period of the data table according to the attribute frequency in the data table; The change capture module is configured to obtain the corresponding change attribute according to the change of the real-time data, and obtain the change plane and the change time of the change attribute according to the hierarchical configuration rule; The processing module is configured to determine the ownership of the change attribute, when the change attribute belongs to the frequent change plane in the hierarchical configuration rule, the change attribute is entered into the frequent change incremental queue, and the change attribute and the change time of the change attribute are processed according to the change time of the change attribute and the first period in the preset processing period, when the change attribute belongs to the non-frequent change plane in the hierarchical configuration rule, the change attribute is entered into the non-frequent change incremental queue, and the change attribute and the change time of the change attribute are processed according to the change time of the change attribute and the second period in the preset processing period.

9. A computer device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the frequency hierarchical based real-time data processing method according to any one of claims 1-7. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the frequency hierarchical based real-time data processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: ​

Images