Massive data processing query method and system for power visualization monitoring

Abstract

This invention belongs to the field of data processing and querying, and provides a method and system for processing and querying massive amounts of data for power visualization monitoring. It includes building a unified basic environment for local and remote data centers, constructing a data synchronization queue and a user business query habit queue; storing changed data simultaneously in both the local and remote data centers, and storing it in the corresponding database according to data type; the remote data center periodically queries data based on historical user query logs to determine the relationship between user query hotspots and corresponding data time-series nanosecond IDs; based on the determined query hotspots and query hotspot-nanosecond index IDs, obtaining the corresponding underlying database nanosecond IDs, and determining the query data based on the underlying database nanosecond IDs. A multi-dimensional index based on user query habits is pre-generated and reverse-synchronized to the user-side server database storage. The application uses the generated business-frequently used multi-dimensional indexes built on the underlying data indexes, facilitating rapid response to user queries.

Description

Technical Field

[0001] This invention belongs to the field of data processing and query technology, specifically relating to a method and system for processing and querying massive amounts of data for power visualization monitoring. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Data for power transmission visualization services originates from IoT data sources such as terminal monitoring and surveillance equipment, user usage data generated by the user side, and algorithm result data, resulting in a diverse range of data types and significant incremental growth. Given this massive amount of data, the stability of the underlying data for software applications is paramount. Currently, the commonly used methods include regular data backups or targeted disk mirroring backups of servers, storing the data in off-site data centers. This approach has played a certain role in data protection.

[0004] Furthermore, as applications accumulate data volumes reaching hundreds of millions, it is common for such large tables to have built-in indexes to improve query efficiency.

[0005] The aforementioned methods of periodically backing up data or mirroring servers may result in data loss from the time of the last backup in the event of a server outage. Furthermore, off-site data centers only serve the purpose of data backup or mirroring, not fully utilizing their capabilities, and scheduled backups suffer from latency in data synchronization. If data transfer is not timely after a server outage, data integrity will be low, impacting normal software operation.

[0006] Even if indexes on hundreds of millions of data points improve query efficiency to some extent, the effect is not significant. Furthermore, when the data table is updated or attributes are added, the index update time is long, which is not conducive to data maintenance. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a method and system for processing and querying massive amounts of data for power visualization monitoring. This invention simultaneously solves the problems mentioned above and addresses the inefficiency of massive data queries. Data backup is no longer scheduled, and it fully utilizes data from remote data centers to provide multi-dimensional indexes based on user query habits for efficient queries. These indexes are pre-generated and then reverse-synchronized to the user-side server database storage. The software application uses the generated multi-dimensional indexes commonly used in business operations, built on top of the underlying data indexes. The multi-level indexes reduce the query scope of large tables, facilitating rapid response for user queries.

[0008] According to some embodiments, the first solution of the present invention provides a method for processing and querying massive amounts of data for power visualization monitoring, which adopts the following technical solution:

[0009] Massive data processing and querying methods for power visualization monitoring include:

[0010] A unified basic environment is built for local and remote data centers, and a data synchronization queue and a user business query habit queue are constructed.

[0011] Real-time power visualization data is collected and simultaneously stored in local and remote data centers, and then stored in the corresponding database according to the data type.

[0012] The remote data center periodically queries data based on historical user query logs to determine the relationship between user query hot words and corresponding data time-series nanosecond IDs, forming query hot word-nanosecond index IDs and sending them to the local data center for index synchronization.

[0013] Obtain the user's query instruction and determine the query hot words based on the user's query instruction; based on the determined query hot words and query hot words-nanosecond index ID, obtain the corresponding underlying database nanosecond ID, and determine the query data based on the underlying database nanosecond ID.

[0014] Furthermore, the unified basic environment setup for local and remote data centers, and the construction of data synchronization queues and user business query habit queues, include:

[0015] Set up the same version of database, backup data transmission queue and backup data consumption strategy for local data center and remote data center;

[0016] Establish the corresponding data synchronization queue and its agreed data format;

[0017] Establish a corresponding queue of user business query habits and its agreed data format.

[0018] Furthermore, the keywords of the data synchronization queue are DataBaseType (database type), DataType (data type), and Data (specific data);

[0019] The keywords for the user business query habit queue are BusinessType (business function module), QueryKeyWords (query keywords), and DATASHEET (data table).

[0020] Furthermore, real-time power visualization data is collected and simultaneously stored in both local and remote data centers, and then stored in the corresponding databases according to data type, including:

[0021] Establish a business function module dictionary, which includes the function types of all power visualization data within the application;

[0022] The local data center stores the power visualization data into the corresponding database according to the data type. When data entry fails, the local data center removes the failed data and sends the remaining data to the Mq data synchronization queue in the remote data center.

[0023] The remote data center also stores the power visualization data in the remote data center's Mq data synchronization queue into the corresponding database according to the data type.

[0024] Furthermore, the local data center stores the power visualization data in the corresponding database according to the data type. When data import fails, the local data center removes the failed data and sends the remaining data to the Mq data synchronization queue in the remote data center, including:

[0025] For relational data changes, the local data center uses Mybatis, the persistence layer framework corresponding to the relational database, to intercept the relational data changes, extract the relational SQL statements of the relational data changes, form JSON format relational data changes corresponding to the business function module dictionary, and store them in the local data center's Mq temporary storage queue.

[0026] For changes to non-relational data, Spring-AOP is used to encapsulate special annotations to collect non-relational change data, integrate and standardize it into JSON format, and store it in the local data center's MQ temporary storage queue;

[0027] If data import fails, the same data is combined and sent to the local data center's Mq temporary storage queue and marked as -1.

[0028] The application-side unified data processing program consumes the local data center's Mq temporary storage queue. If there is overlapping data and the data is marked as -1, the data entry fails, and the remaining data in the local data center's Mq temporary storage queue is sent to the remote data center's Mq data synchronization queue.

[0029] Furthermore, the remote data center periodically queries historical user query logs to determine the relationship between user query hot words and corresponding data time-series nanosecond IDs, forming query hot word-nanosecond index IDs, which are then sent to the local data center for index synchronization, including:

[0030] The local data center periodically collects user query logs and stores them in a local Mq cache queue. The local data center then sends the user query logs stored in the local Mq cache queue to the Mq user business query habit queue in the remote data center.

[0031] The remote data center consumption Mq user business query habit queue aggregates user query data and stores it in the remote data center database;

[0032] When the database activity in the remote data center decreases daily, user query data is collected. Based on the business function module, sub-function, and corresponding dictionary item, the number of queries for the function module is counted, and the query conditions appearing under the function module are counted. If it is a multi-condition query, the combined conditions are treated as one item, and the number of queries for the function module is counted.

[0033] The query is sorted in reverse order based on the number of queries for each functional module, and the corresponding data in the database is queried using the underlying index of the database based on priority and time range, while determining the time-series nanosecond ID corresponding to the data.

[0034] Based on the number of queries for the functional modules and the corresponding time-series nanosecond IDs of the data, a query hot word-nanosecond index ID is formed;

[0035] The remote data center reorders the data based on the query hot keywords - nanosecond index ID, and combines the historical summary results. Finally, according to the sorting results, the data is forwarded to the local data center application-side Mq temporary storage queue for index synchronization.

[0036] Furthermore, the method also includes:

[0037] When the application is running normally, the local data center and the remote data center synchronize operation data, and the data in both data centers is consistent; when the local data center goes down, the monitoring center immediately switches the upper-layer gateway and forwards user requests to the remote data center; the operation and maintenance personnel restore the local data center server;

[0038] During the recovery of the local data center server, the data generated by the remote data center is temporarily stored in the local Mq cache queue of the remote data center; when the local data center server recovers, the cached data continues to be distributed, and at the same time, the remote data center will synchronize the data in the local Mq cache queue of the remote data center to the local data center.

[0039] According to some embodiments, the second aspect of the present invention provides a massive data processing and querying system for power visualization monitoring, employing the following technical solution:

[0040] A massive data processing and query system for power visualization monitoring includes:

[0041] The basic environment setup module is configured to build a unified basic environment for local and remote data centers, and to construct a data synchronization queue and a user business query habit queue.

[0042] The data change summary module is configured to collect power visualization data in real time and store it in both the local and remote computer rooms, and store it in the corresponding database according to the data type.

[0043] The index building module is configured to periodically query data based on historical user query logs in the remote data center, determine the relationship between user query hot words and corresponding data time-series nanosecond IDs, form query hot word-nanosecond index IDs, and send them to the local data center for index synchronization;

[0044] The data query module is configured to obtain user query instructions, determine query hot words based on user query instructions, obtain the corresponding underlying database nanosecond ID based on the determined query hot words and query hot words-nanosecond index ID, and determine the query data based on the underlying database nanosecond ID.

[0045] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0046] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the massive data processing and querying method for power visualization monitoring as described in the first aspect above.

[0047] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0048] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the massive data processing and querying method for power visualization monitoring as described in the first aspect above.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] This invention addresses both the aforementioned problems and the inefficiency of querying massive amounts of data. Data backup is no longer scheduled, and data from remote data centers is fully utilized to provide efficient querying. Multi-dimensional indexes based on user query habits are pre-generated and reverse-synchronized to the user-side server database storage. Software applications use the generated multi-dimensional indexes commonly used in business operations, built on the underlying data indexes. Multi-level indexes reduce the query scope of large tables, facilitating rapid response for user queries.

[0051] This invention provides enhanced disaster recovery capabilities for service applications and a superior user experience through off-site backup. In terms of data protection, off-site backup ensures data is stored in multiple locations, reducing the risk of data loss. The design for efficient data querying employs a multi-layered structural design for the reception, storage, and use of massive amounts of data. It generates multi-key indexes based on business function classification, providing efficient query methods for complex business operations. This design offers underlying support for user experience, R&D, and maintenance, resulting in significant long-term benefits. Attached Figure Description

[0052] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0053] Figure 1 This is a flowchart of a massive data processing and query method for power visualization monitoring in an embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0056] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0057] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a method for processing and querying massive amounts of data for power visualization monitoring. In this embodiment, the method includes the following steps:

[0060] Massive data processing and querying methods for power visualization monitoring include:

[0061] A unified basic environment is built for local and remote data centers, and a data synchronization queue and a user business query habit queue are constructed.

[0062] Real-time power visualization data is collected and stored in local and remote data centers, and then stored in the corresponding database according to the data type.

[0063] The remote data center periodically queries data based on historical user query logs to determine the relationship between user query hot words and corresponding data time-series nanosecond IDs, forming query hot word-nanosecond index IDs and sending them to the local data center for index synchronization.

[0064] Obtain the user's query instruction and determine the query hot words based on the user's query instruction; based on the determined query hot words and query hot words-nanosecond index ID, obtain the corresponding underlying database nanosecond ID, and determine the query data based on the underlying database nanosecond ID.

[0065] Furthermore, the unified basic environment setup for local and remote data centers, and the construction of data synchronization queues and user business query habit queues, include:

[0066] Set up the same version of database, backup data transmission queue and backup data consumption strategy for local data center and remote data center;

[0067] Establish the corresponding data synchronization queue and its agreed data format;

[0068] Establish a corresponding queue of user business query habits and its agreed data format.

[0069] Furthermore, the keywords of the data synchronization queue are DataBaseType (database type), DataType (data type), and Data (specific data);

[0070] The keywords for the user business query habit queue are BusinessType (business function module), QueryKeyWords (query keywords), and DATASHEET (data table).

[0071] Furthermore, real-time power visualization data is collected and simultaneously stored in both local and remote data centers, and then stored in the corresponding databases according to data type, including:

[0072] Establish a business function module dictionary, which includes the function types of all data changes within the application;

[0073] The local data center stores the power visualization data into the corresponding database according to the data type. When data entry fails, the local data center removes the failed data and sends the remaining data to the Mq data synchronization queue in the remote data center.

[0074] The remote data center also stores the power visualization data in the remote data center's Mq data synchronization queue into the corresponding database according to the data type.

[0075] Furthermore, the local data center stores the power visualization data in the corresponding database according to the data type. When data import fails, the local data center removes the failed data and sends the remaining data to the Mq data synchronization queue in the remote data center, including:

[0076] For relational data changes, the local data center uses Mybatis, the persistence layer framework corresponding to the relational database, to intercept the relational data changes, extract the relational SQL statements of the relational data changes, form JSON format relational data changes corresponding to the business function module dictionary, and store them in the local data center's Mq temporary storage queue.

[0077] For changes to non-relational data, Spring-AOP is used to encapsulate special annotations to collect non-relational change data, integrate and standardize it into JSON format, and store it in the local data center's MQ temporary storage queue;

[0078] If a data transaction rollback occurs, the same data is combined and sent to the local data center's Mq temporary storage queue and marked as -1;

[0079] The application-side unified data processing program consumes the local data center's Mq temporary storage queue. If there is overlapping data and the data is marked as -1, the data entry fails, and the remaining data in the local data center's Mq temporary storage queue is sent to the remote data center's Mq data synchronization queue.

[0080] Furthermore, the remote data center periodically queries historical user query logs to determine the relationship between user query hot words and corresponding data time-series nanosecond IDs, forming query hot word-nanosecond index IDs, which are then sent to the local data center for index synchronization, including:

[0081] The local data center periodically collects user query logs and stores them in a local Mq cache queue. The local data center then sends the user query logs stored in the local Mq cache queue to the Mq user business query habit queue in the remote data center.

[0082] The remote data center consumption Mq user business query habit queue aggregates user query data and stores it in the remote data center database;

[0083] When the database activity in the remote data center decreases daily, user query data is collected. Based on the business function module, sub-function, and corresponding dictionary items, the number of queries for the function module is counted, and the query conditions appearing under the function module are counted. If it is a multi-condition query, the combined conditions are treated as one item, and the combined conditions are separated into individual conditions as one item. The number of queries for the function module is counted.

[0084] The query is sorted in reverse order based on the number of queries for each functional module, and the corresponding data in the database is queried using the underlying index of the database based on priority and time range, while determining the time-series nanosecond ID corresponding to the data.

[0085] Based on the number of queries for the functional modules and the corresponding time-series nanosecond IDs of the data, a query hot word-nanosecond index ID is formed;

[0086] The remote data center reorders the data based on the query hot keywords - nanosecond index ID, and combines the historical summary results. Finally, according to the sorting results, the data is forwarded to the local data center application-side Mq temporary storage queue for index synchronization.

[0087] Furthermore, the method also includes:

[0088] When the application is running normally, the local data center and the remote data center synchronize operation data, and the data in both data centers is consistent; when the local data center goes down, the monitoring center immediately switches the upper-layer gateway and forwards user requests to the remote data center; the operation and maintenance personnel restore the local data center server;

[0089] During the recovery of the local data center server, the data generated by the remote data center is temporarily stored in the local Mq cache queue of the remote data center; when the local data center server recovers, the cached data continues to be distributed, and at the same time, the remote data center will synchronize the data in the local Mq cache queue of the remote data center to the local data center.

[0090] Specifically, the method described in this embodiment includes:

[0091] Step 1, Basic Environment Setup:

[0092] Deploy the same version of software applications and databases (including relational and non-relational databases) in both local and remote data centers. Deploy a dedicated message queue (MQ) for backup data transmission and a backup data consumption program.

[0093] Establish a corresponding data synchronization queue SYNCDATA. The data format is agreed to be JSON. The keywords within the JSON are DataBaseType (database type), DataType (data type - SQL statement, file stream, serialized JSON, etc.), and Data (the actual data).

[0094] Establish a corresponding user business query habit queue, USER-QUERY-DATA. The data format should be JSON. The keywords within the JSON are: BusinessType (business function module), QueryKeyWords (query keywords), and DATASHEETe (data table).

[0095] Step Two: Data Change Summary and Processing

[0096] 1. Establish a dictionary of business function modules. Summarize all functions within the application that involve data changes and categorize them.

[0097] 2: Built-in data repeater.

[0098] 1) Relational Database Data Changes: A SQL interceptor is built using Mybatis, the persistence layer framework for relational databases. This interceptor operates before the SQL statement is sent to the database, intercepting it. At this point, an embedded data aggregation function extracts the relational SQL statement, generates a JSON dictionary, and forwards it to the local data center's MQ temporary queue.

[0099] 2) Non-relational data changes: Spring-AOP is used to encapsulate special annotations, which are then annotated in the code to collect non-relational data changes, integrate them into standardized JSON, and forward them to the local data center's Mq temporary queue.

[0100] 3) If data import fails, the same data is combined and sent to the local data center's Mq temporary storage queue and marked as -1.

[0101] It should be noted that the data entering the local data center is processed asynchronously. One process involves directly storing the data into the local data center's Mq temporary queue for database entry. The other parallel process involves first entering an interceptor to perform data combination and judgment. If the database entry fails, the interceptor will combine the failed data and add a status flag of -1 before forwarding it to the local data center's Mq temporary queue. This results in two duplicate data entries in the local data center's Mq temporary queue, but with a flag of -1, indicating that the database entry failed.

[0102] For example: If a row of data is 2-2-2 and the database storage fails, a rollback will occur. The program will then retrieve the erroneous data as 2-2-2, and recreate the same 2-2-2 row with a status flag of -1 to indicate that the data entry failed. If a data transaction rollback occurs, the data will be sent to the temporary queue again, this time with a status flag of -1.

[0103] 3: The application-side unified data processing program consumes the local data center's Mq temporary queue and sends the data in the local data center's Mq temporary queue to the remote data center's Mq data synchronization queue: SYNCDATA.

[0104] Application data is typically entered into the database in seconds; the volume of data is small, but the frequency is high. Therefore, this processing program caches x seconds of data in memory. If duplicate data is found with a -1 flag, it is discarded; this data is considered a failed entry. In other words, if duplicate data with a -1 flag is found, both records are removed, and the remaining data is entered into the database.

[0105] 4. The remote data center data synchronization processing program consumes the remote data center's MQ data synchronization queue: SYNCDATA data. Based on the DataBaseTyp (database type), it calls different ingestion programs (relational database ingestion, non-relational database ingestion, file data ingestion, etc.) to obtain the Data and store it in the database.

[0106] 5. To achieve the above, the application's data structure design needs to avoid auto-incrementing IDs in the database. It should use time-series nanosecond-level IDs generated within the application to ensure data consistency across local and remote locations. This provides the foundation for building multi-dimensional indexes at the business layer. It's important to note that these time-series nanosecond IDs are generated within the application; the data already possesses these IDs before being cached in the local and remote data centers.

[0107] Step 3: Building and synchronizing multi-dimensional indexes in remote data centers with local data centers

[0108] 1. The local application has a built-in user query log collection program. The collected data includes user ID, business function module, sub-data module, and multiple query conditions. This data is asynchronously forwarded to the local data center's MQ cache queue. The local cache queue then centrally consumes the data and forwards it to the off-site data center's MQ user business query habit queue.

[0109] It should be noted that this is the user business query habit queue (USER-QUERY-DATA).

[0110] 2: Consume the Mq-USER-QUERY-DATA queue in the remote data center. Aggregate user query data and store it in the remote data center database.

[0111] 3. In the remote data center, when database activity decreases daily (generally in the early morning), user query data is analyzed. Based on business function modules, sub-functions, and corresponding dictionary entries (which record the relationship between business functions and database tables), the number of queries for each function module is counted. The query conditions appearing under each function module are also counted. If a query involves multiple conditions, the combined conditions are treated as one item. The final count is then calculated for the total number of queries that occurred under that function module. For example, if the function module is a user information query module, then multiple conditions here would refer to user name, user ID card number, user address, etc.

[0112] A reverse search is performed based on the frequency of all query conditions appearing under each functional module. Then, data from the corresponding database is retrieved according to priority and time range. This query utilizes the database's built-in underlying indexes. Due to a unified design, the index keys for large tables are time-series nanosecond IDs; therefore, this query will retrieve the nanosecond ID range corresponding to the query conditions. This process is time-consuming and puts significant pressure on database read / write operations. Therefore, this step is designed to be handled during periods of low activity in a remote data center.

[0113] 4. The query hot keywords - nanosecond index ID range results generated in the previous step are preferentially stored in the off-site data center. During storage, the data is re-sorted based on historical summary results, and finally forwarded to the application-side MQ in the local data center according to the sorting results. It should be noted that the specific data forwarding process is consistent with the process of synchronizing data from the off-site data center to the local data center mentioned earlier. Both processes involve first temporarily storing the data in the off-site data center's MQ temporary queue, and then forwarding it to the local data center.

[0114] 5. The local application consumes the result queue data generated by the multi-dimensional index using Mq, stores it in a relational database, and then caches the result in a server cache such as Redis.

[0115] Step 4: Efficient User Query Process

[0116] When a visualization application is running, users often use query-based functions to view historical data, etc.

[0117] At this point, the application prioritizes decomposing the business function module to which the query belongs, decomposing the query conditions, obtaining the query time range, and retrieving the data in the cache based on the function and query conditions.

[0118] If the query is successful, retrieve the corresponding underlying database nanosecond ID. Using this underlying index ID, retrieve the detailed data from the underlying database. This step avoids using excessive underlying indexes, directly obtaining the underlying data index key through the multi-dimensional index generated by the upper-layer application, and directly retrieving the data based on the key value.

[0119] Step 5: Off-site mutual backup and disaster recovery process.

[0120] When the application is running normally, the local and remote data centers operate on data almost synchronously, and the data in both data centers is consistent. When the local data center goes down, the monitoring center immediately switches the upper-layer gateway and forwards user requests to the remote data center.

[0121] The operations and maintenance personnel restore the local server. During this restoration process, data generated in the remote data center also needs to be synchronized to the downed local server, but the downed server cannot consume this data. To prevent data loss, the remote data center uses an MQ cache queue, which temporarily stores the data that needs to be synchronized. Once the original server recovers, the cached data is continuously distributed, ensuring rapid data synchronization and eliminating the risk of data loss.

[0122] The remote, ultra-low latency mutual backup uses a message queue (MQ) approach to minimize data synchronization latency, and having separate MQs on each local machine prevents data loss during downtime. Furthermore, the application-generated nanosecond IDs ensure complete consistency between the two sets of data, creating the conditions for the periodic generation of multi-dimensional indexes in the remote data center. This approach neither affects the original application's operation nor reduces computational load.

[0123] Example 2

[0124] This embodiment provides a massive data processing and query system for power visualization monitoring, including:

[0125] According to some embodiments, the second aspect of the present invention provides a massive data processing and querying system for power visualization monitoring, employing the following technical solution:

[0126] A massive data processing and query system for power visualization monitoring includes:

[0127] The basic environment setup module is configured to build a unified basic environment for local and remote data centers, and to construct a data synchronization queue and a user business query habit queue.

[0128] The data change summary module is configured to collect real-time power visualization data and store it in local and remote computer rooms, and store it in the corresponding database according to the data type.

[0129] The index building module is configured to periodically query data based on historical user query logs in the remote data center, determine the relationship between user query hot words and corresponding data time-series nanosecond IDs, form query hot word-nanosecond index IDs, and send them to the local data center for index synchronization;

[0130] The data query module is configured to obtain user query instructions, determine query hot words based on user query instructions, obtain the corresponding underlying database nanosecond ID based on the determined query hot words and query hot words-nanosecond index ID, and determine the query data based on the underlying database nanosecond ID.

[0131] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0132] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0133] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0134] Example 3

[0135] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the massive data processing and querying method for power visualization monitoring as described in Embodiment 1 above.

[0136] Example 4

[0137] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the massive data processing and querying method for power visualization monitoring as described in Embodiment 1 above.

[0138] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0139] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0142] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0143] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for mass data processing query for power visualized monitoring, characterized in that, include: A unified basic environment is built for local and remote data centers, and a data synchronization queue and a user business query habit queue are constructed. Real-time power visualization data is collected and simultaneously stored in local and remote data centers, and then stored in the corresponding database according to the data type. The remote data center periodically queries historical user query logs to determine the relationship between user query hotspots and corresponding data time-series nanosecond IDs, forming query hotspot-nanosecond index IDs, which are then sent to the local data center for index synchronization. This includes: The local data center periodically collects user query logs and stores them in a local Mq cache queue. The local data center then sends the user query logs stored in the local Mq cache queue to the Mq user business query habit queue in the remote data center. The remote data center consumption Mq user business query habit queue aggregates user query data and stores it in the remote data center database; When the database activity in the remote data center decreases daily, user query data is collected. Based on the business function module, sub-function, and corresponding dictionary item, the number of queries for the function module is counted, and the query conditions appearing under the function module are counted. If it is a multi-condition query, the combined conditions are treated as one item, and the number of queries for the function module is counted. The query is sorted in reverse order based on the number of queries for each functional module, and the corresponding data in the database is queried using the underlying index of the database based on priority and time range, while determining the time-series nanosecond ID corresponding to the data. Based on the number of queries for the functional modules and the corresponding time-series nanosecond IDs of the data, a query hot word-nanosecond index ID is formed; The remote data center reorders the data based on the query hot words - nanosecond index ID, and combines the historical summary results. Finally, according to the sorting results, the data is forwarded to the local data center application-side Mq temporary storage queue for index synchronization. Obtain the user's query instruction and determine the query hot words based on the user's query instruction; based on the determined query hot words and query hot words-nanosecond index ID, obtain the corresponding underlying database nanosecond ID, and determine the query data based on the underlying database nanosecond ID.

2. The petabyte processing query method for power visualization monitoring of claim 1, wherein, The aforementioned steps involve establishing a unified basic environment for both local and remote data centers, and constructing data synchronization queues and user business query habit queues, including: Set up the same version of database, backup data transmission queue and backup data consumption strategy for local data center and remote data center; Establish the corresponding data synchronization queue and its agreed data format; Establish a corresponding queue of user business query habits and its agreed data format.

3. The petabyte processing query method for power visualization monitoring of claim 2, wherein, The keywords of the data synchronization queue are DataBaseType, DataType, and Data; The keywords for the user business query habit queue are BusinessType, QueryKeyWords, and DATASHEET.

4. The method for processing and querying massive amounts of data for power visualization monitoring as described in claim 1, characterized in that, Real-time acquisition of power visualization data is performed and simultaneously stored in both local and remote data centers. The data is then stored in the corresponding database based on its data type, including: Establish a business function module dictionary, which includes the function types of all power visualization data within the application; The local data center stores the power visualization data into the corresponding database according to the data type. When data entry fails, the local data center removes the failed data and sends the remaining data to the Mq data synchronization queue in the remote data center. The remote data center also stores the power visualization data in the remote data center's Mq data synchronization queue into the corresponding database according to the data type.

5. The method for processing and querying massive amounts of data for power visualization monitoring as described in claim 4, characterized in that, The local data center stores the power visualization data into the corresponding database according to the data type. When data import fails, the local data center removes the failed data and sends the remaining data to the Mq data synchronization queue in the remote data center, including: For relational data changes, the local data center uses Mybatis, the persistence layer framework corresponding to the relational database, to intercept the relational data changes, extract the relational SQL statements of the relational data changes, form JSON format relational data changes corresponding to the business function module dictionary, and store them in the local data center's Mq temporary storage queue. For changes to non-relational data, Spring-AOP is used to encapsulate special annotations to collect non-relational change data, integrate and standardize it into JSON format, and store it in the local data center's MQ temporary storage queue; If data import fails, the same data is combined and sent to the local data center's Mq temporary storage queue and marked as -1. The application-side unified data processing program consumes the local data center's Mq temporary storage queue. If there is overlapping data and the data is marked as -1, the data entry fails, and the remaining data in the local data center's Mq temporary storage queue is sent to the remote data center's Mq data synchronization queue.

6. The method for processing and querying massive amounts of data for power visualization monitoring as described in claim 1, characterized in that, Also includes: When the application is running normally, the local data center and the remote data center synchronize operation data, and the data in both data centers is consistent; When the local data center goes down, the monitoring center immediately switches the upper-layer gateway and forwards user requests to the remote data center; the operations and maintenance personnel then restore the local data center server. During the local data center server recovery period, the data generated in the remote data center is temporarily stored in the local Mq cache queue of the remote data center; Once the local data center server recovers, the cached data continues to be distributed, and the remote data center will synchronize the data from its local Mq cache queue to the local data center.

7. A massive data processing and query system for power visualization monitoring, characterized in that, The method for processing and querying massive amounts of data for power visualization monitoring as described in any one of claims 1-6 includes: The basic environment setup module is configured to build a unified basic environment for local and remote data centers, and to construct a data synchronization queue and a user business query habit queue. The data change summary module is configured to collect power visualization data in real time and store it in both the local and remote computer rooms, and store it in the corresponding database according to the data type. The index building module is configured to periodically query data based on historical user query logs in the remote data center, determine the relationship between user query hot words and corresponding data time-series nanosecond IDs, form query hot word-nanosecond index IDs, and send them to the local data center for index synchronization; The data query module is configured to obtain user query instructions, determine query hot words based on user query instructions, obtain the corresponding underlying database nanosecond ID based on the determined query hot words and query hot words-nanosecond index ID, and determine the query data based on the underlying database nanosecond ID.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the massive data processing and querying method for power visualization monitoring as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the massive data processing and querying method for power visualization monitoring as described in any one of claims 1-6.

Images