A method and device for processing equipment pseudo telemetry historical data

Abstract

The present invention discloses a method for processing equipment pseudo-telemetry historical data and a corresponding processing device. The method includes: building a pseudo-telemetry point memory cluster; receiving pseudo-telemetry data uploaded by each site in real time; formatting and pre-processing the received pseudo-telemetry data, and managing the pre-processed pseudo-telemetry data in the form of a memory queue; persisting the pseudo-telemetry data in the memory queue to build a historical pseudo-telemetry data storage system; receiving a query request for pseudo-telemetry point historical information, analyzing the query request and generating query conditions; matching a query algorithm according to the query conditions, and sending a search request to the pseudo-telemetry data storage system based on the query algorithm; and receiving the search results. The method and device can improve the call speed of pseudo-telemetry historical data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method for obtaining pseudo telemetry historical data of a processing device. Background Art

[0002] The equipment monitoring system plays an important role in monitoring, controlling and maintaining ground equipment. The equipment's pseudo-telemetry historical information plays a very important role in analyzing the causes of equipment failures or preventing failures. Reasonable use of historical pseudo-telemetry information can help the system control and manage equipment more quickly and intelligently.

[0003] Existing equipment monitoring systems are not ideal for handling historical pseudo-telemetry data. On the one hand, in terms of storage database selection, most current equipment monitoring systems use relational databases to store equipment pseudo-telemetry data. When faced with large amounts of data, relational databases have low read and write throughput and cannot meet the daily monitoring system's demand for historical data. Secondly, during the storage process, in order to reduce the amount of data stored, many systems use sampling to compress and store data. Although this reduces the read and write pressure on the database to a certain extent, the equipment pseudo-telemetry data information is lost, which reduces the accuracy of data analysis and does not make effective use of the historical pseudo-telemetry data. Thirdly, in terms of query methods, most current systems use the method of directly querying data from the database, which has a slow response speed. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention discloses a method for processing pseudo telemetry historical data of a device, comprising:

[0005] Build a pseudo telemetry point memory cluster;

[0006] Receive pseudo telemetry data uploaded by each site in real time;

[0007] Formatting and preprocessing the received pseudo telemetry data, and managing the preprocessed pseudo telemetry data in the form of a memory queue;

[0008] Persistently process pseudo telemetry data in the memory queue and build a historical pseudo telemetry data storage system;

[0009] receiving a query request for pseudo telemetry point historical information, analyzing the query request, and generating query conditions;

[0010] matching a query algorithm according to the query condition, and sending a retrieval request to the pseudo telemetry data storage system based on the query algorithm;

[0011] Receive search results.

[0012] Optionally, the step of building a pseudo telemetry point memory cluster includes:

[0013] Import the device pseudo telemetry point information data into the memory to build a memory data cache cluster; wherein the pseudo telemetry point information data includes: site number, device number, pseudo telemetry point name and pseudo telemetry point number.

[0014] Optionally, the step of receiving pseudo telemetry data uploaded by each site in real time includes:

[0015] Multiple sites upload pseudo telemetry information of their respective site devices through the message middleware Kafka, where the pseudo telemetry information includes: site number, device number, pseudo telemetry point name, collection time and pseudo telemetry value.

[0016] Optionally, the step of formatting and preprocessing the received pseudo telemetry data includes:

[0017] For each pseudo telemetry information received, query the pseudo telemetry point memory cluster for a pseudo telemetry point number that matches the site number, device number, and pseudo telemetry point name;

[0018] The pseudo telemetry collection time and pseudo telemetry value in the received pseudo telemetry data are integrated to format and pre-process the pseudo telemetry information, wherein the formatted pseudo telemetry information includes: pseudo telemetry point number, collection month, collection date and pseudo telemetry value.

[0019] Optionally, the step of performing persistent processing on the pseudo telemetry data in the memory queue and building a historical pseudo telemetry data storage system includes:

[0020] Create a thread to extract pseudo telemetry data from the memory queue;

[0021] Storing the extracted pseudo telemetry data and counting the number of stored pseudo telemetry data;

[0022] When the number reaches a preset number, the stored pseudo telemetry data is persistently stored in batches to accumulate and form a historical pseudo telemetry data storage system.

[0023] Optionally, the preset number is 1000, and the historical pseudo telemetry data storage system is Apache Cassandra.

[0024] Optionally, the step of sending a retrieval request to the pseudo telemetry data storage system based on the query algorithm includes:

[0025] According to the query algorithm, it is compiled into CQL statements that can be recognized by the Cassandra database and a retrieval request is initiated to the Cassandra database.

[0026] In order to solve the above technical problems, the present invention further discloses a device for processing pseudo telemetry historical data of a device, comprising:

[0027] The first building module is used to build a pseudo telemetry point memory cluster;

[0028] The receiving module is used to receive the pseudo telemetry data uploaded by each site in real time;

[0029] a preprocessing module, configured to format and preprocess the received pseudo telemetry data, and manage the preprocessed pseudo telemetry data in the form of a memory queue;

[0030] The second construction module is used to persist the pseudo telemetry data in the memory queue and build a historical pseudo telemetry data storage system;

[0031] A generating module, configured to receive a query request for pseudo telemetry point historical information, analyze the query request, and generate query conditions;

[0032] a sending module, configured to match a query algorithm according to the query condition, and send a retrieval request to the pseudo telemetry data storage system based on the query algorithm;

[0033] The result receiving module is used to receive the search results.

[0034] Optionally, the first building module is specifically used to:

[0035] Import the device pseudo telemetry point information data into the memory to build a memory data cache cluster; wherein the pseudo telemetry point information data includes: site number, device number, pseudo telemetry point name and pseudo telemetry point number.

[0036] Optionally, the second building block includes:

[0037] The first submodule is used to create a thread to extract pseudo telemetry data from the memory queue;

[0038] The second submodule is used to store the extracted pseudo telemetry data and count the number of stored pseudo telemetry data;

[0039] The third submodule is configured to perform batch persistent storage on the stored pseudo telemetry data when the number reaches a preset number, and accumulate the stored pseudo telemetry data to form a historical pseudo telemetry data storage system.

[0040] The method for processing device pseudo-telemetry historical data disclosed in an embodiment of the present invention includes: constructing a pseudo-telemetry point memory cluster; receiving pseudo-telemetry data uploaded by each site in real time; formatting and pre-processing the received pseudo-telemetry data, and managing the pre-processed pseudo-telemetry data in the form of a memory queue; persisting the pseudo-telemetry data in the memory queue to construct a historical pseudo-telemetry data storage system; receiving a query request for pseudo-telemetry point historical information, analyzing the query request and generating query conditions; matching a query algorithm according to the query conditions, and sending a retrieval request to the pseudo-telemetry data storage system based on the query algorithm; and receiving the retrieval results. The method for processing device pseudo-telemetry historical data of this embodiment can improve the speed of calling pseudo-telemetry historical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a method for processing pseudo telemetry historical data of a device provided by an embodiment of the present invention;

[0042] Figure 2 This is a flowchart of storing pseudo telemetry history data of a device in an embodiment of the present invention;

[0043] Figure 3 Flowchart of a device pseudo telemetry history query method according to an embodiment of the present invention;

[0044] Figure 4 1 is a diagram illustrating an LRU page replacement algorithm in an embodiment of the present invention;

[0045] Figure 5 This is a structural block diagram of an apparatus for processing pseudo telemetry history data of a device in an embodiment of the present invention;

[0046] Figure 6 It is the internal storage structure in the embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below based on specific embodiments and in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] Example 1

[0049] like Figure 1 As shown, a method for processing device pseudo telemetry historical data according to an embodiment of the present invention includes the following steps:

[0050] Step 101: Build a pseudo telemetry point memory cluster.

[0051] An optional method for building a pseudo telemetry point memory cluster is as follows: import device pseudo telemetry point information data into memory to build an in-memory data cache cluster. Pseudo telemetry point information data includes: site number, device number, pseudo telemetry point name, and pseudo telemetry point number. This construction method facilitates efficient reuse of basic pseudo telemetry information and improves query speed.

[0052] Step 102: Receive the pseudo telemetry data uploaded by each site in real time.

[0053] An optional method for receiving pseudo telemetry data uploaded by each site in real time can be as follows: multiple sites upload pseudo telemetry information of their own site equipment through the message middleware Kafka, where the pseudo telemetry information includes: site number, device number, pseudo telemetry point name, collection time and pseudo telemetry value.

[0054] Kafka uses an efficient and reliable message delivery mechanism to support cross-platform data interaction. By listening to the topic or title corresponding to Kafka pseudo-telemetry information, it can receive pseudo-telemetry information from multiple sites in real time, mainly including site number, device number, pseudo-telemetry point name, collection time, and pseudo-telemetry value, ensuring that pseudo-telemetry information can be received and processed uniformly and asynchronously.

[0055] Step 103: Format and pre-process the received pseudo telemetry data, and manage the pre-processed pseudo telemetry data in the form of a memory queue.

[0056] In order to avoid the mismatch between the pseudo telemetry data formatting speed and the data persistence speed, which may cause message blocking, the pseudo telemetry data is managed by a memory queue in the embodiment of the present application.

[0057] An optional method for formatting and preprocessing the received pseudo telemetry data can be as follows: for each received pseudo telemetry information, query the pseudo telemetry point number that matches the site number, device number, and pseudo telemetry point name in the pseudo telemetry point memory cluster; integrate the pseudo telemetry collection time and pseudo telemetry value in the received pseudo telemetry data to format and preprocess the pseudo telemetry information, wherein the formatted pseudo telemetry information includes: pseudo telemetry point number, collection month, collection date, and pseudo telemetry value.

[0058] Step 104: Perform persistence processing on the pseudo telemetry data in the memory queue and build a historical pseudo telemetry data storage system.

[0059] An optional way to persist pseudo telemetry data in the memory queue and build a historical pseudo telemetry data storage system is as follows:

[0060] First, create a thread to extract pseudo telemetry data from the memory queue;

[0061] Secondly, the extracted pseudo telemetry data is stored and the number of stored pseudo telemetry data is counted;

[0062] Again, when the number reaches a preset number, the stored pseudo telemetry data is stored persistently in batches to accumulate and form a historical pseudo telemetry data storage system.

[0063] The default number is 1000. The historical pseudo-telemetry data storage system can be Apache Cassandra, a hybrid non-relational database. The storage table structure in the non-relational database uses the pseudo-telemetry point number and the collection month as the unique record identifier, with the collection time as the row key.

[0064] Step 105: Receive a query request for pseudo telemetry point historical information, analyze the query request, and generate query conditions.

[0065] The query request can be an HTTP request, which can be a single-point or multi-point request. When quickly responding to HTTP requests for pseudo telemetry point historical information, a query service interface can be written and provided to other programs for use, to monitor requests and return the data results required by other programs.

[0066] When analyzing the query request and generating query conditions, you can query the service interface to parse the request content and specify the query conditions to be used.

[0067] Step 106: Match the query algorithm according to the query condition, and send a retrieval request to the pseudo telemetry data storage system based on the query algorithm.

[0068] The method of sending a retrieval request to the pseudo telemetry data storage system based on the query algorithm may be: according to the query algorithm, compiling into a CQL statement recognizable by the Cassandra database and initiating a retrieval request to the Cassandra database.

[0069] Step 107: Receive the search results.

[0070] The Cassandra database can return database response results in a unified format.

[0071] The method for processing device pseudo-telemetry historical data provided in the embodiments of this application, on the one hand, analyzes the characteristics of pseudo-telemetry points collected by the device and selects Cassandra as the data storage database based on business needs. The Cassandra database can achieve rapid data writing and can store hundreds of TB of data without sacrificing read efficiency. It is very suitable for storage scenarios such as time-series data that do not require update operations and only focus on reading and writing. Secondly, query efficiency is improved by rationally designing the storage table structure of the Cassandra database. Thirdly, the LRU (Least Recently Used) page replacement algorithm is used to manage the memory cache and optimize query speed.

[0072] Example 2

[0073] Figure 2 This is a flowchart of storing pseudo telemetry history data of a device provided by the second embodiment of the present invention, which is specifically as follows:

[0074] Step 201: monitor and receive real-time telemetry.

[0075] The system receives multiple pseudo telemetry information in real time by monitoring the topic corresponding to the Kafka pseudo telemetry information, which mainly includes the site number, equipment number, pseudo telemetry point name, collection time, and pseudo telemetry value.

[0076] Step 202: Preprocessing of pseudo telemetry data.

[0077] During system initialization, a memory set of all pseudo telemetry points of the device is constructed, including the site number, device number, pseudo telemetry name, and corresponding pseudo telemetry number. This step uses the information stored in the above memory to quickly parse the pseudo telemetry number of the received data, obtain the collection month, and integrate the pseudo telemetry collection time and pseudo telemetry value.

[0078] Step 203: Memory table cache.

[0079] To avoid message blocking caused by the mismatch between the formatting speed and data persistence speed of pseudo telemetry data, a memory queue is used to manage the formatted pseudo telemetry data, realizing asynchronous processing of message formatting and data reading. The default cache size here is 1000 data items.

[0080] Step 204: Store in the database.

[0081] The formatted data, including the pseudo telemetry number, collection month, collection time, and pseudo telemetry value, is persisted to the Cassandra database.

[0082] Table 1 Database storage auxiliary table structure design table

[0083] Column Name property illustrate telemetry_id text Telemetry point number month bigint Pseudo telemetry collection month timestamp time bigint Pseudo telemetry collection timestamp value double Pseudo telemetry values

[0084] The table structure uses the telemetry point number (telemetry_id) and the pseudo-telemetry collection month timestamp (month) as the PartitionKey, and the collection time (time) as the ClusteringKey. Each telemetry point stores one row per month for easy search. The telemetry_id is a 9-digit decimal string in the format "AAABBBCCC." "AAA" represents the site number, such as "100" for Beijing; "BBB" represents the device number at each site, such as "090" for a device in Beijing; and "CCC" represents the telemetry point number for each device, such as "111" for a telemetry point. "100090111" uniquely identifies a telemetry point on a device at a specific site.

[0085] The internal storage structure is Figure 6 .

[0086] For example, inserting data as:

[0087] insert into telemetry(telemetry_id,month,time,value)values('100090111', 1625068800000,1625313771000,26.0);

[0088] insert into telemetry(telemetry_id,month,time,value)values('100090111',16 25068800000,1625313772000,26.1);

[0089] insert into telemetry(telemetry_id,month,time,value)values('100090111',16 25068800000,1625313773000,26.22);

[0090] insert into telemetry(telemetry_id,month,time,value)values('100090111',16 25068800000,1625313774000,26.1);

[0091] insert into telemetry(telemetry_id,month,time,value)values('100090112',16 25068800000,1625313775000,26.4);

[0092] insert into telemetry(telemetry_id,month,time,value)values('100090112',16 25068800000,1625313776000,26.3);

[0093] insert into telemetry(telemetry_id,month,time,value)values('100090112',16 25068800000,1625313777000,26.4);

[0094] The internal storage format of the above data is:

[0095]

[0096] Figure 3 This is a flowchart of a method for calling device pseudo telemetry history provided by an embodiment of the present invention, which is specifically as follows:

[0097] Step 301: Obtain query conditions.

[0098] When a user has a query requirement, they can enter a query request through a browser or client, specifically through the RESTful API. Data query conditions can include site name, device name, pseudo telemetry name, and query time period.

[0099] Step 302: parse the query conditions.

[0100] In this embodiment, after receiving the query conditions input by the user, the query conditions are parsed according to the query content and the memory set of all pseudo telemetry points of the device involved in step 202. The query conditions can include single-condition queries and multi-condition queries, which are set specifically according to user needs. The parsing result is the number of the queried pseudo telemetry point, the collection month timestamp, and the collection time interval.

[0101] Step 303: Obtain a query statement.

[0102] According to the query condition generated in step 302 and combined with the table structure stored in step 204, an example query statement is: select * from telemetry where telemetry_id = 'queried pseudo telemetry point number' and month = 'collection month time timestamp' and time between 'collection time interval'.

[0103] Steps 304-306: check whether the data is in the memory, determine whether the data is complete, and query the database.

[0104] In this embodiment, the LRU algorithm is used to manage memory data.

[0105] The LRU (Least recently used) algorithm eliminates data based on its historical access records. Its core idea is that "if data has been accessed recently, it is more likely to be accessed in the future."

[0106] Figure 4 This is an illustration of the LRU page replacement algorithm in an embodiment of the present invention:

[0107] Step 401: Map the pseudo telemetry point number to be queried and the month to be queried according to the query condition. If there is a cross-month situation, the query condition is split into multiple parts.

[0108] Step 402: According to the mapping result, query whether there is corresponding data in the hash table.

[0109] Step 403: If there is no data meeting the query condition in the hash table, it is determined whether the current cache capacity is full.

[0110] Step 404: If the cache capacity is full, remove the hash table key and the tail node of the linked list; otherwise, go directly to step 405.

[0111] Step 405: query the database according to the mapping conditions.

[0112] Step 406: Add a hash table key.

[0113] Step 407: insert the retrieved key value data into the head of the linked list.

[0114] Step 408: Query specific data information that meets the conditions based on the mapped V value.

[0115] According to the Cassandra database storage table structure design scheme in step 204, the query data is cached as follows: Map <String,List<Pair<long,double> >>

[0116]

[0117] The entire Map structure is managed using the LRU algorithm.

[0118] The specific query process is as follows:

[0119] 1) Map to the K structure according to the query conditions. If it matches in K, the List is directly obtained from V <pair>, and then get the specific value according to time.

[0120] 2) If not found, step 306 is executed again, and the database is directly queried according to the query statement generated in step 303, and the data of the entire month found is loaded into the memory in Map format.

[0121] 3) If there is a partial match, that is, there is a cross-month situation, the query conditions are processed according to K, and the unmatched part is subjected to the above step 2).

[0122] Step 307: Return the query result.

[0123] In this embodiment, based on the results of the database query, combined with the memory set of all pseudo telemetry points of the device involved in step 202, the query results are parsed, and data such as numbers and timestamps are parsed into more intuitive names, dates, etc.

[0124] Example 3

[0125] like Figure 5 As shown, the apparatus for processing device pseudo telemetry history data in an embodiment of the present application includes the following modules:

[0126] The first building module 501 is used to build a pseudo telemetry point memory cluster;

[0127] Receiving module 502, for receiving pseudo telemetry data uploaded by each site in real time;

[0128] A preprocessing module 503 is configured to format and preprocess the received pseudo telemetry data, and manage the preprocessed pseudo telemetry data in the form of a memory queue;

[0129] The second construction module 504 is used to perform persistent processing on the pseudo telemetry data in the memory queue and build a historical pseudo telemetry data storage system;

[0130] A generating module 505 is configured to receive a query request for pseudo telemetry point historical information, analyze the query request, and generate query conditions;

[0131] a sending module 506, configured to match a query algorithm according to the query condition, and send a retrieval request to the pseudo telemetry data storage system based on the query algorithm;

[0132] The result receiving module 507 is used to receive the search results.

[0133] Optionally, the first construction module is specifically used to: import the device pseudo telemetry point information data into the memory to build a memory data cache cluster; wherein the pseudo telemetry point information data includes: site number, device number, pseudo telemetry point name and pseudo telemetry point number.

[0134] Optionally, the second construction module includes: a first sub-module, used to create a thread to extract pseudo telemetry data from a memory queue; a second sub-module, used to store the extracted pseudo telemetry data and count the number of stored pseudo telemetry data; and a third sub-module, used to perform batch persistent storage of the stored pseudo telemetry data when the number reaches a preset number, and accumulate to form a historical pseudo telemetry data storage system.

[0135] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0136] The above description is only the best specific implementation method of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

[0137] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.< / pair>

Claims

1. A method for processing pseudo telemetry historical data of a device, characterized in that: The method comprises: Build a pseudo telemetry point memory cluster; Receive pseudo telemetry data uploaded by each site in real time; Formatting and preprocessing the received pseudo telemetry data, and managing the preprocessed pseudo telemetry data in the form of a memory queue; Persistently process pseudo telemetry data in the memory queue and build a historical pseudo telemetry data storage system; receiving a query request for pseudo telemetry point historical information, analyzing the query request, and generating query conditions; matching a query algorithm according to the query condition, and sending a retrieval request to the pseudo telemetry data storage system based on the query algorithm; Receive search results; The step of building a pseudo telemetry point memory cluster includes: Importing device pseudo telemetry point information data into memory to build a memory data cache cluster; wherein the pseudo telemetry point information data includes: site number, device number, pseudo telemetry point name and pseudo telemetry point number; The step of receiving the pseudo telemetry data uploaded by each site in real time includes: Multiple sites upload pseudo telemetry information of their respective site devices through the message middleware Kafka, where the pseudo telemetry information includes: site number, device number, pseudo telemetry point name, collection time, and pseudo telemetry value; The step of formatting and pre-processing the received pseudo telemetry data includes: For each pseudo telemetry information received, query the pseudo telemetry point memory cluster for a pseudo telemetry point number that matches the site number, device number, and pseudo telemetry point name; Integrating the pseudo telemetry collection time and pseudo telemetry value in the received pseudo telemetry data to format and pre-process the pseudo telemetry information, wherein the formatted pseudo telemetry information includes: pseudo telemetry point number, collection month, collection date, and pseudo telemetry value; The step of performing persistent processing on the pseudo telemetry data in the memory queue and building a historical pseudo telemetry data storage system comprises: Create a thread to extract pseudo telemetry data from the memory queue; Storing the extracted pseudo telemetry data and counting the number of stored pseudo telemetry data; When the number reaches a preset number, the stored pseudo telemetry data is persistently stored in batches to accumulate and form a historical pseudo telemetry data storage system; During the persistence process, the formatted data is persisted to the Cassandra database. The database storage auxiliary table structure design table is: The table structure uses the telemetry point ID (telemetry_id) and the pseudo-telemetry collection month timestamp (month) as the PartitionKey, and the collection time (time) as the ClusteringKey. The telemetry_id is a 9-digit decimal string in the format of "AAABBBCCC," where "AAA" represents the site ID, "BBB" represents the device ID at each site, and "CCC" represents the telemetry point ID of each device. The internal storage structure is: According to the Cassandra database storage table structure design, the following cache structure is designed for query data: Map <String,List<Pair<long,double> >> The entire Map structure is managed using the LRU algorithm; The query process is as follows: 1) Map to the K structure according to the query conditions. If it matches in K, the List is directly obtained from V <pair> , and then get the specific value according to time;< / pair> 2) If not found, re-execute the query database, directly query the database according to the generated query statement, and load the queried data for the entire month into the memory in Map format; 3) If there is a partial match, that is, there is a cross-month situation, the query conditions are processed according to K, and the unmatched part is subjected to the above step 2).

2. The method according to claim 1, characterized in that The preset number is 1000, and the historical pseudo telemetry data storage system is Apache Cassandra.

3. The method according to claim 2, characterized in that The step of sending a retrieval request to the pseudo telemetry data storage system based on the query algorithm comprises: According to the query algorithm, it is compiled into CQL statements that can be recognized by the Cassandra database and a retrieval request is initiated to the Cassandra database.

4. A device for processing pseudo telemetry historical data of equipment, characterized in that: The device comprises: The first building module is used to build a pseudo telemetry point memory cluster; The receiving module is used to receive the pseudo telemetry data uploaded by each site in real time; a preprocessing module, configured to format and preprocess the received pseudo telemetry data, and manage the preprocessed pseudo telemetry data in the form of a memory queue; The second construction module is used to persist the pseudo telemetry data in the memory queue and build a historical pseudo telemetry data storage system; A generating module, configured to receive a query request for pseudo telemetry point historical information, analyze the query request, and generate query conditions; a sending module, configured to match a query algorithm according to the query condition, and send a retrieval request to the pseudo telemetry data storage system based on the query algorithm; A result receiving module, used for receiving search results; The step of building a pseudo telemetry point memory cluster includes: Importing device pseudo telemetry point information data into memory to build a memory data cache cluster; wherein the pseudo telemetry point information data includes: site number, device number, pseudo telemetry point name and pseudo telemetry point number; The step of receiving the pseudo telemetry data uploaded by each site in real time includes: Multiple sites upload pseudo telemetry information of their respective site devices through the message middleware Kafka, where the pseudo telemetry information includes: site number, device number, pseudo telemetry point name, collection time, and pseudo telemetry value; The step of formatting and pre-processing the received pseudo telemetry data includes: For each pseudo telemetry information received, query the pseudo telemetry point memory cluster for a pseudo telemetry point number that matches the site number, device number, and pseudo telemetry point name; Integrating the pseudo telemetry collection time and pseudo telemetry value in the received pseudo telemetry data to format and pre-process the pseudo telemetry information, wherein the formatted pseudo telemetry information includes: pseudo telemetry point number, collection month, collection date, and pseudo telemetry value; The step of performing persistent processing on the pseudo telemetry data in the memory queue and building a historical pseudo telemetry data storage system comprises: Create a thread to extract pseudo telemetry data from the memory queue; Storing the extracted pseudo telemetry data and counting the number of stored pseudo telemetry data; When the number reaches a preset number, the stored pseudo telemetry data is persistently stored in batches to accumulate and form a historical pseudo telemetry data storage system; During the persistence process, the formatted data is persisted to the Cassandra database. The database storage auxiliary table structure design table is: The table structure uses the telemetry point ID (telemetry_id) and the pseudo-telemetry collection month timestamp (month) as the PartitionKey, and the collection time (time) as the ClusteringKey. The telemetry_id is a 9-digit decimal string in the format of "AAABBBCCC," where "AAA" represents the site ID, "BBB" represents the device ID at each site, and "CCC" represents the telemetry point ID of each device. The internal storage structure is: According to the Cassandra database storage table structure design, the following cache structure is designed for query data: Map <String,List<Pair<long,double> >> The entire Map structure is managed using the LRU algorithm; The query process is as follows: 1) Map to the K structure according to the query conditions. If it matches in K, the List is directly obtained from V <pair> , and then get the specific value according to time;< / pair> 2) If not found, re-execute the query database, directly query the database according to the generated query statement, and load the queried data for the entire month into the memory in Map format; 3) If there is a partial match, that is, there is a cross-month situation, the query conditions are processed according to K, and the unmatched part is subjected to the above step 2).

Images