Method, system and equipment for caching real-time data of configuration data source based on RocksDB and medium

Abstract

The invention discloses a RocksDB-based configuration data source real-time data caching method, system, equipment and medium, and the method comprises the following steps: when configuration software requests data, querying a gnRealTimeData column family, when required data is searched from the gnRealTimeData column family, directly returning the searched data, otherwise, calling an external data source API to obtain the data required to be searched; when a storage request is received, the data needing to be stored is written into the gnRealTimeData column family, and the data needing to be stored is connected to an external data source API, the method, the system, the equipment and the medium can improve the response speed of the system, and meanwhile the data access performance is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of data caching, and relates to a RocksDB-based configuration data source real-time data caching method, system, device and medium. Background Art

[0002] In the field of modern industrial automation, configuration software is used to monitor and control various process flows. Configuration software needs to continuously obtain real-time data from data sources for calculation and display. However, frequent data access will cause a large I / O burden on the data source server, thus affecting the response time of the system. In addition, since configuration software runs in a resource-constrained environment, such as a Docker container, the limitations of memory and CPU resources also place higher requirements on data access performance. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, system, device and medium for real-time data caching of a configuration data source based on RocksDB. The method, system, device and medium can improve the response speed of the system and have high data access performance.

[0004] To achieve the above object, the present invention discloses a real-time data caching method for a configuration data source based on RocksDB, comprising:

[0005] When the configuration software requests data, the gnRealTimeData column family is queried. When the required data is found in the gnRealTimeData column family, the found data is directly returned. Otherwise, the external data source API is called to obtain the required data.

[0006] When a storage request is received, the data to be stored is written into the gnRealTimeData column family, and the data to be stored is sent to an external data source API.

[0007] Furthermore, after calling the external data source API to obtain the required data, the process further includes:

[0008] The data to be searched is stored in the gnRealTimeData column family.

[0009] Furthermore, it also includes:

[0010] When the system starts, check and create the RocksDB database file and Column Family;

[0011] Get all the measurement point information and its connection information from the gnKeysAndConnectionInfo column family, and use it to construct the mapping table infoGNMap;

[0012] Traversing the mapping table infoGNMap, obtaining the latest real-time data of the corresponding measuring point from the original data source according to each connection information;

[0013] The latest real-time data is stored in the gnRealTimeData column family.

[0014] Furthermore, the process of checking and creating RocksDB database files and Column Family is as follows:

[0015] Check whether there is a RocksDB database file locally. If it exists, load the database. Otherwise, create a new RocksDB database file and establish two Column Families. One Column Family is used to store measurement point information and its connection information, and the other Column Family is used to store real-time data.

[0016] The present invention discloses a configuration data source real-time data cache system based on RocksDB, comprising:

[0017] The search module is used to query the gnRealTimeData column family when the configuration software requests data. When the required data is found in the gnRealTimeData column family, the found data is directly returned. Otherwise, the external data source API is called to obtain the required data.

[0018] The first storage module is used for writing the data to be stored into the gnRealTimeData column family when receiving a storage request, and sending the data to be stored to an external data source API.

[0019] Furthermore, after calling the external data source API to obtain the required data, the process further includes:

[0020] The data to be searched is stored in the gnRealTimeData column family.

[0021] Furthermore, it also includes:

[0022] Initialization module, used to check and create RocksDB database files and Column Family when the system starts;

[0023] The first acquisition module is used to obtain all measurement point information and its connection information from the gnKeysAndConnectionInfo column family, and construct a mapping table infoGNMap based on it;

[0024] The second acquisition module is used to traverse the mapping table infoGNMap and acquire the latest real-time data of the corresponding measuring point from the original data source according to each connection information;

[0025] The second storage module is used to store the latest real-time data in the gnRealTimeData column family.

[0026] Furthermore, the process of checking and creating RocksDB database files and Column Family is as follows:

[0027] Check whether there is a RocksDB database file locally. If it exists, load the database. Otherwise, create a new RocksDB database file and establish two Column Families. One Column Family is used to store measurement point information and its connection information, and the other Column Family is used to store real-time data.

[0028] The present invention discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the RocksDB-based configuration data source real-time data caching method are implemented.

[0029] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the configuration data source real-time data caching method based on RocksDB are implemented.

[0030] The present invention has the following beneficial effects:

[0031] In the specific operation of the RocksDB-based configuration data source real-time data caching method, system, device and medium described in the present invention, when the configuration software requests data, the gnRealTimeData column family is first queried. When the required data is found in the gnRealTimeData column family, the found data is directly returned. If the data is not found, the external data source API is called to obtain the required data. It should be noted that the present invention greatly reduces the number of accesses to the original data source through a local caching mechanism, significantly improves the response speed of the system, and has high data access performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0033] Figure 1Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0036] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0037] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0038] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0039] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0042] Embodiment 1

[0043] refer to Figure 1 The RocksDB-based configuration data source real-time data caching method of the present invention comprises the following steps:

[0044] 1) Refresh the cache regularly;

[0045] 11) Initialization;

[0046] When the system starts, check and create the RocksDB database file and Column Family.

[0047] Specifically, check whether there is a RocksDB database file locally. If it exists, load the database. Otherwise, create a new RocksDB database file and establish two Column Families. One Column Family is used to store the measurement point information and its connection information (gnKeysAndConnectionInfo), and the other Column Family is used to store the real-time data (gnRealTimeData).

[0048] 12) Start the scheduled task;

[0049] Use a scheduler to execute a cache refresh task every 5 seconds.

[0050] 13) Obtain measurement point information and connection information, and build a mapping table;

[0051] Get all the measurement point information and its connection information from the gnKeysAndConnectionInfo column family and build the mapping table infoGNMap.

[0052] 14) Real-time data acquisition;

[0053] Traversing the mapping table infoGNMap, obtaining the latest real-time data of the corresponding measuring point from the original data source according to each connection information;

[0054] The latest real-time data is stored in the gnRealTimeData column family.

[0055] The pseudo code example for step 1) is:

[0056] / / Initialize the system, check and create RocksDB database files and Column Family initialize_system()

[0057] if not exists(rocksDB_database):

[0058] create_rocksDB_database()

[0059] load_column_families()

[0060] / / Create a scheduled task to refresh the cache every 5 seconds

[0061] create_scheduled_task(interval=5s,task=refresh_cache)

[0062] function refresh_cache():

[0063] / / Get all measurement points and their connection information from the gnKeysAndConnectionInfo column family infoGNMap = get_all_gn_and_connection_info()

[0064] / / Traverse infoGNMap and obtain the latest real-time data of the corresponding measurement point from the original data source according to each connection information

[0065] for each(connection_info,gn_list)in infoGNMap:

[0066] opRealtimeDataList=fetch_realtime_data(connection_info,gn_list)

[0067] / / Store the acquired real-time data in the gnRealTimeData column family in sequence and update the cache

[0068] for each opData in opRealtimeDataList:

[0069] update_gnRealTimeData_cache(opData)

[0070] 2) Data acquisition process;

[0071] 21) receiving data request;

[0072] When the configuration software requests data, it first queries the gnRealTimeData column family;

[0073] 22) When the cache hits;

[0074] When the required data is found in the gnRealTimeData column family, the found data is directly returned;

[0075] 23) When the cache misses;

[0076] When the required data is not found in the gnRealTimeData column family, the external data source API is called to obtain the required data, and then the required data is stored in the gnRealTimeData column family.

[0077] The pseudo code example for step 2) is:

[0078] function get_data(requested_metrics):

[0079] result = {}

[0080] / / Traverse the requested measurement points

[0081] for each metric in requested_metrics:

[0082] / / Find the required measurement point data from the gnRealTimeData column family

[0083] data=get_from_gnRealTimeData_cache(metric)

[0084] if data!=null:

[0085] / / Cache hit, directly return the data in the cache

[0086] result[metric]=data

[0087] else:

[0088] / / Cache miss, call external data source API to get data

[0089] data=fetch_from_external_source(metric)

[0090] / / Store the acquired data in the gnRealTimeData column family and update the cache

[0091] update_gnRealTimeData_cache(data)

[0092] result[metric]=data

[0093] / / Return all the measured point data obtained

[0094] return result

[0095] 3) Data storage process;

[0096] 31) receiving a storage request;

[0097] When the storage request is received, the data to be stored is written into the gnRealTimeData column family;

[0098] 32) Write data to external data source;

[0099] The data to be stored is written into the external data source API.

[0100] The pseudo code example for step 3) is:

[0101]

[0102] It should be noted that the present invention has the following characteristics:

[0103] 1) Reduce I / O burden: Through the local caching mechanism, the number of accesses to the original data source is greatly reduced.

[0104] 2) Improve response speed: When the cache hits, data is returned directly, which significantly improves the response speed of the system.

[0105] 3) Low memory usage: Based on the storage design of RocksDB, it has low memory usage and is suitable for running in resource-constrained environments.

[0106] 4) Efficient data management: Ensure the real-time and effectiveness of cached data through scheduled tasks and reasonable data storage strategies.

[0107] The present invention is applicable to various configuration software that needs to frequently access real-time data, especially in resource-constrained environments, such as application programs running in Docker containers.

[0108] Embodiment 2

[0109] The present invention discloses a configuration data source real-time data cache system based on RocksDB, comprising:

[0110] The search module is used to query the gnRealTimeData column family when the configuration software requests data. When the required data is found in the gnRealTimeData column family, the found data is directly returned. Otherwise, the external data source API is called to obtain the required data.

[0111] The first storage module is used for writing the data to be stored into the gnRealTimeData column family when receiving a storage request, and sending the data to be stored to an external data source API.

[0112] Furthermore, after calling the external data source API to obtain the required data, the process further includes:

[0113] The data to be searched is stored in the gnRealTimeData column family.

[0114] Furthermore, it also includes:

[0115] Initialization module, used to check and create RocksDB database files and Column Family when the system starts;

[0116] The first acquisition module is used to obtain all measurement point information and its connection information from the gnKeysAndConnectionInfo column family, and construct a mapping table infoGNMap based on it;

[0117] The second acquisition module is used to traverse the mapping table infoGNMap and acquire the latest real-time data of the corresponding measuring point from the original data source according to each connection information;

[0118] The second storage module is used to store the latest real-time data in the gnRealTimeData column family.

[0119] Furthermore, the process of checking and creating RocksDB database files and Column Family is as follows:

[0120] Check whether there is a RocksDB database file locally. If it exists, load the database. Otherwise, create a new RocksDB database file and establish two Column Families. One Column Family is used to store measurement point information and its connection information, and the other Column Family is used to store real-time data.

[0121] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0122] Embodiment 3

[0123] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the configuration data source real-time data caching method based on RocksDB are implemented, for example, including: when the configuration software requests data, the gnRealTimeData column family is queried. When the required data is found from the gnRealTimeData column family, the found data is directly returned. Otherwise, an external data source API is called to obtain the required data; when a storage request is received, the data to be stored is written into the gnRealTimeData column family, and the data to be stored is sent to the external data source API. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, the network interface, and the memory are interconnected through an internal bus, which may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard structure bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include memory and nonvolatile memory and provides instructions and data to the processor.

[0124] Embodiment 4

[0125] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the configuration data source real-time data caching method based on RocksDB are implemented, for example, including: when the configuration software requests data, the gnRealTimeData column family is queried, and when the required data is found from the gnRealTimeData column family, the found data is directly returned, otherwise, an external data source API is called to obtain the required data; when a storage request is received, the data to be stored is written to the gnRealTimeData column family, and the data to be stored is sent to the external data source API. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0126] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0131] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0132] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A real-time data caching method for configuration data source based on RocksDB, characterized in that: include: When the configuration software requests data, it queries the gnRealTimeData column family. When the required data is found in the gnRealTimeData column family, it directly returns the found data. Otherwise, it calls the external data source API to obtain the required data. When a storage request is received, the data to be stored is written into the gnRealTimeData column family, and the data to be stored is sent to an external data source API.

2. The method for real-time data caching of a configuration data source based on RocksDB according to claim 1, characterized in that: After calling the external data source API to obtain the required data, the following steps are also included: The data to be searched is stored in the gnRealTimeData column family.

3. The configuration data source real-time data caching method based on RocksDB according to claim 1, characterized in that: Also includes: When the system starts, check and create the RocksDB database file and Column Family; Get all the measurement point information and its connection information from the gnKeysAndConnectionInfo column family, and use it to construct the mapping table infoGNMap; Traversing the mapping table infoGNMap, obtaining the latest real-time data of the corresponding measuring point from the original data source according to each connection information; The latest real-time data is stored in the gnRealTimeData column family.

4. The method for real-time data caching of a configuration data source based on RocksDB according to claim 3, characterized in that: The process of checking and creating RocksDB database files and Column Family is as follows: Check whether there is a RocksDB database file locally. If it exists, load the database. Otherwise, create a new RocksDB database file and establish two Column Families. One Column Family is used to store measurement point information and its connection information, and the other Column Family is used to store real-time data.

5. A real-time data cache system for configuration data sources based on RocksDB, characterized in that: include: The search module is used to query the gnRealTimeData column family when the configuration software requests data. When the required data is found in the gnRealTimeData column family, the found data is directly returned. Otherwise, the external data source API is called to obtain the required data. The first storage module is used for writing the data to be stored into the gnRealTimeData column family when receiving a storage request, and sending the data to be stored to an external data source API.

6. The configuration data source real-time data cache system based on RocksDB according to claim 5, characterized in that: After calling the external data source API to obtain the required data, the following steps are also included: The data to be searched is stored in the gnRealTimeData column family.

7. The configuration data source real-time data cache system based on RocksDB according to claim 5, characterized in that: Also includes: Initialization module, used to check and create RocksDB database files and Column Family when the system starts; The first acquisition module is used to obtain all measurement point information and its connection information from the gnKeysAndConnectionInfo column family, and construct a mapping table infoGNMap based on it; The second acquisition module is used to traverse the mapping table infoGNMap and acquire the latest real-time data of the corresponding measuring point from the original data source according to each connection information; The second storage module is used to store the latest real-time data in the gnRealTimeData column family.

8. The configuration data source real-time data cache system based on RocksDB according to claim 7, characterized in that: The process of checking and creating RocksDB database files and Column Family is as follows: Check whether there is a RocksDB database file locally. If it exists, load the database. Otherwise, create a new RocksDB database file and establish two Column Families. One Column Family is used to store measurement point information and its connection information, and the other Column Family is used to store real-time data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the RocksDB-based configuration data source real-time data caching method as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the RocksDB-based configuration data source real-time data caching method are implemented as described in any one of claims 1 to 4.

Images