A real-time measurement system based on materialized memory
Patent Information
- Application Number
- CN202311690998.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-11
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2043-12-11
AI Technical Summary
如，一种二维电网分布式缓存服务系统，将电网设备对象数据进行抽象，转换为内存对象存储在数据缓存模块，并根据电网设备之间的关系建立相应的关系索引，以此构建服务化的系统，缺点在于需要针对不同的业务需求开发定制化的接口和服务功能，如拓扑分析接口、站线变户接口等
[0023]有益效果：与现有技术相比，本发明具有如下显著优点：1、通过应用组和表管理逻辑单元实现了面向新型电力系统的业务模型匹配与融合，并通过内存与磁盘分离的方式解决了传统关系数据索引存储资源占用过大的问题；2、通过物化内存结构和分段索引法实现了面向对象属性的快速检索能力，避免了频繁的内存读写与磁盘操作，提升了查询效率；3、采用简化的API接口，通过公式化查询条件，替代传统的SQL查询引擎及接口，避免了复杂的关系运算，实现查询条件在接口侧的快速解析，提升面向对象的数据查询访问效率。
Smart Images
Figure CN117688030B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a real-time measurement system, and more particularly to a real-time measurement system based on materialized memory, belonging to the field of computer applications. Background Technology
[0002] The new power system, primarily based on new energy sources, is a crucial component of a clean, low-carbon, safe, and efficient energy system, characterized by its wide scope, numerous stages, high timeliness, and multi-dimensional nature. The data center for this new power system involves measurement data from power grid acquisition systems, non-electrical environments, and carbon extraction platforms. This data comes from diverse sources, large-scale data collection, significant differences in data formats, and stringent timeliness requirements, making the construction and processing of real-time measurement models extremely challenging. Therefore, how to construct real-time measurement models and provide methods for accessing massive amounts of data at high speed to meet the needs of new business applications such as real-time response to massive amounts of data and high-speed cross-disciplinary data sharing has become a critical issue that urgently needs to be addressed.
[0003] Currently, business systems primarily use relational models for data construction. This model, proposed in 1970 by IBM researcher Dr. E.F. Codd, uses two-dimensional tables to represent entities and the relationships between them, and is mostly stored in relational databases. The advantages of relational models are simplicity and ease of understanding and mastery for users; a relation is simply a two-dimensional table, and users can manipulate the database using simple query languages. The disadvantage is that in scenarios with complex relational schemas and large amounts of relational data, the time cost of relational operations generated by model associations is enormous, easily leading to problems such as untimely system response. With the development of big data technology, data warehouses and data marts are increasingly used to process large-scale measurement data and business models. In terms of data structure, a data warehouse is a subject-oriented, integrated collection of data, while a data mart is usually defined as a star schema or snowflake schema, generally composed of a fact table and several dimension tables. However, essentially, their relational models are still stored in larger-scale, multi-level relational databases in the form of two-dimensional tables, and the cost of relational operations remains high.
[0004] With the continuous development of computer hardware technology, technicians are leveraging the fast access capabilities of RAM to alleviate problems, but each approach has its own focus. For example, a two-dimensional power grid distributed caching service system abstracts power grid equipment object data, converts it into memory objects stored in a data caching module, and establishes corresponding relationship indexes based on the relationships between power grid equipment to build a service-oriented system. The drawback is the need to develop customized interfaces and service functions for different business needs, such as topology analysis interfaces and station / line / substation / customer interfaces. Another system and method for quickly materializing Java objects from database data provides a way to transfer data from the database and materialize it into data objects (e.g., Java objects), improving the speed of data transfer from virtual machine (VM) objects to the database. The drawback is that the information in the buffer needs to be known and transparent to both the native buffer and the VM buffer API, which is not very friendly to the flexibility and usability of diverse cross-disciplinary model data. A hash-index-based memory table model improves the speed of data querying in application systems by constructing a hash-index-based memory table model, mainly solving the retrieval efficiency of key operating parameters in the UnionPay system. However, a problem exists: insufficient consideration is given to the joint querying of model data and collected measurement data. Summary of the Invention
[0005] Purpose of the invention: The purpose of this invention is to provide a real-time measurement system based on materialized memory that reduces storage resource consumption and improves query efficiency.
[0006] Technical solution: The real-time measurement system based on materialized memory described in this invention includes: a source data layer, a storage service layer, and an application layer;
[0007] The source data layer includes an object model, which includes model attribute data and collected measurement data of the object. The object is a business entity, and the model attribute data includes static attributes and dynamic attributes.
[0008] The storage service layer includes a memory model service and a real-time database. The memory model service receives and processes model attribute data, and the real-time database receives and processes collected measurement data.
[0009] The memory model service includes a model storage module, a memory service module, and a memory model service API interface;
[0010] The model storage module is used to provide the functions of fixed storage and materialized loading of model attribute data through a relational database;
[0011] The memory service module is used to materialize cached model attribute data, build object indexes, and realize object-oriented attribute retrieval through segmented indexing; it includes an application group management unit, a table management unit, a main table management unit, and a point table management unit, as well as a materialized memory structure and a materialized memory index.
[0012] The memory model service API interface is used to provide a unified interface for object-oriented model storage and access.
[0013] The application layer is used to obtain relevant object models and measurement data through the memory model service API interface for various business systems and applications.
[0014] Furthermore, the static attributes are used to describe the static features of the object, the dynamic attributes are used as measurement points to describe the sampling attributes of the object, and the collected measurement data are used to describe the real-time sampled values of the object.
[0015] Furthermore, the memory model service and the real-time database are correlated through measurement points.
[0016] Furthermore, the application group management unit of the memory service module is used to manage business models of different specialties and realize the classification management of business models; the table management unit is used to manage and coordinate the main table management unit and the point table management unit, and coordinate the model, measurement data flow and related task distribution from a global perspective; the main table management unit is used to store the static attributes of objects; the point table management unit is used to store the dynamic attributes of objects.
[0017] Furthermore, the specific method for materializing and caching model attribute data in the memory service module is as follows: the measurement object to be stored is abstracted into a two-dimensional model table. When each model table is created, the system automatically generates a corresponding global UID to identify the object. When the two-dimensional model table is stored in the relational database, it corresponds to the main table management unit and the point table management unit, respectively. The main table stores all the static attributes of the object, and the point table stores the dynamic attributes or measurement points of the object.
[0018] Furthermore, the main table management unit and the point table management unit are associated through a global UID.
[0019] Furthermore, the constructed object index is built using a memory structure based on a hash map. The index exists in system memory, and the data inside the index is stored in a relational database in the form of a two-dimensional model table. The data corresponding to the index is materialized and loaded when the service starts, and is solidified and written back when the service exits.
[0020] Furthermore, the segmented indexing method creates indexes for the fields with retrieval functions in the two-dimensional model table of the object, with each field's index serving as a segment. Several segmented indexes are created, and each segment is independent of the others. During a query, the range query function for multiple fields is implemented segment by segment according to different query constraints and aggregation methods.
[0021] Furthermore, the primary key of the segmented index is the name of each field, and the value is a list of global UIDs of the device objects associated with each field.
[0022] Furthermore, the memory model service API interface adopts a simplified API interface, and realizes the parsing of query conditions on the interface side through formulaic query conditions.
[0023] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. It realizes the matching and integration of business models for new power systems through application groups and table management logic units, and solves the problem of excessive storage resource consumption of traditional relational data indexes by separating memory and disk; 2. It realizes the ability to quickly retrieve object-oriented attributes through materialized memory structure and segmented indexing method, avoiding frequent memory read and write and disk operations, and improving query efficiency; 3. It adopts a simplified API interface and replaces the traditional SQL query engine and interface with formulaic query conditions, avoiding complex relational operations, realizing fast parsing of query conditions on the interface side, and improving the efficiency of object-oriented data query and access. Attached Figure Description
[0024] Figure 1 This is a system schematic diagram according to an embodiment of the present invention;
[0025] Figure 2 This is a schematic diagram of the memory service module according to an embodiment of the present invention;
[0026] Figure 3 This is a schematic diagram of the segmented index of an example of the present invention;
[0027] Figure 4 This is a schematic diagram of the system query process in an embodiment of the present invention. Detailed Implementation
[0028] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0029] As attached Figure 1 As shown, this invention provides a real-time measurement system based on materialized memory, comprising: a source data layer, a storage service layer, and an application layer;
[0030] The source data layer includes an object model, which includes model attribute data and collected measurement data of the object. The object is a business entity, and the model attribute data includes static attributes and dynamic attributes.
[0031] The storage service layer includes a memory model service and a real-time database. The memory model service receives and processes model attribute data, and the real-time database receives and processes collected measurement data.
[0032] The memory model service includes a model storage module, a memory service module, and a memory model service API interface;
[0033] The model storage module is used to provide the functions of fixed storage and materialized loading of model attribute data through a relational database;
[0034] The memory service module is used to materialize cached model attribute data, build object indexes, and realize object-oriented attribute retrieval through segmented indexing; it includes an application group management unit, a table management unit, a main table management unit, and a point table management unit, as well as a materialized memory structure and a materialized memory index.
[0035] The memory model service API interface is used to provide a unified interface for object-oriented model storage and access.
[0036] The application layer is used to obtain relevant object models and measurement data through the memory model service API interface for various business systems and applications.
[0037] In the technical solution of this embodiment:
[0038] An object model was designed based on the traditional two-dimensional table model. Each object represents a business entity (such as a power meter, a power line, etc.), including static attributes, dynamic attributes, and collected measurement data. Static attributes describe the static characteristics of the object; dynamic attributes describe the sampling attributes of the object, also known as measurement points; and collected measurement data describes the real-time sampled values of the object.
[0039] It mainly relies on in-memory model services and real-time databases to store and manage static attributes, dynamic attributes, and collected measurement data separately. Static attributes and dynamic attributes are stored in the in-memory model service, and the stored data is also called model attribute data; collected measurement data is stored in the real-time database, and the two are linked and fused through measurement points.
[0040] The memory model service consists of a memory service module, a model storage module, and an API interface. The memory service module caches model attribute data using a memory structure materialization cache; the model storage module stores model attribute data in hardware to prevent data loss; and the API interface provides a unified interface for object-oriented model storage and access.
[0041] The memory service module comprises four logical units: application group management, table management, master table management, and point table management. The application group management unit manages business models for different specialties; the table management unit coordinates the master table and point table management units; the master table management unit stores static attributes of objects; and the point table management unit stores dynamic attributes of objects. The master table management unit and the point table management unit are linked via a global UID.
[0042] The memory service module constructs an object index using a hash MAP memory structure, enabling fast retrieval of object-oriented attributes.
[0043] The memory service module implements range queries for object-oriented fields using a segmented indexing method. This segmented indexing method involves creating hash indexes for each of the fields in an object that have retrieval functionality. The primary key is the field name, and the value is a list of global UIDs of the device objects associated with that field. During a query, range queries for multiple fields can be implemented segment by segment based on different query constraints and aggregation methods.
[0044] The memory service module uses a simplified API interface and replaces the traditional SQL query engine and interface with formulaic query conditions, enabling fast parsing of query conditions on the interface side and improving the efficiency of object-oriented data query and access.
[0045] The real-time measurement system based on materialized memory provided in this embodiment is divided into three layers from bottom to top: data source, storage service, and application. The data source layer mainly includes model attribute data and collected measurement data, primarily sourced from business systems, acquisition devices, and access protocols. The storage service layer mainly includes an in-memory model service and a real-time database, used to receive and process model attribute data and collected measurement data, respectively. The in-memory model service is the core component, comprising a model storage module, a memory service module, and a memory model service API interface. The model storage module includes a relational database, providing persistent storage and materialized loading functionality for model attribute data. The memory service module mainly consists of four logical units: application group management, table management, master table management, and point table management, as well as a materialized memory structure and a materialized memory index. The in-memory model service and the real-time database are associated through measurement points, and the in-memory model service API provides unified and rapid read / write access capabilities for model attribute data. The application layer is accessible to various business systems and applications, quickly obtaining relevant model and measurement data through the in-memory model service API interface.
[0046] The application group management unit of the memory service module described in this embodiment is used to manage business models of different specialties. Its main purpose is to achieve classified management of business models. The objects stored within the application group management unit can be specialties or specific business applications, such as marketing specialties or electricity information collection systems. The table management unit is used to manage and coordinate the master table and point table management units, coordinating the flow of models, measurement data, and related task distribution from a global perspective. The master table management unit stores the static attributes of objects, mainly handling the storage and access of model attribute data. The point table management unit stores the dynamic attributes of objects, mainly handling the storage and access of collected measurement data. The master table management unit and the point table management unit are associated through a global UID, which is automatically generated internally by the system and mainly used to identify different measurement objects. In a specific embodiment, the static attributes stored and managed by the master table management unit describe the device's file type data, including but not limited to serial number, attribute name, type, unit, and collection cycle. The dynamic attributes of the devices stored in the point table management unit vary depending on the device's function, including but not limited to three-phase current, three-phase voltage, active power, and reactive power.
[0047] Traditional relational index creation and maintenance are time-consuming, increasing with data volume. Furthermore, these indexes require physical space; not only does the table occupy data space, but each index also requires physical space. When adding, deleting, or modifying data in the table, the index also needs dynamic maintenance. Since the index is also stored on the physical disk, the update process slows down data maintenance. The materialized memory structure design described in this invention uses a hash map-based memory structure to build object indexes. Its key feature is that the index exists only in system memory; the index itself does not occupy physical space. The data within the index is stored in a relational database (i.e., a general relational database) in a two-dimensional table format. The data corresponding to the index is materialized and loaded only when the service starts and written back when the service exits, thus ensuring the separation of index memory and storage and enabling fast retrieval of object-oriented attributes. A specific materialized memory structure diagram is shown in the table below.
[0048] Table 1. Schematic diagram of materialized memory structure
[0049]
[0050] The model storage module of the memory model service primarily relies on a relational database for storage, used to permanently store model attribute data on hardware to prevent data loss. The data structure of the memory service module is as follows: Figure 2As shown, table1...tablei represent the stored measurement objects, which can be abstracted as two-dimensional model tables. Each model table is automatically generated by the system upon creation, using a corresponding global UID to identify the object. When stored in the relational database, the model tables correspond to two types of sub-tables, denoted as the main table and the point table. The main table stores all the static attributes of the object, while the point table stores the dynamic attributes, or measurement points, of the object. Generally, each object has multiple dynamic attributes; therefore, the measurement points stored in the point table are typically a measurement point set structure. Both the main table and the point table structure store the initially allocated global UID for data association.
[0051] The segmented indexing method in the memory service module involves creating hash indexes for each retrieval-enabled field in an object (two-dimensional table). Each field's index forms a segment. If an object has n indexed fields, n segmented indexes are created simultaneously, each segment being independent. The primary key of each segmented index is the field name, and the value is a list of global UIDs of the device objects associated with that field. Simultaneously, for each global UID, a hash map is used to construct the corresponding measurement point name index structure. Since the global UID ensures the uniqueness of object descriptions across various business systems and achieves separate storage association of static and dynamic attributes, it enables unique allocation of index memory resources. Furthermore, because the entire index is stored in the memory structure and does not occupy physical storage, this method reduces index storage resource consumption by approximately 30-50%, further improving retrieval efficiency while optimizing storage. A specific diagram of the segmented index is shown below. Figure 3 As shown.
[0052] The memory service module employs a simplified API interface, replacing the traditional SQL query engine and interface with formulaic query conditions. This enables rapid parsing of query conditions on the interface side, improving the efficiency of object-oriented data query access. During queries, range queries targeting multiple fields can be implemented in segments based on different query constraints and aggregation methods. Query conditions can be selected according to business scenarios, including greater than ">", less than "<", equal to "=", greater than or equal to ">=", less than or equal to "<=", not equal to "!=", and fuzzy query like "?" (supporting single-character wildcard "?" and multi-character wildcard "*"). Condition constraints can include intersection, union, etc. The specific query data structure is illustrated in the table below.
[0053] Table 2 shows the query data structure.
[0054]
[0055] As attached Figure 4In the specific implementation example of the query process shown, taking application A and table T as an example, assuming that it is necessary to query all equipment with a voltage level of 25kV, city code of 32401, and substation code of 10009, the specific query implementation steps are as follows:
[0056] 1. First, construct a query structure QueryStatement Q to store different query conditions;
[0057] 2. Initialize Q, and then insert the field name, query symbol, parameter value and other query conditions in sequence, such as Q.insert(“voltage_level”,”=”,”25”);
[0058] 3. Call the query interface, JReq_MS_Search(A,T,Q)
[0059] 4. Locate application group A, table T;
[0060] 5. For each query condition, perform a segmented search operation, find the index field i, and obtain the corresponding UID set Ui;
[0061] 6. Based on the query constraints, return a set of UIDs. If it is an intersection, return the intersection I; if it is a union, return the union U.
[0062] 7. Search for the set of test point names and the main table fields based on the UID set.
[0063] This invention proposes a real-time measurement system based on materialized memory. It achieves business model matching and fusion for new power systems through application groups and table management logic units, and solves the problem of excessive storage resource consumption in traditional relational data indexes by separating memory and disk. Secondly, it achieves rapid retrieval of object-oriented attributes through a materialized memory structure and segmented indexing, avoiding frequent memory read / write and disk operations and improving query efficiency. Finally, it adopts a simplified API interface, replacing the traditional SQL query engine and interface with formulaic query conditions, avoiding complex relational operations and enabling rapid parsing of query conditions on the interface side, thus improving the efficiency of object-oriented data query access. The materialized memory-based memory model service system generated by this invention is suitable for real-time systems represented by new power systems, providing methods for constructing real-time measurement models and for rapid access to measurement models and collected measurement data, meeting the requirements of new business applications such as real-time response to massive data and high-speed cross-disciplinary data sharing.
Claims
1. A real-time measurement system based on materialized memory, characterized in that, include: Source data layer, storage service layer, and application layer; The source data layer includes an object model, which includes model attribute data and collected measurement data of the object. The object is a business entity, and the model attribute data includes static attributes and dynamic attributes. The storage service layer includes a memory model service and a real-time database. The memory model service receives and processes model attribute data, and the real-time database receives and processes collected measurement data. The memory model service includes a model storage module, a memory service module, and a memory model service API interface; The model storage module is used to provide the functions of fixed storage and materialized loading of model attribute data through a relational database; The memory service module is used to materialize cached model attribute data, build object indexes, and realize object-oriented attribute retrieval through segmented indexing; it includes an application group management unit, a table management unit, a main table management unit, and a point table management unit, as well as a materialized memory structure and a materialized memory index. The memory model service API interface is used to provide a unified interface for object-oriented model storage and access. The application layer is used to obtain relevant object models and measurement data through the memory model service API interface for various business systems and applications.
2. The real-time measurement system based on materialized memory according to claim 1, characterized in that, The static attributes are used to describe the static features of the object, and the dynamic attributes, as measurement points, are used to describe the sampling attributes of the object; the collected measurement data are used to describe the real-time sampled values of the object.
3. The real-time measurement system based on materialized memory according to claim 2, characterized in that, The memory model service and the real-time database are linked through measurement points.
4. The real-time measurement system based on materialized memory according to claim 1, characterized in that, The application group management unit of the memory service module is used to manage business models of different specialties and realize the classification management of business models; the table management unit is used to manage and coordinate the main table management unit and the point table management unit, and coordinate the model, measurement data flow and related task distribution from a global perspective; the main table management unit is used to store the static attributes of objects; the point table management unit is used to store the dynamic attributes of objects.
5. The real-time measurement system based on materialized memory according to claim 1, characterized in that, The specific method for the materialized caching model attribute data in the memory service module is as follows: the measurement object to be stored is abstracted into a two-dimensional model table. When each model table is created, the system automatically generates a corresponding global UID to identify the object. When the two-dimensional model table is stored in the relational database, it corresponds to the main table management unit and the point table management unit, respectively. The main table stores all the static attributes of the object, and the point table stores the dynamic attributes or measurement points of the object.
6. The real-time measurement system based on materialized memory according to claim 5, characterized in that, The main table management unit and the point table management unit are associated through a global UID.
7. The real-time measurement system based on materialized memory according to claim 5, characterized in that, The constructed object index is built using a memory structure based on a hash map. The index exists in system memory, and the data inside the index is stored in a relational database in the form of a two-dimensional model table. The data corresponding to the index is materialized and loaded when the service starts, and is solidified and written back when the service exits.
8. The real-time measurement system based on materialized memory according to claim 7, characterized in that, The segmented indexing method involves creating indexes for the fields with retrieval functions in the two-dimensional model table of the object. Each index is considered a segment, and several segmented indexes are created. Each segment is independent of the others. During a query, the range query function for multiple fields is implemented segment by segment according to different query constraints and aggregation methods.
9. The real-time measurement system based on materialized memory according to claim 8, characterized in that, The primary key of the segmented index is the name of each field, and the value is a list of global UIDs of the device objects associated with each field.
10. The real-time measurement system based on materialized memory according to claim 1, characterized in that, The memory model service API interface adopts a simplified API interface, and realizes the parsing of query conditions on the interface side through formulaic query conditions.