Dynamic integration system and method of electric power large model platform and heterogeneous system

Abstract

The invention relates to the technical field of information, and discloses a dynamic integration system and method for an electric power large model platform and heterogeneous systems, and the dynamic integration system comprises a protocol adaptation layer which is used for processing original heterogeneous data of a plurality of heterogeneous systems through a three-layer conversion engine, obtaining a standardized data stream, and dynamically adapting a transmission protocol of the standardized data stream; the service perception layer is used for generating a real-time routing rule by adopting a three-layer linkage mechanism based on the standardized data flow to control on-demand routing of the standardized data flow so as to obtain a routed data flow; the data pipeline layer is used for acquiring a binary log event of each heterogeneous system in real time, processing the routed data flow through a three-layer guarantee mechanism, generating training data consistent with the binary log event, and outputting the training data to each heterogeneous system; in complex cross-platform data interaction, high consistency and high reliability of data are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, in particular to a dynamic integration system and method of a power large model platform and a heterogeneous system. BACKGROUND

[0002] The current artificial intelligence platform of the power industry adopts an architecture mode of "sample library-model library-operation platform" (i.e., two libraries and one platform). With the rise of large model technology, the power large model platform needs to be deeply integrated with the existing "two libraries and one platform" to realize data sharing, function reuse, and full-process collaboration, but in the traditional integration scheme, each system uses private API protocols and data formats, and needs to customize the development of adapters, which has poor extensibility; when new components or systems are upgraded, the integration logic needs to be adjusted manually, and real-time sensing of service changes cannot be realized; and there is a lack of distributed coordination mechanism for cross-system transactions, which is prone to data version conflicts.

[0003] Therefore, how to solve the problems of lack of dynamic adaptation and weak data consistency guarantee in complex cross-platform data interaction has become a technical problem to be solved by those skilled in the art. SUMMARY

[0004] The present application provides a dynamic integration system and method of a power large model platform and a heterogeneous system, which solves the problems of lack of dynamic adaptation and weak data consistency guarantee in complex cross-platform data interaction.

[0005] To solve the above technical problems, the first aspect of the present application provides a dynamic integration system of a power large model platform and a heterogeneous system, comprising a protocol adaptation layer, a service sensing layer and a data pipeline layer; wherein,

[0006] The protocol adaptation layer is used to process the original heterogeneous data received from a plurality of heterogeneous systems through a three-layer conversion engine to obtain a standardized data stream, and dynamically adapt the transmission protocols of each of the heterogeneous systems;

[0007] The service sensing layer is used to generate real-time routing rules based on the standardized data stream using a three-layer linkage mechanism, and control the on-demand routing of the standardized data stream through the real-time routing rules to obtain a routed data stream;

[0008] The data pipeline layer is used to obtain binary log events of each of the heterogeneous systems in real time, and processes the routed data stream through a three-layer guarantee mechanism to make the generated training data consistent with the binary log events, and sends the training data to a power large model platform.

[0009] The second aspect of the present application provides a dynamic integration method of a power large model platform and a heterogeneous system, comprising:

[0010] The original heterogeneous data received from a plurality of heterogeneous systems is processed by a three-layer conversion engine to obtain a standardized data stream, and the transmission protocols of the plurality of heterogeneous systems are dynamically adapted;

[0011] Based on the standardized data stream, a three-layer linkage mechanism is used to generate real-time routing rules, and the standardized data stream is controlled to be routed on demand through the real-time routing rules to obtain a routed data stream;

[0012] Binary log events of the plurality of heterogeneous systems are obtained in real time, the routed data stream is processed by a three-layer guarantee mechanism to make the generated training data consistent with the binary log events, and the training data is sent to a power large model platform.

[0013] Compared with the prior art, the beneficial effects of the embodiment of the present application are at least one of the following:

[0014] (1) The data standardization during multi-platform interaction is realized by the three-layer conversion engine of the protocol adaptation layer, the transmission protocols of the plurality of heterogeneous systems are dynamically adapted, the interface heterogeneity problem during multi-platform interaction is eliminated, the expansion cost is reduced from exponential to linear, the standardized input is provided for subsequent components, the integration complexity is reduced, and the like;

[0015] (2) The three-layer linkage mechanism of the service perception layer is used to generate real-time routing rules to realize dynamic scheduling of the data stream, and the response to the second-level service change is realized without manual intervention;

[0016] (3) The data pipeline layer relies on the standardized data and stable services provided by the previous two layers to realize efficient and reliable data synchronization, and the routed data stream is processed by the three-layer guarantee mechanism, so that the deviation of the generated training data is close to zero, and the strong consistency between the training data and the source end is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a structural diagram of a dynamic integration system of a power large model platform and a heterogeneous system provided by an embodiment of the present application;

[0019] Figure 2 is a flowchart of a dynamic integration method of a power large model platform and a heterogeneous system provided by an embodiment of the present application;

[0020] REFERENCE NUMERALS:

[0021] Among them, 10, protocol adaptation; 20, service perception layer; 30, data pipeline layer. DETAILED DESCRIPTION

[0022] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0023] In the description of this application, the terms "first," "second," "third," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first," "second," "third," etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0024] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the two components. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the system or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0025] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by those skilled in the art. The terms used in this specification are only for describing specific embodiments and are not intended to limit the present invention. Those skilled in the art will understand the specific meanings of the above terms in this application according to specific circumstances.

[0026] In the prior art, if a dynamic integration system accesses diversified heterogeneous systems, that is, inventory systems (such as CSV / Parquet data of a sample library and ONNX / PMML format of a model library), the traditional static interface binding leads to exponential growth of integration cost with the system scale; and when the data service of the library is newly added or the interface of the model library is upgraded, the dynamic integration system cannot automatically discover and adapt to the changes and needs to be manually intervened to interrupt the service; and the power large model platform relies on real-time data of the sample library during training, but the traditional ETL tool cannot guarantee the strong consistency of cross-platform incremental data, leading to deviation of the training result. Based on this, the application provides a dynamic integration system for realizing seamless connection, efficient collaboration and dynamic interaction between a power large model platform and heterogeneous systems. The protocol adaptation layer in the system can reduce the expansion cost between heterogeneous platforms from exponential to linear; the service awareness layer can realize second-level effect of changes; and the data pipeline layer can ensure that the deviation of training data is close to zero, thereby solving the problems of lack of dynamic adaptation and weak data consistency guarantee in complex cross-platform data interaction.

[0027] In an embodiment, as shown in Figure 1 the first aspect of the application provides a dynamic integration system of a power large model platform and heterogeneous systems, comprising a protocol adaptation layer, a service awareness layer and a data pipeline layer; wherein,

[0028] the protocol adaptation layer is configured to process original heterogeneous data received from a plurality of heterogeneous systems through a three-layer conversion engine to obtain a standardized data stream, and dynamically adapt the transmission protocols of the heterogeneous systems;

[0029] In an embodiment, the protocol adaptation layer comprises a physical layer, a semantic layer and a control layer; wherein,

[0030] the physical layer is configured to provide a unified connection interface to interface the transmission protocols of the heterogeneous systems, and establish an asynchronous non-blocking I / O model to receive original heterogeneous data from the heterogeneous systems;

[0031] the semantic layer is configured to build a general data model, and convert the original heterogeneous data into a standardized data stream through the general data model;

[0032] the control layer is configured to deploy Adapter Operator in Kubernetes, and automatically inject a Sidecar container to carry protocol conversion logic, so as to realize dynamic adaptation of the transmission protocols of the heterogeneous systems.

[0033] Specifically, the application uses the physical layer, the semantic layer and the control layer as a three-layer conversion engine to realize seamless compatibility of the transmission protocols between the plurality of heterogeneous systems and the dynamic integration system, so that the expansion cost between the heterogeneous platforms is reduced from exponential to linear.

[0034] The physical layer is used for decoupling, which can provide a unified connection interface for various heterogeneous systems through customized Connector SDK, connect the transmission protocols (such as gRPC / HTTP2 / RabbitMQ) of various heterogeneous systems, establish an efficient asynchronous non-blocking I / O model channel, receive raw heterogeneous data from various heterogeneous systems, and thus improve the concurrent processing capability.

[0035] The Connector SDK in the physical layer initializes the corresponding heterogeneous system source end connection according to the configured protocol type (such as gRPC, HTTP2, RabbitMQ); for gRPC, a bidirectional stream is established using the HTTP / 2 support of Netty; for RabbitMQ, an AMQP channel is established and an exchange and a queue are declared; when there is data to be transmitted, the physical layer directly obtains raw heterogeneous data from the source memory (such as the receiving buffer) and maps the data directly to the network buffer through DirectByteBuffer without going through user-mode memory copying; at the same time, a zero-copy buffer design is adopted in the physical layer, DirectByteBuffer of Netty is used to directly use off-heap memory, avoiding data copying between JVM heap memory and local memory, and reducing 60% memory copying overhead; an event-driven model (such as EventLoop of Netty) is used, one thread handles multiple connections, and a callback function is triggered when data is ready, without blocking the thread.

[0036] The semantic layer is used to convert data of different formats into a universal data model, realize semantic compatibility, and then convert the received raw heterogeneous data into a standardized data stream through the constructed universal data model.

[0037] The control layer is used to deploy Adapter Operator in Kubernetes to realize the automatic operation and management of protocol conversion in the Kubernetes environment; it customizes KubernetesOperator through Kubernetes Adapter Operator, and thus manages the life cycle of the protocol adaptation component; automatically injects Sidecar to inject the protocol conversion logic into the application Pod in the form of a Sidecar container, so as to realize the dynamic adaptation of the transmission protocols of various heterogeneous systems.

[0038] The control layer intercepts the Pod creation request through the Mutating Webhook mechanism of the Operator based on Kubernetes; whether the protocol adaptation Sidecar needs to be injected is determined according to an annotation; a container (using a customized image) running protocol conversion logic is injected in the Pod to dynamically adjust according to the actual transmission protocol of the heterogeneous system: when the transmission protocol of the heterogeneous system changes, only the protocol conversion logic in the Sidecar container needs to be updated, without modifying the main application code; the dynamic adaptation mechanism adopted enables the system to flexibly cope with the transmission protocol changes of different heterogeneous systems, realizes dynamic adaptation of the transmission protocols of the heterogeneous systems, and configures a shared memory volume or a Unix domain socket for communication; the Sidecar container exposes Prometheus indicators (such as request delay, error rate, and concurrency), uses a token bucket algorithm to realize flow control, and sets an initial rate according to historical data, and dynamically adjusts the filling rate of the token bucket according to real-time indicators (such as an error rate increase) collected by Prometheus (for example, the rate is reduced when the error rate is greater than 5%).

[0039] The control layer is also used to configure a specified injected fault type (such as network delay, packet loss, and network partition), and inject a fault at the network layer of the Pod using Chaos Mesh or a custom network interference tool to simulate network partition and verify robustness.

[0040] The traditional scheme relies on static interface binding, and an adapter (such as an XML / binary conversion module) needs to be customized and developed for each newly added system, and the extension cost increases exponentially with the system size; the three-layer protocol conversion engine of the protocol adaptation layer solves the heterogeneous system access problem, provides a unified interface for data flow conversion, realizes non-invasive dynamic access of heterogeneous systems, and reduces integration complexity; the physical layer eliminates interface heterogeneity, and solves the transmission efficiency problem through zero-copy and non-blocking I / O optimization performance; the semantic layer realizes heterogeneous data unification through dynamic Schema and efficient encoding and decoding, and further solves the data compatibility problem; the control layer realizes automatic deployment and elastic control through the Kubernetes Operator, and solves the operation and maintenance management problem; the three-layer cooperation of the application realizes seamless compatibility and efficient conversion of protocols.

[0041] In an embodiment, the semantic layer includes a model construction module, a model analysis module, and a data mapping module; wherein,

[0042] The model construction module is configured to construct the general data model based on an Apache Avro dynamic Schema;

[0043] The model analysis module is configured to parse the general data model, generate an operator dependency tree, and extract tensor metadata from each of the original heterogeneous data according to the operator dependency tree;

[0044] The data mapping module is configured to perform standardization processing on the tensor metadata to obtain tensor features, and map the tensor features to standardized data streams using a dynamic Schema construction engine.

[0045] Specifically, the model construction module in the semantic layer utilizes the dynamic Schema feature of Apache Avro, dynamically defines the structure of the general data model according to the business requirements and data characteristics of the power industry, and supports complex data types such as nesting, array, and enumeration, which can flexibly describe various entities and relationships in power data. The constructed general data model Schema is registered in the Schema registration center for query and use by other modules. At the same time, the module provides a Schema version management function to ensure the compatibility and traceability of the data model.

[0046] The model analysis module first reads the Schema of the general data model, and generates an operator dependency tree according to the field definition and dependency relationship in the Schema. The operator dependency tree describes each operation step required for extracting tensor metadata from the original heterogeneous data and the dependency relationship between them. Then, according to the operator dependency tree, the model analysis module extracts the required tensor metadata from the original heterogeneous data. During the extraction process, data analysis, field mapping, data conversion, and other operations are performed according to actual requirements to ensure that the extracted metadata meets the requirements of the general data model.

[0047] The data mapping module performs standardized processing on the extracted tensor metadata, including data cleaning, format conversion, unit unification, etc., to generate standardized tensor features; a dynamic Schema construction engine is used to dynamically generate mapping rules based on the standardized tensor features and the Schema of the general data model; wherein the mapping rules define how to map the tensor features to each field in the standardized data stream; finally, the data mapping module maps the standardized tensor features to the standardized data stream according to the generated mapping rules; wherein during the mapping process, data aggregation, calculation, format conversion, etc. are used according to actual needs to ensure that the generated standardized data stream meets business needs and data specifications, such as converting ONNX's TensorProto::DataType to Avro basic types (such as FLOAT32 to avro::FLOAT) in the data type standardization stage, and converting a multi-dimensional shape array (such as [1, 3, 224, 224]) to an Avro array structure {"type": "array", "items": "int"}; and extending the sparse_indices field for sparse tensors to achieve special structure compatibility; the dynamic Schema construction engine automatically generates nested Record based on the tensor features (example Schema contains tensor_name string field, dtype enumeration field and shape integer array field), and avoids special character conflicts through name normalization processing (such as "conv1.weight" to a legal named field).

[0048] In addition, the semantic layer can also automatically generate binary codecs: using zero-copy serialization technology to directly reference the tensor shape memory address and applying Netty's DirectByteBuffer to reduce 60% memory overhead, and integrating LEB128 variable-length compression technology (such as integer 56 encoded as 0x38) and single-byte data type enumeration markers (such as FLOAT32 marked as 0x01) at the encoding strategy level, which has been measured and compared to optimize the effect significantly: the serialization time of thousands of tensors is compressed to, the data volume is reduced, and the memory management is optimized from heap memory caching to direct buffer access. This technology covers the complete link from model input, dependency tree generation, metadata extraction, dynamic Schema construction to codec generation and serialization stream output, its innovation lies in realizing dynamic compatibility of new operators through Schema registration mechanism, automatically adapting semantic changes when ONNX opset is upgraded (such as Opset15 layer normalization operator), and using GPU to offload SHA-256 verification to improve efficiency.

[0049] The semantic layer in the application realizes dynamic Schema construction, efficient analysis and standardized mapping of original heterogeneous data through close cooperation of a model construction module (dynamic Schema support, flexible adaptation to data changes), a model analysis module (operator dependency tree analysis, improved data extraction efficiency) and a data mapping module (standardized tensor features, enhanced data interoperability; dynamic mapping engine, simplified data adaptation process). The architecture design not only improves the flexibility and efficiency of data processing, but also enhances the interoperability and quality of data, providing strong support for system integration and data sharing in the power industry.

[0050] In an embodiment, the model analysis module is further configured to:

[0051] perform Graph analysis on the general data model, identify an input node of the general data model as a root node, and traverse graph nodes layer by layer to establish a connection relationship according to the types of the nodes in the process of traversal;

[0052] build a tree topology structure through recursive operators, dynamically prune and remove redundant operators, and generate independent sub-trees for control flow operators;

[0053] extract key data of each operator, generate an operator dependency tree based on the connection relationship, using the tree topology structure, the pruning result and the independent sub-trees, and extract tensor metadata from each original heterogeneous data according to the operator dependency tree.

[0054] Specifically, the application takes ONNX model access as an example to illustrate the specific working principle of the model analysis module.

[0055] Working principle:

[0056] First, load the model, use the Graph module of ONNX Runtime to load and analyze the ONNX model file (.onnx), the nodes in Graph represent the operators in the model, and the edges represent the connection relationship between the nodes, and then identify the input node of the general data model and take it as the root node; traverse the calculation graph nodes (NodeProto), and establish a connection relationship according to the input and output fields of each node.

[0057] The recursive algorithm is used to recursively analyze the input source of each operator, and a tree-shaped topological structure is constructed according to the connection relationship in the Graph (for example, the input of a convolution layer is the output of the previous layer); wherein each node in the tree-shaped structure represents an operator, and the child node represents the subsequent processing step dependent on the parent node; redundant operators such as Identity that do not change data are removed for dynamic pruning; wherein the redundant operator refers to those operators (such as conditional branching operators, loop operators, etc.) that do not contribute to the final result or can be replaced by other operators: and an independent sub-tree is generated for the control flow operator Loop; wherein the independent sub-tree contains all the subsequent processing steps dependent on the control flow operator, forming a relatively independent processing unit.

[0058] The key data of various operators (such as convolution kernel size, step size) are extracted to generate an operator dependency tree based on the obtained connection relationship, using the tree-shaped topological structure, the pruning result and the independent sub-tree; wherein the operator dependency tree clearly indicates the various operation steps required to extract tensor metadata from the original heterogeneous data and the dependency relationship between them; finally, the tensor metadata (data type and dimension information of the tensor) is extracted from each original heterogeneous data according to the operator dependency tree through data parsing, field mapping, data conversion and other operations to ensure that the extracted metadata meets the requirements of the general data model.

[0059] The model analysis module in the application realizes deep analysis and optimization of the general data model through steps such as Graph analysis, tree-shaped topological structure construction, dynamic pruning and independent sub-tree generation; not only improves the accuracy and efficiency of data processing, but also enhances the flexibility and maintainability of the system, providing strong support for system integration and data sharing in the power industry.

[0060] In addition, the protocol adaptation layer in the application can also use Java Instrumentation API to realize class dynamic loading, add Parquet adapter, and then support adding / removing plugins at runtime, with a service jitter time of less than 100ms; plugin isolation: each adapter runs in an independent ClassLoader, avoiding dependency conflicts, solving the "adapter explosion" problem caused by interface heterogeneity, and significantly improving the extension efficiency.

[0061] The service-aware layer is configured to generate real-time routing rules based on the standardized data stream using a three-layer linkage mechanism, and control the on-demand routing of the standardized data stream through the real-time routing rules to obtain a routed data stream.

[0062] In an embodiment, the service-aware layer includes a service registration module, a change-aware module and a policy enforcement module; wherein,

[0063] The service registration module is configured to automatically submit a service contract to the Nacos registration center through an Envoy Sidecar agent in response to the standardized data stream, and trigger a continuous health check at the same time.

[0064] The change perception module is configured to intelligently analyze the interface signature of each of the heterogeneous systems after the continuous health check passes, and generate real-time routing rules according to the analysis result.

[0065] The policy implementation module is configured to hot load the real-time routing rules, establish a double-flow channel to control the standardized data stream to be divided into an old version service and a new version service and routed as needed, and obtain a routed data stream.

[0066] Specifically, when the power big model platform needs to integrate a new service of the sample library or the model library (that is, the heterogeneous system), the application adopts a service grid to control the flow through a three-layer linkage mechanism (that is, the service registration module, the change perception module and the policy implementation module) to realize second-level dynamic integration; wherein,

[0067] The service registration module responds to the standardized data stream, that is, when a new service of the sample library is online (that is, the standardized data stream is received), a service contract is automatically submitted to the Nacos registration center through the built-in Envoy Sidecar agent, including endpoint path (such as / api / v1 / sensor-data), protocol version (such as gRPC / HTTP2), QPS flow limiting threshold and other metadata, while triggering a continuous health check (verifying the service survival state every 5 seconds, and continuously three times are all survival states, which is passed health check). If the health check fails, the Envoy Sidecar agent will remove the service from the service list to avoid routing the flow to the unavailable service; at the same time, an alarm mechanism is triggered to inform the operation and maintenance personnel to handle.

[0068] The change perception module is used to capture contract change events through a long connection subscription mechanism by Istio control plane after the continuous health check passes, and to perceive the update of service interface in real time by monitoring the interface signature change of heterogeneous systems; wherein the interface signature includes interface name, parameter list, return value type and other information; and then start the intelligent analysis engine: intelligently analyze the interface signature of each heterogeneous system, and generate real-time routing rules (such as routing all requests containing " / sensor-data" path to the new service, and for example, gray shunting and protocol conversion) according to the analysis result. The generated real-time routing rules will be updated to the rule library, and the policy implementation module will be notified through message queue or event bus to load and apply. Subsequently, only the changed configuration fragments (rather than the full configuration) are issued through the incremental xDS protocol to avoid network bandwidth waste. The intelligent analysis process of the interface signature is as follows: first, generate path matching rules according to the endpoint path, such as converting the / sensor-data path to the prefix matching strategy of prefix: " / api / v1 / sensor-data", and automatically deriving the regular expression regex: " / sensor / [a-zA-Z0-9]+" if the path contains dynamic parameters (such as / sensor / {id}); second, dynamically insert protocol conversion middleware according to the protocol type, such as automatically injecting a JSON-Protobuf bidirectional converter when detecting that the new and old services are REST and gRPC protocols respectively, and completing seamless conversion of request and response formats according to the method name (such as GetSensorData) and parameter structure in the interface signature; then initialize the traffic splitting strategy based on the gray release demand to generate the weight splitting rule, that is, the routing rule, and synchronously inject the fault tolerance rule of fuse -by analyzing the health check data to dynamically set the consecutive5xxErrors threshold and retry strategy (such as 3 attempts). The routing rules generated in this process are the same technical features as the configuration of the Envoy proxy loaded by the xDS protocol in the subsequent policy implementation stage, realizing the closed loop from rule automatic generation to second-level dynamic implementation.

[0069] The policy implementation module uses the Envoy proxy deployed in the large model platform to hot load real-time routing rules within 300 milliseconds, so that the routing rules can take effect without restarting the service, realizing seamless upgrade; in order to realize on-demand routing and gray release, a double traffic lane is established to control the standardized data flow to be divided into old version service and new version service (the old version service before change and the new version service after change) and routed on demand, that is, 95% of the production traffic maintains the old version service to ensure business continuity, and 5% of the traffic is introduced into the new version for gray test, and the data flow after routing is obtained.

[0070] To address the defect of traditional solutions that cannot automatically perceive service changes, this solution pioneered a service grid linkage control flow: the service perception layer implements intelligent routing control of standardized data streams through a three-layer linkage mechanism consisting of a service registration module (automatic service registration and health check to improve system reliability), a change perception module (intelligent parsing of interface signatures to generate real-time routing rules), and a policy enforcement module (hot loading and dual traffic lanes to achieve seamless upgrades and on-demand routing). This architecture design not only improves the reliability and flexibility of the system, but also optimizes resource utilization and supports advanced functions such as grayscale releases and A / B testing, providing strong support for system upgrades and operation and maintenance in the power industry.

[0071] In one embodiment, the policy validation module is further configured to:

[0072] When an anomaly is detected in the new version service, the circuit breaker rollback mechanism is triggered to automatically block the new version service with the anomaly and intelligently roll back to the stable new version service based on the topological relationship diagram between the new version services;

[0073] When it is detected that there is no abnormality in the new version service, the full switching mechanism is used to automatically and smoothly transition the traffic ratio of the new version service, and control the old version service to gracefully go offline after completing the existing requests.

[0074] Specifically, the policy validation module in the present invention is also used to detect that a new version of the service is abnormal (such as the error rate exceeds the 10% threshold), and the system immediately triggers the circuit breaker rollback mechanism, automatically blocks the traffic of the problematic version and sends multi-level alarms (SMS / email / monitoring screen), where the blocking method may include routing the traffic to the old version service, returning an error response, or enabling a backup service, etc. At the same time, the policy validation module will maintain the topological relationship diagram between the new version services, record the dependencies and call chains between the services, and after the circuit breaker, the module will analyze the topological relationship diagram to find other new version services that are directly or indirectly related to the abnormal service and evaluate the stability of these services; then, based on the topological relationship diagram between the new version services, the module will intelligently select the stable new version service as the rollback target, without the need for manual intervention throughout the process.

[0075] When the abnormality of the nonexistence of the new version service is detected, and when the new service passes the 2-hour gray verification, a full switching mechanism is started to automatically smoothly transition the traffic proportion of the new version service, that is, the traffic of the new version service is gradually increased according to a preset traffic proportion (such as 10%, 20%, etc.), and the traffic of the old version service is gradually reduced; the adjustment of the traffic proportion can be realized by updating the routing rules or configuring the traffic management strategy. During the process of the smooth transition of the traffic proportion, the strategy taking effect module continuously monitors various indicators of the system to ensure that the system can run smoothly; when the traffic proportion of the new version service reaches 100%, the strategy taking effect module controls the old version service to be gracefully offline after completing the inventory request; the offline process includes stopping receiving new requests, processing all inventory requests, releasing resources and the like, so as to ensure the integrity and consistency of the business data. The whole process realizes zero service interruption.

[0076] In addition, for the protocol change scenario (such as REST to gRPC), the protocol converter built in the grid automatically translates the request format, and the front end does not need to modify any calling code to seamlessly access the new service.

[0077] The strategy taking effect module in the application adopts a fuse rollback mechanism to ensure system stability; the full switching mechanism is used to automatically smoothly transition the traffic proportion of the new version service, so as to avoid the impact of traffic surge on the system and ensure the smooth running of the system; the fuse rollback and full switching mechanisms realize automatic operation and maintenance, reduce the need for manual intervention, and improve the operation and maintenance efficiency.

[0078] The data pipeline layer is used for acquiring binary log events of each of the heterogeneous systems in real time, processing the data stream after routing through a three-layer guarantee mechanism, so that the generated training data is consistent with the binary log events, and sending the training data to a power large model platform;

[0079] In an embodiment, the data pipeline layer includes a change capture module, a transaction coordination module and a conflict detection module; wherein,

[0080] The change capture module is used for capturing binary log events of each of the heterogeneous systems in real time through a CDC technology;

[0081] The transaction coordination module is used for decomposing cross-platform interactions between the dynamic integration system and each of the heterogeneous systems into a sequence of atomic sub-transactions, pre-registering compensation transactions for each atomic sub-transaction, and starting the compensation transaction when the atomic sub-transaction is monitored to fail, so that the cross-platform interactions are all successful;

[0082] The conflict detection module is configured to perform conflict detection on the routed data stream, process the routed data stream with conflicts by using a preset algorithm, combine the processing result with the routed data stream without conflicts into training data consistent with the binary log event, and send the training data to the power large model platform.

[0083] Specifically, the application relies on the standardized data and stable services provided by the first two layers, and uses the change capture module, the transaction coordination module and the conflict detection module as a three-layer guarantee mechanism to guarantee the strong consistency of the training data with the source end; wherein,

[0084] The change capture module uses the CDC (Change Data Capture) technology to capture data change events in real time by listening to the binary logs (such as MySQL binlog, Oracle redo log, etc.) of various heterogeneous systems; and publishes the data change events to the Kafka message queue through the Debezium connector, so that the stream processing layer uses the Flink SQL engine for millisecond-level data processing.

[0085] The transaction coordinator based on the Saga mode built in the transaction coordination module disassembles the cross-platform interaction between the dynamically integrated system and each heterogeneous system into a sequence of atomic sub-transactions (such as sample library data update→training platform model parameter writing), to ensure the atomicity of "writing to the large model platform" and "updating the sample library state"; wherein, each atomic sub-transaction represents an independent operation unit, and has the ACID (Atomicity, Consistency, Isolation and Durability) characteristics; then, a compensation transaction (such as CompensateDelete for rolling back the writing operation) is pre-registered for each atomic sub-transaction; wherein, the compensation transaction is used to perform a rollback operation when the atomic sub-transaction fails, to ensure the consistency of the data, and the definition and implementation of the compensation transaction need to be written according to the specific business logic. Finally, when the atomic sub-transaction fails, the corresponding compensation transaction is started to make the cross-platform interaction successful; that is, when a sub-transaction fails (such as sample library service timeout), the coordinator immediately starts the chain compensation process: triggers the compensation operation of the successful sub-transaction in reverse order, and performs data restoration (such as restoring the historical value of the device voltage) by accurately positioning the power equipment ID and timestamp. The whole process uses an asynchronous retry mechanism and a timeout control to ensure that the rollback response is 120ms on average in the provincial power grid measurement, and the transaction log is persisted to ensure recovery after interruption.

[0086] The conflict detection module is used for conflict detection on the routed data stream, a preset algorithm is used for processing the routed data stream with conflict, and the processing result is combined with the routed data stream without conflict into training data with strong consistency with the binary log event.

[0087] The data pipeline layer in the application realizes efficient processing on the routed data stream through the three-layer guarantee mechanism of the capture module (real-time data capture, improving data timeliness), the transaction coordination module (transaction coordination mechanism, guaranteeing the success rate of cross-platform interaction) and the conflict detection module (conflict detection and processing, ensuring data consistency), and ensures that the generated training data has strong consistency with the binary log event. This architecture design not only improves the timeliness and accuracy of data, but also guarantees the success rate of cross-platform interaction and the consistency of data, and provides strong support for the training and updating of the power big model platform.

[0088] In an embodiment, the conflict detection module is further configured to:

[0089] The routed data stream is marked with a vector clock, and the routed data stream is subjected to conflict detection based on the vector clock through a Version vectors algorithm;

[0090] The routed data stream with conflict is taken as conflict-elimination data, and the conflict-elimination data is processed by using a CRDT merging algorithm based on space-time priority to obtain conflict-elimination data;

[0091] The routed data stream without conflict is combined with the conflict-elimination data to obtain the training data.

[0092] Specifically, the conflict detection module in the application is further configured to initialize a vector clock for each data item when the routed data stream is generated, that is, mark a distributed version {node ID: logical clock} for each record; whenever a data item is updated, the corresponding vector clock is updated, that is, the logical clock value of the updated node is increased by 1, and the logical clock values of other nodes remain unchanged; and the routed data stream is subjected to conflict detection through a Version vectors algorithm, that is, the vector clocks of different data items are compared by using the Version vectors algorithm, if there is a difference (that is, at least one logical clock value is different) between the vector clocks of two data items, it is determined that there is conflict between the two data items; that is, when different nodes concurrently modify the same device data (such as transformer temperature), the system detects whether there is a cross relationship between the version vectors (such as conflict between node A version {A:2} and node B version {B:1}).

[0093] The post-routing data stream with conflicts is taken as conflict-elimination data, and a CRDT merging algorithm based on space-time priority is used to process the conflict-elimination data, specifically: first, the data time stamp is compared, and the latest collected value is preferentially retained; when the time stamp is the same, the data is merged according to the data source type priority decision (SCADA monitoring data > RTU terminal data > AMI electric meter data); in particular for the power load prediction scene, a smoothing weighted strategy is designed for numerical data (such as two conflicting voltage values taking a weighted average), and the latest effective principle is adopted for discrete state data (such as switch state). After processing by the CRDT merging algorithm, conflict-elimination data is generated; wherein the conflict-elimination data is the merged data version, which maintains strong consistency with the binary log event.

[0094] Finally, the post-routing data stream without conflicts is merged with the conflict-elimination data to obtain training data. The conflict detection module in the application realizes efficient conflict detection and processing of the post-routing data stream by introducing vector clocks and the CRDT merging algorithm, improves data accuracy, and guarantees data consistency.

[0095] In an embodiment, the data pipeline layer further comprises a blockchain storage module; wherein the blockchain storage module is configured to:

[0096] The training data is processed by using an NVIDIA CUDA kernel function to generate a SHA-256 fingerprint;

[0097] The SHA-256 fingerprint is written into a blockchain through a chain code smart contract, and a Merkle tree structure is used for optimized storage to realize data traceability.

[0098] Before the training data is sent to the power large model platform, a dynamic verification mechanism is triggered to recalculate the hash value of the training data, compare it with the storage record on the blockchain, and trigger multi-level alarms when the comparison result is abnormal.

[0099] Specifically, the data pipeline layer in the application further comprises a blockchain storage module, which can constitute a four-layer guarantee mechanism with the change capture module, the transaction coordination module and the conflict detection module to guarantee the strong consistency of the training data and the source end.

[0100] The blockchain storage module utilizes GPU parallel computing (NVIDIA CUDA kernel function) to process the training data, generates SHA-256 fingerprints, realizes a hash throughput of 340 Gb / s, writes the SHA-256 fingerprints into the blockchain, and realizes the data traceability of the device level (such as querying the data change history of a certain transformer substation at a certain time period). The construction process of the Merkle tree includes taking the fingerprints as leaf nodes, constructing parent nodes layer by layer upwards through hash operation, and constructing the root node until the root node. The root node hash value of the Merkle tree can be used as the unique identifier of the entire fingerprint set. The root node hash value of the Merkle tree and the related fingerprint information are written into the blockchain through the chain code smart contract. After the writing operation is completed, the nodes on the blockchain will jointly maintain this fingerprint record to ensure that it cannot be tampered with. A dynamic verification mechanism is established, and the mechanism is triggered before the training data is sent to the power big model platform. Then, the hash value of the training data is recalculated by using an efficient hash algorithm (such as SHA-256) and compared with the on-chain record. If the comparison result is consistent, it means that the data has not been tampered with in the transmission and storage process. If the comparison result is inconsistent, it means that the data may have been tampered with or there is an anomaly, and a multi-level alarm (SMS / monitoring large screen / audit system) is triggered when the anomaly occurs.

[0101] The blockchain storage module provides highly secure, traceable and reliable storage services for training data by combining NVIDIA CUDA kernel function, chain code smart contract, Merkle tree structure and dynamic verification mechanism.

[0102] In the embodiment of the application, a dynamic integration system of a power big model platform and a heterogeneous system is designed to solve the problems of lack of dynamic adaptation and weak data consistency guarantee in complex cross-platform data interaction. The system includes a protocol adaptation layer, a service perception layer and a data pipeline layer. The protocol adaptation layer is used as the bottom foundation, the three-layer conversion engine is used to solve the problem of heterogeneous system access, provide a unified interface for data flow, and reduce the expansion cost from exponential to linear; the service perception layer is used as the middle pivot, the three-layer linkage mechanism is used to solve the problem of dynamic service discovery and routing, ensure that new service changes can take effect in real time, and the dynamic routing capability supports flexible scheduling of the protocol adaptation layer and the data pipeline layer; the data pipeline layer is used as the upper guarantee, relies on the standardized data and stable services provided by the previous two layers, uses the three-layer guarantee mechanism to solve the data consistency problem, and guarantees the strong consistency of big model training and sample library data.

[0103] It should be noted that the above each module of the power large model platform and the dynamic integration system of the heterogeneous system can be all or partially realized by software, hardware and a combination thereof. The above each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0104] In another embodiment, as shown in FIG. 2, the second aspect of the present application provides a power large model platform and a dynamic integration method of a heterogeneous system, comprising: Figure 2

[0105] S1, processing the original heterogeneous data received from a plurality of heterogeneous systems by a three-layer conversion engine to obtain a standardized data stream, and dynamically adapting the transmission protocol of each of the heterogeneous systems;

[0106] S2, generating real-time routing rules based on the standardized data stream by using a three-layer linkage mechanism, and controlling the on-demand routing of the standardized data stream by using the real-time routing rules to obtain a routed data stream;

[0107] S3, real-time acquisition of binary log events of each of the heterogeneous systems, processing the routed data stream by using a three-layer guarantee mechanism to make the generated training data consistent with the binary log events, and sending the training data to a power large model platform.

[0108] It should be noted that although each step in the above flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated herein, the execution of these steps has no strict sequence limitation, and these steps can be executed in other orders. For specific limitations of the power large model platform and the dynamic integration method of the heterogeneous system, refer to the limitations of the power large model platform and the dynamic integration system of the heterogeneous system in the above, both of which have the same functions and effects, and will not be described here.

[0109] ​In summary, the present application relates to the field of information technology, and discloses a dynamic integration system and method of a power large model platform and a heterogeneous system, wherein the dynamic integration system comprises a protocol adaptation layer, a service awareness layer, and a data pipeline layer.

[0110] Each of the above-described embodiments is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other, and each of the embodiments mainly describes the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts can be referred to the part of the description of the method embodiments. It should be noted that, each of the technical features of the above-described embodiments can be combined arbitrarily, in order to make the description simple, each of the technical features of the above-described embodiments is not described in all possible combinations, however, as long as the combination of the technical features does not exist contradictory, it should be considered that it is within the scope of the present application.

[0111] The above-described embodiments only express several preferred embodiments of the present application, the description is relatively specific and detailed, but it should not be understood as the limitation of the patent scope of the present application. It should be pointed out that, for the ordinary skilled in the art, without departing from the technical principles of the present application, a number of improvements and replacements can be made, and these improvements and replacements should be considered as the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the protection scope of the claims.

Claims

1. A dynamic integration system of a large power model platform and heterogeneous systems, characterized by: It includes protocol adaptation layer, service perception layer and data pipeline layer; among them, The protocol adaptation layer is used to process the original heterogeneous data received from multiple heterogeneous systems through a three-layer conversion engine to obtain a standardized data stream and dynamically adapt the transmission protocol of each of the heterogeneous systems; The service perception layer is used to generate real-time routing rules based on the standardized data flow using a three-layer linkage mechanism, and control the on-demand routing of the standardized data flow through the real-time routing rules to obtain a routed data flow; The data pipeline layer is used to obtain binary log events of each of the heterogeneous systems in real time, process the routed data stream through a three-layer security mechanism to ensure that the generated training data is consistent with the binary log events, and send the training data to the power big model platform.

2. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 1 is characterized in that: The protocol adaptation layer includes a physical layer, a semantic layer and a control layer; wherein, The physical layer is used to provide a unified connection interface to connect to the transmission protocols of each of the heterogeneous systems, and to establish an asynchronous non-blocking I / O model to receive original heterogeneous data from each of the heterogeneous systems; The semantic layer is used to construct a universal data model and convert the original heterogeneous data into a standardized data stream through the universal data model; The control layer is used to deploy the Adapter Operator in Kubernetes and automatically inject the Sidecar container to carry the protocol conversion logic to achieve dynamic adaptation of the transmission protocols of each of the heterogeneous systems.

3. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 2 is characterized in that: The semantic layer includes a model building module, a model parsing module and a data mapping module; wherein, The model building module is used to build the general data model based on Apache Avro dynamic Schema; The model parsing module is used to parse the universal data model, generate an operator dependency tree, and extract tensor metadata from each of the original heterogeneous data according to the operator dependency tree; The data mapping module is used to standardize the tensor metadata to obtain tensor features, and use a dynamic Schema construction engine to map the tensor features into a standardized data stream.

4. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 3 is characterized in that: The model parsing module is further used to: Performing graph parsing on the general data model, identifying an input node of the general data model as a root node, and traversing the graph nodes layer by layer to establish a connection relationship according to the type of each node during the traversal process; Build a tree topology through recursive operators, dynamically prune to remove redundant operators, and generate independent subtrees for control flow operators; Extract key data of each operator to generate an operator dependency tree based on the connection relationship, using the tree topology structure, the pruning result and the independent subtree, and extract tensor metadata from each original heterogeneous data according to the operator dependency tree.

5. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 1 is characterized in that: The service perception layer includes a service registration module, a change perception module and a policy validation module; wherein, The service registration module is used to automatically submit the service contract to the Nacos registration center through the Envoy Sidecar proxy in response to the standardized data flow, and trigger continuous health checks; The change perception module is used to intelligently analyze the interface signatures of each of the heterogeneous systems after the continuous health check passes, and generate real-time routing rules based on the analysis results; The policy validation module is used to hot-load the real-time routing rules, establish dual traffic lanes to control the standardized data flow to be divided into old version services and new version services and route them on demand to obtain the routed data flow.

6. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 5 is characterized in that: The policy validation module is further configured to: When an anomaly is detected in the new version service, the circuit breaker rollback mechanism is triggered to automatically block the new version service with the anomaly and intelligently roll back to the stable new version service based on the topological relationship diagram between the new version services; When it is detected that there is no abnormality in the new version service, the full switching mechanism is used to automatically and smoothly transition the traffic ratio of the new version service, and control the old version service to gracefully go offline after completing the existing requests.

7. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 1 is characterized in that: The data pipeline layer includes a change capture module, a transaction coordination module and a conflict detection module; wherein, The change capture module is used to capture binary log events of each of the heterogeneous systems in real time using CDC technology; The transaction coordination module is configured to decompose the cross-platform interaction between the dynamic integration system and each of the heterogeneous systems into a sequence of atomic sub-transactions, pre-register a compensation transaction for each atomic sub-transaction, and initiate the compensation transaction when a failure of the atomic sub-transaction is detected, so as to ensure that all cross-platform interactions are successful; The conflict detection module is used to perform conflict detection on the post-routing data stream, process the post-routing data stream with conflicts using a preset algorithm, merge the processing results with the post-routing data stream without conflicts into training data that is consistent with the binary log event, and send the training data to the power big model platform.

8. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 7 is characterized in that: The conflict detection module is further configured to: Marking a vector clock for the routed data flow, and performing conflict detection on the routed data flow using a version vectors algorithm based on the vector clock; The conflicting post-routing data stream is used as the conflict-free data, and the conflict-free data is processed using the time-space-priority CRDT merging algorithm to obtain conflict-free data; The post-routing data stream without conflict is merged with the conflict elimination data to obtain the training data.

9. The dynamic integration system of a large power model platform and heterogeneous systems according to claim 1 is characterized in that: The data pipeline layer also includes a blockchain evidence storage module; wherein the blockchain evidence storage module is used to: Processing the training data using NVIDIA CUDA kernel functions to generate a SHA-256 fingerprint; The SHA-256 fingerprint is written into the blockchain through the chaincode smart contract, and the Merkle tree structure is used to optimize storage to achieve data traceability; Before sending the training data to the power big model platform, a dynamic verification mechanism is triggered to recalculate the hash value of the training data for comparison with the storage record on the blockchain, and when an abnormality is found in the comparison result, a multi-level alarm is triggered.

10. A method for dynamic integration of a large power model platform and heterogeneous systems, characterized in that: include: Processing the original heterogeneous data received from multiple heterogeneous systems through a three-layer conversion engine to obtain a standardized data stream and dynamically adapting the transmission protocol of each of the heterogeneous systems; Based on the standardized data flow, a three-layer linkage mechanism is used to generate real-time routing rules, and the standardized data flow is controlled by the real-time routing rules to route on demand to obtain a routed data flow; Binary log events of each of the heterogeneous systems are acquired in real time, and the routed data stream is processed through a three-layer security mechanism to ensure that the generated training data remains consistent with the binary log events, and the training data is sent to the power big model platform.

Images