Loose coupling data transmission method and device for electric power big data sharing platform

Abstract

The invention provides a loose coupling data transmission method and device for an electric power big data sharing platform, and relates to the technical field of electric power grids. The asynchronous service data is received through the standardized data interface, the theme queue of the message-oriented middleware is used for buffering, a direct dependency chain between the service system and the sharing platform is cut off, time and process decoupling is achieved, cross-system propagation of faults is effectively isolated, and the independence and stability of operation of the power grid service system are improved. According to the invention, dynamic adaptation of a transmission strategy and resource scheduling is realized through intelligent cooperation of a differential forwarding strategy based on a security level and a real-time resource load, so that the system can flexibly cope with peak and valley fluctuations of data traffic, and the overall resource utilization efficiency is improved while key data transmission is guaranteed. According to the invention, the problem that the tight coupling architecture between the business system and the sharing platform affects the stability of the system is solved, and the robustness, elasticity and reliability of the whole power data ecosystem are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power grid, and particularly relates to a loose coupling data transmission method and equipment of a power big data sharing platform. BACKGROUND

[0002] With the deepening of the process of digitalization and intelligentization of the power industry, power big data has penetrated into various aspects of core businesses such as power grid planning, real-time scheduling, equipment operation and maintenance, and customer service. As the core hub of gathering, integrating and distributing these massive data, the robustness, security and flexibility of the data transmission architecture of the power data sharing platform directly determine the efficiency of the entire power data ecosystem.

[0003] At present, the data transmission between the power data sharing platform and various front business systems such as the distribution automation system, the power information acquisition system, and the equipment management system usually adopts a point-to-point tight coupling architecture. The data transmission channel from the business system to the sharing platform is usually tightly integrated and statically configured.

[0004] The deep coupling between the business system and the sharing platform poses a high risk of chain failure. Since the data transmission channel is closely related to the internal logic of both systems, any adjustment of either party - whether it is an upgrade and expansion of the sharing platform to improve performance or a version iteration of the business system itself - may require the other party to make corresponding interface adaptation and synchronous modification, otherwise it may cause data transmission interruption. More seriously, when a system of one party fails, the failure will quickly spread to the other party through the tightly integrated data transmission channel: the service blockage of the sharing platform may slow down or even paralyze the data processing of the front core business system; conversely, the abnormally high load or crash of a business system may also impact the stability of the sharing platform, thereby affecting other data consumers relying on the platform. This architecture feature makes local failure easily evolve into systemic risk, seriously threatening the continuous and stable operation of various business systems of the power grid. SUMMARY

[0005] The present application provides a loose coupling data transmission method and equipment of a power big data sharing platform, which solves the problem of tight coupling architecture between business systems and sharing platforms affecting system stability.

[0006] In a first aspect, the present application provides a loosely coupled data transmission method of a power big data sharing platform, which is applied to a data management unit between a plurality of business systems and a data sharing platform in a power grid. The method comprises the following steps: receiving business data asynchronously sent by the business systems through a standardized data interface; analyzing and classifying the business data of the business systems, determining data topics and security levels of the business data, and the security levels comprising a public level, a restricted level and a confidential level; updating a plurality of topic queues of a message middleware of the data management unit based on the data topics of the business data, decoupling time and process, and obtaining an updated message middleware; determining a differentiated data forwarding strategy based on the security levels of the tasks of the updated message middleware; and sending the business data to the data sharing platform based on the differentiated data forwarding strategy and real-time resource loads of the data management unit and the data sharing platform, and realizing loosely coupled data transmission.

[0007] In a second aspect, the present application provides a loosely coupled data transmission device of a power big data sharing platform, which is applied to a data management unit between a plurality of business systems and a data sharing platform in a power grid. The device comprises a communication module and a processing module. The communication module is configured to receive business data asynchronously sent by the business systems through a standardized data interface. The processing module is configured to analyze and classify the business data of the business systems, determine data topics and security levels of the business data, and the security levels comprising a public level, a restricted level and a confidential level; update a plurality of topic queues of a message middleware of the data management unit based on the data topics of the business data, decouple time and process, and obtain an updated message middleware; determine a differentiated data forwarding strategy based on the security levels of the tasks of the updated message middleware; and send the business data to the data sharing platform based on the differentiated data forwarding strategy and real-time resource loads of the data management unit and the data sharing platform, and realize loosely coupled data transmission.

[0008] In a third aspect, the present application provides an electronic device, which comprises a memory and a processor. The memory stores a computer program. The processor is configured to invoke and run the computer program stored in the memory to execute the steps of the method according to the first aspect and any possible implementation manner of the first aspect.

[0009] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method according to the first aspect and any possible implementation manner of the first aspect are implemented.

[0010] The application provides a loose coupling data transmission method and device of a power big data sharing platform. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the attached drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the attached drawings in the following description are only some embodiments of the present application, and other attached drawings can be obtained by those skilled in the art without any creative labor on the basis of these attached drawings.

[0012] Figure 1 is a power data sharing process schematic diagram provided by the embodiment of the present application; Figure 2 is a loose coupling data transmission method flowchart of a power big data sharing platform provided by the embodiment of the present application; Figure 3 is a loose coupling data transmission device structure schematic diagram of a power big data sharing platform provided by the embodiment of the present application; Figure 4 is an electronic device structure schematic diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0013] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application, such as specific system structures, techniques, etc. However, it should be apparent to those skilled in the art that the present application can be practiced in other embodiments without these specific details. In other cases, well-known system, device, circuit and method details are omitted in order not to obscure the present application with unnecessary details.

[0014] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" herein is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means two or more. "First", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different.

[0015] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. On the contrary, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner, for ease of understanding.

[0016] In addition, the terms "include" and "have" mentioned in the description of the present application and any modification thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but can optionally include other steps or modules not listed, or can optionally include other steps or modules inherent to the process, method, product or device.

[0017] In order to make the purpose, technical scheme and advantages of the present application clearer, specific embodiments will be described below with reference to the drawings of the present application.

[0018] As Figure 1 shown, the embodiment of the present application provides a power data sharing process schematic diagram. The data sharing process includes main links such as demand raising, sharing preparation, sharing implementation, data use, operation and maintenance, etc.

[0019] Among them, the data management party formulates the data sharing demand process, authorizes the sharing use; the data provider clearly defines the use range, permission requirements and data security related requirements of the shared data; the data management party and the data demand party, the data provider jointly determine the data sharing mode, data interaction mode and guarantee measures, etc., and carry out data transmission channel development work; after the data management party agrees to the data sharing application, it should be organized to carry out data collection in time, and when the existing data cannot meet the requirements of the data user, the data provider should be required to carry out data access; according to different data users and demand purposes, a series of processes such as data desensitization, data watermarking, data encryption and other security reinforcement are carried out on the shared data.

[0020] Based on Figure 1The power data sharing process shown as Figure 2 As shown in the figure, the embodiment of the application provides a loosely coupled data transmission method of a power big data sharing platform. The method is applied to a data management unit between a plurality of business systems and a data sharing platform in a power grid. The method comprises steps S101-S105.

[0021] S101, receiving business data asynchronously sent by each business system through a standardized data interface.

[0022] In some embodiments, the business system refers to various core production systems and management systems in the power grid operation system, which generate and need to share data externally. They constitute the data source of the data sharing platform, mainly including: a dispatching automation system: providing real-time running state, switch action, power flow distribution and other data of the power grid. A power utilization information collection system: providing user-side power data, load curve, voltage quality and other data. A distribution automation system: providing fault information, device state, remote control operation and other data of the distribution network. A device management system: providing the account, test data, defect record and other data of the power grid primary equipment such as transformers and circuit breakers. A power market transaction system: providing unit offer, market clearing result, contract power and other data. A new energy monitoring system: providing power generation prediction, actual output, equipment utilization and other data of photovoltaic power stations, wind farms and other data.

[0023] In some embodiments, the standardized data interface is a unified communication contract designed to decouple from each heterogeneous business system. The communication protocol adopts a standard interaction protocol. The data format specifies that the business data must be packaged in a unified structure. The fields, field types, whether to fill in and value range that must be included in the message are clearly defined to ensure the structural consistency of all access data. The authentication and authorization interface call must carry a security token for identity authentication and permission verification to ensure that only authorized business systems can push data. The interaction mode adopts an asynchronous communication mode. After the business system calls the interface to push data, the interface service returns a status code immediately after completing the basic verification, indicating that the request has been accepted and processed, rather than the final processing result. The business system does not need to block and wait for the data sharing platform to complete subsequent storage, calculation and other operations, thereby realizing decoupling.

[0024] S102, analyzing and classifying the business data of each business system to determine the data theme and security level of each business data.

[0025] In the embodiment of the application, the security level includes public level, restricted level and confidential level.

[0026] In some embodiments, the embodiments of the present application can check the structural compliance and integrity of the data, and then match the data inherent business domain tags and key fields with a predefined classification tree to preliminarily classify into macro topics such as power consumption data, equipment status or power grid operation. The security level determination is based on a preconfigured sensitive word library and data identification whitelist to quickly scan and match the data, thereby completing the preliminary classification of public, restricted or confidential levels.

[0027] As a possible implementation, step S102 can be implemented as steps S1021-S1023.

[0028] S1021, parsing the business data of each business system to extract the metadata and content features of each business data.

[0029] In some embodiments, the metadata includes data source system, data generation timestamp and data structure identification; and the content features are used to represent the numerical attributes, identification attributes, space-time attributes and pattern attributes of the data content.

[0030] For example, the extraction of metadata is completed by parsing the header information of the data file, the Header of the API request and the specific fields in the message, so as to accurately obtain the data source system such as system registration ID, data generation timestamp and data structure identification. The extraction of content features: the numerical attributes refer to specific measurement values such as voltage and current and their statistical characteristics; the identification attributes include user ID, equipment asset code and other unique identifiers; the space-time attributes cover GIS coordinates, account area code and event time sequence; and the pattern attributes involve identifying whether the data sequence presents periodic fluctuations or whether there are abnormal spikes.

[0031] S1022, matching based on the metadata and content features of each business data and the preconfigured data topic mapping table to determine the data topic of each business data.

[0032] In some embodiments, the data topic mapping table is a structured knowledge base that records mapping rules such as when the data source system is a power distribution automation system and there is a line load rate field in the content, the topic should be classified as real-time load of distribution line.

[0033] For example, the embodiments of the present application can match the extracted feature set with the rules in the mapping table, determine the most accurate data topic such as 10kV feeder load data through weighted scoring or decision tree model, and thereby realize more refined topic classification.

[0034] S1023, determining the security level of each business data based on the preconfigured data security classification rule library in combination with the data topic and content features of each business data.

[0035] In some embodiments, the data security classification rule base is composed of a series of conditional judgment rules. For example, one rule can stipulate that if the data subject is user detailed electricity consumption data, and the content features contain personal sensitive identifiers such as user address or identity card number, the security level should be determined as restricted level. Another rule can stipulate that if the data subject is power grid key section flow, and the numerical features show that it is real-time operation data, the security level should be determined as confidential level.

[0036] Exemplarily, the embodiment of the present application can execute a rule engine, and logically match the subject and content features of the current data with the rule base one by one. Once a rule is hit, the final security level determination result is output according to the rule. This process realizes the automation, standardization and precision of security classification, and lays a solid foundation for subsequent differentiated security processing.

[0037] S103, based on the data subject of each business data, updating a plurality of subject queues of the message middleware of the data management unit to decouple time and process, and obtaining an updated message middleware.

[0038] In some embodiments, the embodiment of the present application can A subject-queue mapping table is maintained, and the obtained data subject, such as user monthly electricity consumption, will be used as a routing key to automatically route the data to the corresponding subject in Kafka or RabbitMQ. The data is serialized into Avro or Protobuf format messages and stored in the queue. This process enables the business system to be released after receiving the message storage confirmation, thereby decoupling the time and business process from the backend data sharing platform.

[0039] As a possible implementation manner, step S103 can be implemented as steps S1031-S1034.

[0040] S1031, based on the data subject of each business data and the plurality of subject queues in the message middleware, matching is performed to determine the subject queue corresponding to each business data.

[0041] In some embodiments, the subject routing mapping table records rules such as electricity load data subject mapping to message subject, device state data subject mapping to message subject, etc.

[0042] Exemplarily, the embodiment of the present application can determine the corresponding accurate destination of each business data in the message middleware by querying the subject routing mapping table. This design realizes the separation of logical classification of business data and physical transmission channel.

[0043] S1032, respectively perform security encapsulation on each service data based on the security level of each service data, to obtain a message corresponding to each service data.

[0044] For example, the security encapsulation refers to different security enhancement processing according to the data security level. For restricted level data, the encapsulation process calls a data watermark service to embed traceable information containing a timestamp and a source system identifier in the data; for confidential level data, an encryption module is triggered to use pre-set key material to perform end-to-end encryption on the message content using the SM4 algorithm approved by the National Cryptographic Administration. All levels of data are assigned message ID, timestamp and other standard metadata when encapsulated, and are serialized into a format supported by the message middleware.

[0045] S1033, add the message corresponding to each service data to the topic queue corresponding to each service data, to obtain an updated message middleware.

[0046] For example, the producer sends the encapsulated message to the specified topic queue, and the message middleware cluster persists the message to the storage log after receiving the message, and returns a successful confirmation to the producer after completing the replica synchronization. This process safely transfers data from the volatile business system environment to the message bus with persistent capability and high availability features.

[0047] S1034, if the message of a service data is successfully added to the topic queue, a confirmation response is returned to the business system corresponding to the service data to indicate that the decoupling with the business system is successful.

[0048] For example, the embodiment of the application can realize time decoupling. Once the data management unit receives a message write success confirmation from the message middleware, it will immediately send a technical confirmation to the source business system through an asynchronous callback mechanism. This confirmation only indicates that the data has been reliably received and hosted in the message middleware, and does not mean that the data has been completely processed by the data sharing platform. After receiving this confirmation, the business system completes its data transmission responsibility, thereby realizing complete decoupling with the back-end complex processing flow.

[0049] S104, based on the security level of each task of the updated message middleware, a differentiated data forwarding strategy is determined.

[0050] As a possible implementation manner, step S104 can be implemented as steps S1041-S1044.

[0051] S1041, for service data with a security level of public, a high-throughput batch data forwarding strategy is configured.

[0052] Exemplarily, the embodiment of the present application can be implemented by setting optimized batch processing parameters, configuring a batch size threshold significantly larger than other levels for public level data, and enabling a data compression algorithm, so as to maximize the utilization of network bandwidth and overall throughput.

[0053] Exemplarily, step S1041 can be implemented as steps A1-A4.

[0054] A1, identifying a state mutation point and a key steady-state parameter in the business data of the public level.

[0055] In some embodiments, the state mutation point includes the starting point of the power sudden drop event and the voltage out-of-limit event; and the key steady-state parameter includes the load peak value and the voltage steady-state critical value.

[0056] Exemplarily, the embodiment of the present application can be implemented by deploying a real-time event detection algorithm in the stream processing pipeline. The algorithm continuously analyzes the data stream, identifies the point where the first derivative of the power curve exceeds the preset threshold as the state mutation point, and records the statistical extreme value in the measurement sequence as the key steady-state parameter.

[0057] A2, based on the state mutation point and the key steady-state parameter, performing data value density evaluation on the business data of the public level to determine the data value density of the business data of the public level.

[0058] Exemplarily, the embodiment of the present application can calculate a comprehensive score for each data point or data window. The score model gives a higher weight to the state mutation point, and considers the scarcity of the key steady-state parameter, and obtains a quantitative value density index through weighted calculation.

[0059] A3, if the data value density is greater than the value density threshold, determining that the business data of the public level is high-value data, and determining that the batch data forwarding strategy of the high-value data is to encapsulate a high priority label in the message, and adding the message of the high-value data to the first topic queue with high priority.

[0060] Exemplarily, the embodiment of the present application can set a priority level attribute for the identified high-value data message in the message encapsulation stage, and direct it to the independent topic queue specially configured for high-priority data through the routing function of the message middleware. The queue is usually given higher service quality and higher consumption priority.

[0061] A4. If the data value density is less than or equal to the value density threshold, then the business data with the security level of public is determined to be regular steady-state data, and the batch data forwarding strategy is determined to be to add the messages of regular steady-state data to the second topic queue for batch processing after compression and aggregation.

[0062] Among them, the data management unit prioritizes sending business data to the first topic queue.

[0063] For example, in embodiments of the present invention, multiple regular data records can be aggregated in a memory buffer, and a lossless compression algorithm can be used to reduce the data volume to form a merged data block, which is then published as a single message to a regular priority topic queue dedicated to batch processing.

[0064] This invention can configure the consumer group of the message middleware so that when polling messages, it always prioritizes obtaining messages from the high-priority first topic queue, and only consumes data from the second topic queue when there are no messages to be processed in the first topic queue, thereby ensuring low-latency transmission of high-value data.

[0065] S1042. For business data with a restricted security level, configure a streaming data forwarding strategy that performs real-time de-identification processing and records complete access logs.

[0066] For example, embodiments of the present invention can establish a dedicated streaming processing pipeline. This pipeline integrates a data masking engine, which performs real-time masking or replacement processing on sensitive fields (such as user ID numbers and contact information) in the data stream according to predefined masking rules. Simultaneously, the audit log module captures and persistently records complete contextual information, including data source, access time, operation type, and processing result, forming an immutable audit trail.

[0067] S1043. For business data with a security level of confidential, configure a single data forwarding strategy that allows transmission through a dedicated power security channel and confirmation for each data item.

[0068] For example, embodiments of the present invention can establish secure session connections with dedicated networks such as power dispatch data networks. Each piece of confidential data is encapsulated as an independent message. After being transmitted through this secure channel, it will wait for an application layer confirmation message from the receiving end. The local message copy will not be destroyed before receiving the confirmation, thereby ensuring the absolute reliability and traceability of critical data transmission.

[0069] S1044. Based on the number of pending messages in each topic queue in the message middleware, dynamically adjust the batch size and / or sending rate in the data forwarding strategy of each task.

[0070] Exemplarily, the embodiment of the present application can continuously monitor the number of backlog messages of each topic queue, automatically increase the batch size or the number of concurrent threads of the corresponding forwarding task of a specific topic when the backlog number of the specific topic exceeds a first threshold value, and correspondingly lower the sending rate when the backlog number continuously grows beyond a higher second threshold value, so as to realize adaptive balancing of the system transmission load.

[0071] S105, based on the differentiated data forwarding strategy and the real-time resource load of the data management unit and the data sharing platform, sending the service data to the data sharing platform to realize the loose-coupling data transmission.

[0072] As a possible implementation manner, step S105 can be specifically implemented as steps S1051-S1054.

[0073] S1051, continuously monitoring the computing resource utilization rate of the data management unit and the data interface response time of the data sharing platform.

[0074] Exemplarily, the embodiment of the present application can be cooperatively realized by the resource monitoring agent deployed on the data management unit and the probe program facing the API gateway of the data sharing platform. The resource monitoring agent periodically collects the key performance indicators including the central processing unit occupancy rate, the memory usage rate and the network input / output throughput; at the same time, the probe program sends lightweight test requests to the key data interfaces of the data sharing platform at a fixed frequency, and accurately measures the time interval from the request sending to the response first byte receiving, so as to take the interface response delay as the quantitative basis.

[0075] S1052, comparing the computing resource utilization rate and the data interface response time with the preset resource load threshold value to obtain a comparison result.

[0076] In some embodiments, the comparison result is transmission congestion, transmission limitation and transmission normal.

[0077] Exemplarily, the embodiment of the present application can be executed by a load evaluation engine. The engine is preset with a multi-level threshold system, including an upper limit threshold value defining the normal load interval of the system and a higher threshold value identifying the overload of the system. The engine compares the real-time collected indicators with the multi-level threshold value, and outputs the corresponding system state identifier according to the threshold interval where the indicators are located, for example, outputs transmission normal when all indicators are lower than the upper limit threshold value, outputs transmission limitation when any indicator continuously exceeds the upper limit threshold value, and outputs transmission congestion when the key indicators break through the overload threshold value.

[0078] S1053, based on the comparison result and the differentiated data forwarding strategy, performing a resource load adaptation operation to obtain the transmission scheme of each service data.

[0079] When the transmission is limited, the transmission of the public-level service data using the batch data forwarding strategy is throttled; when the transmission is congested, the transmission of part of the non-urgent limited-level service data is temporarily suspended.

[0080] Exemplarily, the embodiment of the application can be completed by a dynamic rule executor. When the system state is transmission limited, the executor first implements transmission throttling on the public-level service data using the batch data forwarding strategy, specifically by reducing the batch processing size or reducing the sending frequency; when the system state is further deteriorated to transmission congestion, the executor starts a more stringent degradation strategy, temporarily suspending the transmission task of part of the limited-level service data marked as non-urgent, thereby preferentially guaranteeing the limited system resources for data transmission with higher security level or higher real-time requirement.

[0081] S1054, based on the data forwarding strategy and the transmission scheme of each service data, scheduling the data sharing platform to pull and process the service data from the topic queue of the message middleware as a consumer.

[0082] Exemplarily, the embodiment of the application can control the consumer service client in the data sharing platform. According to the established data forwarding strategy and the real-time determined transmission scheme, the parameters of the consumer client are dynamically configured, and then the clients are scheduled to obtain messages from the corresponding topic queue of the message middleware in an asynchronous pulling mode. After successfully pulling the messages, the client calls the corresponding processing logic to complete data analysis, verification and final warehousing operation according to the security level and data type information encapsulated in the message, thereby realizing the loosely coupled data transmission of the whole link.

[0083] The application provides a loosely coupled data transmission method and equipment of a power big data sharing platform. The application receives asynchronous service data through a standardized data interface, and buffers the data using a topic queue of a message middleware, thereby fundamentally cutting off the direct dependency chain between a service system and the sharing platform, and realizing decoupling in time and process. When the sharing platform is upgraded, maintained or temporarily fails, the service system can still successfully write data into the middleware without being affected, effectively isolating the cross-system propagation of faults, and greatly improving the independence and stability of the power grid service system operation. The application realizes dynamic adaptation of transmission strategies and resource scheduling through intelligent cooperation based on differentiated forwarding strategies according to security levels and real-time resource loads, so that the system can flexibly cope with data flow peak and valley fluctuations, while ensuring key data transmission, and improving overall resource utilization efficiency. Therefore, the application solves the problem of tight coupling architecture between the service system and the sharing platform affecting system stability, significantly enhances the robustness, flexibility and reliability of the entire power data ecosystem on the basis of ensuring efficient data transmission.

[0084] Optionally, the loose coupling data transmission method of the power big data sharing platform provided by the embodiment of the present application further includes steps S201-S202.

[0085] S201, receiving update information sent by the master message middleware, the update information including change information of each topic queue in the master message middleware.

[0086] In some embodiments, the change information includes new task information and completed task information.

[0087] Illustratively, the embodiment of the present application can be based on the mirror queue mechanism of the message middleware cluster or the cross-cluster data synchronization tool. The master message middleware cluster transmits the operation logs of all persistent topic queues in real time to the standby message middleware cluster through the built-in replication protocol. The update information specifically includes message publishing confirmation, delivery confirmation and metadata change operation instructions, etc., to ensure that the standby cluster can accurately reproduce the queue state and data content of the master cluster.

[0088] S202, synchronously updating the standby message middleware based on the update information.

[0089] Illustratively, the synchronization service in the standby message middleware cluster continuously listens to the data stream from the master cluster. The service parses the received operation log sequence and executes the corresponding data operation in the standby cluster in the same order, including writing new messages to the corresponding mirror queue, marking the confirmed messages as completed state, and maintaining the queue metadata consistent with the master cluster.

[0090] In this way, the present application ensures that the standby cluster always maintains the queue state and data content consistent with the master cluster by real-time streaming of queue operation logs and accurate reproduction of data operation sequences. This design not only enables the standby cluster to take over, but also constitutes a hot backup node, effectively eliminating the risk of data loss caused by single point failure, laying a solid foundation for subsequent fault switching, and greatly improving the resilience to abnormal situations such as hardware failure and network interruption.

[0091] Optionally, the loose coupling data transmission method of the power big data sharing platform provided by the embodiment of the present application further includes steps S203-S205.

[0092] S203, monitoring the connection state and service health of the master message middleware and the standby message middleware.

[0093] Exemplarily, the embodiment of the present application can be completed by a health check service deployed on a data management unit or an independent monitoring node. The service sends a heartbeat detection request to a management interface of a master message middleware cluster at a preset frequency, and comprehensively evaluates the response delay, node survival state and key performance indicators thereof. Meanwhile, by establishing a test production-consumption session, the actual throughput capacity and stability of the data plane are verified, thereby forming a multi-dimensional and stereoscopic judgment on the service health.

[0094] S204, based on the connection state and health of the master message middleware and the standby message middleware, determining whether the master and standby message middlewares meet the switching condition.

[0095] Exemplarily, the embodiment of the present application can be executed by a fault decision engine. The engine is preset with multi-factor switching criteria, including the number of consecutive heartbeat timeouts, the average response delay threshold, the proportion of unavailable nodes, etc. When the monitoring indicators meet any of the preset fault conditions, for example, the master cluster does not respond for a specified time period or the proportion of available nodes thereof is lower than the minimum redundancy requirement, the engine determines that the switching condition is met and prepares to start the failover process.

[0096] S205, if the switching condition is met, switching the data transmission path to the standby message middleware for loose-coupled data transmission.

[0097] Exemplarily, the embodiment of the present application can dynamically update the connection endpoints of all producers and consumers in the data management unit to the access addresses of the standby message middleware cluster by the configuration management service; broadcast the cluster switching notification to the related business systems and data sharing platforms, so that subsequent requests are directed to the new valid endpoints; finally, all data transmission tasks are resumed on the standby message middleware cluster, thereby completing the entire failover process without interrupting the service and ensuring the continuous loose-coupled data transmission capability.

[0098] In this way, the present application can quickly and accurately identify the master cluster anomaly through multi-dimensional health monitoring and intelligent fault decision. The automatic switching mechanism triggered based on the preset conditions realizes seamless migration of the data transmission path. The scheme converts the traditional fault recovery process relying on manual intervention into a minute-level automated response, significantly shortens the service interruption time, ensures the business continuity of the data transmission service, and finally provides a reliable data transmission guarantee with disaster recovery and self-healing capability for the upper-layer business system.

[0099] Optionally, the loose-coupled data transmission method of the power big data sharing platform provided by the embodiment of the present application further includes steps S301-S305.

[0100] S301, receiving a data access request sent by a data consumer.

[0101] In some embodiments, the data access request comprises a target data subject and an access mode, the access mode comprising a real-time query mode, a subscription push mode and a batch download mode; and the data consumer comprises business systems, external third-party institutions and public users.

[0102] For example, the embodiment of the present application can deploy a unified data service gateway as a unique entry, require the data consumer to explicitly specify the unique identifier of the target data subject in the request message, and enumerate and declare the requested service type in the access mode field, the service type comprising three standard service forms of the real-time query mode, the subscription push mode and the batch download mode.

[0103] S302, identity authentication and access permission verification are performed on the data consumer based on the data access request, and a verification result is determined.

[0104] For example, the embodiment of the present application can verify the validity of the identity token of the requestor through the protocol, then query the access control list to verify whether the identity has the operation permission of the specified data subject in the access mode, and finally generate a security context containing the authorized range according to the verification result, to provide a basis for subsequent data processing.

[0105] S303, if the verification result is verified, a data service task is generated based on the access mode and the target data subject.

[0106] For example, the embodiment of the present application can adopt a differentiated task generation strategy according to different access modes. For the real-time query mode, a query task containing an immediate execution instruction is generated; for the subscription push mode, a long-term subscription task containing a terminal callback address is generated; and for the batch download mode, a batch processing task containing a data time range and an output format is generated. All tasks encapsulate complete security context and execution parameters.

[0107] S304, the data service task is added to the subject queue of the message middleware.

[0108] For example, the embodiment of the present application can serialize the generated data service task object into a standard message format, and route it to the corresponding task subject queue in the message middleware according to its task type. The real-time query task enters a high-priority queue to ensure low-latency response, and the subscription and batch tasks enter different business queues to realize resource isolation, and the message persistence mechanism is used to ensure that the task is not lost.

[0109] S305, the data sharing platform acts as a consumer to asynchronously pull and execute the data service task from the subject queue, and returns the processing result data to the data consumer according to the access mode.

[0110] Exemplarily, the embodiment of the present application can asynchronously consume task messages through a task processing service cluster deployed in a data sharing platform. The service calls a corresponding data processing engine according to the task type, and returns the result through a standardized response channel after execution: the real-time query result is returned immediately through an HTTP response, the subscription data is pushed to a preset endpoint, the batch processing result generates a downloadable link and notifies the consumer through a message, thereby completing the end-to-end data service process.

[0111] Thus, the present application uniformly abstracts heterogeneous data access requests into standardized data service tasks, and performs asynchronous scheduling and execution through a message middleware, which not only guarantees the standardization and security of service access, but also significantly improves the throughput and stability of the system under high-concurrency access through resource isolation and asynchronous processing mechanism, and finally provides unified, efficient and reliable data service support for diversified data consumption scenarios.

[0112] Optionally, the loose-coupling data transmission method of the power big data sharing platform provided by the embodiment of the present application further includes steps S401-S405.

[0113] S401, receiving a real-time query request of a data consumer to business data.

[0114] Exemplarily, the embodiment of the present application can receive a standardized query request through a data service gateway, which at least contains a data resource identifier and a query condition parameter. The gateway performs preliminary syntax verification on the request, and then distributes it to a specially designed cache query engine for processing, to ensure the standardization and processability of the query request.

[0115] S402, based on the real-time query request, querying a hot data memory cache to determine whether the to-be-accessed data of the real-time query request exists in the hot data memory cache.

[0116] Exemplarily, the embodiment of the present application can first access a distributed memory database cluster through the query engine. The cluster stores hot data in the form of key-value, wherein the key is determined by generating a hash value from the combination of the data topic identifier and the query parameter. The engine calculates the hash key of the query request and performs an exact match search in the memory database, to determine whether the corresponding cache data copy exists.

[0117] S403, if the to-be-accessed data exists in the hot data memory cache, the to-be-accessed data in the hot data memory cache is directly returned to the data consumer.

[0118] Exemplarily, after the query engine obtains the complete data record from the memory database, it encapsulates the data record into a standard response message, and returns it to the data consumer through the original connection channel of the data service gateway. This process completely avoids accessing the backend database, and realizes a microsecond-level response.

[0119] S404, if not, based on the real-time query request, querying the full data distributed cache to determine whether the real-time query request exists in the full data distributed cache.

[0120] S405, if so, the full data distributed cache is directly returned to the data consumer.

[0121] Exemplarily, the query engine accesses the full data cache layer based on the distributed database. The layer stores all near real-time copies of queryable data, and the query engine performs efficient retrieval through the index combination mechanism to confirm whether the requested data exists in the cache.

[0122] In this way, the application establishes a hierarchical data access path, so that hot data with high frequency access can be directly obtained from the memory to achieve millisecond-level response; and the full distributed cache serves as an effective buffer, which significantly reduces the system pressure of the backend persistent storage while ensuring the integrity of the data query range, and finally provides users with high-throughput, low-latency and high-quality query experience.

[0123] Optionally, the power big data sharing platform loose coupling data transmission method provided by the embodiment of the application further includes steps S406-S408.

[0124] S406, record the query content of multiple real-time query requests within a set period.

[0125] Exemplarily, the embodiment of the application can collect the metadata of all real-time query requests through an asynchronous log service. The service structures the data theme, query parameters, timestamp, cache hit result and other key information of each request in the log database, providing complete data basis for subsequent statistical analysis.

[0126] S407, based on the query content of multiple real-time query requests within a set period, statistical analysis is performed to determine a hot data candidate set.

[0127] Exemplarily, the embodiment of the application can be implemented through a data analysis task executed at a fixed time. The task aggregates and analyzes log data at a preset period, establishes a hotness evaluation model based on query frequency, data theme hotness, time decay factor and other multi-dimensional indicators, and automatically identifies and outputs the data item set reaching the hotness threshold as the candidate hot data set.

[0128] S408, based on the hot data candidate set and the hot data in the hot data memory cache, fusion update processing is performed to obtain an updated hot data memory cache.

[0129] Exemplarily, the cache management service firstly eliminates low access frequency data in the existing cache based on a least recently used algorithm according to a preset memory allocation strategy; subsequently, high priority data items in the hot data candidate set are loaded in parallel from the full data distributed cache and are preset to the in-memory database cluster; meanwhile, the final consistency between the cache data and the data source is maintained, and finally the intelligent update of the hot data in-memory cache is completed.

[0130] In this way, the application can accurately identify the change trend of the user access mode by recording the query features and establishing a data heat analysis model; based on this, the cache content is dynamically adjusted to ensure that the memory resources always serve the data with the highest value, thereby continuously improving the cache hit rate. This adaptive mechanism enables the system to have learning and evolution capabilities, effectively cope with changes in business access patterns, and maintain the optimal performance state in the long term.

[0131] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0132] The following is a device embodiment of the application, and for details not described in detail, reference can be made to the corresponding method embodiments described above.

[0133] Figure 3 A structure schematic diagram of a loosely coupled data transmission device of a power big data sharing platform provided by an embodiment of the application is shown. The data transmission device 500 includes a communication module 501 and a processing module 502.

[0134] The communication module 501 is configured to receive business data asynchronously sent by each business system through a standardized data interface.

[0135] The processing module 502 is configured to analyze and classify the business data of each business system, determine the data theme and the security level of each business data, and the security level includes a public level, a restricted level and a confidential level; based on the data theme of each business data, update a plurality of theme queues of the message middleware of the data management unit, decouple time and process, and obtain an updated message middleware; based on the security level of each task of the updated message middleware, determine a differentiated data forwarding strategy; based on the differentiated data forwarding strategy, and the real-time resource load of the data management unit and the data sharing platform, send the business data to the data sharing platform, and realize loosely coupled data transmission.

[0136] Figure 4is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device 600 includes a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. The processor 601 implements the steps in the above method embodiments when executing the computer program 603. Alternatively, the processor 601 implements the functions of each module / unit in the above device embodiments when executing the computer program 603.

[0137] For example, the computer program 603 can be divided into one or more modules / units, which are stored in the memory 602 and executed by the processor 601 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 603 in the electronic device 600.

[0138] The processor 601 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0139] The memory 602 can be an internal storage unit of the electronic device 600, such as a hard disk or a memory of the electronic device 600. The memory 602 can also be an external storage device of the electronic device 600, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. provided on the electronic device 600. Further, the memory 602 can include both the internal storage unit and the external storage device of the electronic device 600. The memory 602 is used to store the computer program and other programs and data required by the terminal. The memory 602 can also be used to temporarily store data that has been output or will be output.

[0140] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A loosely coupled data transmission method for a power big data sharing platform, characterized in that, A data management unit applied between multiple business systems and a data sharing platform in a power grid, the method comprising: Receive business data asynchronously sent by various business systems through standardized data interfaces; The business data of each business system is analyzed and classified to determine the data theme and security level of each business data, including public, restricted and confidential levels. Based on the data topics of each business data, the message middleware of the data management unit is updated to multiple topic queues, decoupling time and process to obtain the updated message middleware; Based on the updated security level of each task in the message middleware, determine differentiated data forwarding strategies; Based on the differentiated data forwarding strategy and the real-time resource load of the data management unit and the data sharing platform, business data is sent to the data sharing platform to achieve loosely coupled data transmission.

2. The loosely coupled data transmission method of the power big data sharing platform according to claim 1, characterized in that, The process of parsing and classifying business data from various business systems, and determining the data subject and security level of each business data, includes: The business data of each business system is parsed to extract metadata and content features of each business data. The metadata includes the data source system, data generation timestamp, and data structure identifier. The content features are used to characterize the numerical attributes, identifier attributes, spatiotemporal attributes, and pattern attributes of the data content. The data theme of each business data is determined by matching the metadata and content characteristics of each business data with the pre-set data theme mapping table. Based on a pre-built data security classification rule base, and combined with the data themes and content characteristics of each business data, the security level of each business data is determined.

3. The loosely coupled data transmission method of the power big data sharing platform according to claim 1, characterized in that, The data topics based on each business data are updated to multiple topic queues in the message middleware of the data management unit, decoupling time and process to obtain the updated message middleware, including: Based on the data topics of each business data and the multiple topic queues in the message middleware, a matching process is performed to determine the topic queue corresponding to each business data. Based on the security level of each business data, each business data is securely encapsulated to obtain the message corresponding to each business data. Add the messages corresponding to each business data to the topic queue corresponding to each business data to obtain the updated message middleware; If a message for a certain business data is successfully added to the topic queue, an acknowledgment response is sent back to the business system corresponding to that business data to indicate that the decoupling from that business system has been successful.

4. The loosely coupled data transmission method of the power big data sharing platform according to claim 1, characterized in that, The process of determining differentiated data forwarding strategies based on the updated security levels of each task in the message middleware includes: Configure a high-throughput batch data forwarding strategy for business data with a public security level. For business data with a restricted security level, configure a streaming data forwarding strategy that performs real-time data masking and records complete access logs; For business data with a security level of confidential, configure a single data forwarding strategy that transmits through a dedicated power security channel and confirms each data entry individually; Based on the number of pending messages in each topic queue in the message middleware, the batch size and / or sending rate in the data forwarding strategy of each task are dynamically adjusted.

5. The loosely coupled data transmission method of the power big data sharing platform according to claim 4, characterized in that, The configuration of a high-throughput batch data forwarding strategy for business data with a public security level includes: Identify state abrupt change points and key steady-state parameters in business data with a public security level; the state abrupt change points include the starting points of power sag events and voltage over-limit events; the key steady-state parameters include load peak values ​​and voltage steady-state critical values; Based on the aforementioned state mutation points and key steady-state parameters, the data value density of business data with a public security level is assessed to determine the data value density of business data with a public security level. If the data value density is greater than the value density threshold, then business data with a security level of public is determined to be high-value data, and the batch data forwarding strategy for high-value data is determined to be to encapsulate a high-priority tag in the message and add the message of high-value data to the first topic queue with high priority. If the data value density is less than or equal to the value density threshold, then the business data with the security level of public is determined to be regular steady-state data, and the batch data forwarding strategy is determined to be to add the messages of regular steady-state data to the second topic queue for batch processing after compression and aggregation. The data management unit prioritizes sending the business data of the first topic queue.

6. The loosely coupled data transmission method of the power big data sharing platform according to claim 1, characterized in that, Based on the differentiated data forwarding strategy and the real-time resource load of the data management unit and the data sharing platform, business data is sent to the data sharing platform to achieve loosely coupled data transmission, including: Real-time monitoring of the computing resource utilization of the data management unit and the data interface response time of the data sharing platform; The computing resource utilization rate and data interface response time are compared with a preset resource load threshold to obtain a comparison result; the comparison result is transmission congestion, transmission limitation, and transmission normal. Based on the comparison results and the differentiated data forwarding strategy, a resource load adaptation operation is performed to obtain the transmission scheme for each service data; wherein, when transmission is restricted, the transmission rate of public-level service data using the batch data forwarding strategy is limited; when transmission is congested, the transmission of some non-urgent restricted-level service data is temporarily suspended. Based on the data forwarding strategy and transmission scheme of each business data, the data sharing platform is scheduled to pull and process business data from the topic queue of the message middleware as a consumer.

7. The loosely coupled data transmission method for the power big data sharing platform according to any one of claims 1 to 6, characterized in that, The message middleware includes a main message middleware and a backup message middleware; the method further includes: Receive update information sent by the main message middleware, the update information including change information of each topic queue in the main message middleware, the change information including new task information and completed task information; Based on the updated information, the backup message middleware is synchronously updated; Alternatively, the method may further include: Monitor the connection status and service health of the main message middleware and the backup message middleware; Based on the connection status and health of the primary message middleware and the backup message middleware, determine whether the primary and backup message middleware meet the switching conditions. If the switching conditions are met, the data transmission path will be switched to the backup message middleware for loosely coupled data transmission.

8. The loosely coupled data transmission method for the power big data sharing platform according to any one of claims 1 to 6, characterized in that, The method further includes: The system receives data access requests from data consumers, the data access requests including target data topics and access modes, the access modes including real-time query mode, subscription push mode, and batch download mode; the data consumers include various business systems, external third-party organizations, and public users. Based on the data access request, the data consumer is authenticated and access rights are verified, and the verification result is determined. If the verification result is successful, a data service task is generated based on the access mode and the target data topic. Add the data service task to the topic queue of the message middleware; The data sharing platform acts as a consumer, asynchronously pulling and executing the data service task from the topic queue, and then sending the processing result data back to the data consumer according to the access mode.

9. The loosely coupled data transmission method for the power big data sharing platform according to any one of claims 1 to 6, characterized in that, The method further includes: Receive real-time query requests for business data from data consumers; Based on the real-time query request, the hot data memory cache is queried to determine whether the data to be accessed by the real-time query request exists in the hot data memory cache. If it exists, the data to be accessed in the hot data memory cache will be directly sent back to the data consumer; If it does not exist, then based on the real-time query request, query the full data distributed cache to determine whether the data to be accessed by the real-time query request exists in the full data distributed cache; If it exists, the data to be accessed in the full data distributed cache will be directly sent back to the data consumer. Record the query content of multiple real-time query requests within a set time period; Based on the query content of multiple real-time query requests within a set time period, statistical analysis is performed to determine the candidate set of hot data. Based on the hot data candidate set and the hot data in the hot data memory cache, a fusion update process is performed to obtain the updated thermoelectric data memory cache.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to invoke and execute the computer program stored in the memory to perform the method as claimed in any one of claims 1 to 9.

Images