Intelligent fusion terminal edge cloud data acceleration method based on dataset orchestration

Abstract

The application provides an intelligent fusion terminal edge cloud data acceleration method based on dataset arrangement, wherein the method comprises the following steps: after a data server listens to a dataset access request of a cloud platform, synchronizing a dataset stored by the cloud platform with a power business dataset stored locally; the data server preloads the synchronized power business dataset into one or more intelligent fusion terminals; after the cloud platform receives a power grid task instruction, the data server schedules a corresponding target intelligent fusion terminal for the cloud platform according to a power business dataset required by the power grid task instruction synchronized by the cloud platform, so that the target intelligent fusion terminal executes the power grid task instruction and feeds back an execution result to the cloud platform. The application can reduce the delay of end-to-end tasks, improve real-time performance, improve the reliability of power grid services, and reduce the network bandwidth occupied by data transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power grids, in particular to an intelligent fusion terminal edge cloud data acceleration method based on data set arrangement. BACKGROUND

[0002] The intelligent fusion terminal is a key device in the power distribution system, responsible for data collection, monitoring and control, and gradually bearing more and more intelligent analysis services such as power distribution equipment state evaluation, user load clustering analysis, and switch station image inspection. These services usually rely on AI (Artificial Intelligence) models and large historical sample data sets issued by the cloud.

[0003] In the prior art, the intelligent fusion terminal generally only collects data, and the original data is all uploaded to the cloud for analysis. This method requires high network bandwidth and has large response delay, and cannot meet the real-time requirements of high-speed fault isolation and other services. If the model is fixed in the local intelligent fusion terminal, the model update and data iteration will be time-consuming and laborious, and the efficiency is relatively low, and there is a problem of not timely updating. SUMMARY

[0004] The purpose of the present application is to solve the problems of large response delay, poor real-time performance, low efficiency and not timely data update of the intelligent fusion terminal in the prior art by providing an intelligent fusion terminal edge cloud data acceleration method based on data set arrangement.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] In a first aspect, the present application provides an intelligent fusion terminal edge cloud data acceleration method based on data set arrangement, applied to an intelligent fusion terminal system, the intelligent fusion terminal system comprising: a cloud platform, a data server and a plurality of intelligent fusion terminals, the data server being connected between the cloud platform and the intelligent fusion terminals, the data server being used to store power business data sets, an edge K8s cluster being run in the data server, the method comprising:

[0007] After the data server listens to the data set access request of the cloud platform, the data set stored by the cloud platform is synchronized with the locally stored power business data set;

[0008] The data server preloads the synchronized power business data set into the one or more intelligent fusion terminals;

[0009] The data server schedules a corresponding target intelligent fusion terminal for the cloud platform according to a power business data set required by the power grid task instruction synchronized by the cloud platform after the cloud platform receives the power grid task instruction, so that the target intelligent fusion terminal executes the power grid task instruction and feeds back an execution result to the cloud platform.

[0010] Optionally, the data server schedules a corresponding target intelligent fusion terminal for the cloud platform according to a power business data set required by the power grid task instruction synchronized by the cloud platform after the cloud platform receives the power grid task instruction, and the scheduling includes:

[0011] The data server queries a standby intelligent fusion terminal that synchronously has the power business data set required by the power grid task instruction in a local record after the cloud platform receives the power grid task instruction.

[0012] If the data server queries a plurality of standby intelligent fusion terminals, a target intelligent fusion terminal is selected from the standby intelligent fusion terminals according to a preset priority rule, and the preset priority rule includes any one of the following: maximum current available computing power, closest network topology distance to a data consumer, and minimum current storage pressure.

[0013] Optionally, the heterogeneous computing resource states of the plurality of intelligent fusion terminals are different, and the heterogeneous computing resource states include: being used for neural network processing or being used for image processing.

[0014] The data server queries a standby intelligent fusion terminal that synchronously has the power business data set required by the power grid task instruction in a local record after the cloud platform receives the power grid task instruction, and the querying includes:

[0015] The data server queries a standby intelligent fusion terminal that synchronously has the power business data set required by the power grid task instruction and has a heterogeneous computing resource state matched with the power grid task instruction in a local record after the cloud platform receives the power grid task instruction.

[0016] Optionally, the data server stores data according to a preset sharding rule and a data increment period.

[0017] After the data server listens to a data set access request of the cloud platform, the data server synchronizes a data set stored by the cloud platform with a power business data set stored locally, and the synchronizing includes:

[0018] For power grid time sequence type data stored by the cloud platform, the data server periodically listens to a data set access request of the cloud platform, determines a data shard that has data changes, and synchronizes the data shard that has data changes with the cloud platform.

[0019] Optionally, the data server preloads the synchronized power business data set into the one or more intelligent fusion terminals, comprising:

[0020] The data server preloads the synchronized power business data set into the one or more intelligent fusion terminals by using a zero-copy data supply mechanism.

[0021] Optionally, the method further comprises:

[0022] The data server periodically detects the storage space occupation state of each intelligent fusion terminal, and if the remaining cache space of an intelligent fusion terminal is less than a preset threshold, instructs the intelligent fusion terminal to recycle the cache according to a preset recycling strategy. The preset recycling strategy includes data priority, data latest access time, and data size.

[0023] Optionally, the method further comprises:

[0024] After the data server receives the to-be-uploaded data sent by the intelligent fusion terminal, the data server detects whether the to-be-uploaded data is sensitive data according to a security rule.

[0025] If the data server determines that the to-be-uploaded data is sensitive data, the data server marks the to-be-uploaded data as prohibited upload data and stores the to-be-uploaded data in the edge K8s cluster.

[0026] In a second aspect, the application provides an intelligent fusion terminal, wherein the intelligent fusion terminal is configured to perform the steps performed by the intelligent fusion terminal in the intelligent fusion terminal edge cloud data acceleration method based on data set orchestration according to the first aspect when the intelligent fusion terminal is running.

[0027] In a third aspect, the application further provides an intelligent fusion terminal system, comprising a cloud platform, a data server, and a plurality of intelligent fusion terminals according to the second aspect. The data server is connected between the cloud platform and the intelligent fusion terminals. The data server is configured to store power business data sets. An edge K8s cluster is running in the data server. The intelligent fusion terminal system is configured to perform the steps in the intelligent fusion terminal edge cloud data acceleration method based on data set orchestration according to the first aspect when the intelligent fusion terminal system is running.

[0028] In a fourth aspect, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program. The computer program is configured to perform the steps in the intelligent fusion terminal edge cloud data acceleration method based on data set orchestration according to any one of the first aspect when the computer program is run by a processor.

[0029] The beneficial effects of the present application are: through data synchronization based on a data set access request on the data set stored by the cloud platform and the power business data set stored locally by the data server, the consistency of the local data of the data server and the data in the cloud platform can be ensured, and the power business data set is preloaded into the intelligent fusion terminal through the data server, so that when the intelligent fusion terminal performs the power grid task, it does not need to temporarily schedule data from the cloud platform or the data server, but directly uses the preloaded power business data set, thereby greatly reducing the end-to-end task execution delay and improving the real-time performance. And even if the network interruption problem occurs in the transformer substation and the distribution room, since the intelligent fusion terminal has preloaded the power business data set into the intelligent fusion terminal, the offline operation capability of the intelligent fusion terminal can be enhanced, and the reliability of the power grid service can be improved. In addition, the present application combines data incremental synchronization and local preloading, which can only synchronize incremental data instead of full data, and can also significantly reduce the bandwidth occupation of the power communication private network.

[0030] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0032] Figure 1 An architecture diagram of an intelligent fusion terminal system provided by an embodiment of the present application is shown;

[0033] Figure 2 A flowchart of an intelligent fusion terminal edge cloud data acceleration method based on data set orchestration provided by an embodiment of the present application is shown;

[0034] Figure 3 A flowchart of determining a target intelligent fusion terminal provided by an embodiment of the present application is shown;

[0035] Figure 4 A flowchart of performing sensitive data detection provided by an embodiment of the present application is shown;

[0036] Figure 5 Another architecture diagram of an intelligent fusion terminal system provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application and are not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0038] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0039] The intelligent fusion terminal is a key device of the power distribution system, and needs to carry intelligent analysis services such as power distribution equipment state evaluation and user load clustering analysis. Such services rely on AI models and huge historical sample data sets issued by the cloud.

[0040] The current mainstream mode includes two kinds. The first kind is that the terminal only collects raw data and uploads all the raw data to the cloud for analysis. However, the mode of uploading all data to the cloud has high requirements for network bandwidth and large response delay, which cannot meet the real-time requirements such as rapid fault isolation, and in a weak network scenario, the terminal still needs to return to the cloud to access data not stored locally, causing task blocking.

[0041] The second kind is to solidify the AI model locally in the terminal to reduce data upload dependence. However, the way of solidifying the model locally in the terminal makes the model update and data iteration time-consuming and laborious, and there is a problem of not updating in time. Moreover, when the AI model running on the terminal needs to access sample data not stored locally, it still needs to return to the cloud, causing task blocking and network burden. In the scenarios of unstable network substation, power distribution room, etc., this problem is particularly prominent.

[0042] Based on this, the present application proposes an intelligent fusion terminal edge cloud data acceleration method based on data set. By periodically monitoring and synchronizing the data stored in the cloud platform and locally, only the data synchronization of the data slice with data changes can be realized, avoiding full data transmission and reducing bandwidth consumption. By preloading data into the intelligent fusion terminal, the data can be placed in advance, avoiding temporary retrieval of cloud data during task execution. By scheduling the corresponding intelligent fusion terminal based on the data required by the task instruction, the intelligent fusion terminal that is more matched with the task can be screened out, and the efficiency of task processing is improved.

[0043] AsFigure 1 The diagram shown is an architectural schematic of a smart terminal fusion system according to an embodiment of this application. (Refer to...) Figure 1 The system includes a cloud platform, a data server, and multiple intelligent converged terminals. The data server is connected between the cloud platform and the intelligent converged terminals. The data server is used to store power business datasets and runs an edge Kubernetes (container orchestration platform) cluster. For ease of understanding, the following embodiments of this application use a power grid data analysis scenario as an example, but should not be considered as limited thereto.

[0044] The cloud platform can serve as both the initiator of power grid tasks and the raw storage for global datasets, responsible for issuing business instructions and receiving execution results from terminals. In one possible implementation, the cloud platform includes a cloud storage cluster for storing the raw datasets of power business operations, and a business management module for issuing power grid task instructions. The cloud platform can run a master node in a Kubernetes cluster, storing a global fault waveform library, an equipment image sample library, and an AI model library.

[0045] A data server can serve as a node connecting the cloud and terminals, handling core logic such as data synchronization, preloading, task scheduling, and resource management. An edge Kubernetes cluster runs on the data server. The Kubernetes cluster stores historical power grid data and uploaded data sent by smart converged terminals. Within the data server, a custom resource object, PowerDataset, can be created to abstractly describe power grid business datasets. Data management is based on PowerDataset resources, which at least include the dataset's source, size, access mode, and caching strategy.

[0046] The Kubernetes master node running in the cloud platform can communicate with the edge Kubernetes cluster running in the data server, thereby completing data synchronization between the cloud platform and the data server. The cloud platform can also send task commands to the edge Kubernetes cluster in the data server through the Kubernetes master node, and then send task commands to the intelligent converged terminal through the Kubernetes cluster.

[0047] Intelligent converged terminals can serve as both the execution end for power grid tasks and the local cache end for data, as shown in the following example. Figure 1 The intelligent converged terminals are deployed in clusters within substations. In one possible implementation, the intelligent converged terminals can integrate intelligent analytics services, such as AI inference and fault analysis, and be configured with an NPU for AI inference or a GPU for image processing. After completing their tasks, the intelligent converged terminals can upload the results to a data server, which then reports the data to the cloud platform's Kubernetes master node via an edge Kubernetes cluster.

[0048] Next, in combination with Figure 2 , the intelligent fusion terminal edge cloud data acceleration method based on dataset arrangement of the present application is described. The method can be applied to Figure 1 The intelligent fusion terminal system is shown in FIG. 1, and the method is shown in FIG. 2. The method includes: Figure 2

[0049] S201, after the data server listens to the dataset access request of the cloud platform, the dataset stored by the cloud platform is synchronized with the local stored power business dataset.

[0050] Optionally, the dataset access request can be a request initiated by the cloud platform to trigger cloud and data server data synchronization. In one possible implementation, when the power business dataset of the cloud side is updated, or the edge side needs to obtain the historical data that is not synchronized by the cloud side, the cloud platform can send a dataset access request to the data server.

[0051] Optionally, the locally stored power business dataset can be the data stored locally by the data server, and the power business dataset can be the data required by the intelligent fusion terminal to execute the power grid task, which has the characteristics of large volume and high real-time, including but not limited to at least one of the following: historical fault waveform data, power distribution equipment image sample data, user load clustering data, and power grid time series data.

[0052] The data server can listen to the dataset access request of the cloud platform according to a preset period, and determine the changed data based on the dataset access request listened to in the current period, and synchronize the changed data.

[0053] As one possible implementation, when the cloud platform issues a dataset access request that needs to rely on historical sample data, since the historical data needs to be relied on, the data server may not store it locally at this time. At this time, the dataset stored by the cloud platform can be synchronized with the locally stored power business data, and the data stored by the cloud platform can be synchronized to the data server.

[0054] As another possible implementation, when the power grid data needs to be updated periodically or the power grid adds fault data, the cloud platform can issue a dataset access request to store the data of the cloud platform in the data server.

[0055] S202, the data server preloads the synchronized power business dataset into one or more intelligent fusion terminals.

[0056] Optionally, the data server can first determine the preloaded intelligent terminal range in combination with the power grid business demand, for example, a certain area needs to perform fault analysis task, then the synchronized power business dataset can be preloaded into all intelligent fusion terminals in the area.​

[0057] The data server can actively push the synchronized power service data set into the local cache of the smart fusion terminal to realize preloading of the power service data set in the smart fusion terminal.

[0058] For example, the data server can transmit the power service data set to the smart fusion terminal through memory mapping or the like. When the data amount of the power service data set is greater than a preset threshold, the power service data set can also be transmitted to the smart fusion terminal in fragments.

[0059] S203, after receiving the power grid task instruction on the cloud platform, the data server schedules the corresponding target smart fusion terminal for the cloud platform according to the power service data set required by the cloud platform synchronization power grid task instruction, so that the target smart fusion terminal executes the power grid task instruction and feeds back the execution result to the cloud platform.

[0060] The power grid task instruction can be a specific task instruction initiated by the cloud platform for intelligent analysis of the power distribution system, such as a fault recording analysis instruction, a switch station image inspection recognition instruction, and a power distribution equipment state evaluation instruction.

[0061] The power grid task instruction contains the required power service data set information, which is used to indicate the required power service data set. Taking the fault recording analysis instruction as an example, if the instruction is to determine the fault type of line A, the power service data set of line A can be determined as the power service data set required by the power grid task instruction.

[0062] Optionally, the target smart fusion terminal can be an intelligent fusion terminal that meets the execution conditions of the data server according to the power service data set required by the power grid task instruction.

[0063] The power service data set required for executing the power grid task instruction has been preloaded in the smart fusion terminal. When there are multiple smart fusion terminals that have the power service data set corresponding to the power grid task instruction, the target smart fusion terminal that meets the requirements of the power grid task instruction can be further determined from the multiple smart fusion terminals based on the computing power, network resources and the like required for executing the power grid task instruction and the actual computing power, network resources and the like possessed by the smart fusion terminal.

[0064] Optionally, the target smart fusion terminal can feed back the execution result to the data server after executing the power grid task instruction, and the data server feeds back the execution result to the cloud platform.

[0065] In this embodiment, by synchronizing the dataset stored on the cloud platform and the power business dataset stored locally on the data server based on dataset access requests, the consistency between the local data on the data server and the data on the cloud platform can be ensured. Furthermore, the power business dataset is preloaded into the smart converged terminal via the data server. This allows the smart converged terminal to directly use the locally preloaded power business dataset when performing grid tasks without temporarily scheduling data from the cloud platform or data server, thus significantly reducing end-to-end task execution latency and improving real-time performance. Even if network outages occur in substations or distribution rooms, the preloaded power business dataset enhances the offline operation capability of the smart converged terminal, improving the reliability of grid services. In addition, this application, by combining incremental data synchronization and local preloading, can synchronize only incremental data instead of the full data, significantly reducing the bandwidth consumption of the power communication private network.

[0066] The following describes the process by which the data server, after receiving the power grid task instruction from the cloud platform, schedules the corresponding target intelligent fusion terminal for the cloud platform based on the power business dataset required by the synchronized power grid task instruction. Figure 3 As shown, the above step S203 includes:

[0067] S301. After receiving the power grid task instruction from the cloud platform, the data server queries the local records for candidate intelligent fusion terminals that have synchronized the power business dataset required by the power grid task instruction.

[0068] Among them, local records can be the status information of each intelligent converged terminal stored in the data server, including the type of pre-loaded power business dataset, the currently available computing power, the network topology distance to the data server and other business nodes, and the current percentage of remaining local cache space.

[0069] Optionally, the candidate intelligent converged terminal can be an intelligent converged terminal that meets the power business dataset required by the synchronized power grid task instructions, obtained by filtering based on local records.

[0070] In one possible implementation, all smart fusion terminals that have already loaded the power business dataset required by the grid task instructions can be identified, and these smart fusion terminals can be selected as candidate smart fusion terminals.

[0071] For example, assuming the power grid task requires the load data of line B for the most recent month, all smart fusion terminals loaded with the load data of line B for the most recent month can be selected as smart fusion terminals.

[0072] S302, if the data server queries a plurality of candidate intelligent fusion terminals, a target intelligent fusion terminal is selected from the candidate intelligent fusion terminals according to a preset priority rule.

[0073] The preset priority rule includes any one of the following: maximum current available computing power, closest current network topology distance from a data consumer, and minimum current storage pressure.

[0074] Optionally, the maximum current available computing power can be the highest proportion of currently unused computing resources of the intelligent fusion terminal. If the current power grid task is a computing power intensive task, such as AI fault identification, high-definition image inspection, etc., the target intelligent fusion terminal can be selected from all candidate intelligent fusion terminals that have loaded data sets, and the one with the maximum current computing power is selected as the target intelligent fusion terminal.

[0075] The data consumer can be a data server or other intelligent fusion terminal in the system. The closest current network topology distance from the data consumer can be the shortest physical or logical distance in the network topology, such as the fewest network hops or the lowest communication delay. If the current power grid task has a high demand for real-time result feedback, the target intelligent fusion terminal can be selected from all candidate intelligent fusion terminals that have loaded data sets, and the one with the closest network topology distance from the data consumer is selected as the target intelligent fusion terminal.

[0076] Optionally, the minimum current storage pressure can be the highest proportion of remaining space in the local cache of the intelligent fusion terminal, or the smallest amount of stored data. If the current power grid task requires a new data storage scenario during execution, the target intelligent fusion terminal can be selected from all candidate intelligent fusion terminals that have loaded data sets, and the one with the minimum current storage pressure is selected as the target intelligent fusion terminal.

[0077] It should be noted that in the above S302 step, one of the preset priority rules can be used. Multiple rules can also be used for weighted calculation to determine the target intelligent fusion terminal. For example, based on the demand of the current business scenario or the type of power grid task, the weights of each rule can be pre-set, and the priority of each candidate intelligent fusion terminal can be calculated through a weighted decision model, so that the terminal with the highest priority is determined as the target intelligent fusion terminal.

[0078] For example, assuming that the current power grid task is AI fault identification, which is a computing power intensive type, and the real-time performance of the result feedback needs to be guaranteed, the current available computing power and the current storage pressure can be given a higher weight, and each candidate intelligent fusion terminal can be sorted according to the current available computing power and the current storage pressure. The candidate intelligent fusion terminal with the highest priority in the sorting result, i.e. the one with the maximum current available computing power and the minimum current storage pressure, is determined as the target intelligent fusion terminal.

[0079] In the embodiments of the application, the selected intelligent fusion terminal is obtained through local record matching, which can reduce scheduling delay and improve the real-time performance of data processing. The target intelligent fusion terminal is determined from the selected intelligent fusion terminal based on the preset priority rules, which can realize accurate scheduling of the intelligent fusion terminal and improve the efficiency of power grid business execution.

[0080] Optionally, the heterogeneous computing resource states of the plurality of intelligent fusion terminals are different, and the heterogeneous computing resource states include: a state for neural network processing or a state for image processing.

[0081] The heterogeneous computing resource can be a special hardware resource state carried by the intelligent fusion terminal and used for processing a specific type of computing task, and is used to indicate the type of task that the intelligent fusion terminal is good at executing.

[0082] If the heterogeneous computing resource of the intelligent fusion terminal is for neural network processing, the hardware carrier thereof can be an NPU (Neural Processing Unit, neural network processor); if the heterogeneous computing resource of the intelligent fusion terminal is for image processing, the hardware carrier thereof can be a GPU (Graphics Processing Unit, graphics processing unit).

[0083] The step of querying, by the data server, the selected intelligent fusion terminal that synchronously has the power business data set required by the power grid task instruction in the local record after the cloud platform receives the power grid task instruction includes:

[0084] The data server queries, in the local record, the selected intelligent fusion terminal that synchronously has the power business data set required by the power grid task instruction and has a heterogeneous computing resource state matched with the power grid task instruction after the cloud platform receives the power grid task instruction.

[0085] The data server can filter out the intelligent fusion terminal whose heterogeneous computing resource state can efficiently support the task according to the type of the power grid task instruction.

[0086] In a possible implementation, the power grid task instruction includes a task type, and the local record can save the task identification of the heterogeneous computing resource state of each intelligent fusion terminal. The selected intelligent fusion terminal whose heterogeneous computing resource state is matched with the power grid task instruction can be determined based on the matching of the task type and the task identification.

[0087] For example, it is assumed that the power grid task instruction is "identifying the fault type of A line based on an AI model", and the task type is mainly for neural network processing. Therefore, the intelligent fusion terminal that has synchronously synchronized the power business data set and has a heterogeneous computing resource state for neural network processing can be selected as the selected intelligent fusion terminal in the local record.

[0088] In the embodiments of the application, the first re-screening is performed by whether to preload the power service data set, the second re-screening is performed based on the heterogeneous computing resource state of the intelligent fusion terminal, and a more suitable intelligent fusion terminal can be selected based on the demand of the power grid task, so that the hardware capability of the intelligent fusion terminal is aligned with the power grid task, the speed of task execution is improved, the heterogeneous resources of the terminal are used more efficiently, and resource waste is avoided.

[0089] Optionally, the data server stores data according to a preset sharding rule and a data increment period.

[0090] The preset sharding rule can be a preset rule for the data server to store the power service data set in blocks, for example, time series data with large capacity and continuously generated is split into multiple independent data shards according to a specific dimension, so as to facilitate subsequent accurate positioning of changed data.

[0091] In a possible implementation, the data can be blocked according to the time dimension and / or the equipment dimension, for example, time series data of every N hours is split into a shard; or time series data of different equipment and different lines is split into different shards; or the time series data of every N hours is split into a shard according to the time dimension, and then the shard is split into multiple sub-shards according to the equipment dimension.

[0092] Optionally, the data increment period can be a fixed time interval at which the data server listens to whether the cloud platform power grid time series data has new / updated data, that is, how often to check whether the cloud time series data has changed.

[0093] The setting of the data increment period can be based on the collection frequency of the power grid time series data, so that periodic listening can cover the data update frequency and not miss key incremental data.

[0094] After the data server listens to the data set access request of the cloud platform, the process of synchronizing the data set stored by the cloud platform with the locally stored power service data set includes:

[0095] For the power grid time series type data stored by the cloud platform, the data server periodically listens to the data set access request of the cloud platform, determines the data shard with data changes, and synchronizes the data shard with data changes with the cloud platform.

[0096] Optionally, the power grid time series type data stored by the cloud platform can be time series data in the power grid system, for example, PMU phasor data, power quality data, real-time acquisition data of line voltage / current, power change data of distribution equipment, user load fluctuation data, time series waveform data in the fault recording process, etc.

[0097] In a possible implementation, the PowerDataset in the data server can define the incremental synchronization rule. For example, after the first full load, the controller will only pull the latest data segment generated from the cloud every 5 minutes and append it to the edge cache, thereby saving synchronization traffic.

[0098] The data segment where data changes occur can be a time-series data segment that is inconsistent with the local storage content in the cloud platform, which is discovered by the data server through periodic monitoring. The data segment can use the combination of generation time and device identifier to obtain a segment identifier, and the data server can determine whether the segment is new data based on the segment identifier.

[0099] As a possible implementation, when the data server downloads data from the cloud platform, the cloud platform can calculate a hash value (such as SHA-256) for the data or data segment, and record this hash value as a fingerprint on the tamper-proof blockchain. In this way, after the data reaches the data server, the data server can check the integrity of the data based on the fingerprint recorded in the blockchain.

[0100] For example, the data server can perform data segmentation on the power grid time-series type data according to the rule of "segmenting data every 1 hour according to time dimension", and obtain two current data segments of "2024-05-20 12:00-13:00" and "2024-05-20 13:00-14:00" of line A. When the preset monitoring period arrives, the data server can automatically monitor the data set access request of the cloud platform, determine that the power grid time-series type data of line A of the cloud platform needs to be synchronized by analyzing the data set access request, and determine that only the data segment of "2024-05-20 13:00-14:00" has changed by comparing the time-series data segment list of line A of the cloud platform with the locally stored line A segment list. Then, the data server can only pull this data segment from the cloud platform, check the integrity of the data based on the fingerprint of the data segment, and realize incremental update of the data.

[0101] In the embodiments of the present application, by segmenting data and only transmitting the changed data segment, the problem of excessive bandwidth occupation and long time consumption caused by full data update can be avoided. Moreover, the intelligent fusion terminal can obtain the latest time-series data in time.

[0102] The following is a further description of the above-mentioned preloading of the synchronized power business data set into one or more intelligent fusion terminals by the data server, which includes the following steps:

[0103] The data server uses a zero-copy data supply mechanism to pre-load the synchronized power business data set into one or more intelligent fusion terminals.

[0104] The zero-copy data supply mechanism refers to establishing a data transmission channel directly between a storage area of a data server and a local cache of the intelligent fusion terminal by skipping multiple copies or intermediate cache links of data between a user mode and a kernel mode, and realizing one-time writing and direct use of data.

[0105] For example, the intelligent fusion terminal can directly access the data block stored in the server through memory mapping or a direct IO channel, or the data is directly written from the data server storage to the cache of the intelligent fusion terminal.

[0106] In a possible implementation, the power service data set is preloaded to the intelligent fusion terminal, and the power service data can be stored in a high-speed storage area of the intelligent fusion terminal for temporary storage of preloaded data, thereby ensuring that the intelligent fusion terminal can call data with zero delay when performing a power grid task, and maximizing the acceleration effect of preloading.

[0107] The intelligent fusion terminal includes a heterogeneous computing resource for neural network processing or image processing, and the high-speed storage area can be a display memory of an NPU or a GPU, thereby ensuring that the computing unit can directly obtain data and start processing without copying the data from the host memory of the intelligent fusion terminal to the display memory, and further compressing the data preparation time.

[0108] For example, a direct data channel can be established between the storage of the data server and the NPU / GPU in the intelligent fusion terminal through a PCIe bus (Peripheral Component Interconnect Express Bus), thereby enabling the data to be directly stored from the data server to the display memory of the NPU / GPU without passing through CPU processing or host memory caching.

[0109] In the embodiment of the application, the zero-copy data supply mechanism can improve the attack speed of data and further improve the efficiency of the intelligent fusion terminal in performing a task.

[0110] The data server can also control the cache space of the intelligent fusion terminal as a whole to further improve the method of the application.

[0111] The data server periodically detects the storage space occupation state of each intelligent fusion terminal, and if the remaining cache space of the intelligent fusion terminal is less than a preset threshold, instructs the intelligent fusion terminal to recycle the cache according to a preset recycling strategy.

[0112] The preset recycling strategy includes data priority, data latest access time, and data size.

[0113] In a possible implementation, the data server can define a cache space quota for each intelligent fusion terminal in advance, that is, the maximum capacity of the local cache of each intelligent fusion terminal is set in advance, so as to avoid unlimited occupation of terminal storage resources by preloading data. The setting of the cache space quota can be determined according to the type of the service carried by the intelligent fusion terminal. For example, a terminal processing fault analysis tasks can be allocated a larger cache space quota, for example, 20 GB, and a terminal processing daily load statistics can be allocated a smaller cache space quota, for example, 5 GB, to avoid resource waste.

[0114] Optionally, the cache space usage of the intelligent fusion terminal can be monitored in real time by the data server or a control module arranged in the intelligent fusion terminal, the value of the current cache data in the terminal is evaluated by an LRU-K algorithm (Least Recently Used-K Algorithm) according to the data priority of the data, the latest access time of the data, and the size of the data, and the low-value data is automatically recycled when the cache space is insufficient, to ensure that the high-priority service data is preferentially cached in the intelligent fusion terminal. The data priority can be a pre-set importance level of the data to the power service, the latest access time of the data can be the time when the data is last called by the intelligent fusion terminal, and the size of the data can be the cache space occupied by the data.

[0115] It should be noted that the cache recycling of the intelligent fusion terminal can be based on at least one of the above recycling strategies, for example, only the data priority is used for cache recycling, or the value of the data is calculated in combination with the data priority and the latest access size of the data, and the low-value data is recycled, or the value of the data is calculated in combination with the data priority, the latest access time of the data, and the size of the data, and the low-value data is recycled, or other evaluation indexes can be introduced for calculating the value of the data, and the specific mode is not limited herein.

[0116] The storage space occupation state of the intelligent fusion terminal can be the proportion of the cached data in the total cache space, or the proportion of the remaining cache space in the total cache space, or the value of the remaining cache space. The pre-set threshold can be set based on the service demand of each intelligent fusion terminal, for example, intelligent fusion terminals carrying different service types can have different pre-set thresholds. For an intelligent fusion terminal with complex calculation, a lower pre-set threshold can be set to ensure that the terminal has sufficient cache space for calculation and processing. For a terminal that needs to pull long-period historical data for processing, a higher pre-set threshold can be set to ensure that the terminal can store more historical data as much as possible.

[0117] Optionally, the data server can also detect the data uploaded by the intelligent fusion terminal, to further ensure the security of the system, for example, Figure 4As shown, the method of this application further includes:

[0118] S401. After receiving the data to be uploaded from the intelligent fusion terminal, the data server checks whether the data to be uploaded is sensitive data according to the security rules.

[0119] Sensitive data can refer to data that involves power system security, user privacy, or trade secrets, and whose dissemination scope needs to be strictly limited. Examples include parameters of key power grid equipment that could lead to attacks on the power grid system if leaked, user privacy data, and power grid topology and operation data.

[0120] In one possible implementation, the data server can be pre-configured with a sensitive data feature library to store the data characteristics of sensitive data. After receiving the data to be uploaded from the intelligent fusion terminal, the data server can identify the data to be uploaded based on the sensitive data feature library. For example, it can identify whether the data to be uploaded is sensitive data, or whether the data to be uploaded contains sensitive data, through keyword matching, data format recognition, or encryption identifier detection.

[0121] S402. If the data server determines that the data to be uploaded is sensitive data, it marks the data to be uploaded as prohibited data and stores the data to be uploaded to the edge Kubernetes cluster.

[0122] If the data to be uploaded is sensitive, it can be cached on the local storage of the data server, such as the local storage in the edge Kubernetes cluster, or cached in the encrypted cache of the smart converged terminal, and the data to be uploaded should not be uploaded to the cloud platform.

[0123] If the data to be uploaded contains sensitive data, it can be anonymized. For example, only the results of processing and analyzing the sensitive data can be uploaded, while the sensitive data itself can be omitted. For instance, only the aggregated electricity load curve of users in the region can be uploaded, while the electricity load curve of individual users can be omitted. This prevents the leakage of sensitive data without affecting the display of the data results.

[0124] In this embodiment of the application, by performing security checks on the data to be uploaded through a data server, the data security of the power system can be guaranteed, the risk of edge data leakage can be reduced, and the reliability and security of the intelligent converged terminal system can be further improved.

[0125] Optionally, the intelligent fusion terminal system can be deployed in a distributed cluster manner, that is, multiple intelligent fusion terminals form a cluster. If the network of the cloud platform fails and the current intelligent fusion terminal needs to access data set A, the data server can obtain data set A from other intelligent fusion terminal clusters and send data set A to the current intelligent fusion terminal, or the current intelligent fusion terminal can directly establish communication with a nearby intelligent fusion terminal containing data set A and obtain data set A.

[0126] For example, assume that Figure 1 Intelligent fusion terminal 1 in cluster A needs to access the fault waveform data set of B line in the last month, but it is not cached locally, and the network between cluster A and the cloud center cannot perform data synchronization due to rainstorm. At this time, intelligent fusion terminal 1 can send a data acquisition request to the data server, and the data server determines that intelligent fusion terminal 4 in cluster B caches the fault waveform data set of B line in the last month based on the data acquisition request and local records. Then, the data server can obtain the data set from intelligent fusion terminal 4 in cluster B and send the data set to intelligent fusion terminal 1.

[0127] In another example, the intelligent fusion terminal can also scan cluster B based on the power network topology relationship through the edge side local area network. If it is detected that intelligent fusion terminal 4 in cluster B caches the data set, a P2P encrypted transmission channel can be established between intelligent fusion terminal 1 and intelligent fusion terminal 4, and the data set can be transmitted from intelligent fusion terminal 4 to intelligent fusion terminal 1 in a fragmentation transmission manner.

[0128] If multiple clusters simultaneously request data from a certain cluster, the requested cluster can allocate bandwidth according to the business priority of the requesting cluster, so as to avoid the influence of the transmission pressure of the requested cluster on its own business. After completing the data transmission, the intelligent fusion terminal requesting data can check the data integrity based on the fingerprint of the data set.

[0129] Optionally, as Figure 5 shown, the intelligent fusion terminal system can also establish a connection with an external electronic device, and the dispatcher can obtain data and perform data analysis through the electronic device.

[0130] In a possible implementation manner, the electronic device can receive a data requirement intention described by natural language or advanced DSL (Advanced Domain-Specific Language) input by the dispatcher, for example, “I need the fault waveform of A line in the last month and the corresponding weather data”, and send the data requirement intention to the cloud platform. After receiving the data requirement intention, the cloud platform can perform intention analysis on the data requirement intention to determine the data required by the user.

[0131] If all the data required by the user is contained in the cloud platform, the data can be directly called from the cloud platform and returned to the requesting user. If all the data is not included in the cloud platform or the cloud platform network is limited, the data requested by the user can be filtered from the K8s edge cluster of the data server, and the data is returned to the requesting user.

[0132] In a possible implementation, if data scheduling is performed through the K8s edge cluster of the data server, the data server can also construct a PowerDataset object and its cache strategy based on the associated data set obtained through data requirement intention analysis, the PowerDataset object contains the combined associated data set and is marked with a priority, so that the user can directly call the PowerDataset object for data association analysis, reducing the complexity of user management of the data set.

[0133] The embodiment of the present application also provides an intelligent fusion terminal, which can be deployed in a power distribution cabinet, used for collecting power grid data of the deployed area, and performing data calculation, fault processing and remote control.

[0134] Based on the same inventive concept, the embodiment of the present application also provides an intelligent fusion terminal system, the architecture diagram of the system can refer to Figure 1 , including a cloud platform, a data server and a plurality of intelligent fusion terminals. Since the system solving problem principle in the embodiment of the present application is similar to the above-mentioned intelligent fusion terminal edge cloud data acceleration method based on data set arrangement in the embodiment of the present application, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be described in detail.

[0135] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed when the processor is running, and the processor executes the steps in the above-mentioned intelligent fusion terminal edge cloud data acceleration method based on data set arrangement.

[0136] In the embodiment of the present application, the computer program running on the processor can also execute other machine readable instructions to execute the method as described in the embodiment, and the specific method steps and principles executed are described in the embodiment, which will not be described in detail here.

[0137] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. The embodiments described above are merely exemplary, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, and electrical, mechanical or other forms.

[0138] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0139] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0140] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.

[0141] It should be noted that: similar reference numerals and letters in the following drawings represent similar items, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third" and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0142] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, and are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features within the technical scope disclosed by the present application. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. All should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An intelligent converged terminal edge cloud data acceleration method based on dataset orchestration, characterized in that, The application is applied to an intelligent fusion terminal system, which comprises a cloud platform, a data server and a plurality of intelligent fusion terminals. The data server is connected between the cloud platform and the intelligent fusion terminals. The data server is used for storing power service data sets. An edge K8s cluster is run in the data server. The data server comprises a custom resource object for abstractly describing power grid service data sets, and data management is performed based on the custom resource object. The custom resource object at least contains the source, size, access mode and cache strategy of the data set. The method comprises the following steps: For the power grid time series type data stored in the cloud platform, the data server periodically monitors the data set access request of the cloud platform, determines the data shard with data changes, synchronizes the data shard with data changes with the cloud platform, and stores data according to a preset shard rule and data increment period in the data server; The data server preloads the synchronized power service data set into one or more intelligent fusion terminals. The data server adopts a zero-copy data supply mechanism to preload the synchronized power service data set into the high-speed storage area of the one or more intelligent fusion terminals. The zero-copy data supply mechanism refers to establishing a data transmission channel between the storage area of the data server and the local cache of the intelligent fusion terminal, and preloading the power service data set to the intelligent fusion terminal through the data transmission channel. After the cloud platform receives a power grid task instruction, the data server schedules a corresponding target intelligent fusion terminal for the cloud platform according to the power service data set required by the power grid task instruction synchronized by the cloud platform, so that the target intelligent fusion terminal executes the power grid task instruction and feeds back the execution result to the cloud platform.

2. The method of claim 1, wherein, After the cloud platform receives a power grid task instruction, the data server schedules a corresponding target intelligent fusion terminal for the cloud platform according to the power service data set required by the power grid task instruction synchronized by the cloud platform, comprising: After the cloud platform receives a power grid task instruction, the data server queries the standby intelligent fusion terminal which synchronizes the power service data set required by the power grid task instruction in the local record; If the data server queries a plurality of standby intelligent fusion terminals, a target intelligent fusion terminal is selected from the standby intelligent fusion terminals according to a preset priority rule. The preset priority rule comprises any one of the following: the maximum current available computing power, the nearest network topology distance from the data consumer, and the minimum current storage pressure.

3. The method of claim 2, wherein, The heterogeneous computing resource states of the plurality of intelligent fusion terminals are different. The heterogeneous computing resource states comprise a neural network processing or an image processing. After the cloud platform receives a power grid task instruction, the data server queries the standby intelligent fusion terminal which synchronizes the power service data set required by the power grid task instruction in the local record, comprising: The data server queries a local record for a to-be-selected intelligent fusion terminal that has a power service data set required by the power grid task instruction and a heterogeneous computing resource state matching the power grid task instruction after the cloud platform receives the power grid task instruction.

4. The method of claim 1, wherein, The method further includes: The data server periodically detects the storage space occupation state of each intelligent fusion terminal. If the remaining cache space of an intelligent fusion terminal is less than a preset threshold, the intelligent fusion terminal is instructed to recycle the cache according to a preset recycling strategy. The preset recycling strategy includes data priority, data latest access time, and data size.

5. The method of claim 1, wherein, The method further includes: After the data server receives the to-be-uploaded data sent by the intelligent fusion terminal, the data server detects whether the to-be-uploaded data is sensitive data according to a security rule. If the data server determines that the to-be-uploaded data is sensitive data, the data server marks the to-be-uploaded data as prohibited upload data and stores the to-be-uploaded data in the edge K8s cluster.

6. An intelligent converged terminal, characterized by, The intelligent fusion terminal, when running, is configured to perform the steps performed by the intelligent fusion terminal in the intelligent fusion terminal edge cloud data acceleration method based on data set orchestration according to any one of claims 1-5.

7. An intelligent converged terminal system, characterized by, The intelligent fusion terminal system includes a cloud platform, a data server, and a plurality of intelligent fusion terminals according to claim 6. The data server is connected between the cloud platform and the intelligent fusion terminals. The data server is configured to store power service data sets. An edge K8s cluster runs in the data server. When the intelligent fusion terminal system runs, it is configured to perform the steps in the intelligent fusion terminal edge cloud data acceleration method based on data set orchestration according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program. When the computer program is run by a processor, the steps of the intelligent fusion terminal edge cloud data acceleration method based on data set orchestration according to any one of claims 1-5 are performed.

Images