Cloud-edge cooperative multi-modal streaming data processing method and system

Abstract

The invention discloses a cloud-edge cooperative multi-modal streaming data processing method and system, and the method comprises the steps that an edge calculation layer receives a streaming data processing request sent by a sensing detection layer to form an analysis process, and judges and marks the type of the analysis process, and the type of the analysis process comprises a repeated process and a new process; the edge computing layer at least executes one of a resource evaluation step and a step of allocating an analysis carrier and an executor for executing an analysis task to an analysis node in the analysis process according to the type of the labeled analysis process; data processing tasks are executed according to the sequence of all the analysis nodes in the analysis process, and analysis results are transmitted among different analysis carriers when needed. By means of the scheme, the appropriate computing resources are allocated according to the actual requirements of the analysis process, resource waste or resource insufficiency can be avoided, and different processing steps are adopted according to the process type so as to adapt to different data processing requirements and scenes.

Description

Technical Field

[0001] This application generally relates to the field of intelligent operation and maintenance and fault prediction technology for wind turbine generator sets. More specifically, this application relates to a cloud-edge collaborative multimodal streaming data processing method and system. Background Technology

[0002] With the large-scale deployment of new energy power plants and the accelerated construction of new power systems, the installed capacity of wind turbines in the power grid has increased significantly, and their operational stability and reliability are crucial to the power system.

[0003] However, traditional wind farm monitoring systems face numerous challenges in meeting current demands. These systems generally suffer from insufficient processing capabilities for multi-source, heterogeneous data from wind turbines, making effective integration and analysis difficult. Furthermore, due to delays in data processing and analysis, existing systems often exhibit lagging situational awareness, failing to reflect the actual operating status of wind turbines in a timely manner. In addition, traditional monitoring systems typically lack real-time response mechanisms at the edge, resulting in slow response times for abnormal operating conditions requiring rapid handling.

[0004] Currently, many existing technical solutions rely on periodic data upload patterns, centralizing data for analysis and processing in the cloud (i.e., centralized cloud computing). While this model can handle large amounts of data, its data transmission and processing latency makes it difficult to meet real-time requirements when capturing and responding to millisecond-level operational anomalies.

[0005] In view of this, there is an urgent need to provide a cloud-edge collaborative multimodal streaming data processing solution to efficiently process the multimodal data of wind turbine units, achieve rapid and accurate situational awareness and real-time early warning, thereby improving the operational reliability and maintenance economy of wind turbine units. Summary of the Invention

[0006] In order to at least address one or more of the technical problems mentioned above, this application proposes a cloud-edge collaborative multimodal streaming data processing scheme in several aspects.

[0007] In a first aspect, this application provides a cloud-edge collaborative multimodal streaming data processing method, comprising: an edge computing layer receiving a streaming data processing request sent by a sensing and detection layer to form an analysis process, and determining and labeling the type of the analysis process, wherein the type of the analysis process includes a recurring process and a new process; the edge computing layer, according to the labeled type of the analysis process, performing at least one of the following steps: a resource assessment step and an allocation of analysis carriers and executors to the analysis nodes in the analysis process to perform analysis tasks; executing data processing tasks in the order of each analysis node in the analysis process, and transmitting analysis results between different analysis carriers when necessary.

[0008] In some embodiments, during the process of receiving streaming data processing requests from the sensing and detection layer at the edge computing layer to form an analysis process, and determining and labeling the type of the analysis process, the following steps are performed: the process manager in the edge computing layer reads the data analysis method corresponding to the streaming data in the streaming data processing request and forms an analysis process containing one or more analysis nodes; the process manager searches the analysis process in the cloud knowledge base in the cloud computing layer and the edge knowledge base in the edge computing layer to determine whether it is a duplicate process; in response to the analysis process not being a duplicate process, the analysis process is labeled as a new process; in response to the analysis process being a duplicate process, the analysis process is labeled as a duplicate process.

[0009] In some embodiments, when the analysis process is a new process, at least one of the steps of performing a resource assessment step and assigning an analysis carrier and executor to the analysis nodes in the analysis process includes: performing a resource assessment on each analysis node in the new process and calculating the amount of resources required for each analysis node; obtaining the remaining available resources of the edge computing layer and the cloud computing layer from a first resource manager configured in the edge computing layer and a second process manager configured in the cloud computing layer, respectively; determining the analysis carrier for each analysis node in the analysis process based on the remaining available resources of the edge computing layer and the cloud computing layer, and assigning an executor to each analysis node; wherein the analysis carrier is an edge computing layer or a cloud computing layer; and the amount of resources required for each analysis node in the analysis process is not greater than the remaining available resources of its corresponding analysis carrier.

[0010] In some embodiments, when the analysis process is a repeating process, at least one of the steps of performing a resource assessment step and assigning an analysis carrier and executor to the analysis node in the analysis process includes: calling a historical data analysis report from a cloud knowledge base in the cloud computing layer or a historical data analysis report from an edge knowledge base in the edge computing layer to obtain a process matching result and a resource assessment result that match the repeating process.

[0011] In some embodiments, during the transmission of analysis results between different analysis carriers when necessary, the following steps are performed: After the data processing task of each analysis node is completed, it is determined whether the analysis carrier of the next analysis node is the same as the analysis carrier of the current analysis node; in response that the analysis carrier of the next analysis node is the same as the analysis carrier of the current analysis node, the analysis results of the current analysis node are saved, and the data processing task of the next analysis node continues to be executed; in response that the analysis carrier of the next analysis node is different from the analysis carrier of the current analysis node, the communication resources between the analysis carrier of the current analysis node and the analysis carrier of the next analysis node are evaluated to determine whether the communication resources are sufficient to complete the current analysis task. Data transmission of analysis results from the current analysis node to the analysis carrier of the next analysis node; in response to sufficient communication resources to complete the data transmission of analysis results from the current analysis node to the analysis carrier of the next analysis node, saving the analysis results of the current analysis node and sending the analysis results to the analysis carrier of the next analysis node; in response to insufficient communication resources to complete the data transmission of analysis results from the current analysis node to the analysis carrier of the next analysis node, adding the data transmission task of the analysis results of the current analysis node to a waiting queue until sufficient communication resources are available to complete the data transmission of analysis results from the current analysis node to the analysis carrier of the next analysis node, and then sending the analysis results to the analysis carrier of the next analysis node.

[0012] In a second aspect, this application provides a cloud-edge collaborative multimodal streaming data processing system, employing the cloud-edge collaborative multimodal streaming data processing method as described in any embodiment of the first aspect for cloud-edge collaborative multimodal streaming data processing. The system includes: a sensing and detection layer configured to collect wind turbine operating status data and environmental data, and perform preliminary data transfer and storage; an edge computing layer communicating with the sensing and detection layer, configured to receive and preprocess multimodal streaming data from the sensing and detection layer, manage the analysis process and computing resources on the edge side, and perform some real-time data analysis and early warning tasks; and a cloud computing service layer communicating with the edge computing layer, configured to aggregate and analyze data, and manage... The cloud-based analysis process and computing resources are managed, and the cloud-based edge computing layer collaborates with the edge computing layer to complete the situational awareness and early warning of wind turbine units. The sensing and detection layer includes: wind turbine monitoring equipment, composed of a multimodal monitoring sensor network, used to collect real-time operating status data of corresponding points of the wind turbine units; an information system, integrating and accessing the wind farm's SCADA system and environmental meteorological monitoring module, used to acquire electrical parameters and external environmental data, and to transfer operating status data of corresponding points of the wind turbine units and communicate with the edge computing layer; and a database used to store real-time streaming data and historical operating records of the equipment, and to support data retrieval and command issuance between the edge computing layer and the cloud computing service layer.

[0013] In some embodiments, the edge computing layer includes: a gateway node, configured to communicate with the information system of the sensing layer via a wired or wireless interface to build a real-time data transmission service, realize data transmission and synchronization between the edge node and the sensing layer, and between the edge node and the cloud computing service layer, and realize cloud-edge data synchronization and computing task allocation by communicating with the cloud computing service layer; and an edge server, including a data preprocessor, a first process manager, and a first resource manager, wherein the data preprocessor is configured to preprocess the multimodal streaming data uploaded by the sensing layer and upload it to the cloud computing service layer, and the first process manager is configured to collaborate with the cloud computing service layer to realize streaming data processing and analysis. The collaborative computing process includes a first resource manager for coordinating and unifying the management of computing, storage, and network resources of the edge server, providing the first process manager with device resource information of the edge computing layer, and enabling resource assessment at the edge. It also includes an edge model library, comprising a first model processor and an edge knowledge base. The first model processor is used to deploy models at the edge and perform real-time simulations on some streaming data to generate dynamic early warning thresholds. The edge knowledge base stores historical data analysis reports and models and data corresponding to actual operating conditions, supports the first process manager in performing repetitive process judgment and resource assessment, and enables model addition, deletion, updating, and iteration through communication with the cloud computing service layer.

[0014] In some embodiments, the cloud computing service layer includes: a cloud server, which includes a second process manager and a second resource manager. The second process manager collaborates with the first process manager in the edge computing layer to schedule computing resources, process multimodal data fusion tasks and model inference analysis nodes, and form an analysis process according to a resource allocation strategy. The second resource manager is used to achieve coordinated and unified management of cloud center computing resources, storage resources, and network resources, and to collect real-time resources on the cloud and edge sides. It provides the second process manager with resource information of each edge terminal in the cloud and edge computing layers to generate resource scheduling processes and resource evaluation for the cloud. A cloud model library includes a second model processor and a cloud knowledge base. The second model processor is used to execute computational models for data association analysis and to perform trend prediction on corresponding streaming data using the computing resources of the cloud server. The cloud knowledge base is used to construct multimodal spatiotemporal analysis models and fault knowledge graphs, store historical data analysis reports to support repetitive process judgment and resource evaluation, and collaborate with the edge model library in the edge computing layer to complete multimodal streaming data processing and model updates for wind turbine units.

[0015] In some embodiments, the executors of the edge computing layer and the cloud computing service layer are configured to execute data processing tasks sequentially according to the analysis nodes in the analysis process determined by the first process manager and the second process manager, and to evaluate and utilize communication resources to transmit or queue analysis results when the analysis carrier changes.

[0016] In some embodiments, during the deployment of the system, the following steps are performed: resource integration and processing of wind turbine monitoring equipment, information systems, and databases in the sensing and detection layer; generation of unique equipment identification codes for various types of equipment according to unified coding rules; setting up gateway nodes based on the equipment identification codes; constructing edge servers and edge model libraries for the edge computing layer, and deploying data preprocessors and a first process manager within the edge servers; deploying cloud servers for the cloud computing service layer at the data center end, and establishing real-time data interaction channels with each node of the edge computing layer through networks and industrial ring networks, wherein the cloud servers aggregate and integrate preprocessed data transmitted from the nodes of the edge computing layer, and deeply integrate with the database of the sensing and detection layer to construct a cloud model library for the cloud computing service layer; the cloud model library, based on the monitoring information fed back by the edge computing layer, performs model input, real-time updates, and iterations on the edge model library of the edge computing layer, and combines the information in the database of the sensing and detection layer to form an edge knowledge base in the edge model library corresponding to the actual operating status.

[0017] Through the cloud-edge collaborative multimodal streaming data processing scheme provided above, this embodiment of the application receives streaming data processing requests sent by the sensing and detection layer through the edge computing layer to form an analysis process, and judges and labels the type of the analysis process, which can lay the foundation for subsequent differentiated processing. By having the edge computing layer execute at least one step—a resource assessment step and an allocation of analysis carriers and executors to the analysis nodes in the analysis process—based on the labeled analysis process type, it helps to allocate appropriate computing resources according to the actual needs of the analysis process, avoiding resource waste or shortage, ensuring effective resource utilization, and enabling different processing steps according to the process type, thus adapting to different data processing needs and scenarios. By executing data processing tasks sequentially according to the analysis nodes in the analysis process, and transmitting analysis results between different analysis carriers when needed, complex analysis tasks can be decomposed and executed on different carriers that may have specific capabilities, thereby improving data processing efficiency, resource utilization, and the flexibility and intelligence level of data processing.

[0018] Furthermore, in some embodiments, the process manager uses an analysis method to transform the streaming data processing request into an analysis process consisting of analysis nodes, thereby providing a basis for subsequent processing and management. By retrieving the analysis process from the cloud knowledge base and edge knowledge base, it is determined whether the analysis process is a duplicate process, providing a key basis for subsequent differentiated processing.

[0019] Furthermore, in some embodiments, when the analysis process is a new process, by assessing the resource requirements of the analysis nodes and the remaining available resources of the edge computing layer and cloud computing layer, it ensures that tasks are allocated and executed only when resources are sufficient, avoiding execution failures due to insufficient resources, and intelligently determining whether each analysis task is executed at the edge computing layer or the cloud computing layer. This helps to achieve optimal overall resource allocation and avoid overloading edge resources or wasting cloud resources. Through cloud-edge collaboration, the system can handle a wider range of more complex analysis tasks.

[0020] Furthermore, in some embodiments, when the analysis process is a repetitive process, by directly reusing historical resource evaluation results and process configurations, complex steps such as recalculating, querying, and allocating resources are avoided, which greatly shortens the task startup time and improves the response speed.

[0021] Furthermore, in some embodiments, during the transmission of analysis results between different analysis carriers when necessary, the transmission of analysis results between different analysis carriers is managed to ensure that the processing results of the current analysis node can be reliably and effectively transmitted to the next analysis node using a different analysis carrier. Communication resources are assessed before cross-carrier transmission to avoid failures and resource waste caused by blind transmission when resources are insufficient. When communication resources are insufficient, the transmission task is added to a waiting queue until resources are sufficient before transmission is executed. This improves the success rate and reliability of data transmission and avoids data loss or process interruption due to temporary resource issues. Even with cross-carrier transmission and resource limitations, the analysis results are ultimately guaranteed to reach the next node, maintaining the continuity of the entire analysis process. Attached Figure Description

[0022] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0023] Figure 1 An exemplary flowchart of the cloud-edge collaborative multimodal streaming data processing method according to an embodiment of this application is shown;

[0024] Figure 2 An exemplary flowchart illustrating the formation analysis process of an embodiment of this application, and determining and labeling the type of the analysis process;

[0025] Figure 3 An exemplary flowchart of an embodiment of this application is shown, in which the steps of resource evaluation and allocation of analysis carriers and executors to analysis nodes to perform analysis tasks are executed sequentially when the analysis process is of the new process type;

[0026] Figure 4 This illustration shows an exemplary flowchart of an embodiment of the present application, in which data processing tasks are executed sequentially according to the analysis nodes in the analysis process, and analysis results are transmitted between different analysis carriers when necessary.

[0027] Figure 5 An exemplary structural block diagram of the cloud-edge collaborative multimodal streaming data processing system according to an embodiment of this application is shown;

[0028] Figure 6 An exemplary flowchart illustrating the deployment of a cloud-edge collaborative multimodal streaming data processing system according to an embodiment of this application is shown. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be understood that the terms "comprising" and "including" used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0032] Figure 1An exemplary flowchart of a cloud-edge collaborative multimodal streaming data processing method 100 according to an embodiment of this application is shown.

[0033] like Figure 1 As shown, in step S110, the edge computing layer receives the streaming data processing request sent by the sensing and detection layer to form an analysis process, and judges and labels the type of the analysis process. The types of analysis processes include repetitive processes and new processes.

[0034] The specific process involved in step S110 in the embodiments of this application can be found in [reference needed]. Figure 2 .

[0035] Figure 2 An exemplary flowchart illustrating the formation analysis process of an embodiment of this application, and determining and labeling the type of the analysis process, is provided.

[0036] like Figure 2 As shown, in step S210, the process manager in the edge computing layer reads the data analysis method corresponding to the streaming data in the streaming data processing request and forms an analysis process containing one or more analysis nodes. In step S220, the process manager searches for the analysis process in the cloud knowledge base in the cloud computing layer and the edge knowledge base in the edge computing layer to determine whether it is a duplicate process. In response to the analysis process not being a duplicate process, in step S230, the analysis process is marked as a new process. In response to the analysis process being a duplicate process, in step S240, the analysis process is marked as a duplicate process.

[0037] In the embodiments of this application, a specific execution sequence is generated or organized based on the data analysis method corresponding to the streaming data in the streaming data processing request. That is, an analysis process consisting of one or more analysis nodes is used to actually process the streaming data.

[0038] In the embodiments of this application, existing analysis process definitions or records are retrieved from the cloud knowledge base in the cloud computing layer and the edge knowledge base in the edge computing layer. The resulting analysis process is then compared with the retrieved process to determine whether it is a duplicate process. If the requested analysis process already exists in either the cloud knowledge base in the cloud computing layer or the edge knowledge base in the edge computing layer, the process manager can identify it, thereby avoiding unnecessary reconstruction and deployment of the exact same process and saving computing, storage, and network resources.

[0039] After completing step S110, in step S120, the edge computing layer performs at least one of the following steps, based on the type of the labeled analysis process: the resource assessment step and the step of assigning analysis carriers and executors to the analysis nodes in the analysis process to perform analysis tasks.

[0040] In the embodiments of this application, when the analysis process is a new process, the resource assessment step and the step of allocating analysis carriers and executors to the analysis nodes in the analysis process are executed sequentially. The specific process involved in step S120 can be found in [reference needed]. Figure 3 .

[0041] Figure 3 An exemplary flowchart of an embodiment of this application is shown, in which resource evaluation and analysis carrier and executor steps are sequentially performed to allocate analysis tasks to analysis nodes when the analysis process is of the new process type.

[0042] like Figure 3 As shown, in step S310, resource assessment is performed on each analysis node in the new process, and the resource quantity required for each analysis node is calculated. In step S320, the remaining available resources of the edge computing layer and the cloud computing layer are obtained from the first resource manager configured in the edge computing layer and the second process manager configured in the cloud computing layer, respectively. In step S330, based on the remaining available resources of the edge computing layer and the cloud computing layer, the analysis carrier of each analysis node in the analysis process is determined, and an executor is assigned to each analysis node.

[0043] In the embodiments of this application, the analysis carrier is an edge computing layer or a cloud computing layer.

[0044] In the embodiments of this application, the amount of resources required by each analysis node in the analysis process is no greater than the remaining available resources of its corresponding analysis carrier.

[0045] In the embodiments of this application, during the process of determining the analysis carrier for each analysis node in the analysis process, each analysis node in the new process is examined one by one (or in a certain optimized order). For the analysis node i currently being examined, if the resources required by analysis node i are less than or equal to the remaining available resources of the edge computing layer, then the edge computing layer is a potential analysis carrier for analysis node i. If the resources required by analysis node i are less than or equal to the remaining available resources of the cloud computing layer, then the cloud computing layer is a potential analysis carrier for analysis node i. Next, when only one potential analysis carrier exists, that is, when the resources required by analysis node i are less than or equal to the remaining available resources of the edge computing layer, and the resources required by analysis node i are greater than or equal to the remaining available resources of the cloud computing layer, then the analysis carrier for analysis node i is determined to be the edge computing layer. Or, when the resources required by analysis node i are less than or equal to the remaining available resources of the cloud computing layer, and the resources required by analysis node i are greater than or equal to the remaining available resources of the edge computing layer, then the analysis carrier for analysis node i is determined to be the cloud computing layer.

[0046] Then, when two potential analysis carriers exist, that is, when the resources required by analysis node i are both less than or equal to the remaining available resources of the edge computing layer, the current strategy is determined, and the analysis carrier for analysis node i is determined based on the current measurement. Specifically, when the current strategy is an edge-first strategy, it tends to keep tasks at the edge to reduce latency, lower the cost of data transmission to the cloud, and utilize data locality. Therefore, the analysis carrier for analysis node i is determined to be the edge computing layer. When the current strategy is a cloud-first / offload strategy, it tends to offload computationally intensive tasks or tasks requiring more global data to the cloud to free up edge resources or utilize the more powerful computing capabilities of the cloud. Therefore, the analysis carrier for analysis node i is determined to be the cloud computing layer. When the current strategy is a cost-optimization strategy, the analysis carrier for analysis node i is determined to be the one with the lower operating cost between the edge computing layer and the cloud computing layer. When the current strategy is a load-balancing strategy, considering the overall load of the current edge computing layer and the cloud computing layer, the one with the lower load is selected as the analysis carrier. When the current strategy is set to data dependency and affinity, the data flow between analysis node i and its predecessor and successor nodes is considered. If the predecessor node is at the edge, and the current node is also suitable for the edge computing layer, the edge computing layer will be prioritized to reduce cross-carrier data transmission. When the current strategy is set to node characteristic strategy, analysis node i may be more suitable for a certain analysis carrier due to its characteristics (such as dependence on specific hardware or extremely high latency sensitivity).

[0047] In the embodiments of this application, during the process of allocating executors to each analysis node, after the analysis carrier of each analysis node is determined, a specific executor is allocated to each analysis node on the corresponding analysis carrier. Specifically, during the executor allocation process, the following conditions must be met: the allocated executor must be able to handle the analysis task type defined by the analysis node (for example, an image recognition task needs to be allocated to an executor with image processing capabilities). The allocated executor must have sufficient computing resources (CPU, memory, etc.) to meet the needs of the analysis node. The allocated executor must be currently available and have an appropriate load. A load balancing strategy needs to be met to ensure that executors are not overly concentrated on a few physical or virtual units.

[0048] Through steps S310-S330, each analysis node can be intelligently, efficiently and reliably allocated to appropriate computing resources (analysis carrier and executor), thereby maximizing the advantages of cloud-edge collaborative computing and improving system performance, flexibility and resource utilization efficiency.

[0049] In the embodiments of this application, when the analysis process is a repetitive process, only the resource assessment step is executed. Specifically, historical data analysis reports from the cloud knowledge base in the cloud computing layer or the edge knowledge base in the edge computing layer are invoked to obtain process matching results and resource assessment results that match the repetitive process.

[0050] In the embodiments of this application, the aforementioned resource assessment results include historical resource requirements, historical allocation schemes, execution results, etc. Historical resource requirements refer to how many resources were previously assessed to be needed for this process (and its various nodes). Historical allocation schemes refer to how the analysis carrier (which node runs at the edge, which runs in the cloud) and executor were allocated to this process (and its various nodes) in the past. Execution results include the performance data of the last execution or the success or failure indicators to ensure that a good configuration is reused.

[0051] By directly accessing historical data through the above process, resource assessment can be completed extremely quickly, thereby rapidly initiating the analysis process and significantly reducing task preparation time. Simultaneously, unnecessary calculations and frequent interactions with the resource manager are avoided, reducing the overall system load. Furthermore, for the same repetitive tasks, using historically validated resource configurations helps ensure the stability and consistency of analysis results and performance.

[0052] After step S120 is completed, in step S130, data processing tasks are executed in the order of each analysis node in the analysis process, and analysis results are transmitted between different analysis carriers when necessary.

[0053] The specific process involved in step S130 in the embodiments of this application can be found in [reference needed]. Figure 4 .

[0054] Figure 4 This illustration shows an exemplary flowchart of an embodiment of the present application, in which data processing tasks are executed sequentially according to the analysis nodes in the analysis process, and analysis results are transmitted between different analysis carriers when necessary.

[0055] like Figure 4As shown, in step S410, after the data processing task of each analysis node is completed, it is determined whether the analysis carrier of the next analysis node is the same as the analysis carrier of the current analysis node. If the analysis carrier of the next analysis node is the same as the analysis carrier of the current analysis node, in step S420, the analysis result of the current analysis node is saved, and the data processing task of the next analysis node continues to be executed. If the analysis carrier of the next analysis node is different from the analysis carrier of the current analysis node, in step S430, the communication resources between the analysis carrier of the current analysis node and the analysis carrier of the next analysis node are evaluated to determine whether the communication resources are sufficient to complete the data transmission of the analysis result of the current analysis node to the analysis carrier of the next analysis node. If the communication resources are sufficient to complete the data transmission of the analysis result of the current analysis node to the analysis carrier of the next analysis node, in step S440, the analysis result of the current analysis node is saved, and the analysis result is sent to the analysis carrier of the next analysis node. In response to the inability of communication resources to complete the data transmission of the analysis results of the current analysis node to the analysis carrier of the next analysis node, in step S450, the data transmission task of the analysis results of the current analysis node is added to the waiting queue, and the process returns to step S430 until the communication resources are sufficient to complete the data transmission of the analysis results of the current analysis node to the analysis carrier of the next analysis node, at which point the analysis results are sent to the analysis carrier of the next analysis node.

[0056] In embodiments of this application, performing data processing tasks includes executing a model processor to extrapolate streaming data using an artificial intelligence model in order to generate situational awareness and real-time early warning data.

[0057] Through steps S410-S450, when the next analysis node and the current analysis node are on the same analysis carrier, the analysis results of the current node are saved, and then the next analysis node can directly start processing the task. Data does not need to be transmitted over long distances or across networks; it can be accessed directly by subsequent nodes within the same analysis carrier, resulting in high efficiency. When the analysis carriers of the next analysis node and the current analysis node are different, a communication resource assessment is performed. If communication resources are insufficient, the process waits instead of blindly sending data, avoiding forced data transmission under poor network conditions. This reduces the risk of data transmission failure, timeout, or data corruption due to network problems. This ensures that the analysis process will not immediately crash or be interrupted when faced with temporary network fluctuations or resource constraints, but can wait for conditions to improve, thereby improving the robustness of the entire analysis process. Although it waits when resources are insufficient, data will be transmitted immediately once resources allow, ensuring that downstream nodes can obtain the required data as quickly as possible. For nodes on the same carrier, execution continues directly, ensuring maximum efficiency.

[0058] In summary, through the cloud-edge collaborative multimodal streaming data processing scheme provided above, this application embodiment receives streaming data processing requests sent by the sensing and detection layer through the edge computing layer to form an analysis process, and judges and labels the type of the analysis process, which can lay the foundation for subsequent differentiated processing. By having the edge computing layer execute at least one step—the resource assessment step and the step of allocating analysis carriers and executors to the analysis nodes in the analysis process—based on the labeled analysis process type, it helps to allocate appropriate computing resources according to the actual needs of the analysis process, avoiding resource waste or shortage, ensuring effective resource utilization, and enabling different processing steps to be taken according to the process type, thus adapting to different data processing needs and scenarios. By executing data processing tasks sequentially according to the analysis nodes in the analysis process, and transmitting analysis results between different analysis carriers when needed, complex analysis tasks can be decomposed and executed on different carriers that may have specific capabilities, thereby improving data processing efficiency, resource utilization, and the flexibility and intelligence level of data processing.

[0059] Furthermore, in some embodiments, the process manager uses an analysis method to transform the streaming data processing request into an analysis process consisting of analysis nodes, thereby providing a basis for subsequent processing and management. By retrieving the analysis process from the cloud knowledge base and edge knowledge base, it is determined whether the analysis process is a duplicate process, providing a key basis for subsequent differentiated processing.

[0060] Furthermore, in some embodiments, when the analysis process is a new process, by assessing the resource requirements of the analysis nodes and the remaining available resources of the edge computing layer and cloud computing layer, it ensures that tasks are allocated and executed only when resources are sufficient, avoiding execution failures due to insufficient resources, and intelligently determining whether each analysis task is executed at the edge computing layer or the cloud computing layer. This helps to achieve optimal overall resource allocation and avoid overloading edge resources or wasting cloud resources. Through cloud-edge collaboration, the system can handle a wider range of more complex analysis tasks.

[0061] Furthermore, in some embodiments, when the analysis process is a repetitive process, by directly reusing historical resource evaluation results and process configurations, complex steps such as recalculating, querying, and allocating resources are avoided, which greatly shortens the task startup time and improves the response speed.

[0062] Furthermore, in some embodiments, when the analysis process is a new process, during the transmission of analysis results between different analysis carriers when necessary, the transmission of analysis results between different analysis carriers is managed to ensure that the processing results of the current analysis node can be reliably and effectively transmitted to the next analysis node using a different analysis carrier. Communication resources are assessed before cross-carrier transmission to avoid failures and resource waste caused by blind transmission when resources are insufficient. When communication resources are insufficient, the transmission task is added to a waiting queue until resources are sufficient before transmission is executed. This improves the success rate and reliability of data transmission and avoids data loss or process interruption due to temporary resource issues. Even with cross-carrier transmission and resource limitations, the analysis results are ultimately guaranteed to reach the next node, maintaining the continuity of the entire analysis process.

[0063] This application also provides a cloud-edge collaborative multimodal streaming data processing system, which can use the aforementioned cloud-edge collaborative multimodal streaming data processing method 100 to perform cloud-edge collaborative multimodal streaming data processing, or other methods to perform cloud-edge collaborative multimodal streaming data processing. This application does not impose any restrictions on this.

[0064] Figure 5 An exemplary structural block diagram of a cloud-edge collaborative multimodal streaming data processing system according to an embodiment of this application is shown.

[0065] like Figure 5 As shown, the system 500 includes a sensing and detection layer 510, an edge computing layer 520, and a cloud computing service layer 530.

[0066] In the embodiments of this application, the sensing and detection layer 510 is configured to collect wind turbine operating status data and environmental data, and perform preliminary data transfer and storage.

[0067] Specifically, the sensing and detection layer 510 includes wind turbine monitoring equipment 511, an information system 512, and a database 513.

[0068] Specifically, the wind turbine monitoring equipment 511 consists of a multimodal monitoring sensor network, used to collect real-time operating status data of corresponding points on the wind turbine. The information system 512 integrates the wind farm's SCADA system and environmental meteorological monitoring module, used to acquire electrical parameters and external environmental data, and to transfer operating status data of corresponding points on the wind turbine, as well as to communicate with the edge computing layer. The database 513 stores real-time streaming data and historical equipment operation records, and supports data retrieval and command issuance between the edge computing layer and the cloud computing service layer.

[0069] In the embodiments of this application, the wind turbine monitoring device 511 can collect real-time operating status data of key points of equipment such as wind turbine gearbox, blades and generator.

[0070] In the embodiments of this application, the edge computing layer 520 communicates with the sensing and detection layer 510 and is configured to receive and preprocess multimodal streaming data from the sensing and detection layer 510, manage the analysis process and computing resources on the edge side, and perform some real-time data analysis and early warning tasks.

[0071] In the embodiments of this application, the edge computing layer 520 includes a gateway node 521, an edge server 522, and an edge model library 523.

[0072] Specifically, the gateway node 521 is used to communicate with the information system 512 of the sensing and detection layer 510 through a wired or wireless interface to build a real-time data transmission service, realize data transmission and synchronization between the edge node and the sensing and detection layer 510, and between the edge node and the cloud computing service layer 530, and realize cloud-edge data synchronization and computing task allocation by communicating with the cloud computing service layer 530.

[0073] Specifically, the edge server 522 includes a data preprocessor 5221, a first process manager 5222, and a first resource manager 5223.

[0074] Specifically, the data preprocessor 5221 is used to preprocess the multimodal streaming data uploaded by the sensing and detection layer 510 and upload it to the cloud computing service layer 530.

[0075] In the embodiments of this application, the data preprocessor 5221 can perform preprocessing steps including data noise reduction, time-frequency domain feature extraction, and anomaly screening to extract high-quality data.

[0076] Specifically, the first process manager 5222 is used to collaborate with the cloud computing service layer 530 to achieve collaborative computing of the streaming data processing and analysis process.

[0077] Specifically, the first resource manager 5223 is used to achieve coordinated and unified management of the computing resources, storage resources and network resources of the edge server 522, and to provide the first process manager 5222 with device resource information of the edge computing layer 520 to realize resource evaluation on the edge side.

[0078] Specifically, the edge model library 523 includes a first model processor 5231 and an edge knowledge base 5232.

[0079] Specifically, the first model processor 5231 is used to deploy models on the edge side and perform real-time simulations on some streaming data to generate dynamic early warning thresholds.

[0080] Specifically, the edge knowledge base 5232 is used to store historical data analysis reports and models and data corresponding to actual operating conditions, supports the first model processor 5231 to perform repetitive process judgment and resource evaluation, and realizes model addition, deletion, update and iteration through communication with the cloud computing service layer 530.

[0081] In the embodiments of this application, the cloud computing service layer 530 communicates with the edge computing layer 520, is configured to aggregate and analyze data, manage the analysis process and computing resources in the cloud, and collaborate with the edge computing layer 520 to complete the situational awareness and early warning of the wind turbine.

[0082] Specifically, the cloud computing service layer 530 includes a cloud server 531 and a cloud model library 532.

[0083] Specifically, the cloud server 531 includes a second process manager 5311 and a second resource manager 5312. The second process manager 5311 collaborates with the first process manager 5222 of the edge computing layer 520 to schedule computing resources, process multimodal data fusion tasks and model inference analysis nodes, and form analysis processes according to resource allocation strategies. The second resource manager 5312 is used to realize the coordinated and unified management of cloud center computing resources, storage resources and network resources, and to collect real-time resources on the cloud and edge sides. It provides the second process manager 5311 with resource information of each edge terminal on the cloud and edge computing layer 520 to generate resource scheduling processes and resource evaluation for the cloud.

[0084] Specifically, the cloud model library 532 includes a second model processor 5321 and a cloud knowledge base 5322. The second model processor 5321 is used to execute computational models for data association analysis and to perform trend prediction on the corresponding streaming data using the computing resources of the cloud server 531. The cloud knowledge base 5322 is used to construct multimodal spatiotemporal analysis models and fault knowledge graphs, store historical data analysis reports to support repetitive process judgment and resource assessment, and collaborate with the edge model library 523 in the edge computing layer 520 to complete multimodal streaming data processing and model updates for wind turbine units.

[0085] In the embodiments of this application, the executors of the edge computing layer 520 and the cloud computing service layer 530 are configured to execute data processing tasks sequentially according to the analysis nodes in the analysis process determined by the first process manager 5222 and the second process manager 5311, and when the analysis carrier changes, the analysis results are transmitted or queued by evaluating and utilizing communication resources through the aforementioned gateway node 521 and the network.

[0086] Through the close collaboration between the edge computing layer 520 and the cloud computing service layer 530, the streaming data of wind turbine units is deeply mined and efficiently analyzed. This fully leverages the powerful computing capabilities of cloud computing and the real-time advantages of edge computing to achieve accurate perception and real-time early warning of the wind turbine unit's operating status. It provides intelligent and intuitive operation and maintenance decision support to wind power operation and maintenance personnel, significantly improving the operating efficiency and safety of wind turbine units and endowing wind power operation and maintenance with vigorous intelligent vitality.

[0087] When system 500 performs cloud-edge collaborative multimodal streaming data processing using the aforementioned cloud-edge collaborative multimodal streaming data processing method 100, step S110 is executed through the sensing and detection layer 510 and the edge computing layer 520, and steps S120 and S130 are executed through the edge computing layer 520 and the cloud computing service layer 530. The specific execution process can be found above and will not be repeated here.

[0088] In the embodiments of this application, the deployment process of the aforementioned system 500 can be referred to Figure 6 .

[0089] Figure 6 An exemplary flowchart illustrating the deployment of a cloud-edge collaborative multimodal streaming data processing system according to an embodiment of this application is shown.

[0090] like Figure 6 As shown, in step S610, the wind turbine monitoring equipment 511, information system 512, and database 513 in the sensing and detection layer 510 are integrated and processed. Unique equipment identification codes are generated for each type of equipment according to unified coding rules, and gateway nodes 521 are set up based on these identification codes. In step S620, edge servers 522 and edge model libraries 523 of the edge computing layer 520 are constructed, and a data preprocessor 5221 and a first process manager 5222 are deployed within the edge server 522. In step S630, a cloud server 531 of the cloud computing service layer 530 is deployed at the data center end, and a real-time data interaction channel is established with each node of the edge computing layer 520 through the network and industrial ring network. The cloud server 531 aggregates and integrates the preprocessed data transmitted from the nodes of the edge computing layer 520 and deeply integrates it with the database 513 of the sensing and detection layer 510 to construct the cloud model library 532 of the cloud computing service layer 530. In step S640, the cloud model library 532 performs model input, real-time updates and iterations on the edge model library 523 of the edge computing layer 520 based on the monitoring information fed back by the edge computing layer 520, and combines the information in the database 513 of the sensing and detection layer 510 to form an edge knowledge base 5232 in the edge model library 523 that corresponds to the actual operating status.

[0091] In the embodiments of this application, during step S610, the wind turbine monitoring equipment 511, information system 512, and database 513 in the sensing and detection layer 510 are first integrated and classified. Data consistency and redundancy are strictly verified, and a unified naming rule is used to set identification codes for various types of data. A precise data access model is constructed, and a standardized description file is output. After data acquisition, the consistency and redundancy of the data are verified a second time. A unique device identification code is generated according to the unified coding rule, a standardized data access interface system is built, and a highly compatible gateway node 521 is set up to seamlessly connect to various access protocols and methods such as video, images, and industrial data.

[0092] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A cloud-edge collaborative multimodal streaming data processing method, characterized in that, include: The edge computing layer receives streaming data processing requests sent by the sensing and detection layer to form an analysis process, and determines and labels the type of the analysis process, wherein the type of the analysis process includes repetitive processes and new processes; The edge computing layer performs at least one of the following steps, depending on the type of the labeled analysis process: a resource assessment step and an analysis carrier and executor step, to allocate analysis tasks to analysis nodes in the analysis process. Data processing tasks are executed sequentially according to each analysis node in the analysis process, and analysis results are transmitted between different analysis carriers when necessary.

2. The cloud-edge collaborative multimodal streaming data processing method according to claim 1, characterized in that, In the process of receiving streaming data processing requests from the sensing and detection layer at the edge computing layer to form an analysis flow, and determining and labeling the type of the analysis flow, the following steps are performed: In the edge computing layer, the process manager reads the data analysis method corresponding to the streaming data in the streaming data processing request and forms an analysis process containing one or more analysis nodes. The process manager retrieves the analysis process from the cloud knowledge base in the cloud computing layer and the edge knowledge base in the edge computing layer to determine whether it is a duplicate process. In response to the fact that the analysis process is not a repeated process, the analysis process is marked as a new process; In response to the analysis process being a repetitive process, the analysis process is marked as a repetitive process.

3. The cloud-edge collaborative multimodal streaming data processing method according to claim 1 or 2, characterized in that, When the analysis process is a new process, at least one of the following steps is performed: the resource assessment step and the step of assigning analysis carriers and executors to the analysis nodes in the analysis process to perform analysis tasks: Resource assessment is performed on each analysis node in the new process, and the resource requirement for each analysis node is calculated. The remaining available resources of the edge computing layer and the cloud computing layer are obtained from the first resource manager configured in the edge computing layer and the second process manager configured in the cloud computing layer, respectively. Based on the remaining available resources of the edge computing layer and the cloud computing layer, the analysis carrier of each analysis node in the analysis process is determined, and an executor is assigned to each analysis node; The analysis carrier is an edge computing layer or a cloud computing layer; The amount of resources required for each analysis node in the analysis process is no greater than the remaining available resources of its corresponding analysis carrier.

4. The cloud-edge collaborative multimodal streaming data processing method according to claim 1 or 2, characterized in that, When the analysis process is a repetitive process, at least one of the following steps is performed: the resource assessment step and the step of assigning analysis carriers and executors to the analysis nodes in the analysis process to perform analysis tasks: The system retrieves historical data analysis reports from either the cloud knowledge base in the cloud computing layer or the edge knowledge base in the edge computing layer to obtain process matching results and resource evaluation results that match the repetitive process.

5. The cloud-edge collaborative multimodal streaming data processing method according to claim 1, characterized in that, When transferring analysis results between different analytical carriers as needed, the following steps are performed: After the data processing task of each analysis node is completed, determine whether the analysis carrier of the next analysis node is the same as the analysis carrier of the current analysis node. In response to the fact that the analysis carrier of the next analysis node is the same as that of the current analysis node, the analysis results of the current analysis node are saved, and the data processing task of the next analysis node is continued to be executed. In response to the fact that the analysis carrier of the next analysis node is different from the analysis carrier of the current analysis node, the communication resources between the analysis carrier of the current analysis node and the analysis carrier of the next analysis node are evaluated to determine whether the communication resources are sufficient to complete the data transmission of the analysis results of the current analysis node to the analysis carrier of the next analysis node. In response to the fact that the communication resources are sufficient to complete the data transmission of the analysis results of the current analysis node to the analysis carrier of the next analysis node, the analysis results of the current analysis node are saved, and the analysis results are sent to the analysis carrier of the next analysis node; In response to the fact that the communication resources are insufficient to complete the data transmission of the analysis results of the current analysis node to the analysis carrier of the next analysis node, the data transmission task of the analysis results of the current analysis node is added to the waiting queue until the communication resources are sufficient to complete the data transmission of the analysis results of the current analysis node to the analysis carrier of the next analysis node, at which point the analysis results are sent to the analysis carrier of the next analysis node.

6. A cloud-edge collaborative multimodal streaming data processing system, characterized in that, The system employs the cloud-edge collaborative multimodal streaming data processing method as described in any one of claims 1-5 to perform cloud-edge collaborative multimodal streaming data processing, wherein the system comprises: The sensing and detection layer is configured to collect wind turbine operating status data and environmental data, and to perform preliminary data transfer and storage. The edge computing layer communicates with the sensing and detection layer, is configured to receive and preprocess multimodal streaming data from the sensing and detection layer, manage the analysis process and computing resources on the edge side, and perform some real-time data analysis and early warning tasks. The cloud computing service layer communicates with the edge computing layer, is configured to aggregate and analyze data, manage the analysis process and computing resources in the cloud, and collaborates with the edge computing layer to complete the situational awareness and early warning of wind turbine units. The sensing and detection layer includes: The wind turbine monitoring equipment consists of a multi-modal monitoring sensor network, used to collect real-time operating status data of corresponding points of the wind turbine. The information system integrates the wind farm's SCADA system and environmental meteorological monitoring module to acquire electrical parameters and external environmental data, and to transfer operational status data of corresponding wind turbine locations and communicate with the edge computing layer; and The database is used to store real-time streaming data and historical operation records of devices, and supports data retrieval and command issuance between the edge computing layer and the cloud computing service layer.

7. The cloud-edge collaborative multimodal streaming data processing system according to claim 6, characterized in that, The edge computing layer includes: Gateway nodes are used to communicate with the information system of the sensing and detection layer through wired or wireless interfaces to build real-time data transmission services, realize data transmission and synchronization between edge nodes and the sensing and detection layer, and between edge nodes and the cloud computing service layer, and realize cloud-edge data synchronization and computing task allocation by communicating with the cloud computing service layer. An edge server includes a data preprocessor, a first process manager, and a first resource manager. The data preprocessor preprocesses the multimodal streaming data uploaded by the sensing layer and uploads it to the cloud computing service layer. The first process manager collaborates with the cloud computing service layer to achieve collaborative computing in the streaming data processing and analysis process. The first resource manager coordinates and unifies the management of the edge server's computing, storage, and network resources, provides the first process manager with device resource information of the edge computing layer, and enables resource assessment at the edge. The edge model library includes a first model processor and an edge knowledge base. The first model processor is used to deploy models on the edge side and perform real-time simulations on some streaming data to generate dynamic early warning thresholds. The edge knowledge base is used to store historical data analysis reports and models and data corresponding to actual operating conditions. It supports the first process manager in performing repetitive process judgment and resource evaluation, and realizes model addition, deletion, updating and iteration through communication with the cloud computing service layer.

8. The cloud-edge collaborative multimodal streaming data processing system according to claim 7, characterized in that, The cloud computing service layer includes: The cloud server includes a second process manager and a second resource manager. The second process manager collaborates with the first process manager in the edge computing layer to schedule computing resources, process multimodal data fusion tasks and model inference analysis nodes, and form an analysis process according to the resource allocation strategy. The second resource manager is used to realize the coordinated and unified management of cloud center computing resources, storage resources and network resources, and to collect real-time resources on the cloud and edge sides. It provides the second process manager with resource information of each edge terminal in the cloud and edge computing layers to generate resource scheduling processes and resource evaluation for the cloud. The cloud-based model library includes a second model processor and a cloud-based knowledge base. The second model processor is used to execute computational models for data correlation analysis and to perform trend prediction on the corresponding streaming data using the computing power resources of the cloud server. The cloud-based knowledge base is used to construct multimodal spatiotemporal analysis models and fault knowledge graphs, store historical data analysis reports to support repetitive process judgment and resource assessment, and collaborate with the edge model library in the edge computing layer to complete multimodal streaming data processing and model updates for wind turbine units.

9. The cloud-edge collaborative multimodal streaming data processing system according to claim 8, characterized in that, The executors of the edge computing layer and the cloud computing service layer are configured to execute data processing tasks sequentially according to the analysis nodes in the analysis process determined by the first process manager and the second process manager, and to transmit or queue the analysis results through the gateway node and network evaluation and utilize communication resources when the analysis carrier changes.

10. The cloud-edge collaborative multimodal streaming data processing system according to claim 8, characterized in that, During the deployment of the system, the following steps are performed: The monitoring equipment, information system and database of wind turbine units in the sensing and detection layer are integrated and processed. Unique equipment identification codes are generated for various types of equipment according to unified coding rules. Gateway nodes are set up according to the equipment identification codes. Build edge servers and edge model libraries for the edge computing layer, and deploy data preprocessors and first process managers within the edge servers; A cloud server for the cloud computing service layer is deployed at the data center end, and a real-time data interaction channel is established with each node of the edge computing layer through the network and industrial ring network. The cloud server aggregates and integrates the pre-processed data transmitted from the nodes of the edge computing layer and deeply integrates it with the database of the sensing and detection layer to build a cloud model library for the cloud computing service layer. The cloud-based model library inputs, updates, and iterates the edge model library of the edge computing layer in real time based on the monitoring information fed back by the edge computing layer, and combines the information in the database of the sensing and detection layer to form an edge knowledge base in the edge model library that corresponds to the actual operating status.

Images