A cloud computing-based big data processing method and system

Abstract

This application discloses a cloud computing-based big data processing method and system. The method, implemented on a cloud computing platform, includes: decomposing a big data processing task into multiple micro-tasks and distributing them to distributed computing nodes; each computing node selecting neighboring nodes based on a hyperlocality algorithm and constructing a local task processing model using only local data and metadata of neighboring nodes dynamically selected based on a fast game mechanism. The metadata includes the semantic features and task dependencies of the micro-task data of neighboring nodes. The semantic features are extracted using natural language processing techniques, and the task dependencies are determined based on the input-output association of the micro-tasks; according to the local task processing model, the micro-tasks are executed independently on each computing node, and the allocation of micro-tasks is dynamically adjusted through a point-to-point local negotiation mechanism; the micro-task processing results from each computing node are aggregated to generate a global data processing result. This method reduces computational and communication overhead.

Description

Technical Field

[0001] This application relates to the fields of cloud computing and big data processing technology, and in particular to a cloud computing-based big data processing method and system. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, distributed computing systems have been widely used to handle large-scale data tasks. Traditional big data processing methods, such as frameworks based on MapReduce or Spark, typically rely on centralized scheduling or global coordination mechanisms to allocate tasks and optimize resource utilization. However, these methods face challenges in high real-time or low-bandwidth scenarios (such as edge computing and the Internet of Things), mainly due to the high communication overhead and latency caused by global coordination.

[0003] Therefore, there is an urgent need for a big data processing method that does not require global coordination and is based on localized decision-making, in order to reduce communication overhead, improve real-time performance and adaptability, especially in distributed cloud computing environments. Summary of the Invention

[0004] This application provides a cloud computing-based big data processing method and system that reduces computational and communication overhead.

[0005] This application provides the following solution:

[0006] According to the first aspect, a cloud computing-based big data processing method is provided. The method is implemented on a cloud computing platform and includes: decomposing a big data processing task into multiple micro-tasks and allocating them to distributed computing nodes; selecting neighboring nodes for each computing node based on a hyperlocality algorithm, and constructing a local task processing model using only local data and metadata of the neighboring nodes dynamically selected based on a fast game mechanism. The metadata includes the semantic features of the micro-task data of the neighboring nodes and task dependencies. The semantic features are extracted using natural language processing techniques, and the task dependencies are determined based on the input-output association of the micro-tasks; executing the micro-tasks independently on each computing node according to the local task processing model, and dynamically adjusting the micro-task allocation through a point-to-point local negotiation mechanism; and aggregating the micro-task processing results of each computing node to generate a global data processing result.

[0007] According to one achievable method in an embodiment of this application, the decomposition of the big data processing task into multiple micro-tasks includes: decomposing the task into micro-tasks with a granularity smaller than a preset threshold according to the data type and processing requirements of the big data processing task, wherein the decomposition is based on task-based semantic segmentation.

[0008] According to one achievable method in the embodiments of this application, the selection of neighboring nodes for each computing node based on the hyperlocal algorithm includes: each computing node dynamically selects the neighboring nodes by calculating the semantic similarity threshold of the microtask and the dependency chain length, wherein the selection mechanism is based on the local topology optimization algorithm to ensure that the number of neighboring nodes does not exceed a preset upper limit.

[0009] According to one achievable method in this application embodiment, the construction of a local task processing model using only local data and metadata of neighboring nodes dynamically selected based on a fast game mechanism includes: each computing node analyzing local data and the metadata through a fast game mechanism based on micro-task relevance to determine the execution priority of local micro-tasks, wherein the fast game mechanism optimizes task allocation between nodes without global coordination by calculating the semantic relevance and dependency weights between local micro-tasks; and constructing an adaptive local task processing model based on the execution priority and the neighboring nodes, wherein each node dynamically adjusts its cooperation relationship with neighboring nodes according to the real-time changes of micro-tasks to optimize the local execution path of micro-tasks.

[0010] According to one achievable method in the embodiments of this application, the fast game mechanism adopts an iterative algorithm based on local Nash equilibrium, wherein each computing node calculates the optimal task allocation strategy based on semantic relevance and dependency weight within a finite number of iterations, and the number of iterations is limited by a preset number threshold.

[0011] According to one achievable method in this application embodiment, the dynamic adjustment of microtask allocation through a point-to-point local negotiation mechanism includes: each computing node exchanging microtask status and processing capacity information with neighboring nodes via asynchronous message passing; generating a local negotiation result based on the microtask status and processing capacity information through a weighted voting mechanism based on microtask relevance, wherein the weighted voting mechanism allocates weights according to semantic relevance and task priority to determine the allocation ratio of microtasks; and dynamically adjusting the allocation ratio of microtasks based on the local negotiation result.

[0012] According to one achievable method in the embodiments of this application, the aggregation of micro-task processing results of each computing node to generate global data processing results includes: each computing node generating intermediate result metadata of micro-task processing, merging it step by step among neighboring nodes through a tree aggregation protocol to generate global data processing results, wherein the tree aggregation protocol reduces global communication overhead by limiting the communication range.

[0013] According to the second aspect, a cloud computing-based big data processing system is provided. The system is built on a cloud computing platform and includes: a task decomposition module configured to decompose a big data processing task into multiple micro-tasks and allocate the micro-tasks to distributed computing nodes; a neighbor node selection module configured to select neighbor nodes for each computing node based on a hyperlocal algorithm, and to construct a local task processing model using only local data and metadata of neighbor nodes dynamically selected based on a fast game mechanism, wherein the metadata includes the semantic features and task dependencies of the micro-task data of neighbor nodes, the semantic features being extracted using natural language processing technology, and the task dependencies being determined based on the input-output association of the micro-tasks; a task execution module configured to independently execute micro-tasks on each computing node according to the local task processing model, and to dynamically adjust the allocation of micro-tasks through a point-to-point local negotiation mechanism; and a result aggregation module configured to aggregate the micro-task processing results of each computing node to generate a global data processing result.

[0014] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0015] According to a fourth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0017] This application refines task granularity and improves processing flexibility through micro-task decomposition and semantic segmentation; it dynamically selects neighboring nodes using a hyperlocal algorithm, relying only on local data and neighboring node metadata to construct a local task processing model, avoiding global coordination and significantly reducing communication overhead; it optimizes micro-task allocation through a fast game mechanism to improve task execution efficiency; and it further reduces global communication through point-to-point local negotiation and tree aggregation protocols, ensuring real-time performance and adaptability. Compared to traditional methods, this invention exhibits higher real-time performance and efficiency in edge computing and IoT scenarios, and is particularly suitable for dynamic, heterogeneous data processing environments.

[0018] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a system architecture diagram applicable to the embodiments of this application;

[0021] Figure 2 A flowchart illustrating a cloud computing-based big data processing method provided in an embodiment of this application;

[0022] Figure 3 A schematic diagram illustrating the process of a cloud computing-based big data processing method provided in this application embodiment;

[0023] Figure 4 A structural block diagram of a cloud computing-based big data processing system provided in an embodiment of this application;

[0024] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0027] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0028] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0029] Several technologies already exist that optimize the allocation of storage, computing, and training resources through local game theory within logical partitions and global collaborative game theory. These methods divide nodes into predefined logical partitions and rely on a global resource scheduling system for cross-partition coordination, with the game theory based on resource load and task priority. However, these methods have the following drawbacks: reliance on logical partitions and global coordination increases communication and computational overhead, making them unsuitable for dynamic, real-time scenarios; the game theory model focuses on resource scheduling rather than optimizing data processing tasks, making it difficult to adapt to heterogeneous data or complex task dependencies; and global coordination and historical data learning increase system complexity, limiting their application in scenarios such as edge computing.

[0030] In view of this, this application provides a new approach. To facilitate understanding of this application, the system architecture on which this application is based will first be described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture may include: user equipment and a cloud-based big data processing system located on the server side.

[0031] Users can input big data processing tasks through their user devices, which then send these tasks to a cloud-based big data processing system on the server side. The cloud-based big data processing system can use the methods provided in this embodiment to process the data and obtain global data processing results. The server can then send the global data processing results to the user terminal.

[0032] User devices can include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and PCs (Personal Computers). Smart mobile devices can include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and connected cars. Smart home devices can include smart TVs, smart refrigerators, and so on. Wearable devices can include smartwatches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices.

[0033] Cloud-based big data processing systems can be configured as standalone servers, server clusters, or cloud servers. Cloud servers, also known as cloud computing servers or cloud hosts, are a hosting product within the cloud computing service ecosystem, designed to address the management difficulties and weak service scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services. In addition to... Figure 1 In addition to the architecture shown, cloud computing-based big data processing systems can also be set up on computer terminals with strong computing capabilities.

[0034] It should be understood that Figure 1 The user equipment and cloud-based big data processing system shown are merely illustrative. Depending on implementation needs, any number of user equipment and cloud-based big data processing systems can be included.

[0035] Figure 2 This application provides a flowchart of a cloud computing-based big data processing method, which can be implemented by... Figure 1 The cloud-based big data processing system in the system shown is executing. For example... Figure 2 As shown, the method may include the following steps:

[0036] Step 201: Decompose the big data processing task into multiple micro-tasks and distribute them to distributed computing nodes.

[0037] Step 202: Select neighboring nodes for each computing node based on the hyperlocal algorithm, and construct a local task processing model using only local data and metadata of the neighboring nodes dynamically selected based on the fast game mechanism. The metadata includes the semantic features of the micro-task data of the neighboring nodes and the task dependencies. The semantic features are extracted by natural language processing technology, and the task dependencies are determined based on the input-output association of the micro-tasks.

[0038] Step 203: Based on the local task processing model, microtasks are executed independently on each computing node, and the allocation of microtasks is dynamically adjusted through a point-to-point local negotiation mechanism.

[0039] Step 204: Aggregate the microtask processing results of each computing node to generate a global data processing result, wherein each computing node shares only the necessary result metadata by minimizing global communication.

[0040] As can be seen from the above process, this application refines task granularity and improves processing flexibility through micro-task decomposition and semantic segmentation; it dynamically selects neighboring nodes using a hyperlocal algorithm, relying only on local data and neighboring node metadata to construct a local task processing model, avoiding global coordination and significantly reducing communication overhead; it optimizes micro-task allocation through a fast game mechanism to improve task execution efficiency; and it further reduces global communication through point-to-point local negotiation and tree aggregation protocols, ensuring real-time performance and adaptability. Compared with traditional methods, this invention exhibits higher real-time performance and efficiency in edge computing and IoT scenarios, and is particularly suitable for dynamic, heterogeneous data processing environments.

[0041] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments. First, step 201, namely "decomposing the big data processing task into multiple micro-tasks and distributing them to distributed computing nodes", will be described in detail with reference to the embodiments.

[0042] This invention relies on cloud computing platforms, a modern computing infrastructure, to fully utilize their powerful computing resources, storage capabilities, and network connectivity. As a unified resource pool, cloud computing platforms provide virtualization technology support, enabling dynamic allocation of computing resources to meet the demands of big data processing, thereby ensuring the method's efficiency and scalability when facing large-scale, complex data tasks.

[0043] Big data tasks involve massive amounts of data and complex computational processes. By breaking them down into multiple smaller, independent micro-tasks, the overall problem can be decomposed into sub-problems that can be processed in parallel. Figure 3 A schematic diagram illustrating the process of the cloud computing-based big data processing method provided in this application embodiment, as shown below. Figure 3 As shown, this application decomposes big data processing tasks into multiple micro-tasks. This decomposition process is usually based on the inherent logic of the task or the characteristics of the data, such as dividing by time series, data type, or processing requirements, so that each micro-task has specific functions and input / output requirements.

[0044] As an implementable approach, decomposing a big data processing task into multiple micro-tasks includes: decomposing the task into micro-tasks with a granularity smaller than a preset threshold according to the data type and processing requirements of the big data processing task, wherein the decomposition is based on task-based semantic segmentation.

[0045] Data types may include structured data (such as database records), unstructured data (such as text and images), or semi-structured data (such as JSON files), and processing requirements may include computational complexity, real-time requirements, or resource constraints. By combining these factors, the decomposition process can be tailored to different tasks, ensuring that the definition of each microtask matches the actual processing objective, thereby improving efficiency and reducing resource waste.

[0046] A preset threshold can be an upper limit for task processing time, data volume, or computational load. By limiting the size of microtasks, it ensures that each subtask is small enough to facilitate parallel processing by distributed computing nodes. This granular control not only reduces the execution difficulty of individual microtasks but also facilitates subsequent dynamic allocation and optimization operations, avoiding node overload caused by excessively large tasks. Fine-grained microtask settings also provide the system with greater flexibility, enabling it to adapt to different computing environments and varying task loads.

[0047] Semantic segmentation refers to dividing tasks based on the semantic information of the task content (such as topics in text or object categories in images), rather than simply relying on data volume or time segments. This method utilizes natural language processing or computer vision techniques to identify relevant parts of the task. For example, it can segment object detection tasks in video streams according to object categories (such as vehicles and pedestrians), or segment text analysis tasks according to semantic topics (such as sentiment analysis and entity recognition). Semantic-based segmentation not only improves the independence between micro-tasks but also provides a semantic foundation for subsequent dynamic neighbor selection and fast game-theoretic mechanisms, making the allocation and processing more aligned with the inherent logic and needs of the task.

[0048] Distributed computing nodes are computing units within a cloud computing platform. They can be cloud servers, edge servers, or virtual machines. These nodes are interconnected via a network to collaboratively process micro-tasks. The allocation process is typically implemented using a scheduling mechanism, dynamically distributing micro-tasks to suitable nodes based on factors such as computing power, load, or geographical location. This allocation strategy ensures full utilization of resources, while the distributed architecture reduces the risk of single points of failure and improves the overall processing efficiency of the system.

[0049] The following describes in detail step 202, namely, "selecting neighboring nodes for each computing node based on the hyperlocal algorithm, and constructing a local task processing model using only local data and metadata of the neighboring nodes dynamically selected based on the fast game mechanism, wherein the metadata includes the semantic features of the micro-task data of the neighboring nodes and the task dependency relationship, wherein the semantic features are extracted by natural language processing technology, and the task dependency relationship is determined based on the input-output association of the micro-tasks," with reference to the embodiments.

[0050] The core of hyperlocal algorithms lies in making decisions based solely on local information and limited data from neighboring nodes, without requiring global coordination. This approach adapts to task changes by dynamically adjusting neighboring nodes, ensuring a limited scope of cooperation, significantly reducing communication overhead and improving the system's real-time performance.

[0051] Each node utilizes its own stored local data, such as the status of processed tasks or resource information, combined with metadata obtained from neighboring nodes, to optimize decisions through a fast game mechanism. This fast game mechanism rapidly computes the optimal strategy through local interactions, ensuring efficient configuration of task allocation and collaboration relationships without global control. This approach differs from traditional globally optimized distributed systems and is particularly suitable for edge computing or low-bandwidth environments, reducing the complexity of data transmission.

[0052] As an implementable approach, each computing node selects neighboring nodes based on a hyperlocal algorithm, which includes: each computing node dynamically selects neighboring nodes by calculating the semantic similarity threshold of the microtask and the dependency chain length, wherein the selection mechanism is based on a local topology optimization algorithm to ensure that the number of neighboring nodes does not exceed a preset upper limit.

[0053] Specifically, the semantic similarity threshold is derived by analyzing the semantic features of the micro-task content. For example, when two micro-tasks involve the same topic or target category, their similarity is high, and only nodes exceeding the preset threshold are selected as neighboring nodes. The dependency chain length measures the input-output dependencies between micro-tasks, such as the length of dependency paths in a task sequence; shorter dependency chains indicate higher collaboration efficiency. Dynamic selection means that nodes continuously adjust their neighboring nodes based on real-time changes in tasks (such as data updates or load changes), ensuring that collaborating objects always match the current task requirements. Local topology optimization algorithms optimize data flow and communication efficiency by adjusting the collaboration structure between nodes. For example, the k-nearest neighbor algorithm is used to select the optimal neighbor based on semantic similarity and dependency chain length. This algorithm operates locally, considering only the state of the current node and its direct neighbors, avoiding complex calculations in the global network and thus reducing resource consumption. Ensuring that "the number of neighboring nodes does not exceed a preset upper limit" restricts the scope of collaboration; for example, setting the upper limit to 5 nodes effectively controls communication overhead and prevents network overload, enhancing system stability and scalability.

[0054] In a real-time video analytics system, node A detects vehicles, node B detects pedestrians, and node C processes the background scene. Node A, by calculating the semantic similarity of micro-tasks, discovers that B's pedestrian detection task is correlated with vehicle detection, and the dependency chain length is only 1 (B's output can directly support A's further analysis), while C's background processing is irrelevant to A. Based on a local topology optimization algorithm, A selects B as a neighboring node, but due to a preset upper limit of 2 neighbors, A will not select C simultaneously. During task updates, if B's ​​load is too high, A can dynamically switch to another node D with higher similarity, demonstrating the advantages of dynamic selection and optimization.

[0055] A local task processing model is constructed using local data and metadata of neighboring nodes dynamically selected based on a fast game mechanism. The metadata includes the semantic features of micro-task data of neighboring nodes and task dependencies. The semantic features of micro-task data reflect the semantic information of the task content, such as topics in text or object categories in images; these features provide a semantic basis for collaboration between nodes. Task dependencies describe the input-output relationships between micro-tasks; for example, the output of one detection task may be the input of another classification task. This dependency guides the collaboration order and data flow. The concise design of the metadata ensures efficient information exchange while supporting localized decision-making in hyperlocal algorithms.

[0056] Natural language processing techniques, such as word embeddings or semantic analysis, are used to extract semantic features from microtask data, for example, generating semantic vectors for text using the BERT model. Task dependencies are determined by analyzing the input-output relationship graph of the microtasks, such as path lengths or data flow directions in the dependency graph. This approach ensures the accuracy of semantic features and dependencies, providing reliable data support for hyperlocal algorithms, thereby optimizing the construction and execution efficiency of local task processing models.

[0057] As an implementable approach, this application utilizes local data and metadata of neighboring nodes dynamically selected based on a fast game mechanism to construct a local task processing model, including: each computing node analyzes local data and metadata through a fast game mechanism based on micro-task relevance to determine the execution priority of local micro-tasks, wherein the fast game mechanism optimizes task allocation among nodes without global coordination by calculating the semantic relevance and dependency weights between local micro-tasks; based on the execution priority and the neighboring nodes, an adaptive local task processing model is constructed, wherein each node dynamically adjusts its cooperative relationship with neighboring nodes according to the real-time changes of micro-tasks to optimize the local execution path of micro-tasks.

[0058] The fast game mechanism simulates competition and cooperation through local interactions, analyzes the relationships between micro-tasks, and determines which tasks should be prioritized. Semantic relevance measures the similarity of micro-task content; for example, two tasks may involve the same target category. Dependency weights reflect the input-output correlation between tasks; for example, the output of one task directly supports the input of another. Without global coordination, nodes compute the optimal strategy through a finite number of iterations, such as prioritizing tasks with high relevance or high dependency weights, thereby optimizing task allocation, reducing resource waste, and improving processing efficiency. Specifically, nodes first extract semantic features of micro-tasks using a pre-trained BERT model, generating 128-dimensional embedding vectors. Then, they calculate semantic relevance scores between micro-tasks using cosine similarity, ranging from 0 to 1. Dependency weights are calculated by analyzing the input-output dependency graphs of micro-tasks; for example, micro-tasks with dependency path lengths less than 3 are assigned higher weights. The game process employs a local Nash equilibrium algorithm, where nodes assign initial priority scores to each micro-task based on semantic relevance and dependency weights; for example, tasks with scores higher than 0.7 are marked as high priority. In practice, nodes adjust their local policies by exchanging priority scores with neighboring nodes, ensuring that task allocation is completed within 2 seconds without waiting for global synchronization.

[0059] Computing nodes assign tasks to themselves and neighboring nodes based on a defined priority, constructing a processing model that reflects the current collaborative relationships. This model can adjust according to real-time changes in micro-tasks; for example, the highest priority micro-task is processed locally by the node, while the second highest priority task is distributed to the neighboring node with the highest semantic relevance. The model is represented by a graph structure, with nodes as vertices, micro-task allocation relationships as edges, and edge weights reflecting the strength of collaboration. The process of dynamically adjusting collaborative relationships is achieved by monitoring the real-time status of micro-tasks; for example, checking the load of neighboring nodes every second. If the load exceeds 80%, the node will reselect the neighboring node with the second highest semantic similarity. Optimizing the local execution path of micro-tasks is achieved by adjusting the direction of data flow; for example, transmitting intermediate results directly to dependent nodes instead of through multi-hop routing to reduce transmission latency.

[0060] In a smart medical monitoring scenario, computing node A handles heart rate analysis, node B detects respiratory patterns, and node C processes body temperature data. A reads the heart rate data from the last 10 seconds locally, obtaining metadata from B and C, including semantic feature vectors for respiratory patterns and body temperature. A uses BERT to extract the semantic features of the heart rate data, calculating a semantic correlation of 0.85 with B (due to the correlation between heart rate and respiration) and a dependency weight of 2 (B's output supports further analysis by A), while the correlation with C is only 0.3. Through three iterations of game theory, A processes the heart rate analysis task locally and assigns the respiratory pattern detection task to B. After one minute, B's load rises to 90%, and A dynamically adjusts, switching to C to process some data. The optimized path directly transmits B's output to A, reducing data transfer time.

[0061] Furthermore, the fast game mechanism employs an iterative algorithm based on local Nash equilibrium, in which each computing node calculates the optimal task allocation strategy based on semantic relevance and dependency weights within a finite number of iterations, and the number of iterations is limited by a preset threshold.

[0062] Local Nash equilibrium refers to a situation where, without global coordination, each computing node finds a stable combination of strategies based on its own state and that of its neighbors, such that unilateral adjustments by any node cannot further improve the outcome. Iterative algorithms gradually approach this equilibrium state through multiple local adjustments, ensuring that task allocation achieves local optimum in resource-constrained distributed environments.

[0063] Semantic relevance is derived by analyzing the similarity of microtask content. For example, two tasks involving the same data topic are highly relevant, specifically calculated using cosine similarity to score the embedding vectors. Dependency weights reflect the input-output relationship between microtasks; for example, the weight increases when the output of one task directly supports another. In each iteration, nodes adjust task allocation based on these metrics, prioritizing tasks with high relevance or high dependency weights for themselves or neighboring nodes. The finite number of iterations ensures computational efficiency and avoids excessively long convergence times, making it particularly suitable for scenarios with high real-time requirements. Preset thresholds are typically set based on system performance and task urgency, such as 3 or 5 iterations; specific values ​​can be determined experimentally during deployment.

[0064] The following describes step 203, namely, "according to the local task processing model, micro-tasks are executed independently on each computing node, and the allocation of micro-tasks is dynamically adjusted through a point-to-point local negotiation mechanism," in detail with reference to the embodiments.

[0065] Each computing node autonomously completes the microtasks assigned to it based on the previously constructed local task processing model. The local task processing model includes the execution priority and cooperation relationship of the tasks. The node uses local computing resources and data to process the microtasks, such as feature extraction or data classification, according to the path defined by the model.

[0066] The peer-to-peer negotiation mechanism is implemented through direct communication between computing nodes. Nodes exchange microtask status information, such as completion progress or current load, and processing capacity information, such as available CPU or memory resources, via asynchronous message passing. Dynamic adjustment means that nodes reallocate tasks based on real-time data, for example, transferring overloaded tasks to idle nodes. This mechanism eliminates the need for global coordination, reducing communication overhead and improving resource utilization. The negotiation process is typically based on predefined rules, such as prioritizing assignment to semantically relevant neighboring nodes to ensure the continuity of task processing.

[0067] Preferably, dynamically adjusting the allocation of microtasks through a point-to-point local negotiation mechanism includes: each computing node exchanging microtask status and processing capacity information with neighboring nodes through asynchronous message passing; based on the microtask status and processing capacity information, generating a local negotiation result through a weighted voting mechanism based on microtask relevance, wherein the weighted voting mechanism allocates weights according to semantic relevance and task priority to determine the allocation ratio of microtasks; and dynamically adjusting the allocation ratio of microtasks based on the local negotiation result.

[0068] Asynchronous messaging allows nodes to send and receive information at different times. For example, a node can immediately notify neighboring nodes of its status after completing a partial task, without waiting for a synchronization signal. The weighted voting mechanism is jointly executed by participating nodes, each proposing task allocation suggestions based on its own and neighboring nodes' states. Weight allocation is determined based on semantic relevance and task priority. For instance, two microtasks involving the same data topic have higher semantic relevance and thus higher weights, while high-priority tasks such as real-time monitoring tasks also receive higher weights. The voting results generate a local negotiation result by calculating a weighted average, determining the microtask allocation ratio for each node; for example, a node might handle 60% of the workload. This method ensures that the allocation scheme reflects the inherent relevance and urgency of tasks while considering the actual capabilities of nodes. Nodes immediately adjust their task load based on the negotiation results, for example, distributing overloaded tasks to neighboring nodes with stronger processing capabilities. The adjustment process is continuous; nodes renegotiate at fixed time intervals (e.g., every 10 seconds) to ensure the allocation ratio is consistent with the current system state. If task requirements change or node performance degrades, the allocation ratio is updated in real time to maintain system balance. Dynamic adjustment reduces the need for manual intervention and improves the adaptability of the distributed system.

[0069] The following describes step 203, namely "aggregating the micro-task processing results of each computing node to generate global data processing results," in detail with reference to the embodiments.

[0070] Each computing node, after independently executing a microtask, aggregates its processing results. These results typically exist in the form of intermediate outputs, such as statistical data, feature vectors, or detection results. Nodes share these results across the network to support the overall task objective. The aggregation process is designed to ensure a balance between the decentralization of distributed computing and the integrity of the final output, avoiding the high communication burden of traditional centralized systems while adapting to the dynamic nature of cloud computing environments. The global data processing result can be a comprehensive analysis report, a complete data model, or a final prediction, depending on the task type, such as a summary image for object detection or a total topic distribution for text analysis.

[0071] As an implementable approach, aggregating the micro-task processing results of each computing node to generate global data processing results includes: each computing node generating intermediate result metadata of micro-task processing, which is then gradually merged among neighboring nodes through a tree aggregation protocol to generate global data processing results, wherein the tree aggregation protocol reduces global communication overhead by limiting the communication range.

[0072] Specifically, after completing a microtask, each node does not directly transmit the complete data, but instead generates concise metadata. This metadata includes key information about the task processing, such as statistical values, feature summaries, or dependency identifiers. This concise form reduces data transmission while retaining enough information to support subsequent merging. The generation of metadata typically depends on the task type; for example, in image processing, it might be the bounding box coordinates for object detection, while in data analysis, it might be the mean or variance.

[0073] The tree-structured aggregation protocol employs a hierarchical merging strategy. Nodes are organized into a tree structure based on proximity, with data converging from leaf nodes upwards to the root node. The merging process is incremental; for example, neighboring nodes first combine their respective metadata into sub-results, then pass them to higher-level nodes until the final global result is generated. This method leverages the local connectivity of nodes, avoiding direct communication between all nodes and the central node, thus reducing the likelihood of network congestion. Merging operations may include weighted averaging, maximum value selection, or logical combination, depending on task requirements, ensuring the accuracy of the results.

[0074] The limitation of communication range means that nodes only exchange data with their immediate neighbors, rather than interacting with all nodes in the entire network; for example, they are limited to parent or child nodes in a tree structure. This design significantly reduces the number of hops and bandwidth requirements for data transmission, especially in scenarios with a large number of nodes or high network latency. Compared to traditional network-wide broadcasting, tree aggregation protocols can reduce communication overhead by approximately 30% to 50% while maintaining the integrity of the aggregation results.

[0075] The methods provided in this application can be applied to various scenarios, including but not limited to: First, in intelligent traffic management, this method can be used to analyze vehicle flow and traffic signal status in various urban areas in real time. Distributed computing nodes process micro-tasks and dynamically collaborate to generate a global traffic optimization plan, ensuring smooth traffic during peak hours. Second, in the field of intelligent medical monitoring, the system can be deployed in hospitals or home devices to analyze heart rate, respiratory patterns, and body temperature data in real time. Each node quickly generates patient health reports through point-to-point negotiation and tree-structured aggregation, supporting the timeliness of remote diagnosis and treatment. Finally, in environmental monitoring under the Internet of Things (IoT) environment, distributed nodes can monitor air quality, water quality, or noise levels. Local data is processed through semantic segmentation and fast game theory mechanisms, ultimately aggregating a global environmental index to provide data support for urban planning and emergency response. These scenarios fully utilize the method's low communication overhead and high adaptability, making it particularly suitable for edge computing and real-time applications. It can effectively improve resource utilization and processing efficiency, meeting the diverse needs of modern big data processing.

[0076] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0077] According to another embodiment, a cloud computing-based big data processing system is provided. Figure 4 A schematic block diagram of a cloud-based big data processing system according to one embodiment is shown. Figure 4 As shown, the system 400 includes:

[0078] The task decomposition module 401 is configured to decompose a big data processing task into multiple micro-tasks and allocate the micro-tasks to distributed computing nodes.

[0079] The neighbor node selection module 402 is configured to select neighbor nodes for each computing node based on a hyperlocal algorithm, and to construct a local task processing model using only local data and metadata of neighbor nodes dynamically selected based on a fast game mechanism. The metadata includes the semantic features of micro-task data of neighbor nodes and task dependencies. The semantic features are extracted using natural language processing technology, and the task dependencies are determined based on the input-output association of micro-tasks.

[0080] The task execution module 403 is configured to independently execute microtasks on each computing node according to the local task processing model, and dynamically adjust the allocation of microtasks through a point-to-point local negotiation mechanism.

[0081] The result aggregation module 404 is configured to aggregate the micro-task processing results of each computing node to generate global data processing results.

[0082] As an implementable approach, the task decomposition module 401 can be configured to decompose a big data processing task into multiple micro-tasks by: decomposing the task into micro-tasks with a granularity smaller than a preset threshold according to the data type and processing requirements of the big data processing task, wherein the decomposition is based on task-based semantic segmentation.

[0083] As an implementable approach, the neighbor node selection module 402 can be configured to select neighbor nodes for each computing node based on the hyperlocal algorithm as follows: each computing node dynamically selects the neighbor node by calculating the semantic similarity threshold of the microtask and the dependency chain length, wherein the selection mechanism is based on the local topology optimization algorithm to ensure that the number of neighbor nodes does not exceed a preset upper limit.

[0084] As an implementable approach, the neighbor node selection module 402, when constructing a local task processing model using only local data and metadata of neighbor nodes dynamically selected based on a fast game mechanism, can be configured as follows: each computing node analyzes local data and the metadata through a fast game mechanism based on micro-task relevance to determine the execution priority of local micro-tasks, wherein the fast game mechanism optimizes task allocation between nodes without global coordination by calculating the semantic relevance and dependency weights between local micro-tasks; based on the execution priority and the neighbor nodes, an adaptive local task processing model is constructed, wherein each node dynamically adjusts its cooperation relationship with neighbor nodes according to the real-time changes of micro-tasks to optimize the local execution path of micro-tasks.

[0085] As an feasible approach, the fast game mechanism employs an iterative algorithm based on local Nash equilibrium, in which each computing node calculates the optimal task allocation strategy based on semantic relevance and dependency weights within a finite number of iterations, and the number of iterations is limited by a preset threshold.

[0086] As an implementable approach, the task execution module 403 can be configured to dynamically adjust the allocation of microtasks through a point-to-point local negotiation mechanism as follows: each computing node exchanges microtask status and processing capacity information with its neighboring nodes via asynchronous message passing; based on the microtask status and processing capacity information, a local negotiation result is generated through a weighted voting mechanism based on microtask relevance, wherein the weighted voting mechanism allocates weights according to semantic relevance and task priority to determine the allocation ratio of microtasks; and the allocation ratio of microtasks is dynamically adjusted based on the local negotiation result.

[0087] As an implementable approach, the result aggregation module 404 can be configured to generate global data processing results by aggregating the micro-task processing results of each computing node. Each computing node generates intermediate result metadata of the micro-task processing, which is then gradually merged among neighboring nodes through a tree aggregation protocol to generate global data processing results. The tree aggregation protocol reduces global communication overhead by limiting the communication range.

[0088] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. Components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0089] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0090] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0091] And an electronic device comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0092] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0093] in, Figure 5 An exemplary architecture of an electronic device is shown, which may include a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, and a memory 520. The processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520 can communicate with each other via a communication bus 530.

[0094] The processor 510 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0095] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 520 can store the operating system 521 for controlling the operation of the electronic device 500, and the basic input / output system (BIOS) 522 for controlling the low-level operations of the electronic device 500. Additionally, it can store a web browser 523, a data storage management system 524, and a cloud computing-based big data processing system 525, etc. The aforementioned cloud computing-based big data processing system 525 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 520 and executed by the processor 510.

[0096] Input / output interface 513 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0097] Network interface 514 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0098] Bus 530 includes a pathway for transmitting information between various components of the device, such as processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520.

[0099] It should be noted that although the above-described device only shows the processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, memory 520, bus 530, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0100] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0101] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A big data processing method based on cloud computing, characterized in that, The method is implemented based on a cloud computing platform, and the method includes: Big data processing tasks are broken down into multiple micro-tasks and distributed to distributed computing nodes; Based on the hyperlocal algorithm, neighboring nodes are selected for each computing node. A local task processing model is constructed using only local data and metadata of the neighboring nodes dynamically selected based on a fast game mechanism. The metadata includes the semantic features of the micro-task data of the neighboring nodes and the task dependencies. The semantic features are extracted using natural language processing techniques, and the task dependencies are determined based on the input-output association of the micro-tasks. According to the local task processing model, micro-tasks are executed independently on each computing node, and the allocation of micro-tasks is dynamically adjusted through a point-to-point local negotiation mechanism. The microtask processing results of each computing node are aggregated to generate a global data processing result; The selection of neighboring nodes for each computing node based on the hyperlocal algorithm includes: Each computing node dynamically selects neighboring nodes by calculating the semantic similarity threshold and dependency chain length of the microtasks. The selection mechanism is based on a local topology optimization algorithm to ensure that the number of neighboring nodes does not exceed a preset upper limit. The method of constructing a local task processing model by utilizing only local data and metadata of neighboring nodes dynamically selected based on a fast game mechanism includes: Each computing node analyzes local data and metadata through a fast game mechanism based on micro-task correlation to determine the execution priority of local micro-tasks. The fast game mechanism optimizes task allocation between nodes without global coordination by calculating the semantic correlation and dependency weight between local micro-tasks. Based on the execution priority and the neighboring nodes, an adaptive local task processing model is constructed, in which each node dynamically adjusts its cooperation relationship with neighboring nodes according to the real-time changes of the micro-task, and optimizes the local execution path of the micro-task. The fast game mechanism adopts an iterative algorithm based on local Nash equilibrium, in which each computing node calculates the optimal task allocation strategy based on semantic relevance and dependency weight within a finite number of iterations, and the number of iterations is limited by a preset threshold. The dynamic adjustment of micro-task allocation through a point-to-point local negotiation mechanism includes: Each computing node exchanges microtask status and processing capacity information with its neighboring nodes via asynchronous message passing. Based on the microtask status and the processing capacity information, a local negotiation result is generated through a weighted voting mechanism based on microtask relevance, wherein the weighted voting mechanism allocates weights according to semantic relevance and task priority to determine the allocation ratio of microtasks; The allocation ratio of microtasks is dynamically adjusted based on the results of the local negotiation.

2. The method according to claim 1, characterized in that, The process of breaking down big data processing tasks into multiple micro-tasks includes: Based on the data type and processing requirements of the big data processing task, the task is decomposed into micro-tasks with a granularity smaller than a preset threshold, wherein the decomposition is task-based semantic segmentation.

3. The method according to claim 1, characterized in that, The aggregation of micro-task processing results from each computing node to generate global data processing results includes: Each computing node generates intermediate result metadata for microtask processing, which is then gradually merged among neighboring nodes through a tree aggregation protocol to generate global data processing results. The tree aggregation protocol reduces global communication overhead by limiting the communication range.

4. A big data processing system based on cloud computing, characterized in that, The system is built on a cloud computing platform and includes: The task decomposition module is configured to decompose big data processing tasks into multiple micro-tasks and allocate the micro-tasks to distributed computing nodes; The neighbor node selection module is configured to select neighbor nodes for each computing node based on the hyperlocal algorithm, and to construct a local task processing model using only local data and metadata of neighbor nodes dynamically selected based on a fast game mechanism. The metadata includes the semantic features of micro-task data of neighbor nodes and task dependencies. The semantic features are extracted using natural language processing technology, and the task dependencies are determined based on the input-output association of micro-tasks. The task execution module is configured to independently execute microtasks on each computing node according to the local task processing model, and dynamically adjust the allocation of microtasks through a point-to-point local negotiation mechanism. The results aggregation module is configured to aggregate the micro-task processing results of each computing node and generate global data processing results. The selection of neighboring nodes for each computing node based on the hyperlocal algorithm includes: Each computing node dynamically selects neighboring nodes by calculating the semantic similarity threshold and dependency chain length of the microtasks. The selection mechanism is based on a local topology optimization algorithm to ensure that the number of neighboring nodes does not exceed a preset upper limit. The method of constructing a local task processing model by utilizing only local data and metadata of neighboring nodes dynamically selected based on a fast game mechanism includes: Each computing node analyzes local data and metadata through a fast game mechanism based on micro-task correlation to determine the execution priority of local micro-tasks. The fast game mechanism optimizes task allocation between nodes without global coordination by calculating the semantic correlation and dependency weight between local micro-tasks. Based on the execution priority and the neighboring nodes, an adaptive local task processing model is constructed, in which each node dynamically adjusts its cooperation relationship with neighboring nodes according to the real-time changes of the micro-task, and optimizes the local execution path of the micro-task. The fast game mechanism adopts an iterative algorithm based on local Nash equilibrium, in which each computing node calculates the optimal task allocation strategy based on semantic relevance and dependency weight within a finite number of iterations, and the number of iterations is limited by a preset threshold. The dynamic adjustment of micro-task allocation through a point-to-point local negotiation mechanism includes: Each computing node exchanges microtask status and processing capacity information with its neighboring nodes via asynchronous message passing. Based on the microtask status and the processing capacity information, a local negotiation result is generated through a weighted voting mechanism based on microtask relevance, wherein the weighted voting mechanism allocates weights according to semantic relevance and task priority to determine the allocation ratio of microtasks; The allocation ratio of microtasks is dynamically adjusted based on the results of the local negotiation.

5. An electronic device, characterized in that, include: One or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 3.

Images