Cloud data processing system based on artificial intelligence algorithm

Abstract

The invention relates to the field of cloud computing and distributed computing, and discloses a cloud data processing system based on an artificial intelligence algorithm, which comprises a data input module, a task decomposition module, a resource scheduling module, a parallel computing module and a result reconstruction module, the data input module receives high-dimensional data and converts the high-dimensional data into a tensor structure; the task decomposition module decomposes the tensor, generates a core tensor and an orthogonal matrix, and generates a plurality of sub-tasks; the resource scheduling module is used for carrying out resource allocation and scheduling on the sub-tasks, and task execution time and resource consumption are minimized; the parallel computing module distributes subtasks to the computing nodes for execution, and the task execution sequence is guaranteed; and the result reconstruction module combines and verifies the execution results of the subtasks to generate a final calculation result. The cloud data processing efficiency and the resource utilization rate are effectively improved, the accuracy and consistency of calculation results are ensured, and the method is suitable for the fields of artificial intelligence training, large-scale data analysis, scientific calculation and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing and distributed computing, and in particular to a cloud data processing system based on an artificial intelligence algorithm. Background Art

[0002] With the rapid development of cloud computing and distributed computing technologies and artificial intelligence algorithms, large-scale high-dimensional data processing has gradually become a key demand in various industries. In the fields of artificial intelligence model training, big data analysis, and scientific simulation, the rapid growth of data scale and computational complexity has put forward higher requirements for resource allocation, task scheduling, and result processing of cloud computing platforms. Generally, traditional cloud data processing systems adopt fixed resource allocation or centralized task scheduling strategies, which are difficult to cope with complex data structures and dynamic computing environments. This processing mode has significant deficiencies in task parallelism, resource utilization efficiency, and result accuracy. Especially when processing high-dimensional data, it is easy to cause waste of computing resources, task dependency bottlenecks, and a significant increase in processing time.

[0003] The existing technology lacks effective processing of complex dependencies between tasks in task decomposition and resource scheduling, resulting in inadequate optimization of task scheduling and execution order. In addition, data processing tasks usually allocate computing resources in a static manner, ignoring the real-time changes in resource status in a dynamic computing environment, making it difficult to achieve global optimization in resource utilization and execution efficiency. At the same time, the existing technology lacks an effective mechanism for merging and verifying parallel computing results, which can easily affect the reliability and accuracy of the final results due to precision loss or data inconsistency during the calculation process. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a cloud data processing system based on artificial intelligence algorithm, which solves the technical problems in the prior art of low efficiency in high-dimensional data processing, insufficient resource utilization, and difficulty in ensuring the accuracy and consistency of calculation results.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A cloud data processing system based on artificial intelligence algorithm, comprising: A data input module, used for receiving and loading high-dimensional data, and converting the data into a tensor structure; A task decomposition module is used to decompose the high-dimensional data, decompose the high-dimensional tensor into a core tensor and multiple orthogonal matrices based on high-order singular value decomposition, and generate multiple subtasks; A resource scheduling module, used to allocate and schedule resources for the subtasks based on a distributed optimization method to minimize task execution time and resource consumption; A parallel computing module, used for performing parallel computing on the subtasks in the time domain and the space domain, including time parallel pipeline execution and space parallel task distribution; The result reconstruction module is used to merge and reconstruct the subtask results obtained by the parallel computing module to generate the final computing result.

[0006] Preferably, the data input module preprocesses the input data, including data cleaning, formatting and tensor conversion.

[0007] Preferably, the high-order singular value decomposition decomposes the high-dimensional tensor into a core tensor and a plurality of orthogonal matrices, including: Perform matrix expansion on each dimension of the tensor to convert the high-dimensional tensor into matrix form; Decomposing the matrix by singular value decomposition to obtain an orthogonal matrix of corresponding dimension; According to the orthogonal matrix, a core tensor decomposition is performed on the high-dimensional tensor.

[0008] Preferably, the resource scheduling module implements resource allocation through a distributed Lagrangian optimization method, which specifically includes: Construct a resource scheduling objective function to simultaneously optimize task execution time, resource consumption, and data transmission overhead. The objective function is:

[0009] in, is the number of tasks, For the task The execution time, For the task resource consumption, For the task and tasks The data transmission delay between are the weight factors for time, resources, and transmission, respectively; Introduce resource constraints, the constraints are:

[0010] in, For the task The amount of resources allocated, is the total system resources; Construct the Lagrangian function and combine the resource scheduling objective function with the constraints to obtain:

[0011] in, is the Lagrange multiplier; The Lagrangian function is solved based on the distributed iterative optimization method, and the global optimal solution for resource allocation is achieved by updating the Lagrangian multiplier. The iterative update formula is:

[0012] in, is the number of iterations, is the step size parameter; A task allocation scheme is determined according to the global optimal solution, and tasks are allocated to computing nodes.

[0013] Preferably, the resource scheduling module performs task scheduling based on the Lagrangian function optimization result, specifically including: The objective function is decomposed into multiple sub-problems through a distributed solver, and the optimization goal of each sub-problem is:

[0014] in, For the task resource constraints; Solve the subproblems in parallel on each computing node and determine the resource allocation for each task ; Based on task execution time , Resource consumption and data transmission overhead The task allocation scheme is dynamically adjusted based on the trade-off results; According to the task allocation scheme, subtasks are allocated to corresponding computing nodes to minimize the overall execution time and resource consumption of the system.

[0015] Preferably, the parallel computing module includes: The time parallel unit is used to assign subtasks to time windows to form a task pipeline; A spatial parallel unit, used to distribute the subtasks to different computing nodes for parallel execution; The task dependency analysis unit is used to analyze the dependency of tasks based on a directed acyclic graph to ensure execution order and data synchronization.

[0016] Preferably, the task dependency analysis unit is used to model and schedule the dependency relationship of tasks based on a directed acyclic graph, specifically including: Assemble the tasks Represented as a directed acyclic graph ,in is the set of task vertices, is the data dependency edge between tasks, and the weight of the edge represents the data transmission overhead between tasks. ; The directed acyclic graph is topologically sorted to determine the order of task execution, and the topological sorting is performed according to the following conditions: like , i.e., the task Depends on the task As a result, the task Must precede the task implement; Calculate the execution priority of the task, which is determined by the node weight and path length. The node weight is the task execution time. , the path length is the cumulative weight from the starting task to the current task; Based on the execution priority, generating a task scheduling order; During the scheduling process, ensure that tasks are Executes all dependent tasks After completion, the execution conditions are: For any task ,like , then the task Start time Must be later than the task Completion time ; Dynamically adjust the task scheduling order. When the computing resource status or task execution time changes, re-optimize the scheduling order based on the task dependency graph.

[0017] Preferably, the subtask results obtained by the parallel computing module are merged and reconstructed to generate the final computing result, including: Receive all subtask results output by the parallel computing module ,in Indicates The calculation results of each subtask; Based on the block information of the task decomposition module, according to the logical order and dependency relationship of the subtasks, Combined into an overall result; Perform matrix concatenation or tensor reconstruction on the subtask results. If the subtask is a matrix result, concatenate according to the following rules:

[0018] If the subtask is a tensor result, the original tensor is reconstructed based on the multi-dimensional index of the tensor ; Output the final calculation results and feeds the results back to the system or external call interface.

[0019] Preferably, the subtask results obtained by the parallel computing module are merged and reconstructed to generate the final computing result, and the step of: Perform result verification to verify the integrity and consistency of the merged results, and use the following formula to determine whether the calculation accuracy requirements are met:

[0020] in, is the original tensor, is the reconstructed tensor, represents the Frobenius norm, is the accuracy threshold.

[0021] The present invention also provides a cloud data processing method based on an artificial intelligence algorithm, comprising the following steps: Receiving high-dimensional data, and converting the data into a tensor structure; Performing high-order singular value decomposition on the tensor to decompose it into a core tensor and an orthogonal matrix, and generating multiple subtasks; Based on the distributed Lagrangian optimization method, resources are allocated and scheduled for the subtasks to minimize task execution time and resource consumption; Based on the time parallelism and space parallelism mechanism, the subtasks are assigned to different time windows and computing nodes for execution, and the task order is ensured through task dependency analysis; The subtask results obtained by execution are merged and reconstructed to output the final calculation results.

[0022] The present invention provides a cloud data processing system based on artificial intelligence algorithm. It has the following beneficial effects: The present invention performs high-order singular value decomposition on high-dimensional data through a task decomposition module, decomposes complex computing tasks into multiple independent and parallel subtasks, and combines time parallelism and space parallelism mechanisms to effectively reduce the idle time of computing resources and dependency conflicts between tasks, thereby significantly improving the overall efficiency of data processing. It is particularly suitable for scenarios where large-scale high-dimensional data is quickly processed.

[0023] The present invention adopts a distributed Lagrangian optimization method to dynamically allocate and schedule subtasks. By optimizing task execution time, resource consumption, and data transmission delay between tasks at the same time, efficient use of computing resources is achieved, resource waste or overload problems are avoided, performance differences of different computing nodes are adapted, and global optimal allocation of computing resources is ensured.

[0024] The present invention combines and verifies the parallel computing results through the result reconstruction module, and can verify the integrity and accuracy of the final result. Combined with dynamic data synchronization and abnormal result correction mechanism, the accuracy and consistency of the final calculation result are ensured. Even if abnormal situations occur during task scheduling or distributed computing, the reliability of the final output can still be guaranteed through the compensation mechanism.

[0025] The present invention adopts modular design, and each functional module can be independently adjusted and expanded according to application requirements. The task decomposition module supports a variety of input data types, including structured data, unstructured data, etc.; the resource scheduling module can dynamically adapt to different computing environments; the parallel computing module and the result reconstruction module have good portability and scalability, and can be widely used in artificial intelligence model training, big data analysis, scientific simulation and other scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the system architecture of the present invention; Figure 2 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Please see attached Figure 1 The present invention provides a cloud data processing system based on artificial intelligence algorithm, which can efficiently process complex high-dimensional data by receiving, decomposing, allocating resources, parallel computing and reconstructing results of high-dimensional data. The system of the present invention is described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the cloud data processing system based on artificial intelligence algorithm may include a data input module, a task decomposition module, a resource scheduling module, a parallel computing module and a result reconstruction module.

[0030] In this embodiment, the specific implementation of the data input module is as follows: The data input module is used to receive high-dimensional data from external input, preprocess the data, format and convert the data, and generate a tensor structure suitable for subsequent processing.

[0031] In a possible implementation, the data input module first receives high-dimensional data through a network interface. The data may come from a local storage system, a distributed data source, or real-time streaming data. The data format may include but is not limited to structured data (such as tables or CSV files), semi-structured data (such as JSON, XML), and unstructured data (such as images, videos, and text). In practical applications, different data sources may have different data formats and distribution characteristics.

[0032] Specifically, for structured data, the data input module directly extracts the row and column data and represents it in the form of a multidimensional array. For example, the input tabular data can be converted into a two-dimensional matrix, where the rows represent samples and the columns represent features.

[0033] For semi-structured data, the data input module extracts valid information from the data through a parser. For example, the multi-layer nested structure in the JSON format will be unfolded and converted into a matrix or high-dimensional tensor according to predefined mapping rules. For example, the multi-layer fields in JSON can be mapped to different dimensions of a tensor, where the value of each field can be used as the element value of the tensor in the corresponding dimension.

[0034] For unstructured data (such as images or videos), the data input module first decodes and extracts features. For example, for an input image, the data input module can use the pixel values ​​to convert it into a three-dimensional tensor, where the height, width, and number of channels correspond to the three dimensions of the tensor. For video data, the time dimension can be further introduced into the tensor representation.

[0035] As an option, this module can also support data normalization. Specifically, for numerical data, the data input module will normalize the input data. The purpose of normalization is to unify the data value range of different features to [0, 1] or other fixed ranges, so as to avoid the influence of feature value range differences on subsequent calculations.

[0036] In an exemplary implementation, the data input module can fill missing values ​​in the input data. For missing values ​​of numerical data, the mean, mode or interpolation algorithm can be used for filling. For non-numerical data, the most frequently occurring category can be selected as the filling value based on the category distribution law. It should be noted that this missing value filling method can dynamically adjust the strategy according to the specific application scenario.

[0037] After completing data preprocessing, this module converts the data into tensor form to facilitate the processing of subsequent task decomposition modules. Specifically, the input data is converted into a high-dimensional tensor ,in Respectively represent the size of each dimension of the tensor. As a specific implementation method, matrix data is directly input as a two-dimensional tensor, text data can be converted into a two-dimensional or three-dimensional tensor through embedding methods, and image data is directly converted into a three-dimensional or four-dimensional tensor (with batch dimensions added).

[0038] It should be noted that in practical applications, the generation of high-dimensional tensors also involves specific data mapping rules and logic. For example, for a set of input data containing time series data, it can be tensorized into a three-dimensional form, where the time series value is the first dimension, the feature variable is the second dimension, and the sample number is the third dimension.

[0039] It is understandable that the data input module of the present invention also supports the fusion of multi-source data. For example, if image data and corresponding metadata (such as labels) are received at the same time, the data input module can fuse the image tensor and the metadata tensor into a high-dimensional tensor by data splicing. In this implementation, the tensorization of the image data and the embedded representation of the metadata will be completed synchronously, and a unified high-dimensional tensor structure will be generated in the end.

[0040] In order to further expand the scope of application, the data input module of the present invention can also support real-time streaming data processing. In this implementation, the data input module slices the streaming data into blocks of fixed length by means of sliding windows or time segments, and uses each time block as a tensor processing unit. For example, the video stream input can be divided into several video segments with fixed time intervals, and each segment is converted into a four-dimensional tensor.

[0041] It should be added that this module not only supports the input processing of static data, but also can optimize the reception and tensorization efficiency of real-time data through a specific cache mechanism in combination with the characteristics of dynamic data sources. For example, a ring buffer is used to save streaming input data for subsequent module calls.

[0042] Through the above design, the data input module of this embodiment can efficiently receive, clean, convert and tensorize high-dimensional data from multiple sources and multiple formats, provide a unified input interface for subsequent modules, and ensure efficient flow and processing of data in the entire system.

[0043] In this embodiment, the specific implementation of the task decomposition module is as follows: The task decomposition module is used to decompose the input high-dimensional tensor data. It is mainly based on the high-order singular value decomposition (HOSVD) method to decompose the high-dimensional tensor into core tensors and orthogonal matrices, and generate multiple subtasks to facilitate parallel computing and resource scheduling in subsequent modules.

[0044] As an option, the data received by this module is a tensor structure ,in Represent different dimensions of the tensor. The goal of the module is to Dimensionality reduction and block division are performed to extract its main features while reducing computational complexity.

[0045] In one possible implementation, the task decomposition module performs high-order singular value decomposition based on the following steps: Specifically, first, the input tensor Perform an expansion operation on each dimension to obtain the expansion matrix The expansion operation is implemented by expanding the tensor along the n-th dimension into a two-dimensional matrix, as follows:

[0046] in, is a tensor The matrix expanded along the n-th dimension, Represents the product of all dimensions except the n-th dimension.

[0047] It should be noted that this expansion operation converts the high-dimensional tensor into a two-dimensional matrix in each dimension, providing a basis for the subsequent singular value decomposition (SVD).

[0048] Next, for each expanded matrix Perform singular value decomposition (SVD) to extract features of the corresponding dimension. The process of singular value decomposition is as follows:

[0049] in, is the n-th dimension orthogonal matrix, representing the eigenvector of the corresponding dimension, is the singular value matrix, is the right singular vector matrix.

[0050] By performing singular value decomposition on the unfolded matrix of all dimensions, we can obtain the tensor The set of feature matrices in each dimension .

[0051] After completing the orthogonal matrix extraction, the task decomposition module further uses these orthogonal matrices to calculate the core tensor of the original tensor. The formula is as follows:

[0052] in, is the core tensor, which represents the low-rank compressed form of tensor data. Represents the multiplication of a tensor and a matrix in the n-th dimension.

[0053] As an exemplary implementation, the core tensor The core tensor can be further divided into multiple sub-tensors according to certain dimensions. , each sub-tensor corresponds to an independent sub-task. It should be noted that this block division method can be flexibly adjusted according to the complexity of the task, the number of computing nodes and the characteristics of data distribution.

[0054] It can be understood that the core tensors and orthogonal matrices generated by the task decomposition module are not only used for subsequent parallel computing, but can also be used as a compressed representation of the input data, significantly reducing the space occupied by the data during storage and transmission.

[0055] In one possible extension, the task decomposition module also supports rank-constrained decomposition of tensors. In some applications, in order to further simplify the calculation, the rank of the core tensor can be restricted. , the low-rank tensor decomposition is directly solved through optimization methods, reducing the subsequent computational complexity.

[0056] It should be added that this module also supports dynamic task generation. When the dimension or distribution characteristics of the input tensor data change, the task decomposition module can adjust the block strategy in real time, such as dynamically adjusting the number of blocks of the core tensor. Or the size of each sub-tensor to adapt to dynamic task requirements and computing resource allocation.

[0057] For example, in an image processing scenario, the input three-dimensional tensor (height, width, channel) can be decomposed into multiple smaller tensor blocks by the above method, each of which contains local feature information and is distributed to independent computing nodes for processing. This implementation method can effectively utilize parallel computing resources and improve overall computing efficiency.

[0058] Through the above design, the task decomposition module in this embodiment can efficiently complete the dimensionality reduction, decomposition and task generation of high-dimensional tensors, providing sufficient technical support for subsequent resource scheduling and parallel computing.

[0059] In this embodiment, the specific implementation of the resource scheduling module is as follows: The resource scheduling module is used to allocate and schedule resources for multiple subtasks generated by the subtask decomposition module. Through distributed optimization methods and dynamic scheduling strategies, it can achieve efficient utilization of computing resources and minimize task execution time. The core goal of this module is to solve the problem of resource competition and load balancing among multiple tasks, while optimizing task execution efficiency and data transmission overhead.

[0060] In a possible implementation, the resource scheduling module uses a distributed Lagrangian optimization method to construct a resource allocation model. First, the subtask set in the system and computing resource collection Modeling and defining tasks Resource consumption , Execution time and inter-task data transfer overhead .

[0061] As an option, the resource scheduling module constructs an objective function Optimize task scheduling. The objective function is defined as follows:

[0062] in, They are the weight factors for task execution time, resource consumption, and data transmission overhead, respectively, and are used to balance the priorities of different optimization objectives; For the task The estimated execution time; For the task The amount of resources consumed; For the task and Communication delay or data transmission overhead between them.

[0063] It should be noted that the above objective function not only takes into account the execution time of a single task, but also comprehensively evaluates the resource utilization efficiency of the entire system and the communication cost of data transmission between tasks, so as to ensure the optimal global performance of the system.

[0064] Specifically, this module performs resource scheduling through the following steps: First, set resource constraints to ensure that the total resource requirements of the task do not exceed the total available resources of the system. The constraints are defined as follows:

[0065] in, Indicates the task assigned The amount of resources, is the total resource capacity of the system. In addition, the resource allocation of tasks must be non-negative, that is:

[0066] In an exemplary implementation, the resource scheduling module uses the Lagrange multiplier method to optimize the above objective function. , combine the constraints with the objective function and construct the Lagrangian function:

[0067] in, is the Lagrangian function, Represents the weight of the constraint, which is used to balance the relationship between resource allocation and system capacity.

[0068] Next, the Lagrangian function is solved through iterative optimization and the resource allocation scheme is updated. In the iterations, the Lagrange multiplier is updated using the following formula:

[0069] in, is the step size parameter, which is used to control the iterative convergence speed; Represents the Lagrange multiplier value of the current iteration.

[0070] In order to efficiently solve the above optimization problem, the resource scheduling module decomposes the objective function into multiple independent sub-problems through a distributed solver. Each sub-problem is in the form of:

[0071] in, For the task resource constraints, Represents the resource allocation objective function of a single task. On each computing node, solve the corresponding sub-problem separately The solutions to the subproblems are transmitted to the master node through a distributed communication mechanism, and the master node updates the global Lagrange multiplier based on these results. .

[0072] It is understandable that the resource scheduling module will give priority to tasks with strong data dependencies when calculating the data transmission delay between tasks. and When intermediate computation results need to be exchanged frequently, the system will prioritize allocating them to the same computing node or between nodes with lower communication latency.

[0073] During the task execution process, this module supports dynamic adjustment of the task allocation scheme. Specifically, when the actual execution time of the task or resource consumption When the predicted value is deviated, the system will re-optimize the Lagrangian function and dynamically adjust the resource allocation plan. The adjustment process includes the following steps: Recalculate the resource requirements of each task; Update Lagrange multipliers and distribution scheme; Prioritize more resources for critical tasks, such as high-priority tasks or tasks that have a greater impact on the final results.

[0074] In a possible technical implementation, the resource scheduling module can also be combined with artificial intelligence algorithms for predictive scheduling. For example, the execution time and resource requirements of a task can be predicted through a machine learning model, and then the task scheduling scheme can be adjusted based on the prediction results. This extended implementation can significantly improve scheduling efficiency, especially in scenarios with complex task characteristics or strong resource constraints.

[0075] It should be added that when the resource scheduling module outputs the scheduling plan, it will generate a mapping relationship between tasks and computing nodes and pass it to the parallel computing module. Assign to compute nodes , nodes will be used first when tasks are executed computing resources on .

[0076] Through the above design, the resource scheduling module in this embodiment can efficiently complete the resource allocation and scheduling of tasks, which not only improves the utilization efficiency of system resources, but also effectively reduces the data transmission overhead between tasks, and provides an optimized task allocation solution for subsequent parallel computing modules.

[0077] In this embodiment, the specific implementation of the parallel computing module is as follows: The parallel computing module is used to execute the subtasks generated by the task decomposition module in parallel, using a dual strategy of time parallelism and space parallelism, while combining task dependency analysis and dynamic data synchronization to ensure the efficiency and accuracy of the calculation. The goal of the module is to maximize the use of cloud computing resources and reduce computing time and data transmission delays.

[0078] In a possible implementation, the parallel computing module receives the mapping relationship between tasks and computing nodes output by the resource scheduling module, where each task Assigned to the corresponding computing node Before the task is executed, the parallel computing module analyzes and schedules the execution order of the task according to the task dependency graph (DAG).

[0079] Specifically, this module first generates a subtask set based on the task decomposition module. Build task dependency graph Among them, the node set Represents all subtasks, edge sets Represents the data dependency between tasks. The weight of the dependency Representation Task With the task Data transmission delay between.

[0080] As an option, the task dependency analysis unit optimizes the task execution order by topological sorting. The priority of the tasks is determined to ensure that the execution order of any dependent tasks meets the following conditions: like , i.e., the task Depends on the task As a result, the task Must precede the task implement; Based on the topological sorting, the execution priority is calculated for each task, and the execution priority is determined according to the following rules: Node Weight: Task The node weight is its execution time ; Path length: from the starting task to the current task The cumulative weight of the task (including the sum of the weights of all nodes on the path). The higher the priority of the task, the higher the priority of the task will be scheduled first, and the task scheduling order will be generated.

[0081] During the task scheduling process, the following execution conditions are ensured based on the data dependencies between tasks: For any task ,like , then the task Start time Must be later than the task Completion time ,Right now:

[0082] in, For the task The completion time, For the task The start time of the

[0083] During the task execution process, when the computing resource status or task execution time changes, according to the task dependency graph Dynamically adjust the task scheduling order, recalculate task priorities, and update task allocation and time schedule to ensure scheduling rationality and execution efficiency.

[0084] After the task scheduling is completed, the parallel computing module executes the subtasks simultaneously according to the strategies of time parallelism and space parallelism.

[0085] In a possible implementation, the time parallel unit is responsible for allocating tasks to different time windows in a pipeline scheduling manner. Specifically, the time parallel unit divides the execution time of each task into multiple stages and allocates subtasks to different time windows. The staged calculation results are stored in a temporary buffer to provide data for subsequent dependent tasks. The goal of temporal parallel scheduling is to maximize the parallelism of tasks while ensuring that the dependencies between tasks do not conflict.

[0086] As an exemplary implementation, assume that the task set In the task and Can be executed in parallel, and tasks Depends on and In this case, the time parallel unit will and Assign to the first time window and start after it completes .

[0087] The spatial parallel unit is responsible for allocating tasks to different computing nodes. , realizing the physical parallelization of computing tasks. Specifically, the spatial parallel unit allocates each task according to the allocation result of the resource scheduling module. Distribute to compute nodes Each node independently completes the calculation of the assigned task, and the calculation results are transmitted to the nodes where the dependent tasks are located through a high-speed data bus or network.

[0088] It should be noted that in order to further optimize the execution efficiency of tasks, the spatial parallel unit will give priority to the data dependencies between tasks when assigning tasks. and When strong dependencies exist, the spatial parallel units will preferentially assign them to the same node or to physically adjacent nodes, thereby reducing the overhead of data transmission.

[0089] It is understandable that this module also includes a dynamic data synchronization function. During the task execution process, if the dependent task has not been completed, the current task needs to obtain the required data through the data synchronization unit. The data synchronization unit ensures the timely transmission of data between tasks by monitoring the execution status and data flow of the task in real time. Specifically, when the task The output of needs to be passed to the task The data synchronization unit will update the data transmission status in real time and adjust the data transmission priority according to the task priority.

[0090] As a possible technical expansion, the parallel computing module can be combined with artificial intelligence algorithms to predict and optimize the execution efficiency of tasks. For example, the execution time and resource utilization efficiency of tasks can be predicted through machine learning models, and the scheduling strategies of time parallelism and space parallelism can be dynamically adjusted to further improve the overall performance of the system.

[0091] It should be added that this module also supports fault detection and recovery during task execution. When a task fails to complete successfully due to resource failure or network interruption, the module can perform fault tolerance by reallocating tasks. On the compute node If the node fails, the system will automatically migrate it to the standby node. Continue execution.

[0092] Through the above design, the parallel computing module in this embodiment can efficiently process multiple subtasks, maximize the use of time and space resources, and improve the overall execution efficiency of the task.

[0093] In this embodiment, the specific implementation of the result reconstruction module is as follows: The result reconstruction module is used to receive the subtask results output by the parallel computing module, merge, reorganize and verify them, and finally generate the complete calculation results. The main goal of this module is to effectively integrate the decomposed task results in distributed computing to ensure the accuracy and consistency of the calculation.

[0094] In a possible implementation, the result reconstruction module receives the subtask result set from the parallel computing module. ,in Representation Task The module first determines the position and structure of each subtask result in the original task according to the task block information generated by the task decomposition module.

[0095] Specifically, when the input data is a tensor structure, the result reconstruction module concatenates or reorganizes the subtask results according to the decomposition rules. For example, if the input tensor is , subtask results Corresponding to different blocks of the tensor, the reconstruction operation can be expressed as:

[0096] in, Represents the concatenation operation of tensors. The concatenation order is determined by the task division rule of the task decomposition module.

[0097] As an option, for the reconstruction of matrix data, the module can use matrix splicing to combine the results of each subtask Merge into a complete matrix. For example, if the input data is a two-dimensional matrix , the result reconstruction module converts each sub-matrix Concatenate by row or column to generate a complete matrix :

[0098] This splicing method is suitable for block tasks in a specific dimension, such as parallel computing of row blocks or column blocks.

[0099] In one possible implementation, the result reconstruction module also includes a result verification function, which is used to verify whether the merged complete result meets the calculation accuracy requirements. Specifically, the module evaluates the accuracy of the result by comparing the error between the reconstructed result and the reference value of the original task. The error evaluation method can be calculated using the Frobenius norm:

[0100] in, is the original tensor, To reconstruct the results, represents the Frobenius norm, is the error tolerance threshold. If the error exceeds the threshold, the module will return an exception flag and trigger a recalculation or repair process.

[0101] As an exemplary implementation, if the task result contains a large error, the module can correct the result by combining interpolation or optimization algorithms. For example, in an image processing task, if some pixel values ​​are missing or abnormal due to calculation errors, the result reconstruction module can compensate for the missing values ​​through neighborhood interpolation to ensure the integrity of the final result.

[0102] It should be noted that this module supports multiple types of input data formats and structural rules. For example, in natural language processing scenarios, the result reconstruction module can splice multiple block word vector results into a complete sentence embedding vector; in video processing scenarios, the module can combine multiple frame sequence results into a complete video clip.

[0103] In an extended implementation, the result reconstruction module can also incrementally merge some subtask results based on task dependencies. For example, in dynamic streaming data processing, the module can only merge newly generated task results without reprocessing the completed parts, thereby improving system efficiency.

[0104] It is understandable that this module also includes the function of result optimization. After completing the basic merging and verification operations, the module can further optimize the final results. For example, for large-scale tensor data, the reconstruction results can be compressed and stored, and the tensors can be decomposed into core tensors and feature matrices to reduce storage space; for classification task results, post-processing algorithms (such as confidence threshold screening) can be combined to further improve the usability of the results.

[0105] Through the above design, the result reconstruction module in this embodiment can efficiently integrate the sub-task results of parallel computing, ensure the integrity and accuracy of the final result, and provide flexible expansion capabilities to support different types of computing tasks and scenario requirements.

[0106] In general, the system of the present invention receives and preprocesses high-dimensional data through a data input module, converting it into a tensor structure; using a task decomposition module to decompose the tensor based on high-order singular value decomposition, generating a core tensor and an orthogonal matrix, and splitting it into multiple subtasks; using a distributed Lagrangian optimization method through a resource scheduling module, dynamic resource allocation and scheduling of tasks are achieved, minimizing task execution time and resource consumption; in a parallel computing module, time parallelism and space parallelism strategies are combined to efficiently execute subtasks, while ensuring the correctness of tasks through task dependency analysis; finally, a result reconstruction module merges and verifies the subtask results obtained by parallel computing to generate a complete final calculation result. The present invention improves the efficiency of cloud data processing and computing resource utilization, and can be widely used in scenarios such as large-scale data computing, artificial intelligence model training, and scientific simulation.

[0107] In the implementation process of each module of the present invention, by introducing artificial intelligence algorithms, the system can more efficiently process high-dimensional data, optimize resource allocation, and ensure the accuracy of the final calculation results. The following is a step-by-step process to illustrate the role and application scenarios of artificial intelligence algorithms in the specific implementation of the present invention.

[0108] Step 1: Data feature extraction and tensor quantization In the data input module, artificial intelligence algorithms are used to extract features and perform tensor processing for input data of different formats (including structured, semi-structured and unstructured data): For unstructured data such as images and videos, convolutional neural networks (CNNs) are used to extract spatial and channel features and generate three-dimensional or four-dimensional tensor representations.

[0109] For text data, commonly used natural language processing algorithms (such as Word2Vec, GloVe, or BERT) are combined to generate embedded vectors or tensors for subsequent calculations.

[0110] For structured data (such as tables or log data), tensors are directly constructed through simple mapping, and preprocessing operations such as normalization or deduplication are performed on the data. The combination of the above algorithms ensures the compatibility and flexibility of the data input module.

[0111] Step 2: Intelligent Optimization in Task Decomposition The task decomposition module uses artificial intelligence algorithms to decompose and optimize high-dimensional tensors to ensure the balance and effectiveness of the blocks: Based on high-order singular value decomposition (HOSVD), the tensor partitioning strategy is dynamically adjusted in combination with the gradient optimization algorithm to generate core tensors and orthogonal matrices.

[0112] During the tensor decomposition process, deep learning technology is used to further optimize the task block results to ensure that the decomposed tasks can be evenly distributed to the computing nodes and improve the system parallelism.

[0113] Step 3: Intelligent dynamic allocation in resource scheduling The resource scheduling module uses reinforcement learning algorithms to build an intelligent scheduler that dynamically adapts to the resource requirements and system status of different tasks: The scheduler uses a deep Q-learning algorithm to analyze the resource requirements, execution time, and network transmission delay of subtasks in real time, and dynamically adjusts the mapping relationship between tasks and nodes.

[0114] At the same time, the Bayesian optimization method is used to globally optimize the scheduling strategy to ensure the overall optimality of resource allocation and further improve resource utilization efficiency.

[0115] Step 4: Intelligent prediction and dependency analysis during task execution In the parallel computing module, artificial intelligence algorithms are used to analyze the dependencies of tasks and predict the execution time and resource usage of tasks: Use graph neural network (GNN) to model the task dependency graph (DAG), determine the critical path tasks and dependency priorities, and optimize the task scheduling order.

[0116] Based on time series prediction models (such as LSTM or GRU), the task execution time and data transmission delay are predicted to further optimize the task time allocation and parallel execution plan.

[0117] Step 5: Result verification and intelligent repair In the result reconstruction module, the subtask results are verified and optimized through artificial intelligence algorithms: Use unsupervised learning models, such as autoencoders, to detect potential anomalies or errors in computational results.

[0118] Use generative adversarial networks (GANs) or interpolation algorithms to repair abnormal results or missing data to ensure the integrity and accuracy of the final output results.

[0119] Artificial intelligence algorithms are used throughout the data input, task decomposition, resource scheduling, parallel computing and result reconstruction modules, providing the system with efficient data processing capabilities, dynamic resource optimization capabilities, and accurate result verification and repair capabilities. Through the flexible application of artificial intelligence algorithms, the present invention can adapt to complex high-dimensional data processing requirements and significantly improve the performance and reliability of the system.

[0120] Please see attached Figure 2 The present invention also provides a cloud data processing method based on artificial intelligence algorithm. This method is based on the overall architecture and module collaborative working mechanism of the aforementioned system, and sequentially completes the whole process of data reception and processing, task decomposition, resource scheduling, parallel computing and result reconstruction, ensuring the efficiency and accuracy of high-dimensional data processing. The following is a detailed description of each step: S1, receiving high-dimensional data and converting the data into a tensor structure; This step receives external high-dimensional data through the data input module, and preprocesses and converts the data. The input data can include structured data (such as tables), semi-structured data (such as JSON files), or unstructured data (such as images, videos). The data input module standardizes data in different formats, including deduplication, normalization, and format conversion. Subsequently, the input data is mapped to a tensor structure for subsequent task decomposition processing. For example, for image data, the system converts it into a three-dimensional tensor to represent height, width, and channel information; for text data, the system generates a tensor representation based on the embedding method.

[0121] S2. Perform high-order singular value decomposition on the tensor, decompose it into core tensors and orthogonal matrices, and generate multiple subtasks; In this step, the task decomposition module receives the tensor data generated in step S1 and decomposes the tensor based on the high-order singular value decomposition (HOSVD) method. The decomposition process extracts the core information of the tensor, generates the core tensor and the orthogonal matrix of each dimension, and completes the dimensionality reduction and block operations. At the same time, based on the decomposition results, the task decomposition module generates multiple subtasks that can be processed independently and in parallel, and each subtask corresponds to a block of the tensor. The block strategy can be flexibly adjusted according to the size of the tensor, the data distribution, and the status of subsequent computing resources to ensure that the subtasks can be evenly distributed to different computing nodes.

[0122] S3, based on the distributed Lagrangian optimization method, allocate and schedule resources for subtasks to minimize task execution time and resource consumption; In this step, the resource scheduling module allocates and schedules resources for the subtasks generated in step S2. The module constructs an optimization model for task scheduling, takes the execution time of subtasks, resource consumption, and data transmission overhead between tasks as optimization targets, and generates a task allocation plan in combination with the total resource constraints of the system. Through the distributed Lagrangian optimization method, the module dynamically adjusts the task allocation strategy, assigns subtasks to appropriate computing nodes, and prioritizes the optimization of task dependency links with large data transmission delays. The generation result of the scheduling plan includes the mapping relationship between tasks and nodes, which is used to guide the execution of parallel computing.

[0123] S4, based on the time parallelism and space parallelism mechanism, the subtasks are assigned to different time windows and computing nodes for execution, and the task order is guaranteed through task dependency analysis; This step uses the parallel computing module to achieve efficient execution of subtasks. The module uses a dual mechanism of time parallelism and space parallelism: In terms of time parallelism, the module divides subtasks into different time windows based on the pipeline scheduling strategy and executes them sequentially to ensure efficient use of computing resources; In terms of spatial parallelism, the module distributes subtasks to different computing nodes according to the allocation plan generated in step S3, making full use of the parallel capabilities of multi-node resources.

[0124] In order to ensure the correct execution of tasks, the module uses the task dependency analysis function to topologically sort the dependencies between tasks to ensure that dependent tasks are completed first. For example, for a task chain with data transmission dependencies, the system will prioritize the scheduling of predecessor tasks and deliver the calculation results in a timely manner.

[0125] S5, merge and reconstruct the subtask results obtained by execution, and output the final calculation result; In this step, the result reconstruction module receives the subtask results generated by the parallel computing module, and merges and reorganizes them based on the block rules provided by the task decomposition module. The module splices the subtask results into a complete tensor or matrix form, and performs result verification to verify whether the merged results meet the accuracy requirements. If an abnormal result is found, the module can trigger the correction or recalculation process to ensure the accuracy of the final calculation result. Finally, the module outputs the complete calculation result to the external system or user.

[0126] It should be noted that each step in this method works together through modular design and efficient algorithms to ensure the processing efficiency of large-scale high-dimensional data and the reliability of calculation results. This method can be widely used in artificial intelligence training, big data analysis, scientific computing and other fields. Through the dynamic scheduling and efficient allocation of cloud computing resources, it solves the computational complexity and resource optimization problems in high-dimensional data processing.

[0127] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cloud data processing system based on artificial intelligence algorithm, characterized in that: include: A data input module, used for receiving and loading high-dimensional data, and converting the high-dimensional data into a tensor structure; A task decomposition module is used to decompose the high-dimensional data, decompose the high-dimensional tensor into a core tensor and multiple orthogonal matrices based on high-order singular value decomposition, and generate multiple subtasks; A resource scheduling module, used to allocate and schedule resources for the subtasks based on a distributed optimization method to minimize task execution time and resource consumption; A parallel computing module, used for performing parallel computing on the subtasks in the time domain and the space domain, including time parallel pipeline execution and space parallel task distribution; The result reconstruction module is used to merge and reconstruct the subtask results obtained by the parallel computing module to generate the final computing result.

2. The cloud data processing system based on artificial intelligence algorithm according to claim 1, characterized in that: The data input module preprocesses the input data, including data cleaning, formatting and tensor conversion.

3. The cloud data processing system based on artificial intelligence algorithm according to claim 1, characterized in that: The high-order singular value decomposition decomposes the high-dimensional tensor into a core tensor and multiple orthogonal matrices including: Perform matrix expansion on each dimension of the tensor to convert the high-dimensional tensor into matrix form; Decomposing the matrix by singular value decomposition to obtain an orthogonal matrix of corresponding dimension; According to the orthogonal matrix, a core tensor decomposition is performed on the high-dimensional tensor.

4. The cloud data processing system based on artificial intelligence algorithm according to claim 1, characterized in that: The resource scheduling module implements resource allocation through a distributed Lagrangian optimization method, which specifically includes: Construct a resource scheduling objective function to simultaneously optimize task execution time, resource consumption, and data transmission overhead. The objective function is: ， in, is the number of tasks, For the task The execution time, For the task resource consumption, For the task and tasks The data transmission delay between are the weight factors for time, resources, and transmission, respectively; Introduce resource constraints, the constraints are: ， in, For the task The amount of resources allocated, is the total system resources; Construct the Lagrangian function and combine the resource scheduling objective function with the constraints to obtain: ， in, is the Lagrange multiplier; The Lagrangian function is solved based on the distributed iterative optimization method, and the global optimal solution for resource allocation is achieved by updating the Lagrangian multiplier. The iterative update formula of the distributed iterative optimization method is: ， in, is the number of iterations, is the step size parameter; A task allocation scheme is determined according to the global optimal solution, and tasks are allocated to computing nodes.

5. The cloud data processing system based on artificial intelligence algorithm according to claim 4 is characterized in that: The resource scheduling module performs task scheduling based on the Lagrangian function optimization result, specifically including: The objective function is decomposed into multiple sub-problems through a distributed solver, and the optimization goal of each sub-problem is: ， in, For the task resource constraints; Solve the subproblems in parallel on each computing node and determine the resource allocation for each task ; Based on task execution time , Resource consumption and data transmission overhead The task allocation scheme is dynamically adjusted based on the trade-off results; According to the task allocation scheme, subtasks are allocated to corresponding computing nodes to minimize the overall execution time and resource consumption of the system.

6. The cloud data processing system based on artificial intelligence algorithm according to claim 1, characterized in that: The parallel computing module includes: The time parallel unit is used to assign subtasks to time windows to form a task pipeline; A spatial parallel unit, used to distribute the subtasks to different computing nodes for parallel execution; The task dependency analysis unit is used to analyze the dependency of tasks based on a directed acyclic graph to ensure execution order and data synchronization.

7. The cloud data processing system based on artificial intelligence algorithm according to claim 6, characterized in that: The task dependency analysis unit is used to model and schedule the dependency relationship of tasks based on a directed acyclic graph, specifically including: Assemble the tasks Represented as a directed acyclic graph ,in is the set of task vertices, is the data dependency edge between tasks, and the weight of the edge represents the data transmission overhead between tasks. ; The directed acyclic graph is topologically sorted to determine the order of task execution, and the topological sorting is performed according to the following conditions: like , i.e., the task Depends on the task As a result, the task Must precede the task implement; Calculate the execution priority of the task, which is determined by the node weight and path length. The node weight is the task execution time. , the path length is the cumulative weight from the starting task to the current task; Based on the execution priority, generating a task scheduling order; During the scheduling process, ensure that tasks are Executes all dependent tasks After completion, the execution conditions are: For any task ,like , then the task Start time Must be later than the task Completion time ; Dynamically adjust the task scheduling order. When the computing resource status or task execution time changes, re-optimize the scheduling order based on the task dependency graph.

8. The cloud data processing system based on artificial intelligence algorithm according to claim 1, characterized in that: The subtask results obtained by the parallel computing module are merged and reconstructed to generate the final computing result, including: Receive all subtask results output by the parallel computing module ,in Indicates The calculation results of each subtask; Based on the block information of the task decomposition module, according to the logical order and dependency relationship of the subtasks, Combined into an overall result; Perform matrix concatenation or tensor reconstruction on the subtask results. If the subtask is a matrix result, concatenate according to the following rules: ， If the subtask is a tensor result, the original tensor is reconstructed based on the multi-dimensional index of the tensor ; Output the final calculation results and feeds the results back to the system or external call interface.

9. The cloud data processing system based on artificial intelligence algorithm according to claim 8, characterized in that: The subtask results obtained by the parallel computing module are merged and reconstructed to generate the final computing result, which further includes: Perform result verification to verify the integrity and consistency of the merged results, and use the following formula to determine whether the calculation accuracy requirements are met: ， in, is the original tensor, is the reconstructed tensor, represents the Frobenius norm, is the accuracy threshold.

Images