An artificial intelligence-based distributed file generation method and system

Abstract

The application provides a kind of distributed file generation method and system based on artificial intelligence, the application is by integrating dynamic node capability image, task graph intelligent evaluation, global optimization scheduling, micro-batch flow execution and fault recovery and five core modules of meta-learning continuous optimization, constructs an efficient, reliable and adaptive distributed file generation system.The application system can realize the real-time perception and quantification of the processing capacity of cluster nodes, accurately evaluate the calculation and communication overhead inside the task through graph neural network, and generate a sharding scheduling scheme that takes into account the completion time and data migration volume based on dynamic programming optimization algorithm, thereby significantly improving the task scheduling efficiency and resource utilization.The system is suitable for large-scale personalized PDF documents, Excel reports and image file generation tasks, can effectively reduce the global completion time, reduce cross-node data transmission overhead, and improve the overall efficiency and reliability of distributed file generation.

Description

Technical Field

[0001] This invention relates to the field of distributed data processing technology, and specifically to a distributed file generation method and system based on artificial intelligence. Background Technology

[0002] In distributed computing environments, large-scale file generation tasks, such as batch generation of personalized PDF documents, Excel reports, or image files based on templates and variable data, have become common needs in many fields such as finance, e-commerce, and government.

[0003] Such tasks typically involve multiple interdependent steps such as data processing, template rendering, and format conversion. Their efficient completion depends on breaking down the task appropriately and scheduling it to multiple computing nodes in the cluster for parallel execution.

[0004] Existing distributed file generation methods typically rely on static resource allocation or simple heuristics for task scheduling, making it difficult to accurately and in real-time perceive and adapt to dynamic load and performance fluctuations across cluster nodes. This can easily lead to unbalanced loads and low resource utilization. Furthermore, the lack of precise pre-assessment of the computational and communication overhead of internal task modules means that scheduling decisions are often based on experience or rough estimates, hindering global optimization and potentially extending overall completion time and generating unnecessary cross-node data migration overhead. To address these issues, an artificial intelligence-based distributed file generation method and system are proposed. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an artificial intelligence-based distributed file generation method and system, which can effectively reduce global completion time, reduce cross-node data transmission overhead, and improve the overall efficiency and reliability of distributed file generation.

[0006] The technical solution adopted in this invention is as follows:

[0007] An artificial intelligence-based distributed file generation method includes the following steps:

[0008] Step S1: For each node, use a time-series prediction model to generate a future performance baseline curve based on its historical performance data, and combine the deviation of real-time indicators from the baseline and the node's static configuration to generate a dynamic capability profile vector through a neural network model.

[0009] Step S2: Receive the macro file generation task and parse it into a task topology graph with data dependencies; based on historical data, evaluate the computational overhead of each module and the communication overhead between modules in the graph through a graph neural network to obtain the task cost evaluation result;

[0010] Step S3: With the goal of minimizing the global completion time and data migration volume, and with node resources as constraints, construct an optimization model; input the dynamic capability profile vector set and the task cost evaluation results into the model to solve, and output a sharding scheduling scheme that allocates each module in the task topology graph to the target node;

[0011] Step S4: According to the segmentation scheduling scheme, the macro task is divided into sub-task units, and execution instructions are sent to the target node in micro-batch units;

[0012] Step S5: The target node executes in micro-batches. After each micro-batch is completed, an intermediate result file is generated, synchronized to the shared storage, and a completion report is sent to the scheduling center.

[0013] Step S6: The scheduling center updates the progress based on the feedback. If a micro-batch fails, it will reschedule the micro-batch to other nodes based on its checkpoint information.

[0014] Step S7: After all micro-batches are completed, merge the intermediate result files to generate the final output file;

[0015] Step S8: Collect historical execution data to train the meta-learner, and dynamically adjust the parameters of the neural network model according to the cluster load characteristics.

[0016] Further, step S1 specifically includes:

[0017] Step S101: Construct the time-series prediction model based on the Long Short-Term Memory network, using the historical CPU utilization, memory utilization, and disk I / O rate of the node as the input sequence, and predict the baseline value sequence of each indicator within a set future time period to form the performance baseline curve;

[0018] Step S102: Collect the instantaneous performance indicators of the node in real time, calculate the difference between the node and the benchmark value on the performance baseline curve at the corresponding time point, normalize the difference, and obtain the deviation vector characterizing the real-time state fluctuation of the node.

[0019] Step S103: Concatenate the static hardware configuration code of the node, the deviation vector and the real-time performance index vector, and input the result into a multilayer perceptron containing at least two fully connected layers; the last layer of the multilayer perceptron outputs a low-dimensional dense vector of a preset dimension as the dynamic capability profile vector.

[0020] Further, step S2 specifically includes:

[0021] Step S201: Abstract the data processing, template rendering, and format conversion operations in the task into task vertices, and abstract the execution order and data transfer relationship that must be satisfied between operations into directed edges, and construct the task topology graph that represents the complete workflow.

[0022] Step S202: Based on the predefined feature mapping table, encode the type of each task vertex, the estimated number of data entries to be processed, and the rendering complexity level of the associated file template into a fixed-length initial feature vector;

[0023] Step S203: A graph convolutional network is used as the graph neural network to perform multiple rounds of neighborhood aggregation and feature update; in each iteration, the vertex aggregates the features of all its incoming edges and updates its own feature representation through a nonlinear transformation function;

[0024] Step S204: In the last layer of the graph neural network, the final feature vector of each vertex is input into a regression subnetwork, and the floating-point computational cost estimate of the module represented by the vertex is output; at the same time, the final feature of each directed edge is input into another regression subnetwork, and the communication cost estimate of the data transmission corresponding to the edge is output.

[0025] Furthermore, step S3 specifically includes:

[0026] S301: Define the objective function of the optimization model as: minimizing the time of the last node to complete processing among all nodes, and minimizing the weighted sum of the total data migration caused by the allocation of modules to different nodes; the constraints include that the available memory capacity of each node must be greater than the sum of the memory requirements of all the modules to which it is allocated;

[0027] S302: Perform a depth-first search on the task topology graph, identify the dependencies between all vertices, generate a linear sequence of vertices, and ensure that all predecessor vertices of any vertex in the sequence appear before it.

[0028] S303: Define state dp[k][R], which represents the optimal objective function value when considering the first k vertices and the resource occupancy of each node is R. Starting from the initial state, consider each vertex in turn, enumerate all feasible allocation nodes, update the dp table according to the state transition equation, until all vertices are processed, and determine the optimal partitioning scheduling scheme by backtracking.

[0029] Further, step S4 specifically includes:

[0030] Step S401: For each data source corresponding to a subtask unit, the dataset to be processed is sequentially divided into several logical data blocks according to the preset maximum number of data records per micro-batch, and each logical data block corresponds to one micro-batch;

[0031] Step S402: Generate an independent instruction object for each micro-batch; the instruction object includes the micro-batch's data range identifier, the network address of the target processing node, the hash value of the processor to be executed, and the storage path of the intermediate result file output by the previous micro-batch on which it depends;

[0032] Step S403: The scheduling center maintains a priority-based instruction sending queue. Based on the execution order of modules in the sharding scheduling scheme and the dependencies between micro-batches, it determines the sending priority of each instruction object and sends them to the target node in sequence through the network.

[0033] Furthermore, step S8 specifically includes:

[0034] Step S801: Extract multiple complete task scheduling and execution records from the historical logs. Each record contains a macro-level task cost evaluation result, a dynamic capability profile vector of each node at the scheduling time, the actual sharding scheduling scheme used, and the final measured global completion time. Encapsulate each record into a meta-task sample.

[0035] Step S802: Train the meta-learner using the MAML algorithm framework; in the inner loop, for each meta-task sample, perform forward prediction using the current portrait generation neural network model parameters, calculate the loss between the predicted scheduling effect and the measured completion time, and calculate the gradient of this loss with respect to the model parameters to support simulation of internal parameter updates with a small step size; in the outer loop, aggregate the internal update experience of multiple meta-task samples, update the initialization parameters of the meta-learner, and enable it to have fast adaptation capabilities;

[0036] Step S803: When a new specific type of load task is detected in the cluster, the meta-learner is triggered; the meta-learner generates a set of incremental adjustment vectors for the weight parameters of the neural network model based on the current load characteristics; the incremental adjustment vectors are applied to the current model parameters to achieve rapid adaptation of the model to the new load.

[0037] Furthermore, the operational details included in the checkpoint information in step S6 are as follows: the intermediate result file is encapsulated in a specific format, and its content includes a data block and a status block; the data block stores the business data after the micro-batch processing is completed; the status block stores a serialized processing status object, which fully records the stack information of the task execution thread, the snapshot of the allocated memory variables, and the location information of the file output handle; during rescheduling, the new node first loads the intermediate result file, parses the status block and deserializes it, accurately reconstructs the runtime environment before the failure in memory, and then continues to execute the unfinished calculation logic from the end of the data block.

[0038] Furthermore, the macro-level document generation task specifically refers to the task of generating a large number of PDF documents, Excel reports, or JPG and PNG image files with personalized content based on predefined document templates, report styles, or image layer templates, combined with batch variable business data obtained from databases or message queues.

[0039] Artificial intelligence-based distributed file generation systems include:

[0040] The intelligent profile management module is used to execute step S1, and specifically includes a time-series prediction unit, an indicator acquisition unit and a neural network computing unit to generate and update the dynamic capability profile vector of the node.

[0041] The task graph parsing and evaluation module is used to perform step S2, and specifically includes a syntax parser, a graph builder and a graph neural network inference engine to parse the task topology graph and evaluate the task cost through the graph neural network.

[0042] An optimized piecewise solver is used to perform step S3, specifically including a model configurator and a dynamic programming solver to run the optimized model and generate a piecewise scheduling scheme;

[0043] The streaming scheduler and execution controller is used to execute steps S4 to S6, and specifically includes a task splitter, an instruction queue manager and a health monitor to be responsible for micro-batch splitting, instruction issuance, progress monitoring and fault-tolerant rescheduling.

[0044] A distributed streaming engine, deployed on various nodes, is used to execute tasks in micro-batches, generate intermediate result files containing checkpoints, and provide execution reports.

[0045] The result aggregator is used to perform step S7, sorting and splicing the intermediate result files according to the dependency relationship defined in the task topology graph, and generating the final file output.

[0046] The meta-learning optimization module is used to perform step S8, and specifically includes a historical data warehouse and a meta-learning trainer to adjust the parameters of the portrait generation neural network model.

[0047] Furthermore, the distributed streaming processing engine includes a state management component; the state management component maintains a state mapping table in the node memory to record the values ​​of all key variables during the current micro-batch processing; the state management component triggers a checkpoint operation at a preset time interval or at a data processing milestone, the checkpoint operation suspends the current processing thread, serializes the state mapping table and program counter into a byte stream, packages it together with the generated business data and writes it into the intermediate result file, and then resumes thread execution.

[0048] The beneficial effects of this invention are:

[0049] This invention integrates five core modules: dynamic node capability profiling, intelligent task graph evaluation, global optimization scheduling, micro-batch streaming execution and fault-tolerant recovery, and meta-learning continuous optimization. This results in a highly efficient, reliable, and adaptive distributed file generation system. The system enables real-time perception and quantification of the processing capabilities of cluster nodes. It accurately evaluates the computational and communication overhead within tasks through graph neural networks and generates sharding scheduling schemes that balance completion time and data migration volume based on optimization algorithms such as dynamic programming. This significantly improves task scheduling efficiency and resource utilization.

[0050] Meanwhile, the system supports streaming execution at the micro-batch level and a lightweight incremental checkpointing mechanism, enabling fine-grained fault tolerance and rapid fault recovery, thus enhancing system robustness. The system is suitable for generating large-scale personalized PDF documents, Excel reports, and image files, effectively reducing global completion time, minimizing cross-node data transfer overhead, and improving the overall efficiency and reliability of distributed file generation. Attached Figure Description

[0051] Figure 1 This is a flowchart of an artificial intelligence-based distributed file generation method according to an embodiment of the present invention;

[0052] Figure 2 This is a flowchart of task graph analysis and evaluation according to an embodiment of the present invention;

[0053] Figure 3 A flowchart for generating an optimized scheduling scheme according to an embodiment of the present invention;

[0054] Figure 4 This is a flowchart of micro-batch splitting and scheduling according to an embodiment of the present invention;

[0055] Figure 5 This is a block diagram of an artificial intelligence-based distributed file generation system according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1

[0058] like Figures 1-4As shown in the figure, an artificial intelligence-based distributed file generation method according to an embodiment of the present invention includes the following steps:

[0059] Step S1: For each node, use a time-series prediction model to generate a future performance baseline curve based on its historical performance data, and combine the deviation of real-time indicators from the baseline and the node's static configuration to generate a dynamic capability profile vector through a neural network model.

[0060] Step S2: Receive the macro file generation task and parse it into a task topology graph with data dependencies; based on historical data, evaluate the computational overhead of each module and the communication overhead between modules in the graph through a graph neural network to obtain the task cost evaluation result;

[0061] Step S3: With the goal of minimizing the global completion time and data migration volume, and with node resources as constraints, construct an optimization model; input the dynamic capability profile vector set and the task cost evaluation results into the model to solve, and output a sharding scheduling scheme that allocates each module in the task topology graph to the target node;

[0062] Step S4: According to the sharding scheduling scheme, the macro task is divided into sub-task units, and execution instructions are sent to the target node in micro-batch units;

[0063] Step S5: The target node executes in micro-batches. After each micro-batch is completed, an intermediate result file is generated, synchronized to the shared storage, and a completion report is sent to the scheduling center.

[0064] Step S6: The scheduling center updates the progress based on the feedback. If a micro-batch fails, it will reschedule the micro-batch to other nodes based on its checkpoint information.

[0065] Step S7: After all micro-batches are completed, merge the intermediate result files to generate the final output file;

[0066] Step S8: Collect historical execution data to train the meta-learner, and dynamically adjust the parameters of the neural network model according to the cluster load characteristics.

[0067] Furthermore, step S1 specifically includes:

[0068] Step S101: Construct a time-series prediction model based on the Long Short-Term Memory network, using the historical CPU utilization, memory utilization, and disk I / O rate of the nodes as input sequences to predict the baseline value sequence of each indicator within a set future time period, forming a performance baseline curve;

[0069] Step S102: Collect the instantaneous performance indicators of the nodes in real time, calculate the difference between them and the benchmark value on the performance baseline curve at the corresponding time point, normalize the difference, and obtain the deviation vector characterizing the real-time state fluctuation of the nodes.

[0070] Step S103: Concatenate the node's static hardware configuration encoding, deviation vector, and real-time performance index vector, and input them into a multilayer perceptron containing at least two fully connected layers; the last layer of the multilayer perceptron outputs a low-dimensional dense vector of a preset dimension as a dynamic capability profile vector.

[0071] Furthermore, step S2 specifically includes:

[0072] Step S201: Abstract the data processing, template rendering, and format conversion operations in the task into task vertices, and abstract the execution order and data transfer relationship that must be satisfied between operations into directed edges, and construct a task topology graph that represents the complete workflow.

[0073] Step S202: Based on the predefined feature mapping table, encode the type of each task vertex, the estimated number of data entries to be processed, and the rendering complexity level of the associated file template into a fixed-length initial feature vector;

[0074] Step S203: Use a graph convolutional network as the graph neural network to perform multiple rounds of neighborhood aggregation and feature update; in each iteration, a vertex aggregates the features of all its incoming edges associated with its predecessor vertices and updates its own feature representation through a nonlinear transformation function;

[0075] Step S204: In the last layer of the graph neural network, the final feature vector of each vertex is input into a regression subnetwork, and the floating-point computational cost estimate of the module represented by the vertex is output; at the same time, the final feature of each directed edge is input into another regression subnetwork, and the communication cost estimate of the data transmission corresponding to the edge is output.

[0076] The system's starting point lies in the rapid perception and quantification of the instantaneous processing capabilities of computing nodes in a distributed cluster. Traditional methods rely on complex time-series predictions to establish performance baselines, which incur high computational overhead and latency. This invention simplifies this by directly integrating the inherent hardware capabilities of nodes with their real-time operating load to generate a capability profile.

[0077] Specifically, for any node in the cluster, the system continuously collects its core real-time performance metrics, such as CPU utilization. Memory usage Disk I / O throughput and network bandwidth utilization etc., forming a real-time indicator vector Meanwhile, the node's static hardware configuration, including the number of CPU cores... Total memory capacity Disk type (encoded as a scalar) ), etc., are encoded as static configuration vectors. .

[0078] Then, these two vectors are concatenated to form the node's composite feature vector. The combined feature vectors are fed into a lightweight multilayer perceptron for nonlinear transformation and dimensionality reduction. ReLU The final output is a low-dimensional dense vector of a preset dimension. This is the node. This is a dynamic capability profile vector. Instead of mining historical periodic patterns, it focuses on capturing the comprehensive processing capabilities of a node determined by both static resources and dynamic load. Its generation process is fast and computationally lightweight, providing the system with a near real-time snapshot of the node's state.

[0079] Furthermore, step S3 specifically includes:

[0080] S301: Define the objective function of the optimization model as: minimizing the time of the last node to complete processing and minimizing the weighted sum of the total data migration caused by the allocation of modules to different nodes; constraints include that the available memory capacity of each node must be greater than the sum of the memory requirements of all the modules to which it is allocated;

[0081] S302: Perform a depth-first search on the task topology graph, identify the dependencies between all vertices, generate a linear sequence of vertices, and ensure that all predecessor vertices of any vertex in the sequence appear before it.

[0082] S303: Define state dp[k][R], which represents the optimal objective function value when considering the first k vertices and the resource occupancy of each node is R. Starting from the initial state, consider each vertex in turn, enumerate all feasible allocation nodes, update the dp table according to the state transition equation, until all vertices are processed, and determine the optimal partitioning scheduling scheme by backtracking.

[0083] While accurately characterizing node capabilities, the system needs to deconstruct and assess the resource requirements of the submitted macro-level document generation task. The macro-level document generation task is parsed as a directed acyclic graph (DAG). , where the vertex Representative atomic operation modules such as data filtering, template rendering, and format conversion, with directed edges. This represents the strict data dependencies and execution order relationships between modules.

[0084] To quantify the computational complexity of each module and the communication cost of data transfer between modules, this invention uses a graph neural network to model the task topology. First, each vertex... Based on its operation type (categorical variable), estimated data volume (numerical variable), template rendering complexity (numerical variable), and other attributes, it is transformed into an initial feature vector through embedding layers and normalization. Subsequently, a graph convolutional network is used for multi-round message passing. In the... Layer, where each vertex aggregates the feature information of all its predecessor vertices: ,in Represents vertices Find the set of incoming neighbors and update its own characteristics: .go through After layer iterations, each vertex obtains a final feature representation containing its global context information. .

[0085] Based on this, two lightweight regression heads are used for evaluation: the overhead regression head outputs the estimated computational cost of the module. The communication overhead regression head is for each edge Based on the characteristics of its two vertices, the data transmission cost is estimated. At this point, the task has been quantified into a task cost assessment result. It accurately describes the computational and communication requirements within the task.

[0086] At this point, the system has two inputs: one is a set of dynamic capability profiles of nodes describing the instantaneous state of cluster resource supply. Secondly, it describes the task cost assessment results of the task resource requirement map. The next decision-making stage, namely global optimization of fragmented scheduling, aims to optimize the task graph. Each module in Optimally assign to a target node superior.

[0087] Its objective function is designed to minimize the weighted sum of the global completion time (Makespan) and the total amount of data migration across nodes: Among them, the global completion time The amount of data migration depends on when the last node completes its assigned task. This refers to the communication overhead incurred by all modules on both ends of the dependency relationship being assigned to different nodes. The sum. A module. At the node Estimated execution time The computational overhead is derived from a simple mapping function. and node image vector The estimate is derived from, for example This reflects the impact of node processing capacity on task execution speed.

[0088] The optimization problem is also constrained by node resources, such as memory constraints: ,in It is a binary allocation variable. It is a module Memory requirements, It is a node Available memory. To solve this NP-hard problem, the system employs a dynamic programming (DP)-based heuristic solver. First, the task graph is topologically sorted to obtain the vertex sequence. Define the state. For consideration before There are vertices and the current resource usage vector of each node is... The optimal objective value at that time. From Begin by processing each vertex in turn. Enumerate all its feasible allocation nodes Through the state transition equation Update status, among which This includes the increased execution time and its contribution to data migration. The final backtracking yields the sharding scheduling scheme. Clearly define the node to which each module belongs.

[0089] Furthermore, step S4 specifically includes:

[0090] Step S401: For the data source corresponding to each subtask unit, according to the preset maximum number of data records in each micro-batch, the dataset to be processed is sequentially divided into several logical data blocks, and each logical data block corresponds to a micro-batch.

[0091] Step S402: Generate an independent instruction object for each micro-batch; the instruction object contains the micro-batch's data range identifier, the network address of the target processing node, the hash value of the processor to be executed, and the storage path of the intermediate result file output by the previous micro-batch on which it depends;

[0092] Step S403: The scheduling center maintains a priority-based instruction sending queue. Based on the execution order of modules in the sharding scheduling scheme and the dependencies between micro-batches, it determines the sending priority of each instruction object and sends them to the target node in sequence through the network.

[0093] Scheduling scheme After generation, the system enters the streaming execution and fault-tolerant recovery phase. To improve parallelism and resource utilization, each module group assigned to a specific node further divides its data to be processed into smaller micro-batches. Let the total data volume of the subtasks be... Micro-batch size is Then it is divided into Each micro-batch. Encapsulated as a separate instruction object This includes the data range, target node, program hash, and the paths to dependent upstream result files. The scheduling center maintains a priority queue, with priorities... The critical path level of the module where this micro-batch is located and its data volume Decide: This ensures that critical path tasks and high-load tasks are executed first. Target nodes execute micro-batch processing according to instructions.

[0094] To achieve fine-grained fault tolerance, the system employs a lightweight incremental checkpointing mechanism. Each node maintains an incremental log of key state variables (such as counters, aggregate values, and file handle positions) in memory during the execution of each micro-batch. Instead of performing a full state save after each micro-batch, it performs a checkpointing only after each batch is completed. When a micro-batch or a data processing milestone is reached, a checkpoint is triggered: the incremental status log since the last checkpoint, the current program counter (PC) value, and the data output position information are serialized and appended to the status block of the current intermediate result file; business data is written to the data block.

[0095] If a micro-batch fails to execute, the scheduling center, based on its checkpoint information, simply reschedules the micro-batch to another healthy node. The new node loads the corresponding intermediate result file, deserializes the state block, accurately reconstructs the execution context before the failure, and then continues execution from the point of failure. This achieves fast and accurate fault recovery, greatly improving system robustness.

[0096] Furthermore, step S8 specifically includes:

[0097] Step S801: Extract multiple complete task scheduling and execution records from the historical logs. Each record contains a macro-level task cost evaluation result, a dynamic capability profile vector of each node at the scheduling time, the actual sharding scheduling scheme used, and the final measured global completion time. Encapsulate each record into a meta-task sample.

[0098] Step S802: Train the meta-learner using the MAML algorithm framework; in the inner loop, for each meta-task sample, perform forward prediction using the current portrait generation neural network model parameters, calculate the loss between the predicted scheduling effect and the measured completion time, and calculate the gradient of this loss with respect to the model parameters to support simulation of internal parameter updates with a small step size; in the outer loop, aggregate the internal update experience of multiple meta-task samples, update the initialization parameters of the meta-learner, and enable it to adapt quickly.

[0099] Step S803: When a new specific type of load task is detected in the cluster, the meta-learner is triggered; the meta-learner generates a set of incremental adjustment vectors for the weight parameters of the neural network model based on the current load characteristics; the incremental adjustment vectors are applied to the current model parameters to achieve rapid adaptation of the model to the new load.

[0100] Furthermore, the operational details included in the checkpoint information in step S6 are as follows: the intermediate result file is encapsulated in a specific format, and its content includes a data block and a status block; the data block stores the business data after the micro-batch processing is completed; the status block stores a serialized processing status object, which fully records the stack information of the task execution thread, the snapshot of the allocated memory variables, and the location information of the file output handle; during rescheduling, the new node first loads the intermediate result file, parses and deserializes the status block, accurately reconstructs the runtime environment before the failure in memory, and then continues to execute the unfinished calculation logic from the end of the data block.

[0101] Furthermore, the macro-level document generation task specifically involves generating a large number of PDF documents, Excel reports, or JPG / PNG image files with personalized content, based on predefined document templates, report styles, or image layer templates, combined with batch variable business data obtained from databases or message queues.

[0102] To enable the system to adapt to constantly changing cluster loads and new task patterns, this invention introduces a meta-learning-driven continuous optimization module, forming an intelligent closed loop from execution feedback to model parameter adjustment. The system continuously collects historical task execution records, and each record... Constitute a meta-task sample This includes the task cost assessment results, the node profile set at the scheduling time, the actual scheduling scheme adopted, and the measured global completion time.

[0103] A model-independent meta-learning framework is used to train a meta-learner. In the inner loop, for each sample... MLP parameters are generated using the current image. The node profile is regenerated by combining real-time metrics at the time (available from historical data). , and then with Input them together into the scheduling simulator to obtain a simulated scheduling scheme. and its estimated completion time Calculate the loss between the prediction and the actual time. And calculate the loss with respect to the parameters. The gradient is used to simulate a one-step parameter update to obtain the adapted parameters. In the outer loop, the goal of meta-learning is to optimize the initial parameters. This allows it to quickly adapt to new tasks from the task distribution with only a few internal updates. Therefore, the external loss is the expected loss of multiple tasks after internal adaptation: And update the meta-parameters using gradient descent: .

[0104] After training, the meta-learner enables the profiling generation MLP to obtain a set of excellent initialization parameters. When the system detects a significant change in cluster load characteristics or the emergence of new tasks, the meta-learner is triggered to quickly generate a set of parameters for the MLP model based on the currently observed load characteristics. Incremental adjustment vector And perform online updates: This allows the simplified profile generation model to dynamically adapt to environmental changes and continuously improve the guiding value of its generated profile vectors for scheduling decisions.

[0105] Example 2

[0106] like Figure 5 As shown, corresponding to the above embodiment of the distributed file generation method based on artificial intelligence, the present invention also proposes an distributed file generation system based on artificial intelligence, including:

[0107] The intelligent profile management module is used to execute step S1, which specifically includes a time-series prediction unit, an indicator acquisition unit, and a neural network computing unit to generate and update the dynamic capability profile vector of the node.

[0108] The task graph parsing and evaluation module is used to perform step S2. Specifically, it includes a syntax parser, a graph builder, and a graph neural network inference engine to parse the task topology graph and evaluate the task cost through a graph neural network.

[0109] An optimized piecewise solver is used to perform step S3, which specifically includes a model configurator and a dynamic programming solver to run the optimized model and generate a piecewise scheduling scheme.

[0110] The streaming scheduler and execution controller is used to execute steps S4 to S6. Specifically, it includes a task splitter, an instruction queue manager, and a health monitor to be responsible for micro-batch splitting, instruction issuance, progress monitoring, and fault-tolerant rescheduling.

[0111] A distributed streaming engine, deployed on various nodes, is used to execute tasks in micro-batches, generate intermediate result files containing checkpoints, and provide execution reports.

[0112] The result aggregator is used to execute step S7, which sorts and concatenates the intermediate result files according to the dependencies defined in the task topology graph, and generates the final file output.

[0113] The meta-learning optimization module is used to execute step S8, specifically including a historical data warehouse and a meta-learning trainer to adjust the parameters of the portrait generation neural network model.

[0114] Furthermore, the distributed streaming engine includes a state management component; the state management component maintains a state mapping table in the node's memory to record the values ​​of all key variables during the current micro-batch processing; the state management component triggers a checkpoint operation at a preset time interval or at a data processing milestone, the checkpoint operation suspends the current processing thread, serializes the state mapping table and program counter into a byte stream, packages it together with the generated business data and writes it to an intermediate result file, and then resumes thread execution.

[0115] The system integrates five core modules: dynamic node capability profiling, intelligent task graph evaluation, global optimization scheduling, micro-batch streaming execution and fault-tolerant recovery, and meta-learning continuous optimization. It has built an efficient, reliable and adaptive distributed file processing platform. Through the meta-learning framework, it continuously collects historical execution feedback and dynamically adjusts the parameters of the node profiling generation model, enabling the system to adapt to the ever-changing cluster load and task types.

[0116] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0118] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0119] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0120] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0121] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An artificial intelligence-based distributed file generation method, characterized by, Includes the following steps: Step S1: For each node, use a time-series prediction model to generate a future performance baseline curve based on its historical performance data, and combine the deviation of real-time indicators from the baseline and the node's static configuration to generate a dynamic capability profile vector through a neural network model. Step S2: Receive the macro file generation task and parse it into a task topology graph with data dependencies; Based on historical data, the computational overhead of each module and the communication overhead between modules in the graph are evaluated using a graph neural network to obtain the task cost evaluation result. Step S2 specifically includes: Step S201: Abstract the data processing, template rendering, and format conversion operations in the task into task vertices, and abstract the execution order and data transfer relationship that must be satisfied between operations into directed edges, and construct the task topology graph that represents the complete workflow. Step S202: Based on the predefined feature mapping table, encode the type of each task vertex, the estimated number of data entries to be processed, and the rendering complexity level of the associated file template into a fixed-length initial feature vector; Step S203: A graph convolutional network is used as the graph neural network to perform multiple rounds of neighborhood aggregation and feature update; in each iteration, the vertex aggregates the features of all its incoming edges and updates its own feature representation through a nonlinear transformation function; Step S204: In the last layer of the graph neural network, the final feature vector of each vertex is input into a regression subnetwork, and the floating-point computational cost estimate of the module represented by the vertex is output; at the same time, the final feature of each directed edge is input into another regression subnetwork, and the communication cost estimate of the data transmission corresponding to the edge is output. Step S3: With the goal of minimizing the global completion time and data migration volume, and with node resources as constraints, construct an optimization model; input the dynamic capability profile vector set and the task cost evaluation results into the model to solve, and output a sharding scheduling scheme that allocates each module in the task topology graph to the target node; Step S4: According to the segmentation scheduling scheme, the macro task is divided into sub-task units, and execution instructions are sent to the target node in micro-batch units; Step S5: The target node executes in micro-batches. After each micro-batch is completed, an intermediate result file is generated, synchronized to the shared storage, and a completion report is sent to the scheduling center. Step S6: The scheduling center updates the progress based on the feedback. If a micro-batch fails, it will reschedule the micro-batch to other nodes based on its checkpoint information. Step S7: After all micro-batches are completed, merge the intermediate result files to generate the final output file; Step S8: Collect historical execution data to train the meta-learner, and dynamically adjust the parameters of the neural network model according to the cluster load characteristics.

2. The distributed file generation method based on artificial intelligence according to claim 1, characterized in that, Step S1 specifically includes: Step S101: Construct the time-series prediction model based on the Long Short-Term Memory network, using the historical CPU utilization, memory utilization, and disk I / O rate of the node as the input sequence, and predict the baseline value sequence of each indicator within a set future time period to form the performance baseline curve; Step S102: Collect the instantaneous performance indicators of the node in real time, calculate the difference between the node and the benchmark value on the performance baseline curve at the corresponding time point, normalize the difference, and obtain the deviation vector characterizing the real-time state fluctuation of the node. Step S103: Concatenate the static hardware configuration code of the node, the deviation vector and the real-time performance index vector, and input the result into a multilayer perceptron containing at least two fully connected layers; the last layer of the multilayer perceptron outputs a low-dimensional dense vector of a preset dimension as the dynamic capability profile vector.

3. The distributed file generation method based on artificial intelligence according to claim 1, characterized in that, Step S3 specifically includes: S301: Define the objective function of the optimization model as: minimizing the time of the last node to complete processing among all nodes, and minimizing the weighted sum of the total data migration caused by the allocation of modules to different nodes; the constraints include that the available memory capacity of each node must be greater than the sum of the memory requirements of all the modules to which it is allocated; S302: Perform a depth-first search on the task topology graph, identify the dependencies between all vertices, generate a linear sequence of vertices, and ensure that all predecessor vertices of any vertex in the sequence appear before it. S303: Define state dp[k][R], which represents the optimal objective function value when considering the first k vertices and the resource occupancy of each node is R. Starting from the initial state, consider each vertex in turn, enumerate all feasible allocation nodes, update the dp table according to the state transition equation, until all vertices are processed, and determine the optimal partitioning scheduling scheme by backtracking.

4. The distributed file generation method based on artificial intelligence according to claim 1, characterized in that, Step S4 specifically includes: Step S401: For each data source corresponding to a subtask unit, the dataset to be processed is sequentially divided into several logical data blocks according to the preset maximum number of data records per micro-batch, and each logical data block corresponds to one micro-batch; Step S402: Generate an independent instruction object for each micro-batch; the instruction object includes the micro-batch's data range identifier, the network address of the target processing node, the hash value of the processor to be executed, and the storage path of the intermediate result file output by the previous micro-batch on which it depends; Step S403: The scheduling center maintains a priority-based instruction sending queue. Based on the execution order of modules in the sharding scheduling scheme and the dependencies between micro-batches, it determines the sending priority of each instruction object and sends them to the target node in sequence through the network.

5. The distributed file generation method based on artificial intelligence according to claim 1, characterized in that, Step S8 specifically includes: Step S801: Extract multiple complete task scheduling and execution records from the historical logs. Each record contains a macro-level task cost evaluation result, a dynamic capability profile vector of each node at the scheduling time, the actual sharding scheduling scheme used, and the final measured global completion time. Encapsulate each record into a meta-task sample. Step S802: Train the meta-learner using the MAML algorithm framework; in the inner loop, for each meta-task sample, perform forward prediction using the current portrait generation neural network model parameters, calculate the loss between the predicted scheduling effect and the measured completion time, and calculate the gradient of this loss with respect to the model parameters to support simulation of internal parameter updates with a small step size; in the outer loop, aggregate the internal update experience of multiple meta-task samples, update the initialization parameters of the meta-learner, and enable it to have fast adaptation capabilities; Step S803: When a new specific type of load task is detected in the cluster, the meta-learner is triggered; the meta-learner generates a set of incremental adjustment vectors for the weight parameters of the neural network model based on the current load characteristics; the incremental adjustment vectors are applied to the current model parameters to achieve rapid adaptation of the model to the new load.

6. The distributed file generation method based on artificial intelligence according to claim 1, characterized in that, The operational details included in the checkpoint information in step S6 are as follows: the intermediate result file is encapsulated in a specific format, and its content includes a data block and a status block; the data block stores the business data after the micro-batch processing is completed; the status block stores a serialized processing status object, which fully records the stack information of the task execution thread, the snapshot of the allocated memory variables, and the location information of the file output handle; During rescheduling, the new node first loads the intermediate result file, parses and deserializes the state block, accurately reconstructs the runtime environment before the failure in memory, and then continues to execute the unfinished computation logic from the end of the data block.

7. The distributed file generation method based on artificial intelligence according to claim 1, characterized in that, The macro-level document generation task specifically refers to the task of generating a large number of PDF documents, Excel reports, or JPG and PNG image files with personalized content, based on predefined document templates, report styles, or image layer templates, combined with batch variable business data obtained from databases or message queues.

8. An artificial intelligence based distributed document generation system for implementing the method of any one of claims 1-7, characterized by, include: The intelligent profile management module is used to execute step S1, and specifically includes a time-series prediction unit, an indicator acquisition unit and a neural network computing unit to generate and update the dynamic capability profile vector of the node. The task graph parsing and evaluation module is used to perform step S2, and specifically includes a syntax parser, a graph builder and a graph neural network inference engine to parse the task topology graph and evaluate the task cost through the graph neural network. An optimized piecewise solver is used to perform step S3, specifically including a model configurator and a dynamic programming solver to run the optimized model and generate a piecewise scheduling scheme; The streaming scheduler and execution controller is used to execute steps S4 to S6, and specifically includes a task splitter, an instruction queue manager and a health monitor to be responsible for micro-batch splitting, instruction issuance, progress monitoring and fault-tolerant rescheduling. A distributed streaming engine, deployed on various nodes, is used to execute tasks in micro-batches, generate intermediate result files containing checkpoints, and provide execution reports. The result aggregator is used to perform step S7, sorting and splicing the intermediate result files according to the dependency relationship defined in the task topology graph, and generating the final file output. The meta-learning optimization module is used to perform step S8, and specifically includes a historical data warehouse and a meta-learning trainer to adjust the parameters of the portrait generation neural network model. 9.The artificial intelligence-based distributed file generation system of claim 8, wherein, The distributed streaming processing engine includes a state management component. The state management component maintains a state mapping table in the node memory to record the values ​​of all key variables during the current micro-batch processing. The state management component triggers a checkpoint operation at a preset time interval or at a data processing milestone. The checkpoint operation involves pausing the current processing thread, serializing the state mapping table and program counter into a byte stream, packaging it together with the generated business data, and writing it into the intermediate result file. Then, the thread execution is resumed.

Images