A hierarchical reward training system for multi-modal generative node workflows

Abstract

The present application relates to the technical field of multi-modal generation, and discloses a layered reward training system for a multi-modal generation node workflow, comprising a workflow execution unit, a running data processing unit, a reward data generation unit and a model training unit. The workflow execution unit schedules multiple multi-modal generation nodes according to a directed workflow and forms node execution records; the running data processing unit reads node running resource data, forms three-item change combinations, obtains resource inflection degrees and determines actual execution segments; the reward data generation unit assembles layered reward training records based on node scores, resource inflection degrees and overall process scores; and the model training unit trains a layered reward model based on the records, so that the model outputs model-predicted node scores, model-predicted segment scores and model-predicted overall process scores. The present application is used for candidate output scoring, successor node file selection and model version rollback in a multi-modal generation workflow.

Description

Technical Field

[0001] This invention relates to the field of multimodal generation technology, and more specifically, to a hierarchical reward training system for multimodal generation node workflow. Background Technology

[0002] In a multimodal generation platform, business logic typically involves a workflow control service that strings together nodes such as text processing, image generation, audio generation, video generation, and format conversion. Node input and output files are usually stored in object storage devices, node calls are handled by GPU services, and execution logs, GPU memory records, file read / write records, and network transmission records are continuously written by monitoring or scheduling components.

[0003] In engineering practice, workflow control services focus more on node dependency order, task status, and the availability of the final output. When training reward models, a common practice is to write node-level evaluations, process-level evaluations, or human preference samples into the training data, and then assign rewards to intermediate nodes according to the scheduling order or the final output result. This approach is easy to implement, but it doesn't adequately reflect changes in GPU usage, file read / write operations, and network transmission between multimodal generation nodes.

[0004] When the workflow switches from text processing to image generation, or from image generation to video generation, the runtime load, memory usage, and data transfer volume between nodes will change. If the reward data is still distributed evenly according to the scheduling order, nodes with large resource changes and nodes with relatively stable resource changes may fall into the same reward distribution process, resulting in a decrease in the correspondence between the intermediate node scores and the actual execution process on the server.

[0005] Furthermore, multimodal generation nodes are typically distributed across different GPU services or container instances, and node outputs are passed to subsequent nodes via object storage devices and network switching devices. When training a reward model using only the final output score, the training data struggles to distinguish the impact of node content quality, file transfer processes, and device load variations on subsequent outputs, leading to potential fluctuations in the model's scores near cross-modal nodes. Summary of the Invention

[0006] This invention provides a hierarchical reward training system for multimodal generation node workflows, which solves the technical problems mentioned in the background art.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution.

[0008] A hierarchical reward training system for multimodal generation node workflow includes a workflow execution unit, a runtime data processing unit, a reward data generation unit, and a model training unit. The workflow execution unit is used to schedule multiple multimodal generation nodes according to a directed workflow and form a node execution record; The operation data processing unit is used to read node operation resource data representing calculation duration, device storage pressure, and data transfer volume from the node execution record. In each actual execution branch, the relative change conversion of the node operation resource data of adjacent nodes is performed according to the node completion order to form a three-item change combination. The current three-item change combination is compared with the next three-item change combination to obtain the resource turning degree, which is used to characterize the degree of change in the direction of resource change before and after the node. The actual execution segment is formed according to the starting point of the actual execution branch, the ending point of the actual execution branch, and the adjacent local maximum position of the resource turning degree. The reward data generation unit is used to obtain node scores through the node evaluation model, obtain the entire process score through the overall evaluation model, generate actual execution segment scores based on node scores and resource turning degree, and assemble node scores, actual execution segment scores and the entire process score into hierarchical reward training records. The model training unit is used to train a hierarchical reward model based on the hierarchical reward training records, so that the hierarchical reward model outputs the model prediction node score, the model prediction segment score, and the model prediction score for the entire process.

[0009] Compared with the prior art, the present invention has the following substantial features and significant progress: This invention, operating within an environment that integrates a workflow control server, training data server, GPU training server, object storage device, model repository server, and network switching device, utilizes node execution records to obtain node runtime resource data. It then establishes resource turning points and actual execution segments based on actual execution branches, enabling node scores, actual execution segment scores, and overall process scores to correspond to the execution process of multimodal generation nodes. Compared to methods that only transmit rewards based on scheduling order or final results, this system helps reduce the dilution of cross-modal node scores by adjacent stable nodes, improves the correspondence between hierarchical reward training records and GPU inference, file read / write, and network transmission processes, and allows the trained hierarchical reward model to be used for candidate output ranking, successor node file selection, and model version rollback. Attached Figure Description

[0010] Figure 1 This is a system architecture diagram of a hierarchical reward training system for a multimodal generation node workflow according to the present invention. Detailed Implementation

[0011] The following description is provided in conjunction with the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the invention; those skilled in the art can make equivalent substitutions or combinations for the specific implementations.

[0012] Example 1: This implementation method applies to directed workflows consisting of text, image, audio, video, and format conversion-related generation tasks. In actual operation, this directed workflow is typically executed by multiple multimodal generation nodes scheduled by a workflow control server. The node output files are written to object storage devices via network switching equipment and then read by subsequent multimodal generation nodes. Because different multimodal generation nodes invoke different generation models, input file sizes, output file types, and utilize different devices, the number of workflow topology layers only indicates the software dependency order and cannot directly indicate the common changing positions of computation duration, device storage pressure, and data transfer volume within the server chain.

[0013] The system of this embodiment includes a workflow execution unit 100, a multimodal generation node 200, an object storage device 300, a runtime data processing unit 400, a reward data generation unit 500, a model training unit 600, a model repository server 700, and a network switching device 800.

[0014] In this embodiment, the system only uses the node running resource data that can be obtained from the scheduler timestamp, GPU driver monitoring interface and network interface counter in the node execution record, and forms resource turning degree, actual execution segment, actual execution segment score and hierarchical reward training record in the training data server and GPU training server, so that the training objective of the hierarchical reward model corresponds to the execution stage in the actual server chain.

[0015] Example 2: The workflow execution unit is used to schedule multiple multimodal generation nodes according to a directed workflow and form node execution records. The workflow execution unit can be deployed on the workflow control server and read the node registration file through the existing network connection, writing the node number, device address, input data type, output data type, callable model version, and message queue topic into the node directory, so that each multimodal generation node can be referenced by the directed workflow configuration.

[0016] When parsing a directed workflow configuration, the workflow execution unit first reads the predecessor and successor relationships, input ports, and output ports between each multimodal generation node. Then, it checks whether the output data type of the previous multimodal generation node is consistent with the input data type of the subsequent multimodal generation node. For directed workflow configurations with missing input files, invalid node addresses, or loops, the workflow execution unit records the error and stops the execution of that sample. For directed workflow configurations that pass the checks, the workflow execution unit performs topology sorting to form a node execution queue, where topology sorting is used for execution scheduling.

[0017] When the workflow execution unit sends sample inputs to each multimodal generation node, it can submit multimodal generation nodes with dependencies sequentially according to the node execution queue, or submit multimodal generation nodes without dependencies in parallel based on device availability. For multimodal generation nodes with dependencies, the workflow execution unit submits the subsequent multimodal generation node only after writing the node output file of the predecessor multimodal generation node to the object storage device and completing verification. This process ensures that the reference relationship between node input files and node output files remains consistent with the directed workflow configuration.

[0018] Node execution records are generated by the workflow execution unit during the execution of multimodal generation nodes. Each node execution record is bound to at least the sample number, workflow number, and node number, and records the node start time, node end time, GPU device number, node input file address, node output file address, output file size, model version, random seed, execution status, and error code. In parallel branching scenarios, the workflow execution unit generates separate node execution records for each multimodal generation node; when the output file of the same predecessor multimodal generation node is referenced by multiple subsequent multimodal generation nodes, the node execution record retains the reference relationship for the same node output file.

[0019] Example 3: Multimodal generation nodes are used to perform generation tasks related to text, images, audio, video, or format conversion.

[0020] After receiving sample input from the workflow execution unit, the multimodal generation node reads its input file from the object storage device, calls the deployed generative model to complete inference, and writes the output file back to the object storage device and records it in the node execution log. The object storage device is used to store node input and output files, and can also store user-input text, reference images, reference audio, reference video, the final output file, and the target output. File read / write operations between the object storage device and the multimodal generation node are performed through a network switching device, allowing node input and output files to be reused by multimodal generation nodes on different servers.

[0021] Network switching devices are used to carry regular network communication between multimodal generation nodes, object storage devices, and workflow execution units.

[0022] Example 4: The runtime data processing unit reads node runtime resource data from the node execution record, representing computation duration, device storage pressure, and data transfer volume. The node runtime resource data includes the number of seconds the node runs, the node's maximum video memory usage, and the number of bytes transferred. The number of seconds the node runs represents the computation duration, the node's maximum video memory usage represents the device storage pressure, and the number of bytes transferred represents the data transfer volume.

[0023] Node running time in seconds It means that, among them This represents the node number in an actual execution branch, arranged in the order of node completion. The number of multimodal generation nodes. The node's runtime in seconds is obtained by the workflow scheduler based on the node's end time minus its start time, and is used to represent the number of nodes. The actual duration of computing time occupied by each multimodal generation node. If a multimodal generation node retryes, the workflow execution unit can save the number of seconds each node runs according to its execution status in the node execution record. The data entering the training record is based on the record of successful completion or the record saved according to the execution status.

[0024] The highest memory of the node is This indicates that the node's highest video memory is read by the GPU driver monitoring interface. Peak memory usage during the execution of each multimodal generation node is recorded to reflect the device storage pressure generated when the multimodal generation node loads models, tensors, and intermediate results. For text, image, audio, or video generation models deployed on GPU servers, the node's maximum GPU memory usage can be periodically read from the peak record via the regular GPU driver monitoring interface. For format conversion-related generation tasks, if they do not occupy the GPU, the node's maximum GPU memory usage can be determined based on valid GPU monitoring data in the node execution record; if valid GPU monitoring data is missing, it will be treated as incomplete data collection.

[0025] The number of bytes transmitted by the node This indicates that the number of bytes transmitted by a node is counted by the network interface counter. The amount of data transferred by each multimodal generator node when reading input files from nodes and writing output files from nodes is used to reflect the amount of data movement of media files and tensors between the server and the object storage device. When the same node output file is referenced by multiple subsequent multimodal generator nodes, the number of bytes transferred by each node is entered into the respective node execution record according to the corresponding actual read and write operations.

[0026] To ensure that node runtime resource data corresponds to specific samples and multimodal generation nodes, the workflow execution unit and runtime data processing unit use sample number, workflow number, and node number as a joint index. Time records can use timestamps generated by a unified clock source, and the node input file address, node output file address, and file checksum are written to the log database along with the node execution status.

[0027] Example 5: The runtime data processing unit performs relative change calculations on the node runtime resource data of adjacent nodes in each actual execution branch according to the node completion order, forming three change combinations. It then compares the current three-change combination with the next three-change combination in terms of directional consistency to obtain the resource turning degree, which characterizes the degree of change in the direction of resource changes before and after a node. Based on the start point of the actual execution branch, the end point of the actual execution branch, and the adjacent local maximum position of the resource turning degree, it forms an actual execution segment. The runtime data processing unit can be deployed on a training data server and reads node runtime resource data from the node execution records output by the workflow execution unit.

[0028] For an actual execution branch from a root node to a leaf node, the runtime data processing unit arranges the multimodal generation nodes according to their completion order. This order describes the actual execution path on the server chain and is not directly equivalent to the number of topology layers. The number of topology layers is only used by the workflow execution unit to form a node execution queue, while the actual execution segments are determined by the relative changes in adjacent node execution resource data. For workflows containing parallel branches, the runtime data processing unit determines the actual execution segments for each actual execution branch from the root node to a leaf node and processes workflows containing parallel branches based on node execution records.

[0029] When calculating the degree of resource turning, the running data processing unit only compares the relative changes of adjacent nodes, without using fixed resource thresholds preset for node running seconds, node maximum video memory, or node transfer bytes. This limitation eliminates the need for the system to pre-determine whether a node's running seconds exceed a fixed time threshold, node's maximum video memory exceeds a fixed video memory threshold, or node transfer bytes exceeds a fixed transfer threshold. Instead, it identifies local turning points in the actual execution phase by the common direction of change between adjacent nodes.

[0030] The running data processing unit uses running calculations to protect small quantities at different calculation locations with different dimensions. Memory protection for small amounts Transmission calculation protection small quantity And direction length calculation to protect small quantities The runtime computation protection quantity has the dimension of seconds, and its value is greater than zero and not greater than the time resolution of the workflow scheduler; the memory computation protection quantity has the same storage capacity dimension as the node's highest memory, and its value is greater than zero and not greater than the memory sampling resolution of the GPU driver monitoring interface; the transmission computation protection quantity has the dimension of bytes, and its value is greater than zero and not greater than the byte counting resolution of the network interface counter; the directional length computation protection quantity is a dimensionless positive quantity used for the numerical stabilization of the product of the forward and backward directional lengths.

[0031] Adjacent operational changes The calculation method is as follows: ; in, Indicates changes in adjacent operations. Indicates the first The number of seconds each multimodal generation node runs. Indicates the first The number of seconds each multimodal generation node runs. This indicates that the calculation process is protected against small changes. Since both the numerator and denominator are in units of seconds, their ratio is dimensionless. Changes between adjacent operations are used to eliminate the influence of the absolute order of magnitude of the number of seconds a node operates, while preserving the relative changes in the calculation duration between adjacent nodes.

[0032] Adjacent memory changes The calculation method is as follows: ; in, This indicates changes in adjacent memory locations. Indicates the first The highest video memory of a multimodal generation node. Indicates the first The highest video memory of a multimodal generation node. This indicates a small amount of memory protection. Since both the numerator and denominator are in the same storage capacity dimension, their ratio is dimensionless. Changes in adjacent memory are used to eliminate the absolute magnitude differences caused by different GPU memory capacities and different generation model sizes, while preserving the relative changes in device storage pressure between adjacent nodes.

[0033] Adjacent transmission changes The calculation method is as follows: ; in, Indicates changes in adjacent transmissions. Indicates the first Number of bytes transmitted by each multimodal generation node. Indicates the first Number of bytes transmitted by each multimodal generation node. This indicates that the transmission calculation protects a small amount. Since both the numerator and denominator are in bytes, their ratio is dimensionless. Adjacent transmission changes are used to preserve the relative changes in data transfer between adjacent nodes.

[0034] The three changes are combined to It is indicated that it is formed in the following way: ; in, This represents a combination of three changes. Indicates changes in adjacent operations. This indicates changes in adjacent memory locations. This represents adjacent transmission changes. The combination of these three changes places adjacent changes in computation duration, device storage pressure, and data transfer volume into the same three-dimensional representation, used to describe the direction of resource changes when a node in a server chain jumps.

[0035] Intra-directional product The calculation method is as follows: ; in, Indicates the dot product in the forward and backward directions. This indicates the current combination of three changes. This represents the combination of the last three changes. The inward product is used to measure whether the resource changes before and after a node are in the same direction.

[0036] The product of lengths in the front and rear directions The calculation method is as follows: ; in, Represents the product of lengths in the forward and backward directions. This represents the L2 norm of the current combination of three changes. The L2 norm represents the combination of the last three terms. This indicates that the calculation of directional lengths protects against small quantities. Since the combination of the three changes forms a dimensionless vector, the product of the lengths of the preceding and following directions is also dimensionless, and is used to normalize the inner product of the preceding and following directions.

[0037] Directional similarity The calculation method is as follows: ; in, Indicates directional similarity. Indicates the dot product in the forward and backward directions. This represents the product of lengths in the preceding and following directions. Directional similarity is used to compare the degree of consistency in the direction of resource changes between two segments before and after a node.

[0038] The degree of resource detour The calculation method is as follows: ; in, Indicates the degree of resource diversion. This represents directional similarity. The resource turning degree is a dimensionless quantity used to characterize the degree of change in the direction of resource changes before and after a node. When the combined changes in computation duration, device storage pressure, and data transfer volume are generally in the same direction, the resource turning degree is small; when the direction of the combination of these three changes shows a significant shift, the resource turning degree increases. For the start point of an actual execution branch, the end point of an actual execution branch, and short branches with fewer than three nodes, the runtime data processing unit assigns a value to the resource turning degree of the corresponding node. This allows the node to continue participating in the calculation of the actual execution segment score and the node training percentage.

[0039] Segmented position set It is indicated that it is formed in the following way: ; in, Represents the set of segment positions. Indicates the total number of nodes. Indicates the first The degree of resource detour at each location, This indicates the degree of resource detour at the previous position. This indicates the degree of resource detour at the next position. The formula satisfies the local maximum condition. Limited to For ordinary intermediate nodes, the start and end points of the actual execution branch are not included in the adjacent local maximum judgment. The running data processing unit uses the start and end points of the actual execution branch, as well as the adjacent local maximum positions of the resource turning degree, as the segment positions.

[0040] The data processing unit sorts the segmented location set in ascending order according to the node completion sequence. to ,in Indicates the starting point of the actual execution branch. This indicates the endpoint of the actual execution branch. For The actual execution branch, the running data processing unit forms an actual execution segment, and the node set contains only this unique node. For The actual execution branch, the running data processing unit forms the actual execution segment according to the positions of two adjacent segments; the first A set of nodes that actually execute segmentation. Includes from Start to Previous nodes, when It also includes Except for the endpoint of the actual execution branch being assigned to the last actual execution segment, each segment position node is assigned to the actual execution segment starting from that segment position, and is not assigned to the previous actual execution segment repeatedly.

[0041] For branch start points, branch end points, and short branches with fewer than three nodes, the execution data processing unit directly uses the start and end points of the actual executed branch as segment endpoints, and adjusts the resource bend degree of the corresponding nodes according to the aforementioned assignment rules. For ordinary intermediate nodes, the execution data processing unit retains the resource bend degree sequence and determines the actual execution segment based on the adjacent local maximum positions of the resource bend degree. For actual execution branches without intermediate adjacent local maximum positions, the execution data processing unit treats the entire actual execution branch as a node set and ensures that each multimodal generation node in this node set enters only one actual execution segment.

[0042] Example 6: The reward data generation unit is used to obtain node scores through a node evaluation model, obtain the overall process score through an overall evaluation model, generate actual execution segment scores based on node scores and resource detours, and assemble node scores, actual execution segment scores, and overall process scores into hierarchical reward training records. The reward data generation unit can be deployed on a training data server and share sample numbers, workflow numbers, and node numbers with the running data processing unit.

[0043] The reward data generation unit imports historical multimodal task samples. These samples include at least one of user input text, reference images, reference audio, and reference video, the desired output modality, previously executed workflow configurations, and corresponding node output records. After completing sample format and media attribute checks, the reward data generation unit filters available samples and generates a training index for use by the workflow execution unit and the reward data generation unit. Sample format checks may include file readability, encoding type, and file checksum checks; media attribute checks may include image size, audio sampling rate, video frame rate, and media duration checks. These checks are standard engineering processes for importing existing multimodal task samples into the database.

[0044] At the beginning of each training cycle, the reward data generation unit reads the evaluation model configuration file and freezes it along with the training index until the end of the training cycle. The evaluation model configuration file stores the node evaluation model version and node evaluation model version number according to the output data type, and the overall evaluation model version and overall evaluation model version number according to the workflow number and expected output modality. The frozen evaluation model configuration file is used to generate all node scores and the entire process score within that training cycle, avoiding switching evaluation model versions during the generation of the same batch of training records.

[0045] When multiple validated node evaluation model versions exist for the same output data type, the reward data generation unit sorts them by their node evaluation model version numbers from smallest to largest and selects the node evaluation model version with the largest number. Similarly, when multiple validated overall evaluation model versions exist for the same workflow number and the same desired output modality, the reward data generation unit sorts them by their overall evaluation model version numbers from smallest to largest and selects the overall evaluation model version with the largest number. If the evaluation model configuration file does not contain a node evaluation model version corresponding to the output data type, or does not contain an overall evaluation model version corresponding to the workflow number and desired output modality, the reward data generation unit marks the corresponding record as evaluation unavailable and does not generate the corresponding node score training record or the entire process score training record.

[0046] The reward data generation unit invokes the node evaluation model based on the output modality of the multimodal generation node. The inputs to the node evaluation model include the node input file address, node output file address, output data type, target output address or reference file address, model version of the multimodal generation node, node input file checksum, and node output file checksum. The node input file, node output file, and target output or reference file are provided by an object storage device, and the reward data generation unit reads the corresponding files through a network switching device. For text, image, audio, and video files, the reward data generation unit invokes the existing modality reading program corresponding to the output data type to form the evaluation model input. For format conversion-related generation tasks, the reward data generation unit reads the readability, media attributes, and file checksum of the pre-conversion and post-conversion files as the evaluation model input.

[0047] The node evaluation model outputs node pass / fail propensity values ​​and node fail / fail propensity values, both of which are dimensionless raw output values. If both the target output and the reference file are missing, the reward data generation unit does not generate a node score training record for that multimodal generation node; if the node output file is unreadable or the node output file checksum is inconsistent with the node execution record, the reward data generation unit marks the corresponding record of that node output file as unavailable for evaluation. The reward data generation unit generates a node score based on the node pass / fail propensity values ​​and node fail / fail propensity values. The node score is expressed as... This indicates that the node's qualification propensity value is based on This indicates that the node's non-compliance tendency value is... The calculation method is as follows: ; in, Indicates the node score. Indicates the node's qualification tendency value. This indicates the node's tendency to fail. This represents an exponential function. Node scores are used to represent the... The normalized reward for the output file of the multimodal generation node is bound to the node evaluation model version, the model version of the multimodal generation node, the node input file check value, and the node output file check value.

[0048] The reward data generation unit invokes the overall evaluation model based on the final output file, the original multimodal inputs, and the target output. The inputs to the overall evaluation model include the original multimodal input address, the final output file address, the target output address, the desired output modality, the workflow number, the final output file checksum, and the overall evaluation model version. The overall evaluation model outputs an overall pass / fail tendency value and an overall fail / fail tendency value, both of which are dimensionless raw output values. The reward data generation unit generates the overall process score based on the overall pass / fail tendency value and the overall fail / fail tendency value. The overall process score is expressed as follows: This indicates that the overall propensity to pass is based on This indicates that the overall tendency to fail is based on The calculation method is as follows: ; in, This indicates the score for the entire process. Indicates the overall pass / fail tendency value. Indicates the overall tendency to fail. This represents an exponential function. The overall process score is used to represent the normalized reward of the final output file of the complete workflow relative to the original multimodal inputs and the target output.

[0049] The reward data generation unit generates an actual execution segment score within each actual execution segment based on node scores and resource inflection levels. The actual execution segment score is... The calculation method is as follows: ; in, This indicates the actual score for each segment. Indicates the first The set of nodes within each actual execution segment. Indicates the degree of resource diversion. This represents the node score. The calculation assigns the node score of each node within the actual executed segment to... Normalization and aggregation are performed so that nodes with greater resource inflection have a higher data contribution in their respective actual execution segments. However, the contribution size is directly calculated from the node's running resource data, without the need to manually set a fixed resource threshold.

[0050] The reward data generation unit assembles node scores, actual execution segment scores, and overall process scores into hierarchical reward training records. These hierarchical reward training records can also save original inputs, node outputs, node dependencies, node execution status, actual execution segments, node input file addresses, node output file addresses, final output file addresses, and checksums. Object storage device 300 provides node output files and final output files to the reward data generation unit 500. The hierarchical reward training records save file addresses and file checksums, thus avoiding the duplication of large media files on the training data server.

[0051] Valid GPU monitoring data requires the GPU driver monitoring interface to return at least one peak sampling record between the start and end times of the multimodal generation node. The GPU device number bound to the sampling record must match the GPU device number in the node execution record, and the sampling timestamp must fall within the execution time range of the multimodal generation node. If the GPU driver monitoring interface returns an error, the sampling record is empty, the sampling timestamp does not overlap with the node execution time range, or the GPU device number is inconsistent, the reward data generation unit marks the record corresponding to the highest video memory of that node as incomplete.

[0052] Effective network counting requires the network interface counter to generate count records before the multimodal generation node reads the node's input file and after writing the node's output file. These two count records must originate from the same network interface and correspond to the node's input file address, output file address, and execution time range. If the network interface counter experiences a wraparound that can be confirmed by the counter's bit width, the data processing unit performs a standard wraparound correction based on the counter's bit width. If the network interface counter returns an error, the interface is inconsistent, the file address cannot be matched, or the wraparound cannot be confirmed, the reward data generation unit marks the record corresponding to the number of bytes transmitted by that node as incomplete. Incomplete records can be used for node and overall reward training but are not included in segmented reward training.

[0053] Example 7: The model training unit is used to train a hierarchical reward model based on hierarchical reward training records, enabling the hierarchical reward model to output the model's predicted node scores, predicted segment scores, and predicted overall process scores. The model training unit can be deployed on a GPU training server, which reads batch data from the training data server and connects to the model repository server to store the hierarchical reward model.

[0054] The model training unit feeds text, image, audio, and video inputs into the modal encoder and encodes node numbers, predecessor relationships, successor relationships, the actual execution segment in which the node resides, and the degree of resource inflection into node context. The modal encoder can employ existing text encoders, image encoders, audio encoders, and video encoders to read different modal inputs and output corresponding modal encoding results. The node context is aligned according to the sample number, workflow number, and node number in the hierarchical reward training record. Node numbers are represented by trainable embedding vectors, predecessor and successor relationships are represented by directed workflow adjacency masks, the actual execution segment in which the node resides is represented by an embedding vector of the actual execution segment number, and the degree of resource inflection is used as a dimensionless scalar input.

[0055] The hierarchical reward model concatenates the modal coding results with the node context and inputs them into the shared coding layer. It then generates corresponding model-predicted node scores, model-predicted segment scores, and model-predicted overall process scores through the node score output, the actual execution segment score output, and the overall process score output, respectively. The model-predicted node score is... This indicates that the model predicts segmented scores based on... This indicates that the model predicts the score for the entire process. This indicates that the outputs of the three output terminals mentioned above are all dimensionless probabilistic scores, and are constrained by existing nonlinear probability mappings. to This allows the training objective to cover the output quality of nodes, the actual execution of segmented rewards, and the final output quality of the complete workflow.

[0056] The model training unit calculates the node training percentage based on the resource curvature. The node training percentage is... The calculation method is as follows: ; in, Indicates the percentage of nodes trained. Indicates the first The degree of resource deflection at each node, Indicates the first The degree of resource deflection at each node, This represents the total number of nodes. The training percentage of a node is used to represent the number of nodes trained. The proportion of each node's contribution to the node training error is determined so that nodes with significant resource changes receive a training contribution that matches their actual changes.

[0057] The model training unit uses a small amount of predicted scores for truncation. Numerical protection is applied to the model's predicted scores, truncating the predicted scores to dimensionless positive insignificance that is less than a certain value. The truncated model prediction score is... The calculation method is as follows: ; in, This represents the model's predicted score after truncation. This represents the model's predicted score. This indicates that the predicted score is truncated by a small amount. The model training unit uses the one-way contrast error to compare a target score with a model predicted score. The one-way contrast error is expressed as... The calculation method is as follows: ; in, Indicates the error in a single comparison. Indicates the target score. This represents the model's predicted score. This represents the model's predicted score after truncation. This represents a logarithmic function. The single-item comparison error is used to calculate the prediction error corresponding to the node score, the actual execution segment score, and the overall process score, respectively.

[0058] Node training error The calculation method is as follows: ; in, Indicates the node training error. Indicates the percentage of nodes trained. Indicates the node score. This indicates the model's predicted node score. This represents the total number of nodes. The node training error is used to train the node score output.

[0059] Segmented training error The calculation method is as follows: ; in, This represents the segmented training error. Indicates the total number of segments. This indicates the actual score for each segment. This represents the model's predicted segmented score. The segmented training error is used to train the actual segmented score output.

[0060] Overall training error The calculation method is as follows: ; in, This represents the overall training error. This indicates the score for the entire process. This represents the model's predicted score for the entire process. The overall training error is used to train the output of the entire process score.

[0061] The total training error of the new scheme is The calculation method is as follows: ; in, This represents the total training error of the new scheme. Indicates the node training error. This represents the segmented training error. This represents the overall training error. The GPU training server performs forward computation, backpropagation, and parameter updates based on the total training error of the new scheme, and then updates the parameters of the model to be trained. Write to the model repository server.

[0062] Example 8: The model repository server is used to store tiered reward models.

[0063] The model repository server stores the validation error, repeated scoring difference, inference latency, model validation memory usage, and model file verification values ​​of the stratified reward model. It selects candidate versions of the stratified reward model based on these metrics. Validation error measures the difference between the model's predicted score and the training record score; repeated scoring difference measures the stability of multiple scores on the same validation sample; inference latency measures the response time of the scoring service; model validation memory usage measures the device storage pressure during candidate version execution; and model file verification values ​​confirm the integrity of the stratified reward model files.

[0064] The model repository server reads the validation configuration at the start of each training cycle. The validation configuration includes the validation sample set identifier, hardware environment identifier, validation error cap, repeated score difference cap, inference latency cap, and model validation memory usage cap. The validation configuration for the first training cycle is written by the system deployment file. If a previous stable version of the hierarchical reward model already exists, the validation error cap, repeated score difference cap, inference latency cap, and model validation memory usage cap are all taken from the previous stable version.

[0065] When a candidate version is selected by the model repository server as a validated stratified reward model, the model repository server writes the validation error, repetition score difference, inference latency, and model validation memory usage of that candidate version into the validation configuration for the next training cycle. This configuration serves as the upper limit for validation error, repetition score difference, inference latency, and model validation memory usage for the next training cycle. If no candidate version is selected as a validated stratified reward model, the model repository server retains the validation configuration from the previous training cycle. If the validation sample set identifier or hardware environment identifier changes, the model repository server generates a new validation configuration and prohibits direct comparison of validation errors, repetition score differences, inference latency, and model validation memory usage under different validation sample set identifiers or different hardware environment identifiers.

[0066] The model repository server first confirms the integrity of the tiered reward model files for candidate versions based on model file verification values. Then, it compares the verification error, duplicate score difference, inference latency, and model verification memory usage with the upper limits for verification error, duplicate score difference, inference latency, and model verification memory usage stored in the verification configuration, respectively. Only candidate versions with identical model file verification values ​​and all four metrics not exceeding their corresponding upper limits are included in the candidate version ranking. During candidate version ranking, the model repository server prioritizes candidate versions with smaller verification errors; if verification errors are the same, it selects candidate versions with smaller duplicate score differences; if duplicate score differences are still the same, it sequentially selects candidate versions with smaller inference latency and smaller model verification memory usage.

[0067] When a candidate version of the tiered reward model experiences issues such as corrupted model files, insufficient GPU memory, inference interruption, inconsistent model file validation values, validation errors exceeding the validation error limit, duplicate scoring differences exceeding the duplicate scoring difference limit, inference latency exceeding the inference latency limit, or model validation GPU memory usage exceeding the model validation GPU memory usage limit, the model repository server retains the previous stable version of the tiered reward model and provides the validated tiered reward model to the workflow execution unit. During production execution, the workflow execution unit reads the validated tiered reward model, sorts multiple candidate outputs generated by the same multimodal generation node, and delivers the selected node's output file to the subsequent multimodal generation node. When the scoring service is unavailable, the workflow execution unit switches to the previous stable version of the tiered reward model or maintains the existing node output path.

[0068] As an optional implementation, a multimodal generation workflow sequentially executes text processing, image generation, image post-processing, and format conversion-related generation tasks. The runtime data processing unit reads the node runtime in seconds, the node's maximum video memory, and the number of bytes transferred for each multimodal generation node. It then uses runtime protection values, video memory protection values, and transfer protection values ​​to calculate adjacent runtime variations, adjacent video memory variations, adjacent transfer variations, and combinations of these three variations. When these three variations are directionally close between consecutive nodes, the resource curvature is minimal, and adjacent segment positions are not incorrectly split due to a high maximum video memory of a single node.

[0069] As an optional implementation, an actual execution branch transitions from text prompt processing to image diffusion generation, and then to image post-processing and object storage writing. During the text prompt processing stage, the node runtime in seconds, the node's maximum video memory usage, and the number of bytes transferred by the node change relatively smoothly. Upon entering image diffusion generation, both the node's maximum video memory usage and the node runtime in seconds change significantly. During image post-processing and object storage writing, the number of bytes transferred by the node changes relatively significantly. The running data processing unit obtains the resource turning point degree by comparing the directional consistency between the current combination of the three changes and the next combination of the three changes, and forms an actual execution segment at the adjacent local maximum position of the resource turning point degree, thus allowing the actual execution segment score to correspond to the cross-modal execution stage.

[0070] As an optional implementation, a workflow executes image generation nodes and audio generation nodes in parallel based on the same input, and then the output is summarized by subsequent format conversion-related generation tasks. The workflow execution unit creates node execution records for each parallel node and retains the node output file reference relationships shared with the predecessor node. The runtime data processing unit determines the actual execution segment for each actual execution branch from the root node to the leaf node; for the start point, end point, and short branch nodes of each actual execution branch, the resource detour degree of the corresponding node is assigned a value. The system determines the set of nodes for each node according to the ascending order of the segment positions, ensuring that multimodal generation nodes within the same actual execution branch do not repeatedly enter two actual execution segments. This process avoids simply merging parallel nodes into the same topology layer for average aggregation, thus preserving the context of the preceding and following nodes for each actual execution branch.

[0071] It should be noted that the workflow execution unit can be implemented by a scheduler on the workflow control server, the running data processing unit and the reward data generation unit can be deployed on the training data server, and the model training unit can be deployed on the GPU training server. Multimodal generation nodes can run on regular GPU servers and deploy the generative models required for text processing, image generation, audio generation, video generation, or format conversion-related generation tasks. Object storage devices, network switching devices, and model repository servers are implemented using existing server or cloud infrastructure. The object storage device stores the files required for training and inference, the network switching device carries regular network communication, and the model repository server stores candidate versions of the hierarchical reward model, validated hierarchical reward models, and validation-related records.

[0072] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A hierarchical reward training system for multimodal generation node workflows, comprising a workflow execution unit, a runtime data processing unit, a reward data generation unit, and a model training unit; characterized in that: The workflow execution unit is used to schedule multiple multimodal generation nodes according to a directed workflow and form a node execution record; The operation data processing unit is used to read node operation resource data representing calculation duration, device storage pressure, and data transfer volume from the node execution record. In each actual execution branch, the relative change conversion of the node operation resource data of adjacent nodes is performed according to the node completion order to form a three-item change combination. The current three-item change combination is compared with the next three-item change combination to obtain the resource turning degree, which is used to characterize the degree of change in the direction of resource change before and after the node. The actual execution segment is formed according to the starting point of the actual execution branch, the ending point of the actual execution branch, and the adjacent local maximum position of the resource turning degree. The reward data generation unit is used to obtain node scores through the node evaluation model, obtain the entire process score through the overall evaluation model, generate actual execution segment scores based on node scores and resource turning degree, and assemble node scores, actual execution segment scores and the entire process score into hierarchical reward training records. The model training unit is used to train a hierarchical reward model based on the hierarchical reward training records, so that the hierarchical reward model outputs the model prediction node score, the model prediction segment score, and the model prediction score for the entire process.

2. The hierarchical reward training system for multimodal generation node workflow according to claim 1, characterized in that, The node operation resource data includes the number of seconds the node runs, the maximum video memory of the node, and the number of bytes transferred by the node. The number of seconds the node runs is used to represent the computation duration, the maximum video memory of the node is used to represent the device storage pressure, and the number of bytes transferred by the node is used to represent the amount of data transferred.

3. The hierarchical reward training system for multimodal generation node workflow according to claim 1, characterized in that, The workflow execution unit is deployed on the workflow control server, the runtime data processing unit and the reward data generation unit are deployed on the training data server, and the model training unit is deployed on the GPU training server. The workflow control server, the training data server, the GPU training server, and multiple multimodal generation nodes are connected via a network switching device. The system also includes an object storage device and a model repository server. The object storage device is used to save node input files and node output files, and the model repository server is used to save hierarchical reward models.

4. A hierarchical reward training system for multimodal generation node workflow according to claim 1, characterized in that, The reward data generation unit is used to import historical multimodal task samples, which include at least one of user input text, reference images, reference audio and reference video, expected output modality, executed workflow configurations and corresponding node output records. After completing sample format checks and media attribute checks, it filters available samples and generates a training index for the workflow execution unit and the reward data generation unit to call.

5. A hierarchical reward training system for multimodal generation node workflow according to claim 1, characterized in that, The workflow execution unit is used to parse the directed workflow configuration, check whether the output data type of the previous multimodal generation node is consistent with the input data type of the next multimodal generation node, and perform topological sorting on the directed workflow configurations that pass the check to form a node execution queue, wherein the topological sorting is used for execution scheduling rather than for actual execution of segmented score generation.

6. A hierarchical reward training system for multimodal generation node workflow according to claim 3, characterized in that, The workflow execution unit is used to send sample inputs to each multimodal generation node; each multimodal generation node is used to read the node input file from the object storage device, call the deployed generation model to complete inference, and write the node output file back to the object storage device and write it into the node execution record.

7. A hierarchical reward training system for multimodal generation node workflow according to claim 1, characterized in that, The running data processing unit is used to compare only the relative changes of adjacent nodes when calculating the degree of resource turning, without using fixed resource thresholds preset for node running seconds, node maximum video memory, or node transmission bytes; the running data processing unit is also used to determine the actual execution segment for each actual execution branch from the root node to the leaf node, and process workflows containing parallel branches based on node execution records.

8. A hierarchical reward training system for multimodal generation node workflow according to claim 1, characterized in that, The reward data generation unit is used to call the node evaluation model according to the output modality of the multimodal generation node, obtain the node pass / fail tendency value and the node fail / fail value, and form the node score based on the node pass / fail tendency value and the node fail / fail value; the reward data generation unit is also used to call the overall evaluation model according to the final output file, the original multimodal input and the target output, obtain the overall pass / fail tendency value and the overall fail / fail value, and form the entire process score based on the overall pass / fail tendency value and the overall fail / fail value.

9. A hierarchical reward training system for multimodal generation node workflow according to claim 1, characterized in that, The model training unit is used to input text, images, audio and video into the modal encoder, and encode the node number, predecessor relationship, successor relationship, actual execution segment where the node is located and the degree of resource turning into node context, so that the hierarchical reward model can generate the corresponding model predicted node score, model predicted segment score and model predicted whole process score through the node score output end, the actual execution segment score output end and the whole process score output end respectively.

10. A hierarchical reward training system for multimodal generation node workflow according to claim 3, characterized in that, The model repository server is used to store the verification error, duplicate score difference, inference latency, model verification memory usage, and model file verification value of the tiered reward model. Based on the verification error, duplicate score difference, inference latency, model verification memory usage, and model file verification value, it selects candidate versions of the tiered reward model. When a candidate version of the tiered reward model has a corrupted model file, insufficient memory, or inference interruption, it retains the previous stable version of the tiered reward model and provides the verified tiered reward model to the workflow execution unit.

Images