Data processing method, device, storage medium and program product

Abstract

The present application discloses a data processing method, device, storage medium and program product, which relate to the field of computer technology, including: obtaining load data of multiple data processing networks, and using the characteristic vectors of the data to be processed and the characteristic vectors of the load data as inputs of a pre-built allocation strategy network, predicting the action probability distribution of the multiple data processing networks, and determining the target processing network to be processed based on the action probability distribution, thereby solving the technical problem of unbalanced loads of multiple data processing networks and achieving the technical effect of improving data processing efficiency, reducing hardware resource waste and deployment costs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, storage medium, and program product. Background Art

[0002] As the size of deep learning models continues to grow, Mixture of Experts (MoE) networks have attracted widespread attention due to their ability to efficiently scale model capacity. MoEs utilize multiple expert networks and a gating network to handle complex tasks, allowing each expert network to specialize in a specific type of input data, thereby improving overall performance without significantly increasing computational cost.

[0003] However, in practical applications, a major challenge facing MoE is the imbalanced load on expert networks. Currently, MoE relies primarily on a single gating network to distribute input data to different expert networks for processing. This often leads to some expert networks being overloaded due to frequent selection, while others remain idle for long periods of time. This load imbalance not only reduces data processing efficiency but also wastes hardware resources and increases deployment costs. Summary of the Invention

[0004] The present application provides a data processing method, device, storage medium and program product to at least solve the problem of unbalanced expert network load in related technologies.

[0005] This application provides a data processing method, including:

[0006] Acquire target data to be processed, perform feature extraction on the target data, and generate a first feature vector;

[0007] generating a second feature vector based on acquired target load data of at least one data processing network, wherein the target load data includes historical information of the data processing network;

[0008] Inputting the first eigenvector and the second eigenvector into a pre-built allocation strategy network, and obtaining an action probability distribution output by the allocation strategy network, wherein the action probability distribution is used to represent the probability of at least one data processing network being assigned the eigenvector;

[0009] At least one target processing network is determined in at least one data processing network according to the action probability distribution, and the first feature vector is processed by the at least one target processing network.

[0010] The present application also provides a data processing device, comprising:

[0011] A first acquisition module is used to acquire target data to be processed, perform feature extraction on the target data, and generate a first feature vector;

[0012] a second acquisition module, configured to generate a second feature vector based on acquired target load data of at least one data processing network, wherein the target load data includes historical information of the data processing network;

[0013] an allocation module, configured to input the first eigenvector and the second eigenvector into a pre-built allocation strategy network, and obtain an action probability distribution output by the allocation strategy network, wherein the action probability distribution is used to represent the probability of at least one data processing network being allocated the eigenvector;

[0014] The data processing module is configured to determine at least one target processing network in the at least one data processing network according to the action probability distribution, and process the first feature vector through the at least one target processing network.

[0015] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data processing methods when executing the computer program.

[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0017] The present application also provides a computer program product, comprising a computer program, which implements the steps of any of the above-mentioned data processing methods when executed by a processor.

[0018] This application obtains load data from multiple data processing networks and uses the feature vectors of the data to be processed and the load data as input to a pre-built allocation strategy network. This predicts the probability distribution of actions across the multiple data processing networks and, based on the action probability distribution, determines the target processing network for data processing. This solves the technical problem of unbalanced loads across multiple data processing networks, achieving the technical benefits of improving data processing efficiency, reducing hardware resource waste, and reducing deployment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1A flowchart of a data processing method provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0023] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;

[0024] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0028] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends, the specific application environment architecture or specific hardware architecture is described here.

[0030] In recent years, as deep learning models continue to scale, hybrid expert networks (HENs) have garnered widespread attention for their ability to effectively increase model capacity. The core concept of HENs is to categorize tasks and then assign them to individual expert networks. A HEN is a sparse gate-controlled deep learning model primarily composed of a set of expert networks and a gating network. By introducing multiple expert networks and a gating network, HENs distribute input samples to different expert networks for processing, ultimately aggregating the outputs of each expert network to produce the final result. This structure allows each expert network to focus on processing input data, thereby improving overall model performance and significantly enhancing the model's expressiveness without significantly increasing computational cost.

[0031] However, hybrid expert models face the problem of unbalanced expert network load in practical applications. Related hybrid expert models typically utilize a gating network, which is updated along with other parameters during training. During inference, it predicts the actual expert network that will process the input data, thereby assigning each task to a specific small model. Each small model can be understood as an expert in solving a specific problem, hence the name expert network or expert model. However, the gating network tends to concentrate tasks on a few expert networks, causing these networks to become overloaded while others remain idle. This load imbalance not only reduces the model's training efficiency but can also cause some expert networks to overfit, thereby affecting the model's generalization performance.

[0032] In addition, in order to solve the problem of unbalanced expert load in the hybrid expert model, related methods have also proposed the following typical strategies:

[0033] 1) By setting a maximum upper limit on the number of tasks that each expert model can handle, a hard constraint is imposed on the task allocation of a single expert network, preventing it from being overloaded and affecting overall performance. However, this approach has capacity limitations and requires manual setting of capacity thresholds, making it difficult to adapt to changes in different tasks and data distributions, and lacking flexibility.

[0034] 2) Distribute tasks to specific expert networks based on pre-set fixed rules (such as round-robin or classification). While this approach can achieve even distribution of tasks to a certain extent, it lacks the ability to dynamically respond to task characteristics and the real-time load of the expert network, making it difficult to adapt to complex and changing application scenarios.

[0035] In response to the above technical problems, this application provides a data processing method. By using a reinforcement learning algorithm, a reward function is set according to the real-time load status of each expert network to automatically learn and optimize the task allocation strategy, thereby effectively alleviating the load imbalance problem in the hybrid expert model and further improving the efficiency of the model during training and its generalization ability on unknown data.

[0036] Before explaining this application in detail, it is first necessary to explain the technical terms involved, including:

[0037] A mixture of experts (MOE) is a machine learning architecture consisting of multiple sub-models (called "experts") and a gating mechanism. Experts excel at processing different types of input data or tasks; the gating mechanism dynamically selects one or more experts for processing based on input features and weightedly combines their outputs.

[0038] Large Language Models (LLMs) refer to deep learning models with large parameters, which are specifically designed for understanding and generating natural language.

[0039] Reinforcement learning (RL) is a machine learning method that uses an agent to continuously experiment and interact with an environment, learning how to make decisions to maximize long-term rewards. The agent is the subject that performs actions; the environment is the external system or task scenario in which the agent resides; the state is data describing the current state of the environment; the action is the action the agent can perform; the reward is the feedback signal from the environment after the agent performs an action; and the policy is the rule by which the agent chooses an action based on the state. The goal of reinforcement learning is to learn an optimal policy that maximizes cumulative rewards.

[0040] Transformer is a deep learning model architecture based on the self-attention mechanism.

[0041] Multilayer Perceptron (MLP) is a feedforward artificial neural network model consisting of multiple layers of nodes (or neurons), including an input layer, at least one hidden layer, and an output layer.

[0042] Proximal Policy Optimization (PPO) is a policy gradient method widely used in the field of reinforcement learning, which aims to solve the training instability and inefficiency problems in policy gradient algorithms.

[0043] Prioritized Experience Replay (PER) is an experience utilization mechanism in reinforcement learning. Its core idea is to assign different sampling priorities based on the importance of experience, and give priority to replaying experience data that is more helpful for strategy updates.

[0044] Offline experience replay (ErReplay) is a commonly used technique in reinforcement learning, used to improve data utilization and training stability. Its principle is that when an agent interacts with the environment, it stores each experience (i.e., state, action, reward, and next state) in a data structure called an "experience pool" or "replay buffer." During the learning process, the agent randomly samples a batch of data from the experience pool for learning, rather than relying solely on the most recent experience. This sampled data is used to update the agent's value function or policy function. Its advantage is that through random sampling, offline experience replay breaks the temporal correlation between data, making learning more stable. Each data point can be used multiple times, improving data utilization efficiency.

[0045] The embodiments of the present application provide a data processing method, and the method is described in detail in conjunction with the execution flow of the data processing method.

[0046] Figure 1 A flow chart of a data processing method provided in an embodiment of the present application specifically includes the following steps: Figure 1 The following steps are shown:

[0047] The data processing method provided in this application is executed by a data processing system, which includes four parts: a dynamic load sensing module, a reinforcement learning strategy module, an expert resource pool, and a load balancing executor. The dynamic load sensing module is used to collect multi-dimensional load data of each expert network in real time. The multi-dimensional load data includes computing resource occupancy indicators, task queue status indicators, and target load data in expert performance evaluation indicators. The computing resource occupancy indicators reflect the consumption of hardware resources by the expert network during task execution, such as GPU / CPU utilization, video memory / memory occupancy, I / O data transmission rate, etc. The task queue status indicators are used to measure the current task processing pressure and response capabilities of the expert network, such as the current number of tasks to be processed, the average task processing delay, the task completion rate, etc. The expert performance evaluation indicators characterize the historical performance and task adaptability of the expert network, such as task processing accuracy, task type preference distribution, historical scheduling success rate, etc.

[0048] The reinforcement learning strategy module includes a state space (State), an action space (Action), and a reward function (Reward). The state space consists of the expert load feature vector and the current task feature vector. The action space is used to define the task allocation strategy, that is, to assign an expert network to each task. The reward function is used to determine the weighted reward based on the load balance (for example, the difference in tasks processed by the expert network) and model performance (for example, the task processing accuracy of the expert network).

[0049] The expert resource pool consists of multiple expert networks and supports the integration of heterogeneous model architectures, such as different types of deep learning networks including convolutional neural networks and attention networks. In addition, each expert network can be configured with differentiated hardware resources based on its model complexity and computing requirements, such as the number of GPU cores, memory usage limit, batch size limit, etc., to achieve efficient processing of different types of tasks and optimized resource utilization.

[0050] The load balancing executor is used to dynamically allocate tasks to the target expert network according to the task allocation results output by the reinforcement learning strategy module, monitor the execution status of the target expert network, and feed back the monitoring information to the load perception module.

[0051] S101: Obtain target data to be processed, perform feature extraction on the target data, and generate a first feature vector.

[0052] It can be understood that the target data is the input data of the hybrid expert model, which can be large-scale multimodal data. The number of target data is not limited. For example, the target data is mixed data of text and images, etc. The target data is the task to be assigned to the expert network for data processing. Subsequently, the target data is feature extracted to generate the first feature vector. The first feature vector can be understood as the task feature vector. Specifically, the first feature vector can be generated by a single-layer perception network. The first feature vector is recorded as , where x is the target data, d is the feature dimension, and the first feature vector includes features such as semantic features and complexity features captured from the target data. In addition, a third feature vector of the task type label can be added to the first feature vector. The task type label is used to characterize the processing method of the target data, for example, recognition processing, segmentation processing, fuzzy processing, etc. of the target data. Specifically, the task type label is encoded to obtain the third feature vector. Subsequently, the third feature vector and the first feature vector are spliced ​​to output the task feature vector (that is, the updated first feature vector). The task feature vector is recorded as , where k is the number of task type labels.

[0053] S102: Generate a second feature vector according to the acquired target load data of at least one data processing network.

[0054] The target load data includes historical information of the data processing network.

[0055] It is understandable that, based on the above S101, the load data of at least one data processing network is collected in real time to obtain target load data, wherein the data processing network refers to an expert network, and the data processing network can be an expert network pre-selected from multiple preset expert networks included in the hybrid expert model. In other words, at least one data processing network can be pre-selected from multiple preset expert networks based on the actual processing requirements of the target data. The target load data can be the load data of a single data processing network, or the comprehensive load data of at least one data processing network. The load data includes historical information of the data processing network, reflecting the historical load conditions of the data processing network when processing historical data. Subsequently, feature extraction is performed on the target load data to generate a second feature vector, wherein the processing order of the first feature vector and the second feature vector and the network for feature extraction are not limited.

[0056] The target load data includes at least one of computing resource occupancy, processed data volume, and average processing delay. The processed data volume refers to the historical data volume processed by the data processing network within a historical period.

[0057] It is understandable that the target load data includes at least one load data, wherein the computing resource occupancy rate is recorded as The amount of processed data refers to the amount of historical data processed or to be processed by the data processing network in the historical time. The amount of processed data can also be understood as the length of the task queue assigned to the data processing network. The amount of processed data is recorded as , the average processing delay refers to the average processing time or average delay time of all tasks processed by the data processing network. The average processing delay is recorded as .in, is the i-th data processing network, N is the number of data processing networks, is the kth task.

[0058] Optionally, generating a second characteristic vector according to the acquired target load data of at least one data processing network can be specifically implemented by the following steps:

[0059] Normalizing target load data of at least one data processing network; extracting features of the at least one normalized load data to generate at least one load feature vector; concatenating the at least one load feature vector to obtain a second feature vector; or, adding a preference feature vector to the at least one load feature vector and concatenating the two vectors to obtain a second feature vector, wherein the preference feature vector is obtained by extracting features of preference information of a data type processed by the at least one data processing network.

[0060] It is understandable that, when the target load data is the load data of a single data processing network, each load data is normalized. Subsequently, the features of each load data after normalization are extracted to generate a load feature vector for each load data. The load feature vectors of each load data are concatenated to generate a second feature vector, which can be understood as an expert load feature vector. Each load data is recorded as , N data processing networks constitute the second eigenvector, recorded as matrix . In the case where the target load data is the combined load data of multiple data processing networks, feature extraction is directly performed on the combined load data to generate a second feature vector for characterizing the combined load situation. In addition, preference information of the data processing network can be set according to actual processing requirements. The preference information characterizes the specific task types that the data processing network can process. Subsequently, feature extraction is performed on the preference information to generate preference feature information. The corresponding preference feature vector is added to each load feature vector and then spliced ​​to generate a second feature information.

[0061] S103: Input the first eigenvector and the second eigenvector into a pre-built allocation strategy network, and obtain the action probability distribution output by the allocation strategy network.

[0062] The action probability distribution is used to characterize the probability that at least one data processing network is assigned a feature vector.

[0063] It is understood that, based on the above-mentioned S102, the allocation policy network is constructed based on a reinforcement learning algorithm. Before the allocation policy network predicts the action probability distribution based on the first and second eigenvectors, the reinforcement learning environment is initialized. Specifically, the expert network is initialized, for example, by setting the number of expert networks (such as the aforementioned expert resource pool), expert heterogeneity parameters (such as computing power and memory limit), and task queue capacity (i.e., the maximum number of tasks that the expert network can arrange or the maximum amount of data to be processed). Simultaneously, the expert load indicator is initialized, recording time series load data to serve as the expert network load data for the next data processing step. The parameters of the allocation policy network are initialized, for example, by using the PPO algorithm to construct an actor-critic (action-state) dual network structure, separating the learning of the policy and value function into two independent but collaborative neural networks. The actor network (also known as the policy network / action network) takes as input the state vector (the aforementioned eigenvector) and outputs the action probability distribution. The output action specifically refers to the probability distribution of each expert network. The network structure can be a three-layer fully connected network, and the network structure can be customized according to actual needs. The Critic network (also known as the Value Network) takes a state vector as input and outputs an estimate of the state's value. Its structure is consistent with that of the Actor network. The experience replay buffer is initialized, using a prioritized experience replay mechanism and adjusting sampling weights based on temporal difference error.

[0064] It can be understood that after completing the feature extraction of the input task (such as the image in the image classification task) and at least one load data and the above-mentioned initialization, the extracted task feature vector (i.e., the first feature vector) and the load feature vector (i.e., the second feature vector) are spliced ​​to form a comprehensive state vector, and the state vector is passed as input to the reinforcement learning-based policy network (such as the Actor network using the PPO algorithm mentioned above), which outputs the selection probability corresponding to each data processing network to generate an action probability distribution, which is used to guide the task allocation decision.

[0065] Optionally, before inputting the first eigenvector and the second eigenvector into a pre-built allocation strategy network and obtaining the action probability distribution output by the allocation strategy network, the method further includes:

[0066] The first eigenvector and the second eigenvector are flattened and concatenated to obtain the current state vector as the input of the allocation strategy network.

[0067] It can be understood that the first eigenvector and the second eigenvector matrix are flattened and concatenated to form the final state vector (i.e., the current state vector), which is recorded as Then, the current state vector is used as the input of the Actor network, and the Actor network outputs the action probability distribution , and the final action index, weight distribution or probability distribution is obtained through softmax sampling.

[0068] S104 : Determine at least one target processing network in at least one data processing network according to the action probability distribution, and process the first feature vector through the at least one target processing network.

[0069] It is understood that, based on the above S103, the target processing network is selected based on the action probability distribution. Specifically, the first K data processing networks can be used as the target processing network. Task allocation is performed through the load balancing executor, and the feature units in the first feature vector are assigned as tasks to the target processing network. For example, according to action a, the task is assigned to the target expert network a. At the same time, the task processing results (such as accuracy, latency, and other indicators) of the target processing network and the expert load changes are recorded to calculate the reward.

[0070] Optionally, after processing the first feature vector by at least one target processing network, the method further includes:

[0071] Detect overload data of at least one target processing network within a set time period; if an overload rate calculated based on the overload data is greater than a fourth set threshold, add a new data processing network based on the configuration information of the overloaded network recorded in the overload data; or, if the overload rate is less than a fifth set threshold, reduce the remaining data processing networks except the overloaded network, wherein the fifth set threshold is greater than the fourth set threshold.

[0072] It is understandable that the dynamic expansion and contraction of the expert network can also be achieved based on the task processing results. Specifically, the overload data of at least one target processing network within a set time period is detected. The set time period refers to a preset period of time, for example, it can be 24 hours. The target processing network refers to the expert network currently being scheduled for use. The overload data refers to whether the load of all target processing networks within the time period exceeds the upper limit, that is, the number of target processing networks whose data to be processed exceeds the maximum processable data volume is counted. Subsequently, the overload rate is calculated based on the overload data. The overload rate refers to the time ratio, task ratio or number ratio of the target processing network in an overloaded state. For example, the overload rate within 24 hours is 35%, that is, 35 expert networks out of 100 expert networks are overloaded. Determine whether the overload rate is greater than the fourth set threshold, wherein the fourth set threshold is a numerical standard for determining whether multiple target processing networks are frequently overloaded. For example, the fourth set threshold is 30%. If the overload rate exceeds the fourth threshold, for example, an overload rate of 35% exceeds the fourth threshold of 30%, the expert resource pool is automatically expanded, adding a new data processing network. The new data processing network can be a homogeneous expert network, meaning that an overloaded expert network is added. If the overload rate is less than the fifth threshold, for example, an overload rate of 5% is less than the fifth threshold of 10%, a scaling warning is triggered, reducing the number of data processing networks except the overloaded network. Some data processing networks can be removed after user confirmation.

[0073] This application provides a data processing method for load balancing in the MoE model. Reinforcement learning dynamically adjusts the expert network's allocation strategy through online learning, as task traffic and expert load change in real time. Based on the reinforcement learning method, a reward function is set based on the load of each expert network to dynamically adjust the task allocation strategy, optimizing load balancing in real time and reducing the degree of load imbalance. This alleviates the problem of uneven expert load and resource waste caused by static routing, further improving model training efficiency.

[0074] Based on the above embodiments, Figure 2 A flow chart of a data processing method provided in an embodiment of the present application, optionally, after processing the first feature vector by at least one target processing network, the method further includes: Figure 2 The following steps are shown:

[0075] S201: Obtain a processing result after at least one target processing network processes a first feature vector.

[0076] The processing result is used to characterize the execution status of at least one target processing network when processing the first feature vector.

[0077] As can be understood, the target processing network is the expert network or subnetwork currently selected to participate in task processing. The first eigenvector is the feature representation obtained after encoding or extracting the input data. The processing result is the output of the target processing network after calculating, classifying, and identifying the first eigenvector. It is used to characterize the performance of the target processing network during the data processing process, such as response latency, computing resource consumption, and output confidence. After completing task processing, the processing results of each target processing network are collected to evaluate its operating status during task execution based on the processing results.

[0078] S202: Calculate reward items based on the allocation result, the processing result, and a pre-set reward function.

[0079] The allocation result refers to an allocation result of allocating feature units to at least one target processing network, the first feature vector includes a plurality of feature units, and each target processing network is allocated with at least one feature unit.

[0080] It is understood that, based on S201 above, the allocation result refers to the task scheduling mechanism's division of the first feature vector into several feature units and their allocation to different target processing networks for processing. This means it records the target feature units processed by each target processing network and the number of them. A feature unit is the smallest processing unit after the first feature vector is divided into several parts, for example, by dimension or spatial region. The reward function is a predefined function used to evaluate system performance, generating rewards based on the task allocation strategy and execution results. The reward term is a feedback signal in reinforcement learning, used to guide the allocation strategy network to optimize the expert scheduling strategy. Specifically, based on the current task allocation result and the processing results returned by each target processing network, combined with a preset reward function, the system calculates a reward term to evaluate the performance of the current task scheduling strategy. This reward term is also used as the basis for updating the allocation strategy network (e.g., the actor network) to optimize the next round of task allocation strategies, thereby realizing a dynamic and intelligent expert scheduling mechanism.

[0081] The processing result includes processing accuracy and / or actual processing delay, the processing accuracy is calculated by the loss function, and the reward function includes a first function for measuring load balancing and / or a second function for measuring network performance.

[0082] Understandably, the processing accuracy can be calculated by the cross entropy loss function, which is recorded as , the actual processing delay is recorded as Other metrics can also be added to the processing results, such as computing resource utilization. The first function is used to calculate the load balancing reward (i.e., the load balancing item), and the second function is used to calculate the model performance reward (i.e., the model performance item). These other metrics can be used together with the accuracy as reward and penalty conditions and included in the second function to adjust the behavior of the Actor network.

[0083] Optionally, the reward item is calculated based on the allocation result, the processing result, and a pre-set reward function. This can be achieved through the following steps:

[0084] According to the first function, the product of the variance of the number of feature units allocated to at least one target processing network in the allocation result and the preset load balancing coefficient is calculated to obtain the load balancing item; according to the second function, the processing accuracy and / or actual processing delay and the preset model performance coefficient are calculated to obtain the model performance item; based on the load balancing item and the model performance item, the reward item is obtained.

[0085] It can be understood that the calculation of the reward function is mainly divided into the load balancing reward item and the model performance reward item. The calculation formula of the first function is shown in formula (1), the calculation formula of the second function is shown in formula (2), and the calculation formula of the reward item is shown in formula (3).

[0086] Formula (1)

[0087] Where, is the load balancing item, is the load balancing coefficient, which is used to adjust the weight of load balancing. is N feature units, The variance of the number of feature units assigned to N target processing networks.

[0088] Formula (2)

[0089] Where, is the model performance item, is the model performance coefficient, which is used to adjust the weight of accuracy.

[0090] Formula (3)

[0091] Where R is the reward item.

[0092] S203: Adjust the allocation strategy network according to the reward item.

[0093] It is understandable that, based on the above S202, the reinforcement learning mechanism is used to guide the allocation strategy network to learn a better task scheduling strategy based on the calculated reward items. Specifically, the network parameters of the allocation strategy network can be adjusted through gradient updates or other optimization methods to improve the quality of future task allocation strategies.

[0094] The data processing method provided in this application designs a multi-objective reward function, which comprehensively considers indicators such as load balancing, task efficiency, and model performance. At the same time, it combines lightweight state representation (expert load characteristics and task characteristics, etc.) to minimize performance and accuracy losses while avoiding uneven load and reduce computational overhead.

[0095] Based on the above embodiments, Figure 3 A flow chart of a data processing method provided in an embodiment of the present application, after processing the first feature vector by at least one target processing network, the method further includes: Figure 3 The following steps are shown:

[0096] S301. Record the current state, current action, current reward, and next state to obtain an experience tuple.

[0097] Among them, the current state refers to the state of the environment before the allocation strategy network outputs the current action, the current action refers to the action to be executed based on the output of the allocation strategy network based on the current state, the current action corresponds to an action probability, the current reward refers to the reward given by the environment after executing the current action, and the next state refers to the state of the environment after executing the current action.

[0098] It can be understood that the current state refers to the state of the environment perceived by the allocation policy network before it outputs the current action, which is usually composed of the concatenation of the task feature vector and the expert load feature vector, that is, the current state vector mentioned above. The current action is the task allocation decision made by the allocation policy network based on the current state, for example, selecting an expert network to process a feature unit. The action probability represents the probability of taking the action in the current state, and is the output of the allocation policy network, that is, the probability of assigning feature units to different expert networks. For example, feature unit 1 is assigned to expert network 1 according to the probability. The current reward is the feedback signal calculated by the system based on the task execution effect after executing the action (expert network 1 processes feature unit 1), which is used to evaluate the quality of the action, that is, to evaluate whether expert network 1 is suitable for processing feature unit 1. The next state is the new state of the environment after the current action is executed, reflecting the change in expert load or the update of task progress. In the process of interaction between the reinforcement learning agent and the environment, the interaction result of each time step is recorded as an experience tuple for storage, usually expressed as , where s is the current state, a is the current action, and R is the current reward. is the next state, that is, the state of the environment after the current action a is executed.

[0099] S302. Calculate the advantage function based on the experience tuple.

[0100] Among them, the advantage function is used to measure the performance of the current action in the current state.

[0101] As can be understood, based on S301 above, the experience tuple represents a complete interaction between the agent and the environment. The advantage function is used to measure the relative merits of the current action taken relative to the average strategy in the current state. The specific calculation process for the advantage function based on the experience tuple is described in the following embodiments.

[0102] As you can understand, temporal difference error (TD-error) is a metric used in reinforcement learning to measure the difference between predicted and actual values. It is primarily used to update estimates of the advantage function, or action-value function. In reinforcement learning, an agent learns how to choose actions to maximize cumulative rewards by interacting with the environment. The advantage function estimates the expected cumulative reward that the agent can obtain in a given state, and TD-error is used to measure the accuracy of this estimate and make updates accordingly.

[0103] Optionally, the advantage function is calculated based on the experience tuple, which can be achieved by the following steps:

[0104] Calculate the initial state value function and the final state value function, where the initial state value function is used to represent the expected value that can be obtained in the current state, and the final state value function is used to represent the evaluation value in the next state; calculate the advantage function based on the current reward, the initial state value function and the final state value function.

[0105] It can be understood that the initial state value function represents the expected long-term return that can be obtained based on the current strategy in the current state, and the final state value function represents the expected return in the next state after executing the action. For example, the critic network can be used to estimate the value of the current state and the next state respectively. The specific calculation process is not described in detail. The advantage function is used to measure the quality of the current action relative to the average strategy in the current state. It is used to guide the update of the strategy network, thereby optimizing the task scheduling strategy (i.e., the current strategy in the allocation strategy network) and improving the reasoning efficiency and load balancing capability of the MoE model. The calculation process of the advantage function is shown in Formula (4).

[0106] Formula (4)

[0107] Where A is the advantage function, R is the cumulative reward observed during the above intermediate behavior (that is, the current reward), is the initial state value function, is the final state value function of the evaluation.

[0108] It is understandable that the advantage estimate reflects the relative merits of the actions taken by the evaluator in a given state. If the actual cumulative reward is higher than expected, the advantage value is positive, indicating that the action is better than the average level of the current strategy, and the system will be more inclined to select this action in this state, that is, it will be more inclined to have a specific expert network process a specific feature unit; conversely, if the actual reward is lower than expected, the advantage value is negative, indicating that the action performs poorly, and the system will reduce the probability of selecting the action again, that is, the effect of expert network 1 processing feature unit 1 is relatively poor, and feature unit 1 will be allocated to other expert networks for processing next time.

[0109] S303: Update the network parameters of the allocation strategy network based on the advantage function.

[0110] As can be understood, after calculating the advantage function in S302 above, the network parameters of the allocation strategy network are updated based on the advantage function. Specifically, the parameters of the actor network and / or the critic network may be updated to achieve dynamic task allocation and load balancing optimization in the MoE model. The specific update process is described in the following embodiment.

[0111] Optionally, the network parameters of the allocation strategy network are updated based on the advantage function, which can be achieved through the following steps:

[0112] Based on the current strategy of the allocation strategy network, the first probability distribution of the current action in the current state is calculated, wherein the current strategy is the allocation strategy adopted by the allocation strategy network to output the current action; based on the optimization strategy of the allocation strategy network, the second probability distribution of the current action in the current state is calculated; the action loss function is calculated based on the first probability distribution, the second probability distribution and a pre-set clipping range, wherein the action loss function is used to characterize the difference in the possibility of executing the allocation strategy before and after the optimization of the same action; the value loss function is calculated based on the advantage function, the current reward, the first probability distribution and the second probability distribution; the joint loss function is calculated based on the action loss function, the value loss function, a pre-set entropy function and a pre-set hyperparameter; the network parameters of the allocation strategy network are updated based on the joint loss function.

[0113] It can be understood that the action probability distribution under the current strategy is obtained. The current strategy is the strategy used by the current allocation strategy network to implement task allocation. The current state specifically refers to the current state vector composed of the first eigenvector and the second eigenvector. The current action refers to the specific task allocation decision output based on the current strategy in the current state. The first probability distribution is the probability distribution of selecting the current action in the current state based on the current strategy, that is, the probability of the current strategy selecting the current action. The action probability distribution under the optimization strategy is obtained. The optimization strategy is the updated strategy in the allocation strategy network, which is used for comparison with the current strategy and is often used to calculate the ratio of new and old strategies in algorithms such as PPO. The second probability distribution is the probability distribution of selecting the current action in the current state based on the optimization strategy, that is, the probability of the optimization strategy selecting the current action. It is used to measure the change in the probability of selecting the current action before and after the strategy update. It can also be understood as measuring the probability ratio of the updated strategy to continue to execute the action relative to the previous strategy. It can also be understood as the difference in the probability of strategy execution before and after the same action. An action loss function is constructed based on the first probability distribution (action probability under the current policy), the second probability distribution (action probability under the optimized policy), and a pre-set clipping range to constrain the policy update amplitude. The clipping range is used to limit the policy update amplitude. For example, the clipping range can be defined as 0.2. The action loss function is used to measure the difference in the change in the probability of selecting the same action before and after the policy update, which can prevent the update amplitude from being too large and causing training instability. Specifically, as shown in Formula (5). A value loss function is constructed based on the advantage function, the current reward, the first probability distribution, and the second probability distribution to optimize the state value estimation. The value loss function is used to measure the error in the state value estimation and optimize the Critic network's ability to estimate the state value. Specifically, as shown in Formula (6). The action loss, value loss, and policy entropy are combined to construct a joint loss function. The entropy function is used to measure the exploratory nature of the policy and prevent the policy from converging to the local optimum too early. The hyperparameter is used to balance the coefficients of the weights of each loss term. The joint loss function is an objective function that combines the action loss, value loss, and policy entropy. As the final optimization goal, it guides the network parameter update by unifying multiple optimization goals. The specific calculation formula of the joint loss function is shown in Formula (7).

[0114] Formula (5)

[0115] Where, is the action loss function, that is, the loss function of the Actor network, ,in, is the probability distribution of a under state S, specifically the second probability distribution, is the first probability distribution, In order to trim the range and avoid excessive fluctuations in the strategy, E is the mean square error loss function.

[0116] Formula (6)

[0117] Where, is the value loss function, which is also the loss function of the Critic network.

[0118] Formula (7)

[0119] Where, is the joint loss function, c1 and c2 are hyperparameters used to measure the importance of strategy evaluation and exploration accuracy, respectively. H is the entropy function that encourages exploration to encourage the exploration of the entropy of the strategy output distribution. The standard entropy function is shown in formula (8).

[0120] Formula (8)

[0121] Understandable, is the standard entropy function. After obtaining the joint loss function, the optimizer is used to update the allocation strategy network parameters. Specifically, the Adam optimizer can be used to update the network parameters.

[0122] The data processing method proposed in this application designs a composite reward function that simultaneously considers key metrics such as latency, energy consumption, and computational accuracy. This helps overcome load imbalances, improve performance, and maintain model accuracy. Furthermore, the PPO algorithm uses a pruning mechanism to ensure training stability, further optimizing training results.

[0123] Based on the above embodiments, Figure 4 A flow chart of a data processing method provided in an embodiment of the present application, after determining at least one target processing network in at least one data processing network according to the action probability distribution, the method further includes: Figure 4 The following steps are shown:

[0124] S401: Allocate multiple feature units included in a first feature vector to at least one target processing network.

[0125] It is understandable that the first feature vector includes multiple feature units, and multiple feature units can be divided when extracting features from the target data. Alternatively, taking the target data including multiple images as an example, a feature unit can be extracted from each image, and at least one feature unit can be understood as a task to be assigned to the expert network for processing. Subsequently, the target processing network for the feature unit to be processed is determined according to the action probability distribution, and the feature unit is then assigned to the corresponding target processing network. The number of feature units assigned to each target processing unit is not limited. For example, all feature units extracted from the image are assigned to the first type of target processing network, and all feature units extracted from the text are assigned to the second type of target processing network. That is, multimodal data can be assigned to different target processing networks according to the data type or processing method. The specific allocation is not described in detail.

[0126] Optionally, after allocating the plurality of feature units included in the first feature vector to at least one target processing network, the method further includes:

[0127] Obtaining the current amount of feature units to be processed by at least one target processing network; allocating the feature units to be allocated to the first target processing network to the second target processing network, wherein the first target processing network refers to a data processing network whose feature unit amount reaches a maximum processing amount, and the first target processing network has a higher priority than the second target processing network.

[0128] It is understood that during the process of allocating at least one feature processing unit to a target processing network via the load balancing executor, the data volume of the feature units currently to be processed by the target processing network is obtained in real time, which is the feature unit quantity. If there is a first target processing network in at least one target processing network whose feature unit quantity has reached its maximum processing capacity, the feature units to be allocated to the first target processing network are allocated to the second target processing network, wherein the priority of the first target processing network is higher than that of the second target processing network. In other words, if the task queue of a target processing network is full, the remaining feature units to be allocated to the target processing network are allocated to the suboptimal target processing network (i.e., the second target processing network). The second target processing network and the first target processing network can be understood as similar networks, and the number of second target processing networks is not limited.

[0129] S402: Acquire real-time load data of at least one target processing network.

[0130] The real-time load data includes the total number of target feature units allocated to the target processing network.

[0131] It is understandable that, based on the above S401, real-time load data of the target processing network in processing the feature unit is obtained. The real-time load data includes multiple indicators such as computing resource occupancy, task queue depth, and historical response delay.

[0132] Optionally, after obtaining real-time load data of at least one target processing network, the method further includes:

[0133] If the load rate calculated based on the real-time load data is less than a third set threshold, the resources of at least one target processing network of the first number are merged; or, if the load rate is greater than or equal to the third set threshold, the resources of at least one target processing network of the second number are split so that the second number of target processing networks can process the feature units in parallel.

[0134] It is understandable that the limitations of static expert networks can be reduced based on real-time load data, and the number and network structure of expert networks can be dynamically adjusted. Reinforcement learning can also optimize the number of experts and expert selection at the same time. Specifically, the load rate of each target processing network, or the comprehensive load of all target processing networks, or the class load rate of a certain type of target processing network is calculated based on real-time load data. Subsequently, if the load rate is less than the third set threshold, the resources of a certain number of target processing networks are merged, that is, the expert networks are merged at low loads to reduce computing overhead, such as merging expert networks for image recognition. If the load rate is greater than or equal to the third set threshold, a certain number of target processing networks are split, and the multiple sub-target processing networks after the split process feature units in parallel, that is, the expert network is split at high loads to improve parallelism.

[0135] S403: Calculate the variance of the total number of at least one target processing network, and / or count the number of times at least one target processing network is continuously overloaded.

[0136] The number of consecutive overloads refers to the number of times the target processing network is in an overload state when being assigned feature units.

[0137] As can be understood, based on the above S402, the allocation strategy network is deployed in a real-world scenario to monitor the load fluctuation indicators of the expert network. Specifically, the variance of the target processing network's queue length is calculated, where the queue length refers to the number of feature units to be processed, and / or the number of consecutive overloads of the target processing network is counted. Continuous overload means that the number of feature units to be processed in the target processing network has reached the maximum processing data volume, and no feature units can be allocated to it multiple times in a row, indicating that the target processing network is overflowing.

[0138] S404: If the variance is greater than a first set threshold and / or the number of times is greater than a second set threshold, adjust the allocation strategy network in real time.

[0139] It is understood that, based on the above S403, if the variance exceeds the first set threshold and / or the number of consecutive overloads exceeds the second set threshold, online fine-tuning of the allocation network is triggered. During the adjustment process, the Critic network is frozen, that is, the Critic network gradient is not updated, and only the Actor network parameters are updated (for example, the learning rate is reduced), and training steps are continued for a given number of steps. The first set threshold can be 0.5, and the second set threshold can be 3. This means that if tasks cannot be allocated to the target processing network for more than three consecutive times, there is a task overflow phenomenon.

[0140] This application proposes a data processing method that uses a reinforcement learning-based allocation strategy network to make dynamic decisions based on real-time load data. Through online learning, it can perceive the changing characteristics of task distribution over time and adjust the allocation strategy network in real time, thereby optimizing the uneven load of the expert network and improving model performance.

[0141] Based on the above embodiments, Figure 5 A flow chart of a data processing method provided in an embodiment of the present application specifically includes the following steps: Figure 5 The following steps are shown:

[0142] 1) System architecture and data initialization: Obtain input data and perform data preprocessing and feature extraction on the input data; 2) State representation and action generation: The encoder constructs relevant features of the expert workload, generates task allocation decisions based on reinforcement learning, assigns tasks to the target expert network based on the task allocation decisions, controls the target expert network to execute tasks and obtain processing results, and updates the state and reward at the same time; 3) Policy network optimization: Iteratively optimizes the allocation policy network; 4) Online adaptive adjustment: Model deployment.

[0143] It is understandable that the detailed descriptions of the above 1) to 4) refer to the above embodiments and are not repeated here.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0145] The embodiment of the present application further provides a data processing device, Figure 6 This is a structural diagram of a data processing device provided in an embodiment of the present application. The data processing device 600 includes a first acquisition module 601, a second acquisition module 602, an allocation module 603 and a data processing module 604, wherein:

[0146] A first acquisition module 601 is used to acquire target data to be processed, perform feature extraction on the target data, and generate a first feature vector;

[0147] A second acquisition module 602 is configured to generate a second feature vector based on acquired target load data of at least one data processing network, wherein the target load data includes historical information of the data processing network;

[0148] an allocation module 603 configured to input the first eigenvector and the second eigenvector into a pre-built allocation strategy network and obtain an action probability distribution output by the allocation strategy network, wherein the action probability distribution is used to represent the probability of at least one data processing network being allocated the eigenvector;

[0149] The data processing module 604 is configured to determine at least one target processing network in the at least one data processing network according to the action probability distribution, and process the first feature vector through the at least one target processing network.

[0150] The target load data includes at least one of computing resource occupancy, processed data volume, and average processing delay. The processed data volume refers to the historical data volume processed by the data processing network within a historical period.

[0151] Optionally, the second obtaining module 602 is configured to:

[0152] normalizing target load data of at least one data processing network;

[0153] extracting features of at least one normalized load data to generate at least one load feature vector;

[0154] The second feature vector is obtained by concatenating at least one load feature vector; or, the second feature vector is obtained by adding a preference feature vector to at least one load feature vector and then concatenating the resultant feature vectors, wherein the preference feature vector is obtained by extracting feature information of preference information of a data type processed by at least one data processing network.

[0155] Optionally, the data processing device 600 is further configured to:

[0156] The first eigenvector and the second eigenvector are flattened and concatenated to obtain the current state vector as the input of the allocation strategy network.

[0157] Optionally, the data processing device 600 is further configured to:

[0158] Obtaining a processing result after at least one target processing network processes the first feature vector, wherein the processing result is used to represent an execution status of the at least one target processing network when processing the first feature vector;

[0159] Calculating a reward item based on the allocation result, the processing result, and a pre-set reward function, wherein the allocation result refers to an allocation result of allocating a feature unit to at least one target processing network, the first feature vector includes a plurality of feature units, and each target processing network is allocated at least one feature unit;

[0160] Adjust the allocation strategy network according to the reward term.

[0161] The processing result includes processing accuracy and / or actual processing delay, the processing accuracy is calculated by the loss function, and the reward function includes a first function for measuring load balancing and / or a second function for measuring network performance.

[0162] Optionally, the data processing device 600 is further configured to:

[0163] According to the first function, a product of a variance of the number of characteristic units allocated to at least one target processing network in the allocation result and a preset load balancing coefficient is calculated to obtain a load balancing term;

[0164] Calculating the processing accuracy and / or actual processing delay and a preset model performance coefficient according to the second function to obtain a model performance item;

[0165] Reward items are obtained based on the load balancing item and the model performance item.

[0166] Optionally, the data processing device 600 is further configured to:

[0167] Record the current state, current action, current reward, and next state to obtain an experience tuple. The current state refers to the state of the environment before the allocation strategy network outputs the current action. The current action refers to the action to be executed based on the output of the allocation strategy network. The current action corresponds to an action probability. The current reward refers to the reward given by the environment after executing the current action. The next state refers to the state of the environment after executing the current action.

[0168] Calculate the advantage function based on the experience tuple, where the advantage function is used to measure the performance of the current action in the current state;

[0169] Update the network parameters of the allocation policy network based on the advantage function.

[0170] Optionally, the data processing device 600 is further configured to:

[0171] Calculate the initial state value function and the final state value function, where the initial state value function is used to represent the expected value that can be obtained in the current state, and the final state value function is used to represent the estimated value in the next state;

[0172] Calculate the advantage function based on the current reward, the initial state value function, and the final state value function.

[0173] Optionally, the data processing device 600 is further configured to:

[0174] Based on the current policy of the allocation policy network, calculate the first probability distribution of the current action in the current state, where the current policy is the allocation policy adopted by the allocation policy network to output the current action;

[0175] Based on the optimization strategy of the allocation strategy network, the second probability distribution of the current action in the current state is calculated;

[0176] An action loss function is calculated based on the first probability distribution, the second probability distribution, and a preset clipping range, wherein the action loss function is used to characterize the difference in the likelihood of executing the allocation strategy before and after optimization for the same action;

[0177] The value loss function is calculated based on the advantage function, the current reward, the first probability distribution and the second probability distribution;

[0178] The joint loss function is calculated based on the action loss function, the value loss function, the pre-set entropy function and the pre-set hyperparameters;

[0179] Update the network parameters of the allocation policy network based on the joint loss function.

[0180] Optionally, the data processing device 600 is further configured to:

[0181] assigning a plurality of feature units included in the first feature vector to at least one target processing network;

[0182] acquiring real-time load data of at least one target processing network, wherein the real-time load data includes a total number of target characteristic units allocated to the target processing network;

[0183] Calculating the variance of the total number of at least one target processing network and / or counting the number of consecutive overloads of at least one target processing network, wherein the number of consecutive overloads refers to the number of consecutive times that the target processing network is in an overload state when being assigned a feature unit;

[0184] If the variance is greater than a first set threshold and / or the number of times is greater than a second set threshold, the allocation strategy network is adjusted in real time.

[0185] Optionally, the data processing device 600 is further configured to:

[0186] If the load rate calculated based on the real-time load data is less than a third set threshold, merging resources of at least one target processing network of the first number; or,

[0187] If the load rate is greater than or equal to a third set threshold, splitting a second number of at least one target processing network so that the second number of target processing networks process the feature units in parallel.

[0188] Optionally, the data processing device 600 is further configured to:

[0189] Obtaining a number of feature units currently to be processed by at least one target processing network;

[0190] The feature units to be allocated to the first target processing network are allocated to the second target processing network, wherein the first target processing network refers to a data processing network whose feature unit quantity reaches the maximum processing quantity, and the priority of the first target processing network is higher than that of the second target processing network.

[0191] Optionally, the data processing device 600 is further configured to:

[0192] detecting overload data of at least one target processing network within a set time period;

[0193] If the overload rate calculated based on the overload data is greater than a fourth set threshold, a new data processing network is added based on the configuration information of the overloaded network recorded in the overload data; or

[0194] If the overload rate is less than a fifth set threshold, the remaining data processing networks except the overloaded network are reduced, wherein the fifth set threshold is greater than the fourth set threshold.

[0195] For the description of the features in the embodiments corresponding to the data processing device, reference can be made to the relevant description of the embodiments corresponding to the data processing method, and no further details will be given here.

[0196] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above data processing method embodiments.

[0197] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data processing method embodiments when run.

[0198] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0199] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above data processing method embodiments are implemented.

[0200] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data processing method embodiments are implemented.

[0201] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] The above is a detailed introduction to a data processing method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire target data to be processed, perform feature extraction on the target data, and generate a first feature vector, wherein the target data is multimodal data mixed with text and images and is a task to be assigned to an expert network for data processing; generating a second feature vector based on target load data of at least one data processing network obtained, wherein the target load data includes historical information of the data processing network, the target load data including at least one of computing resource occupancy, processed data volume, and average processing delay, the processed data volume being a historical amount of data processed by the data processing network within a historical period; and the data processing network is an expert network; Inputting the first feature vector and the second feature vector into a pre-built reinforcement learning-based allocation strategy network, and obtaining an action probability distribution output by the allocation strategy network, wherein the action probability distribution is used to represent the probability of the at least one data processing network being assigned a feature vector; determining at least one target processing network in the at least one data processing network according to the action probability distribution, and processing the first feature vector by the at least one target processing network to calculate a reward according to a task processing result of the target processing network and real-time load data, and adjusting the allocation strategy network; After processing the first feature vector by the at least one target processing network, the method further includes: Record the current state, current action, current reward, and next state to obtain an experience tuple, wherein the current state refers to the state of the environment before the allocation strategy network outputs the current action, the current action refers to the action to be executed output by the allocation strategy network based on the current state, the current action corresponds to an action probability, the current reward refers to the reward given by the environment after executing the current action, and the next state refers to the state of the environment after executing the current action; Calculating an advantage function based on the experience tuple, wherein the advantage function is used to measure the performance of the current action under the current state; The network parameters of the allocation strategy network are updated based on the advantage function.

2. The method according to claim 1, characterized in that Generating a second feature vector according to the acquired target load data of at least one data processing network includes: normalizing target load data of the at least one data processing network; extracting features of at least one normalized load data to generate at least one load feature vector; The at least one load feature vector is concatenated to obtain a second feature vector; or, a preference feature vector is added to the at least one load feature vector and then concatenated to obtain the second feature vector, wherein the preference feature vector is obtained by feature extraction of preference information of a data type processed by the at least one data processing network.

3. The method according to claim 1, characterized in that Before inputting the first feature vector and the second feature vector into a pre-built allocation strategy network and obtaining the action probability distribution output by the allocation strategy network, the method further includes: The first eigenvector and the second eigenvector are subjected to matrix flattening and concatenation processing to obtain a current state vector as an input of the allocation strategy network.

4. The method according to claim 1, wherein After processing the first feature vector by the at least one target processing network, the method further includes: Obtaining a processing result after the at least one target processing network processes the first feature vector, wherein the processing result is used to represent an execution status of the at least one target processing network when processing the first feature vector; Calculating a reward item based on an allocation result, the processing result, and a pre-set reward function, wherein the allocation result refers to an allocation result of allocating a feature unit to the at least one target processing network, the first feature vector includes a plurality of feature units, and each target processing network is allocated at least one feature unit; The allocation strategy network is adjusted according to the reward item.

5. The method according to claim 4, characterized in that The processing result includes a processing accuracy rate and / or an actual processing delay, wherein the processing accuracy rate is calculated by a loss function. The reward function includes a first function for measuring load balancing and / or a second function for measuring network performance. The reward item is calculated based on the allocation result, the processing result, and a preset reward function, including: According to the first function, calculating the product of the variance of the number of feature units allocated to the at least one target processing network in the allocation result and a preset load balancing coefficient to obtain a load balancing term; Calculating the processing accuracy and / or the actual processing delay and a preset model performance coefficient according to the second function to obtain a model performance item; The reward item is obtained according to the load balancing item and the model performance item.

6. The method according to claim 1, characterized in that The calculating of the advantage function according to the experience tuple comprises: Calculating an initial state value function and a final state value function, wherein the initial state value function is used to represent the expected value that can be obtained in the current state, and the final state value function is used to represent the estimated value in the next state; An advantage function is calculated based on the current reward, the initial state value function, and the final state value function.

7. The method according to claim 1, characterized in that The updating of the network parameters of the allocation strategy network based on the advantage function includes: Calculating a first probability distribution of the current action in the current state based on a current policy of the allocation policy network, wherein the current policy is the allocation policy adopted by the allocation policy network to output the current action; Calculating a second probability distribution of the current action in the current state based on the optimization strategy of the allocation strategy network; An action loss function is calculated based on the first probability distribution, the second probability distribution, and a preset clipping range, wherein the action loss function is used to characterize the difference in the likelihood of executing the allocation strategy before and after optimization for the same action; Calculating a value loss function based on the advantage function, the current reward, the first probability distribution, and the second probability distribution; Calculating a joint loss function according to the action loss function, the value loss function, a preset entropy function and preset hyperparameters; Network parameters of the allocation strategy network are updated based on the joint loss function.

8. The method according to claim 1, characterized in that After determining at least one target processing network in the at least one data processing network according to the action probability distribution, the method further includes: assigning a plurality of feature units included in the first feature vector to the at least one target processing network; Acquiring real-time load data of the at least one target processing network, wherein the real-time load data includes a total number of target feature units allocated to the target processing network; Calculating the variance of the total number of at least one target processing network and / or counting the number of consecutive overloads of at least one target processing network, wherein the number of consecutive overloads refers to the number of consecutive times that the target processing network is in an overload state when being assigned a feature unit; If the variance is greater than a first set threshold and / or the number of times is greater than a second set threshold, the allocation strategy network is adjusted in real time.

9. The method according to claim 8, characterized in that After acquiring the real-time load data of the at least one target processing network, the method further includes: If the load rate calculated according to the real-time load data is less than a third set threshold, merging a first number of resources of the at least one target processing network; or, If the load rate is greater than or equal to the third set threshold, splitting a second number of the at least one target processing network so that the second number of target processing networks process feature units in parallel.

10. The method according to claim 8, characterized in that After allocating the plurality of feature units included in the first feature vector to the at least one target processing network, the method further includes: Obtaining a current number of feature units to be processed in the at least one target processing network; The feature units to be allocated to the first target processing network are allocated to the second target processing network, wherein the first target processing network refers to a data processing network whose feature unit quantity reaches the maximum processing quantity, and the priority of the first target processing network is higher than that of the second target processing network.

11. The method according to claim 1, wherein After processing the first feature vector by the at least one target processing network, the method further includes: detecting overload data of the at least one target processing network within a set time period; If the overload rate calculated according to the overload data is greater than a fourth set threshold, adding a new data processing network according to the configuration information of the overloaded network recorded in the overload data; or If the overload rate is less than a fifth set threshold, the remaining data processing networks except the overloaded network are reduced, wherein the fifth set threshold is greater than the fourth set threshold.

12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 11.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 11 are implemented.

Images