Data processing method and device, storage medium and program product

Abstract

The invention discloses a data processing method and device, a storage medium and a program product, and relates to the technical field of computers, and the method comprises the steps: obtaining load data of a plurality of data processing networks, and taking a feature vector of to-be-processed data and a feature vector of the load data as the input of a pre-constructed distribution strategy network, the action probability distribution of the multiple data processing networks is predicted, and the target processing network to be subjected to data processing is determined according to the action probability distribution, so that the technical problem of load imbalance of the multiple data processing networks is solved, and the technical effects of improving data processing efficiency and reducing hardware resource waste and deployment cost are achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technologies, and in particular, to a data processing method, device, storage medium, and program product. Background Art

[0002] With the continuous growth of the scale of deep learning models, the Mixture of Experts (MoE) has received extensive attention due to its ability to efficiently expand the model capacity. MoE processes complex tasks by introducing multiple expert networks and a gating network, aiming to enable each expert network to focus on specific types of input data, thereby improving the overall performance without significantly increasing the computational cost.

[0003] However, in practical applications, a major challenge faced by MoE is the unbalanced load of expert networks. Currently, MoE mainly relies on a single gating network to allocate input data to different expert networks for processing, which often leads to some expert networks being overloaded due to frequent selection, while other expert networks are idle for a long time. This unbalanced load phenomenon not only reduces the data processing efficiency but also may cause waste of hardware resources and an increase in deployment costs. Summary of the Invention

[0004] The present application provides a data processing method, device, storage medium, and program product to at least solve the problem of unbalanced load of expert networks in related technologies.

[0005] The present application provides a data processing method, including: Obtaining target data to be processed, and performing feature extraction on the target data to generate a first feature vector; Generating a second feature vector according to the target load data of at least one data processing network obtained, where the target load data includes the historical information of the data processing network; Inputting the first feature vector and the second feature vector into a pre-constructed allocation policy network, and obtaining the action probability distribution output by the allocation policy network, where the action probability distribution is used to represent the probability that at least one data processing network is allocated the feature vector; Determining at least one target processing network from at least one data processing network according to the action probability distribution, and processing the first feature vector through the at least one target processing network.

[0006] The present application also provides a data processing device, including: A first acquisition module, configured to obtain target data to be processed, and perform feature extraction on the target data to generate a first feature vector; A second acquisition module, configured to generate a second feature vector according to the acquired target load data of at least one data processing network, where the target load data includes the historical information of the data processing network; An allocation module, configured to input the first feature vector and the second feature vector into a pre-constructed allocation policy network, and obtain the action probability distribution output by the allocation policy network, where the action probability distribution is used to represent the probability that at least one data processing network is allocated a feature vector; A data processing module, configured to determine at least one target processing network from at least one data processing network according to the action probability distribution, and process the first feature vector through the at least one target processing network.

[0007] This application also provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above data processing methods when executing the computer program.

[0008] This application also provides a computer-readable storage medium, in which a computer program is stored, where the computer program implements the steps of any one of the above data processing methods when executed by a processor.

[0009] This application also provides a computer program product, including a computer program, where the computer program implements the steps of any one of the above data processing methods when executed by a processor.

[0010] Through this application, the load data of multiple data processing networks is acquired, and the feature vector of the data to be processed and the feature vector of the load data are used as the input of a pre-constructed allocation policy network to predict the action probability distribution of multiple data processing networks, and the target processing network to be subjected to data processing is determined according to the action probability distribution. Therefore, the technical problem of unbalanced loads of multiple data processing networks is solved, and the technical effects of improving data processing efficiency, reducing waste of hardware resources, and deployment costs are achieved. Description of the Drawings

[0011] To more clearly illustrate the embodiments of this application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0012] Figure 1 It is a schematic flowchart of a data processing method provided by an embodiment of this application; Figure 2 It is a schematic flowchart of a data processing method provided by an embodiment of this application; Figure 3Schematic flowchart of a data processing method provided by an embodiment of the present application; Figure 4 Schematic flowchart of a data processing method provided by an embodiment of the present application; Figure 5 Schematic flowchart of a data processing method provided by an embodiment of the present application; Figure 6 Schematic structural diagram of a data processing device provided by an embodiment of the present application. Detailed implementation manners

[0013] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0014] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0016] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends, the specific application environment architecture or specific hardware architecture will be described herein.

[0017] In recent years, with the continuous expansion of the scale of deep learning models, the mixture of experts network has received extensive attention because it can effectively improve the model capacity. The core idea of the mixture of experts network is to classify tasks first and then distribute them to each expert network for processing. Among them, the mixture of experts network is a deep learning model with sparse gate control, mainly composed of a group of expert networks and a gating network. The mixture of experts network introduces multiple expert networks (Experts) and a gating network (Gating Network), distributes input samples to different expert networks for processing, and finally aggregates the outputs of each expert network to obtain the final result. This structure enables each expert network to focus on processing the input data, thereby improving the overall performance of the model and significantly enhancing the model's expressive ability without significantly increasing the computational cost.

[0018] However, the mixture of experts model faces the problem of unbalanced load among expert networks in practical applications. Related mixture of experts models usually adopt a gating network, which is updated together with other parameters during training and is responsible for predicting the actual expert network for processing input data during inference, so as to distribute each task to a specific small model. Each small model can be understood as an expert in solving certain specific problems, so it is called an expert network or an expert model. However, the gating network tends to concentrate the tasks on a few expert networks, resulting in overloading of these expert networks while other expert networks are idle. This unbalanced load phenomenon not only reduces the training efficiency of the model but also may lead to overfitting of some expert networks, thus affecting the generalization performance of the model.

[0019] In addition, to solve the problem of unbalanced expert load in the mixture of experts model, the following typical strategies have been proposed in related methods: 1) By setting the maximum upper limit of the number of tasks that each expert model can handle, a hard constraint is imposed on the task allocation of a single expert network to prevent it from affecting the overall performance due to overload. However, this method has a capacity limitation, requires manual setting of the capacity threshold, and is difficult to adapt to changes in different tasks and data distributions, with poor flexibility.

[0020] 2) According to preset fixed rules (such as round-robin mechanism, classification by category, etc.), tasks are distributed to specific expert networks for processing. Although this method can achieve uniform distribution of tasks to a certain extent, it lacks the ability to dynamically respond to task characteristics and the real-time load status of expert networks and is difficult to adapt to complex and changing application scenarios.

[0021] To address the above technical problems, the present application provides a data processing method. By applying a reinforcement learning algorithm, according to the real-time load conditions of each expert network, a reward function is set to automatically learn and optimize the task allocation strategy, thereby effectively alleviating the load imbalance problem in the mixture of experts model and further improving the efficiency of the model during training and its generalization ability on unknown data.

[0022] Before elaborating on the present application in detail, technical terms involved are first explained, where: The mixture of experts model (MOE) is a machine learning architecture composed of multiple sub-models (referred to as "experts") and a gating mechanism. The experts are good at handling different types of input data or tasks; the gating mechanism dynamically selects one or more experts for processing according to the input features and combines their outputs with weights.

[0023] Large Language Models (LLMs) refer to deep learning models with a large number of parameters, specifically designed to understand and generate natural language.

[0024] Reinforcement Learning (RL) is a machine learning method where an agent continuously tries and interacts with the environment to learn how to make decisions to maximize long-term rewards. The agent is the entity that executes actions; the environment is the external system or task scenario where the agent is located; the state is the data describing the current situation of the environment; the action is the operation that the agent can execute; the reward is the feedback signal from the environment to the agent after the agent executes an action; the policy is the rule by which the agent selects actions according to the state; the goal of reinforcement learning is to obtain an optimal policy through learning to maximize the cumulative reward.

[0025] Transformer is a deep learning model architecture based on the self-attention mechanism.

[0026] The Multilayer Perceptron (MLP) is a feedforward artificial neural network model composed of multiple layers of nodes (or neurons), including an input layer, at least one hidden layer, and an output layer.

[0027] Proximal Policy Optimization (PPO) is a widely used policy gradient method in the field of reinforcement learning, aiming to solve the problems of training instability and inefficiency in policy gradient algorithms.

[0028] Prioritized Experience Replay (PER) is an experience utilization mechanism in reinforcement learning. Its core idea is to assign different sampling priorities according to the importance of experiences, and preferentially replay those experience data that are more helpful for policy update.

[0029] The offline Experience Replay mechanism is a commonly used technique in reinforcement learning, which is used to improve data utilization and training stability. Its principle is that when the agent interacts with the environment, it stores each experience (i.e., state, action, reward, and next state) in a data structure called an "experience pool" or "replay buffer". During the learning process, the agent randomly samples a batch of data from the experience pool for learning, rather than just using the latest experience. The extracted data is used to update the value function or policy function of the agent. Its advantage is that through random sampling, the offline Experience Replay breaks the temporal correlation between data, making the learning more stable. Each data can be used multiple times, improving the data utilization efficiency.

[0030] The embodiments of the present application provide a data processing method, which is described in detail in combination with the execution process of the data processing method.

[0031] Figure 1 It is a schematic flowchart of a data processing method provided by the embodiments of the present application, specifically including the following steps as Figure 1 shown: The data processing method provided by the present application is executed by a data processing system, which includes a dynamic load awareness module, a reinforcement learning policy module, an expert resource pool, and a load balancing executor. Among them, the dynamic load awareness module is used to collect multi-dimensional load data of each expert network in real time. The multi-dimensional load data includes target load data such as computing resource occupancy metrics, task queue status metrics, and expert performance evaluation metrics. The computing resource occupancy metrics reflect the consumption of hardware resources by the expert network during task execution, such as GPU / CPU utilization rate, video memory / memory occupancy, I / O data transfer rate, etc.; the task queue status metrics are used to measure the current task processing pressure and response ability of the expert network, such as the number of current tasks to be processed, the average task processing delay, the task completion rate, etc.; the expert performance evaluation metrics characterize the historical performance and task adaptability of the expert network, such as task processing accuracy, task type preference distribution, historical scheduling success rate, etc.

[0032] The reinforcement learning policy module includes a state space (State), an action space (Action), and a reward function (Reward). Among them, the state space is composed of an expert load feature vector and a current task feature vector. The action space is used to define the task allocation policy, that is, to allocate an expert network for each task. The reward function is used to determine the weighted reward according to the load balance degree (for example, the task difference processed by the expert network) and the model performance (for example, the task processing accuracy of the expert network).

[0033] The expert resource pool consists of multiple expert networks, supporting the integration of heterogeneous model architectures. For example, it includes different types of deep learning networks such as convolutional neural networks and attention networks. In addition, each expert network can be configured with different hardware resources according to its model complexity and computing requirements, such as the number of GPU cores, the upper limit of memory usage, and the batch size limit, etc., to achieve efficient processing of different types of tasks and optimal utilization of resources.

[0034] The load balancer is used to dynamically allocate tasks to the target expert network according to the task allocation result output by the reinforcement learning policy module, monitor the execution status of the target expert network, and feedback the monitoring information to the load awareness module.

[0035] S101. Obtain the target data to be processed, and perform feature extraction on the target data to generate a first feature vector.

[0036] Understandably, the target data is the input data of the mixture of experts model, which can specifically be large-scale multi-modal data. The quantity of the target data is not limited. For example, the target data is text, image mixed data, etc. The target data is also the task to be allocated to the expert network for data processing. Subsequently, perform feature extraction on the target data to generate a first feature vector. The first feature vector can be understood as a task feature vector. Specifically, the first feature vector can be generated by a single-layer perceptron network, and the first feature vector is denoted as , where x is the target data, d is the feature dimension, and the first feature vector includes features such as semantic features and complexity features captured from the target data. In addition, a third feature vector with task type labels can also be added to the first feature vector. The task type label is used to characterize the processing method of the target data. For example, perform recognition processing, segmentation processing, blurring processing, etc. on the target data. Specifically, encode the task type label to obtain a third feature vector. Subsequently, splice the third feature vector and the first feature vector to output a task feature vector (i.e., the updated first feature vector), and the task feature vector is denoted as , where k is the number of types of task type labels.

[0037] S102. Generate a second feature vector according to the target load data of at least one data processing network obtained.

[0038] Among them, the target load data includes the historical information of the data processing network.

[0039] Understandably, based on the above S101, the load data of at least one data processing network is collected in real time to obtain the target load data. Among them, the data processing network refers to the expert network, and the data processing network can be an expert network pre-selected from multiple preset expert networks included in the mixture of experts model. That is to say, at least one data processing network can be pre-selected from multiple preset expert networks according to the actual processing requirements of the target data. The target load data can be the load data of a single data processing network or the comprehensive load data of at least one data processing network. The load data includes the historical information of the data processing network, reflecting the historical load situation of the data processing network when processing historical data. Subsequently, feature extraction is performed on the target load data to generate a second feature vector. Among them, the processing order of the first feature vector and the second feature vector and the network for performing feature extraction are not limited.

[0040] Among them, the target load data includes at least one of the data such as the computing resource occupancy rate, the amount of processed data, and the average processing delay. The amount of processed data refers to the amount of historical data processed by the data processing network within the historical time.

[0041] Understandably, the target load data includes at least one type of load data. Among them, the computing resource occupancy rate is denoted as , the amount of processed data refers to the quantity of historical data processed or to be processed by the data processing network within the historical time. The amount of processed data can also be understood as the length of the task queue allocated to the data processing network, and the amount of processed data is denoted as , the average processing delay refers to the average processing time or the average delay time for the data processing network to process all tasks, and the average processing delay is denoted as . Among them, is the i-th data processing network, N is the number of data processing networks, is the k-th task.

[0042] Optionally, according to the obtained target load data of at least one data processing network, a second feature vector is generated, which can be specifically implemented through the following steps: Perform normalization processing on the target load data of at least one data processing network; extract the features of at least one load data after normalization processing to generate at least one load feature vector; splice at least one load feature vector to obtain a second feature vector; or, add a preference feature vector to at least one load feature vector and then splice to obtain a second feature vector, where the preference feature vector is obtained by performing feature extraction on the preference information of the data types processed by at least one data processing network.

[0043] Understandably, in the case where the target load data is the load data of a single data processing network, each type of load data is normalized. Subsequently, the features of each type of load data after normalization are extracted to generate a load feature vector for each type of load data. The load feature vectors of each type of load data are concatenated to generate a second feature vector, which can be understood as an expert load feature vector, and each type of load data is denoted as , and N data processing networks form a second feature vector, denoted as a matrix . In the case where the target load data is the comprehensive load data of multiple data processing networks, feature extraction is directly performed on the comprehensive load data to generate a second feature vector for characterizing the comprehensive load situation. In addition, the preference information of the data processing network can also be set according to actual processing requirements. The preference information characterizes the specific task types that the data processing network can handle. Subsequently, feature extraction is performed on the preference information to generate preference feature information. After adding the corresponding preference feature vectors to each load feature vector, they are concatenated to generate a second feature information.

[0044] S103. Input the first feature vector and the second feature vector into a pre-constructed allocation policy network, and obtain the action probability distribution output by the allocation policy network.

[0045] Among them, the action probability distribution is used to characterize the probability that at least one data processing network is assigned a feature vector.

[0046] It is understandable that, based on the above S102, the allocation policy network is a network constructed based on the reinforcement learning algorithm. Before predicting the action probability distribution according to the first feature vector and the second feature vector through the allocation policy network, the reinforcement learning environment is initialized. Specifically, the expert network is initialized. For example, the number of expert networks (such as the above-mentioned expert resource pool), the expert heterogeneity parameters (such as computing power, memory limit), and the task queue capacity (that is, the maximum number of arranged tasks or the maximum amount of data to be processed by the expert network) are set, etc. At the same time, the expert load metrics are initialized, and the load data in the time series is recorded to be used as the load data of the expert network when the next data is processed. The parameters of the allocation policy network are initialized. For example, the PPO algorithm is adopted to construct an Actor-Critic (action-state) dual network structure, that is, the learning of the policy and the value function is separated into two independent but cooperative neural networks. Among them, the input of the Actor network (also known as the policy network / action network) is the state vector (the above-mentioned feature vector), and the output is the action probability distribution. The specific actions output refer to the probability distribution of each expert network. The network structure can be a 3-layer fully connected network, and the network structure can be configured according to actual needs. The input of the Critic network (also known as the value network) is the state vector, and the output is the state value estimation. The network structure is the same as that of the Actor network. The experience replay buffer is initialized, and the prioritized experience replay mechanism is adopted to adjust the sampling weights according to the temporal difference error.

[0047] It is understandable that after the feature extraction of the input task (such as the image in the image classification task) and at least one load data and the above initialization are completed, the extracted task feature vector (that is, the first feature vector) is concatenated with the load feature vector (that is, the second feature vector) to form a comprehensive state vector, and this state vector is used as the input and passed to the policy network based on reinforcement learning (such as the Actor network using the PPO algorithm mentioned above). It outputs the selection probability corresponding to each data processing network, generates the action probability distribution, and the action probability distribution is used to guide the task allocation decision.

[0048] Optionally, before inputting the first feature vector and the second feature vector into the pre-constructed allocation policy network and obtaining the action probability distribution output by the allocation policy network, the method further includes: The first feature vector and the second feature vector are subjected to matrix flattening and concatenation processing to obtain the current state vector used as the input of the allocation policy network.

[0049] It is understandable that after the first feature vector and the second feature vector are flattened and concatenated, the final state vector (that is, the current state vector) is formed, denoted as Subsequently, the current state vector is used as the input of the Actor network, and the Actor network outputs an action probability distribution. Through softmax sampling, the final action index, weight distribution, or probability distribution is obtained.

[0050] S104. Determine at least one target processing network in at least one data processing network according to the action probability distribution, and process the first feature vector through the at least one target processing network.

[0051] It can be understood that, based on S103 above, the target processing network is selected according to the action probability distribution. Specifically, the first K data processing networks can be used as the target processing networks, and the task allocation is performed through the load balancing executor. The feature units in the first feature vector are used as tasks and allocated to the target processing networks. For example, the task is allocated to the target expert network a according to the action a. At the same time, record the task processing results (such as accuracy, latency, etc.) of the target processing network and the change in expert load to calculate the reward.

[0052] Optionally, after processing the first feature vector through the at least one target processing network, the method further includes: Detect the overload data of at least one target processing network within a set period; if the overload rate calculated based on the overload data is greater than the fourth set threshold, then according to the configuration information of the overloaded network recorded in the overload data, add a new data processing network; or, if the overload rate is less than the fifth set threshold, then reduce the remaining data processing networks except the overloaded network, where the fifth set threshold is greater than the fourth set threshold.

[0053] It is understandable that the dynamic expansion and contraction of the expert network can also be implemented according to the task processing results. Specifically, the overload data of at least one target processing network within a set period is detected. The set period refers to a preset time range, for example, it can be 24 hours. The target processing network refers to the expert network currently being scheduled for use. The overload data refers to the data indicating whether the load of all target processing networks exceeds the upper limit within this period, that is, the number of target processing networks with the pending data exceeding the maximum processable data volume is counted. Subsequently, the overload rate is calculated based on the overload data. The overload rate refers to the time ratio, task ratio, or quantity ratio of the target processing network being in the overload state. For example, the overload rate within 24 hours is 35%, that is, 35 out of 100 expert networks are overloaded. It is determined whether the overload rate is greater than the fourth set threshold. Here, the fourth set threshold is a numerical standard for determining whether multiple target processing networks are frequently overloaded. For example, the fourth set threshold is 30%. If the overload rate is greater than the fourth set threshold, for example, the overload rate of 35% is greater than the fourth set threshold of 30%, the expert resource pool is automatically expanded, that is, new data processing networks are added. The new data processing networks can be homogeneous expert networks, that is, the expert networks with overload phenomena are added. If the overload rate is less than the fifth set threshold, for example, the overload rate of 5% is less than the fifth set threshold of 10%, a contraction warning is triggered, and the remaining data processing networks except the overloaded networks are reduced. Some data processing networks can be removed after the user confirms.

[0054] In the data processing method provided by this application, in the load balancing of the MoE model, the task traffic and expert load change in real time. Reinforcement learning can dynamically adjust the allocation strategy of the expert network through online learning. Based on the reinforcement learning method, according to the load conditions of each expert network, a reward function is set to dynamically adjust the task allocation strategy, optimize the load balancing in real time, reduce the degree of load imbalance, thereby alleviating the problem of uneven expert load and resource waste caused by static routing, and further improving the training efficiency of the model.

[0055] Based on the above embodiments, Figure 2 It is a schematic flowchart of a data processing method provided by an embodiment of this application. Optionally, after processing the first feature vector through at least one target processing network, the method further includes as Figure 2 shown in the following steps: S201. Obtain the processing result after at least one target processing network processes the first feature vector.

[0056] Among them, the processing result is used to represent the execution situation of at least one target processing network when processing the first feature vector.

[0057] It is understandable that the target processing network is the expert network or sub-network currently selected to participate in task processing. The first feature vector is the feature representation obtained after encoding or extraction of the input data. The processing result is the result output after the target processing network performs calculations, classifications, identifications, etc. on the first feature vector, and is used to characterize the performance of the target processing network during the data processing process, such as response latency, computing resource consumption, output confidence, etc. After completing the task processing, the processing results of each target processing network are collected to evaluate its running state during the task execution based on the processing results.

[0058] S202. Calculate a reward item according to the allocation result, the processing result, and a preset reward function.

[0059] Among them, the allocation result refers to the allocation result of allocating feature units to at least one target processing network. The first feature vector includes multiple feature units, and each target processing network is allocated at least one feature unit.

[0060] It is understandable that, based on S201 above, the allocation result refers to the allocation result that the task scheduling mechanism divides the first feature vector into several feature units and allocates them to different target processing networks for processing, that is, it records the target feature units processed by each target processing network and the quantity. The feature unit is the smallest processing unit after dividing the first feature vector into several parts. For example, it can be divided by dimension, by spatial region, etc. The reward function is a function predefined for evaluating the system performance, so as to generate a reward based on the task allocation strategy and the execution effect. The reward item is a feedback signal in reinforcement learning, which is used to guide the allocation policy network to optimize the expert scheduling policy. Specifically, based on the allocation result of the current task and the processing results returned by each target processing network, combined with the preset reward function, the system calculates a reward item to evaluate the pros and cons of the current task scheduling policy. At the same time, this reward item is used as the basis for updating the allocation policy network (such as the Actor network) to optimize the task allocation policy for the next round, so as to implement a dynamic and intelligent expert scheduling mechanism.

[0061] Among them, the processing result includes the processing accuracy rate and / or the actual processing latency. The processing accuracy rate is calculated through a loss function. The reward function includes a first function for measuring load balancing and / or a second function for measuring network performance.

[0062] It is understandable that the processing accuracy rate can be calculated through the cross-entropy loss function, denoted as , and the actual processing latency is denoted as , other metric items can also be added to the processing results. For example, the computing resource occupancy rate, etc. The first function is used to calculate the load balancing reward item (i.e., the load balancing item), and the second function is used to calculate the model performance reward item (i.e., the model performance item). The above other metric items can be used as reward and punishment conditions together with the accuracy rate and incorporated into the second function to adjust the behavior of the Actor network.

[0063] Optionally, calculate the reward item according to the allocation result, the processing result, and a pre-set reward function, which can be specifically implemented through the following steps: According to the first function, calculate the product of the variance of the number of feature units allocated to at least one target processing network in the allocation result and the pre-set load balancing coefficient to obtain the load balancing item; according to the second function, calculate the processing accuracy rate and / or the actual processing delay and the pre-set model performance coefficient to obtain the model performance item; obtain the reward item according to the load balancing item and the model performance item.

[0064] It can be understood that the calculation of the reward function is mainly divided into the load balancing reward item and the model performance reward item. Among them, the calculation formula of the first function is as shown in formula (1), the calculation formula of the second function is as shown in formula (2), and the calculation formula of the reward item is as shown in formula (3).

[0065] Formula (1) In the formula, is the load balancing item, is the load balancing coefficient, which is used to adjust the weight of load balancing, is N feature units, is the variance of the number of feature units allocated to N target processing networks.

[0066] Formula (2) In the formula, is the model performance item, is the model performance coefficient, which is used to adjust the weight of the accuracy rate.

[0067] Formula (3) In the formula, R is the reward item.

[0068] S203. Adjust the allocation policy network according to the reward item.

[0069] It can be understood that on the basis of the above S202, using the reinforcement learning mechanism, according to the calculated reward item, guide the allocation policy network to learn a better task scheduling policy. Specifically, the network parameters of the allocation policy network can be adjusted through gradient update or other optimization methods to improve the quality of future task allocation policies.

[0070] The data processing method provided by this application designs a multi-objective reward function, comprehensively considering indicators such as load balancing, task efficiency, and model performance in the reward function, and combining lightweight state representations (such as expert load characteristics and task characteristics), minimizing performance and accuracy losses while avoiding uneven loads and reducing computational overhead.

[0071] Based on the above embodiments, Figure 3 is a schematic flowchart of a data processing method provided by an embodiment of this application. After processing the first feature vector through at least one target processing network, the method further includes as Figure 3 shown in the following steps: S301. Record the current state, current action, current reward, and next state to obtain an experience tuple.

[0072] Among them, the current state refers to the environmental state before the allocation policy network outputs the current action, the current action refers to the action to be executed output by the allocation policy network based on the current state, the current action corresponds to an action probability, the current reward refers to the reward given by the environment after executing the current action, and the next state refers to the environmental state after executing the current action.

[0073] It can be understood that the current state refers to the environmental state perceived by the allocation policy network before outputting the current action, usually composed of the concatenation of the task feature vector and the expert load feature vector, that is, the above-mentioned current state vector. The current action is the task allocation decision made by the allocation policy network based on the current state. For example, select a certain expert network to process a certain feature unit. The action probability represents the probability of taking this action in the current state, which is the output of the allocation policy network, that is, the probability of allocating the feature unit to different expert networks. For example, allocate feature unit 1 to expert network 1 according to the probability. The current reward is the feedback signal calculated by the system according to the task execution effect after executing this action (expert network 1 processes feature unit 1), used to evaluate the quality of the action, that is, to evaluate whether expert network 1 is suitable for processing feature unit 1. The next state is the new environmental state after executing the current action, reflecting changes in expert load or task progress updates. During the interaction between the reinforcement learning agent and the environment, the interaction result at each time step is recorded as an experience tuple for storage, usually expressed as , where s is the current state, a is the current action, R is the current reward, is the next state, that is, the environmental state after executing the current action a.

[0074] S302. Calculate the advantage function according to the experience tuple.

[0075] Among them, the advantage function is used to measure the performance of the current action in the current state.

[0076] It can be understood that, based on the above S301, an experience tuple represents a complete process of interaction between an agent and the environment. The advantage function is used to measure the quality of taking the current action in the current state relative to the average policy. The specific calculation process of calculating the advantage function according to the experience tuple can be seen in the following embodiments.

[0077] It can be understood that the temporal difference error (TD-error) is a metric used in reinforcement learning to measure the difference between the predicted value and the actual value, mainly used to update the estimation of the advantage function or the action value function. In reinforcement learning, an agent learns how to select actions to maximize the cumulative reward by interacting with the environment. The advantage function estimates the expected cumulative reward that the agent can obtain in a certain state, and the TD-error is used to measure the accuracy of this estimation and update accordingly.

[0078] Optionally, calculating the advantage function according to the experience tuple can be specifically implemented through the following steps: Calculate the initial state value function and the final state value function, where the initial state value function is used to represent the expected value that can be obtained in the current state, and the final state value function is used to represent the evaluation value in the next state; calculate the advantage function according to the current reward, the initial state value function, and the final state value function.

[0079] It can be understood that the initial state value function represents the expected long-term return that can be obtained in the current state according to the current policy, and the final state value function represents the expected return in the next state entered after executing the action. For example, the Critic network can be used to estimate the values of the current state and the next state respectively, and the specific calculation process will not be elaborated. The advantage function is used to measure the quality of the current action in the current state relative to the average policy, and is used to guide the update of the policy network, so as to optimize the task scheduling policy (that is, allocate the current policy in the policy network), and improve the inference efficiency and load balancing ability of the MoE model. The calculation process of the advantage function is specifically shown in formula (4).

[0080] Formula (4) In the formula, A is the advantage function, R is the cumulative reward observed in the above intermediate behavior process (that is, the current reward), is the initial state value function, is the evaluated final state value function.

[0081] It is understandable that the advantage estimation reflects the relative superiority or inferiority of the actions taken by the evaluator in a given state. If the actual cumulative reward obtained is higher than expected, the advantage value is positive, indicating that this action is better than the average level of the current policy. The system will be more inclined to select this action in this state, that is, it is more inclined to use a specific expert network to process a specific feature unit. Conversely, if the actual reward is lower than expected, the advantage value is negative, indicating that this action performs poorly. The system will reduce the probability of selecting this action again. That is, the effect of expert network 1 in processing feature unit 1 is relatively poor, and feature unit 1 will be assigned to other expert networks for processing during the next allocation.

[0082] S303. Update the network parameters of the allocation policy network based on the advantage function.

[0083] It is understandable that, based on the above S302, after calculating the advantage function, the network parameters of the allocation policy network are updated based on the advantage function. Specifically, the parameters of the Actor network and / or the Critic network can be updated to achieve dynamic task allocation and load balancing optimization in the MoE model. The specific update process is described in the following embodiments.

[0084] Optionally, updating the network parameters of the allocation policy network based on the advantage function can be specifically implemented through the following steps: Based on the current policy of the allocation policy network, calculate the first probability distribution of the current action in the current state, where the current policy is the allocation policy adopted by the allocation policy network to output the current action; based on the optimized policy of the allocation policy network, calculate the second probability distribution of the current action in the current state; calculate the action loss function according to the first probability distribution, the second probability distribution, and the preset clipping range, where the action loss function is used to characterize the difference in the execution probabilities of the allocation policies before and after optimization for the same action; calculate the value loss function according to the advantage function, the current reward, the first probability distribution, and the second probability distribution; calculate the joint loss function according to the action loss function, the value loss function, the preset entropy function, and the preset hyperparameters; update the network parameters of the allocation policy network based on the joint loss function.

[0085] It is understandable that the action probability distribution under the current policy is obtained. The current policy is the policy used by the current assignment policy network for task assignment. The current state specifically refers to the current state vector formed by concatenating the first feature vector and the second feature vector. The current action refers to the specific task assignment decision output based on the current policy in the current state. The first probability distribution is the probability distribution of selecting the current action in the current state based on the current policy, that is, the probability that the current policy selects the current action. The action probability distribution under the optimized policy is obtained. The optimized policy is the updated policy in the assignment policy network, used for comparison with the current policy, and is often used in the calculation of the ratio of the old and new policies in algorithms such as PPO. The second probability distribution is the probability distribution of selecting the current action in the current state based on the optimized policy, that is, the probability that the optimized policy selects the current action, used to measure the change in the probability of selecting the current action before and after the policy update. It can also be understood as measuring the likelihood ratio of the updated policy to continue executing the action relative to the previous policy, and can also be understood as the difference in the execution likelihood of the same action before and after the policy. An action loss function is constructed based on the first probability distribution (the action probability under the current policy), the second probability distribution (the action probability under the optimized policy), and a pre-set clipping range to constrain the policy update amplitude. Among them, the clipping range is used to limit the policy update amplitude. For example, the clipping range can be defined as 0.2. The action loss function is used to measure the difference in the probability of selecting the same action before and after the policy update, and can prevent the training from being unstable due to too large an update amplitude, as shown in formula (5) specifically. A value loss function is constructed based on the advantage function, the current reward, the first probability distribution, and the second probability distribution to optimize the state value estimation. The value loss function is used to measure the error of the state value estimation and optimize the ability of the Critic network to estimate the state value, as shown in formula (6) specifically. The combined action loss, value loss, and policy entropy are integrated to construct a joint loss function. Among them, the entropy function is used to measure the exploration of the policy and prevent the policy from converging to the local optimum prematurely. The hyperparameter is the coefficient used to balance the weights of each loss term. The joint loss function is the objective function that integrates the action loss, value loss, and policy entropy, and is used as the final optimization goal to guide the update of network parameters through unifying multiple optimization goals. The specific calculation formula of the joint loss function is shown in formula (7).

[0086] Formula (5) In the formula, is the action loss function, that is, the loss function of the Actor network, , where, is the probability distribution of a in state S, specifically the second probability distribution, is the first probability distribution, is the clipping range to avoid excessive fluctuations in the policy, and E is the mean square error loss function.

[0087] Formula (6) Wherein is the value loss function, that is, the loss function of the Critic network.

[0088] Formula (7) Wherein is the joint loss function, c1 and c2 are hyperparameters, which are used to measure the importance of policy evaluation and exploration accuracy respectively, H is the entropy function to encourage exploration, which is used to encourage the exploration of the entropy of the policy output distribution, and the standard entropy function is shown in Formula (8).

[0089] Formula (8) It can be understood that is the standard entropy function. After obtaining the joint loss function, the optimizer is used to update the parameters of the allocation policy network. Specifically, the Adam optimizer can be used to update the network parameters.

[0090] The data processing method provided by this application designs a composite reward function, which can simultaneously consider key indicators such as delay, energy consumption, and computing accuracy, helps to overcome the problem of uneven load and improve performance while ensuring the model accuracy, etc. In addition, the PPO algorithm ensures the stability of training through a clipping mechanism, further optimizing the training effect.

[0091] Based on the above embodiments Figure 4 is a schematic flowchart of a data processing method provided by an embodiment of this application. After determining at least one target processing network in at least one data processing network according to the action probability distribution, the method further includes as Figure 4 shown in the following steps S401. Assign multiple feature units included in the first feature vector to at least one target processing network.

[0092] It can be understood that the first feature vector includes multiple feature units. When extracting features from the target data, multiple feature units can be divided. Or, taking the target data including multiple images as an example, one feature unit can be extracted from each image. At least one feature unit can be understood as the task to be assigned to the expert network for processing. Subsequently, according to the action probability distribution, the target processing network of the feature unit to be processed is determined, and then the feature unit is assigned to the corresponding target processing network. The number of feature units assigned to each target processing unit is not limited. For example, all the feature units extracted from the image are assigned to the first type of target processing network, and all the feature units extracted from the text are assigned to the second type of target processing network. That is, multimodal data can be assigned to different target processing networks according to the data type or processing method, and the specific assignment situation will not be elaborated.

[0093] Optionally, after allocating a plurality of feature units included in the first feature vector to at least one target processing network, the method further includes: Obtaining the amount of feature units currently to be processed by at least one target processing network; allocating the feature units to be allocated to the first target processing network to the second target processing network, where the first target processing network refers to a data processing network whose amount of feature units reaches the maximum processing capacity, and the priority of the first target processing network is higher than that of the second target processing network.

[0094] It can be understood that during the process of allocating at least one feature processing unit to the target processing network by the load balancer, the amount of data of the feature units currently to be processed by the target processing network is obtained in real time, which is the amount of feature units. If there is a first target processing network among at least one target processing network whose amount of feature units reaches its maximum processing capacity, then the feature units to be allocated to the first target processing network are allocated to the second target processing network, where the priority of the first target processing network is higher than that of the second target processing network, that is, if the number of task queues of a certain target processing network is full, then the remaining feature units to be allocated to the certain target processing network are allocated to the sub-optimal target processing network (i.e., the second target processing network). The second target processing network and the first target processing network can be understood as similar networks, and the number of the second target processing networks is not limited.

[0095] S402. Obtaining real-time load data of at least one target processing network.

[0096] Among them, the real-time load data includes the total number of target feature units allocated to the target processing network.

[0097] It can be understood that on the basis of the above S401, the real-time load data of the target processing network during the processing of feature units is obtained. The real-time load data includes multiple indicators such as the computing resource occupancy rate, the task queue depth, and the historical response latency.

[0098] Optionally, after obtaining the real-time load data of at least one target processing network, the method further includes: If the load rate calculated according to the real-time load data is less than the third set threshold, then combining the resources of at least one target processing network with the first quantity; or, if the load rate is greater than or equal to the third set threshold, then splitting at least one target processing network with the second quantity so that the target processing networks with the second quantity process the feature units in parallel.

[0099] It is understandable that, based on real-time load data, the limitations of the static expert network can also be reduced, the number and network structure of the expert network can be dynamically adjusted, and reinforcement learning can also optimize the number of experts and expert selection, etc. Specifically, calculate the load rate of each target processing network according to real-time load data, or the comprehensive load of all target processing networks, or the class load rate of a certain type of target processing network. Subsequently, if the load rate is less than the third set threshold, merge the resources of a certain number of target processing networks, that is, merge the expert network at low load to reduce the computing overhead, such as merging the expert network for image recognition. If the load rate is greater than or equal to the third set threshold, split a certain number of target processing networks, and the multiple sub-target processing networks after splitting process the feature units in parallel, that is, split the expert network at high load to improve the parallelism, etc.

[0100] S403. Calculate the variance of the total number among at least one target processing network, and / or count the number of consecutive overloads of at least one target processing network.

[0101] Wherein, the number of consecutive overloads refers to the number of times that the target processing network is continuously in an overloaded state when being assigned feature units.

[0102] It is understandable that, based on the above S402, deploy the allocation policy network to the real scenario and monitor the load fluctuation index of the expert network. Specifically, calculate the variance of the queue length of the target processing network, where the queue length refers to the number of feature units to be processed, and / or count the number of consecutive overloads of the target processing network. Consecutive overload means that the number of feature units to be processed by the target processing network has reached the maximum processing data volume, and it has been unable to allocate any feature units to it for several consecutive times, that is, there is an overflow phenomenon in this target processing network.

[0103] S404. If the variance is greater than the first set threshold and / or the number is greater than the second set threshold, adjust the allocation policy network in real time.

[0104] It is understandable that, based on the above S403, if the variance exceeds the first set threshold and / or the number of consecutive overloads exceeds the second set threshold, trigger the online fine-tuning of the allocation network. At the same time, during the adjustment process, freeze the Critic network, that is, do not perform gradient update on the Critic network, only update the Actor network parameters (such as reducing the learning rate), and continue for a given number of training steps. The first set threshold can be 0.5, and the second set threshold can be 3, that is, if tasks cannot be allocated to the target processing network for more than 3 consecutive times, there is a task overflow phenomenon.

[0105] The data processing method provided by this application enables the allocation policy network based on reinforcement learning to make dynamic decisions according to real-time load data. Through an online learning mechanism, it can perceive the changing characteristics of task distribution in different time periods, and adjust the allocation policy network in real time, thereby optimizing the problem of uneven load on the expert network and improving the model performance.

[0106] Based on the above embodiments, Figure 5 is a schematic flowchart of a data processing method provided by an embodiment of this application, which specifically includes the following steps as Figure 5 shown: 1) System architecture and data initialization: Obtain input data, and perform data preprocessing and feature extraction on the input data; 2) State representation and action generation: The encoder constructs relevant features of the expert load, generates task allocation decisions based on reinforcement learning, allocates tasks to the target expert network according to the task allocation decisions, controls the target expert network to execute tasks and obtains processing results, and at the same time updates the state and rewards; 3) Policy network optimization: Iteratively optimize the allocation policy network; 4) Online adaptive adjustment: Model deployment.

[0107] It can be understood that for the specific descriptions of the above 1) to 4), please refer to the above embodiments, and details are not described herein again.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0109] An embodiment of this application also provides a data processing device, Figure 6 is a schematic structural diagram of a data processing device provided by an embodiment of this application. The data processing device 600 includes a first acquisition module 601, a second acquisition module 602, an allocation module 603, and a data processing module 604, where The first acquisition module 601 is used to acquire target data to be processed, perform feature extraction on the target data, and generate a first feature vector; The second acquisition module 602 is used to generate a second feature vector according to the acquired target load data of at least one data processing network, where the target load data includes historical information of the data processing network; The allocation module 603 is used to input the first feature vector and the second feature vector into a pre-constructed allocation policy network, and obtain the action probability distribution output by the allocation policy network, where the action probability distribution is used to represent the probability that at least one data processing network is allocated the feature vector; A data processing module 604 is configured to determine at least one target processing network in at least one data processing network according to the action probability distribution, and process the first feature vector through the at least one target processing network.

[0110] Among them, the target load data includes at least one of the computing resource occupancy rate, the processed data volume, and the average processing delay. The processed data volume refers to the historical data volume processed by the data processing network within the historical time. Optionally, the second acquisition module 602 is configured to: Perform normalization processing on the target load data of at least one data processing network; Extract the features of the at least one load data after the normalization processing to generate at least one load feature vector; Concatenate the at least one load feature vector to obtain a second feature vector; or, add a preference feature vector to the at least one load feature vector and then concatenate them to obtain the second feature vector, where the preference feature vector is obtained by extracting the features of the preference information for the data processing types of the at least one data processing network.

[0111] Optionally, the data processing device 600 is further configured to: Perform matrix flattening and concatenation processing on the first feature vector and the second feature vector to obtain a current state vector as the input of the allocation policy network.

[0112] Optionally, the data processing device 600 is further configured to: Obtain the processing result after the at least one target processing network processes the first feature vector, where the processing result is used to characterize the execution situation of the at least one target processing network when processing the first feature vector; Calculate a reward term according to the allocation result, the processing result, and a preset reward function, where the allocation result refers to the allocation result of allocating feature units to the at least one target processing network. The first feature vector includes multiple feature units, and each target processing network is allocated at least one feature unit; Adjust the allocation policy network according to the reward term.

[0113] Among them, the processing result includes the processing accuracy rate and / or the actual processing delay. The processing accuracy rate is calculated through a loss function. The reward function includes a first function for measuring load balancing and / or a second function for measuring network performance.

[0114] Optionally, the data processing device 600 is further configured to: Calculate the product of the variance of the number of feature units allocated to the at least one target processing network in the allocation result and a preset load balancing coefficient according to the first function to obtain a load balancing term; Calculate a model performance term based on a second function, the processing accuracy rate and / or the actual processing latency, and a preset model performance coefficient. Obtain a reward term based on the load balancing term and the model performance term.

[0115] Optionally, the data processing device 600 is further configured to: Record the current state, the current action, the current reward, and the next state to obtain an experience tuple, where the current state refers to the environmental state before the allocation policy network outputs the current action, the current action refers to the action to be executed output by the allocation policy network based on the current state, the current action corresponds to an action probability, the current reward refers to the reward given by the environment after executing the current action, and the next state refers to the environmental state after executing the current action; Calculate an advantage function according to the experience tuple, where the advantage function is used to measure the performance of the current action in the current state; Update the network parameters of the allocation policy network based on the advantage function.

[0116] Optionally, the data processing device 600 is further configured to: Calculate an initial state value function and a final state value function, where the initial state value function is used to represent the expected value that can be obtained in the current state, and the final state value function is used to represent the evaluation value in the next state; Calculate the advantage function according to the current reward, the initial state value function, and the final state value function.

[0117] Optionally, the data processing device 600 is further configured to: Calculate a first probability distribution of the current action in the current state based on the current policy of the allocation policy network, where the current policy is the allocation policy adopted by the allocation policy network to output the current action; Calculate a second probability distribution of the current action in the current state based on the optimized policy of the allocation policy network; Calculate an action loss function according to the first probability distribution, the second probability distribution, and a preset clipping range, where the action loss function is used to represent the difference in the execution possibility of the allocation policies before and after optimization for the same action; Calculate a value loss function according to the advantage function, the current reward, the first probability distribution, and the second probability distribution; Calculate a joint loss function according to the action loss function, the value loss function, a preset entropy function, and a preset hyperparameter; Update the network parameters of the allocation policy network based on the joint loss function.

[0118] Optionally, the data processing device 600 is further configured to: Assign a plurality of feature units included in the first feature vector to at least one target processing network; Obtain real-time load data of at least one target processing network, where the real-time load data includes the total number of target feature units assigned to the target processing network; Calculate the variance of the total number among at least one target processing network, and / or count the number of consecutive overloads of at least one target processing network, where the number of consecutive overloads refers to the number of times the target processing network is continuously in an overloaded state when being assigned feature units; If the variance is greater than the first set threshold and / or the number is greater than the second set threshold, then adjust the allocation policy network in real time.

[0119] Optionally, the data processing device 600 is further configured to: If the load rate calculated based on the real-time load data is less than the third set threshold, then merge the resources of at least one target processing network with the first quantity; or, If the load rate is greater than or equal to the third set threshold, then split at least one target processing network with the second quantity so that the target processing networks with the second quantity process feature units in parallel.

[0120] Optionally, the data processing device 600 is further configured to: Obtain the amount of feature units currently to be processed by at least one target processing network; Assign the feature units to be assigned to the first target processing network to the second target processing network, where the first target processing network refers to the data processing network whose feature unit amount reaches the maximum processing amount, and the priority of the first target processing network is higher than that of the second target processing network.

[0121] Optionally, the data processing device 600 is further configured to: Detect the overload data of at least one target processing network within a set time period; If the overload rate calculated based on the overload data is greater than the fourth set threshold, then add a new data processing network according to the configuration information of the overloaded network recorded in the overload data; or, If the overload rate is less than the fifth set threshold, then reduce the remaining data processing networks except the overloaded network, where the fifth set threshold is greater than the fourth set threshold.

[0122] For the description of the features in the embodiments corresponding to the data processing device, reference may be made to the relevant descriptions in the embodiments corresponding to the data processing method, which will not be elaborated here one by one.

[0123] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the embodiments of the above data processing method.

[0124] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and the computer program is configured to execute the steps in any one of the above data processing method embodiments when running.

[0125] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media that can store computer programs such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0126] An embodiment of the present application further provides a computer program product, the above computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any one of the above data processing method embodiments are implemented.

[0127] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the above data processing method embodiments are implemented.

[0128] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0129] The above has introduced in detail a data processing provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data processing method, characterized in that, Including: Obtain target data to be processed, and perform feature extraction on the target data to generate a first feature vector; Generate a second feature vector according to the obtained target load data of at least one data processing network, where the target load data includes historical information of the data processing network; Input the first feature vector and the second feature vector into a pre-constructed allocation policy network, and obtain the action probability distribution output by the allocation policy network, where the action probability distribution is used to represent the probability that the at least one data processing network is allocated the feature vector; Determine at least one target processing network from the at least one data processing network according to the action probability distribution, and process the first feature vector through the at least one target processing network.

2. The method according to claim 1, wherein The target load data includes at least one of a computing resource occupancy rate, a processed data volume, and an average processing delay, and the processed data volume refers to the historical data volume processed by the data processing network within a historical time; The generating a second feature vector according to the obtained target load data of at least one data processing network includes: Perform normalization processing on the target load data of the at least one data processing network; Extract features of the at least one load data after normalization processing to generate at least one load feature vector; Concatenate the at least one load feature vector to obtain a second feature vector; or, add a preference feature vector to the at least one load feature vector and then concatenate them to obtain the second feature vector, where the preference feature vector is obtained by performing feature extraction on the preference information of the data types processed by the at least one data processing network.

3. The method according to claim 1, characterized in that Before the inputting the first feature vector and the second feature vector into a pre-constructed allocation policy network and obtaining the action probability distribution output by the allocation policy network, the method further includes: Perform matrix flattening and concatenation processing on the first feature vector and the second feature vector to obtain a current state vector serving as the input of the allocation policy network.

4. The method according to claim 1, characterized in that After the processing the first feature vector through the at least one target processing network, the method further includes: Obtain the processing result after the at least one target processing network processes the first feature vector, where the processing result is used to represent the execution situation of the at least one target processing network when processing the first feature vector; Calculate a reward item according to the allocation result, the processing result, and a pre-set reward function, where the allocation result refers to the allocation result of allocating feature units to the at least one target processing network, the first feature vector includes multiple feature units, and each target processing network is allocated at least one feature unit; Adjust the allocation policy network according to the reward item.

5. The method according to claim 4, wherein The processing result includes a processing accuracy rate and / or an actual processing delay. The processing accuracy rate is calculated through a loss function. The reward function includes a first function for measuring load balancing and / or a second function for measuring network performance. Calculating a reward item according to the allocation result, the processing result, and a preset reward function includes: Calculating a product of a variance of the number of feature units allocated by the at least one target processing network in the allocation result and a preset load balancing coefficient according to the first function to obtain a load balancing item; Calculating a model performance item according to the second function, the processing accuracy rate and / or the actual processing delay, and a preset model performance coefficient; Obtaining the reward item according to the load balancing item and the model performance item.

6. The method according to claim 1, wherein After processing the first feature vector through the at least one target processing network, the method further includes: Recording a current state, a current action, a current reward, and a next state to obtain an experience tuple, where the current state refers to an environmental state before the allocation policy network outputs the current action, the current action refers to an action to be executed output by the allocation policy network based on the current state, the current action corresponds to an action probability, the current reward refers to a reward given by the environment after executing the current action, and the next state refers to an environmental state after executing the current action; Calculating an advantage function according to the experience tuple, where the advantage function is used to measure the performance of the current action in the current state; Updating network parameters of the allocation policy network based on the advantage function.

7. The method according to claim 6, wherein Calculating the advantage function according to the experience tuple includes: Calculating an initial state value function and a final state value function, where the initial state value function is used to represent an expected value that can be obtained in the current state, and the final state value function is used to represent an evaluation value in the next state; Calculating the advantage function according to the current reward, the initial state value function, and the final state value function.

8. The method according to claim 6, characterized in that, Updating the network parameters of the allocation policy network based on the advantage function includes: Calculating a first probability distribution of the current action in the current state based on the current policy of the allocation policy network, where the current policy is an allocation policy adopted by the allocation policy network to output the current action; Calculating a second probability distribution of the current action in the current state based on the optimized policy of the allocation policy network; Calculating an action loss function according to the first probability distribution, the second probability distribution, and a preset clipping range, where the action loss function is used to represent the difference in the execution possibility of the allocation policies before and after optimization for the same action; Calculating a value loss function according to the advantage function, the current reward, the first probability distribution, and the second probability distribution; Calculating a joint loss function according to the action loss function, the value loss function, a preset entropy function, and a preset hyperparameter; Update the network parameters of the allocation policy network based on the combined loss function.

9. The method according to claim 1, wherein After determining at least one target processing network in the at least one data processing network according to the action probability distribution, the method further includes: Assign a plurality of feature units included in the first feature vector to the at least one target processing network; Obtain real-time load data of the at least one target processing network, where the real-time load data includes the total number of target feature units assigned to the target processing network; Calculate the variance of the total number among the at least one target processing network, and / or count the number of consecutive overloads of the at least one target processing network, where the number of consecutive overloads refers to the number of times the target processing network is continuously in an overloaded state when being assigned feature units; If the variance is greater than a first set threshold and / or the number is greater than a second set threshold, adjust the allocation policy network in real time.

10. The method according to claim 9, characterized in that After obtaining the real-time load data of the at least one target processing network, the method further includes: If the load rate calculated according to the real-time load data is less than a third set threshold, merge the resources of the at least one target processing network by a first quantity; or, If the load rate is greater than or equal to the third set threshold, split the at least one target processing network by a second quantity so that the second quantity of target processing networks process feature units in parallel.

11. The method according to claim 9, characterized in that, After assigning a plurality of feature units included in the first feature vector to the at least one target processing network, the method further includes: Obtain the quantity of feature units currently to be processed by the at least one target processing network; Assign the feature units to be assigned to a first target processing network to a second target processing network, where the first target processing network refers to the data processing network whose quantity of feature units reaches the maximum processing capacity, and the priority of the first target processing network is higher than that of the second target processing network.

12. The method according to claim 1, characterized in that, After processing the first feature vector by the at least one target processing network, the method further includes: Detect the overload data of the at least one target processing network within a set time period; If the overload rate calculated according to the overload data is greater than a fourth set threshold, add a new data processing network according to the configuration information of the overloaded network recorded in the overload data; or, If the overload rate is less than a fifth set threshold, reduce the remaining data processing networks except the overloaded network, where the fifth set threshold is greater than the fourth set threshold.

13. An electronic device, characterized in that, including: a memory for storing a computer program; a processor for implementing the steps of the data processing method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program implements the steps of the data processing method according to any one of claims 1 to 12 when being executed by a processor.

15. A computer program product, comprising a computer program, characterized in that, The computer program implements the steps of the data processing method according to any one of claims 1 to 12 when being executed by a processor.

Images