Data distribution method, device and equipment and computer readable storage medium

Abstract

The invention provides a data distribution method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring a plurality of resource clusters and the unit computing power of an arithmetic unit included in each resource cluster; according to the unit computing power of each computing unit, combining each computing unit to obtain a plurality of combined clusters, and determining the cluster computing power of each combined cluster; obtaining multiple pieces of training data; and based on the cluster computing power of each combined cluster, performing grouping processing on the plurality of pieces of training data to obtain a training data set corresponding to each combined cluster, and distributing the training data set to the corresponding combined cluster so as to train a model by using each combined cluster through the corresponding training data set. According to the method and the device, the computing power of different computing units in a plurality of resource clusters can be activated to support the training of the model, so that the training time can be shortened, and the training efficiency can be improved.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a data distribution method, device, equipment and computer-readable storage medium. Background Art

[0002] Large models usually refer to deep learning models with large-scale parameters and complex structures. These models are usually composed of multiple layers of neural networks and can be used to process large amounts of training data and complex tasks. Large models have achieved good results in natural language processing, image recognition, speech recognition and other fields.

[0003] Large model training usually requires more computing resources and time. During the training phase, large models need to process large data sets, and the required graphics processing unit (GPU) computing power will reach more than 10,000 cards. Since large-scale computing cards are accumulated in recent years, the heterogeneous computing power of 10,000 cards will be dispersed in multiple computing clusters.

[0004] In related technologies, the scattered GPU card resources are gathered through resource integration, and then aggregated into a large cluster for unified management and control, and different label management is adapted to different card types to distinguish different computing capabilities and training data processing capabilities. However, there are problems such as high integration difficulty, the need for additional coordination resources, and limited processing capabilities. Summary of the invention

[0005] The embodiments of the present application provide a data distribution method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can activate the computing power of different computing units in multiple resource clusters to support model training, thereby shortening the training time of the model and improving the training efficiency of the model.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present invention provides a data distribution method, which includes:

[0008] Obtain unit computing power of multiple resource clusters and computing units included in each resource cluster;

[0009] Combining the computing units according to the computing power of the computing units to obtain a plurality of combined clusters, and determining the cluster computing power of each combined cluster;

[0010] Get multiple training data;

[0011] Based on the cluster computing power of each combined cluster, the multiple training data are grouped and processed to obtain training data sets corresponding to each combined cluster, and the training data sets are distributed to the corresponding combined clusters, so as to train the model using the corresponding training data sets of each combined cluster.

[0012] The present invention provides a data distribution device, which includes:

[0013] A first acquisition module is used to acquire a plurality of resource clusters and unit computing power of computing units included in each resource cluster;

[0014] A combination module, used for combining the computing units according to the computing power of the computing units to obtain a plurality of combined clusters, and determining the cluster computing power of each combined cluster;

[0015] A second acquisition module is used to acquire multiple training data;

[0016] The distribution module is used to group the multiple training data based on the cluster computing power of each combined cluster, obtain the training data set corresponding to each combined cluster, and distribute the training data set to the corresponding combined cluster, so as to use the corresponding training data set to train the model using the each combined cluster.

[0017] An embodiment of the present application provides a computer device, including:

[0018] A memory for storing computer executable instructions;

[0019] The processor is used to implement the data distribution method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute the data distribution method provided in the embodiment of the present application.

[0021] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data distribution method provided by the embodiment of the present application is implemented.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] In an embodiment of the present application, multiple resource clusters and the unit computing power of the computing units included in each resource cluster are obtained, and then the computing units are combined according to the unit computing power of each computing unit to obtain multiple combined clusters, which include computing units with the same or similar unit computing power; the cluster computing power of each combined cluster is also determined; then, multiple training data are obtained, and the multiple training data are grouped and processed based on the cluster computing power of each combined cluster to obtain the training data set corresponding to the combined cluster, and the training data set is sent to the corresponding combined cluster, so as to use each combined cluster to train the model through the corresponding training data set. In this way, multiple combined clusters associated with the unit computing power of the computing unit can be generated, that is, multiple resource clusters can be recombined according to the unit computing power to achieve the convenience of combination. Moreover, multiple training data are grouped based on the cluster computing power of each combined cluster, and the grouped training data sets are distributed to the corresponding combined clusters. That is, the cluster computing power of the combined cluster itself is taken into consideration when distributing the training data, so that the corresponding training data sets can be distributed according to the cluster computing power, thereby maximizing the computing power of each combined cluster, and ultimately shortening the training time of the model and improving the training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a schematic diagram of a network architecture of a data distribution system 100 provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of the structure of the server 400 provided in an embodiment of the present application;

[0026] Figure 3 This is a schematic diagram of an implementation flow of the data distribution method provided in an embodiment of the present application;

[0027] Figure 4 This is a schematic diagram of an implementation flow of obtaining multiple combination groups provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of an implementation flow of distributing multiple training data provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of another implementation flow of distributing multiple training data provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of an implementation flow of a plurality of training data after grouping and sorting provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of a data distribution implementation framework provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram showing screening training data provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of another implementation process of data distribution provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0038] See also Figure 1 , Figure 1 1 is a schematic diagram of the network architecture of the data distribution system 100 provided in the embodiment of the present application. To realize supporting a data distribution application, the terminal 200 is connected to the server 400 through the network 300, and the server 400 is connected to multiple resource clusters through the network 300. Figure 1 Schematically shows resource cluster 500 - 1 , resource cluster 500 - 2 and resource cluster 500 - 3 , and network 300 may be a wide area network or a local area network, or a combination of the two.

[0039] Taking the model as the recognition mode as an example, when the terminal 200 receives the model training instruction, the terminal 200 is used to obtain multiple training data and send the multiple training data to the server 400; the server 400 is used to obtain the unit computing power of the computing units included in the resource cluster 500-1, the resource cluster 500-2 and the resource cluster 500-3 and each resource cluster; the computing units are combined according to the unit computing power of each computing unit to obtain multiple combined clusters, and the cluster computing power of each combined cluster is determined; the server 400 is used to distribute the multiple training data to each combined cluster based on the cluster computing power of each combined cluster, so as to train the recognition model with the corresponding training data of each combined cluster to obtain a trained recognition model. The server 400 can also send the trained recognition model to the terminal 200 so that the terminal 200 uses the trained recognition model for recognition processing.

[0040] In the embodiment of the present application, the server 400 can generate multiple combined clusters associated with the unit computing power of the computing unit, that is, to realize the re-combination of multiple resource clusters according to the unit computing power, and realize the convenience of combination. In addition, multiple training data are grouped and processed based on the cluster computing power of each combined cluster, and the grouped training data sets are distributed to the corresponding combined clusters, that is, the cluster computing power of the combined cluster itself is taken into account when distributing the training data, and the corresponding training data sets can be distributed according to the cluster computing power, thereby maximizing the computing power of each combined cluster, and finally shortening the training time of the model and improving the training efficiency of the model.

[0041] In some embodiments, the server 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car-mounted smart terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0042] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0043] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The embodiments of this application relate to model training technology in artificial intelligence.

[0044] See also Figure 2 , Figure 2 A schematic diagram of the structure of the server 400 provided in the embodiment of the present application, Figure 2 The server 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0045] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0046] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0047] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0048] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0049] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0050] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0051] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0052] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0053] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0054] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2The data distribution device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a first acquisition module 4551, a combination module 4552, a second acquisition module 4553 and a distribution module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0055] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data distribution method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integ rated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0056] The data distribution method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0057] The following describes the data distribution method provided by the embodiment of the present application. As mentioned above, the computer device that implements the data distribution method of the embodiment of the present application may be a terminal. Therefore, the execution subject of each step will not be repeatedly described below.

[0058] It should be noted that in the example of the data distribution method below, the distribution of training data is used as an example. Those skilled in the art, based on their understanding of the following, can apply the data distribution method provided in the embodiment of the present application to the distribution of other large-scale data.

[0059] See also Figure 3 , Figure 3 This is a schematic diagram of an implementation flow of the data distribution method provided in the embodiment of the present application. Figure 3 The steps shown are used to illustrate the data distribution method provided in the embodiment of the present application. Figure 3 The execution subject is the server.

[0060] In step S101, a plurality of resource clusters and the unit computing power of the computing units included in each resource cluster are obtained.

[0061] In some embodiments, a resource cluster refers to a set of computing resources, which may be CPU, GPU, memory, etc., and is used to train a machine learning model. During the training process, a large amount of computing and storage resources are usually required to process large-scale data sets and perform complex numerical calculations and optimization algorithms.

[0062] In some embodiments, a resource cluster can be composed of multiple computers or servers, which can be connected to the same local area network or cloud platform. Each computer can be responsible for processing a portion of the data and algorithm in parallel to speed up the training process. By using a resource cluster, multiple computing resources can be effectively utilized to improve the efficiency and performance of model training.

[0063] In some embodiments, a computing unit refers to the smallest unit in a computing resource that can perform computing. For example, the computing unit may be a GPU card, a CPU card, etc. Unit computing power refers to the ability of a computing unit to process data. The unit computing power can be represented by a numerical value. The larger the numerical value, the stronger the ability of the computing unit to process data. Assuming that the unit computing power of GPU card 1 is 3 and the unit computing power of GPU card 2 is 6.1, it indicates that the ability of GPU card 2 to process data is stronger than that of GPU card 1.

[0064] In step S102, the computing units are combined according to the unit computing power of each computing unit to obtain a plurality of combined clusters, and the cluster computing power of each combined cluster is determined.

[0065] In an embodiment of the present application, the computing units in the resource cluster are reintegrated based on the unit computing power, disrupting the original combination of computing units in the resource cluster to obtain a combined cluster corresponding to the unit computing power.

[0066] In some embodiments, see Figure 4 , the "combining each computing unit according to the unit computing power of each computing unit to obtain multiple combined clusters" in the above step S102 can be implemented through the following steps S1021 to S1023, which are described in detail below.

[0067] In step S1021, a mapping relationship between preset unit computing power and unit computing power level is obtained.

[0068] In the embodiment of the present application, the mapping relationship is a mapping relationship in which at least one unit computing power corresponds to one unit computing power level.

[0069] In some embodiments, the mapping relationship may be pre-set, and may be one unit computing power corresponding to one unit computing power level, or multiple unit computing powers corresponding to one unit computing power level. For example, unit computing power 1 may correspond to unit computing power level 1, and unit computing power 2 may correspond to unit computing power level 2; or unit computing power 1 and unit computing power 2 may correspond to unit computing power level 1, and unit computing power 3 and unit computing power 4 may correspond to unit computing power level 2.

[0070] In step S1022, based on the mapping relationship and the unit computing power of each computing unit, the unit computing power level of each computing unit is determined.

[0071] In some embodiments, according to the mapping relationship, the unit computing power level corresponding to the unit computing power of each computing unit is determined, and then the unit computing power level is determined as the unit computing power level of each computing unit.

[0072] For example, assuming that unit computing power 1 corresponds to unit computing power level 1, unit computing power 2 corresponds to unit computing power level 2, the unit computing power of computing unit 1 and computing unit 2 are both unit computing power 1, and the unit computing power of computing unit 3, computing unit 4, and computing unit 5 are all unit computing power 2. Then the unit computing power level of computing unit 1 and computing unit 2 is unit computing power level 1, and the unit computing power level of computing unit 3, computing unit 4, and computing unit 5 is unit computing power level 2.

[0073] In step S1023, computing units with the same computing power level among the computing units are determined as candidate computing units, and the candidate computing units are combined to obtain a combined cluster.

[0074] In some embodiments, computing units with the same computing power level are determined as candidate computing units, and the candidate computing units are combined, that is, the candidate computing units are combined into a computing unit set, and the computing unit set is recorded as a combined cluster.

[0075] Continuing with the above example, operation unit 1 and operation unit 2 may be combined to obtain combined cluster 1; and operation unit 3, operation unit 4, and operation unit 5 may be combined to obtain combined cluster 2.

[0076] Through the above steps S1021 to S1023, the unit computing power level of each computing unit is first determined based on the mapping relationship, and then the computing units with the same unit computing power level are combined to obtain a combined cluster, which contains computing units with the same unit computing power level. In this way, each computing unit is combined according to the unit computing power level to obtain a combined cluster containing computing units with the same unit computing power level, thereby achieving the purpose of integrating the computing power of computing units in multiple resource clusters.

[0077] In some embodiments, the implementation method of "determining the cluster computing power of each combined cluster" in the above step S102 can be: first obtain the computing units included in each combined cluster and the unit computing power of each computing unit; then accumulate and sum the unit computing powers of the computing units in the same combined cluster to obtain the cluster computing power of the combined cluster.

[0078] In some embodiments, the computing units included in each combined cluster are first obtained, and the unit computing power of each computing unit is also obtained, wherein the unit computing power is used to characterize the computing capability of a computing unit, and the unit computing power of the computing unit is determined after the manufacturing of the computing unit is completed, that is, the unit computing power is determined by the hardware composition of the computing unit.

[0079] In some embodiments, the unit computing power of the computing units belonging to the same combined cluster is accumulated and summed to obtain the cluster computing power of the combined cluster. For example, assuming that combined cluster 1 includes computing unit 1 and computing unit 2, the unit computing power of computing unit 1 is 3, and the unit computing power of computing unit 2 is 5, then the cluster computing power of the combined cluster is 8. In this way, the cluster computing power of each combined cluster can be accurately determined.

[0080] Continue to see Figure 3 , and the description continues with step S102 above.

[0081] In step S103, a plurality of training data are obtained.

[0082] In some embodiments, when data is distributed to the combined cluster for the first time, the training data is the training sample; when data is distributed to the combined cluster for the second or Nth time, the training data is the intermediate state data generated by the combined cluster during the last training, where N is the total number of training times.

[0083] In some embodiments, the training sample may include training data and training labels corresponding to the training data. The server may obtain the training sample through an external device, for example, by a touch operation on a touch device, or by an input operation on a typing device. Since the intermediate state data is data generated during the training process, that is, the intermediate state data is stored in the memory of the server, the server may directly obtain the intermediate state data from its own storage space.

[0084] In step S104, based on the cluster computing power of each combined cluster, multiple training data are grouped and processed to obtain training data sets corresponding to each combined cluster, and the training data sets are distributed to the corresponding combined clusters, so as to train the model using the corresponding training data sets of each combined cluster.

[0085] In the embodiment of the present application, multiple training data are first grouped based on the cluster computing power of the combined cluster to obtain multiple sets of training data, wherein the multiple sets of training data correspond to the combined cluster; then, each set of training data is distributed to the corresponding combined cluster. That is, multiple training data are first grouped based on the cluster computing power, and then each set of training data obtained after grouping is distributed to the corresponding combined cluster, thereby realizing targeted distribution of training data.

[0086] In some embodiments, see Figure 5 , the above step S104 of "grouping and processing multiple training data based on the cluster computing power of each combined cluster to obtain a training data set corresponding to each combined cluster" can be implemented through the following steps S1041 to S1043, which are described in detail below.

[0087] In step S1041, the plurality of training data are sorted in order of occupied space from small to large to obtain the sorted plurality of training data.

[0088] In some embodiments, the space occupied by each piece of training data in the storage space may be obtained first, and then the training data may be sorted in order of the space occupied by each piece of training data from small to large to obtain a plurality of sorted training data.

[0089] Exemplarily, it is assumed that the plurality of training data include training data 1, training data 2, training data 3, training data 4, training data 5, training data 6, training data 7, training data 8, training data 9, and training data 10; and the space occupied by training data 1 is 180 kilobytes, the space occupied by training data 2 is 120 kilobytes, the space occupied by training data 3 is 210 kilobytes, the space occupied by training data 4 is 260 kilobytes, the space occupied by training data 5 is 90 kilobytes, the space occupied by training data 6 is 320 kilobytes, the space occupied by training data 7 is 100 kilobytes, the space occupied by training data 8 is 400 kilobytes, the space occupied by training data 9 is 287 kilobytes, and the space occupied by training data 10 is 296 kilobytes. Based on this, the plurality of training data after sorting are training data 5, training data 7, training data 2, training data 1, training data 3, training data 4, training data 9, training data 10, training data 6, and training data 8.

[0090] In step S1042, based on the cluster computing power of each combined cluster, the computing power proportion of each combined cluster is determined.

[0091] In some embodiments, the above-mentioned step S1042 can be implemented by accumulating the cluster computing power of each combined cluster to obtain the total cluster computing power; respectively determining the ratio of the cluster computing power of each combined cluster to the total cluster computing power, and determining each ratio as the computing power ratio of the corresponding combined cluster.

[0092] For example, it is assumed that there are five combined clusters, namely, combined cluster 1, combined cluster 2, combined cluster 3, combined cluster 4 and combined cluster 5, and the cluster computing power of combined cluster 1 is 10, the cluster computing power of combined cluster 2 is 40, the cluster computing power of combined cluster 3 is 20, the cluster computing power of combined cluster 4 is 25, and the cluster computing power of combined cluster 5 is 105. Based on this, the sum of cluster computing power 1, cluster computing power 2, cluster computing power 3, cluster computing power 4 and cluster computing power 5 is the total cluster computing power, that is, the total cluster computing power is 200; then the computing power of combined cluster 1 accounts for 10 / 200, that is, 5%, the computing power of combined cluster 2 accounts for 40 / 200, that is, 20%, the computing power of combined cluster 3 accounts for 20 / 200, that is, 10%, the computing power of combined cluster 4 accounts for 25 / 200, that is, 12.5%, and the computing power of combined cluster 5 accounts for 105 / 200, that is, 52.5%.

[0093] In step S1043, the sorted multiple training data are grouped according to the computing power ratio of each combined cluster to obtain a training data set corresponding to each combined cluster.

[0094] In some embodiments, see Figure 6 The above step S1043 can be implemented by the following steps S431 to S433, which are described in detail below.

[0095] In step S431, the unit computing power level of the computing units in each combined cluster is determined as the cluster computing power level of the corresponding combined cluster.

[0096] In some embodiments, the unit computing power level of each computing unit can be determined based on the above step S1022. Since the combined cluster contains computing units with the same unit computing power level, the unit computing power level of any computing unit in the combined cluster can be determined as the cluster computing power level of the combined cluster.

[0097] In step S432, the combined clusters are sorted in order of the computing power levels of the clusters from small to large to obtain sorted combined clusters.

[0098] For example, it is assumed that the cluster computing power level of combined cluster 1 is 5, the cluster computing power level of combined cluster 2 is 2, the cluster computing power level of combined cluster 3 is 3, the cluster computing power level of combined cluster 4 is 1, and the cluster computing power level of combined cluster 5 is 4. And the above cluster computing power levels are 1, 2, 3, 4, and 5 from small to large. Based on this, the sorted combined clusters are combined cluster 4, combined cluster 2, combined cluster 3, combined cluster 5, and combined cluster 1.

[0099] In step S433, the sorted multiple training data are grouped and processed using the computing power proportions of the sorted combined clusters to obtain training data sets corresponding to the sorted combined clusters.

[0100] In some embodiments, see Figure 7 The above step S433 can be implemented by the following steps S4331 to S4334, which are described in detail below.

[0101] In step S4331, the total number of the sorted training data is obtained.

[0102] In some embodiments, the total number of data refers to the number of data of the plurality of training data after sorting. Since sorting is to change the order of the plurality of training data but does not change the number of data of the plurality of training data, the total number of data may also be the number of data of the plurality of training data. For example, the total number of data may be 10,000, 100,000, 1 million, etc.

[0103] In step S4332, the jth starting position is determined based on the sum of the computing power proportions of the first (j-1) sorted combined clusters and the total number of data.

[0104] In the embodiment of the present application, j is an integer greater than 0 and less than T, and T is the total number of computing power proportions.

[0105] In some embodiments, the product of the sum of the computing power proportions of the first (j-1) sorted combined clusters and the total number of data can be determined as the jth starting position. Continuing with the above example, the sorted combined clusters are combined cluster 4, combined cluster 2, combined cluster 3, combined cluster 5, and combined cluster 1; and the computing power proportion of combined cluster 4 is 12.5%, the computing power proportion of combined cluster 2 is 20%, the computing power proportion of combined cluster 3 is 10%, the computing power proportion of combined cluster 5 is 52.5%, and the computing power proportion of combined cluster 1 is 5%. Based on this, the first starting position is 0, and the second starting position can be determined by the following formula (1).

[0106] The second starting position = 12.5% ​​* total number formula (1);

[0107] In formula (1), when the total number is 10,000, the second starting position is 1250.

[0108] Similarly, the third starting position can be determined by the following formula (2).

[0109] The third starting position = (12.5% ​​+ 20%) * total number formula (2);

[0110] In formula (2), when the total number is 10,000, the third starting position is 3250.

[0111] In some embodiments, the method for determining the fourth starting position and the fifth starting position may refer to the method for determining the third starting position described above.

[0112] In step S4333, the j-th end position is determined based on the sum of the computing power proportions of the first j sorted combined clusters and the total number of data.

[0113] In some embodiments, following the example in step S4332, the first end position can be determined by the following formula (3). That is, the process of determining the first end position includes: first determining the first product of the computing power proportion of the first sorted combined cluster and the total number of data, then determining the first difference between the first product and the value 1, and finally determining the first difference as the first end position.

[0114] 1st end position = 12.5% ​​* total number - 1 Formula (3);

[0115] In formula (3), when the total number is 10,000, the first end position is 1249.

[0116] Similarly, the second end position can be determined by the following formula (4). That is, the process of determining the second end position includes: first determining the first cumulative sum of the computing power proportions of the first two sorted combined clusters, then determining the second product of the first cumulative sum and the total number of data, then determining the second difference between the second product and the value 1, and finally determining the second difference as the second end position.

[0117] The second end position = (12.5% ​​+ 20%) * total number - 1 formula (4);

[0118] In formula (4), when the total number is 10,000, the second end position is 3249.

[0119] In some embodiments, the method for determining the third end position, the fourth end position, and the fifth end position may refer to the method for determining the second end position described above.

[0120] In step S4334, the training data located between the j-th starting position and the j-th ending position in the sorted plurality of training data are determined as the training data set corresponding to the j-th combined cluster after sorting.

[0121] In some embodiments, it is assumed that the sorted plurality of training data includes 10,000 pieces of data, the second starting position is 1250, the second ending position is 3249, and the second combined cluster after sorting is combined cluster 2. Based on this, the training data between the 1250th and 3249th pieces of the sorted 10,000 training data are determined as the training data corresponding to combined cluster 2.

[0122] Through the above steps S4331 to S4334, multiple starting positions and ending positions are determined based on the computing power ratio of the sorted combined clusters, that is, the grouping of the sorted multiple training data is achieved to obtain multiple sets of training data, and the number of training data in the training set corresponds to the computing power ratio of the sorted combined clusters, that is, the computing power ratio of each sorted combined cluster is positively correlated with the number of corresponding training data, and the combined cluster with a large computing power ratio corresponds to more training data. In addition, since the sorted multiple training data are the result of sorting in order of occupied space from small to large, the sorted combined clusters are the result of sorting in order of cluster computing power level from small to large, based on this, the computing power level of each sorted combined cluster is positively correlated with the occupied space of the corresponding training data, that is, the combined cluster with a high computing power level corresponds to training data with a large occupied space.

[0123] In some embodiments, after the above-mentioned step S433, the determined training data set will be distributed to the corresponding sorted combination clusters, that is, more training data will be distributed to the combination clusters with a large computing power share, and training data that occupies a large space will be distributed to the combination clusters with a high computing power level. In this way, the corresponding training data can be distributed to the combination clusters according to the actual processing capacity of the combination clusters, and the resources of each combination cluster can be fully utilized to improve the resource utilization rate of each combination cluster.

[0124] Through the above steps S101 to S104, multiple resource clusters and the unit computing power of the computing units included in each resource cluster are obtained, and then the computing units are combined according to the unit computing power of each computing unit to obtain multiple combined clusters, which include computing units with the same or similar unit computing power; the cluster computing power of each combined cluster is also determined; then, multiple training data are obtained, and the multiple training data are sent to each combined cluster based on the cluster computing power of each combined cluster, so as to train the model with the corresponding training data of each combined cluster. In this way, multiple combined clusters associated with the unit computing power of the computing unit can be generated, that is, multiple resource clusters can be recombined according to the unit computing power to achieve the convenience of combination. In addition, multiple training data are distributed to each combined cluster based on the cluster computing power of each combined cluster. That is, when distributing training data, the cluster computing power of the combined cluster itself is taken into consideration, and the corresponding training data can be distributed according to the cluster computing power, thereby reasonably distributing training data to each combined cluster, maximizing the computing power of each combined cluster, and ultimately shortening the training time of the model and improving the training efficiency of the model. In addition, this can distribute the corresponding training data to the combined cluster according to the actual processing power of the combined cluster, make full use of the resources of each combined cluster, and improve the resource utilization rate of each combined cluster.

[0125] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0126] The embodiment of the present application proposes a data distribution method, which is applied to the scenario of training large models. Since the number of parameters in large model training is large and the scale of data exchanged during training is high, the required scale of GPU computing power will reach more than 10,000 cards. In fact, heterogeneous computing power of 10,000 cards will be dispersed in multiple computing power clusters. How to realize the integration of computing power among multiple clusters and activate the computing power of various card types among multiple clusters to support the training calculation of large models is an urgent problem to be solved in large model training. Among them, multiple computing power clusters and multiple clusters correspond to multiple resource clusters in other embodiments.

[0127] The embodiment of the present application starts from the perspective of model data. First, at the basic resource layer, by building a logical virtual cluster, the GPU computing power resources of various card types are integrated. Usually, the same card type will be placed in the same logical virtual cluster. The same card type has the same unit computing power, that is, the GPU computing power resources of the same unit computing power will be integrated into the same logical virtual cluster, and the GPU computing power resources of different unit computing powers will be integrated into different logical virtual clusters. Then, based on the logical virtual cluster, the differentiated distribution of model training data is realized. Here, the differentiated distribution depends on the computing power processing capacity that each logical virtual cluster can provide. The computing power card with high unit computing power has strong processing capacity, so more training data to be processed is allocated. The computing power card with low unit computing power has weaker processing capacity than the same period last year, so the same proportion of model data is allocated to run training calculations. Different logical virtual clusters need to exchange and recalculate the training intermediate data. The embodiment of the present application adopts a reduction method to aggregate and collect the training intermediate state data output by different logical virtual clusters, and redistribute them according to the computing power of the logical virtual cluster. Such cyclic distribution and reduction of model training data activates the computing power of various card types with different unit computing power, realizes the combination of computing power resources for large model training, and reduces the demand for computing cards with high unit computing power. The combined computing power resources can run large model training calculations, which can reduce the cost of large model training. By constructing a logical virtual cluster at the resource layer, integrating multiple heterogeneous computing powers, and grading the computing power processing capabilities of the logical virtual cluster, the distribution and reduction of model training data are realized, and the intermediate state data of the data in the large model training process is distributed and reduced again, so as to achieve the full exchange and calculation of training data or intermediate state data, so as to realize the training of large models. Among them, the model data and the intermediate state data of the data correspond to the training data in other embodiments, the logical virtual cluster corresponds to the combined cluster in other embodiments, and the GPU computing power resources correspond to the computing unit in other embodiments.

[0128] In some embodiments, the schematic diagram of the data distribution framework is as follows: Figure 8 As shown, see Figure 8 The data distribution process includes computing resource integration 801, model training data (re)distribution 802, and model intermediate data reduction and screening 803.

[0129] In some embodiments, computing resource integration 801 is used to implement the integration of multi-card computing resource clusters or multi-cluster computing resource clusters, wherein the multi-card computing resource clusters or multi-cluster computing resource clusters correspond to the multiple resource clusters in other embodiments.

[0130] The embodiment of the present application realizes the integration of multiple cluster computing resource clusters by merging the same unit computing card type. Figure 8, assuming that all cards in resource cluster 1 are of the P series, which are low-unit computing power cards, resource cluster 1 can be integrated into logical virtual cluster A. Based on this, the P series cards in resource cluster 2 will also be integrated into logical virtual cluster A. In some embodiments, if there are non-P series cards in resource cluster 2, such as V series cards in resource cluster 2, the V series cards will be integrated into the new logical virtual cluster.

[0131] Continue to see Figure 8 , all the cards in resource cluster 3 are of the A series, which are high-unit computing power cards. Then the A series cards in resource cluster 3 can be integrated into the logical virtual cluster B. Referring to the unit computing power of GPU training cards in recent years, including M, P, V, A and H series cards, based on this, the number of logical virtual clusters in the embodiment of the present application can be 5. In practice, since the card performance of each unit computing power will be consumed over time, there will be dynamic errors in performance. If the computing power is accurately allocated, there may be a demand for more than 5 logical clusters.

[0132] In some embodiments, since the card scale of each unit computing power is different, the total amount of GPU computing power that each logical virtual cluster can support is also different. The embodiment of the present application takes the single-card computing power of the V series card as the benchmark, and the single-card computing power of other series cards is converted compared with the V series card. The purpose is to anchor the computing power of the single card. The anchored single-card computing power comparison table is shown in Table 1.

[0133] Table 1 Comparison of single card computing power of different card types

[0134]

[0135]

[0136] See Table 1. The computing power of a single card of each series can be determined based on business scenarios, network structures, experience values, etc. The unknown variables in the above table can be determined based on business scenarios, network structures, experience values, etc. For example, the computing power of a single card of an M series computing power card can be 0.4, the computing power of a single card of a P series computing power card can be 0.8, the computing power of a single card of an A series computing power card can be 2.6, and the computing power of a single card of an H series computing power card can be 3.5.

[0137] In some embodiments, the total amount of computing power that can be provided by each logical virtual cluster can be determined by the following formula (5), where the total amount of computing power corresponds to the cluster computing power in other embodiments.

[0138] The total computing power of the logical virtual cluster N = ∑(D i ) formula (5);

[0139] Among them, N is the number of the logical virtual cluster, i is the number of a certain series of cards contained in the logical virtual cluster N. The value obtained by adding up the computing power of each card type is the total computing power of the logical virtual cluster, which can also be called computing power capacity. The total computing power is the basic resource value for model data distribution and reduction.

[0140] In some embodiments, the model training data (re)distribution 802 is mainly based on the total computing power of the logical virtual cluster N, and performs quantitative distribution of model data. The distributed content includes the initial training original data and the intermediate state data of the training process. The intermediate state data will have data attenuation compared to the original data, that is, the data scale decreases. Among them, the initial training original data and the intermediate state data correspond to the training data in other embodiments.

[0141] In some embodiments, the training data is exchanged on each training card in the logical virtual cluster in order to achieve the effect of random sample training and balancing internal resources. The cluster performance data in Table 2 is referenced during the data distribution process, and the performance data in Table 2 will change as the scale of the logical virtual cluster is adjusted.

[0142] Table 2 Logical virtual cluster performance data table

[0143]

[0144]

[0145] Assuming that the total number of data to be distributed is ALL, the amount of data allocated to each logical virtual cluster can be determined by the following formula (6).

[0146]

[0147] Wherein, formula (6) takes the logical virtual cluster M as an example, and the sum of the computing power of all logical virtual clusters can be determined by accumulation, and then the ratio of the total computing power of the logical virtual cluster M to the sum of the computing power is determined as the computing power proportion of the logical virtual cluster M, and finally, the product of the proportion and the total number of data is determined as the number of data distributed to the logical virtual cluster M, which can also be called the number of data to be processed by the logical virtual cluster M. In some embodiments, the method for determining the number of data to be processed by other logical virtual clusters is similar to the method for determining the number of data to be processed by the above-mentioned logical virtual cluster M.

[0148] The model training intermediate data reduction screening 803 is to screen the intermediate state data, and then perform the redistribution process on the screened data, which can be seen from the reference model training data (re)distribution 802. In the embodiment of the present application, the data screening rule is to classify and screen the intervals according to the size of a single data, such as Fig. 9The size of a single piece of data refers to the size of the storage space occupied by a single piece of data. Combined with the computing power of a single card of different card types in Table 1 above, it can be seen that when performing data screening, the size of a single piece of data must be positively correlated with the computing power of a single card in the logical virtual cluster.

[0149] In some embodiments, see Fig. 9 , the training data can be divided into three segments based on the size of a single data, and then the data in each segment can be reduced and filtered into different series of computing cards. Different series of computing cards correspond to different logical virtual clusters, so that each logical virtual cluster can be used to train the large model with the corresponding training data.

[0150] In some embodiments, the threshold of the segment can be configured in advance, and the interval threshold can be configured according to the data set of the large model training and the total computing power provided by the logical virtual cluster. For example, there are 1 million training data entries of the large model, the computing power of the M and P series cards accounts for M%, the computing power of the V series cards accounts for N%, and the computing power of the A and H series cards accounts for P%. After sorting the 1 million data according to the size of a single data, the threshold 1 and threshold 2 corresponding to the V series card can be determined by the following formula (7).

[0151]

[0152] Among them, the fun function refers to taking the 1 million sorted data, threshold 1 refers to the starting position corresponding to the V series card, and the difference between threshold 2 and value 1 can represent the ending position corresponding to the V series card, that is, the data between the starting position corresponding to threshold 1 and the ending position corresponding to threshold 2 of the 1 million sorted data is determined as the data corresponding to the V series card.

[0153] In the embodiments of the present application, Fig.10 As shown, the distribution process of model training data and intermediate state data may include the following steps S1001 to S1009, which are described in detail below.

[0154] In step S1001, the original data for large model training is loaded.

[0155] In some embodiments, the original data for large model training refers to training samples, which include training data and training labels. Exemplarily, the training data may be multiple images, and the training labels may be objects, scenes, or behaviors corresponding to the images.

[0156] In step S1002 , the model data is sorted.

[0157] In the embodiment of the present application, when entering step S1002 from step S1001, the model data refers to the original data for large model training; when entering step S1002 from step S1009, the model data refers to the reduced intermediate data, and the reduced intermediate data refers to the data obtained by mixing the intermediate data generated by each logical virtual cluster together.

[0158] In some embodiments, sorting the model data includes first obtaining the storage space size of each model data, and then sorting the model data in order of data storage space from small to large, thereby obtaining sorted model data.

[0159] In some embodiments, the original training data of the large model can be sorted according to the size of a single piece of data. Since the large model has many training data items, a bitmap sorting algorithm can be used to complete the sorting operation in one poll.

[0160] In step S1003, the computing power ratio of each logical virtual cluster is determined.

[0161] In some embodiments, the computing power share of each logical virtual cluster can be determined by the following formula (8).

[0162]

[0163] Wherein, formula (8) takes the logical virtual cluster M as an example, and the sum of the computing power of all logical virtual clusters can be determined by accumulation, and then the ratio of the total computing power of the logical virtual cluster M to the sum of the computing power is determined as the computing power proportion of the logical virtual cluster M. In some embodiments, the method for determining the computing power proportion of other logical virtual clusters is similar to the method for determining the computing power proportion of the above-mentioned logical virtual cluster M.

[0164] In step S1004 , the threshold of each segment is determined.

[0165] In the embodiment of the present application, the threshold of each segment can be determined by the method of the above formula (7).

[0166] In some embodiments, based on the sorted model data set, the threshold of each segment can be determined by the percentage of computing power of the logical cluster.

[0167] In step S1005, the training data is segmented according to the threshold.

[0168] In the embodiment of the present application, the training data is divided into multiple groups of training data according to the threshold value determined in the above step S1004, wherein the number of groups of training data is the same as the number of logical virtual clusters.

[0169] In step S1006 , the segmented training data is distributed to corresponding logical virtual clusters.

[0170] Exemplarily, the segmented data sets are distributed to the logical virtual clusters corresponding to the M, P, V and A, H series cards, so that the models are trained using the distributed data of each logical virtual cluster.

[0171] In step S1007 , model training is performed using the distributed training data using each logical virtual cluster.

[0172] In step S1008, intermediate state data generated by each logical virtual cluster is obtained.

[0173] In some embodiments, during the training process, the data generated by each logical virtual cluster is called intermediate data.

[0174] In step S1009, the intermediate data is reduced and the process proceeds to step S1002.

[0175] In some embodiments, all the intermediate state data are mixed to obtain the reduced intermediate state data. Then the process goes to step S1002 and continues to perform data sorting, distribution and reduction from step S1002 to step S1008 until the training of the model is completed and the process ends.

[0176] Through the data distribution method provided in the embodiment of the present application, when training a large model control, by building a logical cluster at the resource layer, integrating a variety of heterogeneous computing power, and grading the computing power processing capabilities of the logical cluster, the distribution and reduction of the model training data are realized, and then the intermediate state results of the data in the large model training process are distributed and reduced again, and this cycle is repeated to achieve the full exchange and calculation of training data or intermediate state data, thereby improving the efficiency of large model training. This can not only ensure the active use of computing power resources, but also avoid the challenges of integration and collection at the resource layer, reducing the work of resource integration from monthly to daily level, or even lower time periods. In addition, by activating high and low performance computing cards of different computing power units, there is no need to make new purchases for large model resources, thereby reducing the cost of training.

[0177] It can be understood that in the embodiments of the present application, data related to resource clusters, unit computing power, training data, etc. are involved, and the collection, use and processing of such relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0178] The following further describes the exemplary structure of the data distribution device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data distribution device 455 of the memory 450 may include:

[0179] A first acquisition module 4554 is used to acquire multiple resource clusters and unit computing power of computing units included in each resource cluster;

[0180] A combination module 4552, configured to combine the computing units according to the computing power of the computing units to obtain a plurality of combined clusters, and determine the cluster computing power of each combined cluster;

[0181] The second acquisition module 4553 is used to acquire multiple training data;

[0182] The distribution module 4554 is used to group the multiple training data based on the cluster computing power of each combined cluster, obtain the training data set corresponding to each combined cluster, and distribute the training data set to the corresponding combined cluster, so as to use the corresponding training data set to train the model using the each combined cluster.

[0183] In some embodiments, the combining module 4552 is further configured to:

[0184] Obtaining a preset mapping relationship between unit computing power and unit computing power level, wherein the mapping relationship is a mapping relationship between at least one unit computing power and one unit computing power level;

[0185] Determine the unit computing power level of each computing unit based on the mapping relationship and the unit computing power of each computing unit;

[0186] The computing units with the same computing power level among the computing units are determined as candidate computing units, and the candidate computing units are combined to obtain a combined cluster.

[0187] In some embodiments, the distribution module 4554 is further configured to:

[0188] Sorting the plurality of training data in order of occupied space from small to large to obtain a plurality of sorted training data;

[0189] Based on the cluster computing power of each of the combined clusters, determining the computing power proportion of each of the combined clusters;

[0190] According to the computing power proportions of the respective combined clusters, the sorted plurality of training data are grouped and processed to obtain training data sets corresponding to the respective combined clusters.

[0191] In some embodiments, the distribution module 4554 is further configured to:

[0192] Determine the unit computing power level of the computing units in each combined cluster as the cluster computing power level of the corresponding combined cluster;

[0193] Sorting the combined clusters in ascending order of computing power level of each cluster to obtain sorted combined clusters;

[0194] The plurality of sorted training data are grouped and processed by using the computing power proportions of the sorted combined clusters to obtain training data sets corresponding to the sorted combined clusters.

[0195] In some embodiments, the distribution module 4554 is further configured to:

[0196] Obtaining the total number of the sorted training data;

[0197] Determine the jth starting position based on the sum of the computing power proportions of the first (j-1) sorted combined clusters and the total number of the data, where j is an integer greater than 0 and less than T, and T is the total number of the computing power proportions;

[0198] Determine the jth end position based on the sum of the computing power proportions of the first j sorted combined clusters and the total number of data;

[0199] The training data located between the j-th starting position and the j-th ending position among the sorted plurality of training data are determined as the training data set corresponding to the sorted j-th combined cluster.

[0200] In some embodiments, the distribution module 4554 is further configured to:

[0201] Accumulating the cluster computing power of each combined cluster to obtain a total cluster computing power value;

[0202] The ratios of the cluster computing power of each combined cluster to the total value of the cluster computing power are determined respectively, and each ratio is determined as the computing power proportion of the corresponding combined cluster.

[0203] In some embodiments, the combining module 4552 is further used to:

[0204] Obtain the computing units included in each combined cluster and the unit computing power of each computing unit;

[0205] The unit computing power of the computing units in the same combined cluster is accumulated and summed to obtain the cluster computing power of the combined cluster.

[0206] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the data distribution method described above in the embodiment of the present application.

[0207] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data distribution method provided by the present application embodiment, for example, Figure 3 and Fig.10 The data distribution method shown.

[0208] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0209] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0210] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0211] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0212] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data distribution method, characterized in that: The method comprises: Obtain unit computing power of multiple resource clusters and computing units included in each resource cluster; Combining the computing units according to the computing power of the computing units to obtain multiple combined clusters, and determining the cluster computing power of each combined cluster; Get multiple training data; Based on the cluster computing power of each combined cluster, the multiple training data are grouped and processed to obtain training data sets corresponding to each combined cluster, and the training data sets are distributed to the corresponding combined clusters, so as to train the model using the corresponding training data sets of each combined cluster.

2. The method according to claim 1, characterized in that The respective computing units are combined according to the computing power of the respective computing units to obtain a plurality of combined clusters, including: Obtaining a preset mapping relationship between unit computing power and unit computing power level, wherein the mapping relationship is a mapping relationship between at least one unit computing power and one unit computing power level; Determine the unit computing power level of each computing unit based on the mapping relationship and the unit computing power of each computing unit; The computing units with the same computing power level among the computing units are determined as candidate computing units, and the candidate computing units are combined to obtain a combined cluster.

3. The method according to claim 1, characterized in that The plurality of training data are grouped and processed based on the cluster computing power of each combined cluster to obtain a training data set corresponding to each combined cluster, including: Sorting the plurality of training data in order of occupied space from small to large to obtain a plurality of sorted training data; Based on the cluster computing power of each of the combined clusters, determining the computing power proportion of each of the combined clusters; According to the computing power proportions of the respective combined clusters, the sorted plurality of training data are grouped and processed to obtain training data sets corresponding to the respective combined clusters.

4. The method according to claim 3, characterized in that The sorted multiple training data are grouped and processed according to the computing power proportion of each combined cluster to obtain a training data set corresponding to each combined cluster, including: Determine the unit computing power level of the computing units in each combined cluster as the cluster computing power level of the corresponding combined cluster; Sorting the combined clusters in ascending order of computing power level of each cluster to obtain sorted combined clusters; The plurality of sorted training data are grouped and processed by using the computing power proportions of the sorted combined clusters to obtain training data sets corresponding to the sorted combined clusters.

5. The method according to claim 4, characterized in that The method of grouping the sorted multiple training data using the computing power proportions of the sorted combined clusters to obtain training data sets corresponding to the sorted combined clusters includes: Obtaining the total number of the sorted training data; Determine the jth starting position based on the sum of the computing power proportions of the first (j-1) sorted combined clusters and the total number of the data, where j is an integer greater than 0 and less than T, and T is the total number of the computing power proportions; Determine the jth end position based on the sum of the computing power proportions of the first j sorted combined clusters and the total number of data; The training data located between the j-th starting position and the j-th ending position among the sorted plurality of training data are determined as the training data set corresponding to the sorted j-th combined cluster.

6. The method according to any one of claims 3 to 5, characterized in that The determining, based on the cluster computing power of each combined cluster, the computing power proportion of each combined cluster includes: Accumulating the cluster computing power of each combined cluster to obtain a total cluster computing power value; The ratios of the cluster computing power of each combined cluster to the total value of the cluster computing power are determined respectively, and each ratio is determined as the computing power proportion of the corresponding combined cluster.

7. The method according to any one of claims 1 to 5, characterized in that The determining of the cluster computing power of each combined cluster includes: Obtain the computing units included in each combined cluster and the unit computing power of each computing unit; The unit computing power of the computing units in the same combined cluster is accumulated and summed to obtain the cluster computing power of the combined cluster.

8. A data distribution device, characterized in that: The device comprises: A first acquisition module is used to acquire a plurality of resource clusters and unit computing power of computing units included in each resource cluster; A combination module, used for combining the computing units according to the computing power of the computing units to obtain a plurality of combined clusters, and determining the cluster computing power of each combined cluster; A second acquisition module is used to acquire multiple training data; The distribution module is used to group the multiple training data based on the cluster computing power of each combined cluster, obtain the training data set corresponding to each combined cluster, and distribute the training data set to the corresponding combined cluster, so as to use the corresponding training data set to train the model using the each combined cluster.

9. A computer device, characterized in that: The computer device comprises: A memory for storing computer executable instructions; A processor, configured to implement the data distribution method according to any one of claims 1 to 7 when executing computer executable instructions stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data distribution method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the data distribution method according to any one of claims 1 to 7 is implemented.

Images