Data processing method, device and equipment, readable storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a readable storage medium and a program product, which can be applied to the fields or scenes of cloud technology, artificial intelligence, large models and the like, and the method comprises the following steps: obtaining storage state data of a to-be-trained model and equipment feature data of a plurality of heterogeneous computing equipment, the storage state data comprises a training data volume, and the equipment characteristic data comprises equipment attribute data and equipment state data; and integrating the storage state data and the equipment feature data to obtain spatial dimension statistical data and time dimension statistical data, and performing data analysis on the spatial dimension statistical data and the time dimension statistical data to obtain a data analysis result of the to-be-trained model. The data analysis result is used for adjusting a model training mode of the to-be-trained model and resource configuration of the plurality of heterogeneous computing devices. Through the embodiment of the invention, the data analysis result related to the model training process can be accurately generated, so that cost reduction and benefit improvement of model training are realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a data processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] When training a large artificial intelligence (AI) model, the resource utilization efficiency of the computing resources during the model training process can be analyzed to find the optimization point of model training. At present, the data analysis results related to the model training process are mainly obtained by analyzing the computing power operation during the model training process, but this method cannot truly reflect the overall situation of the model training, resulting in low accuracy of the data analysis results. Therefore, how to accurately generate data analysis results related to the model training process, and then achieve cost reduction and efficiency improvement of model training, is a problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a data processing method, apparatus, device, readable storage medium and program product, which can accurately generate data analysis results related to the model training process, thereby achieving cost reduction and efficiency improvement of model training.

[0004] In a first aspect, the present application provides a data processing method, the method comprising:

[0005] Obtain storage status data of the model to be trained and device characteristic data of multiple heterogeneous computing devices, the storage status data including the amount of training data distributed in each storage cluster, and the device characteristic data including device attribute data and device status data generated by training the model to be trained;

[0006] Performing data integration processing on the storage status data and the device feature data of the plurality of heterogeneous computing devices to obtain spatial dimension statistical data and temporal dimension statistical data corresponding to the model to be trained; the spatial dimension statistical data includes computing power resource load, communication performance load and training data storage volume, and the temporal dimension statistical data includes resource usage status at each statistical moment;

[0007] The spatial dimension statistical data and the time dimension statistical data are subjected to data analysis and processing to obtain data analysis results of the model to be trained. The data analysis results are used to adjust the model training method of the model to be trained and the resource configuration of the multiple heterogeneous computing devices.

[0008] In a second aspect, the present application provides a data processing device, the device comprising:

[0009] An acquisition module is used to acquire storage status data of the model to be trained and device characteristic data of multiple heterogeneous computing devices, wherein the storage status data includes the amount of training data distributed in each storage cluster, and the device characteristic data includes device attribute data and device status data generated by training the model to be trained;

[0010] An integration module is used to perform data integration processing on the storage status data and the device feature data of the plurality of heterogeneous computing devices to obtain spatial dimension statistical data and temporal dimension statistical data corresponding to the model to be trained; the spatial dimension statistical data includes computing resource load, communication performance load and training data storage volume, and the temporal dimension statistical data includes resource usage status at each statistical moment;

[0011] The data analysis module is used to perform data analysis on the above-mentioned spatial dimension statistical data and the above-mentioned time dimension statistical data to obtain the data analysis results of the above-mentioned model to be trained. The above-mentioned data analysis results are used to adjust the model training method of the above-mentioned model to be trained and the resource configuration of the above-mentioned multiple heterogeneous computing devices.

[0012] In a third aspect, the present application provides a computer device, comprising: a processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are interconnected, wherein the storage device stores an executable program code, and the processor is used to call the executable program code to implement the data processing method.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions are executed by a processor to implement the data processing method as described above.

[0014] In a fifth aspect, the present application provides a computer program product, which includes a computer program or computer instructions, and the computer program or computer instructions are executed by a processor to implement the above-mentioned data processing method.

[0015] The embodiment of the present application first obtains the storage state data of the model to be trained and the device feature data of multiple heterogeneous computing devices, then integrates the storage state data and the device feature data of multiple heterogeneous computing devices, obtains the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, and then performs data analysis on it to obtain the data analysis result of the model to be trained. Since the data analysis result is determined based on the spatial dimension statistical data and the time dimension statistical data, and the spatial dimension statistical data and the time dimension statistical data include data of multiple dimensions, such as computing power resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment, this ensures the diversity of the analyzed data and improves the accuracy and credibility of the generated data analysis results related to the model training process. The data analysis results can guide the adjustment and optimization of the model training method for the model to be trained and the resource configuration of multiple heterogeneous computing devices, so that the resources of heterogeneous computing devices can be more reasonably utilized, and the idleness and waste of resources can be reduced, thereby reducing the cost of model training. At the same time, by adjusting the model training method, the training efficiency and training effect of the model can be improved, and finally the cost reduction and efficiency increase of model training are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a schematic diagram of the architecture of a data processing system provided by an exemplary embodiment of the present application;

[0018] Figure 2 It is a flowchart of a data processing method provided by an exemplary embodiment of the present application;

[0019] Figure 3 is a flowchart of another data processing method provided by an exemplary embodiment of the present application;

[0020] Figure 4A is a schematic diagram of determining a target computing power utilization rate provided by an exemplary embodiment of the present application;

[0021] Figure 4B is a schematic diagram of determining a communication performance load provided by an exemplary embodiment of the present application;

[0022] Figure 4C is a schematic diagram of determining the amount of training data storage provided by an exemplary embodiment of the present application;

[0023] Figure 4D It is a schematic diagram of determining the resource usage status at each statistical moment provided by an exemplary embodiment of the present application;

[0024] Figure 4E It is a schematic diagram of generating spatial dimension statistical data and temporal dimension statistical data provided by an exemplary embodiment of the present application;

[0025] Figure 4F It is a schematic diagram of an architecture for constructing data analysis results provided by an exemplary embodiment of the present application;

[0026] Figure 5 is a schematic block diagram of a data processing device provided by an exemplary embodiment of the present application;

[0027] Figure 6 It is a schematic block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] It should be noted that the descriptions of "first", "second", etc. involved in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0030] The embodiments of the present application can be applied to various fields or scenarios such as cloud computing, cloud Internet of Things, cloud games, artificial intelligence, large models, in-vehicle scenes, smart transportation, and assisted driving. Several typical fields or scenarios will be introduced below.

[0031] Cloud computing refers to the delivery and use mode of IT infrastructure, which means obtaining the required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use mode of services, which means obtaining the required services through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the promotion of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model in terms of concept. This application can store data such as storage status data, device feature data, and training data on a cloud server. When the above different data need to be used, it can be directly obtained on the cloud server, which greatly improves the data acquisition speed.

[0032] Cloud IOT aims to connect the information sensed and instructions received by sensor devices in traditional IoT to the Internet, truly realize networking, and achieve massive data storage and computing through cloud computing technology. Due to the characteristics of IoT, objects are connected to each other and the current operating status of each "object" is perceived in real time. A large amount of data information will be generated in this process. How to aggregate this information and how to filter out useful information from the massive information to support decision-making for subsequent development have become key issues affecting the development of IoT. IoT cloud based on cloud computing and cloud storage technology has therefore become a strong support for IoT technology and applications.

[0033] Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games. In a cloud gaming scenario, the game is not played on the player's game terminal, but on a cloud server, which renders the game scene into a video and audio stream and transmits it to the player's game terminal over the network. The player's game terminal does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input commands and send them to the cloud server.

[0034] With the continuous development of AI big models, when executing big model training, due to the large number of parameters in big model training and the high scale of data exchange during training, the scale of GPU computing power required will reach more than 10,000 cards. When running training tasks on such a large-scale GPU computing power device, how to generate data analysis results related to the model training process (indicating the overall characteristics of model training), and perform big model task efficiency and resource efficiency evaluation based on the data analysis results, so as to promote the continuous cost reduction and efficiency improvement of model training, is an urgent problem to be solved in operating big model training. At present, the corresponding data analysis results are mainly obtained by analyzing the operation of computing power during model training, but this method is limited to a single analysis dimension and cannot truly reflect the overall situation of model training, resulting in low accuracy of data analysis results, and unable to formulate cost reduction and efficiency improvement strategies for model training.

[0035] Based on this, the embodiment of the present application proposes to first obtain the storage status data (such as the amount of training data) of the model to be trained and the device feature data (such as device attribute data, device status data) of multiple heterogeneous computing devices; then obtain spatial dimension statistical data and time dimension statistical data by integrating the storage status data and the device feature data; then perform data analysis on the spatial dimension statistical data and the time dimension statistical data to obtain the data analysis results of the model to be trained, which are used to adjust the model training method of the model to be trained and the resource configuration of multiple heterogeneous computing devices. Among them, the spatial dimension statistical data includes the computing power resource load, the communication performance load and the training data storage volume, and the time dimension statistical data includes the resource usage status at each statistical moment.

[0036] The above method analyzes the scenario of large model training from multiple indicators in the spatial dimension and the temporal dimension. Through the resource usage status at each statistical moment in the time dimension, it makes up for the lack of mapping of the model training time dimension in the traditional mode, and improves the accuracy of the generated data analysis results related to the model training process, thereby achieving cost reduction and efficiency improvement of model training based on the data analysis results.

[0037] It can be understood that in the specific implementation of the present application, related data such as storage status data, device characteristic data, training data, computing resource load, communication performance load, training data storage volume, resource usage status at each statistical moment, etc. are involved. When the above embodiments of the present application are applied to specific products or technologies, the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the relevant regions.

[0038] This application will be specifically described by the following embodiments:

[0039] See also Figure 1 , which is a schematic diagram of the architecture of a data processing system provided by an exemplary embodiment of the present application. The data processing system may specifically include a model training device 101 and a server 102. Among them, the model training device 101 and the server 102 are connected through a network, for example, through a local area network, a wide area network, a mobile Internet, etc. The model training device 101 is configured with a model to be trained, and multiple heterogeneous computing devices (such as heterogeneous computing device 1, heterogeneous computing device 2) for performing model training related computing tasks, and may also include multiple storage clusters (such as storage cluster 1, storage cluster 2) for storing training data. The model training device 101 can send the storage status data of the model to be trained during model training and the device feature data of multiple heterogeneous computing devices to the server 102, so that the server 102 can generate the data analysis results of the model to be trained based on the storage status data and the device feature data to guide the subsequent cost reduction and efficiency improvement of the model to be trained. Among them, the model training device 101 can be a terminal device or a server.

[0040] In one embodiment, the model training device 101 can send the storage status data of the model to be trained and the device feature data of multiple heterogeneous computing devices to the server 102, wherein the storage status data includes the amount of training data distributed in each storage cluster, and the device feature data includes device attribute data and device status data generated by training the model to be trained. Then, the server 102 performs data integration processing on the acquired storage status data and the device feature data of multiple heterogeneous computing devices to obtain spatial dimension statistical data and time dimension statistical data corresponding to the model to be trained; wherein the spatial dimension statistical data includes computing power resource load, communication performance load and training data storage volume, and the time dimension statistical data includes resource usage status at each statistical moment. Finally, the server 102 performs data analysis processing on the spatial dimension statistical data and the time dimension statistical data to obtain the data analysis results of the model to be trained, and the data analysis results are used to adjust the model training method of the model to be trained and the resource configuration of multiple heterogeneous computing devices, thereby achieving cost reduction and efficiency improvement of model training.

[0041] The terminal device is also referred to as a terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication device, user agent or user device. The terminal device may be a smart home appliance, a handheld device with wireless communication function (such as a smart phone, a tablet computer), a computing device (such as a personal computer (PC)), a vehicle terminal, an intelligent voice interaction device, a wearable device or other smart device, but is not limited thereto.

[0042] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0043] In one embodiment, the data processing system may also only include the server 102. Based on this architecture, the model to be trained may be configured in the server 102. Then, in the process of training the model to be trained, the server 102 may directly obtain the storage state data of the model to be trained and the device feature data of multiple heterogeneous computing devices, and then perform data integration processing on the storage state data and the device feature data of multiple heterogeneous computing devices to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained; finally, the server 102 performs data analysis processing on the spatial dimension statistical data and the time dimension statistical data to obtain the data analysis result of the model to be trained.

[0044] It is understandable that the system architecture diagram described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, the data processing method provided in the embodiment of the present application can be executed by the server 102, and can also be executed by other servers or server clusters that are different from the server 102 and can communicate with the model training device 101 and / or the server 102. It is known to those skilled in the art that Figure 1The number of model training devices and servers in is only illustrative. According to the business implementation needs, any number of model training devices and servers can be configured. In addition, with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems. In subsequent embodiments, the model training device will refer to the above-mentioned model training device 101, and the server will refer to the above-mentioned server 102, which will not be repeated in subsequent embodiments.

[0045] See also Figure 2 , which is a flow chart of a data processing method provided by an exemplary embodiment of the present application, and takes the method applied to a server as an example for explanation, the method may include the following steps:

[0046] S201. Obtain storage status data of a model to be trained and device feature data of multiple heterogeneous computing devices.

[0047] The model to be trained may refer to a model that needs to analyze the model training process. The model to be trained may be an artificial intelligence model, a natural language model, a deep learning model, a reinforcement learning model, etc. The training data of the model to be trained is stored in multiple storage clusters. The storage cluster may refer to a cluster composed of multiple storage nodes. The storage status data includes the amount of training data distributed in each storage cluster, that is, it records the distribution of the training data of the model to be trained in each storage cluster, such as the amount of training data in each storage cluster at each statistical moment.

[0048] Device feature data is data used to describe the attributes, status and other dimensions of heterogeneous computing devices. Device feature data includes device attribute data and device status data generated by training the model to be trained. Among them, device attribute data can refer to the relevant attribute information of heterogeneous computing devices, such as the model, architecture, memory capacity, and number of heterogeneous computing units included in the heterogeneous computing device; device status data can refer to the working status and operating status of heterogeneous computing devices during training, such as the operating status of the container of the heterogeneous computing device, computing power utilization, the operating temperature of the heterogeneous computing units included in the heterogeneous computing device, data loading bandwidth, data exchange bandwidth, etc.

[0049] In the process of model training, a variety of intensive computing tasks are usually included, such as matrix operations, gradient calculations, and parameter updates, which are relatively time-consuming for training large-scale and complex models. The embodiment of the present application performs computing tasks in the model training process by combining heterogeneous computing devices and heterogeneous computing units, such as the above-mentioned intensive computing tasks. Among them, heterogeneous computing devices refer to multiple computing devices (such as terminal devices, servers, etc. used to perform computing tasks) used in the model training process. Each computing device can have different hardware configurations and computing capabilities, and each heterogeneous computing device can be connected and communicated through a network. A heterogeneous computing unit (also called a heterogeneous computing card) refers to a computing component in a heterogeneous computing device, such as a graphics processing unit (GPU). A heterogeneous computing unit can also include multiple parallel processing units (such as CUDA cores), so that the heterogeneous computing unit has a high degree of parallel computing capabilities, which can be applied to large-scale parallel computing tasks in model training. Heterogeneous computing units can also refer to a variety of computing devices with different computing characteristics, performances, and capabilities, such as TPUs and CPUs.

[0050] By combining heterogeneous computing devices and heterogeneous computing units to perform computing tasks in the model training process, the following advantages are achieved: First, distributed training is achieved. Each heterogeneous computing device independently participates in the training task and processes different data subsets or training subtasks to speed up the training. Second, parallel computing is achieved. Multiple heterogeneous computing units can perform computing tasks in parallel on the same heterogeneous computing device, which can accelerate the model training process and improve computing efficiency. It should be noted that the above method needs to pay attention to resource management and load balancing, and appropriately distribute training data and computing tasks to different heterogeneous computing devices and heterogeneous computing units according to the current computing power and load conditions of the device to maximize the use of computing resources and ensure load balancing.

[0051] In an embodiment of the present application, the server obtains the storage status data of the model to be trained and the device feature data of multiple heterogeneous computing devices for data integration, thereby ensuring the diversity of the dimensions of the analyzed data, and performing data analysis on the integration results after data integration, which can improve the accuracy and credibility of the data analysis results.

[0052] S202: Perform data integration processing on the storage status data and the device feature data of multiple heterogeneous computing devices to obtain spatial dimension statistical data and temporal dimension statistical data corresponding to the model to be trained.

[0053] In an embodiment of the present application, the server can perform data integration processing on the storage state data and the device characteristic data of multiple heterogeneous computing devices. The data integration processing can refer to merging, cleaning, converting, and other processing of the data, so that the integrated data is suitable for the needs of subsequent analysis and processing. Through data integration processing, the storage state data and the device characteristic data of multiple heterogeneous computing devices can be statistically analyzed in two dimensions, namely, the spatial dimension and the time dimension, to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, so as to facilitate subsequent data analysis and processing. The above method makes up for the lack of statistics on the time dimension of model training in the traditional mode, improves the accuracy of the generated data analysis results related to the model training process, and thus realizes cost reduction and efficiency improvement of model training based on the data analysis results.

[0054] Among them, the spatial dimension statistical data includes computing resource load, communication performance load and training data storage volume, and the time dimension statistical data includes the resource usage status at each statistical moment.

[0055] The computing power resource load may refer to: the load of computing resources used by heterogeneous computing devices during model training, such as the target scale of heterogeneous computing units, the target usage of heterogeneous computing devices, the target computing power utilization of heterogeneous computing devices, and the target temperature of heterogeneous computing units. The above-mentioned target scale, target usage, target computing power utilization, and target temperature will be introduced in subsequent embodiments and will not be repeated here. In addition, the computing power resource load may also include the computing power resource conditions of heterogeneous computing units in heterogeneous computing devices, such as the computing power utilization, operating speed, etc. of the CPU, GPU, or TPU in the heterogeneous computing device.

[0056] The communication performance load may refer to: the network bandwidth used to load the training data of the model to be trained (such as data loading bandwidth) and the network bandwidth for data exchange between multiple heterogeneous computing devices (such as data exchange bandwidth). The above-mentioned data loading bandwidth and data exchange bandwidth will be introduced in subsequent embodiments and will not be repeated here. In addition, the communication performance load may also refer to the load conditions of the heterogeneous computing device itself or the communication between heterogeneous computing devices during the model training process, such as data transmission rate, delay, etc.

[0057] The amount of training data storage can refer to: data related to the storage change trend of training data, such as the total amount of training data stored in each storage cluster corresponding to each statistical moment. The amount of training data storage can represent the change trend and fluctuation of the total amount of training data of the model to be trained over time.

[0058] The resource usage status at each statistical moment may refer to: the running statistics of the model to be trained in the time dimension, including the training calculation time period, the non-training calculation time period, and the second superimposed computing power utilization corresponding to multiple statistical moments. The above-mentioned training calculation time period, the non-training calculation time period, and the second superimposed computing power utilization corresponding to multiple statistical moments will be introduced in subsequent embodiments and will not be repeated here. In addition, the resource usage status at each statistical moment may also refer to the changing trends and fluctuations of the load, memory occupancy, etc. of the CPU, GPU, TPU of heterogeneous computing devices over time.

[0059] S203: Perform data analysis on the spatial dimension statistical data and the temporal dimension statistical data to obtain data analysis results of the model to be trained.

[0060] In an embodiment of the present application, the server can perform data analysis and processing on spatial dimension statistical data (including computing power resource load, communication performance load and training data storage volume) and time dimension statistical data (including resource usage status at each statistical moment). The data analysis and processing can be statistics, analysis and mining of spatial dimension statistical data and time dimension statistical data, extracting useful information that can be used to guide the model training method and the resource configuration of heterogeneous computing devices, and finally obtaining the data analysis results of the model to be trained. Since the data analysis results are obtained by data analysis and processing based on multiple types of spatial dimension statistical data and time dimension statistical data, and the spatial dimension statistical data and time dimension statistical data are obtained by data integration processing based on the storage status data of the model to be trained and the device feature data of multiple heterogeneous computing devices, this ensures the diversity of the analyzed data and improves the accuracy and credibility of the data analysis results. The data analysis results can guide the adjustment and optimization of the model training method for the model to be trained and the resource configuration of multiple heterogeneous computing devices, which can reduce the cost of model training and improve the training efficiency and training effect of the model, that is, achieve cost reduction and efficiency improvement of model training.

[0061] In one embodiment, the data analysis result may be an analysis report. The data analysis result may refer to: the result obtained by analyzing the resource efficiency (such as spatial dimension statistics and time dimension statistics) in the model training process to indicate the defects in the model training method and resource configuration. The data analysis result may be a quantitative analysis result of the computing resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment, or a description and evaluation of resource utilization status, data transmission bottlenecks and other issues. The data analysis results are used to adjust the model training method of the model to be trained and the resource configuration of multiple heterogeneous computing devices, and make decisions and guide the subsequent optimization of the model training process.

[0062] In one embodiment, the data analysis results may indicate defects and problems in the model training process, and the server may formulate a strategy for reducing costs and increasing efficiency of model training based on the data analysis results.

[0063] For example, the data analysis results can indicate the resource utilization at various time points during the model training process, such as the large fluctuation of GPU computing power utilization over time. Then, the server can use the data analysis results to optimize the calculation process of the model to be trained and modify the algorithm in the model to be trained.

[0064] For example, the data analysis results may also indicate an imbalance in resource utilization, such as when the computing power utilization of some heterogeneous computing devices is too high, while the computing power utilization of other heterogeneous computing devices is too low (or idle). Then, the server can use the data analysis results to evenly distribute the training tasks to each heterogeneous computing device to balance the resource utilization of each heterogeneous computing device, and can dynamically adjust and distribute the training tasks according to the performance and load of each heterogeneous computing device.

[0065] For example, the data analysis results may also indicate that a bottleneck has occurred in data transmission, such as when the usage of data loading bandwidth (or data exchange bandwidth) reaches the maximum value at a certain moment. Then, the server can use the data analysis results to fully consider the communication and data transmission capabilities between heterogeneous computing devices, and try to reduce the data transmission volume and communication overhead of the task when allocating training tasks. At the same time, the data transmission strategy can be optimized, such as using distributed storage systems, data compression, and incremental data transmission technologies.

[0066] For example, the data analysis results can also indicate the characteristics of changes in the amount of training data storage, such as a sharp increase in the amount of training data storage within a certain time interval. Then, the server can use the data analysis results to schedule idle heterogeneous computing devices to execute training tasks to meet the training needs of larger amounts of training data. At the same time, the amount of training data can be reduced through data screening, cleaning and other technologies to ensure the quality of training data, thereby ensuring the training effect of the model.

[0067] For example, the data analysis results can also indicate the changing characteristics of resource usage status, such as the proportion of training computing time periods is small, and the proportion of non-training computing time periods is large. Then, the server can use the data analysis results to increase the density of training tasks, which can increase the proportion of training computing time periods, thereby improving resource utilization.

[0068] Through the above method, we can achieve cost reduction and efficiency improvement of model training based on data analysis results, and reduce idle and waste resources by more rationally utilizing heterogeneous computing device resources, thereby reducing the cost of model training. At the same time, by optimizing data transmission and resource utilization, reducing transmission time and bandwidth consumption, we can improve the speed and efficiency of model training, thereby ensuring the effect of model training.

[0069] Based on the above embodiments, the beneficial effect of the present application is that: the embodiment of the present application first obtains the storage state data of the model to be trained and the device feature data of multiple heterogeneous computing devices, and then integrates the storage state data and the device feature data of multiple heterogeneous computing devices to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, and then performs data analysis on it to obtain the data analysis result of the model to be trained. Since the data analysis result is determined based on the spatial dimension statistical data and the time dimension statistical data, and the spatial dimension statistical data and the time dimension statistical data include data of multiple dimensions, such as computing power resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment, this ensures the diversity of the analyzed data and improves the accuracy and credibility of the generated data analysis results related to the model training process. The data analysis results can guide the adjustment and optimization of the model training method for the model to be trained and the resource configuration of multiple heterogeneous computing devices, so that the resources of heterogeneous computing devices can be more reasonably utilized, the idleness and waste of resources can be reduced, and the cost of model training can be reduced. At the same time, by adjusting the model training method, the training efficiency and training effect of the model are improved, and finally the cost reduction and efficiency increase of model training are achieved.

[0070] See also Figure 3 , which is a flow chart of another data processing method provided by an exemplary embodiment of the present application, and takes the method applied to a server as an example for explanation, the method may include the following steps:

[0071] S301. Obtain storage status data of the model to be trained and device characteristic data of multiple heterogeneous computing devices, where the device characteristic data includes the number of heterogeneous computing units, the operating status of the container, computing power utilization, the operating temperature of the heterogeneous computing units, data loading bandwidth, and data exchange bandwidth.

[0072] In the embodiment of the present application, the device characteristic data includes device attribute data and device status data. The device attribute data includes the number of the above-mentioned heterogeneous computing units. The device status data includes the operating status of the above-mentioned container, computing power utilization, operating temperature of the heterogeneous computing units, data loading bandwidth, and data exchange bandwidth. The storage status data includes the amount of training data distributed in each storage cluster. The device characteristic data includes device attribute data and device status data generated by training the model to be trained.

[0073] The number of heterogeneous computing units may refer to the number of heterogeneous computing units included in each heterogeneous computing device.

[0074] The running status of the container may refer to the running status of the container of each heterogeneous computing device. When the model training is run in the form of a container on a heterogeneous computing device, it is possible to determine whether the heterogeneous computing device is being used by checking whether there is a running status of the container on the heterogeneous computing device. If the running status of the container of the heterogeneous computing device is running, it means that the heterogeneous computing device is in use.

[0075] The computing power utilization rate may refer to the computing power utilization rate of each heterogeneous computing unit at each statistical moment, or may refer to the computing power utilization rate of all heterogeneous computing devices as a whole included in each heterogeneous computing device at each statistical moment.

[0076] The operating temperature of the heterogeneous computing unit may refer to the device temperature of the heterogeneous computing unit when it is working.

[0077] The data loading bandwidth may refer to the network bandwidth used to load the training data of the model to be trained.

[0078] Data exchange bandwidth may refer to the network bandwidth for data exchange between multiple heterogeneous computing devices.

[0079] In one embodiment, each heterogeneous computing device includes multiple heterogeneous computing units, the device attribute data includes the number of heterogeneous computing units, and the device status data includes the operating status of the container, computing power utilization, operating temperature of the heterogeneous computing units, data loading bandwidth, and data exchange bandwidth. Based on this, the method in step S202 will be described in detail through steps S302-S305:

[0080] S302: Perform data integration processing on the number of heterogeneous computing units, the operating status of the container, the computing power utilization rate, and the operating temperature of the heterogeneous computing units to obtain the computing power resource load.

[0081] In one embodiment, the computing resource load includes the target scale of the heterogeneous computing unit, the target usage of the heterogeneous computing device, the target computing utilization rate of the heterogeneous computing device, and the target temperature of the heterogeneous computing unit. Then, the above step S302 can be implemented according to the following steps:

[0082] (1) Determine the target scale of the heterogeneous computing units based on the number of heterogeneous computing units included in each heterogeneous computing device.

[0083] In one embodiment, the server may respectively count the number of heterogeneous computing units included in each heterogeneous computing device, and then sum the number of heterogeneous computing units included in each heterogeneous computing device to obtain a target scale of the heterogeneous computing units.

[0084] The number of heterogeneous computing units included in each heterogeneous computing device can be obtained by the server from the resource management cluster. Usually, there are 8 heterogeneous computing units (such as heterogeneous cards) in a heterogeneous computing device in a data center. Then the target scale (denoted as M) can be calculated by the following formula:

[0085] M=∑(Node (i) )*8

[0086] Among them, Node (i) is the ith heterogeneous computing device, ∑(Node (i) ) is the cumulative number of heterogeneous computing devices. It should be noted that when the number of heterogeneous computing units in the heterogeneous computing devices is inconsistent, the number of heterogeneous computing units in each heterogeneous computing device can be counted separately and then summed to obtain the target scale.

[0087] (2) Determine the heterogeneous computing devices whose containers are in the running state as the heterogeneous computing devices in use among the multiple heterogeneous computing devices, and determine the target usage of the heterogeneous computing devices according to the number of the heterogeneous computing devices in use.

[0088] In one embodiment, the server may first obtain the running status (such as running state, stopped state, paused state, deleted state) of the containers in each heterogeneous computing device, and then the server determines the heterogeneous computing devices whose running status is running state as the heterogeneous computing devices in use, and finally sums the number of heterogeneous computing devices in use to obtain the target usage of the heterogeneous computing devices.

[0089] (3) The computing power utilization rates of the heterogeneous computing devices in use at the same statistical moment are superimposed to obtain a first superimposed computing power utilization rate, and the first superimposed computing power utilization rate corresponding to each statistical moment is determined as the target computing power utilization rate of the heterogeneous computing device.

[0090] In one embodiment, the statistical moment may refer to multiple time points at fixed time intervals, such as the first second, the second second, and the third second. The server may determine the computing power utilization of each heterogeneous computing device in use at each statistical moment, and the computing power utilization of each heterogeneous computing device in use may refer to the overall computing power utilization of each heterogeneous computing unit included in each heterogeneous computing device in use. Then, the server superimposes the computing power utilization of each heterogeneous computing device in use at the same statistical moment to obtain a first superimposed computing power utilization, and then integrates the first superimposed computing power utilization corresponding to each statistical moment to obtain the target computing power utilization of the heterogeneous computing device.

[0091] For example, Figure 4A As shown, Figure 4A It is a schematic diagram of determining the target computing power utilization provided by an exemplary embodiment of the present application, wherein the horizontal axis is time and the vertical axis is the computing power utilization in percentage. Broken line 1, broken line 2, and broken line 3 respectively represent the changes in the computing power utilization of three heterogeneous computing devices in use over time. Broken line 4 represents the changes in the superimposed computing power utilization obtained after the computing power utilization of the three heterogeneous computing devices in use is superimposed over time. For example, at the statistical time t=1, the computing power utilization of the three heterogeneous computing devices in use is superimposed to obtain the first superimposed computing power utilization (i.e., the superimposed computing power utilization corresponding to broken line 4 at t=1). Then, the first superimposed computing power utilization corresponding to each statistical moment is determined as the target computing power utilization of the heterogeneous computing device, such as the target computing power utilization includes the first superimposed computing power utilization corresponding to 10 statistical moments from 0 to 9 at the statistical moment t.

[0092] (4) Determine the operating temperature of the heterogeneous computing units included in each heterogeneous computing device in use, and determine the highest operating temperature among the operating temperatures of the heterogeneous computing units as the target temperature of the heterogeneous computing units.

[0093] The operating temperature is a key physical property of a heterogeneous computing unit (heterogeneous card). If the operating temperature exceeds a critical value (such as 90 degrees), the heterogeneous card will drop (i.e., the heterogeneous computing unit will stop working). In addition, the server can also take other measures to protect the hardware when the operating temperature exceeds the critical value, such as automatically adjusting the operating frequency, limiting power consumption, terminating the current computing task, etc. These protection measures are intended to prevent hardware damage or performance degradation due to overheating. Based on this, the embodiment of the present application uses the highest operating temperature among the operating temperatures of each heterogeneous computing unit as the target temperature of the heterogeneous computing unit during the model training process.

[0094] The computing power resource load obtained through the above steps (1)-(4) can reflect the resource load of heterogeneous computing power and the computing power consumption of model training business. Based on the computing power resource load, the data analysis results of the model to be trained are generated, and the cost reduction and efficiency improvement strategy of the model can be evaluated. For example, if the target computing power utilization rate of heterogeneous computing devices is unevenly distributed, targeted optimization strategies can be implemented for heterogeneous computing devices (or heterogeneous computing units in heterogeneous computing devices, such as heterogeneous cards) with low utilization.

[0095] S303: Perform data integration processing on the data loading bandwidth and the data exchange bandwidth to obtain a communication performance load.

[0096] In one embodiment, the data loading bandwidth is the network bandwidth used to load the training data of the model to be trained, the data exchange bandwidth is the network bandwidth for data exchange between multiple heterogeneous computing devices, and the communication performance load includes the data loading peak bandwidth and the data loading mean bandwidth corresponding to the data loading bandwidth, and the data exchange peak bandwidth and the data exchange mean bandwidth corresponding to the data exchange bandwidth. Then, the above step S303 can be implemented according to the following steps:

[0097] (1) The data loading bandwidths of the heterogeneous computing devices in use at the same statistical moment are superimposed to obtain the superimposed bandwidth, and the maximum superimposed bandwidth among the superimposed bandwidths corresponding to the statistical moments is determined as the data loading peak bandwidth.

[0098] Data loading bandwidth mainly focuses on the loading communication of training data. Data loading bandwidth is usually transmitted over the network through TCP. This communication usually reads data from the storage layer of training data and loads it into the memory of heterogeneous computing devices. A typical scenario can be: training data is stored in a distributed file system (ceph), and heterogeneous computing units (such as GPUs) read data from the distributed file system through the POSIX interface before calculation to complete the loading operation of training data. The time granularity of collection (such as statistical moment) can be collected at the minute (or second) granularity. The collected data includes data loading peak bandwidth and data loading mean bandwidth. The data loading peak bandwidth is the maximum superposition bandwidth among the superposition bandwidths corresponding to each statistical moment.

[0099] For example, Figure 4B As shown, Figure 4BIt is a schematic diagram of determining the communication performance load provided by an exemplary embodiment of the present application, wherein the horizontal axis is time and the vertical axis is data loading bandwidth. Broken line 5, broken line 6, and broken line 7 respectively represent the data loading bandwidth of three heterogeneous computing devices in use changing over time. Broken line 4 represents the superposition bandwidth obtained after the data loading bandwidth of the three heterogeneous computing devices in use is superimposed and processed over time. For example, at the statistical time t=1, the data loading bandwidths of the three heterogeneous computing devices in use are superimposed to obtain the superposition bandwidth (that is, the superposition bandwidth corresponding to broken line 8 at t=1). Then, the maximum superposition bandwidth among the superposition bandwidths corresponding to each statistical time is determined as the data loading peak bandwidth. For example, if the superposition bandwidth at the statistical time t=6 is the maximum superposition bandwidth, then the superposition bandwidth at the statistical time t=6 is used as the data loading peak bandwidth.

[0100] (2) Performing mean calculation processing on the data loading bandwidth of each heterogeneous computing device in use at each statistical moment, and obtaining the data loading mean bandwidth.

[0101] In one embodiment, the server may calculate the average data loading bandwidth of each heterogeneous computing device in use at each statistical moment, and then average the average data loading bandwidth corresponding to each heterogeneous computing device in use to obtain the data loading average bandwidth. The average data loading bandwidth corresponding to each heterogeneous computing device in use is denoted as D, which can be calculated by the following formula:

[0102]

[0103] Among them, ∑(Net (i) ) is the cumulative value of the data loading bandwidth at each statistical moment, and ∑(i) is the number of statistical moments, such as 10 statistical moments (10 minutes).

[0104] In one embodiment, if Figure 4B As shown, the server can perform mean calculation processing on the data loading bandwidths at each statistical moment corresponding to the broken lines 5, 6 and 7 in the box (including a total of 30 bandwidth values ​​corresponding to the data loading bandwidths) to obtain the data loading mean bandwidth.

[0105] (3) Perform network interruption detection on each heterogeneous computing device in use, determine the data exchange traffic between each heterogeneous computing device, and perform traffic analysis and processing on the data exchange traffic to obtain the data exchange peak bandwidth and the data exchange mean bandwidth.

[0106] In addition to data loading bandwidth, data exchange bandwidth also needs to be counted. Data exchange bandwidth mainly focuses on the exchange performance of model data during training. However, since the data exchange of the model adopts the remote direct memory access (RDMA) communication strategy, this technology allows computing devices (such as heterogeneous computing devices) to directly transfer data between memories, bypassing the limitations of the traditional operating system network protocol stack, thereby achieving low-latency and high-bandwidth data transmission. There is a pulse-like second-level bandwidth in the communication of model data exchange, and bandwidth statistics require the mirroring operation of network port traffic. Specifically, when using the RDMA communication strategy, the communication network card of the heterogeneous device will copy the data exchange traffic to the peripheral CPU device by listening to network interrupts.

[0107] The mirroring operation of the network port traffic can provide visualization and detection of data interaction, and help understand the communication between heterogeneous computing devices, such as communication performance and bandwidth usage. In this way, the peripheral CPU device can obtain the data exchange traffic between heterogeneous devices. Then, the data exchange traffic can be analyzed and processed by the peripheral CPU device to obtain the data exchange peak bandwidth and the data exchange mean bandwidth. Among them, the method for generating the data exchange peak bandwidth and the data exchange mean bandwidth can refer to the method for generating the data loading peak bandwidth and the data loading mean bandwidth in the aforementioned embodiment, which will not be repeated here.

[0108] In one embodiment, the communication performance load also includes data loading peak bandwidth ratio, data loading average bandwidth ratio, data exchange peak bandwidth ratio and data exchange average bandwidth ratio. Then, the server may further perform the following steps:

[0109] (1) According to the data loading peak bandwidth, the data loading average bandwidth, and the sum of the data loading bandwidths of each heterogeneous computing device in use, determine the data loading peak bandwidth ratio and the data loading average bandwidth ratio.

[0110] Exemplarily, the server can divide the data loading peak bandwidth by the sum of the data loading bandwidths of all heterogeneous computing devices in use to obtain the data loading peak bandwidth ratio, and divide the data loading mean bandwidth by the sum of the data loading bandwidths of all heterogeneous computing devices in use to obtain the data loading mean bandwidth ratio.

[0111] (2) Perform traffic analysis on the data exchange traffic to obtain the data exchange peak bandwidth ratio and the data exchange mean bandwidth ratio.

[0112] Exemplarily, the server can perform traffic analysis on the data exchange traffic through a data analysis device (such as a peripheral CPU device) to obtain the data exchange peak bandwidth ratio and the data exchange mean bandwidth ratio. The data exchange peak bandwidth ratio can be obtained by dividing the data exchange peak bandwidth by the sum of the data exchange bandwidths of the heterogeneous computing devices in use, and dividing the data exchange mean bandwidth by the sum of the data exchange bandwidths of the heterogeneous computing devices in use, to obtain the data exchange mean bandwidth ratio.

[0113] In one embodiment, if the difference between the data loading peak bandwidth share and the data loading mean bandwidth share is greater than the first share threshold, a first optimization prompt information is output; the first optimization prompt information is used to indicate that the data loading method of the training model is optimized. If the difference between the data exchange peak bandwidth share and the data exchange mean bandwidth share is greater than the second share threshold, a second optimization prompt information is output, and the second optimization prompt information is used to indicate that the data exchange method of the training model is optimized. The first share threshold and the second share threshold can be the same (such as both are 20%), or they can be different (such as 20% and 30% respectively).

[0114] By using the above method to generate the data loading peak bandwidth ratio, data loading average bandwidth ratio, data exchange peak bandwidth ratio and data exchange average bandwidth ratio, the performance points that can be optimized in communication can be counted. If the difference between the above peak and average ratios is large (such as the difference between the data loading peak bandwidth ratio and the data loading average bandwidth ratio is greater than 20%, and the difference between the data exchange peak bandwidth ratio and the data exchange average bandwidth ratio is greater than 30%), it means that the bandwidth usage is highly volatile, and the computing pipeline of the model to be trained needs to be optimized, such as optimizing the computing process of the model to be trained.

[0115] S304: Perform data integration processing on the storage status data to obtain the storage capacity of training data.

[0116] In one embodiment, the above step S304 can be implemented according to the following steps:

[0117] (1) The amount of training data distributed in each storage cluster at the same statistical moment is summed to obtain the total amount of training data.

[0118] (2) The total amount of training data corresponding to each statistical moment is determined as the training data storage amount.

[0119] In the above steps (1)-(2), the server can sum up the amount of training data distributed in each storage cluster at each statistical moment to obtain the total amount of training data corresponding to each statistical moment. Then, the total amount of training data corresponding to each statistical moment is determined as the training data storage amount. The change in the training data storage amount of the model to be trained can be determined by the trend of the training data storage amount.

[0120] See also Figure 4C , Figure 4C This is a schematic diagram of determining the amount of training data storage provided by an exemplary embodiment of the present application, wherein the horizontal axis is time (such as in days) and the vertical axis is the total amount of training data (usually in units of G or T). Broken line 9 represents how the total amount of training data corresponding to each storage cluster changes over time. For example, from day 0 to day 9, the total amount of training data corresponding to each of these 10 days is used as the amount of training data storage. As can be seen from the figure, the total amount of training data of the model to be trained is stable in the early stage, but the storage amount continues to increase in the later training, indicating that more training data was added in the later stage, and the purpose may be to more accurately calibrate the model to be tested.

[0121] S305: Perform data integration processing on the computing power utilization rate to obtain the resource usage status at each statistical moment.

[0122] In one embodiment, the above step S305 can be implemented according to the following steps:

[0123] (1) The computing power utilization rates of multiple heterogeneous computing devices at the same statistical moment are superimposed to obtain a second superimposed computing power utilization rate.

[0124] Among them, one training task corresponds to multiple heterogeneous computing devices that need to be used, and the above-mentioned multiple heterogeneous computing devices refer to multiple heterogeneous computing devices corresponding to one training task. The method of generating the second superposition computing power utilization rate in the above steps refers to the method of generating the first superposition computing power utilization rate in step S302, which will not be repeated here.

[0125] (2) The utilization rates of the second superposition computing power corresponding to the multiple statistical moments are divided and processed by the utilization rate threshold to obtain the division results, and the training computing time period and the non-training computing time period of the multiple heterogeneous computing devices are determined according to the division results.

[0126] The server divides the second superposition computing power utilization corresponding to multiple statistical moments by a utilization threshold (such as 5%) to obtain a division result, and the division result indicates whether the second superposition computing power utilization corresponding to each statistical moment is greater than the utilization threshold. Then, the server determines the training computing time period and non-training computing time period of multiple heterogeneous computing devices based on the division result, such as determining the time interval corresponding to the statistical moment when the second superposition computing power utilization is greater than the utilization threshold as the training computing time period, and determining the time interval corresponding to the statistical moment when the second superposition computing power utilization is less than or equal to the utilization threshold as the non-training computing time period. Among them, the training computing time period indicates that the heterogeneous computing device is performing heterogeneous computing, and the non-training computing time period indicates that the heterogeneous computing device is not performing heterogeneous computing (or there is a small amount of heterogeneous computing), and may be in the process of data loading, data effect analysis, etc.

[0127] (3) Determine the resource usage status at each statistical moment according to the second superposition computing power utilization rate, the training computing time period, and the non-training computing time period corresponding to the multiple statistical moments.

[0128] Among them, the computing power resource load, communication performance load and training data storage volume in the above steps S302-S304 are spatial dimension statistical data, and the resource usage status at each statistical moment in the above step S305 is time dimension statistical data. Time dimension statistical data can more specifically show the resource usage of heterogeneous computing power during model training. Although the running state of the container that processes the model training task is created on the heterogeneous computing device, whether the heterogeneous computing device is executing the heterogeneous computing task normally requires the construction of the resource usage of the time dimension, that is, the resource usage status at each statistical moment.

[0129] See also Figure 4D , Figure 4D It is a schematic diagram of determining the resource usage status at each statistical moment provided by an exemplary embodiment of the present application, wherein the horizontal axis is time (such as in minutes) and the vertical axis is the superposition computing power utilization. The time period corresponding to the portion of the broken line 10 that is lower than or equal to the utilization threshold (such as 5%) is the non-training calculation time period, and the time period corresponding to the portion of the broken line 10 that is higher than the utilization threshold (such as 5%) is the training calculation time period. The superposition computing power utilization corresponding to different time periods in the figure is different, which can indicate that there are differences in the training data sets corresponding to different time periods. When the training data set is sparse, the utilization rate is low, and when the training data set is dense, the utilization rate is high.

[0130] See also Figure 4E , Figure 4EThis is a schematic diagram of generating spatial dimension statistics and time dimension statistics provided by an exemplary embodiment of the present application. In the process of training the training model, the server first performs the acquisition of computing power resource load. When the computing power resource load can be obtained normally, the target scale, target usage, target computing power utilization, target temperature and other information are recorded (if the computing power resource load cannot be obtained normally, it is necessary to re-acquire). Then, the server performs the acquisition of communication performance load. When the communication performance load can be obtained normally, the data loading bandwidth, data exchange bandwidth and other information are recorded, including the peak value, mean value and proportion corresponding to the data loading bandwidth, and the peak value, mean value and proportion corresponding to the data exchange bandwidth (if the communication performance load cannot be obtained normally, it is necessary to re-acquire). The server then performs the acquisition of training data storage capacity. When the training data storage capacity can be obtained normally, the training data storage capacity and other information are recorded (if the training data storage capacity cannot be obtained normally, it is necessary to re-acquire). Finally, the server performs the acquisition of resource usage status at each statistical moment. When the resource usage status at each statistical moment can be obtained normally, the resource usage status at each statistical moment is recorded (if the resource usage status at each statistical moment cannot be obtained normally, it is necessary to re-acquire). Finally, we obtain the computing resource load, communication performance load, training data storage capacity, and resource usage status at each statistical moment.

[0131] See also Figure 4F , Figure 4F This is an architectural diagram of building data analysis results provided by an exemplary embodiment of the present application. The server first obtains the storage status data of the model to be trained and the device feature data of multiple heterogeneous computing devices, and then obtains the computing power resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment by integrating the storage status data and the device feature data. Among them, the computing power resource load is determined by the target scale of the heterogeneous computing unit, the target usage of the heterogeneous computing device, the target computing power utilization rate of the heterogeneous computing device, and the target temperature of the heterogeneous computing unit; the communication performance load is determined by the data loading peak bandwidth and the data loading mean bandwidth (and their respective corresponding proportions) corresponding to the data loading bandwidth, and the data exchange peak bandwidth and the data exchange mean bandwidth (and their respective corresponding proportions) corresponding to the data exchange bandwidth; the training data storage volume is determined by the amount of training data in each storage cluster at each statistical moment. Finally, the server performs data analysis and processing on the computing power resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment to obtain the data analysis results of the model to be trained.

[0132] S306: Perform data analysis and processing on the computing resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment to obtain data analysis results of the model to be trained.

[0133] Among them, computing resource load, communication performance load, and training data storage volume are spatial dimension statistics, and resource usage status at each statistical moment is temporal dimension statistics. The data analysis results are used to adjust the model training method of the training model and the resource configuration of multiple heterogeneous computing devices.

[0134] Among them, computing power resource load can refer to: the load of computing resources used by heterogeneous computing devices during model training, such as the target scale of heterogeneous computing units, the target usage of heterogeneous computing devices, the target computing power utilization of heterogeneous computing devices, and the target temperature of heterogeneous computing units.

[0135] The communication performance load may refer to: the network bandwidth used to load the training data of the model to be trained (such as data loading bandwidth) and the network bandwidth for data exchange between multiple heterogeneous computing devices (such as data exchange bandwidth).

[0136] The amount of training data storage can refer to: data related to the storage change trend of training data, such as the total amount of training data stored in each storage cluster corresponding to each statistical moment. The amount of training data storage can represent the change trend and fluctuation of the total amount of training data of the model to be trained over time.

[0137] The resource usage status at each statistical moment may refer to: the operation statistics of the model to be trained in the time dimension, including the training calculation time period, the non-training calculation time period, and the second superposition computing power utilization corresponding to multiple statistical moments.

[0138] In one embodiment, if the data analysis results indicate that the target computing power utilization is unevenly distributed, the distribution method of the training data of the heterogeneous computing devices can be optimized. If the data analysis results indicate that there is a long non-heterogeneous computing operation period, the number of heterogeneous computing devices allocated to the model to be analyzed can be increased, thereby achieving cost reduction optimization of model training. The data analysis results can also indicate defects and problems in the model training process, and the server can formulate a strategy for reducing costs and increasing efficiency of model training based on the data analysis results.

[0139] For example, the data analysis results may also indicate an imbalance in resource utilization, such as when the computing power utilization of some heterogeneous computing devices is too high, while the computing power utilization of other heterogeneous computing devices is too low (or idle). Then, the server can use the data analysis results to evenly distribute the training tasks to each heterogeneous computing device to balance the resource utilization of each heterogeneous computing device, and can dynamically adjust and distribute the training tasks according to the performance and load of each heterogeneous computing device.

[0140] For example, the data analysis results may also indicate that a bottleneck has occurred in data transmission, such as when the usage of data loading bandwidth (or data exchange bandwidth) reaches the maximum value at a certain moment. Then, the server can use the data analysis results to fully consider the communication and data transmission capabilities between heterogeneous computing devices, and try to reduce the data transmission volume and communication overhead of the task when allocating training tasks. At the same time, the data transmission strategy can be optimized, such as using distributed storage systems, data compression, and incremental data transmission technologies.

[0141] For example, the data analysis results can also indicate the characteristics of changes in the amount of training data storage, such as a sharp increase in the amount of training data storage within a certain time interval. Then, the server can use the data analysis results to schedule idle heterogeneous computing devices to execute training tasks to meet the training needs of larger amounts of training data. At the same time, the amount of training data can be reduced through data screening, cleaning and other technologies to ensure the quality of training data, thereby ensuring the training effect of the model.

[0142] For example, the data analysis results can also indicate the changing characteristics of resource usage status, such as the proportion of training computing time periods is small, and the proportion of non-training computing time periods is large. Then, the server can use the data analysis results to increase the density of training tasks, which can increase the proportion of training computing time periods, thereby improving resource utilization.

[0143] Through the above method, we can achieve cost reduction and efficiency improvement of model training based on data analysis results, and reduce idle and waste resources by more rationally utilizing heterogeneous computing device resources, thereby reducing the cost of model training. At the same time, by optimizing data transmission and resource utilization, reducing transmission time and bandwidth consumption, we can improve the speed and efficiency of model training, thereby ensuring the effect of model training.

[0144] In one embodiment, the server may further perform the following steps: obtaining preliminary analysis results for the spatial dimension statistical data and the temporal dimension statistical data, wherein the preliminary analysis results are inferred by the operation and maintenance personnel based on the spatial dimension statistical data and the temporal dimension statistical data, or the preliminary analysis results are obtained by performing data analysis processing on the spatial dimension statistical data and the temporal dimension statistical data by a data analysis model.

[0145] Based on this, the above step S306 can be implemented according to the following steps: use an artificial intelligence model or a data analysis algorithm to perform data analysis and processing on the spatial dimension statistical data, the time dimension statistical data and the preliminary analysis results to obtain the data analysis results of the model to be trained.

[0146] Among them, the preliminary analysis results are inferences or conclusions drawn based on the spatial dimension statistical data and the time dimension statistical data, which are used to further understand the trends, laws or key features of the data. The embodiment of the present application uses an artificial intelligence model or a data analysis algorithm to perform data analysis and processing on the spatial dimension statistical data, the time dimension statistical data and the preliminary analysis results, which can further explore the potential value and relevant information of the data, and provide more comprehensive, accurate and reliable data analysis results for model training. The preliminary analysis results are used together with the spatial dimension statistical data and the time dimension statistical data to perform data analysis and processing, thereby improving the accuracy of the data analysis results obtained by the analysis, thereby achieving cost reduction and efficiency improvement of model training based on the data analysis results.

[0147] The following will take the model to be trained as a natural language processing model to explain the process of generating the data analysis results of the model to be trained. The server integrates the storage status data and device feature data of the model to be trained to obtain spatial dimension statistical data (including computing resource load, communication performance load, and training data storage volume) and time dimension statistical data (including resource usage status at each statistical moment), where the computing resource load can be as follows: target scale is 1005, target usage is 300, target computing power utilization is 90%, target temperature is 75 degrees, etc. The communication performance load can be as follows: data loading peak bandwidth is 19GB / s, data loading average bandwidth is 15GB / s, data exchange peak bandwidth is 40GB / s, data exchange average bandwidth is 27GB / s, etc. The training data storage volume can be as follows: the total amount of training data in five days is 3500, 3600, 3650, 3800, and 3850 respectively. The resource usage status at each statistical moment can be as follows: the superimposed computing power utilization within five minutes is 30%, 30%, 0%, 50%, and 20% respectively, the training calculation time period is 0 to 2 minutes and 3 to 5 minutes, and the non-training calculation time period is 2 to 3 minutes.

[0148] At the same time, the server can obtain preliminary analysis results for spatial dimension statistics and time dimension statistics. For example, the preliminary analysis results corresponding to the communication performance load can be as follows: 100% of the data exchange peak bandwidth is used (the maximum bandwidth limit is 40GB / s, and the peak data exchange bandwidth used is also 40GB / s), and there is a large period of communication idleness. The preliminary analysis results corresponding to the training data storage volume can be as follows: the total amount of training data grows slowly. The preliminary analysis results corresponding to the resource usage status at each statistical moment can be as follows: 80% of the time is for training calculations, and 20% of the time there is communication waiting.

[0149] The server can perform data analysis on the above-mentioned spatial dimension statistics, time dimension statistics and corresponding preliminary analysis results to obtain data analysis results related to the model training process of the model to be trained. The data analysis results can be as follows: training tasks of thousands of cards, high load of heterogeneous computing resources during training, continuous training operations can be performed if a card failure occurs, and the bottleneck point of training is RDMA communication bandwidth, etc.

[0150] Among them, the training task of 1,000 cards indicates that the scale of the training task is large and requires a lot of computing resources. The high load of heterogeneous computing resources during training indicates that the utilization rate of computing resources is very high during model training, which means that the training task requires a lot of computing power to complete. If a card fails, the training operation can be continued. If a heterogeneous card (heterogeneous computing unit) fails or fails during the training process, the training operation can be continued, that is, the training process can be restarted and the training can be continued from the last interruption. The bottleneck point of training is the RDMA communication bandwidth. Indicates that during the model training process, the RDMA communication bandwidth during data exchange has become the bottleneck that limits the training speed, that is, the transmission speed of the data required for training is limited by the bandwidth of RDMA communication.

[0151] The data analysis results of the above-mentioned model to be trained can guide the operation and maintenance personnel of the model to be trained to give priority to RDMA communication and the scale of heterogeneous cards when preparing heterogeneous computing resources for the training scenario of the natural language processing model. Secondly, in order to deal with card failures, the operation and maintenance personnel can reserve some buffer resources (referring to idle resources, which are used to replace the failed heterogeneous card when a card failure is detected), thereby ensuring the automatic continuation of model training. Among them, automatic continuation of training means that during model training, when an operational computing power failure or interruption occurs, the system can automatically detect and retain a certain amount of training progress information or buffer data, so that training can be automatically resumed after the failure is resolved, and training can continue from the last interruption without manual intervention or restarting the entire training process. The above method can improve the robustness and stability of model training, reduce training interruptions and data losses caused by operational failures, and improve the training efficiency and reliability of the entire training process.

[0152] It should be noted that the above data analysis results are only exemplary. For different types of models to be trained and training tasks in different business situations, the form of the data analysis results will be different. The form of the data analysis results is not limited in the embodiments of the present application.

[0153] In one embodiment, the multiple heterogeneous computing devices used by the model to be trained include a first heterogeneous computing device and a second heterogeneous computing device, the first heterogeneous computing device is the default heterogeneous computing device used for model training of the model to be trained, and the second heterogeneous computing device is used to replace the first heterogeneous computing device when a failure of the first heterogeneous computing device is detected. Then, when the server performs data integration processing on the storage status data and the device feature data of the multiple heterogeneous computing devices to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, it will only perform statistics on the relevant data of the first heterogeneous computing device, which avoids performing statistics on idle heterogeneous computing devices, improves the accuracy of the spatial dimension statistical data and the time dimension statistical data obtained by statistics, and thus improves the accuracy of the generated data analysis results related to the model training process.

[0154] Exemplarily, taking the target scale as an example, in step S302, according to the number of heterogeneous computing units included in each heterogeneous computing device, the target scale of the heterogeneous computing unit is determined, which can be achieved by the following steps: the total number of heterogeneous computing units included in each first heterogeneous computing device is determined as the target scale of the heterogeneous computing unit. For the specific implementation method, please refer to the relevant description in the above embodiment, which will not be repeated here.

[0155] In one embodiment, the server can provide the intermediate data in the data analysis process (such as line graphs generated based on computing power utilization, superimposed computing power utilization, etc.) to the operation and maintenance personnel, so that the operation and maintenance personnel can visually observe the optimizable points of model training and evaluate the cost reduction and efficiency improvement strategy of the model to be trained based on the optimizable points.

[0156] In one embodiment, the server can also make new expansions based on multiple hybrid dimensions (such as computing resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment), such as adding dimensions such as CPU usage and training time, so as to improve the accuracy of the generated data analysis results related to the model training process by performing data analysis on more types of data, thereby achieving cost reduction and efficiency improvement of model training.

[0157] Based on the above embodiments, the beneficial effect of the present application is that: the embodiment of the present application first obtains the storage state data of the model to be trained and the device feature data of multiple heterogeneous computing devices, and then integrates the storage state data and the device feature data of multiple heterogeneous computing devices to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, and then performs data analysis on it to obtain the data analysis result of the model to be trained. Since the data analysis result is determined based on the spatial dimension statistical data and the time dimension statistical data, and the spatial dimension statistical data and the time dimension statistical data include computing power resource load, communication performance load, training data storage volume and resource usage status at each statistical moment, this ensures the diversity of the analyzed data and improves the accuracy and credibility of the generated data analysis results related to the model training process. The data analysis results can guide the adjustment and optimization of the model training method for the model to be trained and the resource configuration of multiple heterogeneous computing devices, so that the resources of heterogeneous computing devices can be more reasonably utilized, the idleness and waste of resources can be reduced, and the cost of model training can be reduced. At the same time, by adjusting the model training method, the training efficiency and training effect of the model are improved, and finally the cost reduction and efficiency increase of model training are achieved.

[0158] In addition, the data processing method proposed in this application also has the following advantages: First, this application generates data analysis results related to the model training process of the model to be trained based on multiple mixed dimensions (computing resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment), achieving a more accurate analysis of the model training process. Second, due to the diversity of model training tasks, even if heterogeneous computing resources are in the running state of the container, heterogeneous computing resources do not continue to perform heterogeneous calculations. In this process, there may also be processes such as serialization and regularization of training data and integrated analysis of effect data. This application analyzes the model training process not only from the spatial dimension, but also from the time dimension, making up for the lack of analysis on the time dimension of model training in the traditional mode, and improving the accuracy of the generated data analysis results related to the model training process. Third, by providing the intermediate data in the data analysis process to the operation and maintenance personnel, this application can visually observe the optimizable points of the training task, and evaluate the cost reduction and efficiency improvement strategy of the model to be trained based on the optimizable points. Fourthly, from the perspective of operation services, by generating data analysis results related to the model training process through multiple hybrid dimensions, operation and maintenance personnel can intuitively see the usage of different indicators (such as computing power and bandwidth), which is convenient for monitoring and alarming, thereby improving the quality of operation and maintenance services.

[0159] See also Figure 5 , which is a schematic block diagram of a data processing device provided in an embodiment of the present application. The data processing device may specifically include:

[0160] An acquisition module 501 is used to acquire storage status data of a model to be trained and device characteristic data of multiple heterogeneous computing devices, wherein the storage status data includes the amount of training data distributed in each storage cluster, and the device characteristic data includes device attribute data and device status data generated by training the model to be trained;

[0161] An integration module 502 is used to perform data integration processing on the storage status data and the device feature data of the plurality of heterogeneous computing devices to obtain spatial dimension statistical data and temporal dimension statistical data corresponding to the model to be trained; the spatial dimension statistical data includes computing resource load, communication performance load and training data storage volume, and the temporal dimension statistical data includes resource usage status at each statistical moment;

[0162] The data analysis module 503 is used to perform data analysis on the above-mentioned spatial dimension statistical data and the above-mentioned time dimension statistical data to obtain the data analysis results of the above-mentioned model to be trained. The above-mentioned data analysis results are used to adjust the model training method of the above-mentioned model to be trained and the resource configuration of the above-mentioned multiple heterogeneous computing devices.

[0163] Optionally, each of the above-mentioned heterogeneous computing devices includes multiple heterogeneous computing units, the above-mentioned device attribute data includes the number of heterogeneous computing units, and the above-mentioned device status data includes the operating status of the container, computing power utilization, operating temperature of the heterogeneous computing unit, data loading bandwidth, and data exchange bandwidth;

[0164] When the integration module 502 is used to perform data integration processing on the storage state data and the device feature data of the plurality of heterogeneous computing devices to obtain the spatial dimension statistical data and the temporal dimension statistical data corresponding to the model to be trained, it is specifically used to:

[0165] Performing data integration processing on the number of the heterogeneous computing units, the operating status of the containers, the computing power utilization rate, and the operating temperature of the heterogeneous computing units to obtain the computing power resource load;

[0166] Performing data integration processing on the data loading bandwidth and the data exchange bandwidth to obtain the communication performance load;

[0167] Performing data integration processing on the storage state data to obtain the storage volume of the training data;

[0168] The above computing power utilization rate is processed by data integration to obtain the resource usage status at each statistical moment.

[0169] Optionally, the computing power resource load includes a target scale of the heterogeneous computing unit, a target usage of the heterogeneous computing device, a target computing power utilization rate of the heterogeneous computing device, and a target temperature of the heterogeneous computing unit;

[0170] When the integration module 502 is used to perform data integration processing on the number of the heterogeneous computing units, the operating status of the container, the computing power utilization rate, and the operating temperature of the heterogeneous computing units to obtain the computing power resource load, it is specifically used to:

[0171] Determining a target scale of the heterogeneous computing units according to the number of the heterogeneous computing units included in each of the heterogeneous computing devices;

[0172] Determine the heterogeneous computing devices whose containers are in the running state as the heterogeneous computing devices in use among the plurality of heterogeneous computing devices, and determine the target usage of the heterogeneous computing devices according to the number of the heterogeneous computing devices in use;

[0173] The computing power utilization rates of the heterogeneous computing devices in use at the same statistical moment are superimposed to obtain a first superimposed computing power utilization rate, and the first superimposed computing power utilization rate corresponding to each statistical moment is determined as a target computing power utilization rate of the heterogeneous computing device;

[0174] The operating temperature of each heterogeneous computing unit included in the heterogeneous computing device in use is determined, and the highest operating temperature among the operating temperatures of each heterogeneous computing unit is determined as the target temperature of the heterogeneous computing unit.

[0175] Optionally, the data loading bandwidth is the network bandwidth used for loading the training data of the model to be trained, the data exchange bandwidth is the network bandwidth for data exchange between the multiple heterogeneous computing devices, and the communication performance load includes the data loading peak bandwidth and the data loading mean bandwidth corresponding to the data loading bandwidth, and the data exchange peak bandwidth and the data exchange mean bandwidth corresponding to the data exchange bandwidth;

[0176] When the integration module 502 is used to perform data integration processing on the data loading bandwidth and the data exchange bandwidth to obtain the communication performance load, it is specifically used to:

[0177] The data loading bandwidths of the heterogeneous computing devices in use at the same statistical moment are superimposed to obtain a superimposed bandwidth, and the maximum superimposed bandwidth among the superimposed bandwidths corresponding to the statistical moments is determined as the data loading peak bandwidth;

[0178] Performing mean calculation processing on the data loading bandwidth of each of the heterogeneous computing devices in use at each statistical moment to obtain the data loading mean bandwidth;

[0179] A network interruption detection is performed on each of the above-mentioned heterogeneous computing devices in use, the data exchange traffic between each of the above-mentioned heterogeneous computing devices is determined, and the above-mentioned data exchange traffic is analyzed and processed to obtain the above-mentioned data exchange peak bandwidth and the above-mentioned data exchange mean bandwidth.

[0180] Optionally, the above communication performance load also includes data loading peak bandwidth ratio, data loading average bandwidth ratio, data exchange peak bandwidth ratio and data exchange average bandwidth ratio;

[0181] The above integration module 502 is also used for:

[0182] Determine the data loading peak bandwidth ratio and the data loading average bandwidth ratio according to the data loading peak bandwidth, the data loading average bandwidth and the sum of the data loading bandwidths of the heterogeneous computing devices in use;

[0183] Traffic analysis is performed on the data exchange traffic to obtain the peak bandwidth ratio of the data exchange and the average bandwidth ratio of the data exchange.

[0184] Optionally, when the integration module 502 is used to perform data integration processing on the storage state data to obtain the training data storage volume, it is specifically used to:

[0185] The amount of training data distributed in each storage cluster at the same statistical moment is summed up to obtain the total amount of training data;

[0186] The total amount of training data corresponding to each statistical moment is determined as the above-mentioned training data storage amount.

[0187] Optionally, when the integration module 502 is used to perform data integration processing on the computing power utilization to obtain the resource usage status at each statistical moment, it is specifically used to:

[0188] Superimposing the computing power utilization rates of the plurality of heterogeneous computing devices at the same statistical moment to obtain a second superimposed computing power utilization rate;

[0189] The utilization rates of the second superposition computing power corresponding to the multiple statistical moments are divided and processed by the utilization rate threshold to obtain a division result, and the training computing time period and the non-training computing time period of the multiple heterogeneous computing devices are determined according to the division result;

[0190] The resource usage status of each of the above statistical moments is determined according to the second superposition computing power utilization rate corresponding to the above multiple statistical moments, the above training calculation time period, and the above non-training calculation time period.

[0191] Optionally, the acquisition module 501 is further used to:

[0192] Obtaining preliminary analysis results for the above-mentioned spatial dimension statistical data and the above-mentioned temporal dimension statistical data; the above-mentioned preliminary analysis results are inferred by operation and maintenance personnel based on the above-mentioned spatial dimension statistical data and the above-mentioned temporal dimension statistical data, or the above-mentioned preliminary analysis results are obtained by a data analysis model performing data analysis processing on the above-mentioned spatial dimension statistical data and the above-mentioned temporal dimension statistical data;

[0193] The data analysis module 503, when used to perform data analysis on the spatial dimension statistical data and the time dimension statistical data to obtain the data analysis result of the model to be trained, is specifically used to:

[0194] Using an artificial intelligence model or a data analysis algorithm, the above-mentioned spatial dimension statistical data, the above-mentioned time dimension statistical data and the above-mentioned preliminary analysis results are subjected to data analysis processing to obtain the data analysis results of the above-mentioned model to be trained.

[0195] Based on the above-mentioned data processing device, the embodiment of the present application first obtains the storage status data of the model to be trained and the device characteristic data of multiple heterogeneous computing devices, and then integrates the storage status data and the device characteristic data of multiple heterogeneous computing devices to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, and then performs data analysis to obtain the data analysis results of the model to be trained.

[0196] Since the data analysis results are determined based on spatial dimension statistical data and temporal dimension statistical data, which include computing resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment, this ensures the diversity of the analyzed data and improves the accuracy and credibility of the generated data analysis results related to the model training process.

[0197] The data analysis results can guide the adjustment and optimization of the model training method for the model to be trained and the resource configuration of multiple heterogeneous computing devices, so that the resources of heterogeneous computing devices can be used more reasonably, the idleness and waste of resources can be reduced, and the cost of model training can be reduced.

[0198] At the same time, by adjusting the model training method, the model training efficiency and training effect were improved, and ultimately the cost reduction and efficiency improvement of model training were achieved.

[0199] It should be noted that the functions of each functional module of the data processing device in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0200] See also Figure 6 , which is a schematic block diagram of a computer device provided in an embodiment of the present application. Figure 6 The computer device in the embodiment shown may include: a processor 601, a storage device 602, and a communication interface 603. The processor 601, the storage device 602, and the communication interface 603 may exchange data.

[0201] The above-mentioned storage device 602 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the storage device 602 may also include a non-volatile memory (non-volatile memory), such as a flash memory, a solid-state drive (SSD), etc.; the above-mentioned storage device 602 may also include a combination of the above-mentioned types of memory.

[0202] The processor 601 may be a central processing unit (CPU). In one embodiment, the processor 601 may also be a graphics processing unit (GPU). The processor 601 may also be a combination of a CPU and a GPU. In one embodiment, the storage device 602 is used to store program instructions, and the processor 601 may call the program instructions to perform the following operations:

[0203] Obtain storage status data of the model to be trained and device characteristic data of multiple heterogeneous computing devices, the storage status data including the amount of training data distributed in each storage cluster, and the device characteristic data including device attribute data and device status data generated by training the model to be trained;

[0204] Performing data integration processing on the storage status data and the device feature data of the plurality of heterogeneous computing devices to obtain spatial dimension statistical data and temporal dimension statistical data corresponding to the model to be trained; the spatial dimension statistical data includes computing power resource load, communication performance load and training data storage volume, and the temporal dimension statistical data includes resource usage status at each statistical moment;

[0205] The spatial dimension statistical data and the time dimension statistical data are subjected to data analysis and processing to obtain data analysis results of the model to be trained. The data analysis results are used to adjust the model training method of the model to be trained and the resource configuration of the multiple heterogeneous computing devices.

[0206] Optionally, each of the above-mentioned heterogeneous computing devices includes multiple heterogeneous computing units, the above-mentioned device attribute data includes the number of heterogeneous computing units, and the above-mentioned device status data includes the operating status of the container, computing power utilization, operating temperature of the heterogeneous computing unit, data loading bandwidth, and data exchange bandwidth;

[0207] When the processor 601 is used to perform data integration processing on the storage state data and the device feature data of the plurality of heterogeneous computing devices to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, it is specifically used to:

[0208] Performing data integration processing on the number of the heterogeneous computing units, the operating status of the containers, the computing power utilization rate, and the operating temperature of the heterogeneous computing units to obtain the computing power resource load;

[0209] Performing data integration processing on the data loading bandwidth and the data exchange bandwidth to obtain the communication performance load;

[0210] Performing data integration processing on the storage state data to obtain the storage volume of the training data;

[0211] The above computing power utilization rate is processed by data integration to obtain the resource usage status at each statistical moment.

[0212] Optionally, the computing power resource load includes a target scale of the heterogeneous computing unit, a target usage of the heterogeneous computing device, a target computing power utilization rate of the heterogeneous computing device, and a target temperature of the heterogeneous computing unit;

[0213] When the processor 601 is used to perform data integration processing on the number of the heterogeneous computing units, the operating status of the container, the computing power utilization rate, and the operating temperature of the heterogeneous computing units to obtain the computing power resource load, it is specifically used to:

[0214] Determining a target scale of the heterogeneous computing units according to the number of the heterogeneous computing units included in each of the heterogeneous computing devices;

[0215] Determine the heterogeneous computing devices whose containers are in the running state as the heterogeneous computing devices in use among the plurality of heterogeneous computing devices, and determine the target usage of the heterogeneous computing devices according to the number of the heterogeneous computing devices in use;

[0216] The computing power utilization rates of the heterogeneous computing devices in use at the same statistical moment are superimposed to obtain a first superimposed computing power utilization rate, and the first superimposed computing power utilization rate corresponding to each statistical moment is determined as a target computing power utilization rate of the heterogeneous computing device;

[0217] The operating temperature of each heterogeneous computing unit included in the heterogeneous computing device in use is determined, and the highest operating temperature among the operating temperatures of each heterogeneous computing unit is determined as the target temperature of the heterogeneous computing unit.

[0218] Optionally, the data loading bandwidth is the network bandwidth used for loading the training data of the model to be trained, the data exchange bandwidth is the network bandwidth for data exchange between the multiple heterogeneous computing devices, and the communication performance load includes the data loading peak bandwidth and the data loading mean bandwidth corresponding to the data loading bandwidth, and the data exchange peak bandwidth and the data exchange mean bandwidth corresponding to the data exchange bandwidth;

[0219] When the processor 601 is used to perform data integration processing on the data loading bandwidth and the data exchange bandwidth to obtain the communication performance load, it is specifically used to:

[0220] The data loading bandwidths of the heterogeneous computing devices in use at the same statistical moment are superimposed to obtain a superimposed bandwidth, and the maximum superimposed bandwidth among the superimposed bandwidths corresponding to the statistical moments is determined as the data loading peak bandwidth;

[0221] Performing mean calculation processing on the data loading bandwidth of each of the heterogeneous computing devices in use at each statistical moment to obtain the data loading mean bandwidth;

[0222] A network interruption detection is performed on each of the above-mentioned heterogeneous computing devices in use, the data exchange traffic between each of the above-mentioned heterogeneous computing devices is determined, and the above-mentioned data exchange traffic is analyzed and processed to obtain the above-mentioned data exchange peak bandwidth and the above-mentioned data exchange mean bandwidth.

[0223] Optionally, the above communication performance load also includes data loading peak bandwidth ratio, data loading average bandwidth ratio, data exchange peak bandwidth ratio and data exchange average bandwidth ratio;

[0224] The processor 601 is further configured to:

[0225] Determine the data loading peak bandwidth ratio and the data loading average bandwidth ratio according to the data loading peak bandwidth, the data loading average bandwidth and the sum of the data loading bandwidths of the heterogeneous computing devices in use;

[0226] Traffic analysis is performed on the data exchange traffic to obtain the peak bandwidth ratio of the data exchange and the average bandwidth ratio of the data exchange.

[0227] Optionally, when the processor 601 is used to perform data integration processing on the storage state data to obtain the training data storage volume, it is specifically used to:

[0228] The amount of training data distributed in each storage cluster at the same statistical moment is summed up to obtain the total amount of training data;

[0229] The total amount of training data corresponding to each statistical moment is determined as the above-mentioned training data storage amount.

[0230] Optionally, when the processor 601 is used to perform data integration processing on the computing power utilization to obtain the resource usage status at each statistical moment, it is specifically used to:

[0231] Superimposing the computing power utilization rates of the plurality of heterogeneous computing devices at the same statistical moment to obtain a second superimposed computing power utilization rate;

[0232] The utilization rates of the second superposition computing power corresponding to the multiple statistical moments are divided and processed by the utilization rate threshold to obtain a division result, and the training computing time period and the non-training computing time period of the multiple heterogeneous computing devices are determined according to the division result;

[0233] The resource usage status of each of the above statistical moments is determined according to the second superposition computing power utilization rate corresponding to the above multiple statistical moments, the above training calculation time period, and the above non-training calculation time period.

[0234] Optionally, the processor 601 is further configured to:

[0235] Obtaining preliminary analysis results for the above-mentioned spatial dimension statistical data and the above-mentioned temporal dimension statistical data; the above-mentioned preliminary analysis results are inferred by operation and maintenance personnel based on the above-mentioned spatial dimension statistical data and the above-mentioned temporal dimension statistical data, or the above-mentioned preliminary analysis results are obtained by a data analysis model performing data analysis processing on the above-mentioned spatial dimension statistical data and the above-mentioned temporal dimension statistical data;

[0236] The processor 601, when used to perform data analysis on the spatial dimension statistical data and the time dimension statistical data to obtain the data analysis result of the model to be trained, is specifically used to:

[0237] Using an artificial intelligence model or a data analysis algorithm, the above-mentioned spatial dimension statistical data, the above-mentioned time dimension statistical data and the above-mentioned preliminary analysis results are subjected to data analysis processing to obtain the data analysis results of the above-mentioned model to be trained.

[0238] Based on the above-mentioned computer device, the embodiment of the present application first obtains the storage status data of the model to be trained and the device characteristic data of multiple heterogeneous computing devices, and then integrates the storage status data and the device characteristic data of multiple heterogeneous computing devices to obtain the spatial dimension statistical data and the time dimension statistical data corresponding to the model to be trained, and then performs data analysis to obtain the data analysis results of the model to be trained.

[0239] Since the data analysis results are determined based on spatial dimension statistical data and temporal dimension statistical data, which include computing resource load, communication performance load, training data storage volume, and resource usage status at each statistical moment, this ensures the diversity of the analyzed data and improves the accuracy and credibility of the generated data analysis results related to the model training process.

[0240] The data analysis results can guide the adjustment and optimization of the model training method for the model to be trained and the resource configuration of multiple heterogeneous computing devices, so that the resources of heterogeneous computing devices can be used more reasonably, the idleness and waste of resources can be reduced, and the cost of model training can be reduced.

[0241] At the same time, by adjusting the model training method, the model training efficiency and training effect were improved, and ultimately the cost reduction and efficiency improvement of model training were achieved.

[0242] In a specific implementation, the processor 601, the storage device 602, and the communication interface 603 described in the embodiments of the present application can execute the aforementioned embodiments of the present application. Figure 2 or Figure 3 The implementation described in the relevant embodiments of the data processing method provided can also be implemented in the embodiments of the present application. Figure 5 The implementation methods described in the relevant embodiments of the provided data processing device will not be repeated here.

[0243] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0244] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned data processing device, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the method in the aforementioned embodiment, so it will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located at one location, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network, and multiple computer devices distributed in multiple locations and interconnected by a communication network can constitute a blockchain system.

[0245] According to one aspect of the present application, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device can execute the method in the aforementioned embodiment, and therefore, will not be described in detail here.

[0246] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The above-mentioned storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0247] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and making equivalent changes according to the claims of this application are still within the scope of the invention.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire storage status data of a model to be trained and device characteristic data of multiple heterogeneous computing devices, wherein the storage status data includes the amount of training data distributed in each storage cluster, and the device characteristic data includes device attribute data and device status data generated by training the model to be trained; Performing data integration processing on the storage status data and the device feature data of the multiple heterogeneous computing devices to obtain spatial dimension statistical data and time dimension statistical data corresponding to the model to be trained; the spatial dimension statistical data includes computing power resource load, communication performance load and training data storage volume, and the time dimension statistical data includes resource usage status at each statistical moment; Data analysis is performed on the spatial dimension statistical data and the temporal dimension statistical data to obtain data analysis results of the model to be trained, and the data analysis results are used to adjust the model training method of the model to be trained and the resource configuration of the multiple heterogeneous computing devices.

2. The method according to claim 1, characterized in that Each of the heterogeneous computing devices includes a plurality of heterogeneous computing units, the device attribute data includes the number of heterogeneous computing units, and the device status data includes the operating status of the container, computing power utilization, operating temperature of the heterogeneous computing units, data loading bandwidth, and data exchange bandwidth; The step of performing data integration processing on the storage state data and the device feature data of the plurality of heterogeneous computing devices to obtain spatial dimension statistical data and temporal dimension statistical data corresponding to the model to be trained includes: Performing data integration processing on the number of the heterogeneous computing units, the operating status of the container, the computing power utilization rate, and the operating temperature of the heterogeneous computing units to obtain the computing power resource load; Performing data integration processing on the data loading bandwidth and the data exchange bandwidth to obtain the communication performance load; Performing data integration processing on the storage state data to obtain the storage volume of the training data; The computing power utilization rate is subjected to data integration processing to obtain the resource usage status at each statistical moment.

3. The method according to claim 2, characterized in that The computing resource load includes a target scale of the heterogeneous computing unit, a target usage of the heterogeneous computing device, a target computing power utilization rate of the heterogeneous computing device, and a target temperature of the heterogeneous computing unit; The step of performing data integration processing on the number of the heterogeneous computing units, the operating status of the container, the computing power utilization rate, and the operating temperature of the heterogeneous computing units to obtain the computing power resource load includes: Determining a target scale of the heterogeneous computing units according to the number of the heterogeneous computing units included in each of the heterogeneous computing devices; Determine a heterogeneous computing device whose container operation state is in the running state among the multiple heterogeneous computing devices as a heterogeneous computing device in use, and determine a target usage amount of the heterogeneous computing device according to the number of the heterogeneous computing devices in use; The computing power utilization rates of the heterogeneous computing devices in use at the same statistical moment are superimposed to obtain a first superimposed computing power utilization rate, and the first superimposed computing power utilization rate corresponding to each statistical moment is determined as a target computing power utilization rate of the heterogeneous computing device; The operating temperature of each heterogeneous computing unit included in the heterogeneous computing devices in use is determined, and the highest operating temperature among the operating temperatures of each heterogeneous computing unit is determined as the target temperature of the heterogeneous computing unit.

4. The method according to claim 2, characterized in that: The data loading bandwidth is the network bandwidth used for loading the training data of the model to be trained, the data exchange bandwidth is the network bandwidth for data exchange between the multiple heterogeneous computing devices, and the communication performance load includes the data loading peak bandwidth and the data loading mean bandwidth corresponding to the data loading bandwidth, and the data exchange peak bandwidth and the data exchange mean bandwidth corresponding to the data exchange bandwidth; The step of performing data integration processing on the data loading bandwidth and the data exchange bandwidth to obtain the communication performance load includes: Performing superposition processing on the data loading bandwidths of the heterogeneous computing devices in use at the same statistical moment to obtain a superposition bandwidth, and determining the maximum superposition bandwidth among the superposition bandwidths corresponding to the various statistical moments as the data loading peak bandwidth; Performing mean calculation processing on the data loading bandwidth of each of the heterogeneous computing devices in use at each statistical moment to obtain the data loading mean bandwidth; A network interruption detection is performed on each of the heterogeneous computing devices in use, the data exchange traffic between each of the heterogeneous computing devices is determined, and traffic analysis and processing is performed on the data exchange traffic to obtain the data exchange peak bandwidth and the data exchange mean bandwidth.

5. The method according to claim 4, characterized in that The communication performance load also includes the data loading peak bandwidth ratio, the data loading average bandwidth ratio, the data exchange peak bandwidth ratio and the data exchange average bandwidth ratio; Wherein, the method further comprises: Determine the data loading peak bandwidth ratio and the data loading mean bandwidth ratio according to the data loading peak bandwidth, the data loading mean bandwidth and the sum of the data loading bandwidths of each of the heterogeneous computing devices in use; The data exchange traffic is analyzed to obtain the data exchange peak bandwidth ratio and the data exchange mean bandwidth ratio.

6. The method according to claim 2, characterized in that The performing data integration processing on the storage state data to obtain the training data storage volume includes: The amount of training data distributed in each storage cluster at the same statistical moment is summed to obtain the total amount of training data; The total amount of training data corresponding to each statistical moment is determined as the training data storage amount.

7. The method according to claim 2, characterized in that The data integration processing of the computing power utilization to obtain the resource usage status at each statistical moment includes: Superimposing the computing power utilization rates of the plurality of heterogeneous computing devices at the same statistical moment to obtain a second superimposed computing power utilization rate; Dividing the second superposition computing power utilization corresponding to multiple statistical moments by using the utilization threshold to obtain a division result, and determining the training computing time period and the non-training computing time period of the multiple heterogeneous computing devices according to the division result; The resource usage status of each statistical moment is determined according to the second superposition computing power utilization rate corresponding to the multiple statistical moments, the training calculation time period, and the non-training calculation time period.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Obtaining preliminary analysis results for the spatial dimension statistical data and the temporal dimension statistical data; the preliminary analysis results are inferred by operation and maintenance personnel based on the spatial dimension statistical data and the temporal dimension statistical data, or the preliminary analysis results are obtained by performing data analysis processing on the spatial dimension statistical data and the temporal dimension statistical data by a data analysis model; The data analysis and processing of the spatial dimension statistical data and the time dimension statistical data to obtain the data analysis result of the model to be trained includes: Using an artificial intelligence model or a data analysis algorithm, data analysis and processing are performed on the spatial dimension statistical data, the temporal dimension statistical data and the preliminary analysis results to obtain the data analysis results of the model to be trained.

9. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire storage status data of a model to be trained and device characteristic data of multiple heterogeneous computing devices, wherein the storage status data includes the amount of training data distributed in each storage cluster, and the device characteristic data includes device attribute data and device status data generated by training the model to be trained; an integration module, configured to perform data integration processing on the storage status data and the device feature data of the plurality of heterogeneous computing devices to obtain spatial dimension statistical data and temporal dimension statistical data corresponding to the model to be trained; the spatial dimension statistical data includes computing resource load, communication performance load and training data storage volume, and the temporal dimension statistical data includes resource usage status at each statistical moment; A data analysis module is used to perform data analysis on the spatial dimension statistical data and the temporal dimension statistical data to obtain data analysis results of the model to be trained. The data analysis results are used to adjust the model training method of the model to be trained and the resource configuration of the multiple heterogeneous computing devices.

10. A computer device, characterized in that: include: A processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are connected to each other, wherein the storage device stores an executable program code, and the processor is used to call the executable program code to implement the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, they are used to implement the data processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, it is used to implement the data processing method according to any one of claims 1 to 8.

Images