A method, system and related device for multi-hardware hybrid large model training

Abstract

The present application discloses a method, system and related device for training a multi-hardware hybrid large model, which is used to simplify the model structure and improve the model performance. The method of the present application includes: first, collecting multimodal data, constructing a multimodal model architecture, and obtaining the first performance index of multiple hardware by benchmarking the multimodal large model. Then, the received computing task is decomposed into several subtasks, the multimodal large model is divided into several submodels, and the submodels and subtasks are matched to suitable hardware for training according to the first performance index. Then, the information transmission link is pruned according to the calculated comprehensive weight value. Then, the second performance index is evaluated, a performance evaluation report is obtained, a dynamic adjustment strategy is implemented for the pruning operation, and subsequent pruning operations are performed. After the pruning operation is performed, the submodel is reinitialized and locally fine-tuned. Finally, the multimodal large model is comprehensively evaluated using a test data set.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, system and related devices for training a multi-hardware hybrid large model. Background Art

[0002] In recent years, deep learning has made significant progress in the fields of natural language processing and computer vision. Large-scale deep learning models have demonstrated excellent performance in various tasks and are widely used in natural language processing, computer vision and other fields. These models usually contain hundreds of millions to hundreds of billions of parameters and can capture complex patterns and semantic information. However, large models on a single hardware can no longer meet the performance and resource consumption requirements of large-scale models.

[0003] In the prior art, the use of large models with multiple hardware collaboration can break through the upper limit of the computing power of a single hardware, and use distributed decomposition to decompose tasks onto multiple hardware for parallel computing, achieving more efficient computing processing, thereby accelerating the training process. The structure of large multi-hardware models is complex. In order to fully integrate multimodal information such as text, images, and audio, they often contain a large number of neurons and connections, resulting in a huge amount of calculation. Although large multi-hardware models can take advantage of the advantages of different hardware, they are also difficult to cope with such a huge amount of calculation. In order to reduce the complexity of large multi-hardware models, pruning operations are usually used during the training process to reduce redundant connections and neurons so that the model maintains performance.

[0004] However, during the model training process, the model is in a rapid learning stage and is constantly adjusting parameters to adapt to the characteristics of the data. The pruning operation may prune off some useful connections that have not been fully learned, affecting the convergence of the model and reducing the performance of the model. Summary of the invention

[0005] The present application provides a method, system and related devices for training a multi-hardware hybrid large model, which are used to simplify the model structure and improve the model performance.

[0006] The first aspect of the present application provides a method for training a multi-hardware hybrid large model, comprising:

[0007] Collecting multimodal data and performing preprocessing to obtain multimodal data samples, wherein the multimodal data includes text data, image data, audio data, and video data;

[0008] Constructing a multimodal model architecture according to the multimodal data samples;

[0009] In a simulation environment of multiple hardware, benchmark the multimodal large model using the multimodal data sample to obtain first performance indicators of the multiple hardware;

[0010] Receiving a computing task and decomposing the computing task into a plurality of subtasks;

[0011] Dividing the multimodal large model into a plurality of sub-models based on the multimodal model architecture, and matching the sub-models and the sub-tasks to suitable hardware for training according to the first performance indicator;

[0012] Calculating a comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and performing a pruning operation on the information transmission link according to the comprehensive weight value;

[0013] Based on a preconfigured load monitoring model, evaluate the second performance indicators of the plurality of hardware to obtain a performance evaluation report;

[0014] Implementing a dynamic adjustment strategy for the pruning operation according to the performance evaluation report, and performing subsequent pruning operations, wherein the dynamic adjustment strategy includes adjusting the pruning cycle and adjusting the pruning ratio;

[0015] After performing the pruning operation, reinitializing and locally fine-tuning the sub-model;

[0016] When all the subtasks are completed, the multimodal large model is fully evaluated using the test dataset.

[0017] Optionally, calculating a comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and performing a pruning operation on the information transmission link according to the comprehensive weight value, includes:

[0018] Calculate the information entropy value of the sub-model according to the weight matrix of each layer of the sub-model;

[0019] Monitor the gradient change of the sub-model during the training process to obtain a gradient change index;

[0020] Performing weighted summation on the information entropy value and the gradient change index to obtain a comprehensive weight value of the sub-model;

[0021] Sort the comprehensive weight values ​​of all sub-models in ascending order, and determine the sub-model with the smallest weight as the starting sub-model;

[0022] The information transmission link of the starting sub-model is pruned according to a preset initial pruning ratio, and a pruning log is recorded.

[0023] Optionally, the evaluating the second performance indicators of the plurality of hardware based on the preconfigured load monitoring model to obtain a performance evaluation report includes:

[0024] Based on the preconfigured load detection model, the second performance indicator data of each hardware is collected, and multi-dimensional analysis is performed to obtain the second performance indicator;

[0025] The second performance indicator is compared with the first performance indicator to obtain a performance evaluation report, wherein the performance evaluation report includes a hardware performance value and a hardware resource utilization rate.

[0026] Optionally, implementing a dynamic adjustment strategy on the pruning operation according to the performance evaluation report and performing subsequent pruning operations, wherein the dynamic adjustment strategy includes adjusting a pruning cycle and adjusting a pruning ratio, includes:

[0027] After the pruning operation, determining an adjustment target according to the hardware performance value and the hardware resource utilization in the performance evaluation report;

[0028] executing a dynamic adjustment strategy on the adjustment target;

[0029] Record a dynamic adjustment log of executing the dynamic adjustment strategy, and perform subsequent pruning operations.

[0030] Optionally, if the hardware performance value in the performance evaluation report is not within a preset performance threshold, the adjustment target is determined to be adjusting the pruning ratio, and the dynamic adjustment strategy is executed on the pruning ratio;

[0031] If the hardware resource utilization in the performance evaluation report is not within the preset hardware resource threshold, the adjustment target is determined to be adjusting the pruning period, and the dynamic adjustment strategy is executed on the pruning period.

[0032] Optionally, after performing the pruning operation, reinitializing and locally fine-tuning the sub-model includes:

[0033] Traversing the network layers and parameter matrices of the sub-model to determine the sub-model state;

[0034] Determining whether the pruning operation is performed on the sub-model according to the state of the sub-model;

[0035] If so, the sub-model is randomly initialized and locally fine-tuned.

[0036] Optionally, in a simulation environment of multiple hardware, using the multimodal data samples to perform a benchmark test on the multimodal large model to obtain first performance indicators of the multiple hardware includes:

[0037] Using virtualization technology to create a hardware simulation environment for the multiple hardware, and setting parameters of the hardware simulation environment according to the architecture characteristics of the hardware;

[0038] Using the multimodal data samples in each of the hardware simulation environments to independently test the multimodal large model;

[0039] The first performance indicator data of the hardware is collected and analyzed to obtain a first performance indicator.

[0040] The second aspect of the present application provides a system for multi-hardware hybrid large model training, including:

[0041] A collecting unit, used to collect multimodal data and perform preprocessing to obtain multimodal data samples, wherein the multimodal data includes text data, image data, audio data and video data;

[0042] A construction unit, configured to construct a multimodal model architecture according to the multimodal data sample;

[0043] A first evaluation unit is used to perform a benchmark test on the multimodal large model using the multimodal data sample in a simulation environment of multiple hardware to obtain first performance indicators of the multiple hardware;

[0044] A decomposition unit, used for receiving a computing task and decomposing the computing task into a plurality of subtasks;

[0045] an allocation unit, configured to divide the multimodal large model into a plurality of sub-models based on the multimodal model architecture, and match the sub-models and the sub-tasks to suitable hardware for training according to the first performance indicator;

[0046] A pruning unit, used to calculate a comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and perform a pruning operation on the information transmission link according to the comprehensive weight value;

[0047] A second evaluation unit, configured to evaluate the second performance indicators of the plurality of hardware based on a preconfigured load monitoring model to obtain a performance evaluation report;

[0048] A dynamic adjustment unit, configured to implement a dynamic adjustment strategy for the pruning operation according to the performance evaluation report, and perform subsequent pruning operations, wherein the dynamic adjustment strategy includes adjusting the pruning cycle and adjusting the pruning ratio;

[0049] An initialization unit, used for reinitializing and locally fine-tuning the sub-model after performing the pruning operation;

[0050] The third evaluation unit is used to use the test data set to comprehensively evaluate the multimodal large model after all the subtasks are completed.

[0051] Optionally, the pruning unit is specifically used to:

[0052] Calculate the information entropy value of the sub-model according to the weight matrix of each layer of the sub-model;

[0053] Monitor the gradient change of the sub-model during the training process to obtain a gradient change index;

[0054] Performing weighted summation on the information entropy value and the gradient change index to obtain a comprehensive weight value of the sub-model;

[0055] Sort the comprehensive weight values ​​of all sub-models in ascending order, and determine the sub-model with the smallest weight as the starting sub-model;

[0056] The information transmission link of the starting sub-model is pruned according to a preset initial pruning ratio, and a pruning log is recorded.

[0057] Optionally, the second evaluation unit is specifically configured to:

[0058] Based on the preconfigured load detection model, the second performance indicator data of each hardware is collected, and multi-dimensional analysis is performed to obtain the second performance indicator;

[0059] The second performance indicator is compared with the first performance indicator to obtain a performance evaluation report, wherein the performance evaluation report includes a hardware performance value and a hardware resource utilization rate.

[0060] Optionally, the dynamic adjustment unit is specifically used to:

[0061] After the pruning operation, determining an adjustment target according to the hardware performance value and the hardware resource utilization in the performance evaluation report;

[0062] executing a dynamic adjustment strategy on the adjustment target;

[0063] Record a dynamic adjustment log of executing the dynamic adjustment strategy, and perform subsequent pruning operations.

[0064] Optionally, the dynamic adjustment unit is specifically used to:

[0065] If the hardware performance value in the performance evaluation report is not within the preset performance threshold, determining the adjustment target as adjusting the pruning ratio, and executing the dynamic adjustment strategy on the pruning ratio;

[0066] If the hardware resource utilization in the performance evaluation report is not within the preset hardware resource threshold, the adjustment target is determined to be adjusting the pruning period, and the dynamic adjustment strategy is executed on the pruning period.

[0067] Optionally, the initialization unit is specifically used to:

[0068] Traversing the network layers and parameter matrices of the sub-model to determine the sub-model state;

[0069] Determining whether the pruning operation is performed on the sub-model according to the state of the sub-model;

[0070] If so, the sub-model is randomly initialized and locally fine-tuned.

[0071] Optionally, the first evaluation unit is specifically configured to:

[0072] Using virtualization technology to create a hardware simulation environment for the multiple hardware, and setting parameters of the hardware simulation environment according to the architecture characteristics of the hardware;

[0073] Using the multimodal data samples in each of the hardware simulation environments to independently test the multimodal large model;

[0074] The first performance indicator data of the hardware is collected and analyzed to obtain a first performance indicator.

[0075] A third aspect of the present application provides a device for training a multi-hardware hybrid large model, the device comprising:

[0076] Processor, memory, input-output unit, and bus;

[0077] The processor is connected to the memory, the input and output unit, and the bus;

[0078] The memory stores a program, and the processor calls the program to execute the first aspect and any optional method of multi-hardware hybrid large model training in the first aspect.

[0079] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the method for training a large multi-hardware hybrid model in the first aspect and any optional method in the first aspect is performed.

[0080] It can be seen from the above technical solutions that this application has the following advantages:

[0081] Through the multimodal large model architecture constructed by multimodal data samples, combined with the first performance indicators of multiple hardware, several subtasks and several submodels are adapted to the hardware for training. During the training process, the comprehensive weight value is obtained by combining the information entropy value and the gradient change index, which is used as the basis for performing pruning operations on the information transmission link in the multimodal large model. Pruning operations can be performed more effectively, thereby simplifying the model structure, reducing computing resource consumption, and improving overall training efficiency. Pruning operations are dynamically adjusted according to the performance evaluation report, and timely adjustments can be made according to the actual training status and hardware performance of the multimodal large model, avoiding unnecessary pruning operations or missing the best pruning time. Through the dynamic pruning strategy, the multimodal large model can achieve better performance at the end of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0083] Figure 1 A flowchart of an embodiment of the method for multi-hardware hybrid large model training provided in this application;

[0084] Figure 2 A flowchart of another embodiment of the method for multi-hardware hybrid large model training provided in this application;

[0085] Figure 3 A schematic diagram of the structure of an embodiment of a system for multi-hardware hybrid large model training provided in this application;

[0086] Figure 4 A schematic diagram of the structure of an embodiment of the device for training a multi-hardware hybrid large model provided in this application. DETAILED DESCRIPTION

[0087] The present application provides a method, system and related device for training a multi-hardware hybrid large model, which is used to simplify the model structure and improve the model performance. It should be noted that the method for training a multi-hardware hybrid large model provided by the present application is applied to a terminal.

[0088] See also Figure 1 , Figure 1 An embodiment of the method for multi-hardware hybrid large model training provided by the present application includes:

[0089] S101, collecting multimodal data and performing preprocessing to obtain multimodal data samples, where the multimodal data includes text data, image data, audio data, and video data;

[0090] Start with a variety of external data sources, use the corresponding interfaces or collection tools, and obtain multimodal data according to the set rules and scope. For example, crawl text content from various websites as text data through web crawlers, use image acquisition equipment or extract image data from image databases, collect audio data with the help of audio recording equipment and audio libraries, and obtain video data from video storage platforms and surveillance video streams.

[0091] The collected multimodal data is uniformly preprocessed to ensure that the final multimodal data samples have a consistent format and feature representation, which is convenient for inputting into multimodal large models for training in multiple hardware environments. For text data, natural language processing technology is used to remove redundant punctuation and spaces in the text and standardize the text. Image preprocessing uses image algorithms to perform operations such as size normalization, color correction, and noise removal on the image to ensure that the image quality and feature presentation meet the model input requirements. Audio preprocessing performs operations such as noise reduction and resampling to adjust the audio frequency on the audio data so that the audio data meets the specifications in terms of duration and sampling rate. In addition to performing similar image processing on the image frames in the video, video preprocessing also synchronously processes the audio track to ensure the synchronization of audio and video.

[0092] S102, constructing a multimodal model architecture according to the multimodal data samples;

[0093] Analyze the characteristics, structure and relationship of each modal data in the multimodal data sample, extract feature vectors to build a multimodal model architecture. For example, for the sequential and semantic characteristics of text data, use the Transformer architecture with a multi-head attention mechanism to extract text feature vectors; for image data, use a suitable convolutional neural network layer to capture local and global features in the image and extract image feature vectors; for the characteristics of audio data that change over time, you can combine a recurrent neural network to extract audio feature vectors; for video data, combine convolutional neural network layers with network modules that process time series to extract video feature vectors. Then, the extracted feature vectors are fused through a specific fusion layer to obtain a fused feature vector. Using the fused feature vector as input, construct an encoder and decoder to determine the overall architecture of the model, forming a multimodal model architecture that can simultaneously process multimodal data and realize multimodal information interaction and collaborative learning.

[0094] S103, in a simulation environment of multiple hardware, using multimodal data samples to perform a benchmark test on the multimodal large model to obtain first performance indicators of the multiple hardware;

[0095] The constructed multimodal large model is deployed in a multi-hardware simulation environment, and multimodal data samples are called for simulation training as a benchmark test. During the benchmark test, a specific performance analysis algorithm is used to generate a corresponding first performance indicator for each hardware. The first performance indicator is stored in the hardware performance database in the form of structured data. The first performance indicator records the adaptability and performance of the multimodal large model training task under different hardware initial states.

[0096] S104, receiving a computing task, and decomposing the computing task into a plurality of subtasks;

[0097] When a computing task is received, it is analyzed and divided into several subtasks according to the specific content, objectives, and constraints of the computing task. For example, a multimodal data classification task will be divided into multiple subtasks according to the characteristics of the multimodal data and the characteristics of the classification target. Text feature extraction is one subtask, and semantic features are extracted from text data using natural language processing technology. Image feature extraction is another subtask, and the visual features of the image are obtained through image recognition-related algorithms. There will also be subtasks that integrate different modal features for classification judgment, etc. These subtasks lay the foundation for subsequent allocation and execution of training.

[0098] S105, dividing the multimodal large model into a plurality of sub-models based on the multimodal model architecture, and matching the sub-models and sub-tasks to suitable hardware for training according to the first performance indicator;

[0099] According to the functions and processing objects of different modules in the multimodal model architecture, the entire multimodal model is disassembled into several sub-models. For example, the neural network layer responsible for processing text modality is divided into one sub-model, and the network structure related to image modality processing is divided into another sub-model. Then, the first performance indicator of each hardware is read, and the sub-models and corresponding sub-tasks are assigned to the most suitable hardware resources through matching algorithms in combination with the calculation characteristics of each sub-model and sub-task and the data dependency relationship between sub-tasks. For example, for image feature extraction sub-models and corresponding sub-tasks with huge calculation amount and requiring a large amount of parallel calculation, they are preferentially assigned to hardware with strong parallel computing capabilities for execution; for some sub-tasks and sub-models related to logic control and data preprocessing and with relatively low requirements for calculation speed, they are assigned to more versatile hardware for processing. Ensure that each hardware resource can give full play to its advantages and improve the overall training efficiency.

[0100] S106, calculating a comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and performing a pruning operation on the information transmission link according to the comprehensive weight value;

[0101] During the training process, the information entropy value of each sub-model and the gradient change index during the training process are collected. The information entropy value reflects the uncertainty and amount of information output by the sub-model, while the gradient change index reflects the speed and direction of the parameter update of the sub-model in the current training stage. The corresponding comprehensive weight value of each sub-model is calculated by combining the information entropy value and the gradient change index. The higher the comprehensive weight value, the more important the sub-model is in the entire model system and the current training stage, and the corresponding information transmission link has a greater impact on the overall model performance. According to the calculated comprehensive weight value, pruning operations are performed on the information transmission links, such as reducing unnecessary intermediate layer connections in these links, reducing the frequency of data transmission, or truncating some parameter interaction paths that are not helpful for improving the overall performance.

[0102] S107, based on the preconfigured load monitoring model, evaluating the second performance indicators of the plurality of hardware to obtain a performance evaluation report;

[0103] During the model training process, by evaluating the secondary performance indicators of multiple hardware based on the preconfigured load monitoring model, you can understand the working status of the hardware in real time. For example, if you find that the utilization rate of the hardware has dropped suddenly, it may be because of a bottleneck in data transmission or some layers of the model have not fully utilized the parallel computing capabilities of the hardware. By recording and analyzing this data, you can implement the corresponding strategy adjustment training in a timely manner to ensure efficient training.

[0104] S108, implementing a dynamic adjustment strategy for the pruning operation according to the performance evaluation report, and performing subsequent pruning operations, the dynamic adjustment strategy including adjusting the pruning cycle and adjusting the pruning ratio;

[0105] During the training process, the performance of each hardware is not constant. Pruning all sub-models only according to the initial pruning cycle and initial pruning ratio will not result in the optimal multimodal large model. When the hardware performance deteriorates, if the initial pruning strategy is still followed, the hardware may be overloaded, the amount of data in the information transmission link may exceed its processing capacity, and the training efficiency may be reduced or even hardware failure may occur. In order for the pruning operation to effectively optimize the multimodal large model, a dynamic adjustment strategy needs to be implemented for the pruning operation.

[0106] S109, after performing the pruning operation, reinitialize and locally fine-tune the sub-model;

[0107] After the pruning operation, the structure of the sub-model will change. The pruned part is likely to change the distribution of sub-model parameters and information loss, and there may also be potential redundant calculations and reduced adaptability when the data distribution changes. At this time, re-initialization and local fine-tuning are required to provide a new starting point for the model, adjust the parameter distribution to make it more reasonable and restore information loss, optimize the calculation path to reduce redundant calculations, make the model better adapt to hardware resources, improve running speed and energy efficiency, and help the model learn new feature differences when the data distribution changes, comprehensively improve the model performance and adaptability, so that it can perform better in different data scenarios.

[0108] S110. After all subtasks are completed, use the test dataset to comprehensively evaluate the multimodal large model.

[0109] When all subtasks have completed training according to the set requirements, the prepared test data set will be retrieved and input into the complete multimodal large model, allowing the multimodal large model to perform prediction, classification or other related tasks. According to the results output by the multimodal large model and the real labels pre-labeled in the test data set, appropriate evaluation indicators are selected for different types of tasks. For example, for classification tasks, indicators such as accuracy and recall rate are used to measure the classification performance of the model; for regression tasks, indicators such as mean square error are used to evaluate the prediction accuracy of the model. Calculate the specific values ​​of these evaluation indicators and generate a detailed and comprehensive evaluation report, which records the performance of the multimodal large model in various indicators.

[0110] In this embodiment, a multimodal large model architecture is constructed through multimodal data samples, combined with the first performance indicators of multiple hardware, and several subtasks and several submodels are adapted to the hardware for training. During the training process, the comprehensive weight value is obtained by combining the information entropy value and the gradient change index, which is used as the basis for performing pruning operations on the information transmission link in the multimodal large model, so that pruning operations can be performed more effectively, thereby simplifying the model structure, reducing computing resource consumption, and improving overall training efficiency. The pruning operation is dynamically adjusted according to the performance evaluation report, and timely adjustments can be made according to the actual training status and hardware performance of the multimodal large model, avoiding unnecessary pruning operations or missing the best pruning time. Through the dynamic pruning strategy, the multimodal large model can achieve better performance at the end of training.

[0111] See also Figure 2 , Figure 2 Another embodiment of the method for multi-hardware hybrid large model training provided by the present application includes:

[0112] S201, collecting multimodal data and performing preprocessing to obtain multimodal data samples, where the multimodal data includes text data, image data, audio data, and video data;

[0113] S202, constructing a multimodal model architecture according to the multimodal data samples;

[0114] Steps S201 and S202 are similar to the above-mentioned steps S101 and S102 and will not be described in detail here.

[0115] S203, using virtualization technology to create a hardware simulation environment of multiple hardware, and setting parameters of the hardware simulation environment according to the architecture characteristics of the hardware;

[0116] Use virtualization technology to create multiple simulation environments with different hardware configurations, divide the multimodal data samples reasonably into training sets, validation sets, and test sets according to a certain ratio, and then extract a part of the training set as a benchmark test data set and evenly distribute it to each hardware simulation environment. At the same time, deploy the constructed multimodal large model code to the virtual machine instance corresponding to each simulation environment, set the parameters of the simulation environment according to the hardware architecture characteristics, so that the hardware simulation environment can reflect the operating characteristics of the hardware as realistically as possible.

[0117] S204, independently testing the multimodal large model using multimodal data samples in each hardware simulation environment;

[0118] Independent testing is performed in each hardware simulation environment to evaluate the performance of different hardware in processing multimodal data samples. Multimodal data includes different types of data such as text, images, audio, and video. Each hardware has different processing efficiency, data reading speed, and model parameter update speed for these different modal data. Through independent testing, we can clearly distinguish the advantages and disadvantages of each hardware in the face of multimodal data samples, avoid mutual interference and influence between different hardware, and thus obtain the initial performance of each hardware.

[0119] S205, collecting and analyzing first performance indicator data of the hardware to obtain a first performance indicator;

[0120] Collecting the first performance indicator data of the hardware involves multiple dimensions, which may include key indicators such as the hardware's computing speed, memory read and write speed, and data transmission rate. These collected first performance indicator data are analyzed using data analysis methods and tools to obtain the first performance indicator. For example, by analyzing energy consumption data, the energy utilization efficiency of the hardware during the test can be evaluated. The first performance indicator obtained can reflect the initial performance of each hardware in the multimodal large model training scenario, and thus provide a basis for the subsequent reasonable matching of sub-models and sub-tasks to suitable hardware.

[0121] S206, receiving a computing task, and decomposing the computing task into a plurality of subtasks;

[0122] S207, dividing the multimodal large model into a plurality of sub-models based on the multimodal model architecture, and matching the sub-models and sub-tasks to suitable hardware for training according to the first performance indicator;

[0123] Steps S206 and S207 are similar to the above-mentioned steps S104 and S105 and are not described in detail here.

[0124] S208, calculating the information entropy value of the sub-model according to the weight matrix of each layer of the sub-model;

[0125] The sub-model consists of multiple layers, each of which contains a weight matrix. The weight matrix is ​​used to perform linear transformations on the input data and is a key part of the sub-model's learning and representation of data features. The weight matrix of the sub-model can be accessed by looping through it, and the range of the comprehensive weight value can be determined. The minimum and maximum values ​​in the weight matrix can then be found, and the intervals can be divided to calculate the weight probability distribution of each interval. For example, within a certain interval, there are weight values, a total of elements, then the probability of this interval , thus obtaining the probability distribution of weight values ​​in each interval.

[0126] Using the information entropy value calculation formula ,in is the number of intervals, The weight value appears in The probability of an interval. The information entropy value is calculated for each layer. Through a specific integration algorithm, such as weighted average or comprehensive calculation based on the importance of each layer in the model, the information entropy values ​​calculated for each layer are summarized to finally obtain the information entropy value of the entire sub-model. This information entropy value represents the degree of uncertainty of the output information of the sub-model and the amount of information contained. The higher the information entropy value, the greater the diversity and uncertainty of the sub-model output, and vice versa, it is relatively more certain and regular.

[0127] S209, monitoring the gradient change of the sub-model during the training process and obtaining a gradient change index;

[0128] During the model training process, the gradient change indicator represents the derivative of the loss function with respect to the model parameters, which reflects the direction and magnitude of the impact of parameter changes on the loss function. By monitoring the gradient changes of the sub-model during the training process, the update dynamics of the sub-model parameters can be understood.

[0129] During the training of the sub-model, the gradient monitoring module is activated to track the gradient changes of each parameter in the sub-model in real time during each round of training iteration. For each trainable parameter in the neural network, its gradient indicates the sensitivity of the parameter to the change of the loss function and reflects the direction and magnitude of the current parameter adjustment. The gradient changes of each parameter in several consecutive rounds of training iterations are recorded, such as calculating the mean and variance of the gradient statistics, so as to characterize the overall trend and fluctuation of the gradient change. Based on these statistics and pre-set rules, a gradient change index is comprehensively generated. The gradient change index can reflect the stability and convergence speed of the sub-model training process from different angles. For example, if the mean of the gradient is large and the variance is small, it means that the parameters are updating rapidly in a relatively stable and obvious direction, and the model may converge faster; on the contrary, if the gradient fluctuates violently and the mean is unstable, it may mean that the model training encounters problems such as falling into the local optimum and unreasonable learning rate setting.

[0130] S210, performing weighted summation of the information entropy value and the gradient change index to obtain a comprehensive weight value of the sub-model;

[0131] After obtaining the information entropy value and gradient change index corresponding to the sub-model, the pre-configured weight parameters will be read. These weight parameters reflect the importance of the information entropy value and gradient change index in comprehensive consideration. According to the weighted sum calculation method, the information entropy value is , and its corresponding weight is , the gradient change index is , and its corresponding weight is , weighted summation to obtain the comprehensive weight value of the sub-model , which can integrate the two key factors that reflect the amount of information output by the sub-model and the dynamic changes in the training process, so that the weight value can comprehensively measure the relative importance of the sub-model in the entire model system and the current training stage, and provide key decision-making basis for subsequent pruning operations.

[0132] S211, sorting the comprehensive weight values ​​of all sub-models in ascending order, and determining the sub-model with the smallest weight as the starting sub-model;

[0133] Collect the comprehensive weight values ​​calculated by all sub-models, and use sorting algorithms, such as bubble sort and quick sort, to arrange these comprehensive weight values ​​in ascending order. After the sorting is completed, the sub-model at the front of the sequence, that is, the sub-model with the smallest weight value, is determined as the starting sub-model in this application. The sub-model with the smallest weight contributes less to the overall model performance than other sub-models. In theory, pruning it first may have a relatively smaller negative impact on the overall model, and it is easier to start with the relatively less important parts to simplify the model structure.

[0134] S212, performing a pruning operation on the information transmission link of the starting sub-model according to a preset initial pruning ratio, and recording a pruning log;

[0135] The preset initial pruning ratio indicates the proportion of the information transmission link to be pruned in the starting sub-model to the total. After reading the preset pruning ratio, pruning operations will be performed on each information transmission link of the starting sub-model according to the preset initial pruning ratio. For example, for the neuron connection in the neural network, the information transmission link is truncated or weakened according to the preset initial pruning ratio, so as to reduce the redundant connection and unnecessary information transmission in the sub-model, simplify the model structure and reduce the consumption of computing resources. While executing the pruning operation, the relevant information of the pruning operation is recorded, including the specific time of pruning, the identification of the starting sub-model, the adopted pruning ratio and the specific link position pruned, etc., to form a pruning log stored in a special log database. The pruning log facilitates the subsequent backtracking of the pruning process, the analysis of the pruning effect, and provides a complete reference basis for subsequent further pruning adjustments and model optimization.

[0136] S213, based on the preconfigured load detection model, collecting the second performance indicator data of each hardware, and performing multi-dimensional analysis to obtain the second performance indicator;

[0137] The preconfigured load monitoring model has built-in multiple monitoring modules and data acquisition interfaces for different hardware. During the training of the multimodal large model, these monitoring modules will interact with each hardware in real time, use the performance monitoring interface provided by the hardware or obtain the operation data of the corresponding hardware through the underlying instructions, so as to collect the second performance indicator data of each hardware. The collected second performance indicator data is deeply analyzed from multiple dimensions. From the time dimension, analyze the changing trend of each hardware indicator with the training time, observe whether there are periodic fluctuations or abnormal changes in a specific training stage, and judge whether the hardware utilization rate suddenly increases or decreases in the iteration stage of model training; from the resource association dimension, consider the mutual influence of indicators between different hardware, such as whether the read and write frequency of memory changes accordingly when the hardware computing volume increases, and analyze whether there are resource bottlenecks or poor coordination problems; from the task load dimension, combine the characteristics of the subtask currently being executed, and analyze the matching degree between the hardware and the computing requirements of the subtask. By using data analysis algorithms, such as statistical analysis algorithms and anomaly detection algorithms in machine learning, the analysis results of these different dimensions are comprehensively sorted out to obtain the second performance indicators of each hardware in the current training stage, which more comprehensively and accurately reflects the actual performance of the hardware.

[0138] S214, comparing the second performance indicator with the first performance indicator to obtain a performance evaluation report, the performance evaluation report including a hardware performance value and a hardware resource utilization rate;

[0139] Compare the first performance indicator of each hardware with the second performance indicator collected and analyzed during the current training process. For each hardware, compare the changes in its various performance indicators, such as checking the numerical difference between the core utilization rate of the hardware during the benchmark test and the actual training, and the different performance of indicators such as the hardware memory utilization rate in the two stages. By calculating the difference and the change ratio, the specific change range of each hardware performance indicator from the baseline state to the actual training state is obtained.

[0140] The comparison results are comprehensively collated and analyzed to generate a detailed performance evaluation report. In the report, the hardware performance value section will list in detail the actual values ​​of the main performance indicators of each hardware in the current training stage, such as hardware performance value and hardware resource utilization. This information can be used to determine whether hardware resources are idle, wasted or overused. In addition, the performance evaluation report will also analyze the reasons for changes in hardware performance based on the comparison results, such as whether it is due to unreasonable allocation of subtasks that leads to unbalanced hardware load, or performance degradation of the hardware itself, and give corresponding optimization suggestions, which provide a strong reference for subsequent adjustments to training strategies, optimization of hardware resource allocation, and dynamic adjustment of pruning operations.

[0141] S215, after the pruning operation, determining an adjustment target according to the hardware performance value and the hardware resource utilization in the hardware performance evaluation report;

[0142] After completing a pruning operation, the hardware performance evaluation report will be retrieved. The performance evaluation report contains the hardware performance value and hardware resource utilization of each hardware, and the hardware performance value and hardware resource utilization are analyzed in depth. If the hardware performance value of a certain hardware is significantly lower than the expected level, for example, the core utilization rate of the hardware is at a low level for a long time, while the overall training progress is relatively slow, this may mean that the current pruning operation has reduced the amount of computing tasks excessively, and the hardware resources are not fully utilized, resulting in the limitation of the computing power of the multimodal large model. On the contrary, if the hardware resource utilization rate is too high, close to or reaching the upper limit of the hardware load, for example, the hardware memory utilization rate is at a high level for a long time, and the memory explosion occurs frequently, it means that the current model structure is still relatively complex, the demand for hardware resources exceeds the reasonable range, and the hardware has performance bottlenecks, which affects the normal progress of training. According to the analysis results, the specific adjustment target is determined so as to carry out targeted dynamic adjustment strategies so that the hardware performance and resource utilization can reach a relatively better balance and ensure the training process is carried out efficiently and stably.

[0143] S216, executing a dynamic adjustment strategy for the adjustment target;

[0144] After determining the adjustment target, the dynamic adjustment strategy is implemented in combination with the current performance evaluation report.

[0145] In some specific embodiments, if the hardware performance value in the hardware performance evaluation report is not within the preset performance threshold, the adjustment target is determined to be adjusting the pruning ratio, and a dynamic adjustment strategy is executed on the pruning ratio.

[0146] The preset performance threshold is determined based on the ideal working state of the hardware and the expected requirements of the hardware performance for the multi-hardware mixed large model training task. After obtaining the hardware performance evaluation report, the hardware performance value in the performance evaluation report is checked and compared with the preset performance threshold. If the hardware performance value is not within the preset performance threshold, it means that the current pruning operation has an unreasonable impact on the hardware performance, resulting in the hardware failing to achieve the ideal performance level. In order to improve this situation, the adjustment target is determined to be the adjustment of the pruning ratio. The pruning ratio is directly related to the degree of simplicity of the model structure, which in turn affects the load and performance of the hardware when performing computing tasks. If the hardware performance value is lower than the threshold, it may mean that the current pruning is excessive and the model calculation is too small. In this case, the pruning ratio needs to be appropriately reduced to allow the model to retain more structure and calculation, so that the hardware can fully exert its performance. On the contrary, if the hardware performance value is higher than the threshold, it may be that the pruning is insufficient and the model is too complex, resulting in excessive hardware pressure. At this time, the pruning ratio needs to be increased to further simplify the model and reduce the hardware burden, so that the hardware performance returns to a reasonable threshold range.

[0147] If you want to reduce the pruning ratio, it will follow the pre-configured reduction rules, such as setting the pruning ratio to be reduced by a certain percentage each time, and make corresponding adjustments based on the current pruning ratio value; if you want to increase the pruning ratio, it will also be modified according to the increase rules.

[0148] In some specific embodiments, if the hardware resource utilization in the hardware performance evaluation report is not within the preset hardware resource threshold, the adjustment target is determined to be adjusting the pruning period, and a dynamic adjustment strategy is executed on the pruning period.

[0149] Compare the hardware resource utilization data in the performance evaluation report with the preset hardware resource threshold. The preset hardware resource threshold defines the scope of reasonable utilization of hardware resources. If the hardware resource utilization is not within the preset hardware resource threshold, it means that the current pruning operation does not match the actual use of hardware resources, affecting the overall efficiency of training. At this time, the adjustment target will be determined as adjusting the pruning cycle to balance the hardware resource utilization. If the hardware resource utilization is close to or exceeds the upper threshold of the hardware resource threshold, it means that the current computational complexity of the model has too much demand for hardware resources, the hardware is in a high load state for a long time, and performance bottlenecks are prone to occur. At this time, it is necessary to speed up the pruning frequency, that is, shorten the pruning cycle, and simplify the model structure through more pruning operations as soon as possible to reduce the pressure on hardware resources. On the contrary, if the hardware resource utilization is lower than the lower threshold, it means that the pruning operation is too frequent, the model structure is simplified too quickly, and the hardware resources are idle and wasted. In this case, it is necessary to extend the pruning cycle and appropriately slow down the pruning rhythm so that the hardware has more computing tasks to process and improve resource utilization.

[0150] When it is determined that the pruning cycle needs to be shortened, the preset shortening rules are used, for example, the original pruning operation after every 10 training iterations is adjusted to once every 5 rounds; if the pruning cycle needs to be extended, the extension rules are used, such as adjusting the pruning cycle from once every 5 rounds to once every 15 rounds.

[0151] S217, recording a dynamic adjustment log for executing the dynamic adjustment strategy, and performing subsequent pruning operations;

[0152] While executing the dynamic adjustment strategy, the relevant information of the dynamic adjustment strategy execution is recorded. The dynamic adjustment log includes the specific time of the adjustment, the hardware performance evaluation result based on which the adjustment is made, the pruning cycle before the adjustment, the pruning ratio parameter value, the corresponding parameter value after the adjustment, and the adjustment target, etc., and is stored in a special log database to facilitate the subsequent tracing of the dynamic changes of the pruning strategy during the entire training process, providing a reference for further optimizing the training process.

[0153] The subsequent pruning operations are continued according to the updated pruning cycle and pruning ratio after executing the dynamic adjustment strategy. During the execution process, the standard process of the previous pruning operation will also be followed, while the hardware performance and the status of the multimodal large model will be continuously monitored, so that the dynamic adjustment strategy can be executed again according to the performance evaluation report generated later, forming a closed loop of dynamic optimization, ensuring that the multimodal large model training can achieve better performance in continuous adjustment.

[0154] S218, traversing the network layer and parameter matrix of the sub-model to determine the state of the sub-model;

[0155] Although pruning logs are recorded after the pruning operation, the structure or parameters of the sub-model will change after the pruning operation. Relying solely on the pruning log cannot accurately reflect the changes in the sub-model. Therefore, after the pruning operation is completed, it is still necessary to traverse each network layer of the sub-model in turn according to the topological order of the network structure of the sub-model. In the process of traversing each network layer, the state of the sub-model is determined by comparing the changes in the parameter matrix before and after the pruning operation. For example, for the weight matrix, if some of the elements are set to zero or the number of connections corresponding to some rows and columns is significantly reduced after the pruning operation, it is determined that pruning behavior has occurred at these positions of this network layer.

[0156] S219, judging whether the sub-model is pruned according to the sub-model state;

[0157] After traversing all network layers and their parameter matrices of each sub-model and obtaining the corresponding sub-model status, comprehensively analyze the status of each sub-model to determine whether there are network layer parameter matrices with connections pruned and the number of parameters reduced, indicating pruning characteristics. If at least one network layer in a sub-model changes, it is determined that the sub-model has been pruned; conversely, if the parameter matrices of all network layers have no obvious changes before and after the comparison, it is determined that the sub-model has not been pruned. The judgment results for each sub-model are obtained and marked to provide a basis for subsequent decisions on whether to take further actions on the sub-model.

[0158] S220, if yes, then randomly initialize and locally fine-tune the sub-model;

[0159] After determining that a sub-model has been pruned, the sub-model is randomly initialized. For each network layer in the sub-model, new initial values ​​are assigned to its parameters according to the type of network layer and the corresponding initialization strategy. For example, for the weight matrix of the fully connected layer, a common random initialization method such as normal distribution or uniform distribution may be used to generate new weight parameters according to the set mean and variance, and the bias vector is initialized to zero or other appropriate fixed values, so that the sub-model can be prepared for subsequent adjustments in a new and relatively reasonable initial state after pruning, avoiding the model from falling into an unreasonable state due to changes in parameter structure caused by pruning, and laying the foundation for subsequent local fine-tuning.

[0160] After random initialization, according to the current structural characteristics of the sub-model and the goal of the training task, the appropriate optimization algorithm is used to iteratively adjust the parameters in the sub-model locally, so that the sub-model can still accurately extract features and complete the corresponding computing tasks based on the structure after pruning and simplification, ensuring that the performance of the entire multimodal large model will not be greatly reduced due to pruning operations, and may even be improved to a certain extent. For sub-models that have not changed, no reinitialization or local fine-tuning is performed.

[0161] S221. After all subtasks are completed, use the test dataset to comprehensively evaluate the multimodal large model.

[0162] Step S221 is similar to the above step S110 and will not be described again here.

[0163] In this embodiment, virtualization technology is used to create multiple hardware simulation environments, and multimodal data samples are used to independently test the multimodal large model, which can evaluate the initial adaptability of different hardware to the multimodal large model training in a simulation environment. Collecting multi-dimensional hardware first performance indicator data and analyzing and deriving the first performance indicator for the reasonable matching of subsequent sub-models, sub-tasks, and hardware can effectively improve the overall training efficiency and resource utilization, and avoid unreasonable allocation and waste of hardware resources.

[0164] By performing weighted summation of the obtained information entropy value and gradient change index according to the preset weight parameters, the comprehensive weight value of the sub-model is obtained, and its relative importance is comprehensively measured to make the determination of the starting sub-model more convincing. The comprehensive weight values ​​of all sub-models are collected and arranged in ascending order using a sorting algorithm. The sub-model with the smallest weight is determined as the starting sub-model, and the information transmission link of the starting sub-model is pruned according to the preset initial pruning ratio to minimize the initial impact on the overall model performance and facilitate the simplification of the model structure. The pruning log recorded at the same time can trace back the pruning process, which helps to continuously streamline the model structure, improve the model operation efficiency, and reduce the consumption of computing resources while ensuring the performance of the model.

[0165] According to the dynamic changes of hardware performance from benchmark testing to actual training presented in the performance evaluation report, targeted adjustments to training strategies, optimization of resource allocation, and reasonable model pruning operations are made. After the pruning operation, analysis is conducted based on the hardware performance values ​​and hardware resource utilization in the performance evaluation report to determine whether the hardware resource utilization is reasonable and whether the model has performance bottlenecks, etc., and then the adjustment target is clarified. The determined adjustment target is compared with the preset threshold to execute the dynamic adjustment strategy to avoid blind adjustment, so that the dynamic adjustment strategy can focus on solving problems such as limited hardware performance or poor hardware resource utilization caused by unreasonable pruning operations. After updating the pruning cycle and pruning ratio, the pruning operation continues, and the hardware and model status is continuously monitored so that the dynamic adjustment strategy can be executed again according to the new performance evaluation report to form a dynamic optimization closed loop. The multimodal large model training can be continuously optimized as the hardware and model status changes, promoting model training to achieve better performance, improving training efficiency and the stability of the training process.

[0166] Finally, traverse each network layer in the topological order of the sub-model network structure, and determine the state of the sub-model by comparing the changes in the parameter matrix before and after pruning. Based on the comprehensive analysis of the state of each sub-model, identify the sub-model affected by the pruning operation, and make corresponding marks. For the sub-model determined to be pruned, first perform random initialization, and then use the optimization algorithm to perform local fine-tuning, so that the sub-model maintains good performance on the pruned structure, thereby effectively optimizing the overall performance of the multi-modal large model.

[0167] See also Figure 3 , Figure 3 An embodiment of a system for multi-hardware hybrid large model training provided in this application includes:

[0168] The collecting unit 301 is used to collect multimodal data and perform preprocessing to obtain multimodal data samples, where the multimodal data includes text data, image data, audio data, and video data;

[0169] A construction unit 302 is used to construct a multimodal model architecture according to the multimodal data samples;

[0170] A first evaluation unit 303 is used to perform a benchmark test on the multimodal large model using multimodal data samples in a simulation environment of multiple hardware to obtain first performance indicators of the multiple hardware;

[0171] A decomposition unit 304 is used to receive a computing task and decompose the computing task into a plurality of subtasks;

[0172] An allocating unit 305, configured to divide the multimodal large model into a plurality of sub-models based on the multimodal model architecture, and match the sub-models and sub-tasks to suitable hardware for training according to the first performance indicator;

[0173] A pruning unit 306, used to calculate a comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and perform a pruning operation on the information transmission link according to the comprehensive weight value;

[0174] A second evaluation unit 307 is used to evaluate the second performance indicators of the plurality of hardware based on a preconfigured load monitoring model to obtain a performance evaluation report;

[0175] A dynamic adjustment unit 308, configured to implement a dynamic adjustment strategy for the pruning operation according to the performance evaluation report and perform subsequent pruning operations, wherein the dynamic adjustment strategy includes adjusting the pruning cycle and adjusting the pruning ratio;

[0176] An initialization unit 309, used to reinitialize and locally fine-tune the sub-model after performing the pruning operation;

[0177] The third evaluation unit 310 is used to use the test data set to comprehensively evaluate the multimodal large model after all subtasks are completed.

[0178] Optionally, the pruning unit 306 is specifically configured to:

[0179] Calculate the information entropy value of the sub-model according to the weight matrix of each layer of the sub-model;

[0180] Monitor the gradient changes of the sub-model during the training process and obtain the gradient change indicators;

[0181] The information entropy value and the gradient change index are weighted and summed to obtain the comprehensive weight value of the sub-model;

[0182] Sort the comprehensive weight values ​​of all sub-models in ascending order, and determine the sub-model with the smallest weight as the starting sub-model;

[0183] The information transmission link of the starting sub-model is pruned with a preset initial pruning ratio, and the pruning log is recorded.

[0184] Optionally, the second evaluation unit 307 is specifically configured to:

[0185] Based on the preconfigured load detection model, the second performance indicator data of each hardware is collected, and multi-dimensional analysis is performed to obtain the second performance indicator;

[0186] The second performance indicator is compared with the first performance indicator to obtain a performance evaluation report, which includes a hardware performance value and a hardware resource utilization rate.

[0187] Optionally, the dynamic adjustment unit 308 is specifically configured to:

[0188] After the pruning operation, the adjustment target is determined based on the hardware performance value and hardware resource utilization in the performance evaluation report;

[0189] Implement dynamic adjustment strategies for adjustment targets;

[0190] Record the dynamic adjustment log of executing the dynamic adjustment strategy and perform subsequent pruning operations.

[0191] Optionally, the dynamic adjustment unit 308 is specifically configured to:

[0192] If the hardware performance value in the performance evaluation report is not within the preset performance threshold, the adjustment target is determined to be adjusting the pruning ratio, and a dynamic adjustment strategy is implemented for the pruning ratio;

[0193] If the hardware resource utilization in the performance evaluation report is not within the preset hardware resource threshold, the adjustment target is determined to be the adjustment of the pruning period, and a dynamic adjustment strategy is executed on the pruning period.

[0194] Optionally, the initialization unit 309 is specifically used for:

[0195] Traverse the network layers and parameter matrices of the sub-model to determine the sub-model status;

[0196] Determine whether the sub-model is pruned based on the sub-model status;

[0197] If yes, the sub-model is randomly initialized and locally fine-tuned.

[0198] Optionally, the first evaluation unit 303 is specifically configured to:

[0199] Use virtualization technology to create a hardware simulation environment for multiple hardware, and set the parameters of the hardware simulation environment according to the hardware architecture characteristics;

[0200] Independently test the multimodal large model using multimodal data samples in each hardware simulation environment;

[0201] The first performance indicator data of the hardware is collected and analyzed to obtain the first performance indicator.

[0202] This application also provides a device for training a multi-hardware hybrid large model, see Figure 4 , Figure 4 An embodiment of a device for training a multi-hardware hybrid large model provided in the present application includes:

[0203] Processor 401, memory 402, input and output unit 403, bus 404;

[0204] The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404;

[0205] The memory 402 stores a program, and the processor 401 calls the program to execute any of the above methods for training a multi-hardware hybrid large model.

[0206] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods for training a multi-hardware hybrid large model.

[0207] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0208] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0209] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A method for training a multi-hardware hybrid large model, characterized in that: The method comprises: Collecting multimodal data and performing preprocessing to obtain multimodal data samples, wherein the multimodal data includes text data, image data, audio data, and video data; Constructing a multimodal model architecture based on the multimodal data samples; In a simulation environment of multiple hardware, benchmark the multimodal large model using the multimodal data sample to obtain first performance indicators of the multiple hardware; Receiving a computing task and decomposing the computing task into a plurality of subtasks; Dividing the multimodal large model into a plurality of sub-models based on the multimodal model architecture, and matching the sub-models and the sub-tasks to suitable hardware for training according to the first performance indicator; Calculating a comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and performing a pruning operation on the information transmission link according to the comprehensive weight value; Based on a preconfigured load monitoring model, evaluate the second performance indicators of the plurality of hardware to obtain a performance evaluation report; Implementing a dynamic adjustment strategy for the pruning operation according to the performance evaluation report, and performing subsequent pruning operations, wherein the dynamic adjustment strategy includes adjusting the pruning cycle and adjusting the pruning ratio; After performing the pruning operation, reinitializing and locally fine-tuning the sub-model; When all the subtasks are completed, the multimodal large model is fully evaluated using the test dataset.

2. The method according to claim 1, characterized in that The calculating of the comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and performing a pruning operation on the information transmission link according to the comprehensive weight value, includes: Calculate the information entropy value of the sub-model according to the weight matrix of each layer of the sub-model; Monitor the gradient change of the sub-model during the training process to obtain a gradient change index; Performing weighted summation on the information entropy value and the gradient change index to obtain a comprehensive weight value of the sub-model; Sort the comprehensive weight values ​​of all sub-models in ascending order, and determine the sub-model with the smallest weight as the starting sub-model; The information transmission link of the starting sub-model is pruned according to a preset initial pruning ratio, and a pruning log is recorded.

3. The method according to claim 1, characterized in that The method of evaluating the second performance indicators of the plurality of hardware based on the preconfigured load monitoring model to obtain a performance evaluation report includes: Based on the preconfigured load detection model, the second performance indicator data of each hardware is collected, and multi-dimensional analysis is performed to obtain the second performance indicator; The second performance indicator is compared with the first performance indicator to obtain a performance evaluation report, wherein the performance evaluation report includes a hardware performance value and a hardware resource utilization rate.

4. The method according to claim 3, characterized in that The implementing a dynamic adjustment strategy for the pruning operation according to the performance evaluation report and performing subsequent pruning operations, wherein the dynamic adjustment strategy includes adjusting the pruning cycle and adjusting the pruning ratio, includes: After the pruning operation, determining an adjustment target according to the hardware performance value and the hardware resource utilization in the performance evaluation report; executing a dynamic adjustment strategy on the adjustment target; Record a dynamic adjustment log of executing the dynamic adjustment strategy, and perform subsequent pruning operations.

5. The method according to claim 4, characterized in that If the hardware performance value in the performance evaluation report is not within the preset performance threshold, determining the adjustment target as adjusting the pruning ratio, and executing the dynamic adjustment strategy on the pruning ratio; If the hardware resource utilization in the performance evaluation report is not within the preset hardware resource threshold, the adjustment target is determined to be adjusting the pruning period, and the dynamic adjustment strategy is executed on the pruning period.

6. The method according to claim 1, characterized in that After performing the pruning operation, reinitializing and locally fine-tuning the sub-model includes: Traversing the network layers and parameter matrices of the sub-model to determine the sub-model state; Determining whether the pruning operation is performed on the sub-model according to the state of the sub-model; If so, the sub-model is randomly initialized and locally fine-tuned.

7. The method according to claim 1, characterized in that The method of performing a benchmark test on the multimodal large model using the multimodal data sample in the simulation environment of the multiple hardwares to obtain the first performance indicators of the multiple hardwares includes: Using virtualization technology to create a hardware simulation environment for the multiple hardware, and setting parameters of the hardware simulation environment according to the architecture characteristics of the hardware; Using the multimodal data samples in each of the hardware simulation environments to independently test the multimodal large model; The first performance indicator data of the hardware is collected and analyzed to obtain a first performance indicator.

8. A system for training a multi-hardware hybrid large model, characterized in that: The system comprises: A collecting unit, used to collect multimodal data and perform preprocessing to obtain multimodal data samples, wherein the multimodal data includes text data, image data, audio data and video data; A construction unit, configured to construct a multimodal model architecture according to the multimodal data sample; A first evaluation unit is used to perform a benchmark test on the multimodal large model using the multimodal data sample in a simulation environment of multiple hardware to obtain first performance indicators of the multiple hardware; A decomposition unit, used for receiving a computing task and decomposing the computing task into a plurality of subtasks; an allocation unit, configured to divide the multimodal large model into a plurality of sub-models based on the multimodal model architecture, and match the sub-models and the sub-tasks to suitable hardware for training according to the first performance indicator; A pruning unit, used to calculate a comprehensive weight value according to the information entropy value and the gradient change index of the sub-model, and perform a pruning operation on the information transmission link according to the comprehensive weight value; A second evaluation unit, configured to evaluate the second performance indicators of the plurality of hardware based on a preconfigured load monitoring model to obtain a performance evaluation report; A dynamic adjustment unit, configured to implement a dynamic adjustment strategy for the pruning operation according to the performance evaluation report, and perform subsequent pruning operations, wherein the dynamic adjustment strategy includes adjusting the pruning cycle and adjusting the pruning ratio; An initialization unit, used for reinitializing and locally fine-tuning the sub-model after performing the pruning operation; The third evaluation unit is used to use the test data set to comprehensively evaluate the multimodal large model after all the subtasks are completed.

9. A device for training a multi-hardware hybrid large model, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the method according to any one of claims 1 to 7 is performed.

Images