Model optimization method and related device

Abstract

The invention belongs to the field of artificial intelligence, and particularly relates to a model optimization method and a related device. According to the method, the calculation operation meeting the splitting condition in the target model can be identified; splitting the calculation operation to obtain a plurality of sub-calculation operations; in the process of loading the target model into the hardware computing system, reconfiguring the plurality of sub-computing operations into the corresponding target computing units; the hardware computing system at least comprises a plurality of arithmetic units, and each arithmetic unit is composed of hardware chip resources called based on the assembly instruction; executing respective corresponding sub-calculation operations through the target operation units to obtain a plurality of sub-calculation results; and applying the plurality of sub-calculation results to a data processing flow of the target model. According to the method, calculation operation in the neural network model can be identified, split, reconfigured and executed in parallel, so that the calculation efficiency, the hardware utilization rate, the flexibility and the real-time performance are improved, the time delay of the calculation operation is reduced, hardware acceleration operation is executed for the model, and the performance of the neural network model is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and in particular, relates to a model optimization method and related devices. Background Art

[0002] In the field of modern artificial intelligence, especially in machine learning and deep learning applications, the efficiency and performance of algorithms are crucial. Artificial neural network models, such as the Generative Pre-Trained Transformation (GPT) model and the Bloom model in the Large Language Model (LLM), require a lot of computing resources for data processing and model training. These processes are mainly composed of many complex mathematical calculations, so how to efficiently use hardware resources to improve computing efficiency is crucial.

[0003] In traditional technologies, the model calculation process is usually serial. After waiting for the previous calculation operation to complete and obtain the calculation result, the next calculation operation can be started. The waiting process will cause other hardware to be idle, resulting in a waste of hardware resources and greatly reducing the efficiency of model operation.

[0004] Therefore, it is urgent to design a new technical solution to overcome at least one of the above technical problems. Summary of the invention

[0005] The present application provides a model optimization method and related devices for reallocating the computing operations of a neural network model to multiple hardware computing units, thereby improving the computing efficiency, hardware utilization, flexibility and real-time performance of the neural network model, reducing computing operation latency, performing hardware acceleration operations for the model, and improving the performance of the neural network model.

[0006] In a first aspect, the present application provides a model optimization method, which is applied to a hardware computing system deployed with a target model, wherein the target model is a neural network model; comprising: Identifying computing operations in the target model that meet the splitting conditions; Splitting the computing operation to obtain multiple sub-computing operations; In the process of loading the target model into the hardware computing system, multiple sub-computing operations are reconfigured to corresponding target computing units; the hardware computing system includes at least multiple computing units, each of which is composed of hardware chip resources called based on assembly instructions; Execute the corresponding sub-computation operations through the target computing unit to obtain multiple sub-computation results; The multiple sub-calculation results are applied to the data processing flow of the target model.

[0007] In a second aspect, an embodiment of the present application provides a model optimization device, which is applied to a hardware computing system deployed with a target model, wherein the target model is a neural network model; the device includes at least the following units: an identification unit, configured to identify computing operations in the target model that meet the splitting condition; a splitting unit, configured to split the computing operation into a plurality of sub-computing operations; a configuration unit configured to reconfigure the plurality of sub-computing operations into corresponding target computing units during the process of loading the target model into the hardware computing system; the hardware computing system comprises at least a plurality of computing units, each computing unit being composed of hardware chip resources called by assembly instructions; An execution unit is configured to execute the corresponding sub-computation operations through the target operation unit to obtain multiple sub-computation results; The application unit is configured to apply the multiple sub-calculation results to the data processing flow of the target model.

[0008] In a third aspect, an embodiment of the present application provides a chip for implementing the model optimization method described in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the model optimization method described in the first aspect.

[0010] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed, the model optimization method described in the first aspect is implemented.

[0011] In the embodiment of the present application, it is possible to identify the computing operations that meet the splitting conditions in the target model; split the computing operations to obtain multiple sub-computing operations; in the process of loading the target model into the hardware computing system, reconfigure the multiple sub-computing operations to the corresponding target computing units; the hardware computing system includes at least multiple computing units, each of which is composed of hardware chip resources called based on assembly instructions; execute the corresponding sub-computing operations through the target computing units to obtain multiple sub-computing results; apply the multiple sub-computing results to the data processing flow of the target model. The embodiment of the present application can significantly improve the computing efficiency, hardware utilization, flexibility and real-time performance, reduce the computing operation delay, perform hardware acceleration operations for the neural network model, improve the performance of the neural network model, and provide powerful hardware optimization support for various complex computing tasks by identifying, splitting, reconfiguring and executing computing operations in the neural network model in parallel. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a model optimization method according to an embodiment of the present application; Figure 2 A schematic diagram of the principle of a model optimization process in an embodiment of the present application; Figure 3 A schematic diagram of a principle of a model splitting process according to an embodiment of the present application; Figure 4 Another schematic diagram of the principle of a model splitting process in an embodiment of the present application; Figure 5 is a structural block diagram of a model optimization device according to an embodiment of the present application; Figure 6 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0015] In the field of artificial intelligence, especially in machine learning and deep learning applications, algorithm efficiency and computing performance are crucial. Currently, model calculations are mostly done using serial computing. Serial computing requires subsequent computing operations to wait for all dependent data to be ready before they can continue, and the waiting process will leave the hardware resources required for subsequent operations idle.

[0016] In the related art, for the hardware device where the neural network model is deployed, if the hardware device is provided with multiple computing units (such as computing units for processing matrix addition and matrix multiplication), the efficiency of the model operation can be improved through tensor parallelism. At present, data parallelism is usually used to accelerate model operations when multiple graphics processing units (GPUs) are deployed. In this method, a unified neural network model is loaded in each GPU, and all neural network models synchronize their respective states with each other after each iteration. This method can speed up training and solve the problem by investing more GPU resources.

[0017] In related technologies, the premise of data parallelism is that each GPU can accommodate the entire model. If the model is too large and exceeds the memory limit of a single GPU, the model cannot be loaded on the GPU, and training cannot be performed. This means that for very large models (such as some modern deep learning models like GPT-3), a single GPU cannot support it, and using data parallelism will be difficult. In addition, in multi-GPU training, gradient synchronization between GPUs requires a certain amount of communication time. This is especially obvious when there are a large number of GPUs, which may cause the computing nodes to be idle, so in some cases increasing the number of GPUs will introduce additional delays.

[0018] It should also be noted that for tasks with smaller data volumes, data parallelism may be inefficient since training the full model on each GPU may result in duplicate processing of information and waste of resources.

[0019] Therefore, it is urgent to design a new technical solution to overcome at least one of the above technical problems.

[0020] To this end, the embodiment of the present application provides a model optimization method and related devices, which can identify the computing operations that meet the splitting conditions in the target model; split the computing operations to obtain multiple sub-computing operations; in the process of loading the target model into the hardware computing system, reconfigure the multiple sub-computing operations to the corresponding target computing unit; the hardware computing system includes at least multiple computing units, each of which is composed of hardware chip resources called based on assembly instructions; execute the corresponding sub-computing operations through the target computing unit to obtain multiple sub-computing results; apply the multiple sub-computing results to the data processing flow of the target model. The embodiment of the present application can significantly improve the computing efficiency, hardware utilization, flexibility and real-time performance, reduce the computing operation delay, perform hardware acceleration operations for the neural network model, improve the performance of the neural network model, and provide powerful hardware optimization support for various complex computing tasks.

[0021] The model optimization scheme provided in the embodiment of the present application can also be executed by an electronic device, which can be a server, a server cluster, or a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a dedicated device (such as a dedicated terminal device with a model optimization system). These electronic devices can also be equipped with the chips introduced in the above embodiments. Alternatively, these electronic devices can also be installed with a service program for executing the model optimization scheme.

[0022] The model optimization method and related devices provided in the embodiments of the present application are described below in conjunction with the accompanying drawings. Figure 1 A model optimization method provided in an embodiment of the present application, such as Figure 1 As shown, the method includes: S101, identifying computing operations in the target model that meet the splitting conditions; S102, splitting the computing operation to obtain multiple sub-computing operations; S103, during the process of loading the target model into the hardware computing system, reconfiguring multiple sub-computing operations into corresponding target computing units; S104, performing the corresponding sub-computation operations through the target computing unit to obtain multiple sub-computation results; S105: Apply the multiple sub-calculation results to the data processing flow of the target model.

[0023] In the embodiment of the present application, the hardware computing system and the computing unit are key components of the model optimization method. In the embodiment of the present application, the hardware computing system includes at least a plurality of computing units, each of which is composed of hardware chip resources called by assembly instructions.

[0024] A hardware computing system is an integrated computing platform designed to provide efficient computing power to support complex model running and data processing tasks. The system is usually composed of multiple computing units, memory, storage, and other supporting components. Specifically, the processing unit is the core part of the calculation, which can include different types of processors, such as the central processing unit (CPU), which is used for general computing and is suitable for processing complex logic and control tasks. Graphics processing units (GPUs), which are specially designed for parallel processing of graphics calculations, are very suitable for deep learning and large-scale matrix operations. Tensor processing units (TPUs), such as dedicated chips, are designed to accelerate machine learning tasks. And field programmable gate arrays (FPGAs), programmable logic devices that can be efficiently configured for specific tasks. In addition, the hardware computing system also includes memory, such as random access memory (RAM) and cache, which are used to store the data being processed and model parameters for fast access and calculation. Storage, including hard disks and solid-state drives, is used to persist the storage of models, data, and calculation results. Buses and interfaces, used for data transmission between different components, include data buses, address buses, and control buses. And power management modules to ensure stable operation of hardware components and optimize energy consumption.

[0025] The arithmetic unit is the basic computing unit in the hardware computing system, each responsible for a specific computing task. The arithmetic unit is composed of hardware chip resources and can be called using a specific assembly instruction set. These instruction sets provide descriptions of basic operations for the arithmetic unit, making the execution of instructions more efficient. Each arithmetic unit can be configured to perform different types of operations. For example, the CPU is good at serial processing logic and complex calculations, while the GPU is suitable for large-scale parallel computing. FPGA can be reconfigured to implement specific logical operations according to specific needs. The task scheduling and resource allocation of the arithmetic unit are aimed at improving the efficiency of system operation. For example, the allocation of sub-computation operations can be dynamically adjusted according to the load and capacity of the arithmetic unit to ensure efficient use of hardware resources. Efficient data transmission and communication may be required between arithmetic units in order to collaboratively complete more complex computing tasks. In the model optimization method, the results of sub-computation operations can be transferred between different arithmetic units to achieve the integration of results.

[0026] Further optionally, the operation unit supports basic arithmetic operations, matrix multiplication, matrix transposition, synchronization between operation units, data transmission between operation units, etc., or can indirectly realize the above functions.

[0027] In the embodiment of the present application, the hardware computing system as a whole integrates a variety of computing units, and realizes efficient optimization of the model by identifying, splitting and reconfiguring computing operations. In the specific implementation, the performance of various types of computing units can be fully utilized to ensure efficient processing of complex data and computing tasks. This system architecture provides powerful hardware support for fields such as machine learning and deep learning, and can adapt to changing computing needs.

[0028] Exemplarily, the hardware computing system may be a single chip, or a system composed of multiple chips, or a system composed of other hardware resources.

[0029] In practical applications, on a hardware architecture with multiple computing units, the above steps can also be used to divide complex data calculations into multiple data streams to implement distributed stream computing, while fully utilizing hardware resources and improving model operation efficiency through parallel computing.

[0030] Specifically, on the hardware architecture of multiple computing units, by dividing complex data calculations into multiple data streams to achieve distributed stream computing, the operating efficiency of the model can be significantly improved while fully utilizing hardware resources. In practical applications, complex data computing tasks can be divided into multiple small data streams. Each data stream contains one or more sub-computing operations that are logically independent of each other or have data dependencies at specific points. Data streams can be divided according to the model's computational graph, data dependencies, computational complexity, and the characteristics of hardware resources. For example, the computational tasks of different layers or modules in a deep learning model can be divided into different data streams, each of which is responsible for processing a specific part of the data or computing tasks. Each data stream is assigned to a different computing unit for execution. The system assigns and schedules tasks based on the computing power of each computing unit, the current load situation, and the priority of the data stream. The load situation of each computing unit can be dynamically monitored, and tasks can be redistributed based on the real-time load to ensure full utilization of resources. For example, when a computing unit is idle, the system can automatically assign new data stream tasks to it, or migrate some tasks from computing units with higher loads.

[0031] On the hardware architecture of multiple computing units, multiple data streams can be executed in parallel, making full use of the parallel computing capabilities of the hardware. Each data stream is executed in parallel on its own computing unit, and data interaction between different data streams can be synchronized or asynchronously communicated at specific points. For example, in the forward propagation process of a deep learning model, computing tasks at different layers can be executed in parallel on different computing units, thereby speeding up the overall computing speed.

[0032] In some cases, multiple data streams need to be integrated or synchronized at specific points to ensure the correctness and completeness of the calculation results. When a data stream needs the calculation results of other data streams, the system will perform data integration at a specific point and use the output of other data streams as input to continue the calculation. For example, in the back propagation process of the deep learning model, the calculation of the gradient requires the intermediate results of the forward propagation, and data integration will be performed at the appropriate time point. Synchronization operations will be performed at key points (such as splittable calculation operations in the embodiment of the present application) to ensure that all data streams have completed the tasks of the current stage before continuing the calculation. This synchronization mechanism can be an explicit wait operation or an implicit dependency processing. Ultimately, the calculation results of all data streams will be integrated into the overall data processing flow of the model, pushing the model forward to calculate or propagate backward. The integrated calculation results are passed to the next layer or next module of the model as input data to continue the subsequent calculation tasks. For example, the integrated feature map or hidden state will be passed to the next layer of the model for further calculation.

[0033] During the model training process, the calculation results are also used for back propagation to calculate gradients and update parameters. This feedback mechanism ensures that the model can be continuously optimized and improved.

[0034] By dividing complex data calculations into multiple data streams and implementing distributed stream computing on a multi-computing unit hardware architecture, the model's operating efficiency can be significantly improved while fully utilizing hardware resources. This approach not only improves the parallelism and resource utilization of calculations, but also enhances the flexibility and response speed of the system, allowing complex computing tasks to run efficiently on high-performance hardware.

[0035] The following is an introduction based on specific embodiments Figure 1 The specific implementation method of each step is shown.

[0036] As an optional embodiment, in S101, identifying the computing operations in the target model that meet the splitting condition may be implemented as follows: 201. Obtain the calculation type to which each calculation operation in the target model belongs, the data interaction relationship between the calculation operations, and / or the model structure of the target model; 202. Based on the calculation type and / or the data interaction relationship, determine the calculation operations in the target model that meet the splitting condition.

[0037] In the embodiment of the present application, the splitting condition is pre-set based on the model type to which the target model belongs and the model structure.

[0038] In step 201, first, relevant information of the target model needs to be collected and analyzed. Specifically, the computation type (Computation Types), that is, each computation operation (such as convolution, matrix multiplication, activation function, etc.) has its specific computation type. For example, in a deep learning model, convolution operations and matrix multiplication are usually computationally intensive tasks, while activation functions may be relatively simple. By identifying the type of computation operation, it is possible to better determine which operations are suitable for splitting. Data interaction relationships between computation operations (Data Interaction Relationships), analyze the dependencies and data flows between computation operations. For example, the output of some computation operations is the input of other operations. This dependency determines whether these operations can be split and executed in parallel on different computing units. Data interaction relationships are crucial to ensure that the split sub-computation operations can be executed independently or in parallel to a limited extent. Analyzing the mode and frequency of data interaction between computation operations through data interaction relationships helps to optimize data flow and reduce unnecessary communication overhead. The model architecture (Model Architecture) of the target model, such as convolutional neural network (CNN), recurrent neural network (RNN), Transformer, etc., different model structures have different requirements for the dependency and execution order of computation operations. The model structure helps determine which parts can be split and parallelized, identify key nodes and bottlenecks in the model, and better split the computational operations. For example, in CNN, the convolutional layer may contain a large number of parallel computing opportunities, while in RNN, the dependencies between time steps are strong, and the splitting strategy will be different.

[0039] In step 202, after obtaining information such as the calculation type and data interaction relationship, the next step is to determine which calculation operations meet the splitting conditions based on this information. Specifically, based on the calculation type and data interaction relationship, filter out the calculation operations suitable for splitting. For example, some calculation types (such as matrix multiplication) perform better in parallel computing and have less data interaction, and these operations are more likely to be split. Determine the specific split point in the filtered calculation operations. The selection of the split point should take into account the granularity of the calculation operation and the continuity of the data flow to ensure that the split sub-calculation operations can still be executed efficiently.

[0040] It can be understood that the splitting conditions are related to the following characteristics of the computing operations. First, the independence of computing operations. If some computing operations are relatively independent, that is, their execution does not strongly depend on the results of other operations, then these operations are ideal candidates for splitting. For example, in the forward propagation of a neural network, the calculations of different layers can be performed relatively independently, so they are suitable for splitting. Second, computational intensity. Computational intensive operations (such as matrix multiplication, convolution, etc.) are usually performance bottlenecks. Splitting these operations and executing them on hardware with stronger parallel computing capabilities (such as GPU, TPU) can significantly improve performance. Third, data dependency. If the data dependency between computing operations is weak, or the dependency can be resolved by reconfiguring the data flow, then these operations can also be split. For example, in some network structures, operations of non-adjacent layers can be executed in parallel because there is no direct data dependency between them. Fourth, the adaptability of model type and model structure. Depending on the type and structure of the target model, certain specific types of models are naturally suitable for operation splitting. For example, in deep convolutional networks, convolutional layers can be split to different GPUs for parallel processing, and in the Transformer model, multi-head computations in the self-attention mechanism can also be split.

[0041] Exemplarily, the splitting conditions are pre-set based on the model type and model structure of the target model. For example, complex computing operations (such as large-scale matrix multiplication or convolution operations) are split to utilize parallel computing capabilities. For example, computing operations with local data interaction are split into multiple sub-operations to reduce the overhead of global data transmission. For example, in computing-intensive modules (such as certain layers in the model), key computing operations are identified and split to improve overall computing efficiency. For example, computing operations that can be executed in parallel on hardware resources are identified and split to fully utilize the advantages of multi-core or multi-device.

[0042] Through the above steps and the application of splitting conditions, by identifying and splitting complex computing operations and assigning them to multiple computing units for parallel execution, the computing speed and efficiency are significantly improved. According to the computing type and data interaction relationship, the computing tasks are dynamically assigned to the most suitable hardware resources to improve the resource utilization of the overall computing system. By optimizing the data interaction relationship, unnecessary communication between computing operations is reduced, the communication overhead is reduced, and the system performance is further improved. The splitting conditions are pre-set and flexibly adapted to different types of models and their structures, making the optimization method widely applicable and scalable.

[0043] In summary, by analyzing and identifying the computing operations and their relationships of the target model in detail and optimizing them based on preset splitting conditions, the execution efficiency and performance of the model on the hardware computing system can be significantly improved.

[0044] Further optionally, in the above steps, based on the calculation type and / or the data interaction relationship, determining the calculation operation in the target model that meets the splitting condition can be implemented as follows: If the target model is a large floodlight model, determining that the calculation type in the target model is a calculation operation of matrix multiplication is used as a calculation operation that meets the splitting condition. And / or determining that the data interaction structure in the target model is a calculation operation obtained by concatenating matrix multiplication and activation function is used as a calculation operation that meets the splitting condition.

[0045] In an embodiment of the present application, the calculation operations that meet the splitting conditions in the target model can be identified by a specific model type (such as a large-scale model) and a specific data interaction structure. Specifically, if the target model is a large-scale model (also known as a large pre-trained model, such as BERT, GPT, etc.), the calculation operations whose calculation type is matrix multiplication in the target model are determined as calculation operations that meet the splitting conditions. Exemplarily, it is assumed that the target model is a BERT model, which contains multiple self-attention mechanism layers. In these layers, matrix multiplication operations account for most of the calculation amount. By identifying these operations whose calculation type is matrix multiplication, these operations can be further split to improve computational efficiency. Matrix multiplication operations are highly parallel and can be efficiently executed on multiple GPUs or TPUs, thereby significantly improving the calculation speed. Distributing matrix multiplication operations to multiple computing units can make full use of hardware resources, reduce the load of a single computing unit, and improve the efficiency of the overall system.

[0046] Determine the data interaction structure in the target model as a calculation operation obtained by concatenating matrix multiplication and activation function, as a calculation operation that meets the splitting condition.

[0047] For example, in the self-attention mechanism layer of the BERT model, the matrix multiplication operation is usually followed by an activation function (such as ReLU, GELU, etc.). There is a close data interaction relationship between these computing operations. By identifying this data interaction structure, the matrix multiplication and activation function can be spliced ​​into a sub-computation operation, which can be split and executed on multiple operation units. In this way, closely related computing operations (such as matrix multiplication and activation function) are combined together, reducing the transmission overhead of intermediate data and improving the efficiency of data processing. By optimizing the data interaction structure, unnecessary memory access and data rearrangement can be reduced, further improving the performance of the computing system.

[0048] Assume that the target model is a BERT model, and the model contains the following sequence of computational operations: matrix multiplication operation: calculation of attention score; activation function operation: application of GELU activation function. Through the above splitting conditions, the following computational operations can be identified and determined to meet the splitting conditions: splitting and parallel execution of matrix multiplication operations as separate sub-computation operations. Alternatively, splitting and parallel execution of operations consisting of matrix multiplication and activation function concatenation as a composite sub-computation operation.

[0049] As a result, matrix multiplication operations are performed in parallel on multiple GPUs, significantly shortening the computation time. The concatenation of matrix multiplication and activation functions is performed in parallel on multiple computing units, further reducing computational latency. By rationally allocating computational tasks to multiple computing units, hardware resources can be fully utilized to avoid idle or overloaded resources. Closely related computational operations (such as matrix multiplication and activation functions) are performed on the same computing unit, reducing data transmission overhead and communication latency. By identifying and splitting key computational operations, the data processing process is optimized, and the computational efficiency and performance of the overall system are improved. In summary, through splitting conditions based on specific model types and data interaction structures, the system can effectively identify and optimize computational operations and improve the execution efficiency and performance of the model on the hardware computing system.

[0050] As an optional embodiment, in S102, splitting the computing operation to obtain multiple sub-computing operations can be implemented as the following steps: 301. Determine a candidate splitting method for the computing operation. The candidate splitting method is determined based on a computing type of the computing operation.

[0051] In this step, multiple candidate splitting methods are determined for the computational operations to be split. The definition of candidate splitting methods is usually closely related to the type of computational operation, such as matrix multiplication, convolution, etc. Determine the corresponding splitting strategy based on the specific type of computational operation (such as matrix multiplication, convolution, activation function, etc.). For matrix multiplication, common splitting methods include splitting by row, splitting by column, or dividing the entire operation into multiple sub-matrix multiplications. For each type of computation, analyze in which dimensions the split can be performed. For example, in convolution operations, you can consider splitting the input channel, output channel, or spatial dimension. List all possible splitting methods, for example, two-dimensional matrix multiplication can be split into multiple one-dimensional operations, and block partitioning of the input and output can also be considered.

[0052] 302. Based on the data interaction relationship between the computing operation and other computing operations in the target model, obtain the computing efficiency corresponding to the candidate splitting method.

[0053] In this step, the computational efficiency of each candidate split method is analyzed and evaluated in relation to the data interaction with other computing operations. For each candidate split method, the performance indicators such as the time required for execution, the memory bandwidth occupied, and the data transmission overhead are calculated. These efficiency data can be obtained through simulation or benchmarking. Combined with the arrangement of other computing operations in the target model, the impact of the candidate split method on the overall computing process is evaluated. If a split method can effectively reduce data transmission, increase computing speed, or make better use of computing unit resources, its computational efficiency is higher. The computational efficiency of all candidate split methods is compared to find the optimal split strategy. In this process, indicators such as time complexity, memory utilization, and overall computing efficiency can be used for quantitative evaluation.

[0054] One optional embodiment is that in step 302, the number of other computing operations associated with the computing operation in the target model is first obtained. The other computing operations associated with the computing operation are used to indicate the computing operations applied to part or all of the computing results in the computing operation during the computing process. Then, the computing efficiency corresponding to the candidate splitting method is determined based on the number of other computing operations associated with the computing operation. The fewer the number of other computing operations associated with the computing operation, the higher the computing efficiency corresponding to the candidate splitting method.

[0055] Specifically, the fewer associated computing operations, the less frequently the output of the computing operation is used by subsequent computing operations, so a more efficient splitting method can be selected. The core principle of this method is to reduce data dependencies and thus improve the efficiency of parallel computing.

[0056] Another optional embodiment is that in the above step 302, the number of data aggregations to be performed after the calculation operation in the target model is first obtained. Then, the operation efficiency corresponding to the candidate splitting method is determined based on the number of data aggregations to be performed after the calculation operation. The fewer the number of data aggregations to be performed after the calculation operation, the higher the operation efficiency corresponding to the candidate splitting method.

[0057] In this optional embodiment, step 302 evaluates the computational efficiency of different candidate splitting methods by analyzing the number of other computing operations associated with the current computing operation. Specifically, the fewer the associated computing operations, the less frequently the output result of the computing operation is used by subsequent computing operations, so a more efficient splitting method can be selected. The core principle of this method is to reduce data dependencies, thereby improving the efficiency of parallel computing.

[0058] In deep learning models or other complex computing models, there are often dependencies between computing operations. For example, the output of computing operation A may be used by multiple operations such as computing operations B, C, and D. If the output of a computing operation is only used by a small number of other operations, it means that there is less data interaction and the degree of parallelism after splitting is higher.

[0059] For a calculation operation, there can be multiple splitting methods (such as splitting by row, splitting by column, etc.). The system evaluates the computing efficiency of each splitting method by analyzing the number of other computing operations associated with the calculation operation. If a certain splitting method corresponds to a small number of associated computing operations, it means that this method can reduce data transmission and dependence, thereby improving the overall computing efficiency.

[0060] The computational efficiency of the split method is inversely proportional to the number of associated computational operations. The fewer the number of other associated computational operations, the more independent the sub-computation operations can be executed after the split, reducing the overhead of synchronization and data transmission, thereby improving parallelism and computational efficiency.

[0061] Suppose there is a deep learning model, which includes a computational operation called large-scale matrix multiplication (MatMul), whose output is used for other subsequent computational operations. The output of the matrix multiplication operation is used by multiple subsequent operations in the model, such as 5 different convolutional layers and 2 activation function layers, a total of 7 associated computational operations. In the first case, candidate split method 1 is to split the matrix multiplication by rows, and candidate split method 2 is to split the matrix multiplication by columns. Since the result of the matrix multiplication operation needs to be used by 7 other computational operations, it means that no matter which split method is chosen, more data transmission and synchronization operations will be involved. Therefore, the computational efficiency of methods 1 and 2 is relatively low.

[0062] In the second case, there are fewer associated computation operations. The output of the matrix multiplication operation is only used by one subsequent activation function layer. Then, candidate splitting method 1 is to split the matrix multiplication by rows, and candidate splitting method 2 is to split the matrix multiplication by columns. Since the result of the matrix multiplication operation is only used by one activation function layer, data interaction and dependency are low. Splitting by columns may be more suitable because it can better utilize computing resources and reduce unnecessary data transmission, thereby improving computing efficiency.

[0063] For the first case, since there are more associated calculation operations, splitting by rows may slightly improve the calculation efficiency. For the second case, since there are fewer associated calculation operations, splitting by columns is more efficient because it reduces the complexity of data interaction and can better utilize parallel computing resources.

[0064] In the above embodiment, by selecting a splitting method with a smaller number of associated computing operations, the data dependency between computing operations can be effectively reduced, so that each sub-computing operation can be executed more independently, thereby improving the degree of parallelism. In addition, the splitting method with fewer associated computing operations means that the data transmission and synchronization operations required after the splitting will also be reduced accordingly, which will significantly reduce the communication overhead and improve the computing efficiency. Furthermore, reducing data dependency and data transmission allows each sub-computing operation to be executed in parallel on multiple computing units more efficiently, thereby greatly improving the overall computing performance. At the same time, choosing a suitable splitting method can not only improve computing efficiency, but also help to better utilize hardware resources (such as multi-core CPU, GPU, TPU, etc.) to avoid idle or overloaded resources. When dealing with large-scale deep learning models or complex computing tasks, this method can flexibly adapt to different computing needs and optimize overall performance.

[0065] Further optionally, after step 302, the storage space occupancy rate corresponding to the calculation operation in each candidate splitting mode may also be obtained. Then, based on the ranking relationship of the storage space occupancy rate, the numerical value of the operation efficiency corresponding to each candidate splitting mode is re-determined.

[0066] In this embodiment, the effect of the storage space occupancy rate on the computational efficiency of the candidate splitting methods is additionally evaluated after step 302. Specifically, the storage space required for each candidate splitting method during execution is calculated, and by comparing these storage space occupancy rates, the computational efficiency values ​​of each splitting method are readjusted and determined.

[0067] For each candidate split, calculate the memory space required during execution. This includes the storage of input data, storage of intermediate calculation results, and storage of output results. Compare the storage space required for each split with the total available storage space and calculate the storage space occupancy of each split.

[0068] The storage space occupancy rate of each split method is sorted, and the split method with lower occupancy rate has higher priority. Based on the original computing efficiency value, the storage space occupancy rate is adjusted. The computing efficiency value of the split method with lower storage space occupancy rate will be higher accordingly, because lower storage requirements can reduce memory bottlenecks and improve overall computing efficiency.

[0069] For example, suppose there is a deep learning model, which includes a computing operation of large-scale matrix multiplication (MatMul), and its candidate splitting methods include splitting by row, splitting by column, and splitting by block. For these three methods, the original computing efficiency evaluation results are: the computing efficiency value of method 1 (splitting by row) is 0.7, method 2 (splitting by column) is 0.8, and method 3 (splitting by block) is 0.9. At the same time, the storage space occupancy evaluation results show that the storage space occupancy of method 1 is 20%, method 2 is 30%, and method 3 is 15%. When re-determining the computing efficiency, according to the ranking of storage space occupancy, method 3 (15%) has the highest priority, followed by method 1 (20%) and method 2 (30%). Assuming that the computing efficiency value increases by 0.1 for every 10% reduction in storage space occupancy, the adjusted computing efficiency values ​​are: method 3 is adjusted from the original 0.9 to 1.1, method 1 is adjusted from the original 0.7 to 0.8, and method 2 is adjusted from the original 0.8 to 0.7. In this way, not only the storage space utilization is improved, but also the overall computing performance of matrix multiplication operations is optimized.

[0070] In this way, by considering the storage space occupancy rate, the system can choose a splitting method with lower memory requirements, thereby reducing memory bottlenecks and improving computing efficiency. Combining storage space occupancy rate and computing efficiency, the system can better balance the use of memory and computing resources, and avoid performance degradation due to insufficient memory. For hardware environments with limited memory (such as embedded devices, mobile devices, etc.), it is particularly important to choose a splitting method with low storage space occupancy rate. This method can ensure that when the model runs on these hardware, performance will not be limited due to memory problems. By optimizing storage space usage, the system can reduce memory access latency and data transmission overhead, further improving overall computing performance.

[0071] By further evaluating the storage space occupancy after step 302 and adjusting it in combination with the original computing efficiency, this method can more comprehensively evaluate and select the best splitting method. Not only the computing efficiency is considered, but also the memory utilization is taken into account, so that better computing performance can be achieved in different hardware environments. This method is particularly suitable for application scenarios with limited memory resources or strict requirements on memory optimization.

[0072] 303. Use a candidate splitting method with the highest computational efficiency to split the computing operation to obtain multiple sub-computing operations.

[0073] After evaluating all candidate splitting methods, the strategy with the highest computational efficiency is selected to split the original computing operation to generate multiple sub-computing operations. Based on the evaluation result of step 302, the candidate splitting method with the highest computational efficiency is selected and the original computing operation is split using this method. For matrix multiplication, it can be divided into multiple smaller matrix multiplication operations. Use the selected splitting method to perform actual code implementation, generate corresponding sub-computing operations, and assign computing units to these sub-operations. Ensure that each sub-computing operation can still maintain correctness after splitting and that the data is smooth. Schedule the generated multiple sub-computing operations to available computing units to ensure load balancing, so as to optimize resource utilization and improve overall computing efficiency.

[0074] One optional embodiment is that in step 303, if the target model is a floodlight model, the weight matrix is ​​vertically split into a plurality of first weight matrices. Then, the input matrix is ​​vertically split into a plurality of first sub-input matrices. Furthermore, multiplication calculations between the plurality of first weight matrices and the plurality of first sub-input matrices are used as the corresponding plurality of first sub-calculation operations.

[0075] In this way, by splitting a large matrix into multiple small matrices, each sub-computation operation can be executed in parallel on an independent computing unit (such as a multi-core CPU or GPU). This can significantly improve the computing efficiency and reduce the overall computing time. When splitting, the size of each sub-weight matrix and sub-input matrix is ​​small, which reduces the memory usage during the calculation process and avoids the performance bottleneck caused by insufficient memory. It is especially suitable for hardware environments with limited resources. After the matrix is ​​split, the amount of data transmission required is reduced, which can reduce the data transmission delay and bandwidth usage and improve the throughput of the overall system. This helps to efficiently utilize network resources in a distributed environment. For large neural networks, this splitting method can be used to easily perform distributed training or reasoning on the model, so that it can be processed in parallel on multiple machines, which not only improves the training speed, but also meets the computing needs of large models. On different tasks or data sets, the split weight matrix and input matrix can be used to flexibly and quickly implement various experiments without rebuilding the entire model, which helps to quickly iterate research and development.

[0076] By vertically splitting the weight matrix and input matrix in the floodlight model, the computational complexity can be reduced while improving computational efficiency and memory utilization. This strategy not only improves computing performance, but also helps to flexibly respond to different application scenarios, greatly enhancing the adaptability and scalability of the model.

[0077] Further optionally, in the above steps, applying multiple sub-calculation results to the data processing flow of the target model can be implemented as: adding multiple sub-calculation results obtained by multiple first sub-calculation operations to obtain a final calculation result corresponding to the calculation operation.

[0078] In the above steps, multiple sub-calculation results are applied to the data processing flow of the target model, which is specifically implemented by adding the sub-calculation results obtained by the multiple first sub-calculation operations to obtain the final calculation result corresponding to the calculation operation.

[0079] For a large-scale matrix multiplication operation Y = W cdot X, it is split into multiple sub-computation operations Yi = Wicdot Xi, where Wi is a block of the weight matrix and Xi is the corresponding input matrix block. The results Yi of multiple sub-computation operations are added together to obtain the final calculation result Y. Exemplarily, in a large neural network, assume that the dimension of the weight matrix W is 1000×4000 and the dimension of the input matrix X is 4000×500. First, the weight matrix W is split vertically into 4 first weight matrices W1, W2, W3, and W4, each of which has a dimension of 1000×1000. Next, the input matrix X is also split vertically into 4 first sub-input matrices X1, X2, X3, and X4, each of which has a dimension of 1000×5001000×500. Then, perform multiplication operations Y1=W1·X1, Y2=W2·X2, Y3=W3·X3, and Y4=W4·X4 to obtain sub-calculation results Y1, Y2, Y3, and Y4. Finally, add these sub-calculation results to obtain the final calculation result Y=Y1+Y2+Y3+Y4, thus completing the entire calculation process. This method not only improves the parallel computing capability, but also optimizes memory usage and data transmission, simplifies the calculation process, and enhances the flexibility and scalability of the model.

[0080] In this way, by splitting and executing sub-computation operations in parallel, multiple computing units (such as multi-core CPUs or GPUs) can process different sub-computation tasks at the same time, greatly improving computing efficiency and reducing overall computing time. The matrix size involved in each sub-computation operation is small, thereby reducing memory usage and avoiding memory bottlenecks, especially in environments with limited memory resources. The split computing operations involve less data, which can reduce data transmission delays and bandwidth usage, and improve system throughput, especially in distributed computing environments, where the optimization effect is obvious. The simple addition operation of sub-computation results avoids complex merging operations, simplifies the calculation process, reduces implementation complexity, and also reduces computing overhead. By splitting and combining sub-computation results, the model can be flexibly deployed in different hardware environments, adapt to different computing resource configurations, and improve the scalability and adaptability of the system.

[0081] Assume that during the calculation process, the result of the first sub-calculation operation is Y1=W1·X1=[1, 2, 3], the result of the second sub-calculation operation is Y2=W2·X2=[4, 5, 6], the result of the third sub-calculation operation is Y3=W3·X3=[7, 8, 9], and the result of the fourth sub-calculation operation is Y4=W4·X4=[10, 11, 12]. Adding these sub-calculation results together, we can get the final calculation result YY: Y=Y1+Y2+Y3+Y4=[1+4+7+10, 2+5+8+11, 3+6+9+12]=[22, 26, 30].

[0082] In this way, by aggregating multiple sub-calculation results, a comprehensive final result is obtained, demonstrating the effectiveness of achieving performance optimization through effective splitting and combination of calculations.

[0083] By adding up the results of multiple sub-calculations, the results of each sub-calculation operation can be effectively integrated to obtain the final calculation result. This method not only improves computing efficiency, optimizes memory usage and data transmission, but also simplifies the calculation process and enhances the flexibility and scalability of the model. In scenarios dealing with large-scale data processing, this method can significantly improve the overall performance and adaptability of the system.

[0084] Another optional embodiment is that in the above step 303, if the target model is a large floodlight model, the weight matrix is ​​split horizontally into multiple second weight matrices. Then, the multiplication calculation between each of the multiple second weight matrices and the input matrix is ​​used as the corresponding multiple second sub-calculation operations.

[0085] For large matrix operations, the parallel computing capabilities of multi-core processors or GPUs can be fully utilized through horizontal splitting and parallel computing. Each sub-computation operation Yi can be performed independently, significantly reducing the overall calculation time. After horizontal splitting, the size of each second weight matrix Wi is reduced, which reduces memory usage, avoids memory overflow when memory resources are limited, and uses computing resources more efficiently. Splitting the weight matrix horizontally makes the execution of the model more flexible on different hardware platforms. For example, the same model can be executed on GPUs with different video memory sizes without reconstructing the overall structure of the model. By using the entire input matrix X, multiple data transfers in the calculation can be avoided, data utilization can be improved, and unnecessary data copy operations can be reduced, thereby improving the transmission efficiency of the overall calculation.

[0086] This splitting method allows dynamic adjustment of each sub-computation result, such as adjusting the participation of each component according to the amount of calculation or memory load at runtime, so as to flexibly respond to different computing needs.

[0087] Correspondingly, in the above steps, applying the multiple sub-computation results to the data processing flow of the target model can be implemented as: splicing the multiple sub-computation results obtained by the multiple second sub-computation operations to obtain the final calculation result corresponding to the calculation operation.

[0088] In practical applications, for a large input matrix XX (such as image data containing many samples), the following sub-calculation results may be obtained through the above splitting and calculation: Y1=[1, 2, 3, ..., 1000] Y2=[4, 5, 6, ..., 2000] Y3=[7, 8, 9, ..., 3000] Y4=[10, 11, 12,..., 4000] Finally, the concatenation operation combines these results together, and the result is: Y = [1, 2, 3, ..., 1000; 4, 5, 6, ..., 2000; 7, 8, 9, ..., 3000; 10, 11, 12, ..., 4000] This processing approach ensures efficient computation and optimal use of memory, while also helping to improve the performance of the model in practical applications.

[0089] Another optional embodiment is that in the above step 303, if the target model is a large floodlight model, the input matrix is ​​split, and the GeLU operation associated with the split input matrix is ​​used as the corresponding sub-computation operation.

[0090] In this way, by splitting the input matrix and applying the GeLU operation independently, we can make full use of parallel computing resources, such as multi-core CPUs or GPUs, thereby greatly improving the computational efficiency. Each sub-computation operation can be executed simultaneously, reducing the overall computation time. After splitting the input matrix, the size of each sub-input matrix is ​​reduced, which can effectively reduce memory usage, especially in environments with limited memory resources, to avoid memory bottlenecks and make more efficient use of available memory resources. For large GeLU operations, splitting into multiple sub-operations can reduce the computational complexity of each operation. The GeLU function involves nonlinear calculations, and the amount of operations on each sub-matrix is ​​reduced after splitting, thereby reducing the complexity of the entire computational process.

[0091] During the training process of neural networks, especially in models that use the GeLU activation function such as Transformer, by splitting and executing GeLU operations in parallel, the speed of forward propagation and backpropagation can be significantly accelerated, thereby improving the training speed of the model.

[0092] This splitting method allows the size of the submatrix to be dynamically adjusted according to the amount of data in practical applications, so as to flexibly cope with input data of different sizes. This flexibility is particularly important when dealing with large amounts of data, and can improve the adaptability and practicality of the model.

[0093] For example, assume that in an input matrix containing multiple samples, each sample is represented as a row vector, and the input matrix X has a size of 4000×5004000×500 and is split into four blocks: X1=[x1, 1x1, 2…x1, 500x1000, 1x1000, 2…x1000, 500] X2=[x1001, 1x1001, 2…x1001, 500x2000, 1x2000, 2…x2000, 500] X3=[x2001, 1x2001, 2…x2001, 500x3000, 1x3000, 2…x3000, 500] X4=[x3001, 1x3001, 2…x3001, 500x4000, 1x4000, 2…x4000, 500] Then, apply the GeLU operation to each sub-input matrix, for example, Y1=GeLU(X1), Y2=GeLU(X2), Y3=GeLU(X3), Y4=GeLU(X4). The final concatenation result is: Y=[Y1; Y2; Y3; Y4].

[0094] This processing method can not only effectively improve computing efficiency and optimize memory usage, but also significantly improve the operating speed and data processing capabilities of large neural networks without adding additional hardware resources.

[0095] Through the execution of these three steps, the best splitting method can be determined based on the nature and performance diagnosis of the computing operation to be split, and the computing operation can be optimized into multiple efficiently executed sub-computing operations. This not only improves computing performance, but also optimizes resource usage and adapts to the needs of different hardware environments. Such a process ensures that in complex computing tasks, the model can maximize the potential of computing resources and improve overall execution efficiency.

[0096] S103. When the target model is loaded into the hardware computing system, multiple sub-computing operations are reconfigured into corresponding target computing units.

[0097] In an optional example, during the process of loading the target model into the hardware computing system, before reconfiguring the multiple sub-computing operations to the corresponding target computing units, the first data transmission efficiency between the candidate computing units to be configured for the multiple sub-computing operations and the second data transmission efficiency between the candidate computing units and the surrounding computing units can also be obtained. Then, the candidate computing unit corresponding to the optimal solution of the first data transmission efficiency and the second data transmission efficiency is used as the target computing unit corresponding to each of the multiple sub-computing operations.

[0098] In the above optional example, in the process of loading the target model into the hardware computing system, the computing efficiency and system performance can be significantly improved by optimizing the configuration of the sub-computing operations on the hardware computing units. Specifically, before reconfiguring multiple sub-computing operations to corresponding computing units, the system first obtains the first data transmission efficiency between the candidate computing units of each sub-computing operation, and the second data transmission efficiency between these candidate computing units and other surrounding computing units. Then, by solving the optimal combination of the two, the most suitable computing unit is selected to perform each sub-computing operation.

[0099] By selecting the computing unit with the highest data transmission efficiency, the data transmission delay between sub-computing operations can be reduced, thereby speeding up the overall computing speed, especially when processing large-scale neural network models. This optimization effect is particularly significant. By comprehensively considering the first and second data transmission efficiencies, the system can allocate computing resources more evenly, avoiding the situation where some computing units are overloaded while other computing units are idle, thereby improving the overall utilization of hardware resources. Efficient data transmission and balanced resource utilization can reduce unnecessary data movement and repeated calculations, thereby reducing the energy consumption of the system, which is particularly important for large-scale computing tasks and high-performance computing environments. By optimizing the selection of computing units, system congestion or failures caused by data transmission bottlenecks or resource contention can be reduced, thereby enhancing the stability and reliability of the system. This approach allows the system to dynamically adjust the configuration of computing units according to the current load conditions at runtime, ensuring optimal performance under different computing tasks and load conditions.

[0100] For example, assume that the sub-computing operations Y1, Y2, Y3, and Y4 of a neural network model need to be configured to four computing units U1, U2, U3, and U4 in the hardware system. The system first measures the first data transmission efficiency between each sub-computing operation and the candidate computing unit (for example, the efficiency from Y1 to U1 is 90%, the efficiency to U2 is 85%, etc.), and the second data transmission efficiency between these computing units and other surrounding units (for example, the efficiency between U1 and U2 is 95%, etc.). By comprehensively analyzing these efficiency data, the system selects the optimal combination of computing units, such as configuring Y1 to U2, Y2 to U3, etc. This optimized configuration not only speeds up the data transmission speed, but also balances the computing load, thereby significantly improving the overall computing performance.

[0101] In another optional example, during the process of loading the target model into the hardware computing system, before reconfiguring the multiple sub-computing operations to the corresponding target computing units, the distances between the candidate computing units to be configured for the multiple sub-computing operations can be obtained based on the data interaction relationship between the computing operation and other computing operations in the target model. Then, the candidate computing units whose distances are within the preset range are used as the target computing units corresponding to the multiple sub-computing operations.

[0102] In this example, the goal is to optimize the configuration of sub-computing operations on hardware computing units based on the data interaction relationship between computing operations during the process of loading the target model into the hardware computing system. Specifically, before selecting the target computing unit, the data interaction relationship between each sub-computing operation and other computing operations in the target model is analyzed, and based on this, the distances between the candidate computing units to be configured for each of the multiple sub-computing operations are obtained. Then, those computing units whose distances are within a preset range are selected as the target computing units corresponding to each sub-computing operation.

[0103] Specifically, first, the system analyzes the data interaction relationship between each sub-computation operation in the target model and other computing operations in the model. This includes data dependencies, the direction of data flow, and data transmission requirements between computing operations. Based on the data interaction relationship, the system obtains the physical distance or logical distance between the candidate computing units to be configured for each sub-computation operation. The distance here can be the distance in physical position (such as the position on the hardware chip) or the logical communication delay (such as the transmission time in the network). Further optionally, a preset range is defined, which can be based on an empirical value or the best performance area obtained through experiments. For example, a suitable distance threshold is set, indicating that the computing units within this distance range are considered to have high data transmission efficiency. The candidate computing units whose distances are within the preset range are screened out and used as the target computing units corresponding to each sub-computation operation. For example, for the candidate computing unit of the sub-computation operation Yi, the system calculates the distance between the candidate computing unit and other computing units. If the distance meets the distance threshold, the candidate computing unit is selected as the target computing unit of Yi.

[0104] In this way, by selecting computing units that are closer, the time delay of data transmission can be significantly reduced. Especially in computing operations that require frequent data interaction, this optimization can greatly improve the overall computing efficiency. This method makes the geographical or logical distribution of computing resources more reasonable, avoids the bottleneck problem that may be caused by long-distance data transmission, and thus improves the overall utilization of hardware resources. Reducing the need for long-distance data transmission can reduce the load on the network or hardware transmission link, thereby enhancing the stability and reliability of the system. By reducing unnecessary long-distance data transmission, the energy consumption of the system can be reduced, which is particularly important for large-scale computing tasks and high-performance computing environments. This method allows the system to dynamically adjust the configuration of the computing unit according to the current data interaction mode at runtime, so as to flexibly respond to different computing tasks and load conditions.

[0105] Assume that a neural network model contains four sub-computation operations, which need to be executed on four computing units of the hardware system. After analyzing the data interaction relationship, the system determines the data dependency of each sub-computation operation on other operations, and then measures the distance between candidate computing units. Based on these data and the frequency of data interaction between computing units, select the target computing unit for sub-computation operation Y1. Assume that the distances between the target computing unit and computing units U2 and U4 of sub-computation operations Y2 and sub-computation operations Y4 are 5 and 10 respectively, and the distance between U2 and U4 is 3, both within the preset range.

[0106] Through this optimized configuration, data transmission delay can be greatly reduced, computing efficiency and system performance can be improved, while hardware resource utilization and energy efficiency are also improved.

[0107] S104. Execute the corresponding sub-computation operations through the target computing unit to obtain multiple sub-computation results.

[0108] In this step, the most suitable target computing unit has been assigned to each sub-computation operation. Next, the system will execute the respective sub-computation operations through these computing units to obtain the corresponding sub-computation results. According to the previous optimization configuration, each sub-computation operation is assigned to the corresponding target computing unit. These computing units may be different processor cores, GPU cores, TPUs or other computing resources in the hardware system. The target computing units begin to execute the sub-computation operations assigned to them. These operations can be common deep learning computing tasks such as matrix multiplication, convolution, activation functions (such as ReLU, GeLU), pooling, etc. The computing units perform corresponding computing tasks according to their specific computing instruction sets and architectures. These tasks are usually highly parallelized and can make full use of the computing power of the hardware.

[0109] After each computing unit completes the calculation, it generates the corresponding sub-calculation results. These results may be the output of the intermediate layer, feature maps, hidden states, etc. These sub-calculation results are usually stored in the form of tensors, and the dimensions of the tensors depend on the specific calculation operations and the architecture of the model.

[0110] S105: Apply the multiple sub-calculation results to the data processing flow of the target model.

[0111] After generating multiple sub-calculation results, the system needs to reintegrate these results into the overall data processing flow of the model to drive the model forward calculation or backward propagation. This step ensures that the sub-calculation results can correctly participate in subsequent calculations, thereby completing the calculation task of the entire model.

[0112] After obtaining multiple sub-calculation results, they are reassembled according to the model's computational graph or data flow. This may involve operations such as concatenation, addition, and element-by-element product of multiple sub-calculation results. According to the model's architecture and data dependencies, the sub-calculation results are integrated into an input format suitable for processing by the next layer or module. The integrated sub-calculation results are passed as input data to the next layer or module in the model. These results may be used as input to the next layer's calculation, or as part of a feedback mechanism to participate in the back-propagation process in training. In the forward propagation of the neural network, the sub-calculation results are passed to subsequent layers for further calculation. In the back-propagation, these results are used for gradient calculation and parameter update. The application of sub-calculation results will push the calculation process of the entire model forward until the forward or reverse calculation of the entire model is completed. According to the model's computational graph, the integrated results are passed to the subsequent operation units in sequence to ensure that each calculation step can be executed correctly, thereby generating the final model output or completing the back-propagation in training.

[0113] By executing steps S104 and S105, the system can efficiently distribute computing tasks to different computing units and ensure that the computing results can be correctly integrated into the overall data processing flow of the model. This method improves computing efficiency, fully utilizes hardware resources, and ensures the accuracy and completeness of model calculations.

[0114] In an embodiment of the present application, there are many structures in the Bloom model in which matrix multiplication is followed by an activation function. Assuming that there is an input matrix X, an output matrix Y, and a weight matrix A, the structure in which the matrix multiplication in the Bloom model is followed by an activation function is split and calculated to improve the model operation efficiency. Specifically, when there is a splittable matrix multiplication in the model, the weight matrix can be split into multiple operation units by column. Each operation unit independently performs part of the matrix multiplication, and the result can independently enter the activation function to participate in subsequent calculations. The advantage of this splitting method is that it allows parallel processing, thereby significantly improving the calculation speed.

[0115] When choosing a split method, you need to consider the impact of subsequent computing operations to choose a method that is more convenient for subsequent computing and generates fewer data aggregation operations. For example, left-right splitting is usually more efficient because it only requires splitting the weight matrix, and the result matrix can be directly spliced ​​to get the complete result. In contrast, although the upper-lower split requires less computing space, it may require additional data aggregation operations, which will increase the computational overhead.

[0116] In addition, it can also be applied to other splittable computational operations in the floodlight model, even without matrix multiplication. For example, for some computations that do not require complete data (such as the GeLU activation function), the input matrix can also be split to allow multiple computational units to execute in parallel. In this case, the choice of splitting method should be mainly based on the principle of facilitating subsequent calculations and reducing the number of aggregations.

[0117] In general, by reasonably splitting the computing operations in the model, efficient parallel computing on multiple computing units is achieved, which improves the overall operating efficiency of the model while having good versatility and flexibility.

[0118] For example, assuming that the splittable part of the floodlight model is matrix multiplication, Figure 2 Two methods of splitting the weight matrix are shown. If the weight matrix A is split into n operation units by column, and then XA1, XA2, ..., XAn are executed, n calculation results Y1, Y2, ..., Yn will be obtained in the end. These results can be independently entered into the activation function to participate in the calculation. The choice of splitting method is mainly affected by the subsequent calculation operations. The selected splitting method should be more convenient for subsequent calculations and generate fewer data aggregation operations in subsequent calculations.

[0119] Figure 3 middle, represents the upper and lower split matrix A, It represents the splitting of the matrix B into upper and lower parts. In forward propagation, f means that the data passing through here remains unchanged, and g means that the data of each computing unit needs to be synthesized into complete data. The meaning is opposite in reverse propagation.

[0120] In the multi-layer perceptron module, you first need to input matrix X and matrix A for matrix multiplication. Here, matrix A is split left and right. If the result of the upper and lower split is obtained, it cannot continue the subsequent GeLU and needs to be aggregated immediately. This will obviously bring more consumption than the left and right split. After the left and right split and GeLU calculation, Y is obtained. Here, Y is the result of different positions. Then it needs to be matrix multiplied with the B matrix. Here, the B matrix adopts the upper and lower split, and can only be split up and down, because the input matrix at this time is equivalent to having completed the left and right split. The subsequent dropout calculation requires complete data, so it needs to be aggregated first. The choice of splitting method here and its subsequent impact are not easy to explain clearly. You can understand it by combining the description of the two splitting methods in the first picture.

[0121] It can be explained that the calculation process of the self-attention module is similar. The self-attention module only requires two splits and one aggregation of data processing procedures, which will not be repeated here.

[0122] The main modules in the Floodlight model are the multi-layer perceptron module and the self-attention mechanism module. The remaining parts can also be split, but the efficiency improvement is small. If the structure of the self-attention module and the multi-layer perceptron module in other neural network models has not changed, this split method can be used directly. If the structure has changed, some modifications may be required, depending on the actual application scenario.

[0123] exist Figure 2 It can be seen that the final result is still obtained after the left and right split calculations, or each operation unit obtains the final result at a different position. As long as these results are spliced ​​or combined together, the same result as the unsplit can be obtained. If the upper and lower splits can obtain partial results of the same size as the unsplit calculation, these results need to be added to get the final result. Most calculations can continue through the results at different positions, but some results cannot continue most of the calculations. Failure to continue means that aggregation is required, that is, adding the results in each operation unit and then synchronizing them to each operation unit. Aggregation will bring a lot of overhead. For higher model operation efficiency, the choice of splitting method should generate as few aggregation operations as possible. This needs to be selected according to the structure of the model and the required calculations.

[0124] In most cases of matrix multiplication, the left-right split method is more efficient in terms of weight splitting, while the top-bottom split method requires less computational space. The specific reasons are as follows: Figure 2 It can also be roughly seen that the left and right split only needs to split the weights, and after getting the results, you only need to splice to get the complete result, but each operation unit needs to store 14 values ​​​​(X, A1, Y1) at the same time, while the upper and lower splits not only need to split the weights but also the input matrix. The result matrix also needs to be added to get the final result, but this split method only needs to store 12 values ​​​​(X1, A1, Y1) at the same time.

[0125] If a module does not have matrix multiplication, but there are some calculations that do not require all the data, you can also try to split it. For example, if there is no XA matrix multiplication in the second picture, the input directly participates in the GeLU calculation. Because the calculation does not require complete data, you can also split the input matrix to allow multiple operation units to run GeLU calculations at the same time. Here, the splitting method does not affect the calculation efficiency of GeLU. As for which method to choose, it is still based on the convenience of subsequent calculations to reduce the number of aggregations. The partial result obtained is equivalent to the complete result split by column. In the subsequent matrix multiplication, the weight matrix can be split by row, so that the partial result obtained previously can be directly used for calculation without additional communication overhead.

[0126] According to the technical solution given in the above steps, similar computing structures can be split into multiple computing units, and then the data can be aggregated together only when synchronization is required. Figure 3 and Figure 4 As shown in the figure, the data flow diagram of the Self-Attention module and the Multi-Layer Perceptron (MLP) module in the Floodlight model after applying this method. Figure 3 and Figure 4 In , f is the identity operator of forward propagation, indicating that the data does not change after forward propagation passes through it. It is also the aggregation operator of back propagation, that is, the data needs to be synchronized here, while g is the opposite of f. Because the attention (Self-Attention) module in the bloom model adopts the multi-head attention (Multi-Head Attention) mechanism, the multi-head attention (Multi-Head Attention) calculation itself is parallel, because each head is independent, so the calculation in the self-attention (Self-Attention) module is very suitable for split calculation.

[0127] In the embodiment of the present application, it is possible to identify the computing operations that meet the splitting conditions in the target model; split the computing operations to obtain multiple sub-computing operations; in the process of loading the target model into the hardware computing system, reconfigure the multiple sub-computing operations to the corresponding target computing units; the hardware computing system includes at least multiple computing units, each of which is composed of hardware chip resources called based on assembly instructions; execute the corresponding sub-computing operations through the target computing units to obtain multiple sub-computing results; apply the multiple sub-computing results to the data processing flow of the target model. The embodiment of the present application can improve computing efficiency, hardware utilization, flexibility and real-time performance, reduce computing operation latency, perform hardware acceleration operations for the model, and improve the performance of the neural network model by identifying, splitting, reconfiguring and executing computing operations in the neural network model in parallel.

[0128] In particular, the embodiments of the present application can split the complex mathematical calculations in the neural network model into different computing units, reducing the waiting time required between computing operations, and this method can improve the utilization rate of hardware resources per unit time, give full play to hardware performance, help avoid waste caused by idle resources, and enable neural network models (such as large floodlight models, etc.) to be more efficiently implemented and deployed on hardware with multiple computing units.

[0129] Based on the same implementation principle, an embodiment of the present application also provides a model optimization device for implementing the model optimization method in the above embodiment.

[0130] Figure 5This is a structural block diagram of a model optimization device provided in an embodiment of the present application, which is applied to a hardware computing system deployed with a target model, wherein the target model is a neural network model. Figure 5 As shown, the device comprises at least the following units: an identification unit, configured to identify computing operations in the target model that meet the splitting condition; a splitting unit, configured to split the computing operation into a plurality of sub-computing operations; a configuration unit configured to reconfigure the plurality of sub-computing operations into corresponding target computing units during the process of loading the target model into the hardware computing system; the hardware computing system comprises at least a plurality of computing units, each computing unit being composed of hardware chip resources called by assembly instructions; An execution unit is configured to execute the corresponding sub-computation operations through the target operation unit to obtain multiple sub-computation results; The application unit is configured to apply the multiple sub-calculation results to the data processing flow of the target model.

[0131] In an optional embodiment, when the identification unit identifies the computing operation that meets the splitting condition in the target model, it is configured to: Acquire the calculation type to which each calculation operation in the target model belongs, the data interaction relationship between the calculation operations, and / or the model structure of the target model; Based on the calculation type and / or the data interaction relationship, the calculation operations in the target model that meet the splitting condition are determined; the splitting condition is pre-set based on the model type to which the target model belongs and the model structure.

[0132] In an optional embodiment, when the identification unit determines the calculation operation in the target model that meets the splitting condition based on the calculation type and / or the data interaction relationship, it is configured to: If the target model is a large floodlight model, determining a calculation operation whose calculation type in the target model is matrix multiplication as a calculation operation that meets the splitting condition; and / or, Determine that the data interaction structure in the target model is a calculation operation obtained by concatenating matrix multiplication and activation function as a calculation operation that meets the splitting condition.

[0133] In an optional embodiment, the splitting unit, when splitting the computing operation to obtain a plurality of sub-computing operations, is configured to: Determine a candidate splitting method for the computing operation; the candidate splitting method is determined based on a computing type of the computing operation; Based on the data interaction relationship between the computing operation and other computing operations in the target model, obtaining the computing efficiency corresponding to the candidate splitting method; The computing operation is split using a candidate splitting method with the highest computational efficiency to obtain a plurality of sub-computing operations.

[0134] In an optional embodiment, the splitting unit, when acquiring the computing efficiency corresponding to the candidate splitting mode based on the data interaction relationship between the computing operation and other computing operations in the target model, is configured to: Acquire the number of other computing operations associated with the computing operation in the target model; wherein the other computing operations associated with the computing operation are used to indicate computing operations applied to part or all of the computing results in the computing operation during the computing process; The computing efficiency corresponding to the candidate splitting method is determined by the number of other computing operations associated with the computing operation; wherein, the smaller the number of other computing operations associated with the computing operation, the higher the computing efficiency corresponding to the candidate splitting method.

[0135] In an optional embodiment, the splitting unit, when acquiring the computing efficiency corresponding to the candidate splitting mode based on the data interaction relationship between the computing operation and other computing operations in the target model, is configured to: Obtaining the number of data aggregations to be performed after the calculation operation in the target model; The computing efficiency corresponding to the candidate splitting method is determined based on the number of data aggregation times to be performed after the computing operation; wherein, the smaller the number of data aggregation times to be performed after the computing operation, the higher the computing efficiency corresponding to the candidate splitting method.

[0136] In an optional embodiment, the splitting unit, when splitting the computing operation by using the candidate splitting method with the highest computational efficiency to obtain a plurality of sub-computing operations, is configured to: If the target model is a large floodlight model, the weight matrix is ​​vertically split into a plurality of first weight matrices; Splitting the input matrix vertically into a plurality of first sub-input matrices; The multiplication calculations between the plurality of first weight matrices and the plurality of first sub-input matrices are used as the corresponding plurality of first sub-calculation operations.

[0137] In an optional embodiment, the application unit, when applying the multiple sub-calculation results to the data processing flow of the target model, is configured to: A plurality of sub-calculation results obtained by the plurality of first sub-calculation operations are added together to obtain a final calculation result corresponding to the calculation operation.

[0138] In an optional embodiment, the splitting unit, when splitting the computing operation by using the candidate splitting method with the highest computational efficiency to obtain a plurality of sub-computing operations, is configured to: If the target model is a large floodlight model, the weight matrix is ​​split horizontally into a plurality of second weight matrices; The multiplication calculation between each of the plurality of second weight matrices and the input matrix is ​​performed as the corresponding plurality of second sub-calculation operations.

[0139] In an optional embodiment, the application unit, when applying the multiple sub-calculation results to the data processing flow of the target model, is configured to: The multiple sub-computation results obtained by the multiple second sub-computation operations are concatenated to obtain a final calculation result corresponding to the calculation operation.

[0140] In an optional embodiment, the splitting unit, when splitting the computing operation by using the candidate splitting method with the highest computational efficiency to obtain a plurality of sub-computing operations, is configured to: If the target model is a large floodlight model, the input matrix is ​​split, and the GeLU operations associated with the split input matrix are used as corresponding sub-computation operations.

[0141] In an optional embodiment, the splitting unit, after acquiring the computing efficiency corresponding to the candidate splitting mode based on the data interaction relationship between the computing operation and other computing operations in the target model, is further configured to: Obtaining the storage space occupancy rate corresponding to the calculation operation under each candidate splitting mode; Based on the ranking relationship of the storage space occupancy rate, the numerical value of the computational efficiency corresponding to each candidate splitting method is re-determined.

[0142] In an optional embodiment, an optimization unit is further included, which is further configured to: before the configuration unit reconfigures the multiple sub-computing operations to the corresponding target computing units during the process of loading the target model into the hardware computing system, further obtain a first data transmission efficiency between the candidate computing units to be configured for the multiple sub-computing operations, and a second data transmission efficiency between the candidate computing units and the surrounding computing units; The candidate computing units corresponding to the optimal solutions of the first data transmission efficiency and the second data transmission efficiency are used as target computing units corresponding to the multiple sub-computing operations.

[0143] In an optional embodiment, an optimization unit is further included, which is further configured to: in the process of loading the target model into the hardware computing system, before the configuration unit reconfigures the multiple sub-computing operations to the corresponding target computing units, further based on the data interaction relationship between the computing operation and other computing operations in the target model, obtain the distances between the candidate computing units to be configured for the multiple sub-computing operations; The candidate computing units whose distances are within the preset range are used as target computing units corresponding to the multiple sub-computing operations.

[0144] It should be noted that, regarding the model optimization device provided by the implementation of this application, the specific functions and implementation details of each module therein have the same implementation principles as the implementation process of the corresponding steps in the aforementioned method embodiment. For details, please refer to the description of the corresponding parts in the aforementioned method embodiment, which will not be repeated here.

[0145] Based on the same implementation principle, the embodiment of the present application also provides an electronic device, Figure 6 is a structural block diagram of the electronic device 400, such as Figure 6 As shown, the electronic device 400 includes: a processor 401, a memory 402, a communication interface 403, a communication bus 404 and a controller 405; wherein the processor 401, the memory 402, and the communication interface 403 communicate with each other via the communication bus 404; the memory 402 is used to store computer programs; the processor 401 is used to execute the programs stored in the memory 402 to implement corresponding processing functions; the communication interface 404 is used for communication between the electronic device 400 and other devices; and the controller 405 is used to implement the model optimization method described in the above method embodiment.

[0146] In the embodiment of the present application, the communication bus 404 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 404 may be divided into an address bus, a data bus, a control bus, etc., and the specific form is not limited. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] The memory 402 may include a random access memory (RAM) or a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far away from the processor 401 .

[0148] Processor 401 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other editable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the specific form can be determined according to actual needs.

[0149] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0150] It should be noted that although one or more embodiments of the present application provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0151] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, devices (systems) or computer program products. Therefore, the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of one or more embodiments of the present application, rather than to limit them. Although one or more embodiments of the present application have been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of one or more embodiments of the present application, and they should all be included in the scope of the claims and specifications of one or more embodiments of the present application.

[0153] One or more embodiments of the present application are described above in conjunction with optional implementation methods, but these implementation methods are only exemplary and serve only as an illustration. On this basis, multiple replacements and improvements can be made to one or more embodiments of the present application, all of which fall within the scope of protection of one or more embodiments of the present application.

Claims

1. A model optimization method, characterized in that: Applied to a hardware computing system deployed with a target model, the target model being a neural network model; the method comprises: Identifying computing operations in the target model that meet the splitting conditions; Splitting the computing operation to obtain multiple sub-computing operations; In the process of loading the target model into the hardware computing system, multiple sub-computing operations are reconfigured to corresponding target computing units; the hardware computing system includes at least multiple computing units, each of which is composed of hardware chip resources called based on assembly instructions; Execute the corresponding sub-computation operations through the target computing unit to obtain multiple sub-computation results; The multiple sub-calculation results are applied to the data processing flow of the target model.

2. The method according to claim 1, characterized in that The identifying the computing operations in the target model that meet the splitting conditions includes: Acquire the calculation type to which each calculation operation in the target model belongs, the data interaction relationship between the calculation operations, and / or the model structure of the target model; Based on the calculation type and / or the data interaction relationship, the calculation operations in the target model that meet the splitting condition are determined; the splitting condition is pre-set based on the model type to which the target model belongs and the model structure.

3. The method according to claim 2, characterized in that The determining, based on the calculation type and / or the data interaction relationship, the calculation operation in the target model that meets the splitting condition includes: If the target model is a large floodlight model, determining a calculation operation whose calculation type in the target model is matrix multiplication as a calculation operation that meets the splitting condition; and / or, Determine that the data interaction structure in the target model is a calculation operation obtained by concatenating matrix multiplication and activation function as a calculation operation that meets the splitting condition.

4. The method according to claim 1, characterized in that The splitting of the computing operation to obtain multiple sub-computing operations includes: Determine a candidate splitting method for the computing operation; the candidate splitting method is determined based on a computing type of the computing operation; Based on the data interaction relationship between the computing operation and other computing operations in the target model, obtaining the computing efficiency corresponding to the candidate splitting method; The computing operation is split using a candidate splitting method with the highest computational efficiency to obtain a plurality of sub-computing operations.

5. The method according to claim 4, characterized in that The obtaining, based on the data interaction relationship between the computing operation and other computing operations in the target model, the computing efficiency corresponding to the candidate splitting mode includes: Acquire the number of other computing operations associated with the computing operation in the target model; wherein the other computing operations associated with the computing operation are used to indicate computing operations applied to part or all of the computing results in the computing operation during the computing process; The computing efficiency corresponding to the candidate splitting method is determined by the number of other computing operations associated with the computing operation; wherein, the smaller the number of other computing operations associated with the computing operation, the higher the computing efficiency corresponding to the candidate splitting method.

6. The method according to claim 4, characterized in that The obtaining, based on the data interaction relationship between the computing operation and other computing operations in the target model, the computing efficiency corresponding to the candidate splitting mode includes: Obtaining the number of data aggregations to be performed after the calculation operation in the target model; The computing efficiency corresponding to the candidate splitting method is determined based on the number of data aggregation times to be performed after the computing operation; wherein, the smaller the number of data aggregation times to be performed after the computing operation, the higher the computing efficiency corresponding to the candidate splitting method.

7. The method according to claim 4, characterized in that The candidate splitting method with the highest computational efficiency is adopted to split the computing operation to obtain multiple sub-computing operations, including: If the target model is a large floodlight model, the weight matrix is ​​vertically split into a plurality of first weight matrices; Splitting the input matrix vertically into a plurality of first sub-input matrices; The multiplication calculations between the plurality of first weight matrices and the plurality of first sub-input matrices are used as the corresponding plurality of first sub-calculation operations.

8. The method according to claim 7, characterized in that The step of applying the plurality of sub-calculation results to the data processing flow of the target model includes: A plurality of sub-calculation results obtained by the plurality of first sub-calculation operations are added together to obtain a final calculation result corresponding to the calculation operation.

9. The method according to claim 4, characterized in that The candidate splitting method with the highest computational efficiency is adopted to split the computing operation to obtain multiple sub-computing operations, including: If the target model is a large floodlight model, the weight matrix is ​​split horizontally into a plurality of second weight matrices; The multiplication calculation between each of the plurality of second weight matrices and the input matrix is ​​performed as the corresponding plurality of second sub-calculation operations.

10. The method according to claim 9, characterized in that The step of applying the plurality of sub-calculation results to the data processing flow of the target model includes: The multiple sub-computation results obtained by the multiple second sub-computation operations are concatenated to obtain a final calculation result corresponding to the calculation operation.

11. The method according to claim 4, characterized in that The candidate splitting method with the highest computational efficiency is adopted to split the computing operation to obtain multiple sub-computing operations, including: If the target model is a large floodlight model, the input matrix is ​​split, and the GeLU operations associated with the split input matrix are used as corresponding sub-computation operations.

12. The method according to claim 4, characterized in that After obtaining the computational efficiency corresponding to the candidate splitting method based on the data interaction relationship between the computational operation and other computational operations in the target model, the method further includes: Obtaining the storage space occupancy rate corresponding to the calculation operation under each candidate splitting mode; Based on the ranking relationship of the storage space occupancy rate, the numerical value of the computational efficiency corresponding to each candidate splitting method is re-determined.

13. The method according to claim 1, characterized in that In the process of loading the target model into the hardware computing system, before reconfiguring the plurality of sub-computing operations to the corresponding target computing units, the method further includes: Acquire a first data transmission efficiency between candidate computing units to be configured for each of the plurality of sub-computing operations, and a second data transmission efficiency between the candidate computing units and surrounding computing units; The candidate computing units corresponding to the optimal solutions of the first data transmission efficiency and the second data transmission efficiency are used as target computing units corresponding to the multiple sub-computing operations.

14. The method according to claim 1, characterized in that In the process of loading the target model into the hardware computing system, before reconfiguring the plurality of sub-computing operations to the corresponding target computing units, the method further includes: Based on the data interaction relationship between the computing operation and other computing operations in the target model, obtaining distances between candidate computing units to be configured for each of the plurality of sub-computing operations; The candidate computing units whose distances are within the preset range are used as target computing units corresponding to the multiple sub-computing operations.

15. A model optimization device, characterized in that: The device is applied to a hardware computing system deployed with a target model, wherein the target model is a neural network model; the device comprises at least the following units: an identification unit, configured to identify computing operations in the target model that meet the splitting condition; a splitting unit, configured to split the computing operation into a plurality of sub-computing operations; a configuration unit configured to reconfigure the plurality of sub-computing operations into corresponding target computing units during the process of loading the target model into the hardware computing system; The hardware computing system comprises at least a plurality of computing units, each of which is composed of hardware chip resources called by assembly instructions; An execution unit is configured to execute the corresponding sub-computation operations through the target operation unit to obtain multiple sub-computation results; The application unit is configured to apply the multiple sub-calculation results to the data processing flow of the target model.

16. A chip, characterized in that: The chip includes a processor coupled to a transceiver, and is used to execute the model optimization method according to any one of claims 1 to 14.

17. An electronic device, characterized in that: The system comprises a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the model optimization method according to any one of claims 1 to 14.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed, implement the model optimization method described in any one of claims 1 to 14.

Images