Neural network model optimization method and device, equipment, storage medium and program product

Abstract

The invention provides a neural network model optimization method and device, equipment, a storage medium and a program product. The method comprises the following steps: executing first pruning processing on a first neural network model based on first data of a plurality of data fields and first weights of the plurality of data fields to obtain a second neural network model; based on the second data of the plurality of data fields, verifying the second neural network model to obtain a first loss value in each data field; based on the first loss value of each data field, updating the first weights of the plurality of data fields to obtain second weights of the plurality of data fields; and performing second pruning processing on the second neural network model based on the first data of the plurality of data fields and the second weights of the plurality of data fields to obtain a third neural network model. According to the method and the device, the model performance reduction caused by pruning processing can be improved while the model complexity is reduced.

Description

Background Art

[0002] In recent years, as the scale of neural network models has increased dramatically, the computing resources and energy consumption required in the inference process have also increased significantly. This problem is particularly prominent in practical applications, because many application scenarios have limited resources and it is difficult to support the high computing costs of large-scale models. Therefore, how to effectively reduce the computing cost of neural network models has become an important direction of current research. To this end, exploring the training of smaller-scale models has become one of the most important research goals. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, device, storage medium and program product for optimizing a neural network model, which can reduce the complexity of the model while improving the degradation of model performance caused by pruning.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a method for optimizing a neural network model, the method comprising:

[0006] Based on first data of multiple data fields and first weights of the multiple data fields, performing a first pruning process on the first neural network model to obtain a second neural network model;

[0007] Based on the second data in the plurality of data fields, verifying the second neural network model, and obtaining a first loss value in each of the data fields;

[0008] Based on the first loss value of each of the data fields, updating the first weights of the multiple data fields to obtain second weights of the multiple data fields;

[0009] Based on the first data of the multiple data fields and the second weights of the multiple data fields, a second pruning process is performed on the second neural network model to obtain a third neural network model.

[0010] In the above solution, training the fourth neural network model includes multiple iterative training of the fourth neural network model, and obtaining the fourth reference loss value of each data field includes:

[0011] Obtaining the number of model parameters of the fourth neural network model and the number of iterations of the multiple iterative trainings;

[0012] Based on the power-law relationship between the number of model parameters of the fourth neural network model, the number of iterations of the multiple iterative trainings and the fourth reference loss value, a fourth reference loss value for each of the data fields is predicted.

[0013] In the above solution, after determining the fifth weights of the plurality of data fields based on the fifth loss value and the pre-constructed constraint condition, the method further includes:

[0014] Performing weighted sum processing on the fifth weight and the fourth weight according to a preset second weighting parameter to obtain a sixth weight of the plurality of data fields;

[0015] The sixth weights of the multiple data fields are used as new fifth weights, and the third data based on the multiple data fields and the fifth weights of the multiple data fields are executed to train the fourth neural network model to obtain a fifth neural network model.

[0016] In the above scheme, after obtaining the fifth neural network model, the method further includes:

[0017] Acquire training data for a target task, wherein the training data for the target task includes data samples and labeled data;

[0018] Performing target task result prediction processing on the data sample through the fifth neural network model to obtain prediction data;

[0019] Determining a target task loss value based on the labeled data and the predicted data;

[0020] The fifth neural network model is updated based on the target task loss value to obtain a fifth neural network model for the target task.

[0021] The present application provides an optimization device for a neural network model, including:

[0022] A pruning module, configured to perform a first pruning process on the first neural network model based on first data of a plurality of data fields and first weights of the plurality of data fields to obtain a second neural network model;

[0023] An optimization module, configured to verify the second neural network model based on the second data in the plurality of data fields, and obtain a first loss value in each of the data fields;

[0024] The optimization module is further used to update the first weights of the multiple data fields based on the first loss value of each of the data fields to obtain the second weights of the multiple data fields;

[0025] The pruning module is further used to perform a second pruning process on the second neural network model based on the first data in the multiple data fields and the second weights of the multiple data fields to obtain a third neural network model.

[0026] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0027] A memory for storing computer executable instructions or computer programs;

[0028] The processor is used to implement the optimization method of the neural network model provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0029] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the optimization method of the neural network model provided in the embodiment of the present application when executed by a processor.

[0030] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the optimization method of the neural network model provided in the embodiment of the present application is implemented.

[0031] The embodiments of the present application have the following beneficial effects:

[0032] The second neural network model obtained by the first pruning process is verified based on the second data to obtain the first loss value of the second neural network model in each data field. The performance of the second neural network model in each data field represented by the first loss value is used to guide the resource allocation of multiple data fields in the second pruning process. Specifically, the first weights of multiple data fields are updated by the first loss value to obtain the second weights of multiple data fields. On this basis, the resources of multiple data fields are allocated by the second weights, and the second pruning process is performed on the second neural network model to obtain the third neural network model. By updating the first weight to the second weight, the data resources are considered and optimized in the pruning process. Allocation, by more accurately allocating data resources, improves the robustness of the pruned third neural network model to data distribution deviation and its ability to optimize high-potential data areas, thereby reducing the complexity of the model while improving the degradation of model performance caused by pruning. In addition, the pruned third neural network model contains fewer parameters, so the model occupies less storage space and is easier to deploy on devices with limited storage resources. At the same time, due to the reduced model complexity, the model's inference speed is significantly improved, which is suitable for real-time applications and easy to run on low-cost hardware, reducing the cost of model deployment and maintenance, thereby achieving the beneficial effect of taking into account both resource conservation of computer equipment and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural diagram of an optimized system architecture of a neural network model provided in an embodiment of the present application;

[0034] Figure 2 It is a schematic diagram of the structure of the server provided in the embodiment of the present application;

[0035] Figure 3A It is a first flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0036] Figure 3B It is a second flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0037] Figure 3C It is a third flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0038] Figure 3D It is a fourth flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0039] Figure 3E It is a fifth flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0040] Figure 3F It is a sixth flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0041] Figure 3G It is a seventh flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0042] Figure 3H It is an eighth flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0043] Fig. 3I It is a ninth flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0044] Figure 3J is a tenth flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0045] Figure 3K This is an eleventh flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0046] Figure 3L It is a twelfth flow chart of the optimization method of the neural network model provided in the embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of the application process of the optimization method of the neural network model provided in the embodiment of the present application;

[0048] Figure 5It is a schematic diagram of the principle of the optimization method of the neural network model provided in the embodiment of the present application;

[0049] Fig. 6A This is a first effect diagram of the optimization method of the neural network model provided in the embodiment of the present application;

[0050] Figure 6B This is a second effect diagram of the optimization method of the neural network model provided in an embodiment of the present application.

[0051] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0053] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0056] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0057] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0058] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0059] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0060] 2) Multiple data domains refer to different sources, types or characteristics of data. Taking text-based data as an example, multiple data domains may include books, Wikipedia, academic papers, etc.

[0061] 3) Pruning (or model pruning) is a technique used to reduce the size and complexity of a neural network model by removing unimportant parts of the model (such as neural network weights, neurons, or entire layers). This method can increase the model's inference speed and reduce memory consumption while maintaining or even improving the model's predictive performance as much as possible.

[0062] 4) Large Language Model (LLM) is a large-scale language model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models are characterized by their large scale and billions of parameters that help them learn complex patterns in language data. They are usually based on deep learning architectures. Large language models refer to deep learning models trained with large amounts of text data, containing billions or even larger parameters. They can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a huge language knowledge base, thereby simulating human language understanding and generation capabilities. Large language models have the following characteristics:

[0063] Learning ability: Through training on massive amounts of text data, large language models can learn a wealth of language knowledge and expressions, including grammar, semantics, and common expression habits.

[0064] Pattern recognition: Large language models can recognize common text patterns and semantic associations, such as the co-occurrence relationship between words, the logical structure and semantic roles of sentences, etc.

[0065] Contextual understanding: Large language models can capture contextual information in text, understand the impact of previous text on subsequent text, and generate appropriate responses based on the context.

[0066] Generation capability: The large language model can generate relevant natural language text based on the input information, including answering questions, generating articles, and conducting conversations.

[0067] Resolving ambiguity: Despite the polysemy and ambiguity of language, large language models resolve ambiguity through contextual information and language regularities, providing more accurate and appropriate text generation or understanding.

[0068] The application scenarios of large language models are very broad. They can be applied to intelligent customer service, intelligent question and answer, natural language generation, advertising recommendation, games and other fields. They can improve the efficiency and accuracy of human-computer interaction and enhance user experience.

[0069] 5) Transformer refers to a time series model based on the self-attention mechanism. The encoder part can effectively encode time series information. Its processing ability of time series information is much better than that of the long short-term memory network (LSTM), and its speed is fast. It is widely used in natural language processing, computer vision, machine translation, speech recognition and other fields.

[0070] In recent years, as the scale of neural network models has increased dramatically, the computing resources and energy consumption required in the inference process have also increased significantly. This problem is particularly prominent in practical applications, because many application scenarios have limited resources and it is difficult to support the high computing costs of large-scale models. Therefore, how to effectively reduce the computing cost of neural network models has become an important direction of current research. To this end, exploring the training of smaller-scale models has become one of the most important research goals.

[0071] The embodiments of the present application provide a method, apparatus, device, computer-readable storage medium, and computer program product for optimizing a neural network model, which can reduce the complexity of the model while improving the degradation of model performance caused by pruning. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers.

[0072] See also Figure 1 , Figure 1 is a schematic diagram of the structure of the optimized system architecture of the neural network model provided in the embodiment of the present application, Figure 1 The server 100, the terminal device 200 and the network 300 are involved. The terminal device 200 is connected to the server 100 via the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0073] In some embodiments, the embodiments of the present application can be implemented by a server and a terminal device in collaboration. For example, the terminal device 200 sends the first neural network model to the server 100, and the server 100 obtains a third neural network model through the neural network model optimization method provided in the embodiments of the present application, and sends the third neural network model to the terminal device 200.

[0074] For example, the server 100 reads data from the local memory or external storage device, loads the first neural network model into the memory, and after the server 100 prunes the first neural network model through the neural network model optimization method provided in the embodiment of the present application, the pruned model (corresponding to the third neural network model) can be exported to the local memory or external storage device, thereby saving storage space and accelerating the reasoning speed of the model through the pruned model. The terminal device 200 reads the third neural network model sent by the server 100 from the local memory or external storage device, and the terminal device 200 locally runs the third neural network model obtained after pruning, and accelerates the reasoning speed through the third neural network model, reducing the computing resource consumption generated by model reasoning.

[0075] In some other embodiments, the embodiments of the present application can be implemented by a server alone. The terminal device 200 sends a model optimization request to the server 100, and the optimization request carries the identifier of the first neural network model. The server 100 receives the model optimization request, parses the model optimization request, and optimizes the first neural network model according to the identifier of the first neural network model through the neural network model optimization method provided in the embodiment of the present application to obtain a third neural network model, and sends the prompt information that the optimization of the first neural network model is completed and the identifier of the third neural network model as the model optimization result to the terminal device 200.

[0076] Take the server for optimizing the neural network model as an example, see Figure 2 , Figure 2 is a schematic diagram of the structure of the server provided in the embodiment of the present application, Figure 2The server 100 shown includes: at least one processor 110, a memory 130 and at least one network interface 120. The various components in the server 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140 is not described in detail. Figure 2 Various buses are labeled as bus system 140 .

[0077] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0078] The memory 130 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 130 may optionally include one or more storage devices that are physically remote from the processor 110.

[0079] The memory 130 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 130 described in the embodiments of the present application is intended to include any suitable type of memory.

[0080] In some embodiments, memory 130 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0081] Operating system 131, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0082] A network communication module 132, used to reach other electronic devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0083] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The optimization device 133 of the neural network model stored in the memory 130 is shown, which can be software in the form of a program and a plug-in, including the following software modules: a pruning module 1331 and an optimization module 1332. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0084] In some embodiments, the terminal device or server can implement the optimization method of the neural network model provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0085] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the optimization method of the neural network model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0086] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution subject, to illustrate the optimization method of the neural network model provided in the embodiment of the present application. Figure 5 , Figure 5It is a schematic diagram of the principle of the optimization method of the neural network model provided in an embodiment of the present application, including a pruning stage and a training stage. For the pruning stage: based on the first data of multiple data fields and the first weights of the multiple data fields, a first pruning process is performed on the first neural network model to obtain a second neural network model (corresponding to step 101 below); based on the second data of the multiple data fields, the second neural network model is verified to obtain a first loss value in each data field (corresponding to step 102 below); based on the first loss value, the first weight is updated to obtain a second weight (corresponding to step 103 below); based on the first data of the multiple data fields and the second weights of the multiple data fields, a second pruning process is performed on the second neural network model to obtain a third neural network model (corresponding to step 104 below). For the training stage: based on the third data of multiple data fields and the fourth weights of multiple data fields, the third neural network model is trained to obtain a fourth neural network model (corresponding to step 401 below); based on the fourth data of multiple data fields, the fourth neural network model is verified to obtain a fourth loss value (corresponding to step 402 below); based on the fourth loss value, the fourth weight is updated to obtain a fifth weight (corresponding to step 403 below); based on the third data of multiple data fields and the fifth weights of multiple data fields, the fourth neural network model is trained to obtain a fifth neural network model (corresponding to step 404 below).

[0087] See also Figure 3A , Figure 3A is a first flow chart of the optimization method of the neural network model provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0088] In step 101, based on first data of multiple data fields and first weights of multiple data fields, a first pruning process is performed on a first neural network model to obtain a second neural network model.

[0089] For example, the first data in multiple data fields is specifically a collection of data in multiple data fields, so the first data can also be called a first data set.

[0090] Taking the first data in text mode as an example, the multiple data fields may include but are not limited to the following:

[0091] News area: can include various types of news reports, such as economy, sports, entertainment, etc.

[0092] Social Media Domain: Data from social media platforms, such as posts, comments, and conversations on social platforms.

[0093] Literature: including novels, poems, dramas, essays and other literary works.

[0094] Scientific field: such as research papers, academic articles, scientific reports, etc.

[0095] Medical field: including medical research reports, clinical guidelines, etc.

[0096] Business field: including business reports, financial analysis, market research, advertising copywriting, etc.

[0097] Education field: involving educational materials, teaching materials, course handouts, etc.

[0098] Technical field; including technical documents, product manuals, code comments, technical blogs, etc.

[0099] Taking the first data of the image modality as an example, the multiple data fields may include but are not limited to the following:

[0100] Natural scene field: including outdoor landscape images, urban landscape images, natural scenery images, etc.

[0101] Character image field: involves portrait photography images, fashion model images, etc.

[0102] Medical imaging field: such as X-rays and other medical images.

[0103] Astronomical image field: including galaxy images, star images and other celestial body images.

[0104] Traffic scene field: involving road images, vehicle images, traffic sign images, etc.

[0105] Product image area: such as product pictures on e-commerce websites, packaging design images of retail products, etc.

[0106] In some embodiments, the first pruning process includes multiple iterations, see Figure 3B , Figure 3A The step 101 shown can be implemented by executing the following steps 1011 to 1015 for each iterative process, which are described in detail below.

[0107] In step 1011, the network structure parameters in the first neural network model are masked based on the pruning mask to obtain a masked neural network model.

[0108] In some embodiments, the pruning mask includes multiple mask variables, each mask variable corresponds to a network structure parameter in the model, and the network structure parameters of each layer in the first neural network model are masked based on the multiple mask variables to obtain a masked neural network model.

[0109] For example, the first neural network model includes L Snetwork layers, each of which consists of a multi-head attention module (Multi-Head Attention, MHA) and a feed-forward neural network (Feed-Forward Neural Network, FFN). The first neural network model has d S hidden state dimensions, H S attention heads, with m in each FFN S Intermediate dimensions. Multiple mask variables can include layer pruning masks: Hidden dimension pruning mask: Attention head pruning mask: Intermediate dimension pruning mask: Each mask variable controls whether to prune or preserve the associated substructure, for example, if a network layer corresponds to z layer = 0, the layer will be pruned. Otherwise, if the z corresponding to a network layer layer =1, the layer will be retained, and the network structure parameters in the first neural network model are masked in the above manner to obtain a masked neural network model.

[0110] In step 1012, first data in a plurality of data fields are sampled based on a first weight to obtain sampled data.

[0111] In some embodiments, data in each data field in the first data (or training data set) is sampled (such as randomly sampled) according to the first weight to obtain sampled data. That is to say, the first weight is specifically the ratio of the amount of data sampled from each data field in the first data to the amount of data in the corresponding data field in the first data.

[0112] For example, assuming that the first data includes data from multiple data fields: A (data from data field 1), B (data from data field 2) and C (data from data field 3), the first weight is expressed as: data field 1 (0.3), data field 2 (0.3) and data field 3 (0.4), where data field 1 (0.3) means that 30% of the data from data field 1 in the first data is sampled as the sampled data of data field 1, and so on. The sampled data obtained after random sampling can be expressed as: 0.3A+0.3B+0.4C.

[0113] In step 1013, based on the masked neural network model and the sampled data, an inference loss value of the masked neural network model is determined.

[0114] In some embodiments, the sampled data includes labeled data, and the masked neural network model obtained after mask processing performs model inference processing based on the sampled data to obtain an inference result, and determines the inference loss value of the masked neural network model based on the inference result and the labeled data.

[0115] For example, assume that a masked neural network model (a language model) is used for a text classification task, and the sampling data includes text data samples and category annotation data of the text data. The text data samples are used as input of the masked neural network model, and the predicted category (inference result) is obtained through forward reasoning of the masked neural network model. The loss value is calculated (for example, cross entropy loss calculation) based on the predicted category and the category annotation data to obtain the inference loss value of the masked neural network model. The inference loss can be expressed as L(θ,z), where θ represents the network parameters of the masked neural network model (such as weight matrix parameters and bias matrix parameters), and z represents the pruning mask.

[0116] In step 1014, a pruning loss value is determined based on the difference between the network structure parameters of the mask neural network model and the network structure parameters of the target network model.

[0117] In some embodiments, the network structure parameters of the target network model are preset, which can be understood as the network structure parameters of the second neural network model expected to be obtained through pruning processing.

[0118] For example, the network structure parameters of the target network model may include the target number of attention heads Target number of layers Target number of hidden dimensions and the target intermediate dimension

[0119] For example, based on the difference between the network structure parameters of the mask neural network model and the network structure parameters of the target network model, the pruning loss value can be expressed by formula (1) and formula (2):

[0120]

[0121] Among them, λ, φ and z are learnable parameters. Formula (1) shows the constraints when pruning the attention head. The constraints of other substructures (layers, intermediate dimensions, hidden dimensions) can be found in Formula (1).

[0122] Following formula (1), the pruning loss can be expressed as formula (2):

[0123]

[0124] Among them, θ represents the network parameters of the mask neural network model (such as weight matrix parameters, bias matrix parameters), Indicates that L S The difference between the network structure parameters of the pruned masked neural network model and the target number of attention heads in the network layer, Indicates that L S The difference between the network structure parameters of the pruned masked neural network model and the target intermediate dimension number in the network layer, Represents the difference between the network structure parameters of the pruned masked neural network model and the target layer, Represents the difference between the network structure parameters of the pruned masked neural network model and the target hidden dimension number.

[0125] In step 1015, the pruning mask is updated based on the inference loss value and the pruning loss value, wherein the updated pruning mask is used for the next iterative processing, and the mask neural network model obtained by the last iterative processing is used as the second neural network model.

[0126] In some embodiments, the inference loss value and the pruning loss value are used as a combined loss value, gradient information is obtained through the combined loss value, and the pruning mask is updated according to the gradient information.

[0127] For example, the gradient information of the combined loss value for each parameter of the pruning mask is obtained through the back-propagation algorithm, and the parameters of the pruning mask are updated using the obtained gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.).

[0128] In other embodiments, the inference loss value and the pruning loss value are used as a combined loss value, gradient information is obtained through the combined loss value, and the network parameters of the pruning mask and the masked neural network model are updated according to the gradient information.

[0129] In other embodiments, other pruning methods may be used for the first pruning process, such as structured pruning methods such as attention head-based pruning and layer-based pruning, or unstructured pruning methods, which are not limited in the embodiments of the present application.

[0130] Continue to see Figure 3A In step 102, the second neural network model is verified based on the second data in multiple data fields to obtain a first loss value in each data field.

[0131] In some embodiments, the second neural network model is verified using second data (or referred to as a verification data set) to obtain a first loss value in each data field.

[0132] For example, the second data is different from the first data and is also composed of data from multiple data fields, but the data in the second data is different from the data in the first data, that is, there is no intersection between the two data, and the first data and the second data are independent verification data sets.

[0133] Here, the second neural network model can be verified in the following way: an independent verification data set (such as the second data) is used to evaluate the performance of the second neural network model, that is, to obtain the first loss value of the second neural network model in each data field.

[0134] For example, the second data includes data in multiple data fields and corresponding labeled data, and the following processing is performed on the data in each data field in the second data: the data in the data field is used as the input of the second neural network model, and the prediction result is obtained through forward reasoning of the second neural network model, and the difference between the prediction result and the labeled data is calculated according to a preset loss function (such as a cross entropy loss function) to obtain the first loss value of the second neural network model in the data field.

[0135] For example, the second neural network model is verified using the second data, and the first loss value on each data subset (data field) is calculated to evaluate the performance of the model in different fields. Assuming that the second neural network is used for text classification, the second data contains multiple data fields (such as documents, books, Wikipedia, etc.). The second neural network model is verified by the second data, and the first loss value on each data field can be calculated to evaluate the text classification effect of the second neural network model in each data field.

[0136] In step 103, based on the first loss value in each data field, the first weights of the plurality of data fields are updated to obtain the second weights of the plurality of data fields.

[0137] In some embodiments, see Figure 3C , Figure 3A The step 103 shown can be implemented by following the steps 1031 to 1033, which are described in detail below.

[0138] In step 1031, a first reference loss value of each data field is obtained.

[0139] In some embodiments, a first reference loss value for each data field is obtained based on a scaling law, where the scaling law refers to a relationship between the performance of a model (such as a loss value or accuracy) and the size of the model (such as the number of parameters) in certain machine learning tasks. In deep learning, the scaling law generally indicates that as the model size increases, the performance of the model will improve, but this improvement is not linear, but follows a sublinear growth pattern.

[0140] For example, obtaining the first reference loss value for each data field based on the scale law can be achieved in the following way: First, select a benchmark model, such as a smaller model as a benchmark, and train and evaluate its performance on each data field, and record the loss value of the benchmark model on each data field. Next, expand the scale of the model according to a certain rule (such as exponential growth), increase the number of parameters, and retrain and evaluate the performance of the expanded model on each data field, and record the loss value. Analyze the changing trend of the loss value as the model scale increases, and apply the scale law based on the learned changing trend to predict the possible loss value (corresponding to the first reference loss value) of the larger model (corresponding to the second neural network model).

[0141] For example, assume that the loss values ​​of the baseline model on two data domains (domain A and domain B) are as follows: Domain A: baseline loss value = 1.0; Domain B: baseline model loss value = 1.5. According to the scale law, the number of model parameters is doubled to obtain a new model. The loss values ​​of the new model on the two data domains are as follows (the loss value may decrease, but the proportion of decrease will not be as large as the proportion of increase in the number of parameters): Domain A: new model loss value = 0.7 (assuming a decrease of 30%); Domain B: new model loss value = 1.1 (assuming a decrease of 20%) ), double the number of model parameters again, and record the loss values: Field A: larger model loss value = 0.5 (assuming it has dropped by about 28.6%); Field B: larger model loss value = 0.9 (assuming it has dropped by about 18.2%). Now, based on this trend, predict the first reference loss value corresponding to the second neural network model. For example, Field A: predicted loss value = 0.35 (assuming it has dropped by 30%); Field B: predicted loss value = 0.73 (assuming it has dropped by 20%). These predicted loss values ​​are the first reference loss values ​​for each data field.

[0142] In some embodiments, the first pruning process includes multiple iterations, see Figure 3D , Figure 3C The step 1031 shown can be implemented by following the steps 10311 to 10312, which are described in detail below.

[0143] In step 10311, the number of model parameters of the second neural network model and the number of iterations of the multiple iterations are obtained.

[0144] In some embodiments, the functions provided by a deep learning framework (such as PyTorch) can be used to count the number of model parameters (Parameters) in the model and the number of iterations (or iteration steps) of multiple iterations.

[0145] In step 10312, based on the power law relationship between the number of model parameters of the second neural network model, the number of iterations of multiple iterative processing and the first reference loss value, the first reference loss value of each data field is predicted.

[0146] In some embodiments, based on the power law relationship between the number of model parameters of the second neural network model, the number of iterations of multiple iterative processes and the first reference loss value, the first reference loss value of each data field is predicted, which can be expressed by formula (3):

[0147]

[0148] in, Represents the first reference loss value, P represents the number of model parameters of the second neural network model, T represents the number of iterations of multiple iterative processing, A (loss balance term), E (predicted final convergence loss value of training), α and β (exponential terms) are trainable parameters. By fitting the loss trajectory of each data field, the first reference loss value of each data field can be predicted during the training process (corresponding to the process of updating the pruning mask).

[0149] In some embodiments, the median (or percentile, such as 75% percentile, etc.) of the first reference loss values ​​of multiple data fields can be used as the new first reference loss value of each data field.

[0150] For example, the method of obtaining the first reference loss value for each data field based on the scale law and the method of obtaining the first reference loss value through formula (3) can be performed successively in multiple iterations. For example, in the first 20% of the iterations, the method of obtaining the first reference loss value for each data field based on the scale law is adopted, and then the first reference loss value is obtained through the method of formula (3). That is, different methods of obtaining the first reference loss value can be adopted at different stages, and the embodiments of the present application are not limited to this.

[0151] Continue to see Figure 3C In step 1032, for each data field, a first reference loss value corresponding to the data field is subtracted from the first loss value of the data field to obtain a second loss value of the data field.

[0152] In some embodiments, the difference between the first loss value of the data field and the first reference loss value is used as the second loss value of the data field.

[0153] In step 1033, second weights of the plurality of data fields are determined based on the second loss value and the pre-constructed constraints.

[0154] In some embodiments, the optimization objective of distributionally robust optimization (DRO) can be used as a constraint condition to obtain second weights of multiple data fields based on the second loss value and the DRO method. DRO aims to find a second neural network model that performs well on all possible data distribution sets (corresponding to the distribution set of the second data, that is, the domain weights of multiple data fields in the second data), and obtains the second weights (that is, domain weights) of multiple data fields by optimizing the worst-case loss or searching for the optimal solution within the data distribution set under the f-divergence constraint.

[0155] For example, DRO will optimize the possible upper bound loss under a series of possible data distributions Q (i.e., all possible second weights) to minimize the upper bound. Therefore, it is transformed into the minimum-maximum problem of formula (4):

[0156]

[0157] Among them, l(x, y; θ) represents the first loss value when the data x (second data) is input and the predicted value y is output when the network parameter θ of the second neural network is given.

[0158] In order to prevent overly difficult data fields from always occupying the optimization attention, resulting in the reappearance of imbalance between fields, the first loss value l(x, y; θ) of each data field needs to be subtracted from the corresponding first reference loss value, and then refer to the constraint conditions of formula (4) to search for the optimal second weight to prevent overly difficult data fields from always dominating Q.

[0159] Furthermore, it is necessary to restrict The value range of Q can be expressed by formula (5) to prevent it from deviating too much from the reference data ratio (i.e., the first weight) and ensure the balance between data fields.

[0160]

[0161] Among them, q represents A possible data distribution in train represents the data distribution represented by the first weight, D f Represents F divergence, which is used to measure q and ptrain The difference between them, ρ represents the preset threshold.

[0162] In some embodiments, see Figure 3E , Figure 3C The step 1033 shown can be implemented by following the steps 10331 to 10332, which are described in detail below.

[0163] In step 10331, the second loss value of each data field is standardized to obtain the first standardized weights of multiple data fields.

[0164] In some embodiments, the second loss value of each data field is standardized by a normalization method (such as a Softmax function) to obtain first standardized weights of multiple data fields.

[0165] In step 10332, in response to the difference value between the first standardized weight and the first weight being less than or equal to the first difference value threshold, the first standardized weights of the multiple data fields are used as second weights of the multiple data fields.

[0166] In some embodiments, the relationship between the first normalized weight, the difference value between the first weights and the first difference value threshold can be expressed by the above formula (5), and the first difference value threshold corresponds to ρ in formula (5).

[0167] In some embodiments, the second weight of each data field is greater than or equal to a first preset multiple of the first weight of each data field, and less than or equal to a second preset multiple of the first weight of each data field, and the first preset multiple is less than the second preset multiple.

[0168] For example, the first preset multiple may be 1 / n (eg, n=2), and the second preset multiple may be n.

[0169] In some embodiments, see Figure 3F , Figure 3A The illustrated step 103 can be implemented by following the steps 201 to 204 , which are described in detail below.

[0170] In step 201, a data field of a first priority and a data field of a second priority are determined from a plurality of data fields, wherein the first priority is higher than the second priority.

[0171] For example, the multiple data fields include data field A, data field B and data field C, wherein data field A is a data field of first priority, data field B and data field C are data fields of second priority, and the first priority being higher than the second priority can be understood as data field A being a more important data field.

[0172] In step 202, a first reference loss value of each second priority data field is obtained.

[0173] In some embodiments, a first reference loss value is obtained for each second priority data field. The specific acquisition method can be found in the description of step 1031 above and will not be repeated here.

[0174] In step 203, for each second priority level, the first reference loss value of the data field of the second priority level is subtracted from the first loss value of the data field of the second priority level to obtain a second loss value of the data field of the second priority level.

[0175] In some embodiments, the difference between the first loss value of the data field of the second priority and the first reference loss value is used as the second loss value of the data field of the second priority.

[0176] In step 204, based on the second loss value and the pre-constructed constraint condition, the second weight of the data field of the second priority is determined, and the first weight of the data field of the first priority is used as the second weight of the data field of the first priority.

[0177] In some embodiments, the first weight of the data field of the first priority is kept unchanged (as the second weight), and the specific implementation method of obtaining the second weight of the data field of the second priority can be found in the description of step 1033 above, which will not be repeated here.

[0178] In some embodiments, see Figure 3G , Figure 3A The step 103 shown can be implemented by following the steps 301 to 304, which are described in detail below.

[0179] In step 301, a data field of a first priority and a data field of a second priority are determined from a plurality of data fields, wherein the first priority is higher than the second priority.

[0180] For example, the multiple data fields include data field A, data field B and data field C, wherein data field A is a data field of first priority, data field B and data field C are data fields of second priority, and the first priority being higher than the second priority can be understood as data field A being a more important data field.

[0181] In step 302, a second reference loss value of each first-priority data field and a third reference loss value of each second-priority data field are obtained.

[0182] In some embodiments, based on empirical values, a second reference loss value is preset for each first-priority data field (i.e., the second reference loss value for each first-priority data field is fixed by presetting), and the implementation method for obtaining the third reference loss value for each second-priority data field can be found in the description of obtaining the first reference loss value in step 1031 above, which will not be repeated here.

[0183] In step 303, for each first priority level, the first loss value of the data field of the first priority level is subtracted from the second reference loss value of the first priority level to obtain the second loss value of the data field of the first priority level, and for each second priority level, the first loss value of the data field of the second priority level is subtracted from the third reference loss value of the second priority level to obtain the third loss value of the data field of the second priority level.

[0184] In some embodiments, the difference between the first loss value of the data field of the first priority and the second reference loss value is used as the second loss value of the data field of the first priority. The difference between the first loss value of the data field of the second priority and the third reference loss value is used as the second loss value of the data field of the second priority.

[0185] In step 304, second weights of the plurality of data fields are determined based on the second loss value, the third loss value, and the pre-constructed constraint conditions.

[0186] Here, the specific implementation method can refer to the description of step 1033 above, which will not be repeated here.

[0187] In some embodiments, after determining the second weights of multiple data fields based on the second loss value and pre-constructed constraints, the second weights and the first weights can be weighted and summed according to a preset first weighting parameter to obtain third weights of the multiple data fields; the third weights of the multiple data fields are used as new second weights, and the second pruning process is performed on the second neural network model based on the first data of the multiple data fields and the second weights of the multiple data fields to obtain a third neural network model.

[0188] For example, obtaining the third weights of multiple data fields can be expressed by formula (6):

[0189] p t+1 =δ·q t +(1-δ)·p t (6)

[0190] Among them, p t+1 represents the third weight, q t represents the second weight, p t represents the first weight, and δ represents a preset first weighting parameter.

[0191] For example, the third weights of multiple data fields can be applied in any iterative stage of the second pruning process, and the third weights of multiple data fields can be used as new second weights. The second pruning process is performed on the second neural network model based on the first data of multiple data fields and the second weights of multiple data fields to obtain a third neural network model. The embodiments of the present application are not limited to this.

[0192] Continue to see Figure 3A In step 104, based on the first data of the multiple data fields and the second weights of the multiple data fields, a second pruning process is performed on the second neural network model to obtain a third neural network model.

[0193] In some embodiments, the specific implementation of performing the second pruning process on the second neural network model can refer to the description of the first pruning process in step 101 above, which will not be repeated here.

[0194] In some embodiments, see Figure 3H After obtaining the third neural network model, the following steps 401 to 404 may be performed, which are described in detail below.

[0195] In step 401, based on third data in multiple data fields and fourth weights in multiple data fields, the third neural network model is trained to obtain a fourth neural network model.

[0196] In some embodiments, data in each data field in the third data is sampled (such as random sampling) according to the fourth weight, the obtained sampled data is used as a training data set, and the third neural network model is trained based on the training data set to obtain a fourth neural network model.

[0197] In step 402, based on fourth data in multiple data fields, the fourth neural network model is verified to obtain a fourth loss value in each data field.

[0198] In some embodiments, the fourth neural network model is verified using fourth data (or referred to as a verification data set) to obtain a fourth loss value in each data field.

[0199] For example, the fourth neural network model is verified using a verification data set (fourth data), and the fourth loss value in each data field is calculated to evaluate the performance of the model in different fields. It is assumed that the fourth neural network is used for text classification, and the fourth data contains multiple data fields (such as documents, books, Wikipedia, etc.). The fourth neural network model is verified by the verification data set, and the fourth loss value in each data field can be calculated to evaluate the text classification effect of the fourth neural network model in each data field.

[0200] In step 403, based on the fourth loss value in each data field, the fourth weights of the plurality of data fields are updated to obtain fifth weights of the plurality of data fields.

[0201] In some embodiments, see Fig. 3I , Figure 3H The step 403 shown can be implemented by following the steps 4031 to 4033, which are described in detail below.

[0202] In step 4031, a fourth reference loss value of each data field is obtained.

[0203] In some embodiments, a fourth reference loss value for each data field is obtained based on a scaling law, where the scaling law refers to the relationship between the performance of the model (such as loss value or accuracy) and the model size (such as the number of parameters) in certain machine learning tasks. In deep learning, the scaling law usually indicates that as the model size increases, the performance of the model will improve, but this improvement is not linear, but follows a sublinear growth pattern. The specific implementation method can be found in the description of step 1031 above, which will not be repeated here.

[0204] In some embodiments, training the fourth neural network model includes multiple iterative training of the fourth neural network model, see Figure 3J , Fig. 3I The step 4031 shown can be implemented by following the steps 40311 to 40312, which are described in detail below.

[0205] In step 40311, the number of model parameters of the fourth neural network model and the number of iterations of the multiple iterative training are obtained.

[0206] In some embodiments, the functions provided by a deep learning framework (such as PyTorch) can be used to count the number of model parameters (Parameters) of the fourth neural network model and the number of iterations (or iteration steps) of multiple iteration training.

[0207] In step 40312, based on the power law relationship between the number of model parameters of the fourth neural network model, the number of iterations of the multiple iterative training and the fourth reference loss value, the fourth reference loss value of each data field is predicted.

[0208] In some embodiments, based on the power law relationship between the number of model parameters of the fourth neural network model, the number of iterations of multiple iterative training and the fourth reference loss value, the fourth reference loss value for each data field is predicted. Please refer to the description of formula (3) above, which will not be repeated here.

[0209] For example, the method of obtaining the first reference loss value for each data field based on the scale law (see the description of step 1031 above) and obtaining the fourth reference loss value through formula (3) can be performed successively in multiple iterative trainings. For example, in the first 20% of the iterative training, the method of obtaining the fourth reference loss value for each data field based on the scale law is adopted, and then the fourth reference loss value is obtained through the method of formula (3). That is, different methods for obtaining the fourth reference loss value can be adopted at different stages, and the embodiments of the present application are not limited to this.

[0210] By dynamically obtaining the fourth reference loss value for each data field, the robustness to data distribution deviation is enhanced. At the same time, the allocation of training resources is optimized, so that the pruned model (the third neural network model) can focus on optimizing high-potential areas, ultimately achieving overall performance improvement and accelerated convergence speed.

[0211] In some embodiments, the median of the fourth reference loss values ​​corresponding to multiple data fields can be used as the new fourth reference loss value of each data field.

[0212] Continue to see Fig. 3I In step 4032, for each data field, the fourth loss value of the data field is subtracted from the fourth reference loss value of the data field to obtain a fifth loss value of the data field.

[0213] In some embodiments, the difference between the fourth loss value of the data field and the fourth reference loss value is used as the fifth loss value of the data field.

[0214] In step 4033, fifth weights of the plurality of data fields are determined based on the fifth loss value and the pre-constructed constraints.

[0215] In some embodiments, the optimization objective of distributionally robust optimization (DRO) can be used as a constraint condition to obtain the fifth weights of multiple data fields based on the fifth loss value and the DRO method. The specific implementation method can be found in the description of step 1033 above, which will not be repeated here.

[0216] In some embodiments, see Figure 3K , Fig. 3I The step 4033 shown can be implemented by following the steps 40331 to 40332, which are described in detail below.

[0217] In step 40331, the fifth loss value of each data field is standardized to obtain second standardized weights of multiple data fields.

[0218] In some embodiments, the fifth loss value of each data field is standardized by a normalization method (such as a Softmax function) to obtain second standardized weights of multiple data fields.

[0219] In step 40332, in response to the difference value between the second normalized weight and the fourth weight being less than or equal to the second difference value threshold, the second normalized weights of the multiple data fields are used as the fifth weights of the multiple data fields.

[0220] In some embodiments, the relationship between the difference value between the second normalized weight and the fourth weight and the second difference value threshold can be expressed by the above formula (5), and the second difference value threshold corresponds to ρ in formula (5).

[0221] For example, ρ can be adaptively adjusted according to the number of iterations of multiple iterative training to better meet the needs of the model at different stages. For example, in the first 50% of the iterative training, ρ is set to 1 , and the last 50% of the training iterations are set to ρ 2 , ρ can also be adaptively adjusted according to a preset ratio based on the loss value in the iterative training process. The embodiment of the present application does not limit the specific value method of ρ.

[0222] In some embodiments, the fifth weight of each data field is greater than or equal to the third preset multiple of the fourth weight of each data field and less than or equal to the fourth weight of each data field and the third preset multiple is less than the fourth preset multiple.

[0223] For example, the third preset multiple may be 1 / m (eg, m=2), and the fourth preset multiple may be m.

[0224] In some embodiments, after determining the fifth weights of multiple data fields based on the fifth loss value and pre-constructed constraints, the fifth weight and the fourth weight can be weighted and summed according to a preset second weighting parameter to obtain the sixth weights of the multiple data fields; the sixth weights of the multiple data fields are used as new fifth weights, and the third data based on the multiple data fields and the fifth weights of the multiple data fields are used to train the fourth neural network model to obtain a fifth neural network model.

[0225] For example, the fifth weight and the fourth weight are weighted and summed according to the preset second weighting parameter to obtain the sixth weight of multiple data fields. The implementation method can refer to the description of formula (6) above, which will not be repeated here. By dynamically changing the weights of multiple data fields during the iteration process, higher weights are continuously assigned to data fields with optimization potential, so that the model can give priority to training and optimizing data fields with optimization potential.

[0226] For example, the weight update method shown in formula (6) can be performed after 40% of the iterative training is completed to ensure that the reference loss has stabilized and can reasonably guide the dynamic adjustment of the weights of multiple data fields. This strategy enables the pruned model to prioritize the optimization of high-potential fields while maintaining the basic performance of other data fields, thereby effectively improving the overall model performance and accelerating convergence.

[0227] Continue to see Figure 3H In step 404, based on the third data of the multiple data fields and the fifth weights of the multiple data fields, the fourth neural network model is trained to obtain a fifth neural network model.

[0228] In other embodiments, as an alternative to steps 401 to 404, the third neural network model can be trained by other methods to recover the model performance degradation caused by pruning. For example, a knowledge distillation method is used to use the pruned model (the third neural network model) as a student model, and the unpruned model (the first neural network model) is used as a teacher model for knowledge transfer to enhance the performance of the pruned model.

[0229] In some embodiments, see Figure 3L After obtaining the fifth neural network model, the following steps 501 to 504 may be performed, which are described in detail below.

[0230] In step 501, training data of a target task is obtained, wherein the training data of the target task includes data samples and labeled data.

[0231] In some embodiments, taking the fifth neural network model as a language model as an example, the target tasks may include: text classification, such as assigning input text data to one or more predefined categories, such as sentiment analysis (positive, neutral, negative), spam detection, etc.; sentiment analysis, such as judging the emotional tendency of text expression, such as positive, negative or neutral; entity recognition, such as identifying specific types of entities in the text, such as names, places, organizations, dates, etc.; question answering, such as given a question and a text paragraph, find the answer in the paragraph or generate an answer; machine translation, such as translating text in one language into another language; text summarization, such as generating a concise summary of the text, retaining the main information; dialogue generation, such as creating natural language dialogues, which can be used for chatbots, virtual assistants, etc.; speech recognition, such as converting speech to text; text generation, such as generating text based on a given context or prompt, such as stories, poems, news reports, etc.

[0232] Taking text classification as the target task as an example, the training data of the target task may include text data samples and labeled data samples (such as literature, novels, poems, etc.).

[0233] In step 502, the target task result prediction processing is performed on the data sample through the fifth neural network model to obtain predicted data.

[0234] For example, suppose the fifth neural network model (a language model) is used for a text classification task, and the data sample is used as the input of the fifth neural network model. Through the forward reasoning of the fifth neural network model, the predicted data (inference result) is obtained.

[0235] In step 503, the target task loss value is determined based on the labeled data and the predicted data.

[0236] In some embodiments, the difference value between the labeled data and the predicted data is obtained through a preset loss function (such as a cross entropy loss function) as the target task loss value.

[0237] In step 504, the fifth neural network model is updated based on the target task loss value to obtain a fifth neural network model for the target task.

[0238] In some embodiments, gradient information is obtained through the target task loss value, and the fifth neural network model is updated according to the gradient information to obtain an updated fifth neural network model for the target task.

[0239] For example, the gradient information of the target task loss value for each parameter of the fifth neural network model is obtained through the back-propagation algorithm, and the parameters of the fifth neural network model are updated using the obtained gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the fifth neural network model converges, thereby obtaining the trained fifth neural network model for the target task.

[0240] Through steps 101 to 104, the weights of multiple data fields and the reference loss values ​​of multiple data fields are dynamically updated during the pruning stage, and lower reference losses and higher data resource ratios are allocated to data fields with greater improvement potential, so that the allocation of data resources is considered and optimized during the pruning process. By more accurately allocating data resources, the robustness of the pruned third neural network model to data distribution deviation and the ability to optimize high-potential data fields are improved, thereby achieving the beneficial effect of improving the degradation of model performance caused by pruning while reducing model complexity.

[0241] Through step 401 to step 401, it is achieved that lower reference loss and a higher proportion of data resources are allocated to data fields with greater improvement potential in the continued training stage (or performance recovery stage), so that the neural network model can focus on improving the performance of high-potential data fields while maintaining basic capabilities in other data fields, effectively improving the performance of the neural network model in key areas, and providing a more balanced overall data distribution by dynamically adjusting the weights of data fields, promoting balanced training of neural network models in different fields, and ultimately achieving the beneficial effect of comprehensively improving overall performance.

[0242] It should be noted that the optimization method of the neural network model provided in the embodiment of the present application can be applied to various types of neural network models. The following are some types of neural network models: Fully Connected Neural Network (FC NN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Transformer model, etc. Below, the exemplary application of the neural network model provided in the embodiment of the present application in the large language model optimization scenario will be described.

[0243] See also Figure 4 , Figure 4 is a schematic diagram of the application flow of the optimization method of the neural network model provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0244] In step 601, a first language model to be optimized is obtained.

[0245] In some embodiments, one of the existing language models is selected as a first language model, where the first language model is pre-trained and exhibits good performance on a specific task.

[0246] For example, suppose there is a pre-trained language model based on the Transformer architecture, which performs well on multiple natural language processing tasks. After evaluation, it is found that the language model has a slow inference speed when processing long texts, and the model parameters are large, which is not suitable for deployment on devices with limited resources. The optimization goal is to reduce model parameters and increase inference speed while maintaining model performance. This Transformer model is used as the first language model to be optimized, and subsequent pruning, training, and fine-tuning are performed.

[0247] In step 602, the first largest language model to be optimized is pruned by the neural network model optimization method provided in the embodiment of the present application to obtain a second largest language model.

[0248] In some embodiments, based on first training data of multiple data fields and first training data weights of multiple data fields, a first pruning process is performed on the first large language model to obtain a first intermediate large language model.

[0249] For example, the network structure parameters in the first large language model are masked based on the pruning mask to obtain a masked large language model (see the description of step 1011 above); the first training data (corresponding to the first data above) of multiple data fields are sampled based on the first training data weight (corresponding to the first weight above) to obtain sampled data (see the description of step 1012 above); based on the masked large language model and the sampled data, the inference loss value of the masked large language model is determined (see the description of step 1013 above); based on the difference between the network structure parameters of the masked large language model and the network structure parameters of the target large language model (corresponding to the target network model above), the pruning loss value is determined (see the description of step 1014 above); based on the inference loss value and the pruning loss value, the pruning mask is updated, wherein the updated pruning mask is used for the next iterative processing, and the masked large language model obtained by the last iterative processing is used as the first intermediate large language model (corresponding to the second neural network model above).

[0250] In some embodiments, based on first verification data of multiple data fields (corresponding to the second data above), the first intermediate large language model is verified to obtain a first verification loss value in each data field (corresponding to the first loss value above, see the description of step 102 above); based on the first verification loss value in each data field, the first training data weights of multiple data fields are updated to obtain second training data weights of multiple data fields (see the description of step 103 above); based on the first training data of multiple data fields and the second training data weights of multiple data fields, a second pruning process is performed on the first intermediate large language model to obtain a second large language model (see the description of step 104 above).

[0251] For example, the first pruning process includes multiple iterative processes, and based on the first verification loss value in each data field, the first training data weights of multiple data fields are updated, and the second training data weights of multiple data fields are obtained. This can be achieved in the following way: obtaining a reference loss value for each data field (see the description of step 1031 above); for each data field, subtracting the reference loss value corresponding to the data field from the first verification loss value of the data field to obtain a second verification loss value of the data field (see the description of step 1032 above); based on the second verification loss value and pre-constructed constraints, determining the second training data weights of multiple data fields.

[0252] For example, obtaining a reference loss value for each data field can be achieved in the following manner: obtaining the number of model parameters of the first intermediate large language model (corresponding to the second neural network model) and the number of iterations of multiple iterative processing (see the description of step 10311 above); based on the power law relationship between the number of model parameters of the first intermediate large language model, the number of iterations of multiple iterative processing and the reference loss value, predicting the reference loss value for each data field (see the description of step 10312 above).

[0253] For example, obtaining a reference loss value for each data field based on the scale law (see the description in step 1031) and obtaining the reference loss value by using the above formula (3) can be performed successively in multiple iterations. For example, in the first 20% of the iterations, the method of obtaining a reference loss value for each data field based on the scale law is adopted, and then the reference loss value is obtained by using the method of formula (3). That is, different methods of obtaining reference loss values ​​can be used at different stages, and the embodiments of the present application are not limited to this.

[0254] In step 603, the second largest language model is trained by the optimization method of the neural network model provided in the embodiment of the present application to obtain the third largest language model.

[0255] In some embodiments, based on the second training data of multiple data fields (corresponding to the third data above) and the third training data weight (corresponding to the fourth weight above), the second large language model is trained to obtain a second intermediate large language model (corresponding to the fourth neural network model above, see the description of step 401 above); based on the second verification data of multiple data fields (corresponding to the city data above), the second intermediate large language model is verified to obtain a second verification loss value in each data field (corresponding to the fourth loss value above, see the description of step 402 above); based on the second verification loss value in each data field, the third training data weights of multiple data fields are updated to obtain fourth training data weights for multiple data fields (corresponding to the fifth weight above, see the description of step 403 above); based on the second training data of multiple data fields and based on the fourth training data weights of multiple data fields, the second intermediate large language model is trained to obtain a third large language model (corresponding to the fifth neural network model above, see the description of step 404 above).

[0256] In step 604, instruction fine-tuning training is performed on the third largest language model to obtain a fourth largest language model.

[0257] In some embodiments, training data for a target task is acquired according to a task instruction, wherein the training data for the target task includes data samples and labeled data (see the description of step 501 above); the data samples are processed for target task result prediction using a third language model to obtain predicted data (see the description of step 502 above); a target task loss value is determined based on the labeled data and the predicted data (see the description of step 503 above); the third language model is updated based on the target task loss value to obtain a fourth language model for the target task (see the description of step 504 above).

[0258] Below, experimental data are used to illustrate the beneficial effects of the optimization method of the neural network model provided in the embodiments of the present application.

[0259] The pruned data used in the pruning process (corresponding to step 602) of the embodiment of the present application is 40 million input units (tokens), and the data in the continued training phase (corresponding to step 603) is 50 billion input units (tokens). The data comes from the training set of SlimPajama, and the first language model to be optimized uses Llama2. At the same time, tests and evaluations were conducted on 15 downstream tasks. The instruction fine-tuning uses 10,000 pairs of instruction-answer data from the ShareGPT dataset for training, and another 1,000 pairs of instruction-answer data are used for evaluation.

[0260] The pruning strategy of the related technique (Sheared-Llama) is used as a strong baseline for comparison in Table 1.

[0261] Table 1

[0262] method Model size before pruning Model size after pruning Perplexity Downstream task performance Related technologies 7B 1.3B 10.42 34.85 Methods of the present application 7B 1.3B 9.83 35.60 Methods of the present application 13B 1.3B 10.80 34.15 Related technologies 7B 2.7B 7.83 39.98 Methods of the present application 7B 2.7B 7.40 40.18 Methods of the present application 13B 2.7B 8.56 38.00

[0263] See also Fig. 6A , Fig. 6A This is the first effect diagram of the optimization method of the neural network model provided in the embodiment of the present application. Combined with Table 1, it can be seen that with only 3.2k training iterations (corresponding to Fig. 6A At the same time, with the increase of the pruning ratio (such as pruning from 7B (Bil lion) to 1.3B, and pruning from 13B to 1.3B), the optimization method of the neural network model provided in the embodiment of the present application does not lead to a significant increase in perplexity, especially in the later stage of training, it can show greater improvement potential.

[0264] In addition, the optimization method of the neural network model provided in the embodiment of the present application also achieved higher performance in downstream tasks.

[0265] Table 2

[0266]

[0267]

[0268] Among them, the downstream tasks include: Winograd Schema Challenge (WSC), Trivia Question Answering (TriQA), Boolean Question Answering (BoolQ), Natural Questions (NQ), TruthQA), Logical Abduction Benchmarks (LAMB), Science Question Answering (SciQ), Logical Inference Question Answering (LogiQa), Massive Multitask Language Understanding (MMLU), Adversarial Reading Comprehension from English Wikipedia (ARCE), Adversarial Question Answering from Children's Books (ARCC), Preposition and Pronoun Inference in Context (PIQA), Pronoun and Commonsense Knowledge (Winograd Schema with Commonsense Knowledge, WinoG), Reading Comprehension 3 (Stanford Question Answering Dataset, SQuAD) and natural language reasoning (Hel laSwag, HelS).

[0269] It can be seen from Table 2 that although the first largest language model (Llama2) experienced performance loss after pruning, the evaluation results on 15 downstream tasks show that the large language model (corresponding to the fourth largest language model in step 604) optimized by the optimization method of the neural network model provided in the embodiment of the present application is always better than the control group, that is, the optimization method of the neural network model provided in the embodiment of the present application has achieved significantly better performance than the related technology Sheared in 15 downstream tasks, and achieved better results in ten of the fifteen tasks, and significantly surpassed the open source model of similar scale (Pythia), verifying the superiority of the solution.

[0270] In addition, in terms of instruction fine-tuning, Figure 6BThe performance of the model after optimization by the optimization method of the neural network model provided by the embodiment of the present application and after instruction fine-tuning is shown. Compared with the Sheared strategy of the related technical method, it has a significant advantage in average winning rate, which shows that the optimization method of the neural network model provided by the embodiment of the present application has a good foundation in instruction fine-tuning and can significantly improve the generalization ability of the model.

[0271] In order to verify the universality of the optimization method of the neural network model provided in the embodiment of the present application, experiments were conducted in a multilingual setting to prove the effectiveness of the optimization method of the neural network model provided in the embodiment of the present application, and the results are shown in Table 3. Among them, Cont.PT means continuing training from Qwen2-1.5B, and Pruning means restoring the model capability by continuing training after pruning from Qwen2-7B. The experimental results show that the optimization method of the neural network model provided in the embodiment of the present application is significantly better than similar open source large models in all multilingual tasks, and surpasses the effect of Sheared, achieving an average downstream task performance improvement of +1.5. In addition, it can be seen from Table 3 that continuing pre-training from a well-trained small model is more effective than pruning from a large model (here, the small model has fewer model parameters than the large model), especially at a high pruning ratio, pruning from a large model may even have a counter-effect on performance. This result verifies the effectiveness of the optimization method of the neural network model provided in the embodiment of the present application, and provides new inspiration for future small model training.

[0272] Table 3

[0273]

[0274] Finally, the advantages of the optimization method of the neural network model provided in the embodiment of the present application in improving the model performance in the continued pre-training stage (corresponding to the training of the second largest language model in step 603 above) are analyzed. As shown in Table 4, the optimization method of the neural network model provided in the embodiment of the present application can allocate lower reference losses and higher training data ratios to fields with greater improvement potential, which enables the model to focus on improving the performance of high-potential data fields while maintaining the basic capabilities of other data fields, significantly reducing the perplexity (Perplexity, PPL) of these data fields and the average PPL of the data field. This targeted training strategy effectively improves the performance of the model in key data fields, and at the same time, by dynamically adjusting the proportion of reference data fields (corresponding to updating the third training data weights of multiple data fields above, obtaining the fourth training data weights of multiple data fields), it provides a more balanced overall data distribution, promotes the balanced training of the model in different data fields, and ultimately achieves a comprehensive improvement in overall performance.

[0275] Table 4

[0276]

[0277] Through steps 601 to 604, the pruning of the large language model and the efficient recovery of the performance of the large language model after pruning are achieved, and the robustness of the large language model to data distribution deviation is significantly enhanced. By dynamically adjusting the reference loss in the training process (such as predicting the reference loss value of each data field based on the power law relationship between the number of model parameters of the intermediate large language model, the number of iterations of multiple iterative processing and the reference loss value) and the data ratio (such as updating the third training data weights of multiple data fields based on the second validation loss value in each data field to obtain the fourth training data weights of multiple data fields), the model can allocate training resources more accurately, thereby accelerating model convergence and improving overall performance under limited computing resources.

[0278] The following is a description of an exemplary structure of the neural network model optimization device 133 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the optimization device 133 of the neural network model in the memory 130 may include:

[0279] The pruning module 1331 is used to perform a first pruning process on the first neural network model based on the first data of the multiple data fields and the first weights of the multiple data fields to obtain a second neural network model.

[0280] The optimization module 1332 is used to verify the second neural network model based on the second data in the multiple data fields to obtain a first loss value in each of the data fields.

[0281] In some embodiments, the optimization module 1332 is further configured to update the first weights of the plurality of data fields based on the first loss value in each of the data fields to obtain second weights of the plurality of data fields.

[0282] In some embodiments, the pruning module 1331 is further used to perform a second pruning process on the second neural network model based on the first data of the multiple data fields and the second weights of the multiple data fields to obtain a third neural network model.

[0283] In some embodiments, the optimization module 1332 is further used to obtain a first reference loss value for each of the data fields; for each of the data fields, subtract the first reference loss value corresponding to the data field from the first loss value of the data field to obtain a second loss value for the data field; and determine second weights of the multiple data fields based on the second loss value and pre-constructed constraints.

[0284] In some embodiments, the optimization module 1332 is further used to normalize the second loss value of each of the data fields to obtain first standardized weights of the multiple data fields; in response to the difference between the first standardized weight and the first weight being less than or equal to a first difference value threshold, the first standardized weights of the multiple data fields are used as second weights of the multiple data fields; wherein the second weight of each of the data fields is greater than or equal to a first preset multiple of the first weight of each of the data fields, and is less than or equal to a second preset multiple of the first weight of each of the data fields, and the first preset multiple is less than the second preset multiple.

[0285] In some embodiments, the first pruning process includes multiple iterative processes, and the optimization module 1332 is further used to obtain the number of model parameters of the second neural network model and the number of iterations of the multiple iterative processes; based on the power law relationship between the number of model parameters of the second neural network model, the number of iterations of the multiple iterative processes and the first reference loss value, the first reference loss value of each of the data fields is predicted.

[0286] In some embodiments, the optimization module 1332 is further used to perform weighted sum processing on the second weight and the first weight according to a preset first weighting parameter to obtain a third weight of the multiple data fields; use the third weights of the multiple data fields as new second weights, and execute the first data based on the multiple data fields and the second weights of the multiple data fields, perform a second pruning process on the second neural network model, and obtain a third neural network model.

[0287] In some embodiments, the optimization module 1332 is also used to determine a data field of a first priority and a data field of a second priority from the multiple data fields, wherein the first priority is higher than the second priority; obtain a first reference loss value for each data field of the second priority; for each second priority, subtract the first reference loss value of the data field of the second priority from the first loss value of the data field of the second priority to obtain a second loss value for the data field of the second priority; determine a second weight for the data field of the second priority based on the second loss value and pre-constructed constraints, and use the first weight of the data field of the first priority as the second weight of the data field of the first priority.

[0288] In some embodiments, the optimization module 1332 is also used to determine a data field of a first priority and a data field of a second priority from the multiple data fields, wherein the first priority is higher than the second priority; obtain a second reference loss value for each data field of the first priority, and a third reference loss value for each data field of the second priority; for each first priority, subtract the second reference loss value of the first priority from the first loss value of the data field of the first priority to obtain the second loss value of the data field of the first priority, and for each second priority, subtract the third reference loss value of the second priority from the first loss value of the data field of the second priority to obtain the third loss value of the data field of the second priority; determine the second weights of the multiple data fields based on the second loss values, the third loss values ​​and pre-constructed constraints.

[0289] In some embodiments, the first pruning process includes multiple iterative processes, and the pruning module 1331 is also used to perform the following processes for each of the iterative processes: masking the network structure parameters in the first neural network model based on the pruning mask to obtain a masked neural network model; sampling the first data of the multiple data fields based on the first weights to obtain sampled data; determining the inference loss value of the masked neural network model based on the masked neural network model and the sampled data; determining the pruning loss value based on the difference between the network structure parameters of the masked neural network model and the network structure parameters of the target network model; updating the pruning mask based on the inference loss value and the pruning loss value, wherein the updated pruning mask is used for the next iterative process, and the masked neural network model obtained by the last iterative process is used as the second neural network model.

[0290] In some embodiments, the optimization module 1332 is further used to train the third neural network model based on the third data of the multiple data fields and the fourth weights of the multiple data fields to obtain a fourth neural network model; verify the fourth neural network model based on the fourth data of the multiple data fields to obtain a fourth loss value in each of the data fields; update the fourth weights of the multiple data fields based on the fourth loss value in each of the data fields to obtain fifth weights of the multiple data fields; train the fourth neural network model based on the third data of the multiple data fields and the fifth weights of the multiple data fields to obtain a fifth neural network model.

[0291] In some embodiments, the optimization module 1332 is further used to obtain a fourth reference loss value for each of the data fields; for each of the data fields, subtract the fourth reference loss value of the data field from the fourth loss value of the data field to obtain a fifth loss value of the data field; and determine the fifth weights of the multiple data fields based on the fifth loss value and pre-constructed constraints.

[0292] In some embodiments, the optimization module 1332 is further used to perform normalization on the fifth loss value of each of the data fields to obtain second normalized weights of the multiple data fields; in response to the difference between the second normalized weight and the fourth weight being less than or equal to a second difference value threshold, the second normalized weights of the multiple data fields are used as the fifth weights of the multiple data fields; wherein the fifth weight of each of the data fields is greater than or equal to the fourth weight of each of the data fields at a third preset multiple, and is less than or equal to the fourth weight of each of the data fields at a fourth preset multiple, and the third preset multiple is less than the fourth preset multiple.

[0293] In some embodiments, the training of the fourth neural network model includes multiple iterative training of the fourth neural network model, and the optimization module 1332 is further used to obtain the number of model parameters of the fourth neural network model and the number of iterations of the multiple iterative training; based on the power law relationship between the number of model parameters of the fourth neural network model, the number of iterations of the multiple iterative training and the fourth reference loss value, the fourth reference loss value for each of the data fields is predicted.

[0294] In some embodiments, the optimization module 1332 is further used to perform weighted sum processing on the fifth weight and the fourth weight according to a preset second weighting parameter to obtain a sixth weight of the multiple data fields; use the sixth weights of the multiple data fields as new fifth weights, and execute the third data based on the multiple data fields and the fifth weights of the multiple data fields to train the fourth neural network model to obtain a fifth neural network model.

[0295] In some embodiments, the optimization module 1332 is also used to obtain training data for a target task, wherein the training data for the target task includes data samples and labeled data; the data samples are processed by the fifth neural network model to predict the results of the target task to obtain predicted data; a target task loss value is determined based on the labeled data and the predicted data; and the fifth neural network model is updated based on the target task loss value to obtain a fifth neural network model for the target task.

[0296] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the optimization method of the neural network model provided by the embodiment of the present application, for example, Figure 3A The optimization method of the neural network model is shown.

[0297] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0298] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0299] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0300] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0301] In summary, through the embodiments of the present application, the second neural network model obtained by the first pruning process is verified based on the second data to obtain the first loss value of the second neural network model in each data field. The performance of the second neural network model in each data field represented by the first loss value is used to guide the resource allocation of multiple data fields in the second pruning process. Specifically, the first weights of multiple data fields are updated by the first loss value to obtain the second weights of multiple data fields. On this basis, the resources of multiple data fields are allocated by the second weights, and the second pruning process is performed on the second neural network model to obtain the third neural network model. By updating the first weight to the second weight, the third neural network model is considered and optimized in the pruning process. The data resource allocation is improved. By allocating data resources more accurately, the robustness of the pruned third neural network model to data distribution deviation and the ability to optimize high-potential data areas are improved, so that the model performance degradation caused by pruning can be improved while reducing the model complexity. In addition, the pruned third neural network model contains fewer parameters, so the model occupies less storage space and is easier to deploy on devices with limited storage resources. At the same time, due to the reduced model complexity, the model's inference speed is significantly improved, which is suitable for real-time applications and easy to run on low-cost hardware, reducing the cost of model deployment and maintenance, thereby achieving the beneficial effect of taking into account both resource conservation of computer equipment and model performance.

[0302] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for optimizing a neural network model, characterized in that: The method comprises: Based on first data of multiple data fields and first weights of the multiple data fields, performing a first pruning process on the first neural network model to obtain a second neural network model; Based on the second data in the plurality of data fields, verifying the second neural network model, and obtaining a first loss value in each of the data fields; Based on the first loss value of each of the data fields, updating the first weights of the multiple data fields to obtain second weights of the multiple data fields; Based on the first data of the multiple data fields and the second weights of the multiple data fields, a second pruning process is performed on the second neural network model to obtain a third neural network model.

2. The method according to claim 1, characterized in that The updating of the first weights of the plurality of data fields based on the first loss value in each of the data fields to obtain the second weights of the plurality of data fields comprises: Obtaining a first reference loss value for each of the data fields; For each of the data fields, subtract the first reference loss value corresponding to the data field from the first loss value of the data field to obtain a second loss value of the data field; Based on the second loss value and pre-constructed constraints, second weights of the plurality of data fields are determined.

3. The method according to claim 2, characterized in that The determining, based on the second loss value and the pre-constructed constraint condition, second weights of the plurality of data fields comprises: Normalizing the second loss value of each of the data fields to obtain first normalized weights of the multiple data fields; In response to a difference value between the first normalized weight and the first weight being less than or equal to a first difference value threshold, using the first normalized weights of the plurality of data fields as second weights of the plurality of data fields; The second weight of each of the data fields is greater than or equal to the first weight of a first preset multiple of each of the data fields, and less than or equal to the first weight of a second preset multiple of each of the data fields, and the first preset multiple is less than the second preset multiple.

4. The method according to claim 2, characterized in that: The first pruning process includes multiple iterative processes, and the obtaining of the first reference loss value of each of the data fields includes: Obtaining the number of model parameters of the second neural network model and the number of iterations of the multiple iterative processes; Based on the power-law relationship between the number of model parameters of the second neural network model, the number of iterations of the multiple iterative processes and the first reference loss value, the first reference loss value of each of the data fields is predicted.

5. The method according to claim 2, characterized in that: After determining the second weights of the plurality of data fields based on the second loss value and the pre-constructed constraint condition, the method further includes: Performing weighted sum processing on the second weight and the first weight according to a preset first weighting parameter to obtain third weights of the multiple data fields; The third weights of the multiple data fields are used as new second weights, and the first data based on the multiple data fields and the second weights of the multiple data fields are executed, and a second pruning process is performed on the second neural network model to obtain a third neural network model.

6. The method according to claim 1, characterized in that The updating of the first weights of the plurality of data fields based on the first loss value in each of the data fields to obtain the second weights of the plurality of data fields comprises: Determine a data field of a first priority and a data field of a second priority from the plurality of data fields, wherein the first priority is higher than the second priority; Obtain a first reference loss value for each data field of the second priority level; For each of the second priorities, subtract the first reference loss value of the data field of the second priority from the first loss value of the data field of the second priority to obtain a second loss value of the data field of the second priority; Based on the second loss value and the pre-constructed constraint condition, a second weight of the data field of the second priority is determined, and the first weight of the data field of the first priority is used as the second weight of the data field of the first priority.

7. The method according to claim 1, characterized in that The updating of the first weights of the plurality of data fields based on the first loss value in each of the data fields to obtain the second weights of the plurality of data fields comprises: Determine a data field of a first priority and a data field of a second priority from the plurality of data fields, wherein the first priority is higher than the second priority; Obtaining a second reference loss value for each data field of the first priority and a third reference loss value for each data field of the second priority; For each of the first priorities, the first loss value of the data field of the first priority is subtracted from the second reference loss value of the first priority to obtain the second loss value of the data field of the first priority; and for each of the second priorities, the first loss value of the data field of the second priority is subtracted from the third reference loss value of the second priority to obtain the third loss value of the data field of the second priority; Based on the second loss value, the third loss value and pre-constructed constraints, second weights of the plurality of data fields are determined.

8. The method according to claim 1, characterized in that The first pruning process includes multiple iterative processes, and the first pruning process is performed on the first neural network model based on the first data of the multiple data fields and the first weights of the multiple data fields to obtain the second neural network model, including: The following processing is performed for each iteration: Performing mask processing on the network structure parameters in the first neural network model based on the pruning mask to obtain a masked neural network model; Sampling the first data of the plurality of data fields based on the first weight to obtain sampled data; Determining an inference loss value of the masked neural network model based on the masked neural network model and the sampled data; Determining a pruning loss value based on a difference between a network structure parameter of the masked neural network model and a network structure parameter of a target network model; Based on the inference loss value and the pruning loss value, the pruning mask is updated, wherein the updated pruning mask is used for the next iterative processing, and the mask neural network model obtained by the last iterative processing is used as the second neural network model.

9. The method according to any one of claims 1 to 8, characterized in that: After obtaining the third neural network model, the method further includes: Based on the third data in the multiple data fields and the fourth weights in the multiple data fields, the third neural network model is trained to obtain a fourth neural network model; Based on fourth data in the plurality of data fields, verifying the fourth neural network model to obtain a fourth loss value in each of the data fields; Based on a fourth loss value in each of the data fields, updating fourth weights of the plurality of data fields to obtain fifth weights of the plurality of data fields; Based on the third data of the multiple data fields and the fifth weights of the multiple data fields, the fourth neural network model is trained to obtain a fifth neural network model.

10. The method according to claim 9, characterized in that The updating of the fourth weights of the plurality of data fields based on the fourth loss value in each of the data fields to obtain the fifth weights of the plurality of data fields comprises: obtaining a fourth reference loss value for each of the data fields; For each of the data fields, subtract the fourth reference loss value of the data field from the fourth loss value of the data field to obtain a fifth loss value of the data field; Based on the fifth loss value and the pre-constructed constraint condition, fifth weights of the plurality of data fields are determined.

11. The method according to claim 10, characterized in that The step of determining fifth weights of the plurality of data fields based on the fifth loss value and the pre-constructed constraint condition comprises: Normalizing the fifth loss value of each of the data fields to obtain second normalized weights of the multiple data fields; In response to a difference value between the second normalized weight and the fourth weight being less than or equal to a second difference value threshold, using the second normalized weights of the plurality of data fields as fifth weights of the plurality of data fields; The fifth weight of each of the data fields is greater than or equal to the fourth weight of the third preset multiple of each of the data fields, and less than or equal to the fourth weight of the fourth preset multiple of each of the data fields, and the third preset multiple is less than the fourth preset multiple.

12. An optimization device for a neural network model, characterized in that: The device comprises: A pruning module, configured to perform a first pruning process on the first neural network model based on first data of a plurality of data fields and first weights of the plurality of data fields to obtain a second neural network model; An optimization module, configured to verify the second neural network model based on the second data in the plurality of data fields, and obtain a first loss value in each of the data fields; The optimization module is further used to update the first weights of the multiple data fields based on the first loss value of each of the data fields to obtain the second weights of the multiple data fields; The pruning module is further used to perform a second pruning process on the second neural network model based on the first data in the multiple data fields and the second weights of the multiple data fields to obtain a third neural network model.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the method for optimizing the neural network model according to any one of claims 1 to 11 when executing the computer executable instructions or computer program stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the optimization method of the neural network model described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the optimization method of the neural network model described in any one of claims 1 to 11 is implemented.

Images