Network model reasoning method and device, electronic equipment and storage medium

Abstract

The invention provides a network model reasoning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the input data of a target module in a first target layer of a network model in a content reasoning process for a to-be-processed task based on the network model; according to the input data, first coarseness description information of predicted output data of the target module is determined, a first ratio of the first coarseness description information to second coarseness description information of the input data is determined, and the size of the first ratio represents the calculation importance degree of the target module; according to the first ratio, judging whether a reasoning adjustment condition for the target module is met or not; under the condition that the reasoning adjustment condition is met, according to an adjustment strategy corresponding to the reasoning adjustment condition, reasoning operation of the target module for the input data is adjusted. Therefore, the reasoning speed of the network model can be improved, and the performance loss can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a network model reasoning method and device, an electronic device, and a storage medium. Background Art

[0002] The scale of parameters contained in network models is getting larger and larger, which brings challenges to the deployment and application of network models. During the network model reasoning process, problems such as storage space and bandwidth may occur, which will affect the reasoning speed and performance of the network model. Therefore, it is very important to further optimize the reasoning of the network model. Summary of the invention

[0003] The present invention provides a network model reasoning method and device, an electronic device, and a storage medium.

[0004] In a first aspect, the present disclosure provides a network model reasoning method, the network model reasoning method comprising:

[0005] In a content reasoning process for a task to be processed based on a network model, obtaining input data of a target module in a first target layer included in the network model;

[0006] Determine first coarse-grained description information of predicted output data of the target module according to the input data, and determine a first ratio of the first coarse-grained description information to second coarse-grained description information of the input data, wherein the magnitude of the first ratio represents the computational importance of the target module;

[0007] According to the first ratio, determining whether an inference adjustment condition for the target module is met;

[0008] When the inference adjustment condition is met, the inference operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the inference adjustment condition.

[0009] In a second aspect, the present disclosure provides a network model reasoning device, the network model reasoning device comprising:

[0010] An acquisition module, used for obtaining input data of a target module in a first target layer included in the network model during content reasoning for a task to be processed based on the network model;

[0011] A determination module, configured to determine first coarse-grained description information of predicted output data of the target module according to the input data, and determine a first ratio of the first coarse-grained description information to second coarse-grained description information of the input data, wherein the magnitude of the first ratio represents the computational importance of the target module;

[0012] A judging module, configured to judge whether an inference adjustment condition for the target module is met according to the first ratio;

[0013] The adjustment module is used to adjust the reasoning operation of the target module on the input data according to the adjustment strategy corresponding to the reasoning adjustment condition when the reasoning adjustment condition is met.

[0014] In a third aspect, the present disclosure provides an electronic device comprising: multiple processing cores; and an on-chip network configured to exchange data between the multiple processing cores and external data; wherein one or more instructions are stored in one or more of the processing cores, and one or more of the instructions are executed by one or more of the processing cores, so that one or more of the processing cores can execute the above-mentioned network model inference method.

[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned network model inference method when executed by a processor.

[0016] According to the embodiments provided by the present disclosure, in the process of content reasoning for a task to be processed based on a network model, for a target module of the network model, a first ratio of a first coarse-grained description information of predicted output data to a second coarse-grained description information of input data can be determined according to the input data of the target module, thereby effectively determining the importance of the target module in performing reasoning calculations on the input data, and then judging whether the reasoning adjustment conditions for the target module are met according to the first ratio. When the reasoning adjustment conditions are met, the reasoning operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the reasoning adjustment conditions. In this way, not only can the computational importance of the target module be accurately judged, but also different reasoning operations can be adjusted according to the adjustment strategy for target modules of different computational importance, thereby combining the computational importance of the target module, improving the reasoning speed, reducing the consumption of reasoning resources, and reducing the performance loss of the network model caused by the adjustment strategy of reasoning acceleration.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 A flowchart of a network model reasoning method provided by an embodiment of the present disclosure;

[0020] Figure 2 Schematic diagram of the distribution of the ratio between the output norm and the input norm of the residual module in the embodiment of the present disclosure;

[0021] Figure 3 A block diagram of a network model inference device provided in an embodiment of the present disclosure;

[0022] Figure 4 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0024] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0025] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0026] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0028] The scale of parameters contained in network models is getting larger and larger, which brings challenges to the deployment and application of network models. For example, large language models (LLM) have a large number of parameters. In addition, with the increasing demand for privacy protection and timeliness, there is also a demand to deploy network models on terminal devices. However, the hardware resources of terminal devices themselves are limited. During the network model reasoning process, there may be problems such as storage space and bandwidth, which affect the reasoning speed and performance of the network model. Therefore, it is very important to further optimize the reasoning of the network model. In related technologies, a weight compression method for the network model can be used. For example, the terminal multi-layer can be skipped according to certain rules to end the reasoning in advance. However, this method has a significant performance loss and reduces the accuracy of network model reasoning.

[0029] According to the network model inference method of the embodiment of the present disclosure, in the content inference process of the network model, the input data of the target module in the first target layer included in the network model is obtained, and the computational importance of the target module is judged by predicting the first coarse-grained description information of the predicted output data of the target module. The target module can be analyzed more accurately, and then it is determined whether to adjust the reasoning operation of the target module with respect to the input data. In this way, by predicting the first coarse-grained description of the output data, not only the computational importance of the target module can be accurately judged, but also different reasoning operations can be adjusted according to target modules with different computational importance, thereby improving the reasoning speed and reducing the consumption of reasoning resources while reducing the performance loss of the network model.

[0030] The network model reasoning method according to the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method can be executed by a server.

[0031] Figure 1A flowchart of a network model inference method provided by an embodiment of the present disclosure. Figure 1 , the method comprising:

[0032] S101: In a content reasoning process for a task to be processed based on a network model, input data of a target module in a first target layer included in the network model is obtained.

[0033] In the embodiments of the present disclosure, the network model generally includes multiple network layers, each of which may include one or more modules. The embodiments of the present disclosure do not limit the structure of the network model.

[0034] For example, in a network model based on a Transformer structure, the network layer may include an attention module and a feed-forward network (FFN) module, and there is a residual connection. Each module can be expressed as y=x+f(x), where x is the input, y is the output, and f(x) is the Attention operation or the FFN operation, which is generally a nonlinear function containing a large number of parameters.

[0035] It can be seen that the residual amount is obtained after the input is processed by the Attention module or FFN module of each layer. The residual amount is added to the input of the module to obtain the output. The residual amount is generally small and presents a certain distribution. For example, the sample input and corresponding sample output of multiple residual modules can be obtained, and the ratio between the norm of the sample output and the norm of the sample input can be calculated. See Figure 2 As shown in FIG. , it is a schematic diagram of the distribution of the ratio between the output norm and the input norm of the residual module in the embodiment of the present disclosure, such as Figure 2 As shown, the horizontal axis represents the value of the ratio, and the vertical axis represents the number of residual modules belonging to the ratio. Figure 2 It can also be seen that the ratio between the output norm and the input norm of the residual module is roughly normally distributed, the ratios are different, and are basically less than 1, indicating that the ratio is usually small, and the output of the residual module is also a small residual.

[0036] Therefore, in a possible embodiment of the present disclosure, the target module may be a module with a residual connection, for example, the target module is an Attention module or an FFN module in a network model with a Transformer structure. In this way, when the importance of the target module is judged to be relatively small by the ratio, the target module may be skipped or other adjustment strategies may be used to improve the reasoning acceleration of the network model, while at the same time having a relatively small impact on the performance of the network model.

[0037] Of course, in the embodiments of the present disclosure, there is no limitation on the network model, and the target module is not limited to the module with residual connection.

[0038] S102: Determine first coarse-grained description information of predicted output data of the target module according to the input data, and determine a first ratio of the first coarse-grained description information to second coarse-grained description information of the input data, wherein the magnitude of the first ratio represents the computational importance of the target module.

[0039] In the embodiments of the present disclosure, the first coarse-grained description information or the second coarse-grained description information may represent the amount of information of the data or the average amount of information of the possibility of occurrence, etc. In a possible embodiment, the first coarse-grained description information or the second coarse-grained description information is any one of the following: norm, information entropy, mean, wherein the norm may also be 1-norm, 2-norm, infinite norm, etc. Of course, it may also be other forms or forms that can characterize the correlation with other data, and there is no limitation on this.

[0040] For example, if the coarse-grained description information is a norm, then the ratio between the first norm of the predicted output data and the second norm of the input data indicates the importance or necessity of the calculation of the target module this time. The size of the ratio is positively correlated with the importance of the calculation, that is, the smaller the ratio, the lower the importance of executing the computational reasoning of the target module this time, and the larger the ratio, the higher the importance of executing the computational reasoning of the target module this time.

[0041] In the disclosed embodiment, the first coarse-grained description information of the predicted output data of the target module can be determined, that is, the output of the target module can be a prediction of the coarse-grained description information, so a smaller amount of calculation is required to achieve it, and the inference speed and performance of the network model are less affected.

[0042] S103: According to the first ratio, determine whether the inference adjustment condition for the target module is met.

[0043] S104: When the reasoning adjustment condition is met, the reasoning operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the reasoning adjustment condition.

[0044] Among them, in the embodiments of the present disclosure, the inference adjustment conditions are mainly used to characterize whether the network model can be accelerated in reasoning. Correspondingly, the adjustment strategy of the inference adjustment conditions is also used to improve the inference speed of the network model, such as skipping, increasing sparsity, reducing the numerical accuracy of weights, pruning to reduce the dimension of the neuron hidden layer, etc., which is not limited in the embodiments of the present disclosure.

[0045] In addition, different reasoning adjustment conditions and adjustment strategies may be set based on different needs. For example, if the calculation importance is high, an adjustment strategy for improving accuracy or performance may be adopted, which is not limited in the embodiments of the present disclosure.

[0046] It should be noted that in the embodiments of the present disclosure, there is no limitation on the network model, and the network model can be used to perform any of image processing tasks, voice processing tasks, text processing tasks, and video processing tasks. The output data or data to be transmitted of each layer in the network model includes any of images, voices, texts, and videos.

[0047] In an embodiment of the present disclosure, in a process of content reasoning for a task to be processed based on a network model, input data of a target module in a first target layer included in the network model is obtained, and a first ratio of a first coarse-grained description information of predicted output data of the target module to a second coarse-grained description information of the input data is determined. According to the first ratio, when it is determined that an inference adjustment condition is met, the inference operation of the target module on the input data is adjusted according to an adjustment strategy corresponding to the inference adjustment condition. In this way, a coarse-grained description information is predicted for the output of the target module, and the computational importance of the target module is predicted by predicting the first coarse-grained description information of the output data to the first coarse-grained description information of the input data. This method is simple to implement and has high accuracy. Furthermore, according to the first ratio, the inference operation of the target module on the input data is adjusted. This method comprehensively considers the computational importance and the inference adjustment. This method can reduce the consumption of inference resources and improve the inference speed while reducing the performance loss of the network model.

[0048] The network model inference method according to the embodiment of the present disclosure is described below.

[0049] As mentioned above, with respect to the above step S103, the present disclosure provides the following possible embodiments, specifically including:

[0050] In a possible embodiment, judging whether the inference adjustment condition for the target module is met according to the first ratio includes:

[0051] 1) When the first ratio is less than a first threshold, it is determined that the inference adjustment condition for the target module is met, wherein the first thresholds corresponding to different target modules are the same or different.

[0052] In one possible embodiment, the first threshold is a pre-fixed setting.

[0053] In another possible embodiment, the first threshold is set as a dynamic threshold, and the first threshold is obtained by joint training with the network model, so that the network model meets the preset reasoning performance condition based on the trained first threshold.

[0054] 2) When the first ratio is greater than or equal to the first threshold, it is determined that the inference adjustment condition for the target module is not satisfied.

[0055] In the disclosed embodiments, a first threshold may be pre-set based on pre-tests or actual experience, or the first threshold may be trained for each target module respectively, and may be determined by joint training with the network model as a whole. For example, the first threshold may be optimized and trained so that the network model meets a preset reasoning performance condition while satisfying a certain reasoning acceleration performance indicator, such as the average skip ratio. For example, the preset reasoning performance condition is that the reasoning accuracy is greater than or equal to a set value, and when the joint network model is trained with the first threshold, in order to improve accuracy, other network parameters may not be optimized at this stage. Therefore, the amount of parameters to be trained is small, and the training speed may also be improved.

[0056] Among them, when the first ratio is less than the first threshold, it can be explained that the importance of the inference calculation of the target module for the input data is relatively low. Therefore, it is determined that the inference adjustment conditions are met, and it can be adjusted to a strategy that is more inclined to improve the inference speed, such as reducing the amount of parameter transmission and calculation, while at the same time having less impact on the overall performance and accuracy of the network model.

[0057] When the first ratio is greater than or equal to the first threshold, it indicates that the reasoning calculation of the target module for the input data is more important. At this time, no adjustment is required, and the original reasoning calculation of the target module is still used to obtain the output data corresponding to the input data.

[0058] In a possible embodiment, judging whether the inference adjustment condition for the target module is met according to the first ratio includes:

[0059] 1) Determine the information entropy of the input data and / or determine the reasoning resource information currently corresponding to the target module.

[0060] 2) Determine a second threshold based on information entropy and / or reasoning resource information.

[0061] Among them, information entropy can be used to quantify the uncertainty of information or the average amount of information. The larger the information entropy of the input data, the larger the second threshold, and the easier it is to meet the reasoning adjustment conditions, that is, the information entropy is positively correlated with the second threshold.

[0062] The reasoning resource information can indicate the hardware resources required for operation, calculation or transmission, etc. For example, the reasoning resource information may include bandwidth resources, storage resources, etc. The less the reasoning resource information is, the more inference acceleration is needed, which can reduce the number of parameters and the amount of calculation. Therefore, the less or more limited the reasoning resource information is, the larger the second threshold is, which can increase the possibility of adjusting the reasoning operation of the target module, that is, the reasoning resource information is negatively correlated with the second threshold.

[0063] In addition, if both information entropy and reasoning resource information are referred to simultaneously to determine the second threshold, the embodiments of the present disclosure also provide possible implementation methods, for example, the second threshold can be determined based on information entropy and reasoning resource information, and their respective corresponding influence weights.

[0064] 3) When the first ratio is less than the second threshold, it is determined that the inference adjustment condition for the target module is met.

[0065] 4) When the first ratio is greater than or equal to the second threshold, it is determined that the inference adjustment condition for the target module is not satisfied.

[0066] In a possible embodiment, judging whether the inference adjustment condition for the target module is met according to the first ratio includes:

[0067] 1) Obtain a second ratio obtained by dividing the first ratio by a third threshold.

[0068] 2) when the second ratio is greater than a fourth threshold, determining that the inference adjustment condition for the target module is not satisfied;

[0069] 3) When the second ratio is less than or equal to a fourth threshold, it is determined that the inference adjustment condition for the target module is met.

[0070] In the disclosed embodiment, the second ratio obtained by dividing the first ratio by the third threshold value can characterize the predicted probability sampling. The larger the second ratio is, the higher the uncertainty of the first ratio predicted based on the input data is. In order to improve the accuracy of the network model, the possibility of inference acceleration is smaller, that is, the possibility of satisfying the inference adjustment condition is smaller. Therefore, when the second ratio is greater than the fourth threshold value, the inference adjustment condition is not satisfied. When the second ratio is less than or equal to the fourth threshold value, it is determined that the inference adjustment condition is satisfied.

[0071] In the disclosed embodiment, during the content inference process of the network model, before proceeding to the processing of each target module, the first coarse-grained description information of the predicted output data is determined based on the input data and a first ratio of the first coarse-grained description information to the second coarse-grained description information of the input data is determined, and then whether the inference adjustment conditions are met is determined based on the ratio. The judgment threshold corresponding to each target module can be fixed or changed dynamically, which improves the judgment accuracy and better meets the needs.

[0072] In the embodiments of the present disclosure, several possible adjustment strategies are also provided. Based on different adjustment strategies, the following possible embodiments are provided for the above step S104.

[0073] In a possible embodiment, the adjustment strategy includes skipping the calculation of the target module. When the reasoning adjustment condition is met, the reasoning operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the reasoning adjustment condition, including: when the reasoning adjustment condition is met, skipping the calculation of the target module, and using the input data as the output of the target module for the next module of the target module.

[0074] In the disclosed embodiment, when the first ratio is smaller, it indicates that the calculation importance of the target module for the input data is lower, and the calculation of the target module can be skipped, and the input data can be directly used as output to perform the inference calculation of the next module. In this way, the amount of calculation and the amount of transmission can be reduced, and the inference speed of the network model can be improved. Moreover, the skipping process is performed only when it is judged that the importance of the inference calculation is lower. Therefore, while improving the inference speed, the performance loss caused by skipping the process can also be reduced.

[0075] In a possible embodiment, the adjustment strategy includes adjusting the sparsity of the target module. When the reasoning adjustment condition is met, the reasoning operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the reasoning adjustment condition, including: when the reasoning adjustment condition is met, determining the first sparsity of the target module, wherein the first sparsity is greater than the second sparsity of the target module before adjustment; and obtaining output data for the input data based on the target module with the first sparsity.

[0076] Among them, the higher the sparsity, the fewer the parameters involved in the calculation. Therefore, in the embodiment of the present disclosure, when the reasoning adjustment conditions are met, the sparsity of the target module can be adjusted to improve the reasoning calculation speed of the target module or reduce the time to load the target module data.

[0077] In a possible embodiment, the adjustment strategy includes adjusting the numerical precision of the weight of the target module. When the reasoning adjustment condition is met, the reasoning operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the reasoning adjustment condition, including: when the reasoning adjustment condition is met, the numerical precision of the weight of the target module is adjusted from a first precision to a second precision, the second precision being less than the first precision; and based on the adjusted target module, output data for the input data is obtained.

[0078] Among them, the weights in the network model are usually expressed with a certain precision, such as 16-bit floating point numbers, 8-bit integers, 4-bit integers or other lower bit width representations. It can be understood that the higher the precision, the higher the amount of calculation involved in the reasoning calculation, the required transmission resources, etc. Therefore, in the embodiment of the present disclosure, when it is judged that the necessity of the reasoning calculation of the target module for the input data is low, that is, the impact on the reasoning performance and accuracy of the network model is small, the reasoning acceleration can be achieved by reducing the numerical precision of the weight.

[0079] In a possible embodiment, the adjustment strategy includes adjusting the number of weights of the target module. When the reasoning adjustment conditions are met, the reasoning operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the reasoning adjustment conditions, including: when the reasoning adjustment conditions are met, pruning the target module to reduce the number of weights of the target module; and obtaining output data for the input data based on the pruned target module.

[0080] In the embodiment of the present disclosure, pruning processing means discarding unimportant weights in the network model to reduce the number of weights involved in the calculation and reduce the number of neurons in the hidden layer. The hidden layer is all layers in the network model except the input layer and the output layer. Each neuron is used to receive input from the previous layer and perform weighted summation, activation function and other operations to obtain output. Therefore, in the embodiment of the present disclosure, when it is judged that the necessity of inference calculation of the target module for the input data is low, the inference can be accelerated by reducing the number of weights of the neurons, and the performance loss of the network model can also be reduced.

[0081] Furthermore, in the disclosed embodiment, the network model inference method is applied to the judgment network of the target module, and step S102 can also be implemented based on a pre-trained judgment network. A judgment network can be connected before the calculation of the target module. Of course, for a target module that does not need to adjust the reasoning operation, the judgment network can be disconnected. The judgment network is used to predict the first coarse-grained description information of the output data, or the judgment network is used to predict the first ratio of the first coarse-grained description information of the output data to the second coarse-grained description information of the input data. For example, in a network model with a Transformer structure, all Attention modules and FFN modules in each layer correspond to a judgment network.

[0082] And in the embodiment of the present disclosure, the judgment network is a fully connected neural network structure or a convolutional network structure, and the parameter scale of the judgment network is less than the preset parameter scale requirement, so that the judgment network scale is small, such as the judgment network is implemented as a simple linear transformation, the input is the input data of the target module, and the output is a one-dimensional real number, which is a coarse-grained description of the output data, or a ratio of the coarse-grained description of the output and the input. The judgment network can be a fully connected neural network, a convolutional network, etc. with a small number of parameters, and can also include various operations such as batch normalization and layer normalization, which are not limited in the embodiment of the present disclosure.

[0083] Regarding the training of the judgment network, the present disclosure also provides possible implementation methods:

[0084] In one possible implementation, the judgment network may be jointly trained with the network model to optimize the performance of the network model and the judgment network performance, specifically including:

[0085] 1) Obtain a second training data set, where the second training data set includes training tasks.

[0086] 2) Based on the second training data set, the network model is trained, and in the process of content reasoning of the network model for the training task, the first input data and the first output data of the target module included in the network model are obtained.

[0087] 3) Training the judgment network of the target module according to the first input data and the first output data, and the second sample ratio between the third sample coarse-grained description information of the first output data and the fourth sample coarse-grained description information of the first input data.

[0088] For example, when the judgment network and the network model are jointly trained, the network model and the judgment network can be understood as two training task optimizations, and the optimization is based on the same target loss function. The target loss function includes a first loss function and a second loss function, wherein the first loss function is the cross entropy loss function of the network model, and the second loss function is the cross entropy loss function of the judgment network.

[0089] In another possible implementation, for the trained network model, the judgment network of each target module is trained separately, and different judgment networks are pre-trained independently and asynchronously. Each judgment network is used to predict the first coarse-grained description information of the output data of the connected target module, or the judgment network is used to predict the first ratio of the first coarse-grained description information of the output data to the second coarse-grained description information of the input data of the connected target module. In this way, there is no need to conduct joint training with a large network model, and the pre-training and actual execution process of different judgment networks are only related to the target modules to which they are connected, and can be decoupled from other non-connected target modules, which is more efficient, simpler, and improves accuracy.

[0090] Specifically, it includes: 1) obtaining a first training data set, which is obtained based on the inference data generated by the corresponding target module in the network model. The first training data set includes input sample data of the target module, and a first sample ratio between the first sample coarse-grained description information of the output sample data and the second sample coarse-grained description information of the input sample data.

[0091] 2) Based on the first training data set, the judgment network is trained to obtain a trained judgment network, so that the trained judgment network is used to predict the first coarse-grained description information of the output data according to the input data, or the trained judgment network is used to predict the first ratio according to the input data.

[0092] In the disclosed embodiment, a corresponding first training data set can be constructed for each target module, and a judgment network connected to the target module can be trained. The judgment network of each target module is trained separately, and the training can be achieved based on the local information of the target module. End-to-end training of the entire network model is not required, which can improve accuracy, reduce computational complexity, and improve efficiency.

[0093] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0094] In addition, the present disclosure also provides a network model reasoning device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any network model reasoning method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0095] Figure 3 A block diagram of a network model inference device provided in an embodiment of the present disclosure.

[0096] An acquisition module 31 is used to obtain input data of a target module in a first target layer included in the network model during content reasoning for a task to be processed based on the network model;

[0097] A determination module 32, configured to determine first coarse-grained description information of the predicted output data of the target module according to the input data, and determine a first ratio of the first coarse-grained description information to the second coarse-grained description information of the input data, wherein the magnitude of the first ratio represents the computational importance of the target module;

[0098] A judging module 33, configured to judge whether the inference adjustment condition for the target module is met according to the first ratio;

[0099] The adjustment module 34 is used to adjust the reasoning operation of the target module on the input data according to the adjustment strategy corresponding to the reasoning adjustment condition when the reasoning adjustment condition is met.

[0100] In a possible embodiment, a first training module 35 is further included, and the network model inference method is applied to the judgment network of the target module; the first training module 35 obtains the judgment network by pre-training in the following manner:

[0101] Acquire a first training data set, wherein the first training data set is obtained according to the inference data generated in the network model corresponding to the target module, and the first training data set includes input sample data of the target module, and a first sample ratio between first sample coarse-grained description information of the output sample data and second sample coarse-grained description information of the input sample data;

[0102] Based on the first training data set, the judgment network is trained to obtain a trained judgment network, so that the trained judgment network is used to predict the first coarse-grained description information of the output data based on the input data, or the trained judgment network is used to predict the first ratio based on the input data.

[0103] In a possible embodiment, the target module is connected to a corresponding judgment network, and different judgment networks are pre-trained independently and asynchronously, each judgment network is used to predict the first coarse-grained description information of the output data of the connected target module, or the judgment network is used to predict the first ratio of the first coarse-grained description information of the output data of the connected target module to the second coarse-grained description information of the input data.

[0104] In a possible embodiment, the target module is an Attention module or an FFN module in a network model of a Transformer structure.

[0105] In a possible embodiment, the judgment network is a fully connected neural network structure or a convolutional network structure, and a parameter scale of the judgment network is smaller than a preset parameter scale requirement.

[0106] In a possible embodiment, when judging whether the reasoning adjustment condition for the target module is met according to the first ratio, the judging module 33 is used to:

[0107] When the first ratio is less than a first threshold, determining that the inference adjustment condition for the target module is met, wherein the first thresholds corresponding to different target modules are the same or different, and the first threshold is pre-set, or the first threshold is determined by pre-training for the target module;

[0108] When the first ratio is greater than or equal to the first threshold, it is determined that the inference adjustment condition for the target module is not satisfied.

[0109] In a possible embodiment, when judging whether the reasoning adjustment condition for the target module is met according to the first ratio, the judging module 33 is used to:

[0110] Determine the information entropy of the input data, and / or determine the reasoning resource information currently corresponding to the target module;

[0111] Determining a second threshold value according to the information entropy and / or the reasoning resource information;

[0112] When the first ratio is less than the second threshold, determining that an inference adjustment condition for the target module is met;

[0113] When the first ratio is greater than or equal to the second threshold, it is determined that the inference adjustment condition for the target module is not satisfied.

[0114] In a possible embodiment, when judging whether the reasoning adjustment condition for the target module is met according to the first ratio, the judging module 33 is used to:

[0115] determining a second ratio between the first ratio and a set third threshold;

[0116] When the second ratio is greater than a fourth threshold, determining that the inference adjustment condition for the target module is not satisfied;

[0117] When the second ratio is less than or equal to the fourth threshold, it is determined that the inference adjustment condition for the target module is met.

[0118] In a possible embodiment, the adjustment strategy includes skipping the calculation of the target module, and when the reasoning adjustment condition is met, according to the adjustment strategy corresponding to the reasoning adjustment condition, when adjusting the reasoning operation of the target module on the input data, the adjustment module 34 is used to:

[0119] When the inference adjustment condition is met, the calculation of the target module is skipped, and the input data is used as the output of the target module for the next module of the target module.

[0120] In a possible embodiment, the adjustment strategy includes adjusting the sparsity of the target module. When the reasoning adjustment condition is met, according to the adjustment strategy corresponding to the reasoning adjustment condition, when adjusting the reasoning operation of the target module on the input data, the adjustment module 34 is used to:

[0121] When the inference adjustment condition is met, determining a first sparsity of the target module, wherein the first sparsity is greater than a second sparsity of the target module before adjustment;

[0122] Based on the target module of the first sparsity, output data for the input data is obtained.

[0123] In a possible embodiment, the adjustment strategy includes adjusting the weight numerical precision of the target module. When the reasoning adjustment condition is met, according to the adjustment strategy corresponding to the reasoning adjustment condition, when adjusting the reasoning operation of the target module on the input data, the adjustment module 34 is used to:

[0124] When the inference adjustment condition is met, adjusting the numerical precision of the weight of the target module from a first precision to a second precision, where the second precision is less than the first precision;

[0125] Based on the adjusted target module, output data for the input data is obtained.

[0126] In a possible embodiment, the adjustment strategy includes adjusting the weight quantity of the target module. When the reasoning adjustment condition is met, according to the adjustment strategy corresponding to the reasoning adjustment condition, when adjusting the reasoning operation of the target module on the input data, the adjustment module 34 is used to:

[0127] When the inference adjustment condition is met, pruning the target module to reduce the weight quantity of the target module;

[0128] Based on the pruned target module, output data for the input data is obtained.

[0129] In a possible embodiment, the first coarse-grained description information or the second coarse-grained description information is any one of the following: norm, information entropy, and mean.

[0130] Figure 4 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0131] Reference Figure 4 An embodiment of the present disclosure provides an electronic device, which includes multiple processing cores 401 and an on-chip network 402, wherein the multiple processing cores 401 are connected to the on-chip network 402, and the on-chip network 402 is used to exchange data between the multiple processing cores and external data.

[0132] Among them, one or more instructions are stored in one or more processing cores 401, and the one or more instructions are executed by one or more processing cores 401, so that one or more processing cores 401 can execute the above-mentioned network model inference method.

[0133] In some embodiments, the electronic device may be a brain-like chip, which can use vectorized computing and needs to load the weight information and other parameters of the neural network model through an external memory such as a double data rate (DDR) synchronous dynamic random access memory. Therefore, the batch processing used in the embodiments of the present disclosure has a higher computing efficiency.

[0134] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned network model reasoning method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0135] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned network model inference method.

[0136] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0137] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0138] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0139] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0140] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0141] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0142] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0143] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0144] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0145] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A network model reasoning method, characterized in that: include: In a content reasoning process for a task to be processed based on a network model, obtaining input data of a target module in a first target layer included in the network model; Determine first coarse-grained description information of predicted output data of the target module according to the input data, and determine a first ratio of the first coarse-grained description information to second coarse-grained description information of the input data, wherein the magnitude of the first ratio represents the computational importance of the target module; According to the first ratio, determining whether an inference adjustment condition for the target module is met; When the inference adjustment condition is met, the inference operation of the target module on the input data is adjusted according to the adjustment strategy corresponding to the inference adjustment condition.

2. The network model inference method according to claim 1, characterized in that: The network model inference method is applied to the judgment network of the target module; the judgment network is pre-trained in the following way: Acquire a first training data set, wherein the first training data set is obtained according to the inference data generated in the network model corresponding to the target module, and the first training data set includes input sample data of the target module, and a first sample ratio between first sample coarse-grained description information of the output sample data and second sample coarse-grained description information of the input sample data; Based on the first training data set, the judgment network is trained to obtain a trained judgment network, so that the trained judgment network is used to predict the first coarse-grained description information of the output data based on the input data, or the trained judgment network is used to predict the first ratio based on the input data.

3. The network model inference method according to claim 2, characterized in that: The target module is connected to a corresponding judgment network, and different judgment networks are pre-trained independently and asynchronously, each judgment network is used to predict the first coarse-grained description information of the output data of the connected target module, or the judgment network is used to predict the first ratio of the first coarse-grained description information of the output data of the connected target module to the second coarse-grained description information of the input data.

4. The network model reasoning method according to claim 1, characterized in that: The target module is an attention module or a feed-forward neural network FFN module in a network model of a Transformer structure.

5. The network model inference method according to claim 3, characterized in that: The judgment network is a fully connected neural network structure or a convolutional network structure, and a parameter scale of the judgment network is smaller than a preset parameter scale requirement.

6. The network model reasoning method according to claim 1, characterized in that: The determining, based on the first ratio, whether an inference adjustment condition for the target module is met includes: When the first ratio is less than a first threshold, determining that the inference adjustment condition for the target module is met, wherein the first thresholds corresponding to different target modules are the same or different; When the first ratio is greater than or equal to the first threshold, it is determined that the inference adjustment condition for the target module is not satisfied.

7. The network model reasoning method according to claim 6, characterized in that: The first threshold is a preset fixed setting; or, The first threshold is set as a dynamic threshold, and the first threshold is obtained by joint training with the network model, so that the network model meets a preset reasoning performance condition based on the trained first threshold.

8. The network model inference method according to claim 1, characterized in that: The determining, based on the first ratio, whether an inference adjustment condition for the target module is met includes: Determine the information entropy of the input data, and / or determine the reasoning resource information currently corresponding to the target module; Determining a second threshold value according to the information entropy and / or the reasoning resource information; When the first ratio is less than the second threshold, determining that an inference adjustment condition for the target module is met; When the first ratio is greater than or equal to the second threshold, it is determined that the inference adjustment condition for the target module is not satisfied.

9. The network model inference method according to claim 1, characterized in that: The determining, based on the first ratio, whether an inference adjustment condition for the target module is met includes: Obtaining a second ratio obtained by dividing the first ratio by a third threshold; When the second ratio is greater than a fourth threshold, determining that the inference adjustment condition for the target module is not satisfied; When the second ratio is less than or equal to the fourth threshold, it is determined that the inference adjustment condition for the target module is met.

10. The network model inference method according to any one of claims 1 to 9, characterized in that: The adjustment strategy includes skipping the calculation of the target module, and adjusting the reasoning operation of the target module on the input data according to the adjustment strategy corresponding to the reasoning adjustment condition when the reasoning adjustment condition is met, including: When the inference adjustment condition is met, the calculation of the target module is skipped, and the input data is used as the output of the target module for the next module of the target module.

11. The network model inference method according to any one of claims 1 to 9, characterized in that: The adjustment strategy includes adjusting the sparsity of the target module, and when the inference adjustment condition is met, adjusting the inference operation of the target module on the input data according to the adjustment strategy corresponding to the inference adjustment condition, including: When the inference adjustment condition is met, determining a first sparsity of the target module, wherein the first sparsity is greater than a second sparsity of the target module before adjustment; Based on the target module of the first sparsity, output data for the input data is obtained.

12. The network model inference method according to any one of claims 1 to 9, characterized in that: The adjustment strategy includes adjusting the weight numerical precision of the target module, and when the reasoning adjustment condition is met, adjusting the reasoning operation of the target module on the input data according to the adjustment strategy corresponding to the reasoning adjustment condition, including: When the inference adjustment condition is met, adjusting the numerical precision of the weight of the target module from a first precision to a second precision, where the second precision is less than the first precision; Based on the adjusted target module, output data for the input data is obtained.

13. The network model inference method according to any one of claims 1 to 9, characterized in that: The adjustment strategy includes adjusting the weight quantity of the target module, and when the reasoning adjustment condition is met, adjusting the reasoning operation of the target module on the input data according to the adjustment strategy corresponding to the reasoning adjustment condition, including: When the inference adjustment condition is met, pruning the target module to reduce the weight quantity of the target module; Based on the pruned target module, output data for the input data is obtained.

14. The network model inference method according to claim 1, characterized in that: The first coarse-grained description information or the second coarse-grained description information is any one of the following: norm, information entropy, and mean.

15. A network model reasoning device, characterized in that: include: An acquisition module, used for obtaining input data of a target module in a first target layer included in the network model during content reasoning for a task to be processed based on the network model; A determination module, configured to determine first coarse-grained description information of predicted output data of the target module according to the input data, and determine a first ratio of the first coarse-grained description information to second coarse-grained description information of the input data, wherein the magnitude of the first ratio represents the computational importance of the target module; A judging module, configured to judge whether an inference adjustment condition for the target module is met according to the first ratio; The adjustment module is used to adjust the reasoning operation of the target module on the input data according to the adjustment strategy corresponding to the reasoning adjustment condition when the reasoning adjustment condition is met.

16. An electronic device, characterized in that: include: Multiple processing cores; as well as A network on chip is configured to exchange data between the multiple processing cores and external data; wherein one or more instructions are stored in one or more of the processing cores, and one or more of the instructions are executed by one or more of the processing cores, so that one or more of the processing cores can execute the network model inference method described in any one of claims 1-14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the network model reasoning method according to any one of claims 1 to 14.

Images