Information processing method, apparatus, computer-readable storage medium, and computer program product

Abstract

The present disclosure relates to an information processing method, device, computer readable storage medium and computer program product, and relates to the technical field of computer. The information processing method comprises: inputting first information into a first machine learning model to output a candidate processing result, the first machine learning model being composed of a plurality of specified layers in a second machine learning model, a difference between the input and the output of each specified layer in the plurality of specified layers being greater than a threshold value; and outputting a final processing result according to the candidate processing result by using the second machine learning model. The technical scheme of the present disclosure can reduce the resources required for processing, so as to improve the information processing performance.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an information processing method, an information processing apparatus, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Machine learning models based on artificial intelligence technology have been widely applied in various practical information processing applications. Machine learning models generally require significant inference costs. During the inference process of machine learning models, major performance bottlenecks include autoregressive decoding.

[0003] In related technologies, in order to alleviate the inference pressure of machine learning models, acceleration methods such as model pruning, model quantization, and knowledge distillation can be used. Summary of the Invention

[0004] According to some embodiments of this disclosure, an information processing method is provided, comprising: inputting first information into a first machine learning model to output candidate processing results, wherein the first machine learning model is composed of multiple specified layers in a second machine learning model, and the difference between the input and output of each specified layer is greater than a threshold; and using the second machine learning model to output a final processing result based on the candidate processing results.

[0005] In some embodiments, the angular distance between the input vector and the output vector of each specified layer is greater than a threshold.

[0006] In some embodiments, the information processing method further includes: inputting second information into a second machine learning model for processing to obtain inputs and outputs of multiple layers in the second machine learning model, for determining multiple specified layers from the multiple layers.

[0007] In some embodiments, the second information is obtained from multiple types of subtask datasets.

[0008] In some embodiments, inputting first information into a first machine learning model to output candidate processing results includes: inputting first information into the first machine learning model to determine candidate outputs of a plurality of specified layers; and using a second machine learning model to output a final processing result based on the candidate processing results includes: using the second machine learning model to determine the final processing result based on the candidate outputs of a plurality of specified layers.

[0009] In some embodiments, using a second machine learning model to determine the final processing result based on the candidate outputs of multiple specified layers includes: inputting a specified candidate output from the candidate outputs into the first non-specified layer in the second machine learning model to determine the final processing result, wherein the specified candidate output is the candidate output of the specified layer preceding the first non-specified layer.

[0010] In some embodiments, inputting the first information into a first machine learning model to output candidate processing results includes: inputting the first information into a second machine learning model to output cached values ​​of key vectors and value vectors for each specified layer; and using the first machine learning model to output candidate processing results based on the cached values.

[0011] According to some other embodiments of this disclosure, an information processing apparatus is provided, comprising: a first processing unit, configured to input first information into a first machine learning model to output candidate processing results, wherein the first machine learning model is composed of multiple specified layers in a second machine learning model, and the difference between the input and output of each specified layer is greater than a threshold; and a second processing unit, configured to use the second machine learning model to output a final processing result based on the candidate processing results.

[0012] In some embodiments, the angular distance between the input vector and the output vector of each specified layer is greater than a threshold.

[0013] In some embodiments, the second processing unit inputs the second information into the second machine learning model for processing to obtain the inputs and outputs of multiple layers in the second machine learning model, for determining multiple specified layers from the multiple layers.

[0014] In some embodiments, the second information is obtained from multiple types of subtask datasets.

[0015] In some embodiments, the first processing unit inputs first information into the first machine learning model to determine candidate outputs of multiple specified layers; the second processing unit uses the second machine learning model to determine the final processing result based on the candidate outputs of the multiple specified layers.

[0016] In some embodiments, the second processing unit inputs a specified candidate output from the candidate outputs into the first unspecified layer in the second machine learning model to determine the final processing result, wherein the specified candidate output is the candidate output of the specified layer preceding the first unspecified layer.

[0017] In some embodiments, the second processing unit inputs the first information into the second machine learning model to output cached values ​​of the key vector and value vector for each specified layer; the first processing unit uses the first machine learning model to output candidate processing results based on the cached values.

[0018] According to further embodiments of the present disclosure, an information processing apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the information processing method of any of the above embodiments based on instructions stored in the memory.

[0019] According to further embodiments of the present disclosure, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the information processing method of any of the above embodiments.

[0020] According to further embodiments of this disclosure, a computer program product is also provided, including instructions that, when executed by a processor, cause the processor to perform an information processing method according to any of the foregoing embodiments. Attached Figure Description

[0021] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0022] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0023] Figure 1 Flowcharts illustrating some embodiments of the information processing methods of this disclosure;

[0024] Figure 2 Schematic diagrams illustrating some embodiments of the information processing methods of this disclosure;

[0025] Figure 3 Block diagrams illustrating some embodiments of the information processing apparatus of this disclosure;

[0026] Figure 4 Block diagrams illustrating other embodiments of the information processing apparatus of this disclosure;

[0027] Figure 5 Block diagrams illustrating further embodiments of the information processing apparatus of this disclosure are shown. Detailed Implementation

[0028] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0029] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0030] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0031] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0032] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0034] The inventors of this disclosure have discovered the following problem in the aforementioned related technologies: the implementation process of accelerated processing is complex, which leads to a decrease in information processing performance.

[0035] As mentioned earlier, while acceleration methods such as model pruning, model quantization, and knowledge distillation improve computational efficiency to some extent, these methods usually require changes to the model architecture, changes to the training procedure, retraining or fine-tuning the model, and cannot keep the output the same before and after acceleration.

[0036] To address the aforementioned technical issues, the speculative decoding method can quickly generate candidate outputs using a relatively small model (called a draft model). Then, using a relatively large original model to be deployed (called a target model), the accuracy of the candidate outputs is checked through forward propagation to ensure that the outputs of the draft model and the target model are consistent.

[0037] However, speculative inference methods require a suitable draft model to ensure that the draft model and the target model have similar outputs. If the target model is a fine-tuned model, the aforementioned acceleration process becomes even more complex.

[0038] Therefore, finding or training a draft model that can effectively mimic the output of the target model is an important task. Furthermore, introducing an additional draft model, especially on devices with limited memory, increases the memory overhead of devices such as GPUs (Graphics Processing Units) and complicates deployment.

[0039] In view of this, the present disclosure proposes an information processing technology solution that can simplify and accelerate the processing implementation process, thereby improving information processing performance. For example, the technical solution of the present disclosure can be implemented through the following embodiments.

[0040] Figure 1Flowcharts illustrating some embodiments of the information processing methods of this disclosure are shown.

[0041] like Figure 1 As shown, in step 110, the first information is input into the first machine learning model to output candidate processing results. The first machine learning model consists of multiple specified layers in the second machine learning model, where the difference between the input and output of each specified layer is greater than a threshold. For example, the first information can be input into the first machine learning model multiple times to output multiple candidate processing results.

[0042] For example, the first information may include image information, natural language information, audio information, text information, etc.; the first information may be information used to realize interaction between humans and computers, such as information to be processed input by users to computers through interfaces such as interactive platforms, and the processing result output by the computer is the reply information corresponding to the first information.

[0043] In some embodiments, the number of layers in the first machine learning model is less than the number of layers in the second machine learning model. The first machine learning model can be called a draft model, and the second machine learning model can be called a target model. The first machine learning model can be generated offline in advance, and the first machine learning model consists of multiple specified layers from the multiple layers of the second machine learning model.

[0044] In step 120, a second machine learning model is used to output the final processing result based on the candidate processing results. For example, the second machine learning model can be used to verify multiple candidate processing results to determine the final processing result based on the multiple candidate processing results.

[0045] In the above embodiments, a portion of the layers of the second machine learning model is used to construct the first machine learning model, which assists the second machine learning model in inference. In this way, by sharing a portion of the same machine learning model, the complexity of the computer acceleration process is reduced, thereby reducing the computing and storage resources required for computer processing and improving the computer's information processing performance.

[0046] The following examples illustrate the method for generating the first machine learning model (i.e., the draft model).

[0047] In some embodiments, the second information is input into a second machine learning model for processing to obtain the inputs and outputs of multiple layers in the second machine learning model, used to determine multiple specified layers from the multiple layers. For example, the second information is obtained from multiple types of subtask datasets.

[0048] For example, the second information may include image information, natural language information, audio information, text information, etc.; the second information may be information used to realize interaction between humans and computers, such as information to be processed input by the user to the computer through an interface such as an interactive platform, and the computer outputs the processing result as the reply information corresponding to the first information.

[0049] For example, a portion of data can be extracted from various types of subtask datasets to construct a more complete dataset called a calibration dataset; each data point in the calibration dataset can be used as secondary information and input into the target model to obtain the output activation value of each layer in the target model.

[0050] In this way, by constructing a complete dataset, important layers can be accurately selected from the second machine learning model to construct the first machine learning model, thereby improving the accuracy of information processing.

[0051] In some embodiments, the angular distance between the input and output vectors of each specified layer is greater than a threshold. For example, the angular distance is the angle between the input and output vectors. Specified layers can also be determined based on other physical quantities that characterize the difference between the input and output vectors (such as vector distance).

[0052] For example, the output of the l-th layer of the second machine learning model is a (l) The input of the l-th layer is the output a of the (l-1)-th layer. (l -1) a (l) With a (l-1) The angular distance is:

[0053]

[0054] "·" represents the dot product of vectors, and "‖‖" represents the L2 norm of a vector.

[0055] For example, the angular distance θ corresponding to each layer of the second machine learning model can be calculated. (l) To obtain the angular distance sequence {θ} corresponding to layers 1 to L in the second machine learning model arranged in the processing order. (1) ,θ (2) ,…,θ (l) ,…,θ (L) ,}。 In response to θ (l) ≤τ, the l-th layer is marked as an "unimportant layer" and is not included in the first machine learning model, where τ is a preset threshold; in response to θ (l) >τ, the l-th layer is marked as an "important layer", and this layer is included in the first machine learning model.

[0056] In this way, the draft model is determined by a subset of the layers in the target model through the offline preprocessing described above. On the one hand, the draft model can be generated without training an additional machine learning model; on the other hand, only the target model needs to be deployed during the deployment phase. This reduces the deployment cost of machine learning models and improves the efficiency of computer information processing and human-computer interaction.

[0057] For example, it can be done through Figure 2 The example in the document generates a first machine learning model.

[0058] Figure 2 Schematic diagrams illustrating some embodiments of the information processing methods of this disclosure.

[0059] like Figure 2 As shown, the second machine learning model includes layers 210 to 270. The solid arrows in the figure represent the inference process of the second machine learning model, namely layer 210-layer 220-layer 230-layer 240-layer 250-layer 260-layer 270. By calculating the angular distance corresponding to each layer of the second machine learning model, it can be confirmed that the layers with angular distances greater than the threshold include layers 210, 220, 240 and 270. It can be determined that the first machine learning model consists of layers 210, 220, 240 and 270. The dashed arrows in the figure represent the inference process of the first machine learning model, namely layer 210-layer 220-layer 240-layer 270.

[0060] For example, the output of the l-th layer of the second machine learning model is a (l) The input of the l-th layer is the output a of the (l-1)-th layer. (l -1) a (l) With a (l-1) The angular distance is θ (l) It can calculate the angular distance θ corresponding to each layer of the second machine learning model. (l) To obtain the angular distance sequence {θ} corresponding to layers 210 to 270 in the second machine learning model arranged in the processing order. (210) ,θ (220) ,…,θ (l) ,…,θ (270) ,}。 In response to θ (l) ≤τ, the l-th layer is marked as an "unimportant layer" (such as layer 230, layer 250, layer 260, etc.), and these layers are not included in the first machine learning model, where τ is a preset threshold; in response to θ (l)>τ, the l-th layer is marked as an “important layer” (such as layer 210, layer 220, layer 240, layer 270, etc.), and these layers are included in the first machine learning model.

[0061] In this way, the draft model is determined by a subset of layers in the target model through the aforementioned offline preprocessing. On the one hand, a draft model can be generated without training an additional machine learning model; on the other hand, only the target model needs to be deployed during the deployment phase, thereby reducing the deployment cost of the machine learning model and improving information processing efficiency. For example, by selecting important layers as draft models based on the angular distance differences in the output of each layer during offline preprocessing, it is unnecessary to train an additional model as a draft model, reducing the complexity of accelerating inference; and by eliminating the need to deploy two independent models, the computational and storage resources required for model deployment are alleviated.

[0062] For example, the input information is obtained from multiple types of subtask datasets. A portion of data can be extracted from each type of subtask dataset to construct a more complete dataset, called a calibrate dataset. Each data point in the calibrate dataset is used as input information and fed into a second machine learning model to obtain the output activation values ​​of layers 210-270 in the second machine learning model. These activation values ​​are used to determine layers 210, 220, 240, and 270 as important layers to generate the first machine learning model.

[0063] In this way, by constructing a complete dataset, important layers can be accurately selected from the second machine learning model to construct the first machine learning model, thereby improving the accuracy of information processing.

[0064] The following examples illustrate the reasoning process of the first machine learning model (i.e., the draft model).

[0065] In some embodiments, the first information is input into the second machine learning model to output cached values ​​(i.e., KV-cache) of key vectors and value vectors for each specified layer; the first machine learning model is then used to output candidate processing results based on the cached values.

[0066] For example, firstly, the second machine learning model can be used to reason about the information to be processed in order to determine the KV-cache corresponding to each layer in the second machine learning model; then, in the reasoning stage of the second machine learning model generating candidate processing results (also known as draft processing results), each layer in the second machine learning model can use the pre-determined KV-cache corresponding to that layer to output the candidate processing results.

[0067] In this way, by taking advantage of the fact that the layers included in the draft model are a subset of the layers included in the target model, each layer of the draft model can reuse the KV-cache of the corresponding layer in the target model, thereby reducing the amount of computation and improving the efficiency of information processing.

[0068] For example, it can be done through Figure 2 The embodiments described above enable the first machine learning model to reuse cached values ​​of the key vector and value vector.

[0069] like Figure 2 As shown, the first machine learning model includes layers 210, 220, 240, and 270, and the second machine learning model includes layers 210 to 270. That is, the layers in the first machine learning model are a subset of the layers in the second machine learning model. The second machine learning model can be used to infer the information to be processed to determine the KV-cache corresponding to each layer from 210 to 270. Then, during the inference stage of generating candidate processing results in the second machine learning model, the KV-cache corresponding to layers 210, 220, 240, and 270 is reused to output the candidate processing results.

[0070] The following examples illustrate the reasoning process of the second machine learning model (i.e., the target model).

[0071] In some embodiments, first information is input into a first machine learning model to determine candidate outputs (also known as draft outputs) of multiple specified layers; a second machine learning model is then used to determine the final processing result based on the candidate outputs of the multiple specified layers. For example, a specified candidate output from the candidate outputs is input into the first unspecified layer in the second machine learning model to determine the final processing result, where the specified candidate output is the candidate output of the specified layer preceding the first unspecified layer.

[0072] For example, in the second machine learning model, multiple layers are arranged according to the processing order of the inference process, with layers closer to the input end of the second machine learning model being the earlier layers and layers closer to the output end being the later layers. The second machine learning model can reuse the draft output of the layer preceding the first non-specified layer.

[0073] This ensures that the intermediate processing results of the draft model reused by the target model have a small deviation, which improves the processing efficiency of the target model and ensures the processing accuracy of the target model.

[0074] For example, it can be done through Figure 2The embodiments described above allow the second machine learning model to reuse the output of some specified layers.

[0075] like Figure 2 As shown, the first machine learning model includes layers 210, 220, 240, and 270 in sequence, while the second machine learning model includes layers 210 through 270 in sequence. That is, the layers in the first machine learning model are a subset of the layers in the second machine learning model. Layers 210 and 220 are shared layers (i.e., designated layers) in both the first and second machine learning models. Layer 230 is the first non-shared layer (i.e., non-designated layer) in the second machine learning model. Layer 220 is the designated layer preceding the non-designated layer 230. The second machine learning model can reuse the draft output of layer 220 as the input of layer 230.

[0076] For example, during the inference process of the second machine learning model, the draft output of layer 220 generated during the inference process of the first machine learning model can be input into layer 230 of the second machine learning model to achieve output sharing of a specified layer.

[0077] For example, in the inference process of the second machine learning model, the inference process of layers 210 to 220 of the first machine learning model is directly reused, that is, the draft output of layer 220 is used as the input of layer 230 of the second machine learning model; then through layers 230-240-250-260-270, the inference process of the second machine learning model is completed to determine the final processing result.

[0078] In the above embodiments, there is no need to train an additional model as a draft model; instead, a portion of the target model's layers are directly used as the draft model. This saves the computational and storage resources required to deploy two models. Moreover, the draft model can share inter-layer outputs and inputs with the target model. During the verification of candidate processing results in the target model, the overlapping layers between the draft model and the target model can be utilized, thereby improving the computer's information processing efficiency.

[0079] Figure 3 Block diagrams illustrating some embodiments of the information processing apparatus of this disclosure are shown.

[0080] like Figure 3As shown, the information processing device 3 includes: a first processing unit 31, used to input first information into a first machine learning model to output candidate processing results, wherein the first machine learning model is composed of multiple specified layers in a second machine learning model, and the difference between the input and output of each specified layer is greater than a threshold; and a second processing unit 32, used to use the second machine learning model to output the final processing result based on the candidate processing results.

[0081] In some embodiments, the angular distance between the input vector and the output vector of each specified layer is greater than a threshold.

[0082] In some embodiments, the second processing unit 32 inputs the second information into the second machine learning model for processing to obtain the inputs and outputs of multiple layers in the second machine learning model, for determining multiple specified layers from the multiple layers.

[0083] In some embodiments, the second information is obtained from multiple types of subtask datasets.

[0084] In some embodiments, the first processing unit 31 inputs first information into the first machine learning model to determine candidate outputs of multiple specified layers; the second processing unit 32 uses the second machine learning model to determine the final processing result based on the candidate outputs of the multiple specified layers.

[0085] In some embodiments, the second processing unit 32 inputs a specified candidate output from the candidate outputs into the first unspecified layer in the second machine learning model to determine the final processing result, wherein the specified candidate output is the candidate output of the specified layer preceding the first unspecified layer.

[0086] In some embodiments, the second processing unit 32 inputs the first information into the second machine learning model to output the cached values ​​of the key vector and value vector of each specified layer; the first processing unit 31 uses the first machine learning model to output candidate processing results based on the cached values.

[0087] In the above embodiments, the importance of each layer is ranked by calculating and analyzing the differences between the input and output of each layer of the second machine learning model; layers with angular distance changes less than a preset threshold are retained to generate the first machine learning model; the original second machine learning model to be deployed is used as a validation model, and the first machine learning model is used as a draft model, and self-speculative sampling is performed. In this way, inference speed can be effectively improved and computational resources saved without training an additional draft model.

[0088] Figure 4 Block diagrams illustrating other embodiments of the information processing apparatus of this disclosure are shown.

[0089] like Figure 4As shown, the information processing apparatus 4 of this embodiment includes a memory 41 and a processor 42 coupled to the memory 41. The processor 42 is configured to execute the information processing method of any embodiment of this disclosure based on instructions stored in the memory 41.

[0090] The memory 41 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, boot loader, database, and other programs.

[0091] In some embodiments, the processor 42 is configured to input first information into a first machine learning model to output candidate processing results. The first machine learning model consists of multiple specified layers in a second machine learning model, wherein the difference between the input and output of each specified layer is greater than a threshold. Using the second machine learning model, the processor outputs a final processing result based on the candidate processing results.

[0092] In some embodiments, the angular distance between the input vector and the output vector of each specified layer is greater than a threshold.

[0093] In some embodiments, the processor 42 is configured to input second information into a second machine learning model for processing to obtain inputs and outputs of multiple layers in the second machine learning model, for determining multiple specified layers from the multiple layers.

[0094] In some embodiments, the second information is obtained from multiple types of subtask datasets.

[0095] In some embodiments, the processor 42 is configured to input first information into a first machine learning model to determine candidate outputs of a plurality of specified layers; and to use a second machine learning model to determine the final processing result based on the candidate outputs of the plurality of specified layers.

[0096] In some embodiments, the processor 42 is configured to input a specified candidate output from the candidate outputs into the first unspecified layer in the second machine learning model to determine the final processing result, wherein the specified candidate output is the candidate output of the specified layer preceding the first unspecified layer.

[0097] In some embodiments, the processor 42 is configured to input first information into a second machine learning model to output cached values ​​of key vectors and value vectors for each specified layer; and to output candidate processing results based on the cached values ​​using the first machine learning model.

[0098] In the above embodiments, the importance of each layer is ranked by calculating and analyzing the differences between the input and output of each layer of the second machine learning model; layers with angular distance changes less than a preset threshold are retained to generate the first machine learning model; the original second machine learning model to be deployed is used as a validation model, and the first machine learning model is used as a draft model, and self-speculative sampling is performed. In this way, inference speed can be effectively improved and computational resources saved without training an additional draft model.

[0099] Figure 5 Block diagrams illustrating further embodiments of the information processing apparatus of this disclosure are shown.

[0100] like Figure 5 As shown, the information processing apparatus 5 of this embodiment includes a memory 510 and a processor 520 coupled to the memory 510. The processor 520 is configured to execute the information processing method of any of the foregoing embodiments based on instructions stored in the memory 510.

[0101] The memory 510 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0102] The information processing device 5 may also include an input / output interface 530, a network interface 540, and a storage interface 550. These interfaces 530, 540, and 550, as well as the memory 510 and processor 520, can be connected via, for example, a bus 560. The input / output interface 530 provides a connection interface for input / output devices such as a monitor, mouse, keyboard, touchscreen, microphone, and speakers. The network interface 540 provides a connection interface for various networked devices. The storage interface 550 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0103] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] In some embodiments, a computer-readable storage medium stores computer instructions that, when executed by a processor, implement an information processing method, including: inputting first information into a first machine learning model to output candidate processing results, the first machine learning model comprising a plurality of specified layers in a second machine learning model, wherein the difference between the input and output of each specified layer is greater than a threshold; and using the second machine learning model to output a final processing result based on the candidate processing results.

[0105] In some embodiments, the angular distance between the input vector and the output vector of each specified layer is greater than a threshold.

[0106] In some embodiments, second information is input into a second machine learning model for processing to obtain the inputs and outputs of multiple layers in the second machine learning model, for determining multiple specified layers from the multiple layers.

[0107] In some embodiments, the second information is obtained from multiple types of subtask datasets.

[0108] In some embodiments, first information is input into a first machine learning model to determine candidate outputs for multiple specified layers; and a second machine learning model is used to determine the final processing result based on the candidate outputs for multiple specified layers.

[0109] In some embodiments, a specified candidate output from the candidate output is input into the first unspecified layer in the second machine learning model to determine the final processing result, wherein the specified candidate output is the candidate output of the specified layer preceding the first unspecified layer.

[0110] In some embodiments, the first information is input into the second machine learning model to output cached values ​​of the key vector and value vector for each specified layer; the first machine learning model is then used to output candidate processing results based on the cached values.

[0111] In the above embodiments, the importance of each layer is ranked by calculating and analyzing the differences between the input and output of each layer of the second machine learning model; layers with angular distance changes less than a preset threshold are retained to generate the first machine learning model; the original second machine learning model to be deployed is used as a validation model, and the first machine learning model is used as a draft model, and self-speculative sampling is performed. In this way, inference speed can be effectively improved and computational resources saved without training an additional draft model.

[0112] The information processing method, information processing apparatus, computer-readable storage medium, and computer program product according to this disclosure have been described in detail above. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0113] The methods and systems of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the specific order described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0114] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. An information processing method applied to inference acceleration of a machine learning model, comprising: inputting first information into a first machine learning model to output a candidate processing result, the first machine learning model being composed of a plurality of specified layers in a second machine learning model, a difference between an input and an output of each of the plurality of specified layers being greater than a threshold value, wherein the first information comprises any one of image information, audio information, and text information to be processed, and an angular distance between an input vector and an output vector of each of the specified layers is greater than the threshold value; outputting, by the second machine learning model, a final processing result according to the candidate processing result. 2.The information processing method of claim 1, further comprising: inputting second information into the second machine learning model to obtain inputs and outputs of a plurality of layers in the second machine learning model, for determining the plurality of specified layers from the plurality of layers.

3. The information processing method according to claim 2, wherein the second information is obtained from a plurality of types of sub-task data sets.

4. The information processing method according to any one of claims 1 to 3, wherein the inputting first information into a first machine learning model to output a candidate processing result comprises: inputting the first information into the first machine learning model to determine candidate outputs of the plurality of specified layers; the outputting, by the second machine learning model, a final processing result according to the candidate processing result comprises: determining, by the second machine learning model, the final processing result according to the candidate outputs of the plurality of specified layers.

5. The information processing method according to claim 4, wherein the determining, by the second machine learning model, the final processing result according to the candidate outputs of the plurality of specified layers comprises: inputting a specified candidate output in the candidate outputs into a non-specified layer arranged at a top position in the second machine learning model to determine the final processing result, the specified candidate output being a candidate output of a previous specified layer of the non-specified layer arranged at the top position.

6. The information processing method according to any one of claims 1 to 3, wherein the inputting first information into a first machine learning model to output a candidate processing result comprises: inputting the first information into the second machine learning model to output cache values of key vectors and value vectors of each of the specified layers; outputting, by the first machine learning model, the candidate processing result according to the cache values. 7.An information processing apparatus applied to inference acceleration of a machine learning model, comprising: a first processing unit configured to input first information into a first machine learning model to output a candidate processing result, the first machine learning model being composed of a plurality of specified layers in a second machine learning model, a difference between an input and an output of each of the plurality of specified layers being greater than a threshold value, wherein the first information comprises any one of image information, audio information, and text information to be processed, and an angular distance between an input vector and an output vector of each of the specified layers is greater than the threshold value; a second processing unit configured to output, by the second machine learning model, a final processing result according to the candidate processing result. 8.An information processing apparatus, comprising: a memory; and a processor. a processor coupled to the memory, the processor configured to perform the information processing method of any one of claims 1-6 based on instructions stored in the memory.

9. A computer readable storage medium having stored thereon computer instructions, the instructions, when executed by a processor, implementing the information processing method of any one of claims 1-6.

10. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the information processing method according to any one of claims 1-6.

Images