Data processing method, computing device, and storage medium

By compressing the parameters of the deep learning model into a quantitative model and converting it into local model parameters, the problems of high memory consumption and large communication losses in the deep learning model computing service are solved, and efficient utilization of memory resources and saving of computing resources are achieved.

WO2024230428A1PCT designated stage expired Publication Date: 2024-11-14CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/088054
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-05
Filing Date
2024-04-16
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

When supporting computing services of deep learning models, the prior art requires a large amount of computing resources, resulting in increased memory consumption and high communication losses, making it difficult to effectively save memory resources and reduce computing resource consumption.

Method used

By compressing the model parameters of the target model into quantitative model parameters and loading the quantitative model parameters in the video memory, we determine the local quantitative model parameters required for the forward calculation of the target model, convert it into local model parameters, and perform forward calculations to avoid video memory Expansion and communication loss.

Benefits of technology

Effectively save video memory resources, reduce communication losses between graphics cards, improve video memory resource utilization, reduce communication needs across graphics cards, and ensure model prediction accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088054_14112024_PF_FP_ABST
    Figure CN2024088054_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present specification are a data processing method, a computing device, and a storage medium, wherein the data processing method comprises: in response to a data processing request corresponding to a target model, in a video memory loaded with quantization model parameters of the target model, determining a local quantization model parameter corresponding to forward computation of the target model, wherein the quantization model parameters are obtained by compressing model parameters of the target model; converting the local quantization model parameter into a local model parameter; and using the local model parameter to perform the forward computation on data to be processed associated with the data processing request, so as to serve said data as a response to the data processing request. The usage amount of the video memory is reduced by means of compressing the model parameters; and in the application, only a parameter required for the forward computation is converted, so that not only can the model prediction precision be ensured, but also parameter expansion caused by the conversion can be avoided, thereby effectively saving the consumption of video memory resources.
Need to check novelty before this filing date? Find Prior Art

Description

Data processing method, computing device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 5, 2023, with application number 202310511586.7 and invention name “Data Processing Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification relate to the field of computer technology, and in particular to data processing methods, computing devices, and storage media. Background Art

[0003] With the advancement of computer technology, artificial intelligence (AI) has been applied in an increasing number of fields. Furthermore, with the development of AI, various deep learning frameworks have emerged. General-purpose deep learning frameworks such as TensorFlow and PyTorch are used in fields such as natural language processing, computer vision, and speech processing, as well as in industries such as machine translation, smart finance, smart healthcare, and autonomous driving. They are among the most widely used deep learning frameworks today. However, computing services supporting deep learning models typically require a significant amount of computing resources. This is particularly true for models with complex and large neural network structures, which consume a significant amount of computing resources. Improving prediction accuracy by using more computing resources is crucial. This not only consumes a large amount of computing resources, but also requires extensive communication between the device nodes that support these resources, further increasing the consumption of computing resources. Therefore, an effective solution is urgently needed to address this issue.

[0004] Summary of the Invention

[0005] In view of this, embodiments of this specification provide data processing methods, computing devices, and storage media. One or more embodiments of this specification also relate to a data processing device, a text processing method, a text processing device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a data processing method is provided, comprising: in response to a data processing request corresponding to a target model, determining, in a video memory that loads the quantization model parameters of the target model, local quantization model parameters corresponding to a forward calculation of the target model, wherein the quantization model parameters are obtained by compressing the model parameters of the target model; converting the local quantization model parameters into local model parameters; and performing the forward calculation on the to-be-processed data associated with the data processing request using the local model parameters as a response to the data processing request.

[0007] According to the second aspect of the embodiments of this specification, another data processing method is provided, which is applied to a server, including: selecting a target model based on a model selection request submitted by a client; in response to a data processing request corresponding to the target model, determining local quantization model parameters corresponding to the forward calculation of the target model in the video memory that loads the quantization model parameters of the target model, wherein the quantization model parameters are obtained by compressing the model parameters of the target model; converting the local quantization model parameters into local model parameters; using the local model parameters to perform the forward calculation on the data to be processed associated with the data processing request, determining the target data based on the calculation results, and feeding back the target data to the client.

[0008] According to a third aspect of the embodiments of this specification, a text processing method is provided, comprising: in response to a text processing request corresponding to a target language model, determining, in a video memory that loads the quantization model parameters of the target language model, local quantization model parameters corresponding to a forward calculation of the target language model, wherein the quantization model parameters are obtained by compressing the model parameters of the target language model; converting the local quantization model parameters into local model parameters; and performing the forward calculation on the to-be-processed text associated with the text processing request using the local model parameters as a response to the text processing request.

[0009] According to a fourth aspect of the embodiments of this specification, a data processing device is provided, comprising: a determination module, configured to, in response to a data processing request corresponding to a target model, determine, in a video memory in which the quantization model parameters of the target model are loaded, local quantization model parameters corresponding to the forward calculation of the target model, wherein the quantization model parameters are obtained by compressing the model parameters of the target model; a conversion module, configured to convert the local quantization model parameters into local model parameters; and a calculation module, configured to perform the forward calculation on the to-be-processed data associated with the data processing request using the local model parameters as a response to the data processing request.

[0010] According to the fifth aspect of the embodiments of this specification, another data processing device is provided, which is applied to a server, including: a model selection module, configured to select a target model according to a model selection request submitted by a client; a parameter determination module, configured to respond to a data processing request corresponding to the target model, and determine local quantization model parameters corresponding to the forward calculation of the target model in the video memory that loads the quantization model parameters of the target model, wherein the quantization model parameters are obtained by compressing the model parameters of the target model; a parameter conversion module, configured to convert the local quantization model parameters into local model parameters; a data calculation module, configured to perform the forward calculation on the to-be-processed data associated with the data processing request using the local model parameters, determine the target data according to the calculation result, and feed the target data back to the client.

[0011] According to a sixth aspect of the embodiments of this specification, a text processing device is provided, comprising: a parameter determination module configured to, in response to a text processing request corresponding to a target language model, determine, in a video memory in which the quantization model parameters of the target language model are loaded, local quantization model parameters corresponding to a forward calculation of the target language model, wherein the quantization model parameters are obtained by compressing the model parameters of the target language model; a parameter conversion module configured to convert the local quantization model parameters into local model parameters; and a forward calculation module configured to perform the forward calculation on the to-be-processed text associated with the text processing request using the local model parameters as a response to the text processing request.

[0012] According to the seventh aspect of the embodiments of this specification, a computing device is provided, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to implement any step of the above-mentioned data processing method or text processing method when executing the computer-executable instructions.

[0013] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which implement the steps of the above-mentioned data processing method or text processing method when executed by a processor.

[0014] According to a ninth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method or text processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a schematic diagram of a data processing method provided by one embodiment of this specification;

[0016] FIG2 is a flow chart of a data processing method provided by one embodiment of this specification;

[0017] FIG3 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0018] FIG4 is a flow chart of another data processing method provided by one embodiment of this specification;

[0019] FIG5 is a flowchart of a text processing method provided by one embodiment of this specification;

[0020] FIG6 is a schematic diagram of the structure of a data processing device provided by one embodiment of this specification;

[0021] FIG7 is a schematic structural diagram of another data processing device provided by an embodiment of this specification;

[0022] FIG8 is a schematic diagram of the structure of a text processing device provided by one embodiment of this specification;

[0023] FIG9 is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0024] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0025] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0026] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0027] First, the terms involved in one or more embodiments of this specification are explained.

[0028] LLM: (Large Language Model) is a probabilistic model that learns natural language patterns, using more computation to train larger models on more data, thereby achieving higher prediction accuracy.

[0029] Transformer model: A deep learning model that uses an attention mechanism to differentially weight the importance of each part of the input data and is widely used in various natural language processing tasks.

[0030] The BERT model (Bidirectional Encoder Representations from Transformer) is a bidirectional encoder representation based on the Transformer. The BERT model is based on the Transformer and is derived from the concept that attention is all you need. The bidirectional design means that when processing a word, it considers information about both preceding and following words to capture contextual semantics.

[0031] In this specification, a data processing method is provided. This specification also relates to a data processing device, a text processing method, a text processing device, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0032] In practical applications, in pursuit of model prediction accuracy, more computation is usually used to train larger models on more data. However, as the size of the model increases, a single graphics card cannot meet the needs of model prediction computation. Therefore, for models whose parameters exceed the memory size of a single graphics card, in order to implement reasoning for such models on fewer graphics cards, it is necessary to compress the model parameters before model inference, and load the compressed model parameters into the memory of a single card or as few graphics cards as possible to implement subsequent reasoning calculations. However, this compression method not only reduces the model prediction accuracy, but also generates additional operations. For example, feature extraction requires conversion to the set parameter format before it can be performed, and it also needs to be converted to the compressed parameter format for other processing. The consumption of computing resources increases significantly, so there is an urgent need for an effective solution to solve the above problems.

[0033] Referring to the schematic diagram shown in FIG1 , the data processing method provided in this specification can save more video memory resources and thus reduce communication loss between graphics cards. When the target model is deployed, the model parameters of the target model can be compressed into quantized model parameters and loaded into the video memory, thereby reducing the video memory resources occupied by the quantized model parameters compared to the model parameters. On this basis, in order to ensure the prediction accuracy of the target model and save more video memory resource consumption, in response to the data processing request corresponding to the target model, the local quantized model parameters required for the forward calculation of the target model can be determined in the video memory where the quantized model parameters are loaded, and then the local quantized model parameters are converted into local model parameters, thereby only converting a small amount of quantized model parameters for the forward calculation into local model parameters instead of converting all quantized model parameters, which can effectively avoid the video memory expansion problem in the data processing stage of the model; and on this basis, the local model parameters are used to perform forward calculation on the data to be processed corresponding to the request, thereby serving as a response to the data processing request; prediction calculation can be achieved in the compressed parameter format, thereby reducing the communication loss of the graphics card compared to the parameter conversion to the pre-compression format, effectively improving the utilization of video memory resources and reducing the communication requirements across graphics cards.

[0034] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] FIG2 shows a flow chart of a data processing method provided according to an embodiment of this specification, which specifically includes the following steps.

[0036] Step S202, in response to a data processing request corresponding to a target model, determining local quantization model parameters corresponding to the forward calculation of the target model in a video memory that loads the quantization model parameters of the target model, wherein the quantization model parameters are obtained by compressing the model parameters of the target model.

[0037] The target model in the data processing method provided in this embodiment specifically refers to a language model deployed according to actual service needs, such as a Transformer model, a model corresponding to a convolutional neural network architecture, a BERT model, etc. The predictive ability of the model can be trained according to actual service needs, such as for keyword extraction, text translation, abstract extraction, typo recognition, etc. This embodiment does not make any restrictions here. This embodiment describes the data processing method using a Chinese-English text translation model as an example. The data processing process of other types of target models can refer to the same or corresponding description in this embodiment, and this embodiment will not go into too much detail here.

[0038] Specifically, a data processing request refers to a request triggered when the target model performs calculation processing at any stage. The request is used to trigger the forward calculation of the model, that is, the calculation processing that needs to be performed in the next calculation cycle after the end of the previous calculation cycle, and the data that needs to be processed in the current calculation cycle comes from the previous calculation cycle; for example, the target model has two stages, encoding and decoding. After the calculation is completed in the encoding stage, the encoding vector will be obtained. If the current calculation cycle is a decoding cycle, the encoding vector will be loaded at this time as the data that needs to be processed in the current decoding cycle. The request triggered at this time is a decoding request.

[0039] Correspondingly, the local quantization model parameters specifically refer to a small number of quantization model parameters that need to be used in the calculation stage corresponding to the data processing request in the quantization model parameters, which are part of the quantization model parameters; accordingly, the forward calculation of the target model can be a calculation process that generates a feature map. For example, if the target model is a Transformer model, the forward calculation is the corresponding calculation when the encoder performs feature extraction in the Transformer model; for example, if the target model is a model with a convolutional neural network architecture, the forward calculation is the corresponding calculation when the convolution layer performs convolution processing in the convolution model; that is to say, the forward calculation can be the calculation involved in feature extraction in the target model. The calculation process involves the generation of feature maps. If forward calculation is performed based on the compressed quantized model parameters, after the feature maps are generated, the feature maps need to be converted into feature maps with the same parameter format as the quantized model parameters, thereby introducing additional calculations. At the same time, because more graphics card memory is used to complete the forward calculation, more communication losses will be generated. Therefore, in order to avoid introducing additional calculations and reduce communication losses, the local quantized model parameters involved in each calculation stage can be determined in the video memory before the forward calculation, and then these part of the quantized model parameters can be converted into local model parameters with the same parameter format as the model parameters, and then calculated. After each calculation stage is completed, the forward calculation of the target model can be completed.

[0040] In addition, it should be noted that during the model startup phase, a data processing request can be understood as a model startup request, that is, a request submitted for model startup data processing. In this case, the request carries the data to be processed and is submitted by a user who needs to use the target model. For example, if the target model is a text translation model, the data processing request is a text translation processing request carrying the corresponding source language text; if the target model is a summary extraction model, the data processing request is a summary extraction processing request carrying the text to be processed; if the target model is a typo recognition model, the data processing request is a typo recognition processing request carrying the text to be processed.

[0041] Based on this, after determining the data processing request corresponding to the target model, it is indicated that the target model needs to be used to request the corresponding prediction processing at this time, so as to obtain the required data for use or for subsequent prediction calculations. In order to save more video memory resources and reduce communication losses between graphics cards, the model parameters of the target model can be compressed into quantized model parameters when the target model is deployed and loaded into the video memory. This reduces the video memory resources occupied by the quantized model parameters compared to the model parameters.

[0042] On this basis, after determining the data processing request corresponding to the target model, in order to ensure the prediction accuracy of the target model and save more video memory resource consumption, the local quantization model parameters required for the forward calculation of the target model can be determined in the video memory that loads the quantization model parameters, and then the local quantization model parameters are converted into local model parameters, so that only a small number of quantization model parameters for forward calculation are converted into local model parameters, rather than converting all quantization model parameters. This can effectively avoid the problem of video memory expansion in the data processing stage of the model.

[0043] In other words, to enable inference calculations, the quantized model parameters obtained by compressing the model parameters of the target model are loaded into the video memory. Furthermore, considering that the forward calculation of the target model involves feature extraction, encoding, decoding, convolution, or fully connected processing, performing forward calculations directly on the basis of the quantized model parameters will not only affect accuracy but also generate additional computing resources. For example, converting the model parameters from FP16 precision to INT8 precision and then loading them into the video memory reduces the use of video memory resources. Subsequently, if INT8 matrix multiplication is performed, the feature map must also be quantized to INT8 precision. This introduces additional calculations and potential precision loss to convert the feature map to INT8. Furthermore, after the INT8 matrix multiplication, the feature map must be converted back to FP16 for subsequent calculations, further wasting computing resources.

[0044] Therefore, in order to avoid the introduction of additional resources during conversion and to avoid affecting the accuracy of inference calculations, in response to a data processing request, in the video memory that loads the quantized model parameters of the target model, only the local quantized model parameters that need to be calculated and used in the forward calculation can be determined, so as to facilitate the subsequent conversion of the local quantized model parameters into local model parameters, so as to complete the calculation corresponding to the stage of the data processing request in the parameter format corresponding to the model parameters. Each stage is completed in this way, which can avoid occupying more video memory resources and also avoid the need to introduce additional computing resources to convert the feature maps generated in the calculation into the parameter format corresponding to the quantized model parameters, so as to reduce the consumption of computing resources. Other local quantized model parameters are not needed in the calculation stage of the data processing request, so these quantized model parameters that are not needed do not need to be converted. When needed, the parameters that have been converted and no longer needed for calculation are released from the video memory, and new parameters are loaded for conversion and then used in the next calculation stage. In other words, each computational stage triggers a different data processing request. Once a request is triggered, only the local quantization model parameters corresponding to that request are determined and converted, while parameters not required for that stage are not converted, effectively utilizing graphics memory resources. This allows for dynamic determination and conversion of parameters required for the computational stage, allowing the forward computation of the target model to be completed once all stages are completed.

[0045] In summary, local model parameters are used to perform forward calculations on the data to be processed, thereby obtaining the target data as a response to the data processing request; predictive calculations can be implemented on the compressed format of the parameters, thereby reducing the communication loss of the graphics card compared to the conversion of the parameters to the pre-compression format, effectively improving the utilization of graphics memory resources, and reducing the communication requirements across graphics cards.

[0046] In addition, before deploying the target model, in order to save more video memory resources and thus reduce communication loss between graphics cards during the model inference phase, the model parameters can be compressed and quantized to reduce the proportion of video memory used. In this embodiment, the specific implementation method is as follows:

[0047] The model parameters of the target model are loaded; and the model parameters are compressed to obtain the quantized model parameters.

[0048] Specifically, model parameters refer to the parameters required by the target model during the inference phase. For example, if the target model is a Transformer model, the corresponding parameters include but are not limited to the parameters required by the encoder during the encoding phase and the parameters required by the decoder during the decoding phase.

[0049] In specific implementation, it can be understood as obtaining the target model and loading the model parameters of the target model, wherein the model parameters correspond to the first parameter format; compressing the model parameters in the first parameter format to obtain quantized model parameters corresponding to the second parameter format, wherein the video memory resources occupied by the quantized model parameters are less than the video memory resources occupied by the model parameters.

[0050] Among them, the first parameter format specifically refers to the format type corresponding to the model parameters of the target model after the training is completed, including but not limited to floating point numbers, integers, etc. For example, if the target model is in FP16 format, the model parameters of the model are all 16-bit floating point types; for example, if the target model is in INT8 format, the model parameters of the model are all 8-bit integer types. Correspondingly, the second parameter format specifically refers to the format type presented after the model parameters of the target model are compressed, including but not limited to floating point numbers, integers, etc., and it should be noted that the model parameters in the second parameter format must be smaller than the video memory bits occupied by the model parameters in the first parameter format, so as to save video memory resources. Correspondingly, the quantized model parameters specifically refer to the model parameters obtained after the model parameters are compressed, and the video memory resources occupied by the quantized model parameters are smaller than the video memory resources occupied by the model parameters.

[0051] Based on this, in order to reduce the excessive video memory resources occupied by model parameters in the prediction stage, the trained target model can be determined first, and then the model parameters of the target model can be loaded, where the model parameters correspond to the first parameter format; thereafter, the model parameters in the first parameter format can be compressed to obtain quantized model parameters corresponding to the second parameter format, and the video memory resources occupied by the quantized model parameters are less than the video memory resources occupied by the model parameters, so that the prediction processing can be completed using fewer video memory resources in the application stage.

[0052] In other words, for a trained target model, the model parameters in the model can be compressed using low-bit quantization. This allows the model parameters to be compressed into a numerical value type that occupies less space resources. For example, the low-bit value can be 8 bits, 4 bits, or any number of bits, which is not limited in this embodiment.

[0053] The above description is illustrated by taking the text translation model as the target model. Assume that the total number of parameters of the text translation model is n, and the parameter format of the model is FP16. At this time, it is determined that the video memory occupied by the n parameters is 350G, and 22 graphics cards are required to complete the prediction calculation. In order to reduce the consumption of video memory resources, the model parameters in FP16 format can be compressed to INT8 format, that is: the model parameters are quantized and compressed with 8 bits, and the quantized model parameters in INT8 format are obtained after compression. It is determined that the video memory occupied by the parameters in this format is 175G, and 11 graphics cards are required to complete the prediction calculation.

[0054] In summary, by compressing the model parameters into quantized model parameters that occupy less video memory resources, the consumption of video memory resources can be effectively saved.

[0055] Furthermore, when performing model parameter compression processing, in order to ensure a more reasonable compression ratio and to integrate the influence of multiple dimensions, the compression processing can be performed after determining the information of different dimensions based on the compression request. In this embodiment, the specific implementation is as follows:

[0056] Receive a compression request submitted for the target model; determine the model accuracy information and model size information of the target model based on the compression request, and read the hardware capacity information corresponding to the hardware on which the target model is deployed; generate a model compression strategy based on the model accuracy information, the model size information and the hardware capacity information; compress the model parameters according to the model compression strategy to obtain the quantized model parameters.

[0057] Specifically, the compression request specifically refers to a request triggered when compressing the model parameters of the target model. The request can be triggered based on a preset compression mechanism or by the user. This embodiment does not impose any restrictions on this. Accordingly, the model accuracy information specifically refers to the prediction accuracy requirement information when using the model. The larger the model compression ratio, the lower the accuracy; accordingly, the model size information specifically refers to the description information of the model size. The larger the model size, the larger the number of parameters; accordingly, the hardware capacity information specifically refers to the attribute information corresponding to the hardware on which the target model is deployed, such as the available video memory of the graphics card. Accordingly, the compression strategy specifically refers to a strategy generated by combining the information of the above three dimensions, which is used to comprehensively consider the information of each dimension mentioned above to generate a strategy with a reasonable compression ratio, so as to reduce the size of the video memory space occupied by the parameters while ensuring the prediction accuracy of the model.

[0058] Based on this, when a compression request is received for the target model, it means that the trained target model is too large and will occupy more video memory resources if deployed. Therefore, in order to reasonably compress the model parameters of the target model and reduce the usage of video memory resources while ensuring the model prediction accuracy, the model accuracy information and model size information of the target model can be determined according to the compression request, and the hardware capacity information corresponding to the hardware on which the target model is deployed can be read; the required information on the prediction accuracy of the model after deployment, the required information on the model volume, and the required information on the hardware on which the model can be deployed can be determined, and the model accuracy information, model size information and hardware capacity information of the three dimensions can be integrated to generate a model compression strategy; thereafter, the model parameters can be compressed according to the model compression strategy to obtain the corresponding quantized model parameters.

[0059] That is to say, considering that different bits will affect the model prediction accuracy when the model parameters are compressed, and will also affect the computational efficiency when subsequently converted back to the first parameter format accuracy, the compression ratio of the model parameters is determined to be related to the number of quantization bits. For example, if the model parameters are in FP16 format and quantized to INT8 format, using 8 bits can approximately compress the model by 50%; using 4 bits can approximately compress the model by 75%. On this basis, when compressing the model parameters of the target model, in addition to considering the compression ratio, factors such as the size of the model itself, the accuracy requirements, and the graphics card hardware can also be considered. The number of bits selected for compression can be comprehensively considered to obtain quantized model parameters that meet usage requirements and occupy fewer computing resources.

[0060] Continuing with the above example, after determining that the video memory occupied by the n parameters is 350G and 22 graphics cards are required to complete the prediction calculation, we can first load the model volume information, translation accuracy requirement information, and the video memory size information of each graphics card, and determine the number of bits to select 8-bit for compression. This can reduce the video memory occupancy rate by 50%, and the prediction accuracy can still be guaranteed. Therefore, the model parameters can be quantized and compressed with 8 bits. After compression, the quantized model parameters in the INT8 format are obtained. It is determined that the video memory occupied by the parameters in this format is 175G, and 11 graphics cards are required to complete the prediction calculation.

[0061] In summary, by combining information from different dimensions to generate a compression strategy for compressing model parameters, we can dynamically control the accuracy of the compressed model and the memory usage, and achieve compression of model parameters by comprehensively considering multiple factors. We can then obtain quantized model parameters for easy use in the inference phase of the target model.

[0062] After the data processing request corresponding to the target model is determined, the target model needs to be used for prediction processing, and the calculation cycle triggered by the data processing request may be any node. Before this, in order to complete the calculation using the compressed parameters, the data processing device group to be used can be determined in response to the request submitted by the user, and then the quantized model parameters of the model are loaded into the video memory of each data processing device in the data processing device group to facilitate subsequent calculation in the video memory. In this embodiment, the specific implementation method is as follows:

[0063] A data processing device group associated with the target model is determined in a data processing device cluster; and quantized model parameters of the target model are loaded into a video memory corresponding to a target data processing device included in the data processing device group.

[0064] Specifically, a data processing device cluster refers to a device cluster that can deploy a target model, such as a computer cluster equipped with a graphics card, which is used to store the quantized model parameters of the target model; correspondingly, a data device processing group refers to a collection of data processing devices used to store the quantized model parameters of the target model.

[0065] Based on this, after receiving the data processing request submitted by the user when starting the model, in order to complete the calculation in the inference stage, the data processing device group associated with the target model can be determined in the data processing device cluster according to the data processing request; it is used to determine the data processing device that needs to load the quantization model parameters, and then the target data processing device included in the data processing device group can be determined, and then the quantization model parameters of the target model can be loaded into the video memory corresponding to the target data processing device.

[0066] That is to say, during the inference phase of the target model, the compressed quantized model parameters can be loaded into the video memory. Assuming the compression ratio is 50%, during the inference phase, half the number of graphics cards before compression can be used to complete the loading of the quantized model parameters for inference use.

[0067] Continuing with the above example, after determining that the quantized model parameters in INT8 format occupy 175GB of video memory and require 11 graphics cards, we can first identify 11 computers C1 to C11 equipped with 11 graphics cards in the computer cluster equipped with graphics cards. After receiving a text translation request carrying the text to be translated {Hello, I am...}, we can load the quantized model parameters corresponding to the text translation model into the graphics cards corresponding to computers C1 to C11, respectively, to facilitate the completion of text translation processing during the inference phase using less video memory resources than before compression.

[0068] In summary, by loading the quantized model parameters into the corresponding data processing device, less video memory resources can be occupied compared to the model parameters before compression. Subsequent reasoning based on this can avoid consuming more video memory and at the same time reduce the communication loss between data processing devices, thereby improving reasoning efficiency and accuracy.

[0069] Furthermore, in order to reduce the extra calculations caused by the conversion, it is necessary to first determine the local quantization model parameters corresponding to the calculation stage of the data processing request in the video memory, and then convert them into local model parameters before processing. This can avoid the introduction of extra calculations and the reduction of prediction accuracy. In this embodiment, the specific implementation method is as follows:

[0070] In response to a data processing request corresponding to a target model, a feature calculation unit is determined in at least two data processing units included in the target model; in a video memory that loads the quantization model parameters of the target model, target quantization model parameters corresponding to the feature calculation unit are determined; and the target quantization model parameters corresponding to the feature calculation unit are used as local quantization model parameters corresponding to the forward calculation of the target model.

[0071] Specifically, the data processing unit refers to the unit corresponding to the different stages of processing in the target model. For example, if the target model is a Transformer model, it at least includes an encoding unit and a decoding unit. For another example, if the target model is a convolutional model, it at least includes an input unit, a convolutional unit, a deconvolutional unit, an output unit, etc. Correspondingly, the feature calculation unit is the unit in the target model that needs to perform forward calculations, such as the encoding unit in the Transformer model and the convolutional unit in the convolutional model. Correspondingly, the target quantization model parameters are the quantization model parameters involved in the feature calculation unit.

[0072] Based on this, in order to reduce the introduction of additional calculations, the current calculation stage can be determined in response to the data processing request corresponding to the target model, so as to select the feature calculation unit corresponding to the current stage from at least two data units contained in the target model according to the current calculation stage; it is determined to be the data processing unit corresponding to the forward calculation, and then the target quantization model parameters corresponding to the feature calculation unit can be determined in the video memory that loads the quantization model parameters of the target model; and the target quantization model parameters corresponding to the feature calculation unit can be used as the local quantization model parameters corresponding to the forward calculation of the target model, so that only the local quantization model parameters are converted in the subsequent conversion, and other quantization model parameters do not need to be converted, which can effectively avoid the introduction of additional calculations.

[0073] Continuing with the above example, 175GB of quantized model parameters are loaded into the video memory. When the text translation model is needed for text translation processing, the local quantized model parameters required for the forward calculation can be determined in the video memory first. The local quantized model parameters corresponding to the forward calculation are 100MB and correspond to the INT8 format, which facilitates the subsequent conversion of the 100MB local quantized model parameters into FP16 format for feature extraction processing.

[0074] In summary, by selecting the local quantization model parameters required for the calculation stage of the data processing request for subsequent conversion, the problem of not being able to store the video memory due to the conversion of all quantization model parameters can be avoided. In each calculation stage, only the local parameters required for calculation are converted, instead of directly converting all the parameters required for forward calculation, thereby effectively saving the consumption of computing resources and avoiding occupying more video memory resources.

[0075] Step S204: convert the local quantization model parameters into local model parameters.

[0076] Specifically, after determining the local quantization model parameters corresponding to the data processing request triggered by the current calculation stage from the video memory, in order to avoid additional calculations and to avoid occupying more video memory resources, the local quantization model parameters can be converted into local model parameters, so as to convert the compressed local quantization model parameters into the pre-compression local model parameters, so as to facilitate the direct use of the local model parameters to perform the calculation corresponding to the data processing request, thereby avoiding the additional calculation introduced by converting the feature map during the forward calculation process. The local model parameters are the model parameters obtained after decompressing the local quantization model parameters, and have the same model parameter format as the target model.

[0077] Furthermore, when converting local quantization model parameters into local model parameters, in order to enable use in subsequent forward calculations and prediction calculations without introducing additional computing resources for format conversion, this can be accomplished according to a set strategy. In this embodiment, the specific implementation is as follows:

[0078] Determine that the model parameter compression is a target model compression strategy corresponding to the quantization model parameter; decompress the local quantization model parameter according to the target model decompression strategy corresponding to the target model compression strategy to obtain the local model parameter; wherein the local model parameter has the same parameter format as the model parameter.

[0079] Specifically, the target model compression strategy refers to a strategy that combines information from different dimensions to generate a strategy with a reasonable compression ratio. This strategy reduces the amount of memory occupied by parameters while ensuring model prediction accuracy, and is used to compress the target model parameters into quantized model parameters. Correspondingly, the target model decompression strategy refers to a strategy that is the opposite of the target model compression strategy and can convert quantized model parameters into model parameters.

[0080] Based on this, in order to avoid introducing additional calculations, we can first determine the corresponding target model compression strategy when the model parameters are compressed into quantized model parameters; at this time, we can choose to take the reverse approach and generate the target model decompression strategy according to the target model compression strategy, and then decompress the local quantized model parameters according to the target model decompression strategy to obtain the local model parameters; and the parameter format of the local model parameters is the same as that of the model parameters.

[0081] In other words, when using the target model for inference, instead of performing inference based on quantized parameters, the quantized model parameters that have been noisyed in the video memory can be converted to the format corresponding to the model parameters as needed for forward inference calculations of the neural network model. This allows forward calculations to be performed at the accuracy of the format corresponding to the model parameters, without having to perform calculations on compressed low-bit data.

[0082] Continuing with the above example, after determining the 100MB local quantization model parameters, the 100MB local quantization model parameters in INT8 format can be converted to FP16 format in the video memory. This will result in 200MB local model parameters in FP16 format. This can then be used for the first calculation using the 200MB local model parameters in FP16 format, enabling fast and accurate text translation.

[0083] In summary, by converting local quantization model parameters into local model parameters, the parameters required for forward calculation can be restored to the format corresponding to the model parameters. Calculations are performed on this basis, which can avoid the problem of secondary conversion of feature maps generated during the calculation process, thereby avoiding the introduction of additional calculations and effectively saving the consumption of computing resources.

[0084] Step S206: performing the forward calculation on the to-be-processed data associated with the data processing request using the local model parameters as a response to the data processing request.

[0085] Specifically, after converting the local quantization model parameters into local model parameters, the model parameters currently stored in the video memory can be used for prediction processing. That is, the local model parameters in the video memory can be used to calculate the data to be processed in the data processing request, and after the calculation is completed, the next calculation is triggered. The data to be processed in the next calculation is the result of the previous calculation, and the next calculation will trigger a new data processing request, and the above processing process will be executed until all calculation stages are completed, that is, the forward calculation of the target model is completed, and then the quantization model parameters in the video memory are used to predict the forward calculation results, so as to obtain the final prediction result. Among them, the data to be processed specifically refers to the data required for each calculation stage, which comes from the result of the previous calculation stage, and when the data processing request is a request to start the target model, the data can be understood as the data that needs to be input into the target model for processing. For example, in the text translation scenario, the data to be processed is the text to be translated; for example, in the summary extraction scenario, the data to be processed is the original text.

[0086] Furthermore, when performing calculations, local model parameters are used for feature extraction, and quantized model parameters are used for prediction. In this embodiment, the specific implementation is as follows:

[0087] Determine the data to be processed associated with the data processing request; use the local model parameters corresponding to the target model in the video memory to perform the forward calculation on the data to be processed, obtain target data as a response to the data processing request.

[0088] Specifically, the target data refers to the data obtained after processing the data to be processed using the target model, and when the calculation stage of the data processing request is not the last calculation stage, the target data will be used as the data to be processed corresponding to the next calculation stage, and after triggering a new data processing request, subsequent calculation processing will be performed.

[0089] Based on this, after converting the local quantization model parameters into local model parameters, the data to be processed associated with the data processing request can be determined. At this time, the local model parameters corresponding to the target model in the video memory are used to perform forward calculations on the data to be processed, and the target data is obtained as a response to the data processing request.

[0090] Continuing with the above example, after the text to be translated {Hello, I am...} is input into the target model, the local model parameters in FP16 format of 200MB in the video memory can be used to extract features of the text to be translated {Hello, I am...}, thereby obtaining text features. Then, translation processing is performed based on the text features to obtain the target text {Hello, I am...} as a response to the text translation request, which can be sent to the client.

[0091] In summary, by combining local model parameters and quantitative model parameters to process the data to be processed, we can effectively avoid introducing additional calculations and ensure the prediction accuracy of the model to facilitate the use of downstream services.

[0092] In addition, in order to meet actual service needs, the target model needs to be fully trained. In this embodiment, the training of the target model includes:

[0093] Extract target samples from a preset training sample set and determine the sample labels corresponding to the target samples; input the target samples into the initial model for processing to obtain predicted data; use the predicted data and the sample labels to adjust the parameters of the initial model until the target model that meets the training stop conditions is obtained.

[0094] Specifically, the target sample refers to the sample used to train the initial model. Different scenarios correspond to different sample types. For example, in the text translation scenario, the target sample is the source language text, and the sample label is the target language text. For example, in the keyword extraction scenario, the target sample is the original text, and the sample label is the target keyword. Correspondingly, the predicted data refers to the result obtained after the target sample is predicted by the initial model. Correspondingly, the training stop condition refers to the condition for stopping the training of the initial model, including but not limited to the loss value comparison condition, the number of iterations condition, or the validation set verification condition. This embodiment does not impose any restrictions on this.

[0095] Based on this, when training the model, you can first extract the target samples from the preset training sample set and determine the sample labels corresponding to the target samples; then input the target samples into the initial model for processing to obtain predicted data; thereafter, use the predicted data and sample labels to adjust the parameters of the initial model. If the adjusted model does not meet the conditions, you can continue to select new samples from the training sample set to continue training until the target model that meets the training stop conditions is obtained.

[0096] The data processing method provided in this specification can save more video memory resources and thus reduce communication loss between graphics cards. When deploying the target model, the model parameters of the target model can be compressed into quantized model parameters and loaded into the video memory, thereby reducing the video memory resources occupied by the quantized model parameters compared to the model parameters. On this basis, in order to ensure the prediction accuracy of the target model and save more video memory resource consumption, in response to a data processing request corresponding to the target model, the local quantized model parameters required for the forward calculation of the target model can be determined in the video memory where the quantized model parameters are loaded, and then the local quantized model parameters are converted into local model parameters, thereby only converting a small number of quantized model parameters for the forward calculation into local model parameters instead of converting all quantized model parameters, which can effectively avoid the video memory expansion problem in the data processing stage of the model; and on this basis, the local model parameters are used to perform forward calculation on the data to be processed corresponding to the request, thereby serving as a response to the data processing request; prediction calculation can be achieved in the compressed parameter format, thereby reducing the communication loss of the graphics card compared to the conversion of the parameters into the uncompressed format, effectively improving the utilization of video memory resources and reducing the communication requirements across graphics cards.

[0097] The following further illustrates the data processing method provided in this specification using the application of the data processing method in a text processing scenario as an example, with reference to FIG3 . FIG3 shows a flow chart of a data processing method provided in one embodiment of this specification, specifically including the following steps.

[0098] Step S302: Load the model parameters of the target model and receive a compression request submitted for the target model.

[0099] Step S304: determining the model accuracy information and model size information of the target model according to the compression request, and reading the hardware capacity information corresponding to the hardware on which the target model is deployed.

[0100] Step S306: Generate a model compression strategy based on the model accuracy information, model size information, and hardware capacity information.

[0101] Step S308: compress the model parameters according to the model compression strategy to obtain quantized model parameters.

[0102] Step S310: determining a data processing device group associated with the target model in the data processing device cluster.

[0103] Step S312 : loading the quantized model parameters of the target model into the video memory corresponding to the target data processing device included in the data processing device group.

[0104] Step S314 , in response to a data processing request corresponding to the target model, determining local quantization model parameters corresponding to the forward calculation of the target model in the video memory that loads the quantization model parameters of the target model.

[0105] Step S316: Determine the target model compression strategy for compressing the model parameters to correspond to the quantized model parameters.

[0106] Step S318 : Decompress the local quantized model parameters according to the target model decompression strategy corresponding to the target model compression strategy to obtain local model parameters.

[0107] Step S320: Determine the data to be processed associated with the data processing request.

[0108] Step S322: using the local model parameters corresponding to the target model in the video memory, forward calculation is performed on the data to be processed to obtain the target data as a response to the data processing request.

[0109] The data processing method provided in this specification can save more video memory resources and thus reduce communication loss between graphics cards. When deploying the target model, the model parameters of the target model can be compressed into quantized model parameters and loaded into the video memory, thereby reducing the video memory resources occupied by the quantized model parameters compared to the model parameters. On this basis, in order to ensure the prediction accuracy of the target model and save more video memory resource consumption, in response to a data processing request corresponding to the target model, the local quantized model parameters required for the forward calculation of the target model can be determined in the video memory where the quantized model parameters are loaded, and then the local quantized model parameters are converted into local model parameters, thereby only converting a small number of quantized model parameters for the forward calculation into local model parameters instead of converting all quantized model parameters, which can effectively avoid the video memory expansion problem in the data processing stage of the model; and on this basis, the local model parameters are used to perform forward calculation on the data to be processed corresponding to the request, thereby serving as a response to the data processing request; prediction calculation can be achieved in the compressed parameter format, thereby reducing the communication loss of the graphics card compared to the conversion of the parameters into the uncompressed format, effectively improving the utilization of video memory resources and reducing the communication requirements across graphics cards.

[0110] Corresponding to the above method embodiment, this specification also provides another data processing method embodiment. Figure 4 shows a flow chart of another data processing method provided by one embodiment of this specification. As shown in Figure 4, the method is applied to the server and includes:

[0111] Step S402: selecting a target model according to the model selection request submitted by the client;

[0112] Step S404: In response to a data processing request corresponding to the target model, determining, in a video memory in which the quantized model parameters of the target model are loaded, local quantized model parameters corresponding to the forward calculation of the target model, wherein the quantized model parameters are obtained by compressing the model parameters of the target model;

[0113] Step S406, converting the local quantization model parameters into local model parameters;

[0114] Step S408: performing the forward calculation on the to-be-processed data associated with the data processing request using the local model parameters, determining target data according to the calculation result, and feeding back the target data to the client.

[0115] It should be noted that the data processing method provided in this embodiment corresponds to the data processing method provided in the above embodiment. Any content not described in detail in this embodiment can refer to the same or corresponding description in the above embodiment, and this embodiment will not be described in detail here.

[0116] Specifically, the server side refers to the end that provides model services. It deploys multiple target models that can provide services to users holding client devices. In response to data processing requests submitted by the client, it selects a target model for subsequent inference. The client side specifically refers to the terminal device held by the user, including but not limited to mobile phones, computers, wearable smart devices, etc.

[0117] Corresponding to the above method embodiment, this specification also provides a text processing method embodiment. FIG5 shows a flowchart of a text processing method provided by one embodiment of this specification. As shown in FIG5 , the method includes:

[0118] Step S502: In response to a text processing request corresponding to a target language model, determining, in a video memory in which quantized model parameters of the target language model are loaded, local quantized model parameters corresponding to a forward calculation of the target language model, wherein the quantized model parameters are obtained by compressing model parameters of the target language model;

[0119] Step S504, converting the local quantization model parameters into local model parameters;

[0120] Step S506: Using the local model parameters, perform the forward calculation on the to-be-processed text associated with the text processing request as a response to the text processing request.

[0121] It should be noted that the text processing method provided in this embodiment corresponds to the data processing method provided in the above embodiment. Any content not described in detail in this embodiment can refer to the same or corresponding description in the above embodiment, and this embodiment will not be elaborated here.

[0122] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment. FIG6 shows a schematic diagram of the structure of a data processing device provided in one embodiment of this specification. As shown in FIG6, the device includes:

[0123] a determination module 602 configured to, in response to a data processing request corresponding to a target model, determine, in a video memory in which quantized model parameters of the target model are loaded, local quantized model parameters corresponding to a forward calculation of the target model, wherein the quantized model parameters are obtained by compressing model parameters of the target model;

[0124] A conversion module 604 is configured to convert the local quantization model parameters into local model parameters;

[0125] The calculation module 606 is configured to perform the forward calculation on the to-be-processed data associated with the data processing request using the local model parameters as a response to the data processing request.

[0126] In an optional embodiment, the device further includes:

[0127] The model parameter acquisition module is configured to load the model parameters of the target model; compress the model parameters to obtain the quantized model parameters.

[0128] In an optional embodiment, the module for obtaining model parameters is further configured to:

[0129] Receive a compression request submitted for the target model; determine the model accuracy information and model size information of the target model based on the compression request, and read the hardware capacity information corresponding to the hardware on which the target model is deployed; generate a model compression strategy based on the model accuracy information, the model size information and the hardware capacity information; compress the model parameters according to the model compression strategy to obtain the quantized model parameters.

[0130] In an optional embodiment, the device further includes:

[0131] The device determination module is configured to determine a data processing device group associated with the target model in a data processing device cluster; and load the quantitative model parameters of the target model into the video memory corresponding to the target data processing device included in the data processing device group.

[0132] In an optional embodiment, the determining module 602 is further configured to:

[0133] In response to a data processing request corresponding to a target model, a feature calculation unit is determined in at least two data processing units included in the target model; in a video memory that loads the quantization model parameters of the target model, target quantization model parameters corresponding to the feature calculation unit are determined; and the target quantization model parameters corresponding to the feature calculation unit are used as local quantization model parameters corresponding to the forward calculation of the target model.

[0134] In an optional embodiment, the conversion module 604 is further configured to:

[0135] Determine that the model parameter compression is a target model compression strategy corresponding to the quantization model parameter; decompress the local quantization model parameter according to the target model decompression strategy corresponding to the target model compression strategy to obtain the local model parameter; wherein the local model parameter has the same parameter format as the model parameter.

[0136] In an optional embodiment, the calculation module 606 is further configured to:

[0137] Determine the data to be processed associated with the data processing request; use the local model parameters corresponding to the target model in the video memory to perform the forward calculation on the data to be processed, obtain target data as a response to the data processing request.

[0138] In an optional embodiment, the training of the target model includes:

[0139] Extract target samples from a preset training sample set and determine the sample labels corresponding to the target samples; input the target samples into the initial model for processing to obtain predicted data; use the predicted data and the sample labels to adjust the parameters of the initial model until the target model that meets the training stop conditions is obtained.

[0140] The data processing device provided in this specification can, in order to save more video memory resources and thus reduce communication loss between graphics cards, compress the model parameters of the target model into quantized model parameters when deploying the target model and load them into the video memory, thereby reducing the video memory resources occupied by the quantized model parameters compared to the model parameters. On this basis, in order to ensure the prediction accuracy of the target model and save more video memory resource consumption, in response to a data processing request corresponding to the target model, first determine the local quantized model parameters required for the forward calculation of the target model in the video memory where the quantized model parameters are loaded, and then convert the local quantized model parameters into local model parameters, thereby converting only a small number of quantized model parameters for the forward calculation into local model parameters instead of converting all quantized model parameters, which can effectively avoid the problem of video memory expansion in the model processing data stage; and on this basis, use the local model parameters to perform forward calculation on the data to be processed corresponding to the request, thereby serving as a response to the data processing request; predictive calculation can be achieved in the compressed parameter format, thereby reducing the communication loss of the graphics card compared to the conversion of the parameters into the uncompressed format, effectively improving the utilization of video memory resources and reducing the communication requirements across graphics cards.

[0141] The above is a schematic diagram of a data processing device according to this embodiment. It should be noted that the technical solution of the data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.

[0142] Corresponding to the above method embodiment, this specification also provides another data processing device embodiment. FIG7 shows a schematic diagram of the structure of another data processing device provided in one embodiment of this specification. As shown in FIG7, the device includes:

[0143] The model selection module 702 is configured to select a target model according to a model selection request submitted by a client;

[0144] a parameter determination module 704 configured to, in response to a data processing request corresponding to the target model, determine, in a video memory in which quantized model parameters of the target model are loaded, local quantized model parameters corresponding to the forward calculation of the target model, wherein the quantized model parameters are obtained by compressing the model parameters of the target model;

[0145] a parameter conversion module 706 configured to convert the local quantization model parameters into local model parameters;

[0146] The data calculation module 708 is configured to perform the forward calculation on the to-be-processed data associated with the data processing request using the local model parameters, determine target data according to the calculation result, and feed the target data back to the client.

[0147] The above is a schematic diagram of another data processing device of this embodiment. It should be noted that the technical solution of this data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.

[0148] Corresponding to the above method embodiment, this specification also provides a text processing device embodiment. FIG8 shows a schematic diagram of the structure of a text processing device provided by one embodiment of this specification. As shown in FIG8, the device includes:

[0149] a parameter determination module 802 configured to, in response to a text processing request corresponding to a target language model, determine, in a video memory loaded with quantization model parameters of the target language model, local quantization model parameters corresponding to a forward calculation of the target language model, wherein the quantization model parameters are obtained by compressing model parameters of the target language model;

[0150] A parameter conversion module 804 is configured to convert the local quantization model parameters into local model parameters;

[0151] The forward calculation module 806 is configured to perform the forward calculation on the to-be-processed text associated with the text processing request using the local model parameters as a response to the text processing request.

[0152] The above is a schematic diagram of a text processing device according to this embodiment. It should be noted that the technical solution of the text processing device and the technical solution of the above-mentioned text processing method are based on the same concept. For details not described in detail in the technical solution of the text processing device, please refer to the description of the technical solution of the above-mentioned text processing method.

[0153] Figure 9 shows a block diagram of a computing device 900 according to one embodiment of this specification. Components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0154] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0155] In one embodiment of the present application, the aforementioned components of the computing device 900 and other components not shown in FIG9 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG9 is for illustrative purposes only and does not limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0156] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0157] The processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned data processing method or text processing method.

[0158] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the aforementioned data processing method or text processing method are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned data processing method or text processing method.

[0159] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned data processing method or text processing method.

[0160] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the aforementioned data processing method or text processing method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned data processing method or text processing method.

[0161] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method or text processing method.

[0162] The above is an illustrative solution of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned data processing method or text processing method are based on the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the aforementioned data processing method or text processing method.

[0163] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0165] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0166] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0167] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: In response to a data processing request corresponding to a target model, determining, in a video memory in which quantized model parameters of the target model are loaded, local quantized model parameters corresponding to forward calculation of the target model, wherein the quantized model parameters are obtained by compressing model parameters of the target model; Converting the local quantization model parameters into local model parameters; The forward calculation is performed on the to-be-processed data associated with the data processing request using the local model parameters as a response to the data processing request.

2. The method according to claim 1, wherein in response to a data processing request corresponding to a target model, before the step of determining, in a video memory for loading the quantization model parameters of the target model, the local quantization model parameters corresponding to the forward calculation of the target model is executed, further comprising: Loading model parameters of the target model; The model parameters are compressed to obtain the quantized model parameters.

3. The method according to claim 2, wherein compressing the model parameters to obtain the quantized model parameters comprises: receiving a compression request submitted for the target model; Determining model accuracy information and model size information of the target model according to the compression request, and reading hardware capacity information corresponding to hardware on which the target model is deployed; Generate a model compression strategy according to the model accuracy information, the model size information and the hardware capacity information; The model parameters are compressed according to the model compression strategy to obtain the quantized model parameters.

4. The method according to claim 1, further comprising: Determining a data processing device group associated with the target model in a data processing device cluster; The quantized model parameters of the target model are loaded into a video memory corresponding to a target data processing device included in the data processing device group.

5. The method according to claim 1, wherein in response to a data processing request corresponding to a target model, determining, in a video memory in which quantization model parameters of the target model are loaded, local quantization model parameters corresponding to forward calculation of the target model comprises: In response to a data processing request corresponding to the target model, determining a feature calculation unit from at least two data processing units included in the target model; Determine the target quantization model parameters corresponding to the feature calculation unit in the video memory in which the quantization model parameters of the target model are loaded; The target quantization model parameters corresponding to the feature calculation unit are used as the local quantization model parameters corresponding to the forward calculation of the target model.

6. The method according to claim 1, wherein converting the local quantization model parameters into local model parameters comprises: Determine that the model parameter compression is a target model compression strategy corresponding to the quantized model parameter; Decompressing the local quantization model parameters according to the target model decompression strategy corresponding to the target model compression strategy to obtain the local model parameters; The local model parameters have the same parameter format as the model parameters.

7. The method according to any one of claims 1 to 6, wherein the forward calculation of the to-be-processed data associated with the data processing request using the local model parameters as a response to the data processing request comprises: Determining the data to be processed associated with the data processing request; The local model parameters corresponding to the target model in the video memory are used to perform the forward calculation on the data to be processed, and the target data is obtained as a response to the data processing request.

8. According to the method according to any one of claims 1 to 6, the training of the target model comprises: Extracting target samples from a preset training sample set and determining sample labels corresponding to the target samples; Inputting the target sample into the initial model for processing to obtain prediction data; The initial model is adjusted using the predicted data and the sample labels until the target model that meets the training stop condition is obtained.

9. A data processing method, applied to a server, comprising: Select a target model according to a model selection request submitted by a client; In response to a data processing request corresponding to the target model, determining, in a video memory in which quantized model parameters of the target model are loaded, local quantized model parameters corresponding to forward calculation of the target model, wherein the quantized model parameters are obtained by compressing model parameters of the target model; Converting the local quantization model parameters into local model parameters; The forward calculation is performed on the to-be-processed data associated with the data processing request using the local model parameters, target data is determined according to the calculation result, and the target data is fed back to the client.

10. A text processing method, comprising: In response to a text processing request corresponding to a target language model, determining, in a video memory in which quantization model parameters of the target language model are loaded, local quantization model parameters corresponding to a forward calculation of the target language model, wherein the quantization model parameters are obtained by compressing model parameters of the target language model; Converting the local quantization model parameters into local model parameters; The local model parameters are used to perform the forward calculation on the to-be-processed text associated with the text processing request, A response to the request for said text processing.

11. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Neural network model compression method and device

    CN106485316A

  • Neural network difference compression method and device, electronic equipment and storage medium

    CN114418098A

  • Data processing method and device

    CN116702798A

  • Method and electronic device for quantizing DNN model

    US20230068381A1