Model processing method, device and device for model processing

By processing the self-attention mechanism of business data and distilling knowledge, the problem of mismatch between the pre-trained model and the initial business model modeling unit is solved, the performance of the business model is improved, and the business model is optimized.

CN114154395BActive Publication Date: 2024-11-08BEIJING SOGOU TECHNOLOGY DEVELOPMENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111301382.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-04
Publication Date
2024-11-08
Estimated Expiration
2041-11-04

AI Technical Summary

Technical Problem

In some scenarios, pre-trained models cannot be used to optimize the business model, which makes the business model unable to improve the effectiveness.

Method used

By obtaining business data and inputting it into the pre-trained model and the initial business model respectively, the data is processed using the self-attention mechanism, matching the modeling units of the pre-trained model and the initial business model, and knowledge distillation is performed to obtain the target business model .

Benefits of technology

When the modeling units of the pretrained model and the initial business model are different, the optimization of the pretrained model and the initial business model is achieved through knowledge distillation, which improves the performance of the business model and avoids the loss of information during knowledge distillation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114154395B_ABST
    Figure CN114154395B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention provide a model processing method, device and device for model processing. The method includes: obtaining business data; inputting the business data into a pre-trained model and an initial business model respectively, processing the business data through a self-attention mechanism, and obtaining the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; matching the modeling unit of the pre-trained model with the modeling unit of the initial business model, and determining the target characters in the modeling unit of the pre-trained model; performing knowledge distillation on the pre-trained model and the initial business model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target characters, and obtaining a target business model. The embodiments of the present invention can avoid the loss of information during the knowledge distillation process and improve the model performance of the business model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model processing method, device and a device for model processing. Background Art

[0002] Currently, pre-trained models are developing rapidly. Pre-training can use self-supervised learning to generate task-independent pre-trained models from large amounts of data. Furthermore, pre-trained models can transfer the knowledge learned from large amounts of data to specific business applications. In other words, using pre-trained models to optimize business models can lead to even better performance.

[0003] However, in some scenarios, pre-trained models cannot be used to optimize business models, which makes it impossible to improve the effectiveness of the business models. Summary of the Invention

[0004] The embodiments of the present invention provide a model processing method, device and device for model processing, which can use a pre-trained model to optimize the initial business model and improve the model performance of the business model.

[0005] In order to solve the above problems, an embodiment of the present invention discloses a model processing method, which includes:

[0006] Obtain business data;

[0007] Input the business data into the pre-trained model and the initial business model respectively, process the business data through the self-attention mechanism, and obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation, one modeling unit of the pre-trained model corresponds to one fully connected layer output, and one modeling unit of the initial business model corresponds to one fully connected layer output;

[0008] Matching the modeling units of the pre-training model with the modeling units of the initial business model to determine the target characters in the modeling units of the pre-training model;

[0009] According to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character, knowledge distillation is performed on the pre-trained model and the initial business model to obtain a target business model.

[0010] In another aspect, an embodiment of the present invention discloses a model processing device, comprising:

[0011] Business data acquisition module, used to obtain business data;

[0012] A business data processing module, configured to input the business data into the pre-trained model and the initial business model respectively, process the business data through a self-attention mechanism, and obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation; one modeling unit of the pre-trained model corresponds to one fully connected layer output, and one modeling unit of the initial business model corresponds to one fully connected layer output;

[0013] A modeling unit matching module, configured to match the modeling units of the pre-trained model with the modeling units of the initial business model, and determine target characters in the modeling units of the pre-trained model;

[0014] The knowledge distillation module is used to perform knowledge distillation on the pre-trained model and the initial business model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character to obtain a target business model.

[0015] On the other hand, an embodiment of the present invention discloses a device for model processing, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for executing one or more model processing methods as described above.

[0016] In yet another aspect, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes a device to perform one or more of the model processing methods described above.

[0017] An embodiment of the present invention also discloses a computer program product, which includes computer instructions stored in a computer-readable storage medium and suitable for being read and executed by a processor, so that a computer device having the processor executes one or more of the model processing methods described above.

[0018] The embodiments of the present invention include the following advantages:

[0019] After acquiring the business data, the embodiment of the present invention inputs the business data into the pre-trained model and the initial business model respectively, processes the business data through the self-attention mechanism, and obtains the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model. The modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output. In order to perform knowledge distillation on the pre-trained model and the initial business model, the embodiment of the present invention matches the modeling unit of the pre-trained model with the modeling unit of the initial business model, determines the target character in the modeling unit of the pre-trained model, and then performs knowledge distillation on the pre-trained model and the initial business model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character to obtain a target business model.

[0020] In an embodiment of the present invention, when the modeling units of the pre-trained model and the initial business model are different, the modeling units of the pre-trained model can be matched with the modeling units of the initial business model to achieve knowledge distillation of the pre-trained model and the initial business model, thereby obtaining an optimized target business model and improving business performance. In addition, before matching the fully connected layer output of the pre-trained model with the fully connected layer output of the initial business model, the embodiment of the present invention first processes all the modeling units of each model based on the self-attention mechanism of the two models, so that each fully connected layer output refers to all the modeling units of the model, thereby avoiding the loss of information during the knowledge distillation process, which is conducive to further improving the model performance of the target business model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 is a flowchart of steps of an embodiment of a model processing method of the present invention;

[0023] Figure 2 It is a schematic diagram of a model processing flow of the present invention;

[0024] Figure 3 is a structural block diagram of an embodiment of a model processing device of the present invention;

[0025] Figure 4 is a block diagram of an apparatus 800 for model processing according to the present invention;

[0026] Figure 5 It is a schematic diagram of the structure of the server in some embodiments of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] Method Example

[0029] Reference Figure 1 , shows a flowchart of a model processing method embodiment of the present invention, the method may specifically include the following steps:

[0030] Step 101: Obtain business data.

[0031] Step 102: Input the business data into the pre-trained model and the initial business model respectively, process the business data through the self-attention mechanism, and obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein, the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output.

[0032] Step 103: Match the modeling unit of the pre-training model with the modeling unit of the initial business model to determine the target characters in the modeling unit of the pre-training model.

[0033] Step 104: Perform knowledge distillation on the pre-trained model and the initial business model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character to obtain a target business model.

[0034] It should be noted that, in an embodiment of the present invention, the pre-trained model and the initial business model may be obtained by pre-training. Exemplarily, the pre-trained model may be obtained by training based on pre-training data and business data. Specifically: the initial pre-trained model may be obtained by training using pre-training data, and then the initial pre-trained model may be fine-tuned (Finetune) using business data to obtain a business-based pre-trained model. The pre-trained data may be training data that is not related to the business, and the pre-trained model may include but is not limited to BERT (Bidirectional Encoder Representations from Transformers, deep language representation model), GPT (Generative Pre-Training, generative pre-training) model and ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately, efficiently learning an encoder that accurately classifies token replacements).

[0035] In embodiments of the present invention, the initial business model can be trained using business data. It is understandable that the amount of business data cannot be expanded indefinitely, especially for emerging businesses, where data is extremely limited. Therefore, the accuracy of the initial business model is often not particularly high. However, optimizing the initial business model using a pre-trained model through knowledge distillation can effectively improve the accuracy of the optimized target business model without requiring additional business data for model training.

[0036] A prerequisite for using knowledge distillation is that the sequence length of the fully connected layer output of the pre-trained model is consistent with the sequence length of the fully connected layer output of the initial business model. This is because when performing knowledge distillation, the relative entropy divergence loss (KLD loss) is calculated based on the fully connected layer output of the pre-trained model and the fully connected layer output of the business model. For example: the fully connected layer output of the pre-trained model includes three vectors, namely w1, w2, and w3, and the fully connected layer output of the initial business model includes three vectors, namely L1, L2, and L3. Then KLD loss = w1*L1+w2*L2+w3*L3.

[0037] In some scenarios, such as automatically adding punctuation to text, the modeling unit of the initial business model is a word segmentation, while the modeling unit of the pre-trained model is a single character. A word segmentation can consist of one or more characters, so one modeling unit in the initial business model may correspond to multiple modeling units in the pre-trained model. Since one modeling unit corresponds to one fully-connected layer output, this results in a mismatch between the sequence lengths of the fully-connected layer outputs of the business model and those of the pre-trained model, making knowledge distillation impossible. For example, for the text "I am Chinese," the pre-trained model input includes five modeling units: "I," "I am," "China," "China," and "People," while the initial business model input includes three modeling units: "I," "I am," and "Chinese." Assuming that the fully-connected layer output corresponding to each modeling unit is a 5-dimensional vector, the pre-trained model's fully-connected layer outputs five 5-dimensional vectors, while the business model's fully-connected layer outputs three 5-dimensional vectors. Clearly, the sequence lengths of the fully-connected layer outputs of the initial business model and the pre-trained model are inconsistent.

[0038] Since it is unlikely that each modeling unit of the initial business model includes only one character, in most cases, the sequence length of the fully connected layer output of the pre-trained model is greater than the sequence length of the fully connected layer output of the initial business model. Therefore, knowledge distillation cannot be performed using the fully connected layer outputs of the pre-trained model and the fully connected layer outputs of the initial business model. In view of this, an embodiment of the present invention provides a model processing method that introduces a self-attention mechanism to soft-align the fully connected layer outputs of the pre-trained model and the fully connected layer outputs of the initial business model, thereby obtaining a target business model through knowledge distillation.

[0039] Reference Figure 2 , shows a schematic diagram of a model processing flow provided by an embodiment of the present invention. Figure 2 The specific implementation steps of the model processing method provided by the present invention are described.

[0040] like Figure 2 As shown in the figure, the business data is input into the pre-trained model and the initial business model respectively. The pre-trained model maps the characters in the business data to obtain character vectors c1 to c5. The pre-trained model then processes the character vectors c1 to c5 based on the self-attention mechanism to obtain fully connected layer outputs S1 to S5. Each fully connected layer output of the pre-trained model is obtained based on the character vectors c1 to c5.

[0041] The initial business model maps the word segments in the business data to obtain word segmentation vectors w1 to w3. Then, the initial business model processes the word segmentation vectors w1 to w3 based on the self-attention mechanism to obtain fully connected layer outputs L1 to L3. Each fully connected layer output of the initial business model is obtained based on the word segmentation vectors w1 to w3.

[0042] It should be noted that in the embodiment of the present invention, the fully connected layer outputs of the pre-trained model and the initial business model are unnormalized label probability value vectors. For example, in the scenario of adding punctuation to text, there are four optional punctuation marks, and the fully connected layer output corresponding to each modeling unit is a 5-dimensional vector. The specific values ​​of the 5-dimensional vector are used to indicate the probabilities of the four punctuation marks and the probability of no punctuation marks. The fully connected layer output can also be called Logits output.

[0043] Assume that the input business data is: "I am Chinese", Figure 2 The three word segmentation vectors of the initial business model correspond to "I," "Is," and "Chinese," respectively. Therefore, the fully connected layer outputs of the initial business model are: L1 corresponds to the label probability value vector for "I," L2 corresponds to the label probability value vector for "Is," and L3 corresponds to the label probability value vector for "Chinese." The five character vectors of the pre-trained model correspond to "I," "Is," "China," "Country," and "People," respectively. Therefore, the five fully connected layers of the pre-trained model output label probability value vectors corresponding to "I," "Is," "China," "Country," and "People," respectively.

[0044] To perform knowledge distillation on the pre-trained model and the initial business model, it is necessary to find the fully connected layer outputs in the pre-trained model that match the fully connected layer outputs of the initial business model, and calculate KLD loss and CEloss (Cross Entropy Loss).

[0045] In an embodiment of the present invention, target characters that match each modeling unit of the initial business model can be determined from the modeling units of the pre-trained model, and then knowledge distillation can be performed using the fully connected layer output of the initial business model and the fully connected layer output of the target characters that match each modeling unit of the initial business model in the modeling units of the pre-trained model.

[0046] For example, Figure 2As shown, the target character "I" in the pre-trained model is determined to correspond to the modeling unit "I" in the initial business model; the target character "I" in the pre-trained model is determined to correspond to the modeling unit "I" in the initial business model; and the target character "Ren" in the pre-trained model is determined to correspond to the modeling unit "Zhongren" in the initial business model. Then, knowledge distillation is performed using L1, L2, L3 and S1, S2, and S5 to obtain the target business model.

[0047] Depend on Figure 2 It can be seen that when performing knowledge distillation on the pre-trained model and the initial business model, after alignment processing, when calculating the KLD loss, only S1, S2, and S5 in the fully connected layer output of the pre-trained model are actually referenced, and the fully connected layer outputs S3 and S4 are discarded. If the fully connected output of the pre-trained model is obtained only by the corresponding word segmentation vector, that is, S1 is obtained only by c1, S2 is obtained only by c2, and so on, S5 is obtained only by c5, then in the process of knowledge distillation, the information of the word segmentation vectors c3 and c4 will be lost, affecting the model performance of the target business model finally obtained.

[0048] Therefore, in an embodiment of the present invention, before matching the fully connected layer output of the pre-trained model with the fully connected layer output of the initial business model, all modeling units of each model are processed separately based on the self-attention mechanism of the two models, so that each fully connected layer output refers to all the modeling units of the model, thereby avoiding the loss of word segmentation information during the knowledge distillation process, which is beneficial to improving the model performance of the target business model.

[0049] The initial business model and the target business model can be used to process related businesses. In one example, the initial business model and the target business model can be used to add punctuation to text. Considering that text data automatically recognized by speech does not include punctuation, in some embodiments, the text data can be text data automatically recognized by speech.

[0050] In an optional embodiment of the present invention, step 102 includes inputting the business data into a pre-trained model and an initial business model, processing the business data through a self-attention mechanism, and obtaining a fully connected layer output of the pre-trained model and a fully connected layer output of the initial business model, including:

[0051] Step S11: input the business data into the self-attention network layer of the pre-trained model for processing to obtain a first hidden layer vector corresponding to the business data;

[0052] Step S12: inputting the first hidden layer vector into the fully connected layer of the pre-trained model for calculation to obtain the fully connected layer output of the pre-trained model;

[0053] Step S13: Input the service data into the self-attention network layer of the initial service model for processing to obtain the second hidden layer vector corresponding to the service data;

[0054] Step S14: Input the second hidden layer vector into the fully connected layer of the initial service model for calculation to obtain the output of the fully connected layer of the pre-trained model.

[0055] In an embodiment of the present invention, both the pre-trained model and the initial service model may include a self-attention network layer and a fully connected layer. For the input service data, the two models respectively map the service data to obtain the encoding vectors corresponding to their respective modeling units. Among them, the pre-trained model maps the service data to obtain character vectors, and the initial service model maps the service data to obtain word segmentation vectors. Then, based on its own self-attention network layer, the encoding vectors corresponding to the modeling units are processed to obtain hidden layer vectors, and then the hidden layer vectors are input into its own fully connected layer for calculation to obtain the output of the fully connected layer.

[0056] In an optional embodiment of the present invention, step 103 of matching the modeling unit of the pre-trained model with the modeling unit of the initial service model to determine the target character in the modeling unit of the pre-trained model includes:

[0057] If the first character set forms a first word segmentation, determine the last character of the first character set as the target character, where the first character set includes multiple characters, and all of the multiple characters are modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial service model.

[0058] In an embodiment of the present invention, for the case where the initial service model and the target service model can be used to add symbols to the text, considering that for a word segmentation, it is impossible to add punctuation marks inside the word segmentation, and punctuation marks may be added after the last character of the word segmentation. For example, for the word segmentation "Chinese people", it is impossible to add punctuation marks between the character "中" and the character "国"; it is also impossible to add punctuation marks between the character "国" and the character "人". However, it is possible to add punctuation marks after the character "人". Therefore, in the specific implementation of step 103, it may be: match each modeling unit of the initial service model with each modeling unit of the pre-trained model. If multiple modeling units of the pre-trained model form a certain modeling unit of the initial service model, then determine the last modeling unit among the multiple modeling units as the character matching the aforementioned "certain modeling unit".

[0059] For convenience of description, for example, the "certain modeling unit" is referred to as the "first word segment", and the "multiple modeling units of the pre-trained model" are referred to as the "first character set". Then: If the first character set constitutes the first word segment, the last character of the first character set is determined as the target character matching the first word segment. For example: The first character set is {"中", "国", "人"}, and the first word segment is "中国人", then the last character "人" of the first character set is determined as the target character matching the first word segment.

[0060] Based on the target characters in the modeling units of the pre-trained model, sequences with the same length as the sequences output by the fully connected layer of the initial business model can be filtered out from the sequences output by the fully connected layer of the pre-trained model, so as to perform knowledge distillation, use the pre-trained model to optimize the initial business model, obtain the target business model, and improve the business performance.

[0061] In an optional embodiment of the present invention, the service data includes the text data to be processed and the punctuation symbol labels corresponding to the text data; step 104 of performing knowledge distillation on the pre-trained model and the initial business model according to the output of the fully connected layer of the initial business model and the output of the fully connected layer corresponding to the target character to obtain the target business model includes:

[0062] Step S21: Calculate the first loss value of the pre-trained model according to the output of the fully connected layer of the initial business model and the output of the fully connected layer corresponding to the target character;

[0063] Step S22: Calculate the second loss value of the initial business model according to the output of the fully connected layer of the initial business model and the punctuation symbol labels corresponding to the text data;

[0064] Step S23: Perform weighted summation on the first loss value and the second loss value to obtain the joint loss value of the pre-trained model and the initial business model;

[0065] Step S24: Perform knowledge distillation on the pre-trained model and the initial business model according to the joint loss value to obtain the target business model.

[0066] Among them, the first loss value may be the relative entropy of the output of the fully connected layer of the initial business model and the output of the fully connected layer corresponding to the target character, that is, the KLD loss. For Figure 2 example, assuming that in the pre-trained model, the outputs of the fully connected layer corresponding to the target characters are S1, S2, and S5 respectively, the first loss value can be calculated according to the following formula:

[0067]

[0068] Among them, L(x) represents the fully connected layer outputs L1 to L3 of the initial business model, and S(x) represents the fully connected layer outputs S1, S2 and S5 corresponding to the target character in the pre-training model.

[0069] The second loss value may be the cross entropy between the fully connected layer output of the initial business model and the punctuation mark label corresponding to the text data, namely, CE-loss. Figure 2 For example, assuming that in the initial business model, the fully connected layer outputs corresponding to the initial business model are L1 to L3, the second loss value can be calculated according to the following formula:

[0070] CE=∑ x∈X P(x)logL(x) (2)

[0071] Among them, P(x) represents the punctuation mark label corresponding to the text data, and L(x) represents the fully connected layer outputs L1 to L3 of the initial business model.

[0072] After respectively calculating the first loss value and the second loss value, a weighted sum is performed on the first loss value and the second loss value to obtain a joint loss value of the pre-trained model and the initial business model, and knowledge distillation is performed based on the joint loss value to obtain a target business model.

[0073] It should be noted that the weighting coefficients of the first loss value and the second loss value can be set according to actual needs. For example, the weighting coefficients of the first loss value and the second loss value can both be set to 0.5. This embodiment of the present invention does not make any specific limitations on this.

[0074] In an optional embodiment of the present invention, after obtaining the target business model, the target business model can be used to process related business. For example, the target business model can be used to add punctuation marks to text, and the method further includes:

[0075] Step S31: Acquire voice data;

[0076] Step S32: performing speech recognition processing on the speech data to obtain text data corresponding to the speech data;

[0077] Step S33: using the target service model, adding punctuation marks to the text data corresponding to the voice data;

[0078] Step S34: output the text data after adding punctuation marks.

[0079] The voice data may be data input by the user in real time through a microphone, or data input and stored by the user in advance, which is not limited here.

[0080] In the embodiment of the present invention, speech recognition technology can be used to identify the specific content of the speech data, thereby obtaining the text corresponding to the speech data. The speech recognition technology will not be described in detail here.

[0081] Since the text obtained by speech recognition does not include punctuation marks, after obtaining the text data corresponding to the speech data, the target business model can be used to automatically add punctuation marks to the text data corresponding to the speech data, and further output the text data after the punctuation marks are added.

[0082] For example, in an instant messaging scenario, if it is inconvenient for the user to manually enter text, they can use a microphone to record voice data. Then, the device installed with the instant messaging software can receive the voice data recorded by the user, further recognize the voice data, and use the target business model to add punctuation marks to the text data corresponding to the voice data. Then, the text data with punctuation marks is entered in the input area of ​​the instant messaging page.

[0083] In summary, the embodiment of the present invention can achieve knowledge distillation of the pre-trained model and the initial business model by matching the modeling units of the pre-trained model with the modeling units of the initial business model when the modeling units of the pre-trained model and the initial business model are different, thereby obtaining an optimized target business model and improving business performance. In addition, before matching the fully connected layer output of the pre-trained model with the fully connected layer output of the initial business model, the embodiment of the present invention first processes the respective modeling units based on the self-attention mechanism of the two models, so that each fully connected layer output refers to all the modeling units of the model, thereby avoiding the loss of word segmentation information during the knowledge distillation process, which is conducive to further improving the model performance of the target business model.

[0084] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0085] Device embodiment

[0086] Reference Figure 3 , shows a structural block diagram of an embodiment of a model processing device of the present invention, the device may include:

[0087] Business data acquisition module 301, used to acquire business data;

[0088] The business data processing module 302 is used to input the business data into the pre-trained model and the initial business model respectively, and process the business data through the self-attention mechanism to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation. One modeling unit of the pre-trained model corresponds to one fully connected layer output, and one modeling unit of the initial business model corresponds to one fully connected layer output;

[0089] A modeling unit matching module 303 is used to match the modeling unit of the pre-training model with the modeling unit of the initial business model to determine the target character in the modeling unit of the pre-training model;

[0090] The knowledge distillation module 304 is used to perform knowledge distillation on the pre-trained model and the initial business model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character to obtain a target business model.

[0091] Optionally, the business data processing module includes:

[0092] A first hidden layer vector determination submodule is used to input the business data into the self-attention network layer of the pre-trained model for processing to obtain a first hidden layer vector corresponding to the business data;

[0093] A first fully connected layer output submodule, configured to input the first hidden layer vector into the fully connected layer of the pre-trained model for calculation to obtain the fully connected layer output of the pre-trained model;

[0094] A second hidden layer vector determination submodule is used to input the business data into the self-attention network layer of the initial business model for processing to obtain a second hidden layer vector corresponding to the business data;

[0095] The second fully connected layer output submodule is used to input the second hidden layer vector into the fully connected layer of the initial business model for calculation to obtain the fully connected layer output of the pre-trained model.

[0096] Optionally, the modeling unit matching module includes:

[0097] A modeling unit matching submodule is used to determine that the last character of the first character set is a target character if the first character set constitutes a first word segmentation, wherein the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.

[0098] Optionally, the initial business model and the target business model are used to add punctuation marks to text.

[0099] Optionally, the business data includes text data to be processed and punctuation labels corresponding to the text data; and the knowledge distillation module includes:

[0100] A first loss value calculation submodule, configured to calculate a first loss value of the pre-trained model based on the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character;

[0101] A second loss value calculation submodule, configured to calculate a second loss value of the initial business model based on the fully connected layer output of the initial business model and the punctuation mark labels corresponding to the text data;

[0102] a joint loss value calculation submodule, configured to perform a weighted summation on the first loss value and the second loss value to obtain a joint loss value of the pre-trained model and the initial business model;

[0103] The knowledge distillation submodule is used to perform knowledge distillation on the pre-trained model and the initial business model according to the joint loss value to obtain a target business model.

[0104] Optionally, the device further comprises:

[0105] A voice data acquisition module, used to acquire voice data;

[0106] A speech recognition processing module, configured to perform speech recognition processing on the speech data to obtain text data corresponding to the speech data;

[0107] a punctuation mark adding module, configured to add punctuation marks to the text data corresponding to the voice data using the target service model;

[0108] The text data output module is used to output text data with punctuation added.

[0109] Optionally, the business data includes text data obtained after performing speech recognition processing on speech data.

[0110] In summary, the embodiment of the present invention can achieve knowledge distillation of the pre-trained model and the initial business model by matching the modeling units of the pre-trained model with the modeling units of the initial business model when the modeling units of the pre-trained model and the initial business model are different, thereby obtaining an optimized target business model and improving business performance. In addition, before matching the fully connected layer output of the pre-trained model with the fully connected layer output of the initial business model, the embodiment of the present invention first processes the respective modeling units based on the self-attention mechanism of the two models, so that each fully connected layer output refers to all the modeling units of the model, thereby avoiding the loss of word segmentation information during the knowledge distillation process, which is conducive to further improving the model performance of the target business model.

[0111] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0113] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0114] An embodiment of the present invention provides a device for model processing, the device comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0115] A11. Obtain business data;

[0116] A12. Input the business data into the pre-trained model and the initial business model respectively, and process the business data through the self-attention mechanism to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation, and one modeling unit of the pre-trained model corresponds to one fully connected layer output, and one modeling unit of the initial business model corresponds to one fully connected layer output;

[0117] A13. Matching the modeling unit of the pre-training model with the modeling unit of the initial business model to determine the target character in the modeling unit of the pre-training model;

[0118] A14. Based on the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character, knowledge distillation is performed on the pre-trained model and the initial business model to obtain a target business model.

[0119] Optionally, the step of inputting the business data into a pre-trained model and an initial business model respectively, processing the business data through a self-attention mechanism, and obtaining a fully connected layer output of the pre-trained model and a fully connected layer output of the initial business model includes:

[0120] Input the business data into the self-attention network layer of the pre-trained model for processing to obtain a first hidden layer vector corresponding to the business data;

[0121] Inputting the first hidden layer vector into the fully connected layer of the pre-trained model for calculation to obtain the fully connected layer output of the pre-trained model;

[0122] Inputting the business data into the self-attention network layer of the initial business model for processing to obtain a second hidden layer vector corresponding to the business data;

[0123] The second hidden layer vector is input into the fully connected layer of the initial business model for calculation to obtain the fully connected layer output of the pre-trained model.

[0124] Optionally, matching the modeling unit of the pre-trained model with the modeling unit of the initial business model to determine the target character in the modeling unit of the pre-trained model includes:

[0125] If the first character set constitutes a first word segmentation, the last character of the first character set is determined to be the target character, wherein the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.

[0126] Optionally, the initial business model and the target business model are used to add punctuation marks to text.

[0127] Optionally, the business data includes text data to be processed and punctuation labels corresponding to the text data; and performing knowledge distillation on the pre-trained model and the initial business model based on the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character to obtain a target business model includes:

[0128] Calculating a first loss value of the pre-trained model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character;

[0129] Calculating a second loss value of the initial business model according to the fully connected layer output of the initial business model and the punctuation mark labels corresponding to the text data;

[0130] Performing a weighted summation on the first loss value and the second loss value to obtain a joint loss value of the pre-trained model and the initial business model;

[0131] Knowledge distillation is performed on the pre-trained model and the initial business model according to the joint loss value to obtain a target business model.

[0132] Optionally, the apparatus is further configured to execute, by one or more processors, the one or more programs comprising instructions for performing the following operations:

[0133] Get voice data;

[0134] Performing speech recognition processing on the speech data to obtain text data corresponding to the speech data;

[0135] Using the target service model, adding punctuation marks to text data corresponding to the voice data;

[0136] Outputs text data with punctuation added.

[0137] Optionally, the business data includes text data obtained after performing speech recognition processing on speech data.

[0138] Figure 4 FIG8 is a block diagram of an apparatus 800 for model processing according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0139] Reference Figure 4 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0140] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0141] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0142] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0143] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0144] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0145] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0146] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, the sensor assembly 814 can also detect changes in the position of the voice processing device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0147] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0148] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0149] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the apparatus 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0150] Figure 5 1 is a schematic diagram of the structure of a server in some embodiments of the present invention. The server 1900 may vary greatly due to different configurations or performance, and may include one or more central processing units (CPUs) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be temporary storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 to execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0151] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or one or more operating systems 1941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0152] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of a device (server or terminal), enables the device to perform Figure 1 The model processing method shown.

[0153] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the device (server or terminal), enables the device to perform the above Figure 1 The description of the model training method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in the present invention, please refer to the description of the method embodiment of the present invention.

[0154] In addition, it should be noted that: the embodiment of the present invention also provides a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device performs the above-mentioned Figure 1 The description of the model training method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in the present invention, please refer to the description of the method embodiment of the present invention.

[0155] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0156] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0157] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0158] The above is a detailed introduction to a model processing method, a model processing device and a device for model processing provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A model processing method, It is characterized in that The method comprises: Get business data; The business data is input into the pre-trained model and the initial business model respectively, and the business data is processed by the self-attention mechanism to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation, and one modeling unit of the pre-trained model corresponds to one fully connected layer output, and one modeling unit of the initial business model corresponds to one fully connected layer output; Matching the modeling unit of the pre-training model with the modeling unit of the initial business model to determine the target character in the modeling unit of the pre-training model; According to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character, the pre-trained model and the initial business model are subjected to knowledge distillation to obtain a target business model; The initial business model and the target business model are used to add punctuation marks to the text; the business data includes text data to be processed and punctuation mark labels corresponding to the text data; the pre-trained model and the initial business model are subjected to knowledge distillation according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character to obtain the target business model, including: Calculating a first loss value of the pre-trained model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character; Calculating a second loss value of the initial business model according to the fully connected layer output of the initial business model and the punctuation mark labels corresponding to the text data; Performing a weighted summation on the first loss value and the second loss value to obtain a joint loss value of the pre-trained model and the initial business model; Knowledge distillation is performed on the pre-trained model and the initial business model according to the joint loss value to obtain a target business model.

2. The method according to claim 1, It is characterized in that The business data is input into the pre-trained model and the initial business model respectively, and the business data is processed by the self-attention mechanism to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model, including: Input the business data into the self-attention network layer of the pre-trained model for processing to obtain a first hidden layer vector corresponding to the business data; Inputting the first hidden layer vector into the fully connected layer of the pre-trained model for calculation to obtain the fully connected layer output of the pre-trained model; Inputting the business data into the self-attention network layer of the initial business model for processing to obtain a second hidden layer vector corresponding to the business data; The second hidden layer vector is input into the fully connected layer of the initial business model for calculation to obtain the fully connected layer output of the pre-trained model.

3. The method according to claim 1, It is characterized in that The matching of the modeling unit of the pre-training model with the modeling unit of the initial business model to determine the target character in the modeling unit of the pre-training model includes: If the first character set constitutes the first word segmentation, the last character of the first character set is determined to be the target character, wherein the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.

4. The method according to claim 1, It is characterized in that The method further comprises: Get voice data; Performing speech recognition processing on the speech data to obtain text data corresponding to the speech data; Using the target service model, adding punctuation marks to text data corresponding to the voice data; Outputs text data with punctuation added.

5. The method according to any one of claims 1 to 4, It is characterized in that The business data includes text data obtained after performing speech recognition processing on speech data.

6. A model processing device, It is characterized in that The device comprises: Business data acquisition module, used to acquire business data; A business data processing module, used to input the business data into the pre-trained model and the initial business model respectively, and process the business data through the self-attention mechanism to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation, one modeling unit of the pre-trained model corresponds to one fully connected layer output, and one modeling unit of the initial business model corresponds to one fully connected layer output; A modeling unit matching module, used to match the modeling unit of the pre-training model with the modeling unit of the initial business model, and determine the target character in the modeling unit of the pre-training model; A knowledge distillation module, used to perform knowledge distillation on the pre-trained model and the initial business model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character, so as to obtain a target business model; The initial business model and the target business model are used to add punctuation marks to the text; the business data includes the text data to be processed and the punctuation mark labels corresponding to the text data; the knowledge distillation module includes: A first loss value calculation submodule, used to calculate a first loss value of the pre-trained model according to the fully connected layer output of the initial business model and the fully connected layer output corresponding to the target character; A second loss value calculation submodule, used to calculate a second loss value of the initial business model according to the fully connected layer output of the initial business model and the punctuation mark labels corresponding to the text data; A joint loss value calculation submodule, used to perform a weighted summation on the first loss value and the second loss value to obtain a joint loss value of the pre-trained model and the initial business model; The knowledge distillation submodule is used to perform knowledge distillation on the pre-trained model and the initial business model according to the joint loss value to obtain a target business model.

7. The device according to claim 6, It is characterized in that The business data processing module includes: A first hidden layer vector determination submodule, used for inputting the business data into the self-attention network layer of the pre-trained model for processing to obtain a first hidden layer vector corresponding to the business data; A first fully connected layer output submodule, used for inputting the first hidden layer vector into the fully connected layer of the pre-trained model for calculation to obtain the fully connected layer output of the pre-trained model; A second hidden layer vector determination submodule, used for inputting the business data into the self-attention network layer of the initial business model for processing to obtain a second hidden layer vector corresponding to the business data; The second fully connected layer output submodule is used to input the second hidden layer vector into the fully connected layer of the initial business model for calculation to obtain the fully connected layer output of the pre-trained model.

8. The device according to claim 6, It is characterized in that The modeling unit matching module comprises: A modeling unit matching submodule is used to determine that the last character of a first character set is a target character if the first character set constitutes a first word segmentation, wherein the first character set includes a plurality of characters, the plurality of characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.

9. A device for model processing, It is characterized in that The device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for executing the model processing method as described in any one of claims 1 to 5.

10. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes a device to perform the model processing method according to any one of claims 1 to 5.

11. A computer program product, It is characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes the model processing method according to any one of claims 1 to 5.