A data processing method and device
By matching the modeling units of the pre-trained model and the initial business model and distilling knowledge, the problem that the business model cannot be optimized is solved and the performance of the business model is improved.
Patent Information
- Application Number
- CN202111094328.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-09-17
AI Technical Summary
When the business model cannot be optimized using the pre-trained model, the effectiveness of the business model cannot be improved.
By matching the modeling unit of the pre-trained model with the modeling unit of the initial business model, and knowledge distillation is performed based on the full connection layer output of the initial business model and the full connection layer output of the matching characters, the target business model is obtained, so as to Optimize the business model without changing the model structure.
It realizes that the accuracy of the target business model is effectively improved without increasing the business data used for model training, and solves the problem of knowledge distillation caused by inconsistent output sequence lengths of the pre-trained model and business model full connection layer.
Smart Images

Figure CN113807540B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning, and in particular to a data processing method and device. Background Art
[0002] At present, pre-training models are developing by leaps and bounds. Pre-training can obtain pre-training models that are not related to specific tasks from large-scale data through self-supervised learning. In addition, pre-training models can transfer knowledge learned from large-scale data to specific businesses. In other words, if a pre-training model is used to optimize a business model, the effect of the business model can be improved. However, in some scenarios, the pre-training model cannot be used to optimize the business model, so the effect of the business model cannot be improved.
[0003] Therefore, a solution is urgently needed to solve the above problems. Summary of the invention
[0004] The technical problem to be solved by the present application is that when the pre-trained model cannot be used to optimize the business model, the effect of the business model cannot be improved. The present application provides a data processing method and device.
[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:
[0006] Get business data;
[0007] Inputting the business data into the pre-trained model and the initial business model to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output;
[0008] Matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively;
[0009] Knowledge distillation is performed based on the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0010] Optionally, the initial business model and the target business model are used to add punctuation marks to text.
[0011] Optionally, matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively, includes:
[0012] If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
[0013] Optionally, the method further includes:
[0014] Get voice data;
[0015] Recognize the voice data to obtain text corresponding to the voice data;
[0016] Using the target service model, adding punctuation marks to the text corresponding to the voice data;
[0017] Output the text corresponding to the voice data to which the punctuation marks are added.
[0018] Optionally, the business data is text obtained through automatic speech recognition.
[0019] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:
[0020] A first acquisition unit, used to acquire business data;
[0021] An input unit, used to input business data into a pre-trained model and an initial business model, to obtain a fully connected layer output of the pre-trained model and a fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output;
[0022] A matching unit, used for matching the modeling unit of the pre-trained model with the modeling unit of the initial business model, obtaining characters in the modeling unit of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively;
[0023] The determination unit is used to perform knowledge distillation according to the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0024] Optionally, the initial business model and the target business model are used to add punctuation marks to text.
[0025] Optionally, the matching unit is used to:
[0026] If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
[0027] Optionally, the device further comprises:
[0028] A second acquisition unit, used to acquire voice data;
[0029] A recognition unit, used to recognize the voice data and obtain text corresponding to the voice data;
[0030] A processing unit, configured to add punctuation marks to the text corresponding to the voice data by using the target service model;
[0031] An output unit is used to output the text corresponding to the voice data with the punctuation marks added.
[0032] Optionally, the business data is text obtained through automatic speech recognition.
[0033] In a third aspect, an embodiment of the present application provides a data processing device, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:
[0034] Get business data;
[0035] Inputting the business data into the pre-trained model and the initial business model to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output;
[0036] Matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively;
[0037] Knowledge distillation is performed based on the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0038] Optionally, the initial business model and the target business model are used to add punctuation marks to text.
[0039] Optionally, matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively, includes:
[0040] If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
[0041] Optionally, the operation further includes:
[0042] Get voice data;
[0043] Recognize the voice data to obtain text corresponding to the voice data;
[0044] Using the target service model, adding punctuation marks to the text corresponding to the voice data;
[0045] Output the text corresponding to the voice data to which the punctuation marks are added.
[0046] Optionally, the business data is text obtained through automatic speech recognition.
[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the device to execute any of the methods described in the first aspect above.
[0048] Compared with the prior art, the embodiments of the present application have the following advantages:
[0049] The embodiment of the present application provides a data processing method. Specifically: business data can be obtained, and then the business data can be input into a pre-trained model and an initial business model to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model. In the embodiment of the present application, the modeling unit of the pre-trained model is different from the modeling unit of the initial business model. The modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a segmentation. Since one modeling unit corresponds to one fully connected layer output, this results in the sequence length of the fully connected layer output of the pre-trained model being different from the sequence length of the fully connected layer output of the initial business model. In this case, in order to enable the initial business model and the pre-trained model to be knowledge distilled, in the embodiment of the present application, the modeling unit of the pre-trained model can be matched with the modeling unit of the initial business model to obtain the characters in the modeling unit of the pre-trained model, and the characters are matched with each modeling unit of the initial business model respectively, and then, according to the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit, knowledge distillation is performed to obtain the target business model. That is, in the embodiment of the present application, even if the length of the sequence output by the fully connected layer of the pre-trained model is greater than the length of the sequence output by the fully connected layer of the initial business model, by using this solution, a sequence with the same length as the sequence output by the fully connected layer of the initial business model can be screened out from the sequence output by the fully connected layer of the pre-trained model, thereby performing knowledge distillation. It can be seen that by using this solution, the initial business model can be optimized using the pre-trained model to obtain the target business model, thereby improving business performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0051] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0052] Figure 2 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0053] Figure 3 A schematic diagram of the structure of the client provided in the embodiment of the present application;
[0054] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0056] After research, the inventors of this application found that the business model can be optimized using a pre-trained model by means of knowledge distillation. However, there is a prerequisite for using knowledge distillation, that is, the sequence length of the fully connected layer output of the pre-trained model is consistent with the sequence length of the fully connected layer output of the business model. This is because when performing knowledge distillation, the relative entropy divergence loss (Kullback–Leibler divergence loss, KLD loss) is calculated based on the fully connected layer output of the pre-trained model and the fully connected layer output of the business model. For example: the fully connected layer output of the pre-trained model includes 3 vectors, namely w1, w2 and w3, and the fully connected layer output of the business model includes 3 vectors, namely L1, L2 and L3, then KLDloss = w1*L1+w2*L2+w3*L3.
[0057] In some scenarios, for example, when automatically adding punctuation to text, the modeling unit of the corresponding business model is word segmentation, while the modeling unit of the pre-trained model is a single character. One modeling unit corresponds to a fully connected layer output, and the sequence lengths of the fully connected layer outputs corresponding to each modeling unit are the same (for example, the dimensions of the aforementioned w1, w2, w3, L1, L2, and L3 are the same). This results in the sequence length of the fully connected layer output of the business model being inconsistent with the sequence length of the fully connected layer output of the pre-trained model, making it impossible to perform knowledge distillation. For example:
[0058] For the text "I am Chinese", the input of the pre-trained model includes 5 modeling units, namely: "I", "Is", "China", "Country", "People". The input of the business model includes 3 modeling units, namely "I", "Is", "Chinese". Assuming that the fully connected layer output corresponding to each modeling unit is a 5-dimensional vector, for the pre-trained model, its fully connected layer output is 5 5-dimensional vectors, while for the business model, its fully connected layer output is 3 5-dimensional vectors. That is: the sequence length of the fully connected layer output of the business model is inconsistent with the sequence length of the fully connected layer output of the pre-trained model.
[0059] It should be noted that:
[0060] The knowledge distillation mentioned here refers to: transferring the knowledge of the pre-trained model to the business model, thereby optimizing the business model.
[0061] The pre-trained models mentioned here include but are not limited to BERT, GPT, and ELECTRA.
[0062] The business model mentioned here may be, for example, a bidirectional long short term memory network (Bi-directional Long-Short Term Memory, BiLSTM) model.
[0063] The modeling unit mentioned here refers to the granularity of the model input, which is generally a single character or a word. A word can include one or more characters.
[0064] The fully connected layer output mentioned here is an unnormalized label probability value vector. For example, in the scenario of adding punctuation to text, there are 4 optional punctuation marks, and the fully connected layer output corresponding to each modeling unit is a 5-dimensional vector. The specific value of the 5-dimensional vector is used to indicate the probability of the 4 punctuation marks and the probability of no punctuation marks. The fully connected layer output can also be called Logits output.
[0065] It is understandable that if the pre-trained model and the business model cannot perform knowledge distillation, the pre-trained model cannot be used to optimize the business model, so that the effect of the business model cannot be improved. In order to solve the above problem, the embodiment of the present application provides a data processing method and device.
[0066] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.
[0067] Exemplary Methods
[0068] See also Figure 1 , which is a flow chart of a data processing method provided in an embodiment of the present application. In this embodiment, the method can be implemented, for example, by following the steps S101-S104.
[0069] S101: Acquire business data.
[0070] The business data mentioned in the embodiments of the present application refers to data related to a specific business. The embodiments of the present application do not specifically limit the business data. In one example, the business data may be text automatically recognized based on speech.
[0071] S102: Input the business data into the pre-trained model and the initial business model to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model, wherein the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output.
[0072] In an embodiment of the present application, the pre-trained model and the initial business model may be obtained by pre-training. In one example, the pre-trained model may be trained based on pre-training data and business data. Specifically: the initial pre-trained model may be obtained by training using pre-training data, and then the initial pre-trained model may be fine-tuned (Finetune) using business data to obtain a business-based pre-trained model. In other words, the pre-trained model mentioned in S101 may be a business-based pre-trained model. The pre-training data mentioned here may be training data that is not related to the business. Generally, the amount of data of the pre-trained model is relatively large, and direct use will affect business processing efficiency.
[0073] In an embodiment of the present application, the initial business model can be obtained by training using business data. It is understandable that the amount of business data cannot be infinitely expanded. Especially for some emerging businesses, their business data is very limited. Therefore, the accuracy of the initial business model is often not particularly high. By optimizing the initial business model using a pre-trained model in the manner of knowledge distillation, the accuracy of the optimized target business model can be effectively improved without adding additional business data for model training.
[0074] In an embodiment of the present application, the modeling unit of the pre-trained model is a single character, and the modeling unit of the initial business model is a word segmentation, which may include one or more characters. For the pre-trained model and the initial business model, one modeling unit corresponds to one fully connected layer output. Moreover, the sequence lengths of the fully connected layer outputs corresponding to the modeling units of the pre-trained model and the initial business model are the same. Therefore, the sequence length of the fully connected layer output of the pre-trained model is greater than or equal to the sequence length of the fully connected layer output of the initial business model. Moreover, since it is less likely that each modeling unit of the initial business model includes only one character, in most cases, the sequence length of the fully connected layer output of the pre-trained model is greater than the sequence length of the fully connected layer output of the initial business model.
[0075] S103: Matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with the modeling units of the initial business model respectively.
[0076] S104: Perform knowledge distillation according to the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0077] Regarding S102 and S103, it should be noted that the embodiment of the present application does not specifically limit the execution order of S102 and S103, as long as S102 and S103 are executed between S101 and S104. S102 can be executed before S103, S102 can also be executed at the same time as S103, or S102 can also be executed after S103.
[0078] Regarding S103 and S104, it should be noted that, in most cases, the sequence length of the fully connected layer output of the pre-trained model is greater than the sequence length of the fully connected layer output of the initial business model. Therefore, knowledge distillation cannot be performed using the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model. In view of this, in an embodiment of the present application, the characters that match the various modeling units of the initial business model can be determined from the modeling units of the pre-trained model, and then, the fully connected layer output of the initial business model and the fully connected layer output of the characters that match the various modeling units of the initial business model in the modeling units of the pre-trained model are used for knowledge distillation. In other words, in this scheme, a sequence with the same sequence length as the fully connected layer output of the initial business model can be screened out from the sequence output of the fully connected layer of the pre-trained model, thereby performing knowledge distillation and obtaining the target business model.
[0079] For example:
[0080] The business data is: "I am Chinese". The initial business model includes three modeling units, namely: "I", "Is", and "Chinese", and their corresponding fully connected layer outputs are: a1, a2, and a3. The pre-trained model includes five modeling units, namely: "I", "Is", "China", "China", and "People", and their corresponding fully connected layer outputs are: b1, b2, b3, b4, and b5. From the modeling units "I", "Is", "China", "China", and "People" of the pre-trained model, determine the characters corresponding to each modeling unit of the initial business model. For example, determine that the character "I" in the pre-trained model corresponds to the modeling unit "I" of the initial business model; determine that the character "Is" in the pre-trained model corresponds to the modeling unit "Is" of the initial business model; determine that the character "People" in the pre-trained model corresponds to the modeling unit "Chinese" of the initial business model. Then, use a1, a2, a3 and b1, b2, and b5 for knowledge distillation to obtain the target business model. When performing knowledge distillation, the KLD loss can be calculated using the formula a1*b1+a2*b2+a3*b5.
[0081] Another example:
[0082] The business data is: "I am Chinese". The initial business model includes three modeling units, namely: "I", "Is", "Chinese", and their corresponding fully connected layer outputs are: a1, a2, and a3. The pre-trained model includes five modeling units, namely: "I", "Is", "China", "China", and "People", and their corresponding fully connected layer outputs are: b1, b2, b3, b4, and b5. From the modeling units "I", "Is", "China", "China", and "People" of the pre-trained model, determine the characters corresponding to each modeling unit of the initial business model. For example, determine that the character "I" in the pre-trained model corresponds to the modeling unit "I" of the initial business model; determine that the character "Is" in the pre-trained model corresponds to the modeling unit "Is" of the initial business model; determine that the character "China" in the pre-trained model corresponds to the modeling unit "Chinese" of the initial business model. Then, use a1, a2, a3 and b1, b2, b3 for knowledge distillation to obtain the target business model. When performing knowledge distillation, the KLD loss can be calculated using the formula a1*b1+a2*b2+a3*b3.
[0083] It should be noted that the initial business model and the target business model in the embodiments of the present application can be used to process related businesses. In one example, the initial business model and the target business model can be used to add symbols to text. Considering that the text automatically recognized by voice does not include punctuation marks. Therefore, in some embodiments, the text mentioned here can be text automatically recognized by voice.
[0084] In the embodiments of the present application, for the case where the initial business model and the target business model can be used to add symbols to text, considering that for a word segment, it is impossible to add punctuation marks inside the word segment, and punctuation marks may be added after the last character of the word segment. For example, for the word segment "Chinese people", it is impossible to add punctuation marks between the characters "中" and "国"; it is also impossible to add punctuation marks between the characters "国" and "人". However, it is possible to add punctuation marks after the character "人". Therefore, in specific implementation, S103 can be: matching each modeling unit of the initial business model with each modeling unit of the pre-trained model. If multiple modeling units of the pre-trained model constitute a certain modeling unit of the initial business model, then the last modeling unit among the multiple modeling units is determined as the character matching the aforementioned "certain modeling unit".
[0085] For ease of description, the "certain modeling unit" is referred to as the "first word segment", and the "multiple modeling units of the pre-trained model" are referred to as the "first character set". Then: if the first character set constitutes the first word segment, then the last character of the first character set is determined as the character matching the first word segment. For example: the first character set is {"中", "国", "人"}, and the first word segment is "Chinese people", then the last character "人" of the first character set is determined as the character matching the first word segment.
[0086] From the above description, it can be seen that even if the sequence length of the output of the fully connected layer of the pre-trained model is greater than the sequence length of the output of the fully connected layer of the initial business model, it is possible to screen out a sequence with the same length as the sequence length of the output of the fully connected layer of the initial business model from the sequence output by the fully connected layer of the pre-trained model, so as to perform knowledge distillation. Thus, it can be seen that by using this solution, it is possible to optimize the initial business model with the pre-trained model to obtain the target business model, thereby improving business performance.
[0087] In one implementation manner of the embodiments of the present application, after obtaining the target business model, the target business model can be used to process related services. In one example, if the target business model adds punctuation marks to text. Then the method may further include the following steps A - D.
[0088] Step A: Obtain speech data.
[0089] The speech data may be data that the user inputs in real time through a microphone, or data that the user inputs and stores in advance, and is not limited here.
[0090] Step B: Identify the speech data to obtain the text corresponding to the speech data.
[0091] In the embodiment of the present application, the speech recognition technology can be used to recognize the specific content of the speech data, so as to obtain the text corresponding to the speech data. The speech recognition technology will not be introduced in detail here.
[0092] Step C: using the target service model, adding punctuation marks to the text corresponding to the voice data.
[0093] Step D: Output the text corresponding to the voice data with the punctuation marks added.
[0094] Since the text obtained according to speech recognition does not include punctuation marks, after obtaining the text corresponding to the speech data, the target business model can be used to add punctuation marks to the text corresponding to the speech data, and further output the text corresponding to the speech data with the punctuation marks added.
[0095] The outputting of the text corresponding to the voice data with the punctuation marks added mentioned here may, for example, be displaying the text corresponding to the voice data with the punctuation marks added in the text input area. Through steps AD, the text including the punctuation marks can be automatically obtained according to the voice data.
[0096] Regarding steps A and D, let's take a look at some specific scenarios:
[0097] In the instant messaging scenario, if it is inconvenient for the user to manually input text, the user can call a microphone to record voice data, and then the device installed with the instant messaging software can receive the voice data input by the user, further recognize the voice data, and use the target business model to add punctuation marks to the text corresponding to the voice data. Then, in the input area of the instant messaging page, the text corresponding to the voice data with the punctuation marks added is entered.
[0098] It should be noted that the above example is only one application scenario of the present application, and the scenarios to which the solutions provided in the embodiments of the present application are applicable are not limited to those described above.
[0099] Exemplary Devices
[0100] Based on the method provided in the above embodiment, the embodiment of the present application further provides a device, which is described below in conjunction with the accompanying drawings.
[0101] See also Figure 2 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The device 200 may specifically include: a first acquisition unit 201, an input unit 202, a matching unit 203 and a determination unit 204.
[0102] A first acquisition unit 201 is used to acquire business data;
[0103] An input unit 202 is used to input business data into a pre-trained model and an initial business model to obtain a fully connected layer output of the pre-trained model and a fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a segmentation, and one modeling unit corresponds to one fully connected layer output;
[0104] A matching unit 203 is used to match the modeling unit of the pre-trained model with the modeling unit of the initial business model to obtain characters in the modeling unit of the pre-trained model, and to match the characters with each modeling unit of the initial business model respectively;
[0105] The determination unit 204 is used to perform knowledge distillation according to the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0106] Optionally, the initial business model and the target business model are used to add punctuation marks to text.
[0107] Optionally, the matching unit 203 is used to:
[0108] If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
[0109] Optionally, the device further comprises:
[0110] A second acquisition unit, used to acquire voice data;
[0111] A recognition unit, used to recognize the voice data and obtain text corresponding to the voice data;
[0112] A processing unit, configured to add punctuation marks to the text corresponding to the voice data by using the target service model;
[0113] An output unit is used to output the text corresponding to the voice data with the punctuation marks added.
[0114] Optionally, the business data is text obtained through automatic speech recognition.
[0115] Since the device 200 is a device corresponding to the method provided in the above method embodiment, the specific implementation of each unit of the device 200 is based on the same concept as the above method embodiment. Therefore, for the specific implementation of each unit of the device 200, reference can be made to the description part of the above method embodiment, which will not be repeated here.
[0116] The method provided in the embodiment of the present application can be executed by a client or a server. The client and server that execute the above method are described below respectively.
[0117] Figure 3 1 shows a block diagram of a client 300. For example, the client 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0118] Reference Figure 3 The client 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output (I / O) interface 33 , a sensor component 314 , and a communication component 316 .
[0119] The processing component 302 generally controls the overall operation of the client 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.
[0120] The memory 304 is configured to store various types of data to support operations on the client 300. Examples of such data include instructions for any application or method operating on the client 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0121] The power supply component 306 provides power to the various components of the client 300. The power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the client 300.
[0122] The multimedia component 308 includes a screen that provides an output interface between the client 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the client 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0123] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the client 300 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.
[0124] The I / O interface provides an interface between the processing component 302 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0125] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the client 300. For example, the sensor assembly 314 can detect the open / closed state of the device 300, the relative positioning of components, such as the display and keypad of the client 300, and the sensor assembly 314 can also detect the position change of the client 300 or a component of the client 300, the presence or absence of user contact with the client 300, the orientation or acceleration / deceleration of the client 300, and the temperature change of the client 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0126] The communication component 316 is configured to facilitate wired or wireless communication between the client 300 and other devices. The client 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0127] In an exemplary embodiment, the client 300 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the following methods:
[0128] Get business data;
[0129] Inputting the business data into the pre-trained model and the initial business model to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output;
[0130] Matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively;
[0131] Knowledge distillation is performed based on the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0132] Optionally, the initial business model and the target business model are used to add punctuation marks to text.
[0133] Optionally, matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively, includes:
[0134] If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
[0135] Optionally, the method further includes:
[0136] Get voice data;
[0137] Recognize the voice data to obtain text corresponding to the voice data;
[0138] Using the target service model, adding punctuation marks to the text corresponding to the voice data;
[0139] Output the text corresponding to the voice data to which the punctuation marks are added.
[0140] Optionally, the business data is text obtained through automatic speech recognition.
[0141] Figure 44 is a schematic diagram of the structure of the server in the embodiment of the present application. The server 400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 422 (for example, one or more processors) and memory 432, and one or more storage media 430 (for example, one or more mass storage devices) storing application programs 442 or data 444. Among them, the memory 432 and the storage medium 430 can be short-term storage or permanent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 422 can be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the server 400.
[0142] Furthermore, the CPU 422 may execute the following method:
[0143] Get business data;
[0144] Inputting the business data into the pre-trained model and the initial business model to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output;
[0145] Matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively;
[0146] Knowledge distillation is performed based on the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0147] Optionally, the initial business model and the target business model are used to add punctuation marks to text.
[0148] Optionally, matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively, includes:
[0149] If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
[0150] Optionally, the method further includes:
[0151] Get voice data;
[0152] Recognize the voice data to obtain text corresponding to the voice data;
[0153] Using the target service model, adding punctuation marks to the text corresponding to the voice data;
[0154] Output the text corresponding to the voice data to which the punctuation marks are added.
[0155] Optionally, the business data is text obtained through automatic speech recognition.
[0156] The server 400 may also include one or more power supplies 426, one or more wired or wireless network interfaces 450, one or more input and output interfaces 456, one or more keyboards 456, and / or one or more operating systems 441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0157] The present application also provides a computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes the device to perform the following method:
[0158] Get business data;
[0159] Inputting the business data into the pre-trained model and the initial business model to obtain the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, and one modeling unit corresponds to one fully connected layer output;
[0160] Matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively;
[0161] Knowledge distillation is performed based on the fully connected layer output of the initial business model and the fully connected layer output of the characters matched by each modeling unit to obtain a target business model.
[0162] Optionally, the initial business model and the target business model are used to add punctuation marks to text.
[0163] Optionally, matching the modeling units of the pre-trained model with the modeling units of the initial business model to obtain characters in the modeling units of the pre-trained model, and matching the characters with each modeling unit of the initial business model respectively, includes:
[0164] If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
[0165] Optionally, the method further includes:
[0166] Get voice data;
[0167] Recognize the voice data to obtain text corresponding to the voice data;
[0168] Using the target service model, adding punctuation marks to the text corresponding to the voice data;
[0169] Output the text corresponding to the voice data to which the punctuation marks are added.
[0170] Optionally, the business data is text obtained through automatic speech recognition.
[0171] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0172] It should be understood that the present application is not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
[0173] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire business data, where the business data refers to data related to a specific business; Inputting the business data into the pre-trained model and the initial business model, obtaining the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, one modeling unit corresponds to one fully connected layer output, the pre-trained model is trained according to the pre-trained data and the business data, and the initial business model is trained using the business data; Determining, from the modeling units of the pre-trained model, characters that match the respective modeling units of the initial business model; Knowledge distillation is performed using the fully connected layer output of the initial business model and the fully connected layer output of the characters in the modeling units of the pre-trained model that match the various modeling units of the initial business model to obtain a target business model.
2. The method according to claim 1, characterized in that The initial business model and the target business model are used to add punctuation marks to text.
3. The method according to claim 2, characterized in that Determining, from the modeling units of the pre-trained model, characters that match the respective modeling units of the initial business model, including: If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
4. The method according to claim 2, characterized in that: The method further comprises: Get voice data; Recognize the voice data to obtain text corresponding to the voice data; Using the target service model, adding punctuation marks to the text corresponding to the voice data; Output the text corresponding to the voice data to which the punctuation marks are added.
5. The method according to claim 1, characterized in that: The business data is text obtained through automatic speech recognition.
6. A data processing device, characterized in that: The device comprises: A first acquisition unit, configured to acquire business data, wherein the business data refers to data related to a specific business; An input unit, used to input business data into a pre-trained model and an initial business model, to obtain a fully connected layer output of the pre-trained model and a fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, one modeling unit corresponds to one fully connected layer output, the pre-trained model is trained according to the pre-training data and the business data, and the initial business model is trained using the business data; A matching unit, used to determine, from the modeling units of the pre-trained model, characters that match the modeling units of the initial business model respectively; A determination unit is used to perform knowledge distillation using the fully connected layer output of the initial business model and the fully connected layer output of the characters in the modeling units of the pre-trained model that match the various modeling units of the initial business model to obtain a target business model.
7. The device according to claim 6, characterized in that The initial business model and the target business model are used to add punctuation marks to text.
8. The device according to claim 7, characterized in that The matching unit is used for: If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
9. The device according to claim 7, characterized in that The device also includes: A second acquisition unit, used to acquire voice data; A recognition unit, used to recognize the voice data and obtain text corresponding to the voice data; A processing unit, configured to add punctuation marks to the text corresponding to the voice data by using the target service model; An output unit is used to output the text corresponding to the voice data with the punctuation marks added.
10. The device according to claim 6, characterized in that The business data is text obtained through automatic speech recognition.
11. A data processing device, characterized in that: The system comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations: Acquire business data, where the business data refers to data related to a specific business; Inputting the business data into the pre-trained model and the initial business model, obtaining the fully connected layer output of the pre-trained model and the fully connected layer output of the initial business model; wherein the modeling unit of the pre-trained model is a single character, the modeling unit of the initial business model is a word segmentation, one modeling unit corresponds to one fully connected layer output, the pre-trained model is trained according to the pre-trained data and the business data, and the initial business model is trained using the business data; Determining, from the modeling units of the pre-trained model, characters that match the respective modeling units of the initial business model; Knowledge distillation is performed using the fully connected layer output of the initial business model and the fully connected layer output of the characters in the modeling units of the pre-trained model that match the various modeling units of the initial business model to obtain a target business model.
12. The device according to claim 11, characterized in that The initial business model and the target business model are used to add punctuation marks to text.
13. The device according to claim 12, characterized in that Determining, from the modeling units of the pre-trained model, characters that match the respective modeling units of the initial business model, including: If the first character set constitutes the first word segmentation, the last character of the first character set is determined as the character matching the first word segmentation, wherein: the first character set includes multiple characters, the multiple characters are all modeling units of the pre-trained model, and the first word segmentation is any modeling unit of the initial business model.
14. The device according to claim 12, characterized in that The operations also include: Get voice data; Recognize the voice data to obtain text corresponding to the voice data; Using the target service model, adding punctuation marks to the text corresponding to the voice data; Output the text corresponding to the voice data to which the punctuation marks are added.
15. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes an apparatus to perform the method of any one of claims 1 to 5.
Citation Information
Patent Citations
Punctuation mark adding method, device and computer device
CN107767870A
Text labeling method and device based on adversarial learning
CN110110327A
