Deep learning model training method and device, text data processing method and device, electronic device, storage medium, and computer program

Abstract

To provide a training method and apparatus for a deep learning model, a text data processing method and apparatus, an electronic device, a storage medium, and a computer program.SOLUTION: A training method comprises processing sample source data and corresponding sample target data using a deep learning model to obtain a first output value and a second output value, respectively, identifying a regularization function value based on the first output value and the second output value, and adjusting parameters of the deep learning model based on the regularization function value to obtain a pre-trained deep learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing and machine translation. More specifically, the present disclosure provides a method and device for training a deep learning model, a method and device for processing text data, an electronic device, a storage medium, and a computer program. [Background technology]

[0002] Based on the technology of natural language processing, one kind of natural language text can be translated into another kind of natural language text. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides a method and device for training a deep learning model, a method and device for processing text data, an electronic device, a storage medium, and a computer program. [Means for solving the problem]

[0004] According to a first aspect, there is provided a method for training a deep learning model, the method including: processing sample source data and corresponding sample target data using a deep learning model to obtain first output values ​​and second output values, respectively; determining a regularization function value based on the first output values ​​and the second output values; and adjusting parameters of the deep learning model based on the regularization function value to obtain a pre-trained deep learning model.

[0005] In some embodiments, the sample source data includes first language text data and the sample target data includes second language text data.

[0006] According to a second aspect, there is provided a method for processing text data, comprising inputting text data into a deep learning model to obtain a processing result, the deep learning model being trained using a method provided by the present disclosure.

[0007] According to a third aspect, there is provided an apparatus for training a deep learning model, the apparatus including: a first acquisition module for processing sample source data and corresponding sample target data using a deep learning model, respectively, to obtain first output values ​​and second output values; an identification module for identifying a regularization function value based on the first output value and the second output value; and a first adjustment module for adjusting parameters of the deep learning model based on the regularization function value to obtain a pre-trained deep learning model.

[0008] According to a fourth aspect, there is provided a text data processing device including a fourth acquisition module for inputting text data into a deep learning model to obtain a processing result, the deep learning model being trained using the device provided by the present disclosure.

[0009] According to a fifth aspect, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform a method provided by the present disclosure.

[0010] According to a sixth aspect, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform a method provided by the present disclosure.

[0011] According to a seventh aspect, there is provided a computer program which, when executed by a processor, implements the method provided by the present disclosure.

[0012] It should be understood that the contents described in this section are not intended to represent key points or important features of the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood from the following description.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS Here, the drawings are used for a better understanding of the invention of the present disclosure and are not intended to limit the present disclosure. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a flowchart of a method for training a deep learning model according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 is a flowchart of a method for training a deep learning model according to one embodiment of the present disclosure. [Diagram 3] FIG. 3 is a flowchart of a method for training a deep learning model according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a flowchart of a method for training a deep learning model according to one embodiment of the present disclosure. [Diagram 5] FIG. 5 is a principle diagram of a method for training a deep learning model according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a flowchart of a text data processing method according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a block diagram of a deep learning model training apparatus according to one embodiment of the present disclosure. [Figure 8] FIG. 8 is a block diagram of a text processing device according to one embodiment of the present disclosure. [Figure 9]FIG. 9 is a block diagram of an electronic device to which a deep learning model training method and / or a text data processing method according to an embodiment of the present disclosure can be applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. Hereinafter, various details of the embodiments of the present disclosure are included for easier understanding, and they should be considered as examples. Therefore, it should be understood that a person skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description omits descriptions of well-known functions and configurations.

[0016] Based on the technology of natural language processing, translation can be performed using a deep learning model. The performance of the associated deep learning model can be improved by increasing the amount of training data. However, since the number of existing corpus training samples is limited, it is difficult to further improve the performance of the deep learning model.

[0017] FIG. 1 is a flowchart of a method for training a deep learning model according to one embodiment of the present disclosure.

[0018] As shown in FIG. 1, the method 100 may include operations S110 to S130. In operation S110, the sample source data and the corresponding sample target data are processed using the deep learning model to obtain a first output value and a second output value, respectively.

[0019] In an embodiment of the present disclosure, the sample source data may include first language text data, and the sample target data may include second language text data.

[0020] For example, the first language text data may be text in a natural language, and the second language text data may be text in another natural language.

[0021] For example, the sample source data is Chinese text data, and the sample target data is English text data. In one example, the sample source data x is "今天天气很好(今日はいい天気です)", and the sample target data y is "The weather is good today".

[0022] Also for example, the sample source data x may be input into a deep learning model to obtain a first reference output value f(x, y; Θ). Also, the sample target data y may be input into a deep learning model to obtain a second reference output value f(y, x; Θ). The first reference output value f(x, y; Θ) and the second reference output value f(y, x; Θ) may be used as the first output value. In one example, Θ may represent one or more parameters of the deep learning model.

[0023] In an embodiment of the present disclosure, the sample source data may be processed to obtain the processed sample source data. The sample target data may be processed to obtain the processed sample target data.

[0024] For example, the processed sample source data x_proc may be input into a deep learning model to obtain a first processed output value f(x_proc, y_proc; Θ). The processed sample target data y_proc may be input into a deep learning model to obtain a second processed output value f(y_proc, x_proc; Θ).

[0025] The first processed output value f(x_proc, y_proc; Θ) and the second processed output value f(y_proc, x_proc; Θ) may be used as the second output value.

[0026] In operation S120, a regularization function value is specified based on the first output value and the second output value.

[0027] For example, a difference between a first reference output value f(x,y;Θ) and a first processed output value f(x_proc,y_proc;Θ) may be determined, and a difference between a second reference output value f(y,x;Θ) and a second processed output value f(y_proc,x_proc;Θ) may be determined. A regularization function value may be determined based on the two differences. In one example, the regularization function value may be a consistency regularization function value.

[0028] In operation S130, parameters of the deep learning model are adjusted based on the regularization function value to obtain a pre-trained deep learning model.

[0029] For example, a pre-trained deep learning model may be obtained by adjusting parameters of the deep learning model so that the regularization function value satisfies a predetermined condition.

[0030] The embodiments of the present disclosure can further improve the performance of deep learning models, especially those based on bidirectional pre-training.

[0031] In some embodiments, the above-mentioned predetermined condition may be less than a predetermined threshold.

[0032] In some embodiments, various text data expansion methods may be used to obtain processed sample source data or processed sample target data based on the sample source data or sample target data.

[0033] In some embodiments, the sample source data includes a first language text data and the sample target data includes a second language text data. For example, the first language text data may be Chinese text data and the second language text data may be English text data. For example, the first language text data may be English text data and the second language text data may be Chinese text data.

[0034] FIG. 2 is a flowchart of a method for training a deep learning model according to one embodiment of the present disclosure.

[0035] 2, the method 210 may use a deep learning model to process sample source data and corresponding sample target data to obtain a first output value and a second output value, which will be described in detail below with reference to operations S211 to S214.

[0036] In operation S211, the sample source data and the sample target data are respectively input into a deep learning model to obtain a first reference output value and a second reference output value as a first output value.

[0037] For example, as described above, sample source data x may be input to a deep learning model to obtain a first reference output value f(x,y;Θ), and sample target data y may be input to the deep learning model to obtain a second reference output value f(y,x;Θ).

[0038] In operation S212, word segmentation is performed on the sample source data and the sample target data, respectively, to obtain a plurality of source data segments and a plurality of target data segments.

[0039] For example, for the sample source data x, which is "Today's weather is nice (今日はいい天気です)", perform word segmentation to obtain a plurality of source data segments. In one example, one of the plurality of source data segments is "weather (天気)".

[0040] For example, for the sample target data y, which is "The weather is good today", perform word segmentation to obtain a plurality of target data segments. In one example, one of the plurality of target data segments is "today".

[0041] In operation S213, at least a part of the plurality of source data segments and at least a part of the plurality of target data segments are respectively masked to obtain masked source data and masked target data.

[0042] For example, the source data segment "weather (天気)" can be masked to obtain masked source data x_cut. The target data segment "today" can be masked to obtain masked target data y_cut.

[0043] In operation S214, the masked source data and the masked target data are respectively input into a deep learning model to obtain a first masked output value and a second masked output value as the second output value.

[0044] For example, the masked source data x_cut can be input into a deep learning model to obtain a first masked output value f(x_cut, y_cut; Θ). The masked target data y_cut can be input into a deep learning model to obtain a second masked output value f(y_cut, x_cut; Θ).

[0045] Note that operations S212 to S214 may be executed after operation S211 is executed. However, the embodiment of the present disclosure is not limited to this, and these two sets of operations may be executed in other orders, for example, operation S211 and operations S212 to S214 may be executed in parallel, or operation S211 may be executed after operations S212 to S214 are executed.

[0046] In some embodiments, the sample source data has a first label and the sample target data has a second label.

[0047]

number

[0048] FIG. 3 is a flowchart of a method for training a deep learning model according to another embodiment of the present disclosure.

[0049] 3, the method 320 may determine a regularization function value based on the first output value and the second output value, which will be described in detail below with reference to operations S321 to S325.

[0050] In operation S321, a first relative entropy between a first reference output value and a first masked output value is determined.

[0051]

number

[0052] In one example, KL(*) is the Kullback-Leibler divergence function. In operation S322, a second relative entropy between the second reference output value and the second masked output value is determined.

[0053]

number

[0054] In operation S323, a first cross-entropy between the first reference output value and the first label is determined.

[0055]

number

[0056] In operation S324, a second cross-entropy between the second reference output value and the second label is determined.

[0057]

number

[0058] In operation S325, a regularization function value is determined based on the first cross-entropy, the second cross-entropy, the first relative entropy, and the second relative entropy.

[0059] For example, the regularization function value E may be specified by the following formula:

[0060]

number

[0061]

number

[0062] FIG. 4 is a flowchart of a method for training a deep learning model according to another embodiment of the present disclosure.

[0063] 4, the method 400 may be performed after the above-mentioned operation S130, which will be described in detail below with reference to operations S440 to S460.

[0064] In operation S440, the sample source data is input into the pre-trained deep learning model to obtain a third reference output value.

[0065] For example, the sample source data x is input into the pre-trained deep learning model to obtain a third reference value f(x, y; Θ'), where Θ' may represent one or more parameters of the pre-trained deep learning model.

[0066] In operation S450, the masked source data is input to a pre-trained deep learning model to obtain a third masked output value.

[0067] For example, the masked source data x_cut is input into a pre-trained deep learning model to obtain a third masked output value f(x_cut, y_cut; Θ').

[0068] In operation S460, parameters of the pre-trained deep learning model are adjusted based on the third reference output value and the third masking output value to obtain a trained deep learning model.

[0069] For example, the function value E' may be specified by the following formula:

[0070]

number

[0071] Embodiments of the present disclosure allow for unidirectional fine-tuning of the pre-trained deep learning model, thereby further improving the model's performance.

[0072] FIG. 5 is a principle diagram of a method for training a deep learning model according to another embodiment of the present disclosure.

[0073] As shown in FIG. 5, the deep learning model 500 includes a processing layer 510 and a Softmax layer 520. The processing layer 510 may include a self-attention layer 511 and a feed-forward layer 512.

[0074] For example, the deep learning model 500 may output a first reference output value f(x, y; Θ) and a second reference output value f(y, x; Θ) by taking, as inputs, the above-described sample source data x, "The weather is good today", and the above-described sample target data y, "今日はいい天気です", respectively.

[0075] Also, for example, the deep learning model may output a first masked output value f(x_cut, y_cut; Θ) and a second masked output value f(y_cut, x_cut; Θ) by taking, as inputs, the above-described masked source data x_cut and the above-described masked target data y_cut, respectively. In one example, the masked source data x_cut may be, for example, "今天 <mask>Good (Today is good <mask>The masked target data y_cut may be, for example, "The weather is good. <mask>" may also be used.

[0076] In one example, the self-attention layer 511 and the feedforward layer 512 may process the sample source data x in sequence to output one intermediate output value. The softmax layer 520 processes the intermediate output value to output a first reference output value f(x,y;Θ). Similarly, the deep learning model 500 may process the sample target data y, the masked source data x_cut, and the masked target data y_cut.

[0077] FIG. 6 is a flowchart of a text data processing method according to an embodiment of the present disclosure. As shown in FIG. 6, the method 600 may include operation S610.

[0078] In operation S610, the text data is input into the deep learning model to obtain a processing result.

[0079] For example, the deep learning model is trained using the methods provided by the present disclosure.

[0080] For example, Chinese text data is input to a deep learning model, and the processing result obtained is English text data.Also, for example, English text data is input to a deep learning model, and the processing result obtained is Chinese text data.

[0081] FIG. 7 is a block diagram of a deep learning model training apparatus according to one embodiment of the present disclosure.

[0082] As shown in FIG. 7, the apparatus 700 may include a first acquisition module 710, an identification module 720, and a first adjustment module 730.

[0083] The first acquisition module 710 is for respectively processing the sample source data and the corresponding sample target data using the deep learning model to obtain a first output value and a second output value.

[0084] The determining module 720 is for determining a regularization function value based on the first output value and the second output value.

[0085] The first adjustment module 730 is for adjusting parameters of the deep learning model based on the regularization function value to obtain a pre-trained deep learning model.

[0086] In some embodiments, the first acquisition module includes: a first acquisition sub-module for inputting the sample source data and the sample target data into the deep learning model, respectively, to obtain a first reference output value and a second reference output value as first output values; a word segmentation sub-module for performing word segmentation on the sample source data and the sample target data, respectively, to obtain a plurality of source data segments and a plurality of target data segments; a masking module for masking at least a portion of the plurality of source data segments and at least a portion of the plurality of target data segments, respectively, to obtain masked source data and masked target data; and a second acquisition sub-module for inputting the masked source data and the masked target data into the deep learning model, respectively, to obtain a first masked output value and a second masked output value as second output values.

[0087] In some embodiments, the identification module includes a first identification sub-module for identifying a first relative entropy between the first reference output value and the first masked output value, a second identification sub-module for identifying a second relative entropy between the second reference output value and the second masked output value, and a third identification sub-module for identifying the regularization function value based on the first relative entropy and the second relative entropy.

[0088] In some embodiments, the sample source data has a first label and the sample target data has a second label, and the identification module further includes a fifth identification sub-module for determining a first cross-entropy between the first reference output value and the first label, a sixth identification sub-module for determining a second cross-entropy between the second reference output value and the second label, and a seventh identification sub-module for determining the regularization function value based on the first cross-entropy, the second cross-entropy, the first relative entropy, and the second relative entropy.

[0089] In some embodiments, the apparatus 700 further includes a second acquisition module for inputting the sample source data into the pre-trained deep learning model to obtain a third reference output value, a third acquisition module for inputting the masked source data into the pre-trained deep learning model to obtain a third masked output value, and a second adjustment module for adjusting parameters of the pre-trained deep learning model based on the third reference output value and the third masked output value to obtain a trained deep learning model.

[0090] In some embodiments, the second adjustment module includes an eighth identification submodule for identifying a third cross-entropy between the third reference output value and the first label, a ninth identification submodule for identifying a third relative entropy between the third reference output value and the third masked output value, and an adjustment submodule for adjusting parameters of the pre-trained deep learning model based on the third cross-entropy and the third relative entropy.

[0091] FIG. 8 is a block diagram of a text data processing apparatus according to another embodiment of the present disclosure. As shown in FIG. 8, the apparatus 800 may include a fourth acquisition module 810 .

[0092] The fourth acquisition module 810 is for inputting text data into the deep learning model to obtain processing results.

[0093] However, the deep learning model is trained using the device provided by the present disclosure.

[0094] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of the user's personal information are all in accordance with the provisions of relevant laws and regulations, have the necessary security measures, and are not contrary to public order and morals. In the technical solution disclosed herein, the permission or consent of the user is obtained before obtaining or collecting the user's personal information.

[0095] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0096] 9 shows a schematic block diagram of an example electronic device 900 capable of implementing embodiments of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0097] 9, the device 900 includes a computing means 901, which may perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 902 or loaded from a storage means 908 into a random access memory (RAM) 903. The RAM 903 may further store various programs and data necessary for the operation of the device 900. The computing means 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0098] A number of components in the device 900 are connected to an I / O interface 905, which includes input means 906 such as a keyboard, a mouse, etc., output means 907 such as various types of displays, speakers, etc., storage means 908 such as a magnetic disk, an optical disk, etc., and communication means 909 such as a network card, a modem, a wireless communication transceiver, etc. The communication means 909 enables the device 900 to exchange information and data with other devices via a computer network such as the Internet and / or various electric networks.

[0099] The computing means 901 may be various general-purpose and / or dedicated processing modules having processing and computing capabilities. Some examples of the computing means 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, computing units running various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing means 901 performs each of the methods and processes described in the preceding sentence, such as the deep learning model training method and / or the text data processing method. For example, in some embodiments, the deep learning model training method and / or the text data processing method may be realized as a computer software program tangibly included in a machine-readable medium, such as the storage unit 908. In some embodiments, some or all of the computer programs may be loaded and / or installed in the device 900 via the ROM 902 and / or the communication means 909. When the computer program is loaded into the RAM 903 and executed by the computing means 901, it may perform one or more steps of the method for training a deep learning model and / or the method for processing text data described in the preamble. Alternatively, in other embodiments, the computing means 901 may be configured to perform the method for training a deep learning model and / or the method for processing text data in any other suitable manner (e.g. via firmware).

[0100] Various embodiments of the systems and techniques described herein above may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may include capable of receiving data and instructions from and transmitting data and instructions to a storage system, at least one input device, and at least one output device.

[0101] The program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program codes may be fully executed on the device, partially executed on the device, partially executed on the device as a separate software package and partially executed on a remote device or fully executed on a remote device or server.

[0102] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program for use in or in combination with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include electrical connections through one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0103] To provide for interaction with a user, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the computer. Other types of devices may also provide for interaction with a user, for example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and may receive input from the user in any form (including voice input, speech input, or tactile input).

[0104] The systems and techniques described herein may be implemented in a computing system including background components (e.g., a data server), or a computing system including middleware components (e.g., an application server), or a computing system including front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user may interact with embodiments of the systems and techniques described herein), or a computing system including any combination of such background, middleware, or front-end components. The components of the system may be connected to each other by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks illustratively include a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] The computer system may include clients and servers. The clients and servers are generally remote and typically interact through a communication network. The relationship between clients and servers is created by computer programs running on the corresponding computers and having a client-server relationship. The server may be a cloud server, a server in a distributed system, or a server in combination with a blockchain.

[0106] It should be understood that various types of flows shown above may be used, operations may be reordered, added, or removed, etc. For example, each operation described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the desired results of the invention of this disclosure are achieved, and this specification is not limited thereto.

[0107] The above specific embodiments do not limit the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.< / mask> < / mask> < / mask>

Claims

1. A method for detecting a first reference output value and a second reference output value by inputting sample source data having a first label and sample target data having a second label into a deep learning model, respectively, and obtaining a first reference output value and a second reference output value as a first output value; performing word segmentation on the sample source data and the sample target data, respectively, to obtain a plurality of source data segments and a plurality of target data segments; masking at least a portion of the plurality of source data segments and at least a portion of the plurality of target data segments to obtain masked source data and masked target data, respectively; Inputting the masked source data and the masked target data into the deep learning model respectively to obtain a first masked output value and a second masked output value as second output values; determining a first relative entropy between the first reference output value and the first masked output value; determining a second relative entropy between the second reference output value and the second masked output value; determining a first cross-entropy between the first reference output value and the first label; determining a second cross-entropy between the second reference output value and the second label; determining, as the regularization function value, an expectation value of the first cross entropy, the second cross entropy, the first relative entropy, and the second relative entropy according to a regularization function; and adjusting parameters of the deep learning model based on the regularization function value to obtain a pre-trained deep learning model. How to train a deep learning model.

2. inputting the sample source data into the pre-trained deep learning model to obtain a third reference output value; inputting the masked source data into the pre-trained deep learning model to obtain a third masked output value; and unidirectionally fine-tuning parameters of the pre-trained deep learning model based on the third reference output value and the third masking output value to obtain a trained deep learning model. The method of claim 1.

3. adjusting parameters of the pre-trained deep learning model based on the third reference output value and the third masking output value to obtain a trained deep learning model; determining a third cross entropy between the third reference output value and a first label; determining a third relative entropy between the third reference output value and the third masked output value; and unidirectionally fine-tuning parameters of the pre-trained deep learning model based on the third cross-entropy and the third relative entropy. The method of claim 2.

4. 1. A method for processing text data, comprising inputting text data into a deep learning model to obtain a processing result, The deep learning model is trained using the method of claim 1. A method for processing text data.

5. A first acquisition submodule for inputting sample source data having a first label and sample target data having a second label into a deep learning model, respectively, to obtain a first reference output value and a second reference output value as a first output value; a word segmentation sub-module for performing word segmentation on the sample source data and the sample target data, respectively, to obtain a plurality of source data segments and a plurality of target data segments; a masking module for masking at least a portion of the plurality of source data segments and at least a portion of the plurality of target data segments to obtain masked source data and masked target data, respectively; a second obtaining submodule for respectively inputting the masked source data and the masked target data into the deep learning model to obtain a first masked output value and a second masked output value as a second output value; a first determination sub-module for determining a first relative entropy between the first reference output value and the first masked output value; a second determination sub-module for determining a second relative entropy between the second reference output value and the second masked output value; a fifth identification sub-module for determining a first cross-entropy between the first reference output value and the first label; a sixth identification sub-module for determining a second cross-entropy between the second reference output value and the second label; a seventh identification submodule for identifying an expectation value of the first cross entropy, the second cross entropy, the first relative entropy, and the second relative entropy as the regularization function value by a regularization function; and a first adjustment module for adjusting parameters of the deep learning model based on the regularization function value to obtain a pre-trained deep learning model. A training device for deep learning models.

6. a second acquisition module for inputting the sample source data into the pre-trained deep learning model to obtain a third reference output value; a third acquisition module for inputting the masked source data into the pre-trained deep learning model to obtain a third masked output value; and a second adjustment module for unidirectionally fine-tuning parameters of the pre-trained deep learning model based on the third reference output value and the third masking output value to obtain a trained deep learning model.

6. The apparatus of claim 5.

7. The second adjustment module includes: an eighth identification sub-module for determining a third cross-entropy between the third reference output value and a first label; a ninth identification sub-module for identifying a third relative entropy between the third reference output value and the third masked output value; and a tuning submodule for unidirectionally fine-tuning parameters of the pre-trained deep learning model based on the third cross-entropy and the third relative entropy.

7. The apparatus of claim 6.

8. A text data processing apparatus including a fourth acquisition module for inputting text data into a deep learning model to obtain a processing result, The deep learning model is trained using the apparatus of claim 5. Text data processing device.

9. At least one processor; an electronic device including a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor; The instructions are executed by the at least one processor such that the at least one processor performs the method of claim 1. electronic equipment.

10. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions are used to cause a computer to carry out the method of claim 1 . storage medium.

11. A computer program product which, when executed by a processor, implements the method of claim 1.

Images