Translation model determination method and apparatus, and storage medium

By aligning the random initialized translation model to the pretrained translation model and fusing it, the problems of poor effect and insufficient generalization in high-resource scenarios are solved, and the translation effect and generalization are improved.

WO2024244552A1PCT designated stage expired Publication Date: 2024-12-05BEIJING JINGDONG ZHENSHI INFORMATION TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/078499
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2024-02-26
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

在高资源场景的翻译任务中,基于预训练模型的翻译模型表现不佳,泛化性差,且随机参数初始化训练的翻译模型虽然翻译效果较好但泛化性较差。

Method used

By aligning the random initialization translation model to the pretrained translation model, the aligned random initialization translation model is obtained and fused with the pretrained translation model to generate the final fusion translation model to improve translation effect and generalization. The specific method includes aligning the model using a transmission mapping method, and the transmission matrix of each layer determines the optimal matrix through the point multiplying the cost matrix, and performs weighted summing to generate fusion parameters.

Benefits of technology

Through model fusion, the advantages of pre-trained parameter initialization and random parameter initialization are effectively combined, which improves the translation effect and generalization of the translation model, alleviates the problems of poor translation effect and insufficient generalization in high-resource scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024078499_05122024_PF_FP_ABST
    Figure CN2024078499_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to the field of computers. Provided are a translation model determination method and apparatus, and a storage medium. An embodiment of the present disclosure comprises: aligning a randomly initialized translation model to a pre-trained translation model so as to obtain an aligned randomly initialized translation model, fusing the aligned randomly initialized translation model and the pre-trained translation model so as to obtain a fused translation model, and, on the basis of the fused translation model, obtaining a final translation model. Therefore, the model fusion effectively combines the advantages of the translation models trained by using two methods, i.e. pre-training parameter initialization and random parameter initialization, and simultaneously improves the translation effect and generalization of the translation model, thus ameliorating the problems of poor translation effect of translation models trained by means of pre-training parameter initialization and poor generalization of translation models trained by means of random parameter initialization with respect to translation tasks in high-resource scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for determining translation model and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority to an application with CN application number 202310630832.0 and filing date May 30, 2023. The disclosed content of the CN application is hereby incorporated into this application as a whole. Technical Field

[0003] The present disclosure relates to the field of computers, and in particular to a method and device for determining a translation model and a storage medium. Background Art

[0004] Pre-trained models are general-purpose models trained on large-scale monolingual data using self-supervised methods. By effectively leveraging large-scale monolingual data, pre-trained models have achieved success in various natural language processing fields.

[0005] For translation tasks in low-resource scenarios, that is, scenarios with limited supervised training data, translation models based on pre-trained models can improve translation performance in low-resource scenarios. However, for translation tasks in high-resource scenarios, translation models based on pre-trained models can actually lead to a decrease in translation performance.

[0006] In translation tasks in high-resource scenarios, translation models trained with random parameter initialization perform better than those trained with pre-trained parameter initialization, but their generalization is poor, that is, they perform poorly when faced with data from multiple fields and noisy data.

[0007] Summary of the Invention

[0008] Some embodiments of the present disclosure provide a method for determining a translation model, including:

[0009] Initialize the network model with random parameters and pre-trained parameters to obtain a random initialized model and a pre-trained model respectively;

[0010] The randomly initialized model and the pre-trained model are trained using the translation dataset to obtain the randomly initialized translation model and the pre-trained translation model respectively;

[0011] Align the randomly initialized translation model to the pre-trained translation model to obtain an aligned randomly initialized translation model;

[0012] The aligned randomly initialized translation model and the pre-trained translation model are fused to obtain a fused translation model.

[0013] A final translation model for performing the translation task is determined based on the fused translation model.

[0014] In some embodiments, aligning the randomly initialized translation model to the pre-trained translation model to obtain an aligned randomly initialized translation model includes:

[0015] Each layer of the randomly initialized translation model is aligned to the corresponding layer of the pre-trained translation model using the transfer mapping method to obtain the aligned randomly initialized translation model.

[0016] In some embodiments, aligning each layer of the randomly initialized translation model to a corresponding layer of the pre-trained translation model using a transfer mapping method includes:

[0017] The input weight matrix of each current layer of the randomly initialized translation model is right-multiplied by the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, and then left-multiplied by the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model to obtain the aligned input weight matrix of the current layer of the randomly initialized translation model, where:

[0018] The input weight matrix of the current layer of the randomly initialized translation model refers to the weight matrix of the full connection between each neuron in the current layer of the randomly initialized translation model and each neuron in the previous layer.

[0019] The transfer matrix of each layer refers to the correspondence matrix between the neurons of each layer of the randomly initialized translation model and the neurons of the layer of the pre-trained translation model.

[0020] In some embodiments, the transmission matrix of each layer meets the following requirements:

[0021] The right multiplication of the transmission matrix of each layer by the first length of the all-1 vector is equal to the importance vector of each neuron in the layer of the randomly initialized translation model.

[0022] The transpose of the transfer matrix of each layer multiplied by the full 1 vector of the second length is equal to the importance vector of each neuron in this layer of the pre-trained translation model.

[0023] The second length and the first length are respectively the number of neurons in the layer of the randomly initialized translation model and the number of neurons in the layer of the pre-trained translation model.

[0024] In some embodiments, a method for determining the optimal transmission matrix for each layer includes:

[0025] Calculate the total cost of each layer's cost matrix by multiplying the transmission matrices of each layer that meet the requirements;

[0026] The target transmission matrix of the layer that minimizes the total cost is determined as the optimal transmission matrix of the layer.

[0027] In some embodiments, a method for determining the cost matrix of each layer includes:

[0028] A distance matrix formed by the distance information between the activation values ​​of each neuron in each layer of the randomly initialized translation model and the activation values ​​of each neuron in the layer of the pre-trained translation model is determined as the cost matrix of the layer.

[0029] In some embodiments, the importance vectors of the neurons in each layer of the randomly initialized translation model are uniformly distributed; or, the importance vectors of the neurons in each layer of the pre-trained translation model are uniformly distributed.

[0030] In some embodiments, after right-multiplying the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, right-multiplying it by the first diagonal matrix for normalization, the diagonal elements of the first diagonal matrix are: the reciprocal of the number of neurons in the previous layer of the pre-trained translation model.

[0031] In some embodiments, after left-multiplying the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model, it is then left-multiplied by a second diagonal matrix for normalization, and the diagonal elements of the second diagonal matrix are: the reciprocal of the number of neurons in the current layer of the pre-trained translation model.

[0032] In some embodiments, fusing the aligned randomly initialized translation model and the pre-trained translation model to obtain a fused translation model includes:

[0033] The model parameters of the aligned randomly initialized translation model and the pre-trained translation model at the corresponding position are weighted summed to determine the fusion parameters of the fusion translation model at that position.

[0034] In some embodiments, determining a final translation model for performing the translation task based on the fused translation model includes: training the fused translation model using a translation dataset to obtain the final translation model.

[0035] In some embodiments, the network model comprises a Transformer model.

[0036] Some embodiments of the present disclosure provide a device for determining a translation model, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a method for determining a translation model based on instructions stored in the memory.

[0037] Some embodiments of the present disclosure provide a device for determining a translation model, including:

[0038] An initialization module is configured to initialize the network model with random parameters and pre-trained parameters to obtain a random initialization model and a pre-trained model respectively;

[0039] A training module is configured to use the translation dataset to train the randomly initialized model and the pre-trained model to obtain a randomly initialized translation model and a pre-trained translation model respectively;

[0040] an alignment module configured to align the randomly initialized translation model to the pre-trained translation model to obtain an aligned randomly initialized translation model;

[0041] A fusion module is configured to fuse the aligned randomly initialized translation model and the pre-trained translation model to obtain a fused translation model;

[0042] The model determination module is configured to determine a final translation model for performing the translation task based on the fusion translation model.

[0043] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the method for determining a translation model are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The following briefly introduces the drawings required for describing the embodiments or related technologies. The present disclosure can be more clearly understood based on the following detailed description with reference to the drawings.

[0045] Obviously, the drawings described below are only some embodiments of the present disclosure. A person skilled in the art can derive other drawings based on these drawings without inventive effort.

[0046] FIG1 is a schematic flow chart showing a method for determining a translation model according to some embodiments of the present disclosure.

[0047] FIG2 is a schematic diagram showing the translation effects of a randomly initialized translation model and a pre-trained translation model according to some embodiments of the present disclosure.

[0048] FIG3 is a schematic diagram showing a layer-by-layer alignment of a randomly initialized translation model to a pre-trained translation model according to some embodiments of the present disclosure.

[0049] FIG4 shows a schematic diagram of obtaining a fused translation model by fusing an aligned randomly initialized translation model and a pre-trained translation model in some embodiments of the present disclosure.

[0050] FIG5 shows a schematic structural diagram of a device for determining a translation model according to some embodiments of the present disclosure.

[0051] FIG6 shows a schematic structural diagram of a device for determining a translation model according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure.

[0053] Unless otherwise specified, descriptions such as “first” and “second” in the present disclosure are used to distinguish different objects and are not used to indicate meanings such as size or time sequence.

[0054] To alleviate the problems of poor translation performance of translation models trained with pretrained parameter initialization and poor generalization of translation models trained with random parameter initialization in high-resource translation tasks, the disclosed embodiments align a randomly initialized translation model with a pretrained translation model to obtain an aligned randomly initialized translation model. This aligned randomly initialized translation model and the pretrained translation model are then fused to form a fused translation model, which is then used to generate a final translation model. This model fusion effectively combines the advantages of translation models trained with both pretrained parameter initialization and random parameter initialization, while simultaneously improving the translation performance and generalization of the translation model.

[0055] FIG1 is a schematic flow chart showing a method for determining a translation model according to some embodiments of the present disclosure.

[0056] As shown in FIG. 1 , the method for determining a translation model in this embodiment includes steps 110 - 150 .

[0057] In step 110, the network model is initialized with random parameters and pre-trained parameters to obtain a randomly initialized model (110a) and a pre-trained model (110b), respectively.

[0058] In some embodiments, the network model includes, but is not limited to, a neural network model such as a Transformer model.

[0059] Among them, the pre-training parameters are universal network parameters obtained by training through self-supervision methods on large-scale monolingual data. Correspondingly, the network model endowed with universal network parameters is the upstream universal model, that is, the pre-training model.

[0060] In step 120, the randomly initialized model and the pre-trained model are trained using the translation dataset to obtain a randomly initialized translation model (120a) and a pre-trained translation model (120b).

[0061] In some embodiments, using a translation dataset to train a randomly initialized model to obtain a randomly initialized translation model (120a) includes, for example: (1) randomly sampling multiple sentence pairs (source language sentences, target language sentences) from the translation dataset, and splicing them to form a batch of training samples; (2) inputting a batch of training samples into the randomly initialized model, and calculating the cross entropy loss between the sentences output by the randomly initialized model and the target language sentences based on a loss function (such as a cross entropy loss function); (3) based on the derivative of the loss function, propagating the error along the direction of minimum gradient and updating the network parameters of the randomly initialized model; (4) continuously repeating the first three steps until the loss of the model on the validation set is minimized, and determining the trained randomly initialized model as the randomly initialized translation model.

[0062] In some embodiments, using a translation dataset to train a pre-trained model to obtain a pre-trained translation model (120b) includes, for example: (1) randomly sampling multiple sentence pairs (source language sentences, target language sentences) from the translation dataset, and splicing them to form a batch of training samples; (2) inputting a batch of training samples into the pre-trained model, and calculating the cross-entropy loss between the sentences output by the pre-trained model and the target language sentences based on a loss function (such as a cross-entropy loss function); (3) based on the derivative of the loss function, returning the error along the direction of minimum gradient and updating the network parameters of the pre-trained model; (4) continuously repeating the first three steps until the loss of the model on the validation set is minimized, and determining the trained pre-trained model as the pre-trained translation model.

[0063] Taking the Transformer model as an example, the randomly initialized translation model and the pre-trained translation model are the randomly initialized Transformer model and the pre-trained Transformer model respectively.

[0064] Both randomly initialized and pretrained translation models can translate source language sentences into target language sentences. As shown in Figure 2, both randomly initialized translation models (such as the randomly initialized Transformer model) and pretrained translation models (such as the pretrained Transformer model) can translate Chinese sentences into English sentences with the same meaning.

[0065] In step 130 , the randomly initialized translation model is aligned to the pre-trained translation model to obtain an aligned randomly initialized translation model.

[0066] In some embodiments, aligning the randomly initialized translation model to the pre-trained translation model to obtain the aligned randomly initialized translation model includes: aligning each layer of the randomly initialized translation model to the corresponding layer of the pre-trained translation model using a transfer mapping method to obtain the aligned randomly initialized translation model.

[0067] In some embodiments, aligning each layer of the randomly initialized translation model to the corresponding layer of the pre-trained translation model using a transfer mapping method includes: right-multiplying the input weight matrix of each current layer of the randomly initialized translation model by the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, and then left-multiplying it by the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model, to obtain the aligned input weight matrix of the current layer of the randomly initialized translation model.

[0068] The input weight matrix of the current layer of the randomly initialized translation model refers to the weight matrix of the full connection between each neuron of the current layer of the randomly initialized translation model and each neuron of the previous layer.

[0069] The transfer matrix of each layer refers to the correspondence matrix between the neurons of each layer of the randomly initialized translation model and the neurons of the layer of the pre-trained translation model.

[0070] In some embodiments, after right-multiplying the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, right-multiplying it by the first diagonal matrix for normalization, the diagonal elements of the first diagonal matrix are: the reciprocal of the number of neurons in the previous layer of the pre-trained translation model.

[0071] In some embodiments, after left-multiplying the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model, it is then left-multiplied by a second diagonal matrix for normalization, and the diagonal elements of the second diagonal matrix are: the reciprocal of the number of neurons in the current layer of the pre-trained translation model.

[0072] The requirements that the transfer matrix of each layer must meet include: the right-multiplication of the transfer matrix of each layer by an all-ones vector of a first length is equal to the importance vector of each neuron in that layer of the randomly initialized translation model; the right-multiplication of the transpose of the transfer matrix of each layer by an all-ones vector of a second length is equal to the importance vector of each neuron in that layer of the pre-trained translation model, where the second length is the number of neurons in that layer of the randomly initialized translation model and the first length is the number of neurons in that layer of the pre-trained translation model. The importance vectors of each neuron in each layer of the randomly initialized translation model are uniformly distributed. The importance vectors of each neuron in each layer of the pre-trained translation model are uniformly distributed.

[0073] In some embodiments, the method for determining the optimal transmission matrix of each layer includes: calculating the total cost of each transmission matrix of each layer that meets the requirements and multiplying the cost matrix of the layer; and determining the target transmission matrix of the layer that minimizes the total cost as the optimal transmission matrix of the layer.

[0074] In some embodiments, a method for determining the cost matrix of each layer includes: determining a distance matrix formed by distance information between activation values ​​of each neuron in each layer of a randomly initialized translation model and activation values ​​of each neuron in the layer of a pre-trained translation model as the cost matrix of the layer.

[0075] In order to make the expression of the above solution clearer, the formula expression of the above solution is given below.

[0076] For the randomly initialized translation model (set as RI) the network layer l ri , define a discrete probability measure based on activation values in, (abbreviated as α) represents the network layer l ri The importance vectors of each neuron are evenly distributed. Indicates the network layer l ri The activation values ​​of each neuron.

[0077] For the network layer l of the pre-trained translation model (set as PT) pt , define a discrete probability measure based on activation values (abbreviated as β) represents the network layer l pt The importance vectors of each neuron are evenly distributed. Indicates the network layer l pt The activation values ​​of each neuron.

[0078] The input weight matrix of each current layer l of the randomly initialized translation model (the weight matrix of the full connection between each neuron in the current layer l of the randomly initialized translation model and each neuron in the previous layer (l-1)) Right multiply the transfer matrix T from the previous layer (l-1) of the randomly initialized translation model to the previous layer (l-1) of the pre-trained translation model l-1 , then right multiply the first diagonal matrix Normalize and get the intermediate result The elements on the diagonal of the first diagonal matrix are: the reciprocal of the number of neurons in the previous layer of the pre-trained translation model β(l-1) Then multiply the transfer matrix T from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model by l The transpose of T l′ , then multiply the second diagonal matrix on the left Normalized, the elements on the diagonal of the second diagonal matrix are: the reciprocal of the number of neurons β(l) in the current layer of the pre-trained translation model Get the input weight matrix of the current layer of the aligned randomly initialized translation model Please refer to formula (1) and formula (2) for details.

[0079] According to the optimal transmission objective function, the optimal transmission matrix of any layer (assuming it is the lth layer) is calculated, see formula (3):

[0080] Among them, <.,.> F represents the Frobenius product, C is a predefined cost matrix, represents all possible transmission matrices on two measures, represents a vector of all 1s of length m, where m represents the number of neurons in layer l of the pre-trained translation model. represents an all-one vector of length n, n represents the number of neurons in layer l of the randomly initialized translation model, α represents the importance vector of each neuron in layer l of the randomly initialized translation model, showing a uniform distribution, β represents the importance vector of each neuron in layer l of the pre-trained translation model, showing a uniform distribution, and OT is the calculated optimal transmission matrix.

[0081] The method for determining the cost matrix C of any layer (assuming it is the lth layer) includes: determining the distance matrix formed by the distance information (such as Euclidean distance) between the activation values ​​of each neuron in the lth layer of the randomly initialized translation model and the activation values ​​of each neuron in the lth layer of the pre-trained translation model as the cost matrix C of the lth layer, and each element C in the cost matrix ij is the cost from the i-th neuron in layer l of the randomly initialized translation model to the j-th neuron in layer l of the pre-trained translation model (that is, the Euclidean distance between the activation value of the i-th neuron in layer l of the randomly initialized translation model and the activation value of the j-th neuron in layer l of the pre-trained translation model).

[0082] For a better understanding, here is an example: the number of neurons in the previous layer of model RI and model PT is 3 and 2 respectively, and the number of neurons in the current layer is 5 and 4 respectively. Then the size of the input weight matrix RI is (5,3) and PT is (4,2). In order to align the RI model with the PT model, the size of RI needs to be transformed to be consistent with PT. The specific operation is: (1) Multiply the transmission matrix of the previous layer (size is (3,2)) by right, and get it after normalization The size is (5*2); (2) Left multiply the transpose of the current layer's transfer matrix and normalize it to get The size becomes the required (4,2).

[0083] Figure 3 illustrates a schematic diagram of layer-by-layer alignment of a randomly initialized translation model to a pre-trained translation model in accordance with some embodiments of the present disclosure. As shown in Figure 3, the randomly initialized translation model is aligned layer-by-layer to the pre-trained translation model, ultimately yielding the aligned randomly initialized translation model and the pre-trained translation model.

[0084] In step 140 , the aligned randomly initialized translation model and the pre-trained translation model are fused to obtain a fused translation model.

[0085] Figure 4 shows a schematic diagram of a fused translation model obtained by fusing an aligned randomly initialized translation model and a pre-trained translation model in some embodiments of the present disclosure. As shown in Figure 4 , the aligned randomly initialized translation model has been transformed to the same size as the pre-trained translation model, and the parameters at the same position in the two models correspond. Therefore, the model parameters at corresponding positions in the aligned randomly initialized translation model and the pre-trained translation model are weighted and summed to determine the fused parameters of the fused translation model at that position. The weighted values ​​are hyperparameters and can be manually adjusted to achieve this.

[0086] For example, fused_model = RI , *0.1+PT*0.9, where fused_model represents the fused translation model, RI , represents the randomly initialized translation model after alignment, PT represents the pre-trained translation model, and 0.1 and 0.9 represent the weighted values ​​of the two models, respectively.

[0087] At step 150 , a final translation model for performing the translation task is determined based on the fused translation model.

[0088] In some embodiments, determining a final translation model for performing the translation task based on the fusion translation model includes: training the fusion translation model using a translation dataset, and fine-tuning the fusion translation model to obtain the final translation model.

[0089] The disclosed embodiment aligns a randomly initialized translation model with a pretrained translation model to obtain an aligned randomly initialized translation model, then fuses the aligned randomly initialized translation model with the pretrained translation model to obtain a fused translation model, and then generates a final translation model based on the fused translation model. Thus, through model fusion, the advantages of both pretrained and random parameter initialization methods are effectively combined, while simultaneously improving the translation performance and generalization of the translation model. This alleviates the issues of poor translation performance of translation models trained with pretrained parameter initialization and poor generalization of translation models trained with random parameter initialization in high-resource translation tasks.

[0090] FIG5 shows a schematic structural diagram of a device for determining a translation model according to some embodiments of the present disclosure.

[0091] As shown in FIG5 , the translation model determination device 500 of this embodiment includes:

[0092] An initialization module 510 is configured to initialize the network model using random parameters and pre-trained parameters to obtain a random initialization model and a pre-trained model respectively;

[0093] A training module 520 is configured to train the randomly initialized model and the pre-trained model using the translation dataset to obtain a randomly initialized translation model and a pre-trained translation model respectively;

[0094] An alignment module 530 is configured to align the randomly initialized translation model to the pre-trained translation model to obtain an aligned randomly initialized translation model;

[0095] A fusion module 540 is configured to fuse the aligned randomly initialized translation model and the pre-trained translation model to obtain a fused translation model;

[0096] The model determination module 550 is configured to determine a final translation model for performing the translation task based on the fusion translation model.

[0097] In some embodiments, the alignment module 530 is configured to align each layer of the randomly initialized translation model to the corresponding layer of the pre-trained translation model using a transfer mapping method to obtain an aligned randomly initialized translation model.

[0098] In some embodiments, the alignment module 530 is configured to: multiply the input weight matrix of each current layer of the randomly initialized translation model by the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, and then multiply it by the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model, to obtain an aligned input weight matrix of the current layer of the randomly initialized translation model, where:

[0099] The input weight matrix of the current layer of the randomly initialized translation model refers to the weight matrix of the full connection between each neuron in the current layer of the randomly initialized translation model and each neuron in the previous layer.

[0100] The transfer matrix of each layer refers to the correspondence matrix between the neurons of each layer of the randomly initialized translation model and the neurons of the layer of the pre-trained translation model.

[0101] The transmission matrix of each layer must meet the following requirements:

[0102] The right multiplication of the transmission matrix of each layer by the first length of the all-1 vector is equal to the importance vector of each neuron in the layer of the randomly initialized translation model.

[0103] The transpose of the transfer matrix of each layer multiplied by the full 1 vector of the second length is equal to the importance vector of each neuron in this layer of the pre-trained translation model.

[0104] The second length and the first length are respectively the number of neurons in the layer of the randomly initialized translation model and the number of neurons in the layer of the pre-trained translation model.

[0105] The method for determining the optimal transmission matrix of each layer includes:

[0106] Calculate the total cost of each layer's cost matrix by multiplying the transmission matrices of each layer that meet the requirements;

[0107] The target transmission matrix of the layer that minimizes the total cost is determined as the optimal transmission matrix of the layer.

[0108] The method for determining the cost matrix of each layer includes:

[0109] A distance matrix formed by the distance information between the activation values ​​of each neuron in each layer of the randomly initialized translation model and the activation values ​​of each neuron in the layer of the pre-trained translation model is determined as the cost matrix of the layer.

[0110] The importance vectors of the neurons in each layer of the randomly initialized translation model are uniformly distributed; or the importance vectors of the neurons in each layer of the pre-trained translation model are uniformly distributed.

[0111] In some embodiments, the alignment module 530 is configured to: after right-multiplying the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, right-multiply by the first diagonal matrix for normalization, where the diagonal elements of the first diagonal matrix are: the reciprocal of the number of neurons in the previous layer of the pre-trained translation model.

[0112] In some embodiments, the alignment module 530 is configured to: after left-multiplying the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model, left-multiply by a second diagonal matrix for normalization, where the diagonal elements of the second diagonal matrix are: the reciprocal of the number of neurons in the current layer of the pre-trained translation model.

[0113] In some embodiments, the fusion module 540 is configured to perform a weighted summation of the model parameters of the aligned randomly initialized translation model and the pre-trained translation model at corresponding positions to determine the fusion parameters of the fused translation model at the position.

[0114] In some embodiments, the model determination module 550 is configured to train the fusion translation model using the translation dataset to obtain a final translation model.

[0115] FIG6 shows a schematic structural diagram of a device for determining a translation model according to some embodiments of the present disclosure.

[0116] As shown in FIG6 , the apparatus 600 for determining a translation model of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610 . The processor 620 is configured to execute the method for determining a translation model in any of the aforementioned embodiments based on instructions stored in the memory 610 .

[0117] The apparatus 600 may further include an input / output interface 630 , a network interface 640 , a storage interface 650 , etc. These interfaces 630 , 640 , 650 , the memory 610 , and the processor 620 may be connected via a bus 660 , for example.

[0118] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0119] The processor 620 may be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, or discrete hardware components such as discrete gates or transistors.

[0120] Among them, the input and output interface 630 provides a connection interface for input and output devices such as a display, mouse, keyboard, and touch screen. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 660 can use any of a variety of bus structures. For example, bus structures include but are not limited to the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0121] The embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the method for determining a translation model of each embodiment are implemented.

[0122] (1) A method for determining a translation model, comprising:

[0123] Initialize the network model with random parameters and pre-trained parameters to obtain a random initialized model and a pre-trained model respectively;

[0124] The randomly initialized model and the pre-trained model are trained using the translation dataset to obtain the randomly initialized translation model and the pre-trained translation model respectively;

[0125] Align the randomly initialized translation model to the pre-trained translation model to obtain an aligned randomly initialized translation model;

[0126] The aligned randomly initialized translation model and the pre-trained translation model are fused to obtain a fused translation model.

[0127] A final translation model for performing the translation task is determined based on the fused translation model.

[0128] (2) According to (1), the randomly initialized translation model is aligned to the pre-trained translation model to obtain the aligned randomly initialized translation model including:

[0129] Each layer of the randomly initialized translation model is aligned to the corresponding layer of the pre-trained translation model using the transfer mapping method to obtain the aligned randomly initialized translation model.

[0130] (3) According to (2), each layer of the randomly initialized translation model is aligned to the corresponding layer of the pre-trained translation model using the transfer mapping method, including:

[0131] The input weight matrix of each current layer of the randomly initialized translation model is right-multiplied by the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, and then left-multiplied by the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model to obtain the aligned input weight matrix of the current layer of the randomly initialized translation model, where:

[0132] The input weight matrix of the current layer of the randomly initialized translation model refers to the weight matrix of the full connection between each neuron in the current layer of the randomly initialized translation model and each neuron in the previous layer.

[0133] The transfer matrix of each layer refers to the correspondence matrix between the neurons of each layer of the randomly initialized translation model and the neurons of the layer of the pre-trained translation model.

[0134] (4) According to (3), the requirements that the transmission matrix of each layer must meet include:

[0135] The right multiplication of the transmission matrix of each layer by the first length of the all-1 vector is equal to the importance vector of each neuron in the layer of the randomly initialized translation model.

[0136] The transpose of the transfer matrix of each layer multiplied by the full 1 vector of the second length is equal to the importance vector of each neuron in this layer of the pre-trained translation model.

[0137] The second length and the first length are respectively the number of neurons in the layer of the randomly initialized translation model and the number of neurons in the layer of the pre-trained translation model.

[0138] (5) According to (4), the method for determining the optimal transmission matrix of each layer includes:

[0139] Calculate the total cost of each layer's cost matrix by multiplying the transmission matrices of each layer that meet the requirements;

[0140] The target transmission matrix of the layer that minimizes the total cost is determined as the optimal transmission matrix of the layer.

[0141] (6) According to (5), the method for determining the cost matrix of each layer includes:

[0142] A distance matrix formed by the distance information between the activation values ​​of each neuron in each layer of the randomly initialized translation model and the activation values ​​of each neuron in the layer of the pre-trained translation model is determined as the cost matrix of the layer.

[0143] (7) According to (4) or (5) or (6), the importance vectors of the neurons in each layer of the randomly initialized translation model are uniformly distributed; or, the importance vectors of the neurons in each layer of the pre-trained translation model are uniformly distributed.

[0144] (8) According to any one of (3)-(7), after right-multiplying the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, right-multiply by the first diagonal matrix for normalization, where the diagonal elements of the first diagonal matrix are the reciprocal of the number of neurons in the previous layer of the pre-trained translation model.

[0145] (9) According to any one of (3)-(8), after left-multiplying the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model, left-multiplying the matrix by the second diagonal matrix for normalization, the diagonal elements of the second diagonal matrix are: the reciprocal of the number of neurons in the current layer of the pre-trained translation model.

[0146] (10) According to any one of (1) to (9), fusing the aligned randomly initialized translation model and the pre-trained translation model to obtain a fused translation model includes: performing weighted summation on the model parameters of the aligned randomly initialized translation model and the pre-trained translation model at corresponding positions to determine the fusion parameters of the fused translation model at the position.

[0147] (11) According to any one of (1) to (10), determining a final translation model for performing a translation task based on the fusion translation model includes: training the fusion translation model using a translation dataset to obtain a final translation model.

[0148] (12) According to any one of (1) to (11), the network model includes a Transformer model.

[0149] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more non-transitory computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0150] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0151] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0153] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for determining a translation model, comprising: Initialize the network model with random parameters and pre-trained parameters to obtain a random initialization model and a pre-trained model respectively; The randomly initialized model and the pre-trained model are trained using the translation data set to obtain a randomly initialized translation model and a pre-trained translation model respectively; Aligning the randomly initialized translation model to the pre-trained translation model to obtain an aligned randomly initialized translation model; The aligned randomly initialized translation model and the pre-trained translation model are fused to obtain a fused translation model; A final translation model for performing the translation task is determined based on the fused translation model.

2. According to the method of claim 1, aligning the randomly initialized translation model to the pre-trained translation model to obtain the aligned randomly initialized translation model comprises: Each layer of the randomly initialized translation model is aligned to the corresponding layer of the pre-trained translation model using the transfer mapping method to obtain the aligned randomly initialized translation model.

3. According to the method of claim 2, aligning each layer of the randomly initialized translation model to the corresponding layer of the pre-trained translation model by a transfer mapping method comprises: The input weight matrix of each current layer of the randomly initialized translation model is right-multiplied by the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, and then left-multiplied by the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model to obtain the aligned input weight matrix of the current layer of the randomly initialized translation model, where: The input end weight matrix of the current layer of the randomly initialized translation model refers to the weight matrix of the full connection between each neuron of the current layer of the randomly initialized translation model and each neuron of the previous layer. The transfer matrix of each layer refers to: the correspondence matrix between each neuron of each layer of the randomly initialized translation model and each neuron of the layer of the pre-trained translation model.

4. According to the method of claim 3, the requirements that the transmission matrix of each layer meets include: The transfer matrix of each layer is multiplied by the first length of the all-1 vector, which is equal to the random initialization of the translation model. The importance vector of the neuron, The transpose of the transfer matrix of each layer multiplied by the all-1 vector of the second length is equal to the importance vector of each neuron in this layer of the pre-trained translation model. The second length and the first length are respectively the number of neurons in the layer of the randomly initialized translation model and the number of neurons in the layer of the pre-trained translation model.

5. According to the method of claim 4, the method for determining the optimal transmission matrix of each layer comprises: Calculate the total cost of each layer's cost matrix by multiplying each transmission matrix that meets the requirements; The target transmission matrix of the layer that minimizes the total cost is determined as the optimal transmission matrix of the layer.

6. According to the method of claim 5, the method for determining the cost matrix of each layer comprises: A distance matrix formed by the distance information between the activation values ​​of each neuron in each layer of the randomly initialized translation model and the activation values ​​of each neuron in the layer of the pre-trained translation model is determined as the cost matrix of the layer.

7. The method according to any one of claims 4 to 6, wherein: The importance vectors of each neuron in each layer of the randomly initialized translation model are uniformly distributed; or, The importance vectors of each neuron in each layer of the pre-trained translation model are evenly distributed.

8. The method according to any one of claims 3 to 7, further comprising: After right-multiplying the transfer matrix from the previous layer of the randomly initialized translation model to the previous layer of the pre-trained translation model, right-multiply it by the first diagonal matrix for normalization. The diagonal elements of the first diagonal matrix are: the reciprocal of the number of neurons in the previous layer of the pre-trained translation model.

9. The method according to any one of claims 3 to 8, further comprising: After left-multiplying the transpose of the transfer matrix from the current layer of the randomly initialized translation model to the current layer of the pre-trained translation model, left-multiply the second diagonal matrix for normalization, where the diagonal elements of the second diagonal matrix are the reciprocal of the number of neurons in the current layer of the pre-trained translation model.

10. According to the method according to any one of claims 1 to 9, fusing the aligned randomly initialized translation model and the pre-trained translation model to obtain a fused translation model comprises: The model parameters of the aligned randomly initialized translation model and the pre-trained translation model at the corresponding position are weighted summed to determine the fusion parameters of the fusion translation model at the position.

11. According to the method according to any one of claims 1 to 10, determining the final translation model for performing the translation task based on the fusion translation model comprises: The fusion translation model is trained using the translation dataset to obtain the final translation model.

12. According to the method according to any one of claims 1-11, the network model comprises a Transformer model.

13. A translation model determination device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the method for determining a translation model according to any one of claims 1 to 12 based on instructions stored in the memory.

14. A device for determining a translation model, comprising: An initialization module is configured to respectively initialize the network model with random parameters and pre-trained parameters to obtain a random initialization model and a pre-trained model; A training module is configured to use the translation data set to train the random initialization model and the pre-trained model to obtain a random initialization translation model and a pre-trained translation model respectively; an alignment module, configured to align the randomly initialized translation model to the pre-trained translation model to obtain an aligned randomly initialized translation model; A fusion module, configured to fuse the aligned randomly initialized translation model and the pre-trained translation model to obtain a fused translation model; The model determination module is configured to determine a final translation model for performing the translation task based on the fusion translation model.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for determining a translation model according to any one of claims 1 to 12.

16. A computer program comprising: Instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1-12.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Citation Information

Patent Citations

  • Mongolian-Chinese translation method based on transfer learning

    CN110688862A

  • Scarce resource neural machine translation training method based on pre-training

    CN111178094A

  • Neural machine translation method based on pre-training bilingual word vector

    CN113297841A

  • Machine translation method based on context recognition

    CN114118113A

  • Multi-language translation model construction method and storage medium

    CN115688815A