Learning representations of cross-lingual text

Abstract

The present disclosure proposes a method and apparatus for representation learning of cross-language texts. A source language text and a target language text may be obtained. An initial joint representation of the source language text and the target language text may be generated. A relationship between a plurality of words in the source language text and the target language text may be identified. A joint representation of the source language text and the target language text may be generated based on the initial joint representation and the relationship. The joint representation may be mapped to at least a target language representation corresponding to the target language text.

Description

Background Art

[0001] Natural Language Understanding (NLU) is a technology that uses natural language to communicate with computers. It aims to enable computers to understand and use natural language to achieve communication between humans and computers, thereby replacing humans to perform various tasks related to natural language, such as machine reading comprehension tasks, classification tasks, question-answering tasks, etc. NLU tasks can be performed by trained machine learning models. The performance of machine learning models in performing NLU tasks depends on a large amount of reliable training data. For resource-rich languages ​​such as English, there is large-scale human-annotated training data for some NLU tasks. Therefore, these NLU tasks have excellent performance on resource-rich languages. Summary of the invention

[0002] This Summary is provided to introduce a group of concepts that will be further described in the following Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure propose a method and apparatus for representation learning of cross-language texts. A source language text and a target language text may be obtained. An initial joint representation of the source language text and the target language text may be generated. A relationship between a plurality of words in the source language text and the target language text may be identified. A joint representation of the source language text and the target language text may be generated based on the initial joint representation and the relationship. The joint representation may be mapped to at least a target language representation corresponding to the target language text.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. The following description and drawings set forth certain illustrative features of the one or more aspects in detail. These features are merely indicative of the various ways in which the principles of various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0006] Figure 1 is a schematic diagram showing an exemplary source language text, a target language text, and the relationship between them according to an embodiment of the present disclosure.

[0007] Figure 2An exemplary process for constructing a graph corresponding to a source language text and a target language text according to an embodiment of the present disclosure is shown.

[0008] Figure 3 An exemplary diagram according to an embodiment of the present disclosure is shown.

[0009] Figure 4 An exemplary process for representation learning of cross-language text according to an embodiment of the present disclosure is shown.

[0010] Figure 5 An exemplary process for updating a previous joint representation according to an embodiment of the present disclosure is shown.

[0011] Figure 6 An exemplary process for obtaining current attention information according to an embodiment of the present disclosure is shown.

[0012] Figure 7 An exemplary process for pre-training a representation acquisition model according to an embodiment of the present disclosure is shown.

[0013] Figure 8 An exemplary diagram of a shielding strategy according to an embodiment of the present disclosure is shown.

[0014] Fig. 9 is a flowchart of an exemplary method for representation learning of cross-language text according to an embodiment of the present disclosure.

[0015] Fig.10 An exemplary apparatus for representation learning of cross-language texts according to an embodiment of the present disclosure is shown.

[0016] Fig.11 An exemplary apparatus for representation learning of cross-language texts according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that the discussion of these embodiments is only used to enable those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0018] It is desirable to extend NLU tasks such as machine reading comprehension tasks, classification tasks, question-answering tasks, etc. to resource-scarce languages, such as German, Spanish, French, etc. However, for resource-scarce languages, there are only a few or even no reliable training data, which restricts the performance of machine learning models when performing NLU tasks for resource-scarce languages. The problem of lack of training data for resource-scarce languages ​​can be solved by cross-lingual NLU. Cross-lingual NLU can transfer the knowledge learned from the text of resource-rich languages ​​to the text of resource-scarce languages ​​by means of machine translation to help enhance the understanding of the text of resource-scarce languages, so as to obtain a more accurate representation of the text of resource-scarce languages, and further improve the performance of performing NLU tasks for the text of the resource-scarce language. In this article, the resource-rich language that provides knowledge can be referred to as the source language, and the resource-scarce language that learns knowledge can be referred to as the target language. Accordingly, the text whose language is the source language can be referred to as the source language text, and the text whose language is the target language can be referred to as the target language text. There are some existing cross-lingual NLU methods. For example, the training data in the source language for a specific NLU task can be translated into training data in the target language, and the translated training data in the target language can be used to train the machine learning model. When the trained machine learning model is actually deployed, it can generate a representation of the target language text and use the generated representation to perform the specific NLU task.

[0019] Embodiments of the present disclosure propose an improved method for obtaining a representation of a target language text through cross-language NLU. In this article, representation can refer to a set of information generated based on raw data in a form that is conducive to machine learning model processing. A source language text and a target language text can be obtained, and a joint representation of the source language text and the target language text is generated based on the relationship between multiple words in the source language text and the target language text, and the joint representation is mapped to a representation corresponding to the source language text and / or a representation corresponding to the target language text. In this article, a word can broadly refer to a basic language unit that constitutes a text in different languages, and the relationship between words can refer to an association based on a predetermined criterion between words. The relationship between multiple words in the source language text and the target language text can include, for example, an alignment relationship between a word in the source language text and a word in the target language text. Two words with an alignment relationship can generally have similar semantic meanings. Considering the alignment relationship when generating a joint representation of the source language text and the target language text can help understand the semantics of each word in the source language text and the target language text. In addition, the relationship between the multiple words in the source language text and the target language text may include a dependency relationship related to the syntactic structure of each text, such as the dependency relationship between the multiple words in the source language text and the dependency relationship between the multiple words in the target language text. The correct alignment relationship between the source language text and the target language text can be found by means of the dependency relationship. Considering the alignment relationship and the dependency relationship when generating the representation of the source language text and the target language text can enhance the knowledge transfer between the source language text and the target language text to generate a better joint representation of the source language text and the target language text, thereby obtaining a more accurate representation corresponding to the source language text and / or a more accurate representation corresponding to the target language text. Through the method according to the embodiment of the present disclosure, both the representation of the source language text and the representation of the target language text can be improved, so the method can also be referred to as a representation learning method for cross-language texts. The representation of the source language text and / or the representation of the target language text can be further used to perform various types of NLU tasks. When performing tasks using the source language representation and / or the target language representation, a more accurate representation can help obtain more accurate results.

[0020] In one aspect, the embodiments of the present disclosure propose to explicitly model the relationship between multiple words in the source language text and the target language text and the multiple words by constructing a graph corresponding to the source language text and the target language text, so that the semantic association between these words can be better captured. In the graph, the words in the source language text and the target language text can be represented by nodes, and the relationship between the words can be represented by the edges between the nodes. The edges between the nodes can correspond to the relationship between the words. For example, when the relationship between the two words is an alignment relationship, the edge between the two nodes corresponding to the two words can be an alignment edge; and when the relationship between the two words is a dependency relationship, the edge between the two nodes corresponding to the two words can be a dependency edge. In this article, the node with the edge of the current node can be referred to as the neighbor node of the current node. For example, the node with the alignment edge with the current node can be the alignment neighbor node of the current node, and the node with the dependency edge with the current node can be the dependency neighbor node of the current node. Since the graph corresponding to the source language text and the target language text constructed according to the embodiment of the present disclosure integrates the dependency and alignment relationships related to the syntactic structure, the graph can also be called a Syntax-Enhanced and Alignment-Aware graph, or SA graph for short.

[0021] In another aspect, the embodiments of the present disclosure propose that when obtaining the representation of the source language text and the joint representation of the target language text, the representation of at least one neighbor node of each node corresponding to each word in the source language text and the target language text is obtained, and the representation of the node is updated using the representation of at least one neighbor node obtained, and the updated representations of each node can be combined into a joint representation. The neighbor nodes can be identified from the SA graph. In this way, the intrinsic semantic associations between neighbor nodes can be propagated via the edge connections in the SA graph, which helps to generate more accurate representations of each node.

[0022] In yet another aspect, an embodiment of the present disclosure proposes that when the representation of a neighboring node is used to update the representation of the node, when the neighboring node is a neighboring node with an alignment edge with the node, the representation of the node can be updated based on the semantic difference between the neighboring node and the node. For example, when the semantic difference between the neighboring node and the node is large, that is, when there is an alignment error, the weight of the representation of the neighboring node can be reduced when updating the representation of the node, thereby reducing the impact caused by the alignment error. In addition, when the neighboring node is a neighboring node with a dependency edge with the node, the representation of the node can be updated based on the importance of the neighboring node relative to the node. For example, when the neighboring node is more important relative to the node, when updating the representation of the node, a larger weight corresponding to the representation of the neighboring node can be given, thereby allowing the representation of the neighboring node to be considered more when updating the representation of the node.

[0023] In another aspect, the embodiment of the present disclosure proposes to obtain the representation of the source language text and the representation of the target language text through a model based on deep learning, and proposes a pre-training strategy that guides the model to learn the text representation from the semantic associations such as the alignment relationship embedded between the source language text and the target language text, the dependency relationship between multiple words in the source language text, and the dependency relationship between multiple words in the target language text. In this article, the model for obtaining the representation of the source language text and the representation of the target language text can be referred to as a representation acquisition model. In a pre-training strategy, one or more nodes in a node set in a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample can be masked, and for each node, at least the representation of the node's aligned neighbor node is used to restore the node to pre-train the representation acquisition model. The pre-training strategy can guide the representation acquisition model to learn text representation from the semantic association embedded in the alignment relationship. In another pre-training strategy, the representation acquisition model can be pre-trained by masking one or more node pairs with aligned edges in a source language node set corresponding to the source language text sample and a target language node set corresponding to the target language text sample, and for each node, restoring the node using the representation of the node's dependent neighbor node. The pre-training strategy can guide the representation acquisition model to learn text representations from semantic associations embedded in dependency relationships.

[0024] Figure 11 is a schematic diagram 100 showing an exemplary source language text, a target language text and the relationship therebetween according to an embodiment of the present disclosure. The source language text 102 may be a text "We should protect the environment" in English, which may include a set of source language words, namely "we", "should", "protect", "the" and "environment". In this article, the words included in the source language text, whose language is the source language, may be referred to as source language words. The target language text 104 may be a text "Wir sollten die Umwelt schützen" in German, which may include a set of target language words, namely "wir", "sollten", "die", "Umwelt" and "schützen". In this article, the words included in the target language text, whose language is the target language, may be referred to as target language words. The source language text 102 may be obtained by translating the target language text 104. Alternatively, the target language text 104 may be obtained by translating the source language text 102.

[0025] According to an embodiment of the present disclosure, the relationship between a plurality of words in the source language text 102 and the target language text 104 may be identified.

[0026] The alignment relationship between the source language words in the source language text 102 and the target language words in the target language text 104 can be identified. In one embodiment, if the source language word and the target language word have similar semantic meanings, there is an alignment relationship between the source language word and the target language word. The alignment relationship can be identified by known alignment recognition techniques such as GIZA++. In the schematic diagram 100, the alignment relationship between the source language words in the source language text 102 and the target language words in the target language text 104 is shown by a solid line. For example, there can be an alignment relationship between the source language word "we" and the target language word "wir", there can be an alignment relationship between the source language word "should" and the target language word "sollten", there can be an alignment relationship between the source language word "protect" and the target language word "schützen", there can be an alignment relationship between the source language word "the" and the target language word "die", and there can be an alignment relationship between the source language word "environment" and the target language word "Umwelt".

[0027] In addition, the dependency relationship between a group of words in the same language text can be identified, such as the dependency relationship between a group of source language words, the dependency relationship between a group of target language words, etc. In one embodiment, a syntax tree corresponding to the text can be constructed based on the syntactic structure of the text, and the dependency relationship between the two words can be determined based on the association between the two words in the syntax tree. The dependency relationship can be identified by known dependency recognition techniques such as Stanza. A syntax tree 106 corresponding to the source language text 102 and a syntax tree 108 corresponding to the target language text 104 are shown in the schematic diagram 100. In the syntax trees 106 and 108, the dependency relationship between the source language words in the source language text 102 and the dependency relationship between the target language words in the target language text 104 are shown by dotted lines, respectively. For example, there may be a dependency relationship between the source language word “protect” and the source language words “we”, “should” and “environment”, and there may be a dependency relationship between the source language word “environment” and the source language word “the”; and there may be a dependency relationship between the target language word “schützen” and the target language words “Umwelt”, “sollten” and “wir”, and there may be a dependency relationship between the target language word “Umwelt” and the target language word “die”.

[0028] It should be understood that the above Figure 1 The criteria and methods described for identifying the relationship between multiple words in the source language text and the target language text are only exemplary. According to actual application requirements, other criteria or methods can also be used to identify the relationship between words.

[0029] According to an embodiment of the present disclosure, multiple words in the source language text and the target language text and the relationship between the multiple words can be explicitly modeled by constructing a graph corresponding to the source language text and the target language text, so that the semantic association between the words can be better captured. Figure 2 An exemplary process 200 for constructing a graph corresponding to a source language text and a target language text according to an embodiment of the present disclosure is shown.

[0030] At 210 , a group of source language words in the source language text and a group of target language words in the target language text may be arranged into a plurality of nodes.

[0031] At 220, for each two nodes in the plurality of nodes, it may be determined whether there is a relationship between the two words corresponding to the two nodes. Figure 1As described, the relationship may include an alignment relationship between a source language word in a set of source language words and a target language word in a set of target language words. If the source language word and the target language word have similar semantic meanings, then there is a relationship between the source language word and the target language word, and the relationship is an alignment relationship. In addition, the relationship may also include a dependency relationship between a set of source language words and a dependency relationship between a set of target language words. A syntactic tree corresponding to the text may be constructed based on the syntactic structure of the text, and the relationship between the two words may be determined based on the association between the two words in the syntactic tree, and the relationship is a dependency relationship.

[0032] At 230, in response to determining that there is a relationship between two words, an edge corresponding to the relationship between two nodes corresponding to the two words may be determined. For example, when there is an alignment relationship between the two words, it may be determined that there is an alignment edge between the two nodes corresponding to the two words. In addition, when there is a dependency relationship between the two words, it may be determined that there is a dependency edge between the two nodes corresponding to the two words.

[0033] Steps 220 and 230 may be performed for every two nodes of the plurality of nodes corresponding to a set of source language words in the source language text and a set of target language words in the target language text. At 240, a set of edges between the plurality of nodes may be obtained. Therefore, through the exemplary process of steps 220 to 240 in process 200, a set of edges between the plurality of nodes may be determined based on the relationship between the plurality of words in the source language text and the target language text.

[0034] At 250, the plurality of nodes and the obtained set of edges may be combined into a graph corresponding to the source language text and the target language text. As described above, since the graph corresponding to the source language text and the target language text constructed according to the embodiment of the present disclosure incorporates dependency relationships and alignment relationships related to syntactic structures, the graph may also be referred to as a Syntax-Enhanced and Alignment-Aware graph, or SA graph for short.

[0035] It should be understood that Figure 2 The process 200 in is merely an example of a process for constructing an SA graph corresponding to a source language text and a target language text based on the relationship between multiple words in the source language text and the target language text. Depending on the actual application requirements, the process for constructing an SA graph may include any other steps, and may include more or fewer steps.

[0036] Figure 3 An exemplary SA graph 300 according to an embodiment of the present disclosure is shown. The SA graph 300 may be, for example, Figure 2 The process 200 in the Figure 1The SA graph corresponding to the source language text 102 and the target language text 104 in FIG.

[0037] Nodes 302 to 310 may correspond to the source language words "protect", "we", "should", "environment" and "the" in the source language text 102, respectively. In this article, the nodes corresponding to the source language words in the source language text may be referred to as source language nodes. In other words, nodes 302 to 310 may be referred to as source language nodes. Nodes 312 to 320 may correspond to the target language words "wir", "sollten", "Umwelt", "die", and "schützen" in the target language text 104, respectively. In this article, the nodes corresponding to the target language words in the target language text may be referred to as target language nodes. In other words, nodes 312 to 320 may be referred to as target language nodes.

[0038] In diagram 300, alignment edges between each of source language nodes 302 to 310 and corresponding target language nodes 312 to 320 are shown by solid lines. For example, there may be an alignment edge between source language node 302 and target language node 320, there may be an alignment edge between source language node 304 and target language node 312, there may be an alignment edge between source language node 306 and target language node 314, there may be an alignment edge between source language node 308 and target language node 316, and there may be an alignment edge between source language node 310 and target language node 318.

[0039] In addition, in the diagram 300, dependency edges between source language nodes 302 to 310 and dependency edges between target language nodes 312 to 320 are shown by dotted lines. For example, there may be dependency edges between source language node 302 and source language nodes 304 to 308, and there may be dependency edges between source language node 308 and source language node 310; and there may be dependency edges between target language node 320 and target language nodes 312 to 316, and there may be dependency edges between target language node 316 and target language node 318.

[0040] Each node can have one or more neighbor nodes. In this article, the node with the edge of the current node can be called the neighbor node of the current node. As previously mentioned, the edge can include an alignment edge and a dependent edge. Accordingly, the node with the alignment edge of the current node can be the alignment neighbor node of the current node, and the node with the dependent edge of the current node can be the dependent neighbor node of the current node. For example, node 302 can be the alignment neighbor node of node 320, and node 314 can be the dependent neighbor node of node 320.

[0041] It should be understood that Figure 3 The SA graph 300 shown in FIG. 3 is merely an example of a constructed SA graph corresponding to a source language text and a target language text. According to actual application requirements, the SA graph corresponding to a source language text and a target language text may have any other structure and may include other elements.

[0042] Figure 4 An exemplary process 400 for representation learning of cross-language texts according to an embodiment of the present disclosure is shown. For example, a source language representation 452 of a source language text 402 and a target language representation 462 of a target language text 404 may be obtained through a representation acquisition model 410. In this document, the representation corresponding to the source language text obtained through the representation acquisition model may be referred to as the source language representation, and the representation corresponding to the target language text obtained through the representation acquisition model may be referred to as the target language representation.

[0043] First, a source language text 402S and a target language text 404T may be obtained. A target language text T may be given, and then the source language text S may be obtained by translating the target language text T. Alternatively, a source language text S may be given, and then the target language text T may be obtained by translating the source language text S. The source language text S and the target language text T may include entries of different lengths. The lengths of the source language text S and the target language text T may be unified to 1 by operations such as padding or truncation. Source language texts with uniform lengths and target language text can be input into the representation acquisition model 410 in parallel, where d is the dimension of the token embedding vector.

[0044] The source language text S can be input into the transformer layer 420 in the representation acquisition model 410. The transformer layer 420 can generate an initial source language representation 422 of the source language text S. The target language text T can be input into the converter layer 430 in the representation acquisition model 410. The converter layer 430 can generate an initial target language representation 432 of the target language text T. The above processing can be expressed by the following formula:

[0045]

[0046]

[0047] Initial source language representation and the initial target language representation can be combined, e.g., concatenated, into an initial joint representation 434 Initial Joint Representation may be provided to a set of improved converter layers 440. The set of improved converter layers 440 may be based on the initial joint representation and the relationship between multiple words in the source language text S and the target language text T to generate a joint representation of the source language text S and the target language text T 442[A s ; A t In one embodiment, the SA graph 452 corresponding to the source language text S and the target language text T can be constructed by the graph construction module 450 based on the relationship between the multiple words in the source language text S and the target language text T. The graph construction module 450 can be, for example, Figure 2 The SA graph 452 is constructed by the process 200 in FIG. Further, a set of improved converter layers 440 can convert the initial joint representation into Updated to joint representation [A s ; A t ].

[0048] The update operation may be iteratively performed through a set of improved transformer layers 440 to transform the initial joint representation Updated to joint representation [A s ; A t ]. The group of improved converter layers 440 may include, for example, N improved converter layers 440-1, 440-2, ... 440-N having the same model structure. The improved converter layer 440-n (n∈[1,N]) may be a previous joint representation of the output of the previous improved converter layer 440-n-1. Update to current joint representation As shown in the following formula:

[0049]

[0050] Will be combined later Figure 5 To illustrate an exemplary process for updating a previous joint representation. s ; A t ], the joint representation [A s ; A t ] is mapped into a source language representation 452A corresponding to the source language text S s In addition, the joint representation [A s ; A t ] is mapped to a target language representation 462A corresponding to the target language text T t .

[0051] Source language representation A s Or the target language represents A tIt can be further used to perform various NLU tasks, such as machine reading comprehension tasks, classification tasks, question answering tasks, etc. t For example, it is used to perform machine reading comprehension tasks, which can find the answer fragment to a specific question from a given paragraph. The target language can be represented by A t Provided to two separate linear layers. After being processed by each linear layer, a softmax operation can be performed separately to produce the final prediction result of the answer segment, that is, the starting position prediction of the answer segment and end position prediction As shown in the following formula:

[0052]

[0053]

[0054] Among them, u s and and b s and are trainable model parameters. For example, when training for machine reading comprehension tasks, the parameter u can be learned by minimizing the standard cross entropy loss as shown below: s 、u e , b s and b e :

[0055]

[0056] in, is the number of training samples, and are the true annotations of the starting and ending positions of the i-th training sample respectively.

[0057] It should be understood that Figure 4 The process 400 in is merely an example of a process for representation learning of cross-language text. Depending on the actual application requirements, the process for representation learning of cross-language text may include any other steps, and may include more or fewer steps. In addition, it should be understood that although in the above description, the source language representation or the target language representation is used to perform a machine reading comprehension task, the source language representation or the target language representation generated according to the embodiment of the present disclosure can also be used to perform various other NLU tasks, such as classification tasks, question-answering tasks, etc.

[0058] Figure 5An exemplary process 500 for updating a previous joint representation according to an embodiment of the present disclosure is shown. For example, a previous joint representation 504 corresponding to a source language text and a target language text may be updated to a current joint representation 512 corresponding to the source language text and the target language text by an improved converter layer 510. The improved converter layer 510 may correspond to, for example, Figure 4 Any of the improved converter layers 440-n in the improved converter layers 440-1, 440-2, ... 440-N in the improved converter layer 440-n. For example, the current attention information 522 corresponding to the source language text and the target language text can be obtained based on the SA graph 502 and the previous joint representation 504 corresponding to the source language text and the target language text, and the previous joint representation 504 can be updated to the current joint representation 512 corresponding to the source language text and the target language text based on the current attention information 522.

[0059] The improved converter layer 510 may include an attention learning module 520. The attention learning module 520 may obtain current attention information 522 based on the SA graph 502 and the previous joint representation 504. The current attention information 522 may be, for example, an attention matrix The element G i,j It can be an attention score indicating the attention that word i exerts on word j. Figure 6 To illustrate an exemplary process for obtaining current attention information. i,j Information about the alignment relationship and / or dependency relationship between word i and word j may be included.

[0060] The current attention information 522G may be provided to a multi-head attention layer 530. The multi-head attention layer 530 may first obtain a triplet for each head by applying a linear transformation to the previous joint representation 504. m For example, the multi-head attention layer 530 can first apply a linear transformation to the previous joint representation 504 and ( and ) to get the target head m The triple of m , key K m Sum value V m (Q m , K m and ), where d h It is head m Dimensions, and and are trainable model parameters. Then, head mThe attention operation can be performed as shown in the following formula:

[0061]

[0062] The outputs of the various heads may be cascaded. Subsequently, after being processed by an additive normalization layer 540, a forward feedback layer 550, and an additive normalization layer 560, a current joint representation 512 may be obtained.

[0063] It should be understood that Figure 5 The process 500 in the figure is merely an example of a process for updating a previous joint representation. Depending on actual application requirements, the process for updating a previous joint representation may include any other steps, and may include more or fewer steps. In addition, the specific order or hierarchy of the steps in the process 500 is merely exemplary, and the process for updating a previous joint representation may be performed in an order different from the order described.

[0064] Figure 6 FIG. 6 shows an exemplary process 600 for obtaining current attention information according to an embodiment of the present disclosure. The process 600 may be performed by, for example, Figure 5 The attention learning module 520 in the process 600 is executed. In the process 600, the current attention information corresponding to the source language text and the target language text can be obtained based on the SA graph corresponding to the source language text and the target language text and the previous joint representation of the source language text and the target language text.

[0065] At 610, the previous word representation corresponding to each word can be obtained from the previous joint representation. The previous word representation corresponding to word i can be labeled as h i .

[0066] Subsequently, a current word representation of each word may be obtained based at least on the SA graph. At 620, one or more neighbor nodes of the node corresponding to the word may be identified from the SA graph. One or more neighbor nodes of node i corresponding to word i may be labeled As mentioned above, neighbor nodes It may include, for example, aligned neighbor nodes, dependent neighbor nodes, etc. The aligned neighbor nodes of node i may be marked as And the dependent neighbor nodes of node i can be marked as

[0067] At 630, the previous word representation may be updated to the current word representation based at least on the one or more representations of the one or more neighboring nodes. The previous word is represented by h i Update to the current word representation f i , as shown in the following formula:

[0068]

[0069] in, It uses neighbor nodes The representation of node i is h i The function that performs the update.

[0070] Neighbor Node One or more alignment neighbor nodes may be included, for example It can be based on at least one or more aligned neighbor nodes One or more representations to represent the previous word h i Update to the current word representation f based on the alignment relationship i a , as shown in the following formula:

[0071]

[0072] in, It uses the alignment of neighbor nodes The representation of node i is h i The function that performs the update.

[0073] Align neighbor nodes It is identified based on the alignment relationship between words corresponding to nodes. Word alignment is a challenging task. Existing alignment methods may have alignment errors. The embodiments of the present disclosure propose a method based on aligning neighbor nodes. The semantic difference between node i and previous word representation h is updated i For example, the alignment error can be mitigated by a gating mechanism in updating the current word representation f i a In the gating mechanism, when the neighbor nodes are aligned When the semantic difference between node i and node i is large, that is, when there is an alignment error, the alignment of neighboring nodes can be reduced when updating the representation of node i. The weight of the representation of can reduce the impact of alignment error. When there are multiple aligned neighbor nodes, the multiple aligned neighbor nodes can be used as a whole to determine the semantic difference between node i. The gating coefficient of the semantic difference with node i To control the alignment of neighbor nodes The weight of the representation of i in updating the representation of word i. i It can be derived, for example, by the following formula:

[0074]

[0075] in, Align neighbor nodes The mean value of the representation, and σ is the sigmoid function. When there is one aligned neighbor node, can be a representation of the aligned neighbor node. and the gating coefficient g i To generate the current word representation f of word i i a , as shown in the following formula:

[0076]

[0077] Among them, ⊙ can represent the element-wise multiplication operation. When aligning neighbor nodes When the semantic difference between node i and gating coefficient g is large, i will be smaller or even close to zero, thus reducing the alignment of neighbor nodes The weight of the representation of i in updating the representation of word i is used to reduce the impact of alignment error.

[0078] Neighbor Node It may also include, for example, one or more dependent neighbor nodes Can be based on one or more dependent neighbor nodes One or more representations to represent the previous word h i Update to the current word representation f based on the dependency relationship i s , as shown in the following formula:

[0079]

[0080] in, It uses dependent neighbor nodes The representation of node i is h i The function that performs the update.

[0081] According to an embodiment of the present disclosure, based on the dependent neighbor nodes The representation of node i is updated relative to the importance of node i. When there are multiple dependent neighbor nodes, the representation of node i can be updated based on the importance of each dependent neighbor node relative to node i. For example, when the dependent neighbor node is more important relative to node i, a larger weight corresponding to the representation of the dependent neighbor node can be given when updating the representation of node i, so that the representation of the dependent neighbor node can be considered more when updating the representation of node i. The dependent neighbor node can be calculated, for example, by the following formula: Dependent neighbor nodes in The importance α relative to node i iu :

[0082]

[0083] Among them, LR can be a Leaky ReLU activation function, and are trainable model parameters.

[0084] Then, based on one or more dependent neighbor nodes One or more representations of and the importance α corresponding to each dependent neighbor node iu To generate the dependency-based current word representation f of word i i s , as shown in the following formula:

[0085]

[0086] After obtaining the current word representation f based on the alignment relationship of word i i a and the current word representation f based on the dependency relationship i s Afterwards, the current word representation f of word i can be generated based on these two current word representations i In one embodiment, the current word can be represented by f i Calculated as the current word representation f i a and the current word represents f i s The average of the sum is given by the following formula:

[0087]

[0088] At 640, current attention scores corresponding to the two words may be calculated based on the two current word representations corresponding to the two words. For example, the current word representation f corresponding to word i may be calculated based on the current word representation f corresponding to word i. i and the current word representation f corresponding to word j j To calculate the current attention score G corresponding to word i and word j i,j , as shown in the following formula:

[0089] G i,j =(W att ·f i +b att )·(W att ·f j +b att ) (16)

[0090] in, and are trainable model parameters.

[0091] Steps 610-640 may be performed for every two words in a set of source language words in the source language text and a set of target language words in the target language text. At 650, a set of current attention scores may be obtained, each current attention score corresponding to two words in a set of source language words and a set of target language words.

[0092] At 660, a set of current attention scores can be combined into current attention information, ie, attention matrix G.

[0093] It should be understood that Figure 6 The process 600 in FIG. 6 is merely an example of a process for obtaining current attention information. According to actual application requirements, the process for obtaining current attention information may include any other steps, and may include more or fewer steps. For example, the current word representation f of word i may be obtained by i a and the current word represents f i s Perform other combination operations, such as weighted summation, maximum pooling, etc., to calculate the current word representation f of word i i Furthermore, the specific order or hierarchy of steps in process 600 is merely exemplary, and the process for obtaining current attention information may be performed in an order different from that described.

[0094] Combined with the above Figures 4 to 6An exemplary process for obtaining a source language representation of a source language text and a target language representation of a target language text according to an embodiment of the present disclosure is described. After obtaining an initial joint representation of the source language text and the target language text, the initial joint representation can be updated to a joint representation by iteratively performing an update operation. The update operation can include, for example, obtaining current attention information based on an SA graph corresponding to the source language text and the target language text and a previous joint representation, and updating the previous joint representation to a current joint representation based on the current attention information. When calculating the current attention information, the alignment relationship between the words in the source language text and the words in the target language text is considered. Two words with an alignment relationship can generally have similar semantic meanings. Considering the alignment relationship when generating a joint representation of the source language text and the target language text can help understand the semantics of each word in the source language text and the target language text. In addition, the relationship between multiple words in the source language text and the target language text can include a dependency relationship related to the syntactic structure of each text, such as a dependency relationship between multiple words in the source language text and a dependency relationship between multiple words in the target language text. The correct alignment relationship between the source language text and the target language text can be found by means of the dependency relationship. Considering alignment and dependency when generating representations of source language text and target language text can enhance knowledge transfer between source language text and target language text to generate a more accurate joint representation of source language text and target language text. The joint representation can be mapped to a source language representation corresponding to the source language text and / or a target language representation corresponding to the target language text. When performing a task using the source language representation and / or the target language representation, a more accurate representation can help obtain a more accurate result.

[0095] According to an embodiment of the present disclosure, in order to enhance the representation of the obtained model, e.g. Figure 4 In order to improve the ability of the representation acquisition model 410 to generate representations, the representation acquisition model can be pre-trained using a training data set. The training data set may include one or more sample pairs, each of which may include a source language text sample and a corresponding target language text sample. Figure 7 An exemplary process 700 for pre-training a representation acquisition model according to an embodiment of the present disclosure is shown. The representation acquisition model may be, for example, Figure 4 The representation in obtains model 410.

[0096] At 702, a source language text sample and a target language text sample may be obtained. In one embodiment, the target language text sample may be obtained first, and then the source language text sample may be obtained by translating the target language text sample. Alternatively, the source language text sample may be obtained first, and then the target language text sample may be obtained by translating the source language text sample.

[0097] At 704, an SA graph corresponding to the source language text sample and the target language text sample may be constructed. Figure 2 The SA graph corresponding to the source language text sample and the target language text sample is constructed by the process 200. The constructed SA graph may include a source language node set corresponding to the source language text sample and a target language node set corresponding to the target language text sample.

[0098] After the SA graph is constructed, according to an embodiment of the present disclosure, one or more nodes in the source language node set and / or the target language node set may be shielded.

[0099] At 706a, only one or more nodes in a node set corresponding to a sample may be shielded. For example, one or more source language nodes in the source language node set may be shielded without shielding the target language nodes in the target language node set, or one or more target language nodes in the target language node set may be shielded without shielding the source language nodes in the source language node set. This shielding strategy may also be referred to as a single language shielding strategy.

[0100] Figure 8 Exemplary SA graphs 800a and 800b are shown using a shielding strategy according to an embodiment of the present disclosure. SA graphs 800a and 800b may correspond to Figure 3 The SA graph 300 in FIG. 800a and 800b may be shielded nodes. The SA graph 800a may, for example, adopt a single language shielding strategy. For example, node 802 and node 804 may be shielded. The two nodes are in the same node set. In addition, for node 804, since node 802 is also shielded, the dependency edge between the two nodes is cut off.

[0101] Alternatively, at 706b, one or more node pairs with aligned edges in the source language node set and the target language node set may be shielded. For example, one or more source language nodes and one or more target language nodes with aligned edges with the one or more source language nodes may be shielded. In other words, whenever a node is shielded, the nodes aligned with it should also be shielded. This shielding strategy may also be referred to as a cross-language shielding strategy. Figure 8 The SA graph 800b in FIG. 800b can, for example, adopt a cross-language shielding strategy. For example, node 804 and node 806 can be shielded. The two nodes are located in different node sets, and there is an alignment edge between the two nodes. In addition, for node 804, since node 806 is also shielded, the alignment edge between the two nodes is cut off.

[0102] In the pre-training stage, the shielding operation using the single language shielding strategy at 706a may be performed in a part of the stage, and the shielding operation using the cross-language shielding strategy at 706b may be performed in another part of the stage. For example, for multiple sample pairs included in the training data set, the operation at 706a may be performed for a part of the sample pairs, and the operation at 706b may be performed for another part of the sample pairs.

[0103] After performing the masking operation at 706a or the masking operation at 706b, at 708, a source language representation of the source language text sample and a target language representation of the target language text sample may be generated. Figure 4 The representation in the representation is obtained by model 410 to generate source language representation and target language representation.

[0104] After the source language representation and the target language representation are generated, the shielded node can be restored based on the representation of each shielded node's neighbor node and the representation of the shielded node, the restoration loss corresponding to the shielded node is calculated based on the restoration result, and the representation model is pre-trained by minimizing the restoration loss. The above process is explained below by taking node i, one of the shielded nodes, as an example.

[0105] At 710, one or more neighbor nodes of node i may be obtained. In the case where the shielding operation at 706a is performed, the neighboring nodes It can include only the aligned neighbor nodes (i) of node i, or include the aligned neighbor nodes and dependent neighbor nodes When the shielding operation at 706b is performed, the neighbor node It can only include the dependent neighbor nodes of node i The neighbor nodes can be obtained from the joint representation of the source language text sample and the target language text sample. The corresponding representation of each neighbor node in .

[0106] At 712, based on one or more neighbor nodes One or more representations of node i are used to restore node i to obtain a first restoration result. For example, one or more neighbor nodes One or more representations of are input into a linear classifier. The linear classifier can output the predicted probabilities for each word in the entire vocabulary based on the representations, where the predicted probability for word i corresponding to node i can be labeled as

[0107] At 714, a first restoration loss corresponding to node i may be calculated based on the first restoration result. In one embodiment, the first recovery loss can be calculated by the cross entropy function As shown in the following formula:

[0108]

[0109] Node i can also be restored through the representation of node i. At 720, the representation h of node i can be obtained. i The representation h corresponding to node i can be obtained from the joint representation of the source language text sample and the target language text sample i .

[0110] At 722, based on the representation h of node i i To recover node i, to obtain a second recovery result. For example, the representation of node i can be input into a linear classifier. The linear classifier can output the predicted probability for each word in the entire vocabulary based on the representation, wherein the predicted probability for word i corresponding to node i can be marked as P(i|h i ).

[0111] At 724, a second restoration loss corresponding to node i may be calculated based on the second restoration result. In one embodiment, the second recovery loss can be calculated by a cross entropy function: As shown in the following formula:

[0112]

[0113] After obtaining the first recovery loss corresponding to node i and second recovery loss Thereafter, at 730, the loss can be recovered based on the first and second recovery loss To calculate the total recovery loss In one embodiment, the total recovery loss can be Calculated as the first recovery loss and second recovery loss The sum of is shown in the following formula:

[0114]

[0115] At 732, the total recovery loss can be Minimize to pre-train the representation to obtain the model.

[0116] In the case of a single-language masking strategy, since the aligned neighbor nodes in the node set of another language are not masked, the representation acquisition model can at least restore the masked node through the representation of the aligned neighbor nodes. In other words, this masking strategy can encourage the representation acquisition model to explore semantic associations through alignment relationships, thereby learning text representations from the semantic associations embedded in the alignment relationships. In the case of a cross-language masking strategy, since the nodes aligned with the current node are also masked and the alignment edges are cut off, the representation acquisition model is forced to use the representations of the dependent neighbor nodes to restore the current node, thereby guiding the representation acquisition model to learn text representations from the semantic associations embedded in the dependency relationships.

[0117] It should be understood that Figure 7 The process 700 in is merely an example of a process for pre-training a representation to obtain a model. Depending on the actual application requirements, the process for implementing the pre-training representation to obtain a model may include any other steps, and may include more or fewer steps. For example, although it is generally described in the above discussion that the shielding operation at 706a can be performed in a portion of the pre-training stage, and the shielding operation at 706b can be performed in another portion of the pre-training stage, according to the actual application requirements, one of the shielding operation at 706a and the shielding operation at 706b can be enhanced, for example, the shielding operation at 706a is performed in most of the pre-training stages, or the shielding operation at 706b is performed in most of the pre-training stages. In addition, the specific order or hierarchy of the steps in process 700 is merely exemplary, and the process for pre-training a representation to obtain a model may be performed in an order different from the described order.

[0118] Fig. 9 is a flowchart of an exemplary method 900 for representation learning of cross-language text according to an embodiment of the present disclosure.

[0119] At 910 , source language text and target language text may be obtained.

[0120] At 920 , an initial joint representation of the source language text and the target language text may be generated.

[0121] At 930 , a relationship between a plurality of words in the source language text and the target language text may be identified.

[0122] At 940 , a joint representation of the source language text and the target language text may be generated based on the initial joint representation and the relationship.

[0123] At 950, the joint representation may be mapped to at least a target language representation corresponding to the target language text.

[0124] In one implementation, the source language text may be obtained by translating the target language text, or the target language text may be obtained by translating the source language text.

[0125] In one embodiment, generating an initial joint representation may include: generating an initial source language representation of the source language text and an initial target language representation of the target language text; and combining the initial source language representation and the initial target language representation into the initial joint representation.

[0126] In one embodiment, the source language text may include a group of source language words. The target language text may include a group of target language words. The identifying relationship may include: identifying an alignment relationship between the group of source language words and the group of target language words; and / or identifying a dependency relationship between the group of source language words and a dependency relationship between the group of target language words.

[0127] In one embodiment, method 900 may further include: constructing a graph corresponding to the source language text and the target language text based on the relationship. Generating the joint representation may include: updating the initial joint representation to the joint representation based on the graph.

[0128] The constructing the graph may include: arranging a group of source language words in the source language text and a group of target language words in the target language text into a plurality of nodes; determining a group of edges between the plurality of nodes based on the relationship; and combining the plurality of nodes and the group of edges into the graph.

[0129] Determining a set of edges may include, for each two nodes of the plurality of nodes: determining whether there is a relationship between two words corresponding to the two nodes; and in response to determining that there is a relationship between the two words, determining an edge between the two nodes corresponding to the relationship.

[0130] The initial joint representation may be updated by iteratively performing an update operation. The update operation may include: obtaining current attention information based on the graph and a previous joint representation; and updating the previous joint representation to a current joint representation based on the current attention information.

[0131] The obtaining of current attention information may include: calculating current attention scores corresponding to every two words in the source language text and the target language text based on the graph to obtain a set of current attention scores; and combining the set of current attention scores into the current attention information.

[0132] The calculating the current attention score may include: obtaining a current word representation of each of the two words based on at least the graph; and calculating the current attention score based on two current word representations corresponding to the two words.

[0133] The obtaining of the current word representation may include: obtaining a previous word representation corresponding to the word from the previous joint representation; identifying at least one neighbor node of the node corresponding to the word from the graph; and updating the previous word representation to the current word representation based at least on the representation of the at least one neighbor node.

[0134] The at least one neighbor node may include at least one aligned neighbor node having an aligned edge with the node. The updating of the previous word representation may also be based on a semantic difference between the at least one aligned neighbor node and the node.

[0135] The at least one neighbor node may include at least one dependent neighbor node having a dependency edge with the node. The updating of the previous word representation may also be based on the importance of the at least one dependent neighbor node relative to the node.

[0136] In one embodiment, the joint representation may be generated by a representation acquisition model. Pre-training of the representation acquisition model may at least include: masking one or more nodes in only one of a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample; and for each of the one or more nodes, restoring the node using at least a representation of at least one aligned neighbor node of the node.

[0137] In one embodiment, the joint representation may be generated by a representation acquisition model. The pre-training of the representation acquisition model may at least include: shielding one or more node pairs with aligned edges in a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample; and for each node in the one or more node pairs, restoring the node using a representation of at least one dependent neighbor node of the node.

[0138] The pre-training of the representation acquisition model may further include: restoring the node using the representation of the node.

[0139] In one implementation, method 900 may further include: mapping the joint representation into a source language representation corresponding to the source language text.

[0140] It should be understood that method 900 may also include any steps / processes for representation learning of cross-language texts according to the above-mentioned embodiments of the present disclosure.

[0141] Fig.10 An exemplary apparatus 1000 for representation learning of cross-language texts according to an embodiment of the present disclosure is shown.

[0142] The device 1000 may include: a text acquisition module 1010, for obtaining a source language text and a target language text; an initial joint representation generation module 1020, for generating an initial joint representation of the source language text and the target language text; a relationship identification module 1030, for identifying the relationship between a plurality of words in the source language text and the target language text; a joint representation generation module 1040, for generating a joint representation of the source language text and the target language text based on the initial joint representation and the relationship; and a mapping module 1050, for mapping the joint representation to at least a target language representation corresponding to the target language text. In addition, the device 1000 may also include any other module configured for representation learning of cross-language texts according to the above-mentioned embodiments of the present disclosure.

[0143] Fig.11 An exemplary apparatus 1100 for representation learning of cross-language texts according to an embodiment of the present disclosure is shown.

[0144] The apparatus 1100 may include: at least one processor 1110; and a memory 1120 storing computer executable instructions. When the computer executable instructions are executed, the at least one processor 1110 may: obtain a source language text and a target language text, generate an initial joint representation of the source language text and the target language text, identify a relationship between a plurality of words in the source language text and the target language text, generate a joint representation of the source language text and the target language text based on the initial joint representation and the relationship, and map the joint representation to at least a target language representation corresponding to the target language text.

[0145] In one embodiment, the computer executable instructions, when executed, may further cause the at least one processor 1110 to: construct a graph corresponding to the source language text and the target language text based on the relationship. The generating of the joint representation may include: updating the initial joint representation to the joint representation based on the graph.

[0146] It should be understood that the processor 1110 may also execute any other steps / processes of the method for representation learning of cross-language text according to the above-mentioned embodiments of the present disclosure.

[0147] The embodiment of the present disclosure proposes a computer program product for representation learning of cross-language texts, including a computer program, which is executed by at least one processor to: obtain a source language text and a target language text, generate an initial joint representation of the source language text and the target language text, identify the relationship between multiple words in the source language text and the target language text, generate a joint representation of the source language text and the target language text based on the initial joint representation and the relationship, and map the joint representation to at least a target language representation corresponding to the target language text. In addition, the computer program can also be executed to implement any other steps / processes of the method for representation learning of cross-language texts according to the above-mentioned embodiment of the present disclosure.

[0148] The embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions, which, when executed, cause one or more processors to perform any operation of the method for representation learning of cross-language text according to the embodiments of the present disclosure as described above.

[0149] It should be understood that all operations in the above-described methods are merely exemplary, and the present disclosure is not limited to any operations in the methods or the order of these operations, but should cover all other equivalent transformations under the same or similar concept. In addition, unless otherwise specified or it is clear from the context that it is directed to a singular form, the articles "a" and "an" as used in this specification and the appended claims should generally be interpreted as meaning "one" or "one or more".

[0150] It should also be appreciated that all modules in the above described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together in function.

[0151] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gated logic unit, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The function of the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using software executed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0152] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the multiple aspects provided in the present disclosure, the memory can also be located inside the processor, such as a cache or a register.

[0153] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents to the elements of the various aspects described in this disclosure that are known or to be known to those of ordinary skill in the art are expressly incorporated herein and covered by the claims.

Claims

1. A method for representation learning of cross-language text, comprising: Obtaining source language text and target language text; generating an initial joint representation of the source language text and the target language text; identifying a relationship between a plurality of words in the source language text and the target language text; constructing a graph corresponding to the source language text and the target language text based on the relationship; Generating a joint representation of the source language text and the target language text based on the initial joint representation and the relationship, wherein generating the joint representation includes: updating the initial joint representation to the joint representation based on the graph; And mapping the joint representation to at least a target language representation corresponding to the target language text.

2. The method according to claim 1, wherein: The source language text is obtained by translating the target language text, or The target language text is obtained by translating the source language text.

3. The method according to claim 1, wherein: Generating the initial joint representation comprises: generating an initial source language representation of the source language text and an initial target language representation of the target language text; and The initial source language representation and the initial target language representation are combined into the initial joint representation.

4. The method according to claim 1, wherein: The source language text includes a set of source language words, the target language text includes a set of target language words, and the identification relationship includes: identifying an alignment relationship between the set of source language words and the set of target language words; and / or Dependencies between the set of source language words and dependencies between the set of target language words are identified.

5. The method according to claim 1, wherein: The construction diagram includes: Arranging a group of source language words in the source language text and a group of target language words in the target language text into a plurality of nodes; determining a set of edges between the plurality of nodes based on the relationships; and The plurality of nodes and the set of edges are combined into the graph.

6. The method according to claim 5, wherein: Determining a set of edges comprises, for every two nodes in the plurality of nodes: determining whether there is a relationship between two words corresponding to the two nodes; as well as In response to determining that a relationship exists between the two words, an edge between the two nodes corresponding to the relationship is determined.

7. The method according to claim 1, wherein: The initial joint representation is updated by iteratively performing an update operation, the update operation comprising: obtaining current attention information based on the graph and the previous joint representation; and The previous joint representation is updated to a current joint representation based on the current attention information.

8. The method according to claim 7, wherein: The obtaining of current attention information comprises: calculating current attention scores corresponding to every two words in the source language text and the target language text based on the graph to obtain a set of current attention scores; and The set of current attention scores are combined into the current attention information.

9. The method according to claim 8, wherein: The calculation of the current attention score includes: obtaining a current word representation for each of the two words based at least on the graph; and The current attention score is calculated based on two current word representations corresponding to the two words.

10. The method according to claim 9, wherein: The obtaining of the current word representation comprises: obtaining a previous word representation corresponding to the word from the previous joint representation; identifying from the graph at least one neighbor node of the node corresponding to the word; and The previous word representation is updated to the current word representation based at least on the representation of the at least one neighbor node.

11. The method according to claim 10, wherein: The at least one neighbor node includes at least one aligned neighbor node having an aligned edge with the node, and the updating the previous word representation is further based on a semantic difference between the at least one aligned neighbor node and the node.

12. The method according to claim 10, wherein: The at least one neighbor node includes at least one dependent neighbor node having a dependency edge with the node, and the updating the previous word representation is further based on an importance of the at least one dependent neighbor node relative to the node.

13. The method according to claim 1, wherein: The joint representation is generated by a representation acquisition model, and the pre-training of the representation acquisition model at least includes: Masking one or more nodes in only one of a source language node set corresponding to the source language text sample and a target language node set corresponding to the target language text sample; and For each node of the one or more nodes, the node is restored using at least a representation of at least one aligned neighbor node of the node.

14. The method according to claim 1, wherein: The joint representation is generated by a representation acquisition model, and the pre-training of the representation acquisition model at least includes: shielding one or more node pairs having aligned edges in a source language node set corresponding to the source language text sample and a target language node set corresponding to the target language text sample; and For each node in the one or more node pairs, the node is restored using a representation of at least one dependent neighbor node of the node.

15. The method according to claim 13 or 14, wherein: The pre-training of the representation acquisition model further includes: The node is restored using the representation of the node.

16. The method according to claim 1, further comprising: The joint representation is mapped into a source language representation corresponding to the source language text.

17. A device for learning representations of cross-language texts, comprising: at least one processor; as well as a memory storing computer executable instructions that, when executed, cause the at least one processor to: Obtain source language text and target language text, generating an initial joint representation of the source language text and the target language text, identifying a relationship between a plurality of words in the source language text and the target language text, and constructing a graph corresponding to the source language text and the target language text based on the relationship, Generating a joint representation of the source language text and the target language text based on the initial joint representation and the relationship, wherein generating the joint representation includes: updating the initial joint representation to the joint representation based on the graph, and The joint representation is mapped to at least a target language representation corresponding to the target language text.

18. A computer program product for representation learning of cross-language text, comprising a computer program, the computer program being executed by at least one processor for: Obtaining source language text and target language text; generating an initial joint representation of the source language text and the target language text; identifying a relationship between a plurality of words in the source language text and the target language text; constructing a graph corresponding to the source language text and the target language text based on the relationship, Generating a joint representation of the source language text and the target language text based on the initial joint representation and the relationship, wherein generating the joint representation includes: updating the initial joint representation to the joint representation based on the graph; as well as The joint representation is mapped to at least a target language representation corresponding to the target language text.

Images