A cross-language idiom equivalent recognition and translation method

Abstract

The present application relates to the field of data processing, especially to a cross-language English idiom equivalence recognition and translation method. First, the received English input text is preprocessed to obtain the word form restored English input text and the dependency tree of the word form restored English input text; the word form restored English input text is scanned to generate a phrase subsequence set; the phrase subsequences in the phrase subsequence set and the standard English idiom are matched to obtain a candidate English idiom set; then, based on the candidate English idiom set, an idiom equivalence translation determination algorithm is introduced to obtain an equivalence translation set; finally, the equivalence translation set is traversed, and the final translation text is generated in combination with the English input text. The technical problems of the processing defects of the idiom overall semantics, the insufficient cultural adaptability of cross-language translation, and the low recognition and translation efficiency of multi-idiom text are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a cross-linguistic method for identifying and translating equivalent English idioms. Background Technology

[0002] In natural language processing (NLP), idioms are highly fixed and semantically holistic expressions, typically composed of several words whose overall meaning cannot be directly deduced from the literal meanings of individual words. Idioms are widely present in various languages, forming an important part of language culture and a key factor influencing the performance of machine translation, cross-language information retrieval, and language understanding systems. Unlike ordinary words or phrases, idioms have strong cultural dependence and semantic metaphor, posing numerous challenges in cross-language processing. Especially in English-to-Chinese or Chinese-to-English translation tasks, due to significant differences in language structure, cultural background, and expression habits between Chinese and English, the identification and equivalent translation of English idioms has become a research hotspot and challenge in the field of NLP. Therefore, achieving high-quality cross-language idiom equivalence identification and translation methods is of great significance for improving the practicality and intelligence of machine translation systems. Summary of the Invention

[0003] This invention provides a cross-linguistic method for identifying and translating equivalent English idioms, in order to solve the technical problems of deficiencies in processing the overall semantics of idioms, insufficient cultural adaptability in cross-linguistic translation, and low efficiency in identifying and translating multi-idiom texts.

[0004] The present invention provides a cross-linguistic method for identifying and translating equivalent English idioms, comprising the following steps: S1. Preprocess the received English input text to obtain the lemmatized English input text and its dependency tree; scan the lemmatized English input text to generate a set of phrase subsequences; match the phrase subsequences in the phrase subsequence set with standard English idioms to obtain a set of candidate English idioms; S2. Based on the candidate English idiom set, introduce an idiom equivalent translation determination algorithm to obtain an equivalent translation set; traverse the equivalent translation set and combine it with the English input text to generate the final translation text.

[0005] Preferably, S1 specifically includes: Preprocessing operations for English input text, including word segmentation, lexical reconstruction, and syntactic analysis.

[0006] Preferably, S1 specifically includes: During the matching process, a balancing parameter is introduced to weight and combine word commonality similarity and phrase structure similarity to calculate the matching degree between phrase subsequences and standard English idioms.

[0007] Preferably, S1 specifically includes: Phrases with a matching degree higher than the matching degree threshold are considered as candidate English idioms and added to the candidate English idiom set.

[0008] Preferably, S2 specifically includes: In the algorithm for determining idiom equivalent translation, a semantic vector of the candidate English idiom is generated based on each candidate English idiom; a semantic vector of the target language idiom is generated based on each target language idiom; and the cosine similarity between the semantic vectors of the candidate English idiom and the semantic vectors of the target language idiom is calculated to obtain the semantic similarity.

[0009] Preferably, S2 specifically includes: In the algorithm for determining the equivalent translation of idioms, the cultural node sets of candidate English idioms and target language idioms are extracted respectively, and the semantic relationships between cultural nodes are analyzed and the cultural context weights are calculated.

[0010] Preferably, S2 specifically includes: Based on a set of candidate English idioms, and combining semantic similarity and cultural context weights, a comprehensive similarity score between candidate English idioms and target language idioms is calculated.

[0011] Preferably, S2 specifically includes: The target language idiom with the highest overall similarity score is the equivalent translation of the candidate English idiom, and the final output is a set of equivalent translations.

[0012] Preferably, S2 specifically includes: Locate the positions of candidate English idioms in the English input text and replace them with equivalent translations of the candidate idioms to generate intermediate translation text; convert the non-idiom parts of the intermediate translation text into the target language; adjust the intermediate translation text and output the final translation text.

[0013] The beneficial effects of the technical solution of the present invention are: 1. By preprocessing the English input text and matching each phrase subsequence in the phrase subsequence set with standard English idioms in the standard English idiom corpus, and combining the phrase structure similarity analysis by the SpaCy dependency parser, we can accurately identify idioms with multiple independent meanings in the English input text, ensuring that the candidate idiom set is comprehensive and accurate, avoiding misidentification due to word order or variants, thereby significantly improving the accuracy of idiom recognition and the semantic correctness of translation.

[0014] 2. By combining semantic similarity with cultural context weights, the ConceptNet knowledge graph is used to quantify the cultural background association between English idioms and target language idioms, which solves the translation bias caused by cultural differences, making the translation results more in line with the cultural pragmatic habits of the target language, and improving the cultural fit and user acceptance of the translation.

[0015] 3. By first replacing idioms and then translating non-idiom parts, and using pre-trained language models such as mBART for syntactic smoothing and pragmatic optimization, the final translated text conforms to the grammar and expression habits of the target language. This ensures that the translation results retain the semantics of idioms while the overall text is fluent and natural, applicable to multiple target languages, thus improving the practicality and user experience of the translation system. Attached Figure Description

[0016] Figure 1 This is a flowchart of a cross-linguistic English idiom equivalence recognition and translation method according to the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a cross-linguistic English idiom equivalence recognition and translation method provided by the present invention.

[0020] See attached document Figure 1 The diagram illustrates a flowchart of a cross-linguistic English idiom equivalence identification and translation method according to an embodiment of the present invention. The method includes the following steps: S1. Preprocess the received English input text to obtain the lemmatized English input text and its dependency tree; scan the lemmatized English input text to generate a set of phrase subsequences; match the phrase subsequences in the set of phrase subsequences with standard English idioms to obtain a set of candidate English idioms.

[0021] Receive a piece of English input text English input text is treated as a sequence of words, where For the first One word, Enter the length of the text in English. The received English input text is preprocessed, including word segmentation, lexical reconstruction, and syntactic analysis, to obtain the lexical-reconstructed English input text and its dependency tree.

[0022] The word segmentation process uses the SpaCy word segmentation tool to divide the English input text into a sequence of words, ensuring that each word is correctly identified. For example, "He spilled the beans" is segmented into "He", "spilled", "the", and "beans".

[0023] The word form reduction process uses WordNet's Lemmatizer tool to convert words into their basic forms, resulting in English input text with reduced word forms. This process eliminates the influence of morphological changes such as tense and singular / plural forms. For example, "spilled" is reduced to "spill", and "beans" remains "beans", ensuring that the variant "spilled the beans" matches the standard idiom "spill the beans".

[0024] The syntactic analysis uses the SpaCy dependency parser to generate a dependency tree of the English input text after word form reduction and marks the syntactic relationships between words, such as subject-verb relationships and verb-object relationships. For example, in "spill the beans", "spill" and "beans" form a verb-object relationship.

[0025] The existing sliding window technique is used to scan the lemmatized English input text and generate a set of phrase subsequences. , .in, Indicates the first Each phrase subsequence is a phrase subsequence. Represents a potential idiom candidate; Indicates the number of phrase subsequences. ; Indicates the first One word; Indicates the starting index of the sliding window; This indicates the size of the sliding window, i.e., the number of words contained in the sliding window. The choice of sliding window size is based on the characteristics of the idiom's word composition, such as 2 to 5 words.

[0026] Each phrase subsequence in the phrase subsequence set is matched against standard English idioms in a pre-built standard English idiom corpus. The standard English idiom corpus is built based on authoritative resources, such as the Oxford Idioms Dictionary, and contains only the standard form of idioms, such as "spill the beans," excluding variant forms.

[0027] The matching process determines whether a phrase subsequence is an idiom by calculating the degree of matching between the phrase subsequence and standard English idioms. The degree of matching is calculated by two parts: word commonality similarity and phrase structure similarity, which are weighted and combined by a balancing parameter.

[0028] Word co-occurrence similarity measures the degree of overlap between a phrase subsequence and a standard English idiom in terms of words. It is obtained by dividing the number of common words between the phrase subsequence and the standard English idiom by the number of words in the union of the phrase subsequence and the standard English idiom.

[0029] Phrase structure similarity is based on dependency parsing to assess the syntactic structural consistency between phrase subsequences and standard English idioms. Specifically, the SpaCy dependency parser is used to generate dependency trees for phrase subsequences and standard English idioms, extracting dependency relation sets. For example, for "spill the beans," "spill" and "beans" form a verb-object relationship, with "the" acting as a determiner for "beans," resulting in the following dependency relation set: The standard English idiom "spill the beans" shares the same set of dependencies. The phrase structural similarity is obtained by dividing the number of common dependencies between the phrase subsequence and the standard English idiom by the number of union dependencies between the phrase subsequence and the standard English idiom.

[0030] The formula for calculating the matching degree is: ; in, Indicates the first The phrase subsequence and the first The degree of matching of a standard English idiom; Indicates the first A phrase subsequence; Indicates the first A standard English idiom; This represents a balancing parameter used to adjust the relative importance of word commonality similarity and phrase structure similarity, optimizing matching accuracy. It is preset by the system developers based on the characteristics of the standard English idiom corpus and matching accuracy requirements. The characteristics of the standard English idiom corpus include fixed forms and syntactic structure, and its value range is [value range missing]. As a specific embodiment, take ; The similarity is represented by the number of words that are shared, and is used to measure the similarity of the first word. The phrase subsequence and the first By observing the similarities in word composition among standard English idioms, we can capture the fixed nature of English idioms. Indicates the first The phrase subsequence and the first The number of common words in a set of standard English idioms, used to quantify the first The phrase subsequence and the first The degree of word overlap in a standard English idiom; Indicates the first The phrase subsequence and the first The number of words in the union of standard English idioms; It represents phrase structure similarity and is used to capture the syntactic structural consistency between phrase subsequences and standard English idioms, making up for the deficiency of word commonality similarity in ignoring word order; Indicates the first The phrase subsequence and the first The number of common dependencies of a standard English idiom; Indicates the first The phrase subsequence and the first The number of union dependencies of a standard English idiom; Indicates for the first Each phrase subsequence is used to extract all dependencies, forming a set. Indicates for the first A set of standard English idioms, extracting all dependencies.

[0031] A matching accuracy threshold is preset based on the required matching precision. The range of the matching accuracy threshold is as follows: This is used to control the strictness of the selection of candidate English idioms. As a specific embodiment, a preset matching threshold of 0.75 is set. Phrases with a matching degree higher than the preset threshold are considered idioms with independent meaning and added to the candidate English idiom set. ,in, Indicates the first A candidate English idiom, Indicates the number of candidate English idioms. .

[0032] S2. Based on the candidate English idiom set, introduce the idiom equivalent translation determination algorithm to obtain the equivalent translation set; traverse the equivalent translation set and combine it with the English input text to generate the final translation text.

[0033] Based on candidate English idiom set An idiom equivalence translation determination algorithm is introduced, which combines semantic similarity and cultural context weights in a weighted manner to calculate the comprehensive similarity score between each candidate English idiom and the target language idiom, ensuring that the translation results are highly matched in both semantics and culture.

[0034] The idiom equivalence translation determination algorithm calculates semantic similarity to reflect the semantic similarity between the semantic vectors of candidate English idioms and the semantic vectors of target language idioms. Specifically, for each candidate English idiom in the candidate idiom set, a pre-trained language model, such as the Transformer-based bidirectional encoding representation model BERT, is used to generate the semantic vector of the candidate English idiom. The Transformer-based bidirectional encoding representation model BERT includes an embedding layer for mapping discrete words to continuous vectors, a Transformer encoding layer structure, and a pooling layer for generating fixed-dimensional semantic vectors. The embedding layer includes a word embedding layer, a sentence embedding layer, and a position embedding layer. The word embedding layer has a vocabulary size of 30522, which determines the number of recognizable words, implicitly... The hidden layer has a dimension of 768, which determines the semantic expressive power; the sentence embedding layer can distinguish 2 sentence types, and the hidden layer dimension, i.e., the vector dimension corresponding to each sentence type, is 768; the maximum supported sequence length in the positional embedding layer, i.e., the maximum number of words that can be processed, is 512, and the hidden layer dimension is 768; the Transformer encoding layer structure adopts the standard configuration: the number of encoder layers, i.e., the number of stacked Transformer layers, is 12, the hidden layer dimension is 768, the number of attention heads is 12, and the feature dimension of each attention head is 64; the pooling layer uses average pooling to generate semantic vectors of candidate English idioms.

[0035] Similarly, using a pre-trained language model, such as the Transformer-based bidirectional encoding representation model BERT mentioned above, semantic vectors of target language idioms are generated for each target language idiom in the target language idiom corpus. The target language idiom corpus is constructed based on authoritative resources, such as the "Chinese Idiom Dictionary," and contains phrases of target language idioms. Semantic similarity is obtained by calculating the cosine similarity between the semantic vectors of candidate English idioms and the semantic vectors of target language idioms.

[0036] The idiom equivalence translation determination algorithm captures the cultural relevance between candidate English idioms and target language idioms by calculating cultural context weights. Using the existing ConceptNet knowledge graph, it extracts cultural node sets for both candidate English idioms and target language idioms. By analyzing the semantic relationships between cultural nodes, it calculates the cultural context weights. Nodes in the ConceptNet knowledge graph are conceptual units representing the semantics and cultural concepts of idioms, and edges represent the semantic relationships and strengths between different nodes, used to quantify the strength of the cultural association between candidate English idioms and target language idioms. Specifically, a cultural node set is extracted for each candidate English idiom and target language idiom. Based on the semantic association strength provided by the ConceptNet knowledge graph, the sum of edge weights between cultural nodes is calculated. A normalized denominator is used to generate the cultural context weights, reflecting the similarity of cultural backgrounds. The normalized denominator is calculated by subtracting the number of common nodes between the two sets of cultural nodes from the sum of the number of cultural nodes in the candidate English idiom's cultural node set and the target language idiom's cultural node set.

[0037] The formula for calculating the overall similarity score is: ; in, Indicates the first The candidate English idioms and the first The overall similarity score of the target language idioms; Indicates the first One candidate English idiom; Indicates the first Target language idioms; The hyperparameter representing the weighting of semantic similarity and cultural context is preset by the system developers based on the cultural sensitivity and semantic requirements of the translation task, with a reference value range of [value range missing]. As a specific embodiment, take ; Represents semantic similarity, i.e., the first... The semantic vector of the candidate English idiom and the first The cosine similarity of the semantic vectors of the nth target language idiom is used to measure the similarity of the th idiom. The semantic vector of the candidate English idiom and the first Semantic similarity between the semantic vectors of target language idioms; Indicates the first The semantic vectors of candidate English idioms are obtained using a pre-trained language model, such as BERT; Indicates the first The semantic vectors of target language idioms are obtained using a pre-trained language model, such as BERT; Indicates the first The semantic vector of the candidate English idiom and the first The dot product of the semantic vectors of each target language idiom; Indicates the first The vector magnitude of the semantic vector of each candidate English idiom; Indicates the first The vector magnitude of the semantic vector of a target language idiom; Represents cultural context weights, used to quantify the first The candidate English idioms and the first The similarity of target language idioms within their cultural background enhances the cultural adaptability of translation. The calculation formula is as follows: ; in, Indicates the first All cultural nodes in the set of candidate English idioms are related to the first The sum of edge weights between all cultural nodes in a set of cultural nodes of a target language idiom is used to quantify the semantic association strength between cultural nodes; Indicates the first Cultural nodes in the set of candidate English idioms and the Cultural nodes in a set of cultural nodes of target language idioms The edges between them are used to reflect the connections between cultural nodes, such as the connection between "secret" and "secret"; Indicates the first The set of cultural nodes of candidate English idioms and the first The set of edges between cultural nodes of a target language idiom; Indicates the first Cultural nodes in the set of candidate English idioms and the Cultural nodes in a set of cultural nodes of target language idioms The edge weights between the edges reflect the semantic association strength provided by the ConceptNet knowledge graph, and their values ​​range from [value range missing]. ; Indicates the first The set of cultural nodes of candidate English idioms and the first The total number of cultural nodes in the union of the sets of cultural nodes of target language idioms, i.e., the normalized denominator, is used to normalize the cultural context weights. Indicates the first A set of cultural nodes for candidate English idioms; Indicates the first The number of cultural nodes in the set of cultural nodes for each candidate English idiom; Indicates the first A set of cultural nodes of target language idioms; Indicates the first The number of cultural nodes in the set of cultural nodes of a target language idiom; Indicates the first The set of cultural nodes for candidate English idioms and the first The number of cultural nodes in the intersection of the sets of cultural nodes of target language idioms, i.e., the number of common nodes, is used to avoid double counting.

[0038] For each candidate English idiom, the target language idiom with the highest comprehensive similarity score is selected as the equivalent translation result of the candidate English idiom, and an equivalent translation set is output, with each candidate English idiom corresponding to one target language idiom.

[0039] The equivalent translation set is traversed, and combined with the English input text, to achieve overall text translation. Specifically, for each candidate English idiom, existing string matching methods, such as the KMP algorithm, are used to locate the occurrence position of the candidate English idiom in the English input text and replace it with the corresponding target language idiom, generating intermediate translation text. For non-idiom parts in the intermediate translation text, such as the conjunction "and", a pre-trained machine translation model, such as Transformer-based mBART, is used to translate the English words or phrases in the non-idiom parts into the target language. At the same time, the word order, grammatical structure, and pragmatic consistency of the intermediate translation text are adjusted to make the output conform to the expression habits of the target language, generating the final translation text.

[0040] In summary, a cross-linguistic method for identifying and translating English idiom equivalences has been developed.

[0041] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0042] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0043] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A cross-linguistic method for identifying and translating equivalent English idioms, characterized in that, Includes the following steps: S1. Preprocess the received English input text to obtain the lemmatized English input text and the dependency tree of the lemmatized English input text; Scan the English input text after word form restoration to generate a set of phrase subsequences; The phrase subsequences in the phrase subsequence set are matched with standard English idioms to obtain a candidate English idiom set; S2. Based on the candidate English idiom set, introduce an idiom equivalent translation determination algorithm to obtain an equivalent translation set; traverse the equivalent translation set and combine it with the English input text to generate the final translation text.

2. The method for cross-linguistic English idiom equivalence identification and translation according to claim 1, characterized in that, S1 specifically includes: Preprocessing operations for English input text, including word segmentation, lexical reconstruction, and syntactic analysis.

3. The method for cross-linguistic English idiom equivalence identification and translation according to claim 1, characterized in that, S1 specifically includes: During the matching process, a balancing parameter is introduced to weight and combine word commonality similarity and phrase structure similarity to calculate the matching degree between phrase subsequences and standard English idioms.

4. The method for cross-linguistic English idiom equivalence identification and translation according to claim 3, characterized in that, S1 specifically includes: Phrase subsequences with a matching degree higher than the matching degree threshold are considered as candidate English idioms and added to the candidate English idiom set.

5. The method for cross-linguistic English idiom equivalence identification and translation according to claim 1, characterized in that, S2 specifically includes: In the algorithm for determining idiom equivalent translation, a semantic vector of the candidate English idiom is generated based on each candidate English idiom; a semantic vector of the target language idiom is generated based on each target language idiom; and the cosine similarity between the semantic vectors of the candidate English idiom and the semantic vectors of the target language idiom is calculated to obtain the semantic similarity.

6. The method for cross-linguistic English idiom equivalence identification and translation according to claim 5, characterized in that, S2 specifically includes: In the algorithm for determining the equivalent translation of idioms, the cultural node sets of candidate English idioms and target language idioms are extracted respectively, and the semantic relationships between cultural nodes are analyzed and the cultural context weights are calculated.

7. The method for cross-linguistic English idiom equivalence identification and translation according to claim 6, characterized in that, S2 specifically includes: Based on a set of candidate English idioms, and combining semantic similarity and cultural context weights, a comprehensive similarity score between candidate English idioms and target language idioms is calculated.

8. The method for cross-linguistic English idiom equivalence identification and translation according to claim 7, characterized in that, S2 specifically includes: The target language idiom with the highest overall similarity score is the equivalent translation of the candidate English idiom, and the final output is a set of equivalent translations.

9. A cross-linguistic method for identifying and translating English idiom equivalences according to claim 8, characterized in that, S2 specifically includes: Locate the positions of candidate English idioms in the English input text and replace them with equivalent translations of the candidate idioms to generate intermediate translation text; convert the non-idiom parts of the intermediate translation text into the target language; adjust the intermediate translation text and output the final translation text.

Images