Text processing method and related equipment

Abstract

The invention provides a text processing method and related equipment. The method comprises the steps of obtaining an input sequence of a to-be-translated text and field information of the to-be-translated text; processing the input sequence through a proper noun recognition model based on context perception, and recognizing proper nouns in the to-be-translated text; searching candidate translation information of candidate proper nouns matched with proper nouns in the to-be-translated text from a dynamic word bank, and determining a target translation of the proper nouns in the to-be-translated text according to the candidate translation information matched with semantic and domain information of the proper nouns in the to-be-translated text; the to-be-translated text is processed through a translation model, a preliminary translation result of the to-be-translated text is obtained, and the preliminary translation result comprises preliminary translation suggestions of proper nouns in the to-be-translated text; and generating a target translation result of the to-be-translated text according to the preliminary translation result and the target translation of the proper nouns in the to-be-translated text.

Description

Technical Field

[0001] The present disclosure relates to the technical field of artificial intelligence and natural language processing, and in particular, to a text processing method, device, electronic device, computer-readable medium, and computer program product. Background Art

[0002] The translation accuracy of the machine translation system in the relevant technology cannot fully meet the needs of users, especially when dealing with knowledge-intensive texts. It is easy to have problems such as mistranslation and omission of translation, which affects the accuracy and professionalism of the overall translation. Summary of the invention

[0003] According to one aspect of the present disclosure, a text processing method is provided, comprising: processing the input sequence through a context-aware proper noun recognition model to identify proper nouns in the text to be translated; retrieving candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated from a dynamic vocabulary, and determining a target translation of the proper nouns in the text to be translated based on the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated; processing the text to be translated through a translation model to obtain a preliminary translation result of the text to be translated, the preliminary translation result including preliminary translation suggestions for the proper nouns in the text to be translated; and generating a target translation result of the text to be translated based on the preliminary translation result and the target translation of the proper nouns in the text to be translated.

[0004] According to one aspect of the present disclosure, a text processing device is provided, comprising: a receiving unit, configured to obtain an input sequence of a text to be translated and domain information of the text to be translated; a processing unit, configured to process the input sequence through a context-aware proper noun recognition model to identify proper nouns in the text to be translated; a retrieval unit, configured to retrieve candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated from a dynamic vocabulary, and determine a target translation of the proper nouns in the text to be translated based on the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated; the processing unit is further configured to process the text to be translated through a translation model to obtain a preliminary translation result of the text to be translated, wherein the preliminary translation result includes preliminary translation suggestions for the proper nouns in the text to be translated; and a generation unit, configured to generate a target translation result of the text to be translated based on the preliminary translation result and the target translation of the proper nouns in the text to be translated.

[0005] According to one aspect of an embodiment of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the text processing method as described in any embodiment of the present disclosure is implemented.

[0006] According to one aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the text processing method as described in any embodiment of the present disclosure.

[0007] According to one aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the text processing method described in any embodiment of the present disclosure when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 The flowchart of the text processing method according to an embodiment of the present disclosure is schematically shown.

[0009] Figure 2 The flowchart of a text processing method according to another embodiment of the present disclosure is schematically shown.

[0010] Figure 3 The following schematically shows an interface diagram of a translation interface according to an embodiment of the present disclosure.

[0011] Figure 4 The following schematically shows an interface diagram of a translation output interface according to an embodiment of the present disclosure.

[0012] Figure 5 The block diagram schematically shows a text processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] The machine translation systems in the related art rely on basic dictionaries or databases when dealing with the translation of proper nouns, and lack a deep understanding of the text context. This limitation makes it easy for proper nouns to be mistranslated or missed when translated, affecting the accuracy and professionalism of the overall translation, especially in knowledge-intensive fields. Specifically, the machine translation systems in the related art have difficulty identifying polysemous proper nouns in a specific context, resulting in the system's lack of accuracy in identifying and translating some key proper nouns. For example, the same word may have different meanings in different contexts, and the machine translation systems in the related art are often unable to deal with this effectively.

[0014] The method provided in any embodiment of the present disclosure may be executed by any computer device or electronic device, for example, by a terminal device and / or a server, and the present disclosure does not limit this.

[0015] like Figure 1 As shown, the text processing method provided by the embodiment of the present disclosure may include the following steps.

[0016] In S110, an input sequence of a text to be translated and domain information of the text to be translated are obtained.

[0017] The text to be translated in the embodiments of the present disclosure can be any text that needs to be translated from the current language into another language. The text to be translated can be a sentence, a document or file containing a sentence, a paragraph containing a sentence, etc., which is not limited in the present disclosure. In some embodiments, the text to be translated refers to a knowledge-intensive text or a long text. Among them, a knowledge-intensive text or a long text refers to a text that contains a large amount of information, details and professional knowledge and is intended to convey complex concepts, theories or data. Such texts are more common in academic, scientific, legal, medical and other fields, such as academic papers, technical manuals, legal documents, medical reports, etc. These texts are highly professional and often involve specific professional knowledge or fields, requiring the use of professional terms and concepts; the structure is complex and may contain multiple chapters, paragraphs and sub-paragraphs, as well as complex logical relationships, such as cause and effect, contrast, progression, etc.

[0018] In the disclosed embodiment, after the machine translation system receives the text to be translated, it may perform preprocessing on it, such as word segmentation, stop word removal, part-of-speech tagging, vector conversion, etc., to obtain an input sequence of the text to be translated.

[0019] In the disclosed embodiment, when the text to be translated is obtained, the field to which the text to be translated belongs can also be obtained as its field information. For example, when a user inputs or uploads a text to be translated, he or she can select whether the text to be translated is in the medical field, the legal field, or the scientific field, etc. This is used to assist the context-aware proper noun recognition model and translation model to accurately identify the proper nouns and their target translations that are unique to the field contained in the text to be translated.

[0020] In S120, the input sequence is processed by a context-aware proper noun recognition model to recognize proper nouns in the text to be translated.

[0021] In the disclosed embodiment, the context-aware proper noun recognition model is a deep learning model that uses natural language processing technology to identify and understand proper nouns in the text to be translated using context information in the input sequence. This deep learning model can analyze the contextual environment in the text to be translated, including the semantic relationship of vocabulary, syntactic structure, and possible field-specific knowledge, so as to more accurately identify proper nouns. In the context-aware proper noun recognition process, the model can analyze the semantic connection between the proper noun and its context vocabulary. For example, when identifying a person's name, the model can consider whether a common surname or job title appears before and after the name. The model can also focus on the syntactic structure in the text to be translated to determine whether a certain word is in the typical position of a proper noun. For example, a noun in a subject or object position in a sentence is more likely to be a proper noun. For texts to be translated in a specific field, the model can also use professional knowledge in the field to assist in identifying proper nouns. The context-aware proper noun recognition model can more accurately identify proper nouns in the text to be translated, especially when the text environment is complex or the proper noun is ambiguous. The model can also learn text patterns in different fields, such as specific vocabulary patterns in the field of science and technology, to achieve cross-domain or specific field proper noun recognition.

[0022] In the disclosed embodiments, a proper noun is a type of noun, which refers to a single specific thing, indicating a specific, unique person or thing, as opposed to a common noun. Proper nouns can be classified according to different fields and characteristics. In the disclosed embodiments, proper nouns are divided into proper nouns in common fields and proper nouns in specific fields. Common fields may include, for example, names of people, places, names of institutions or groups, time, names of works, etc. Specific fields may further include, for example, the medical field, the legal field, the mechanical field, the communication field, the blockchain field, the semiconductor field, etc.

[0023] In an exemplary embodiment, the context-aware proper noun recognition model includes a bidirectional neural recurrent network model and a transformer model (Transformer) based on an attention mechanism; the bidirectional neural recurrent network model includes a forward neural recurrent network model and a reverse neural recurrent network model.

[0024] The Bidirectional Recurrent Neural Networks (BiRNN or Bi-RNNs) model considers both the forward (forward) and backward (reverse) information of the input sequence, so that the completeness of the context can be captured. BiRNN includes a forward RNN and a reverse RNN, and the output combines the outputs of the forward RNN and the reverse RNN, so that the BiRNN model can obtain contextual information from the past and the future at each time step. The forward RNN processes the input sequence from front to back in chronological order, capturing all information before each time point; while the reverse RNN processes the input sequence from back to front in chronological order, capturing all information after each time point. The outputs of the two RNNs at each time step are merged (such as by concatenation or weighted averaging) to form a comprehensive representation that contains both the contextual information of the sequence.

[0025] Bidirectional LSTM (Bidirectional Long Short-Term Memory) is a BiRNN. In the following embodiments, the bidirectional LSTM is used to illustrate the bidirectional recurrent neural network model, but the present disclosure is not limited to this. Bidirectional LSTM includes forward LSTM and reverse LSMT. When processing the input sequence, the forward and reverse information of the input sequence are considered at the same time. This means that the bidirectional LSTM model can obtain contextual information from the past (forward) and the future (reverse) at each time step, so as to more accurately understand the dependencies in the input sequence.

[0026] Transformer is a neural network architecture based on the attention mechanism. It effectively models the input sequence through components such as self-attention and positional encoding. The attention mechanism enables Transformer to dynamically focus on key information in the input sequence (information related to the recognition of proper nouns), thereby showing higher efficiency and accuracy in processing natural language tasks.

[0027] In an exemplary embodiment, the input sequence is processed by a context-aware proper noun recognition model to identify proper nouns in the text to be translated, including: using the forward recurrent neural network model to start processing from the starting position of the input sequence to obtain the forward hidden state of the input sequence at each time step; using the reverse recurrent neural network model to start processing from the end position of the input sequence to obtain the reverse hidden state of the input sequence at each time step; merging the forward hidden state and the reverse hidden state of each time step to obtain the hidden state of the bidirectional neural network model at each time step; using the transformer model based on the attention mechanism to linearly transform the hidden state of each time step to obtain a query matrix, a key matrix and a value matrix respectively; obtaining an attention weight matrix based on the query matrix and the key matrix; processing the value matrix based on the attention weight matrix to obtain the feature vector of each position in the input sequence; processing the feature vector of each position to identify the proper nouns in the text to be translated.

[0028] The context-aware proper noun recognition model (also referred to as a context-aware module) provided in the embodiments of the present disclosure combines the advantages of bidirectional LSTM and Transformer, and can improve the recognition accuracy of proper nouns.

[0029] In S130, candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated is retrieved from the dynamic vocabulary, and a target translation of the proper nouns in the text to be translated is determined based on the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated.

[0030] The method provided in the embodiment of the present disclosure also integrates dynamic vocabulary technology. The dynamic vocabulary in the embodiment of the present disclosure is designed as a hierarchical structure to support fast updating and retrieval. The dynamic vocabulary mainly includes two parts: a common proper noun layer and a domain-specific vocabulary layer. The common proper noun layer covers basic proper nouns (i.e. proper nouns in daily contexts, such as the proper nouns in the common fields mentioned above). The domain-specific vocabulary layer covers professional terms or proper nouns in specific fields, which can be updated in real time according to external data sources. The dynamic vocabulary can also be called a proper noun vocabulary, which includes domain classification and corresponding domain identification. For example: setting a domain field as a domain identification, the domain field can include common fields, mechanical fields, artificial intelligence fields, computer fields, medical fields, etc. The specific domain division can be divided according to actual needs, and the present disclosure does not limit this. Each field has its own corresponding domain vocabulary, that is, the proper nouns of the corresponding field are stored under each domain identification.

[0031] When updating the dynamic vocabulary, first determine whether the new vocabulary is a proper noun in a common field or a specific vocabulary in a specific field. If it is a proper noun in a common field, the new vocabulary is stored in the common proper noun layer; if it is a specific vocabulary in a specific field, the new vocabulary is stored under the corresponding field identifier in the field-specific vocabulary layer.

[0032] In the disclosed embodiment, a real-time update mechanism is designed for the dynamic vocabulary. The system sets a timed task, regularly obtains new proper noun entries from external data sources (such as professional journals, industry technical reports, etc.) and automatically updates them to the domain vocabulary. In an exemplary embodiment, the dynamic vocabulary can be connected to the external data source through an API interface (Application Programming Interface), regularly pulls data from the external data source to update the content in the dynamic vocabulary, and classifies the entries, and stores the new vocabulary in the corresponding field after classification. In some embodiments, the update module of the dynamic vocabulary uses natural language processing technology to clean and tag the new words obtained from the external data source to ensure the accuracy of the dynamic vocabulary. For example, after obtaining a new article from an external professional journal, first perform word segmentation and part-of-speech tagging on it, find all the nouns in it, and then match them one by one with the existing proper nouns in the dynamic vocabulary. If there is no match, it is identified as a new proper noun entry or entry. If the external professional journal is a medical journal, the new noun is stored in the vocabulary of the medical field of the dynamic vocabulary. Alternatively, although the proper noun is matched in the dynamic vocabulary, the field to which it belongs in the dynamic vocabulary is different from the field to which the external professional journal belongs. In this case, the new proper noun entry or term can be stored in the dynamic vocabulary corresponding to the field to which the external professional journal belongs.

[0033] In an exemplary embodiment, the dynamic word library includes a domain identifier and proper nouns corresponding to the domain identifier, their translations, and the number of citations.

[0034] In the disclosed embodiment, each proper noun and its translation in the field are stored in the dynamic vocabulary by field. For a proper noun with multiple meanings, different translations may be provided in different fields, and the same translation may also correspond to different fields. Exemplarily, the number of times each translation is cited in the corresponding field can also be stored in the dynamic vocabulary. The number of citations in the disclosed embodiment refers to the number of times the translation has been adopted (confirmed or uncorrected) in the corresponding field during the previous translation process. The reliability of the translation in the corresponding field can be reflected by the number of citations.

[0035] In an exemplary embodiment, candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated is retrieved from a dynamic vocabulary, and a target translation of the proper nouns in the text to be translated is determined according to the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated, including: if a proper noun matching the proper noun in the text to be translated is queried from the dynamic vocabulary, the matching proper noun is used as the candidate proper noun; if a proper noun matching the proper noun in the text to be translated is not queried from the dynamic vocabulary, similar proper nouns whose semantic similarity with the proper noun in the text to be translated meets a condition are retrieved from the dynamic vocabulary as the candidate proper noun; the translation of the candidate proper noun and the domain and number of citations corresponding to the domain identifier thereof are used as the candidate translation information; and the translation of the candidate proper noun whose domain matches the domain information of the proper noun in the text to be translated and has the largest number of citations is determined as the target translation.

[0036] In the disclosed embodiment, a dynamic interaction mechanism between machine translation and a dynamic vocabulary is designed. During the translation process of the text to be translated, the machine translation system will first retrieve the entries in the dynamic vocabulary and perform full-word matching, that is, the proper nouns in the text to be translated identified by the context-aware proper noun model will be matched with the proper nouns in various fields in the dynamic vocabulary. The full-word matching here means that the proper nouns in the dynamic vocabulary are matched one by one with each character or word in the proper nouns in the text to be translated. For example, if the proper noun in the text to be translated is "instruction set", then the proper noun in the dynamic vocabulary that matches it with the full word is also "instruction set". The proper nouns in the dynamic vocabulary that match the proper nouns in the text to be translated with the full word are used as candidate proper nouns.

[0037] If the full word match fails, that is, no proper noun matching the full word of the proper noun in the text to be translated is found in the dynamic vocabulary, then the semantic similarity of the vocabulary is calculated based on the word embedding technology, and similar entries are found in a vector manner as candidate proper nouns or candidate entries. Specifically, the proper nouns in the text to be translated and the proper nouns in the dynamic vocabulary are converted into word embedding vectors respectively, and then the semantic similarity between the proper nouns in the text to be translated and the word embedding vectors of each proper noun in the dynamic vocabulary is calculated, such as cosine similarity. Then the proper nouns in the dynamic vocabulary that meet the semantic similarity conditions are used as candidate proper nouns. For example, the proper nouns with the maximum semantic similarity or the semantic similarity greater than a preset threshold or the first preset number are used as candidate proper nouns. The value of the preset threshold or the first preset number can be set according to actual needs, and the present disclosure does not limit this.

[0038] If a candidate entry is found, the machine translation system will recommend the most suitable translation as the target translation based on the context. For example, the translation of the candidate proper nouns with the same or similar fields as the proper nouns in the text to be translated and the most cited times will be selected as the target translation.

[0039] If no entry is found in the dynamic vocabulary that matches the full word of the proper noun in the text to be translated or that satisfies the semantic similarity condition, the machine translation system will provide a preliminary translation suggestion for the proper noun in the text to be translated and mark it as "pending verification", i.e., mark it with a pending verification mark information. After the user confirms or corrects the preliminary translation suggestion, the entry (i.e., the proper noun and its preliminary translation suggestion or the corrected translation) is automatically added to the dynamic vocabulary to provide more accurate support for subsequent translation. Exemplarily, the preliminary translation suggestion can be generated by a translation model.

[0040] In S140, the text to be translated is processed by a translation model to obtain a preliminary translation result of the text to be translated, wherein the preliminary translation result includes preliminary translation suggestions for proper nouns in the text to be translated.

[0041] The translation model in the embodiments of the present disclosure refers to a deep learning model that converts concepts or entities in one language into concepts or entities in another language. It can also be called a machine translation model. It uses deep learning and natural language processing technology to achieve automatic cross-language translation. It can use one or more of recurrent neural networks, convolutional neural networks, Transformer models, etc.

[0042] In the disclosed embodiment, the translation obtained by the translation model translating the proper nouns in the text to be translated is called a preliminary translation suggestion.

[0043] In S150, a target translation result of the text to be translated is generated according to the preliminary translation result and the target translation of the proper noun in the text to be translated.

[0044] In an exemplary embodiment, generating a target translation result of the text to be translated based on the preliminary translation result and the target translation of the proper noun in the text to be translated includes: if the target translation is retrieved, replacing the preliminary translation suggestion of the proper noun in the text to be translated in the preliminary translation result with the target translation to generate the target translation result of the text to be translated. That is, if a target translation matching the proper noun in the text to be translated is retrieved in the dynamic vocabulary, the preliminary translation suggestion is replaced with the target translation to obtain a more accurate target translation result.

[0045] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: displaying the target translation result on the translation output interface, and displaying a hidden control at the associated position of the target translation of the proper noun in the target translation result; in response to the triggering operation of the hidden control, displaying the number of citations of the target translation in various fields, the translations of other candidate proper nouns of the proper noun corresponding to the target translation and their fields and the number of citations, and a correction control in sequence according to the number of citations; in response to the triggering operation of the correction control, obtaining feedback data of the target translation of the proper noun in the text to be translated, the feedback data including the annotated fields and feedback translations of the proper noun in the text to be translated; updating the dynamic vocabulary according to the annotated fields and feedback translations of the proper nouns; and training the context-aware proper noun recognition model using the proper nouns and their feedback translations.

[0046] The hidden control in the embodiment of the present disclosure refers to any appropriate indication that the vocabulary in the target translation result is a proper noun, which can be an implicit indication or an explicit indication. For example, an icon is displayed in the upper right corner of the target translation to indicate that the target translation is a translation of a proper noun. For another example, the target translation is highlighted in yellow or bolded in black to indicate that it is a translation of a proper noun. For another example, there is nothing special about the target translation. When the user selects or clicks on the target translation, the number of citations of the target translation in various fields, the translations of other candidate proper nouns of the proper noun corresponding to the target translation and their fields and the number of citations, and the correction control will be displayed in order according to the number of citations. The present disclosure does not limit the representation method of the hidden control.

[0047] In the disclosed embodiment, the number of times the target translation is cited in the field to which the text to be translated belongs, as well as the number of times the target translation is cited in other fields (if any) can be displayed on the translation output interface. If the proper noun has other translations in the field, the other translations of the proper noun in the field and the number of times they are cited can also be displayed. If the translation and / or the other translations can also be used in other fields, the number of times the translation and / or the other translations are cited in other fields can be displayed.

[0048] In the disclosed embodiment, a correction control can also be displayed, which allows the user to correct the target translation of the proper noun in the target translation result provided by the machine translation system. For example, if the displayed fields are different from the field information of the proper noun of the text to be translated, or although the fields are the same, the target translation is wrong, then the user can trigger the correction control, enter the new field and / or the correct translation as the marked field and the feedback translation, and the marked field and the feedback translation are used as the user's feedback data for the proper noun. Exemplarily, the feedback data can be used to update the translation of the proper noun in the corresponding field in the dynamic vocabulary, or add new fields and the proper nouns and translations under them, or correct the original wrong translation in the dynamic vocabulary. Exemplarily, the feedback data can also be used to continue training the context-aware proper noun recognition model completed by the above training, so as to adjust the model parameters of the context-aware proper noun recognition model, and improve the accuracy of the model to the recognition of proper nouns.

[0049] In an exemplary embodiment, in response to the selection operation of the displayed translation of other candidate proper nouns, the target translation in the target translation result can be replaced with the selected translation of other candidate proper nouns. The translation of the other candidate proper nouns can be used to replace the target translation under the corresponding field identification in the dynamic word library, or the number of citations of the target translation can be decreased and the number of citations of the translation of the other candidate proper nouns can be increased.

[0050] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: if the target translation is not retrieved, taking the preliminary translation result as the target translation result, and marking the preliminary translation suggestion with pending verification mark information; displaying the target translation result on the translation output interface, and displaying a hidden control at the associated position of the preliminary translation suggestion of the proper noun in the target translation result; in response to a triggering operation on the hidden control, displaying the pending verification mark information of the preliminary translation suggestion and a correction control; in response to a triggering operation on the correction control, obtaining feedback data on the preliminary translation suggestion of the proper noun in the text to be translated, the feedback data including the annotated fields and feedback translations of the proper noun in the text to be translated; updating the dynamic vocabulary according to the annotated fields and feedback translations of the proper nouns; and training the context-aware proper noun recognition model using the proper nouns and their feedback translations.

[0051] The text processing method provided by the embodiment of the present disclosure can, on the one hand, improve the recognition accuracy of proper nouns in the text to be translated by processing the input sequence of the text to be translated based on the context-aware proper noun recognition model; on the other hand, determine the target translation of the proper noun by combining the dynamic vocabulary and the domain information of the text to be translated and the proper noun output by the above-mentioned context-aware proper noun recognition model, and then comprehensively consider the target translation of the proper noun and the preliminary translation result obtained by the translation model, so as to improve the translation accuracy of the proper noun and avoid mistranslation and omission.

[0052] Below through Figure 2 The method provided in the embodiment of the present disclosure is illustrated by way of example. Figure 2 As shown, the text processing method provided by the embodiment of the present disclosure may include the following steps.

[0053] In S210 , a text to be translated is input.

[0054] In the disclosed embodiment, the input is the original text to be translated, also referred to as input text.

[0055] In S220, the text to be translated is segmented and POS tagged to obtain an input sequence.

[0056] In the disclosed embodiment, the input text is segmented to obtain each word in the input text, and is ready to enter a subsequent language model, including a proper noun recognition model and a translation model based on context awareness.

[0057] In an exemplary embodiment, each word after word segmentation is tagged with a part-of-speech tag, and each word is assigned a part-of-speech tag, for example, each word is marked as a noun, a verb, an adjective, etc., to help the language model understand the grammatical structure of the text to be translated.

[0058] For example, the word segmentation and part-of-speech tagging of the text to be translated can be implemented by a part-of-speech tagger such as NLTK (Natural Language Toolkit), but the present disclosure is not limited thereto.

[0059] In an exemplary embodiment, each word may also be converted into an embedded vector representation, and then the embedded vector representation of each word is arranged in order according to its position in the text to be translated to form an input sequence.

[0060] In S230 , the input sequence is processed by a context-aware proper noun recognition model to recognize proper nouns.

[0061] In the disclosed embodiment, after the machine translation system receives the text to be translated, each sentence in the text to be translated is subjected to word segmentation, syntactic analysis and part-of-speech tagging. The bidirectional LSTM in the context-aware proper noun recognition model is responsible for transferring the historical context of each word or phrase to the current word or phrase or the current sentence, and by memorizing and analyzing the context information (including domain information) in the sentence, the machine translation system makes the judgment of the proper noun more accurate. The attention mechanism in the context-aware proper noun recognition model can dynamically adjust the attention weight (the adjustment of attention weight can be realized by the attention weight matrix, i.e., more attention is paid to the context information related to the proper noun) to specific vocabulary (here referring to the proper noun in the field) according to the context of the sentence or word or phrase, and ensure the semantic consistency between different sentences of the text to be translated.

[0062] In S231, a bidirectional LSTM is used to process the input sequence and output the hidden state of each time step.

[0063] In S232, the hidden state of each time step of the bidirectional LSTM output is processed through the self-attention mechanism in the transformer model to identify proper nouns in the text to be translated.

[0064] The context-aware proper noun recognition model or context-aware model in the disclosed embodiment is based on the attention mechanism in the LSTM and Transformer architecture, combined with the background thought chain arrangement to enhance the semantic parsing ability. LSTM can handle long-distance dependencies of long texts to ensure that context information is fully captured. The attention mechanism can focus on the key content in the context (proper nouns in this specific context) during the translation process, ensuring that the system accurately recognizes proper nouns in long texts and complex contexts.

[0065] In the disclosed embodiment, long-distance dependency is realized by LSTM, and dependency can be captured by cache unit. At the same time, bidirectional LSTM overcomes the problem that a single head only processes a single time step relationship. Bidirectional LSTM transmits information from both the front and back directions of the text to be translated, thereby capturing contextual information more comprehensively. Because the information in the context may be distributed at both ends of the sentence, unidirectional LSTM cannot fully capture it. The memory unit or hidden state output by the bidirectional LSTM saves the key contextual information of the input text / input sequence (i.e., contextual information related to the proper nouns in the field), especially long-distance dependency information, which provides a rich semantic basis for the subsequent attention mechanism and Transformer encoder layer.

[0066] The disclosed embodiment also uses thought chains to enhance memory. Thought chain arrangement refers to the process of machine logical thinking, which can be understood as intervening in the added business logic. The thought chain is used to assist the context-aware model to find the proper nouns in the text to be translated, that is, the added business logic can be input into the context-aware model. The entire process is a logical thought chain, and arrangement means that the process can be customized. You can configure what capabilities are required in which links.

[0067] The disclosed embodiment utilizes the self-attention of the Transformer encoder layer to analyze the memory units of the LSTM output, calculates the similarity and dependency between different words, and processes the context in the input sequence. The Transformer encoder layer captures key semantics through the self-attention mechanism to ensure the accuracy of proper noun recognition, so as to assist the accuracy of translation of the translation model. Exemplarily, position recognition is optimized, and absolute position encoding is used to help the context-aware model understand the position of a word or words in a sentence. Exemplarily, through the multi-head attention mechanism, the context-aware model can simultaneously focus on multiple different positions in the input text, thereby improving the accuracy of proper noun recognition.

[0068] The attention mechanism is a technique used to enhance the ability of a neural network to focus on key information in the input data. In natural language processing tasks, the attention mechanism is used in the encoder-decoder architecture to dynamically adjust the attention weights on different parts of the input sequence. In the task of proper noun recognition, the attention mechanism can dynamically adjust the attention weights on specific words based on the contextual information of the sentence. This means that when the system processes a sentence, it will pay more attention to words or phrases related to proper nouns, thereby ensuring semantic consistency between different sentences.

[0069] The context-aware model in the disclosed embodiment optimizes the recognition of proper nouns through the combination of LSTM, Transformer and background thinking chain. The system combines the context-aware model with the attention mechanism, and generates more accurate proper noun recognition results by globally understanding the context of the input text, thereby obtaining more accurate target translation results. In the natural language processing model (i.e., context-aware model) combining LSTM and attention mechanism, LSTM is responsible for processing long-term dependencies in sequence data and transmitting historical context information of each word. For example, in the same field, the proper noun A was translated as a in the past. Is a also used in the current text to be translated? The attention mechanism is responsible for dynamically adjusting the attention weight of specific vocabulary according to the context information of the sentence. The thinking chain arrangement serves as a background logic analysis module in this process, and realizes the capture of complex contextual relationships by gradually focusing on the key semantic units in the sentence. The output identified in each link is the key semantic unit of the current link. This collaborative working mode enables the model to more accurately understand the overall semantics of the sentence and more accurately identify the proper nouns in the sentence.

[0070] In the disclosed embodiment, the bidirectional LSTM includes a forward LSTM and a reverse LSTM. The forward LSTM starts processing from the beginning of the input sequence, and the reverse LSTM starts processing from the end of the input sequence. They each generate a hidden state at each time step.

[0071] There are two ways to merge the forward and reverse hidden states. One is simple concatenation, in which case the output of the bidirectional LSTM at each time step t is a vector with the dimension of the sum of the forward and reverse hidden states, which contains information from the context and helps capture long-distance dependencies in the sequence. The other way is addition, that is, the output dimension is the same as the single LSTM hidden state dimension. When identifying proper nouns, the hidden state output by the bidirectional LSTM at each time step is used as the input of the subsequent attention mechanism to predict whether each position is a proper noun.

[0072] Taking the self-attention mechanism in Transformer as an example, the input is the hidden state X output by the bidirectional LSTM at each time step. First, the query, key, and value matrices Q, K, and V are obtained by linearly transforming X. Each element vector V in V i Contains the feature information of X after transformation at position i. Then calculate the attention score matrix A, where:

[0073]

[0074] d k is the dimension of the key vector, A ijrepresents the element in the i-th row and j-th column of the attention score matrix A; Q i represents the i-th element in the Q matrix; represents the transpose of the jth element in the K matrix. i and j are both positive integers greater than or equal to 1. ij Perform softmax normalization to obtain the attention weight matrix Finally, the output of the self-attention mechanism is obtained by weighted summation V j represents the j-th element in the V matrix, assuming that j is a positive integer greater than or equal to 1 and less than or equal to n, and n is a positive integer greater than or equal to 1. express The element in the i-th row and j-th column of the matrix. For each position i in the input sequence, the output is a vector with the same dimension as the value vector V, which incorporates the information of each position in the entire input sequence according to the attention weight.

[0075] The self-attention mechanism generates the final output by calculating the attention weights and performing a weighted summation of the elements in V. The attention weights determine the contribution of each element vector in V to generating the output. For example, when processing a sentence, if a word is critical for identifying proper nouns, then the V at the corresponding position i In the weighted summation process, there will be a higher weight, so that the final output will pay more attention to the information related to proper nouns. In the task of processing sentence recognition of proper nouns, this output will emphasize the feature information related to proper nouns and highlight the important parts of the sentence for recognizing proper nouns. The attention mechanism will calculate the weights based on these inputs, re-weight and combine these features, and output a new feature representation that can better focus on the key information related to proper noun recognition, and then input it into the subsequent classification layer for prediction.

[0076] In the disclosed embodiment, before the input sequence is processed by the context-aware model, the context-aware model is trained using training samples. The training samples may be annotated corpus, that is, the training text includes proper nouns in the corresponding field, and the proper nouns are annotated as labels, the training text is input into the context-aware model to predict the proper nouns in the training text, and the predicted proper nouns are compared with the annotated proper nouns to train the context-aware model.

[0077] The context-aware model in the disclosed embodiment is trained on a large amount of cross-domain corpus data (such as legal, technical, medical and other professional texts). During the training process, the model learns to recognize nouns, verbs and other parts of speech, and automatically annotates with the help of the NLTK library to identify and distinguish between proper nouns and common nouns.

[0078] Specifically, sentences containing proper nouns are collected as training data, and the proper nouns in each sentence are marked. Sentences are divided into word or subword units. For example, tools such as NLTK are used. Words are converted to word embedding representations, and word embedding (Word Embedding) such as Word2Vec can be used. At the same time, corresponding labels are generated for each marked proper noun (such as proper nouns are marked as 1, and other words are marked as 0). Build a context-aware model containing bidirectional LSTM and attention mechanism. Bidirectional LSTM can process sequences from both forward and reverse directions to better capture contextual information. The forward LSTM processes from the beginning to the end of the sentence, and the reverse LSTM processes from the end to the beginning of the sentence, and then merges the hidden states in both directions. The attention mechanism allows the model to dynamically focus on different parts of the input sequence when processing each position. In Transformer, the self-attention mechanism calculates the relevance score of each position with all other positions, and then weights the input according to these scores. The output of the bidirectional LSTM is used as the input of the attention mechanism, and then a fully connected layer is connected for classification to identify whether it is a proper noun. Train the model using the prepared training data. Use the test data to evaluate the performance of the model, such as accuracy, recall, and F1 value.

[0079] In S240, candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated is retrieved from the dynamic vocabulary, and a target translation of the proper nouns in the text to be translated is determined based on the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated.

[0080] In S250, the text to be translated is processed by a translation model to obtain a preliminary translation result of the text to be translated, wherein the preliminary translation result includes preliminary translation suggestions for proper nouns in the text to be translated.

[0081] In S260, the target translation replaces the preliminary translation suggestion of the proper noun in the text to be translated in the preliminary translation result to generate a target translation result of the text to be translated.

[0082] In S270, the target translation result is displayed on the translation output interface, and a hidden control is displayed at a location associated with the target translation of the proper noun in the target translation result.

[0083] In S280, in response to the triggering operation of the hidden control, the number of citations of the target translation in each field, the translations of other candidate proper nouns corresponding to the target translation and their fields and citation times, and the correction control are displayed in order according to the number of citations.

[0084] In S290, in response to the triggering operation of the correction control, feedback data of the target translation of the proper noun in the text to be translated is obtained, and the feedback data includes the marked field and the feedback translation of the proper noun in the text to be translated.

[0085] In S2100, the dynamic vocabulary is updated according to the annotated domain and the feedback translation of the proper noun.

[0086] In S2110, the context-aware proper noun recognition model is trained using the proper nouns and their feedback translations.

[0087] The method provided by the embodiment of the present disclosure also provides a user feedback optimization mechanism. Feedback collection is performed first. The system provides users with annotations (i.e., displaying the number of citations of the target translation in various fields in order according to the number of citations, the translations of other candidate proper nouns of the proper nouns corresponding to the target translation and their fields and the number of citations) and correction options (i.e., correction controls) on the translation output interface. Users can make correction suggestions for the target translation or preliminary translation suggestions of the proper nouns in the target translation results, and these feedback data will be collected and transmitted to the feedback module in real time.

[0088] The feedback module can regularly integrate the user's correction data / feedback data into the dynamic vocabulary, and use this data to update the context-aware model. The update process includes tagging the word part and context (i.e., domain) of the word in the user feedback (the original text translation comparison interface has a tagging function, and the user can enter the domain), and retraining the context-aware model based on the alternative vocabulary selected by the user, so that the weight of the context-aware model is gradually adjusted to the correct word, ensuring that the context-aware model can automatically and correctly identify proper nouns in similar contexts in the future, and can apply the correct proper noun translation in combination with the dynamic vocabulary.

[0089] The disclosed embodiment is designed with closed-loop optimization. After each context-aware model and dynamic vocabulary are updated, the system will retrain the corpus of the specific field and further fine-tune the context-aware model using the latest data from user feedback. As user feedback continues to accumulate, the system gradually improves the recognition accuracy of proper nouns and significantly reduces the missed translation of emerging words and field-specific words.

[0090] In the disclosed embodiment, the word fed back by the user can be directly input into the context-aware model for training, or the word and its context can be input into the context-aware model for training. Context means language context.

[0091] In view of the problems existing in the related art, the disclosed embodiment proposes a comprehensive proper noun translation optimization method, which includes a context-aware model, a dynamic vocabulary integration technology and a user feedback optimization mechanism. By combining the advantages of LSTM and Transformer, the ability to accurately identify proper nouns in complex contexts is improved. LSTM is used to process the dependencies of long texts, while the attention mechanism of Transformer can focus on the key content in the context, thereby ensuring the accuracy of the system when processing long texts and complex contexts.

[0092] The disclosed embodiments significantly improve the proper noun translation performance of the machine translation system in the following aspects through the comprehensive application of context-aware models, dynamic word library integration, and user feedback optimization mechanisms:

[0093] On the one hand, the recognition accuracy is significantly improved: through the deep semantic analysis of the context-aware model, it is possible to maintain high recognition accuracy in complex contexts. Compared with the translation system based on the static vocabulary, the use of LSTM and attention mechanism ensures a more comprehensive understanding of long texts and cross-sentence contexts, thereby reducing the omission and mistranslation of proper nouns. The thought chain arrangement characteristics of the context-aware model enable it to automatically adjust the focus on important information in the sentence when parsing the syntactic structure, effectively solving the recognition bias problems that may be caused by polysemous words, homophones, etc. in different contexts, and ensuring the semantic consistency of the translation.

[0094] On the other hand, dynamic vocabulary is updated in a timely manner and has stronger adaptability: dynamic vocabulary integration technology realizes automatic updating of external data sources, and through real-time connection with data sources such as professional journals and industry reports, it ensures that the system vocabulary can quickly adapt to changes in emerging vocabulary and professional terms. This update method solves the problem of delayed vocabulary updates in traditional translation systems and ensures the professionalism and accuracy of the translation. In addition, the hierarchical structure of the dynamic vocabulary not only improves the retrieval efficiency of the vocabulary, but also allows the system to perform priority matching of entries for specific fields during translation (corresponding to the frequency of use or number of citations of the entry), thereby improving the quality of translation. The system can update and apply new professional vocabulary in a short period of time to adapt to complex translation needs in multi-field and multi-language environments.

[0095] On the other hand, the flexibility and adaptability of the system are significantly enhanced: the user feedback optimization mechanism enables the system to have self-adjustment capabilities. Users can directly mark and update the errors in the proper nouns in the translation results in the translation output interface. These feedback data will be automatically integrated by the system and used for adaptive training of the model. The system can improve the vocabulary and model weights (ie, model parameters) in real time based on user feedback to ensure that the same proper nouns are handled more accurately in future translation tasks. This closed-loop optimization mechanism enables the system to respond quickly to user needs, especially in professional field applications that require high translation accuracy, significantly reducing the user's proofreading workload. Compared with the traditional intelligent agent collaboration method, the present disclosure achieves more flexible self-optimization of translation output through a direct feedback closed loop.

[0096] On the other hand, the performance of proper noun translation can be improved in the long term: due to the continuous accumulation of user feedback data, the dynamic vocabulary and context-aware model in the present disclosure can achieve long-term optimization on the basis of continuous improvement. User feedback on proper nouns is automatically integrated by the system, and the content of the vocabulary is continuously enriched, so that the adaptability and accuracy of the system in different fields are continuously improved. The feedback mechanism of the present disclosure allows the system to gradually build a vocabulary containing a large number of high-quality proper nouns, providing reliable data support for subsequent translation tasks. Especially in fields that need to be constantly updated (such as science and technology, law, medicine, etc.), this mechanism ensures that the translation quality of the system always remains at a high level and reduces the efficiency of manual intervention.

[0097] On the other hand, it can improve translation efficiency and reduce manual proofreading costs: the combination of the dynamic word library and the context-aware model disclosed in the present invention makes the recognition and processing of proper nouns in the translation process more automated. The introduction of the user feedback mechanism reduces the occurrence of mistranslation of proper nouns and reduces the cost of manual proofreading, which is particularly suitable for large-scale, multi-language professional document translation tasks.

[0098] On the other hand, the dynamic word library real-time interactive technology and automatic update mechanism greatly reduce the system preparation time before each task starts, improving the overall translation efficiency. Users can quickly obtain highly accurate translation results, and the system maintenance cost is significantly reduced.

[0099] Figure 3 The following is a schematic diagram of a translation interface according to an embodiment of the present disclosure. Figure 3 As shown, the translation interface 300 includes an original text upload control 310 , an input text control 320 , and a translation setting area 330 .

[0100] The user can click the original text upload control 310 to directly upload the text to be translated at the file or document level. The user can also directly input the text to be translated in the input text control 320.

[0101] The translation setting area 330 may include a term prompt control 340 and a model selection control 350, in which a technical field can be selected. When the user selects a corresponding technical field in the drop-down box of the term prompt control 340 and / or the model selection control 350, the field information of the text to be translated can be determined.

[0102] Figure 4 The following schematically shows an interface diagram of a translation output interface according to an embodiment of the present disclosure. Figure 4 As shown, the translation output interface 400 includes an original text display area 410 and a translation display area 420. The original text display area 410 can be used to display the text to be translated. The translation display area 420 can be used to display the target translation result of the text to be translated.

[0103] refer to Figure 4 , the target translation of a proper noun Z in the translation is assumed to be "AAA", then when the user clicks the hidden control of "AAA", the number of times the target translation is cited in each field, the translations of other candidate proper nouns corresponding to the target translation and their fields and the number of times they are cited are displayed in order according to the number of citations 430. For example, it is displayed that the target translation "AAA" is used 30 times in this field (that is, the number of times "AAA" is cited in the field corresponding to the field information of the text to be translated), and 20 times in the medical field; it is also displayed that the translation of the proper noun Z "BBB" is used 20 times in this field and 20 times in the mechanical field; it is also displayed that the translation of the proper noun Z "CCC" is used 10 times in this field and 20 times in the medical field; it is also displayed that the translation of the proper noun Z "DDD" is used 4 times in this field and 20 times in the medical field. That is, a list of candidate words for proper nouns is obtained, and is displayed to the user in order according to weights such as the number of times used or the number of times cited.

[0104] Figure 4 In the embodiment, a correction control 440 is also displayed. "No corresponding term in the current term library, click to add" means that the target translation, the translation of the candidate proper noun, and the field found in the dynamic word library do not match the proper noun Z in the text to be translated, and the user can click the correction control 440 to input the field and translation, and obtain the marked field and feedback translation.

[0105] For example, after a sentence passes through the context-aware model and the translation model, the target translation result will be given. The recognition result / target translation result can be customized according to the context and user needs. For example, the machine recognizes that the proper noun A is translated as a, but the text to be translated is a new field and there is no relevant word in the candidate list. At that time, a new translation can be added as aa. aa is used as the new field translation of the proper noun A. In the text to be translated, the weight of aa is adjusted to the highest priority by user intervention.

[0106] The solution provided by the embodiments of the present disclosure involves the recognition and translation of proper nouns, covering multiple links such as text processing, context-aware recognition of proper nouns, translation, dynamic word library management, user feedback optimization, etc. Through the collaborative work of these technologies and methods, the system can improve the translation accuracy of proper nouns, adapt to the needs of multiple fields, and continuously optimize its own performance through user feedback.

[0107] The following is a brief description of the main technologies and terms involved in the solution provided by the embodiment of the present disclosure. MultipartFile is an object for processing HTTP file uploads, which can receive file data and convert it into a stream for processing on the server side. MultipartFile is mainly used to process file upload functions in Web applications. fetch is a function for asynchronous requests that supports sending HTTP requests to the server to achieve file upload or data transmission. @PostMapping: used to define the mapping path of HTTP POST requests, used for operations such as data submission and file upload. Regular expressions (regex) are string patterns used for text matching and processing. Character patterns are defined to identify specific text formats or remove invalid characters. LSTM (Long Short-Term Memory Network) is a recursive neural network that processes sequence data. It can retain contextual information in long sequences and is suitable for processing dependencies in long texts. Transformer is a neural network model based on the attention mechanism, which is good at capturing important information in sequence data and is used for semantic analysis of complex contexts. StringBuilder is a class used for string processing in Java. It improves the performance of string operations by dynamically splicing and modifying string content. Word segmentation tool is a tool used to segment text into individual words or phrases to facilitate subsequent grammatical analysis and part-of-speech tagging. A part-of-speech tagger is a tool that automatically tags the grammatical roles of words, marking the words in the text as nouns, verbs, adjectives, etc., to facilitate the understanding of syntactic structure. Similarity algorithms are algorithms used to calculate the semantic similarity between two words or phrases, and recommend the words that best fit the context through word embedding or other techniques. Word2Vec is a word embedding model that converts words into numerical vectors so that semantic similarity can be evaluated by comparing the similarity between vectors. TTL (time to live) is a setting in the Redis cache that controls the storage time of data in the cache. Data that exceeds the set time will be automatically deleted to save storage space.

[0108] Hierarchical vocabulary is a database design that stores vocabulary in layers according to basic vocabulary, domain vocabulary, etc., and supports priority management of proper nouns in different fields. Ajax (asynchronous JavaScript and XML) is used to asynchronously obtain data and update the interface without reloading the page to improve user experience. Redis is an efficient in-memory database that is suitable for storing temporary data and cached data, and is often used in high-frequency read and write application scenarios. The Adam optimization algorithm is an optimization algorithm for deep learning models. It improves the training efficiency of the model by dynamically adjusting the learning rate to prevent the model from overfitting. Indexes can be used to set up a fast search mechanism for specific fields in the database to speed up queries. It is often used for frequently queried fields. evaluatePerformance is a custom performance evaluation method used to calculate the translation accuracy of the system, etc., as a basis for system optimization. cleanDictionary is a custom cleanup method that regularly deletes unaccessed entries in the vocabulary to maintain the timeliness and efficiency of the vocabulary.

[0109] The method provided by the embodiment of the present disclosure may include the following steps: First, perform system configuration and initialization.

[0110] Implementation of the file upload interface: that is, on the provided user interface (translation interface), users have two ways to upload the text to be translated, one is to upload the word file directly, and the other is to manually enter the text to be translated (corresponding to the text input interface below). Front-end file selection and upload: After the user selects the file on the front end, the JavaScript fetch method constructs a POST request and sends the file to the server in multipart / form-data format. Back-end file reception and storage: method saveUploadedFile(MultipartFile file, String targetDir). Among them, file represents the received file object. targetDir represents the target directory path for storing files. It is used to write the uploaded file from the memory to the specified directory to avoid losing the file due to server restart. The implementation steps include:

[0111] 1. Verify whether file is empty. If it is empty, return an error.

[0112] 2. Call file.getOriginalFilename() to get the file name and perform format check.

[0113] 3. Create a file path object

[0114] Path targetPath=Paths.get(targetDir,file.getOriginalFilename()).

[0115] 4. Use the Files.copy() method to save the file to the target path.

[0116] Implementation of text input interface. Front-end input limit: front-end JavaScript monitors the character length of the input box. If the length exceeds the system limit, validateTextLength(inputText, maxLength) is used.

[0117] Method for verification. Backend data filtering: method sanitizeText(String inputText). Among them, inputText represents the original text entered by the user, that is, the text to be translated. It is used to filter out special characters and invalid symbols through regular expressions to keep the input text clean. The implementation steps include:

[0118] 1. Define the regular expression String regex = "[^a-zA-Z0-9\u4e00-\u9fa5]" to retain English letters, numbers, and Chinese characters.

[0119] 2. Call inputText.replaceAll(regex, "") to return the cleaned text.

[0120] Implementation of the context-aware module: method translateTextWithContext(String inputText). Among them, inputText represents the text to be translated entered by the user, including the text to be translated uploaded through the file upload interface and the text to be translated uploaded through the text input interface. It is used to call the LSTM and Transformer models for context analysis, and call the translation model to generate the target translation result. The implementation steps include:

[0121] 1. Segment the inputText and mark its parts of speech to generate a grammatical structure.

[0122] 2. Pass the structure into the LSTM model to obtain contextual associations.

[0123] 3: Pass the LSTM output into the Transformer to generate a context-sensitive target translation result based on the attention weights.

[0124] Process the text and perform context analysis. File parsing and preprocessing call the parseDocument(File file). Among them, File represents the DOCX file uploaded by the user. It is used to parse the DOCX file, extract the text content segment by segment and return the plain text. The implementation steps include:

[0125] 1. Check the file format and make sure it is DOCX.

[0126] 2. Use the document parser to read each piece of text and store it in a StringBuilder object.

[0127] 3. Return the complete plain text content for subsequent processing. The complete plain text content refers to all the text content in the DOCX file uploaded by the user.

[0128] The word segmentation and part-of-speech tagging method calls analyzeSyntax(String text). Among them, text represents the text string to be analyzed. It is used to segment the text and tag the part-of-speech, and return the tagged vocabulary list. The implementation steps include:

[0129] 1. Call the word segmentation tool to split the text into a list of words.

[0130] 2. Use a part-of-speech tagger to assign a part-of-speech tag (such as noun, verb, etc.) to each word.

[0131] 3. Return the annotated vocabulary and part-of-speech list.

[0132] 4. Interaction between proper noun translation and dynamic vocabulary.

[0133] The proper noun matching and recommendation method calls findBestMatch(String term). Term represents the proper noun entered by the user. It is used to match the best proper noun translation in the vocabulary. The implementation steps include:

[0134] 1. Through exact matching, check whether there is an exact matching term in the vocabulary.

[0135] 2. If there is no matching result, use the similarity algorithm to calculate the semantically similar terms.

[0136] 3: Sort by context weight and return the best matching terms. That is, matching fields and words, when provided to users, sort by frequency, that is, the number of citations.

[0137] The dynamic word library update calls the updateDictionary(String term,String definition,String category) method. Term indicates a new word. Definition indicates the definition or translation of a word. Category indicates the field or category to which the word belongs. The original classification is similar to the industry classification and IPC classification number. New words are supplemented according to the original classification, and if they do not exist, they are added (a small probability event). It is used to automatically classify new words and store them in the word library. The implementation steps include:

[0138] 1: Check if the term already exists in the vocabulary. If so, update its definition and classification.

[0139] 2: Store new entries into the vocabulary by category and set indexes for quick retrieval later.

[0140] 3: Record the update time for subsequent vocabulary maintenance.

[0141] 4: Translation output and user feedback.

[0142] The method displayTranslation(String translatedText) is called for translation display and user feedback. Here, translatedText represents the translated text. It is used to dynamically display the translation results on the user interface, allowing users to view or modify the translation. The implementation steps include:

[0143] 1: Call Ajax to refresh the interface and display the translatedText on the user side.

[0144] 2: After clicking on a proper noun, the user can view the recommended alternatives in the vocabulary and modify the inaccurate translation. That is, the context-aware model and the translation model output the translation results of the entire article, and for the proper nouns in it, alternative options are also given using the above-mentioned dynamic vocabulary.

[0145] The user feedback collection method collectFeedback(String term, StringuserCorrection) is called. Term is the proper noun to be translated. UserCorrection is the modification suggestion given by the user. It is used to collect user feedback on a specific translation and store it in the cache. The implementation steps include:

[0146] 1: Store term and userCorrection in the cache and mark them as "pending optimization".

[0147] 2: Regularly synchronize cached data to the vocabulary.

[0148] 5: Systematic learning and self-adjustment.

[0149] Regular feedback data training call method trainModelWithFeedback(Map<String,String> feedbackData). feedbackData represents the data of user feedback, the key is the term, and the value is the user's correction suggestion. It is used to fine-tune the model with the user's feedback data so that the model can better adapt to the user's word preference. The implementation steps include:

[0150] 1: Clean feedbackData to remove duplicate and invalid feedback.

[0151] 2: Readjust the model weights based on feedback data to make the model more in line with user needs in subsequent translations.

[0152] The regular maintenance and optimization of the dynamic word library calls the method cleanDictionary(int daysInactive). Among them, daysInactive represents the threshold of the number of days that the entry has not been accessed. It is used to regularly clean up the unaccessed entries in the word library to keep the word library concise and real-time. The implementation steps include:

[0153] 1: Search all unvisited entries in the vocabulary.

[0154] 2: Delete entries that have not been accessed for more than daysInactive days.

[0155] 3: Record the cleanup log to ensure the traceability of the deletion operation.

[0156] System performance evaluation and optimization call method evaluatePerformance(). No parameters. Used to regularly evaluate the accuracy of system translation and feedback response rate, and trigger automatic optimization. The implementation steps include:

[0157] 1: Statistics on translation accuracy and user feedback response time.

[0158] 2: If the system performance is lower than the preset standard, the automatic optimization process (including model fine-tuning and vocabulary update) is triggered.

[0159] Furthermore, an embodiment of the present disclosure also provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method provided in any embodiment of the present disclosure is implemented.

[0160] Furthermore, an embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any embodiment of the present disclosure.

[0161] Furthermore, an embodiment of the present disclosure also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the method provided by any embodiment of the present disclosure is implemented.

[0162] The text processing device of the embodiment of the present disclosure may be arranged in a terminal device, or may be arranged in a server end, or may be arranged in part in a terminal device and in part in a server end. Figure 5As shown, the text processing device 500 provided by the embodiment of the present disclosure includes a receiving unit 510 , a processing unit 520 , a retrieval unit 530 and a generating unit 540 .

[0163] The receiving unit 510 is used to obtain an input sequence of a text to be translated and domain information of the text to be translated.

[0164] The processing unit 520 is configured to process the input sequence through a context-aware proper noun recognition model to recognize proper nouns in the text to be translated.

[0165] The retrieval unit 530 is used to retrieve candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated from the dynamic vocabulary, and determine the target translation of the proper nouns in the text to be translated based on the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated.

[0166] The processing unit 520 is further configured to process the text to be translated through a translation model to obtain a preliminary translation result of the text to be translated, wherein the preliminary translation result includes preliminary translation suggestions for proper nouns in the text to be translated.

[0167] The generating unit 540 is configured to generate a target translation result of the text to be translated according to the preliminary translation result and the target translation of the proper noun in the text to be translated.

[0168] The specific implementation of each module in the text processing device provided by the embodiment of the present disclosure can refer to the content of the above-mentioned text processing method, which will not be repeated here.

[0169] A method and device for improving the accuracy of machine translation of proper nouns

[0170] The present disclosure relates to the field of language processing technology, and in particular to a method and device for improving the accuracy of proper noun translation in machine translation based on a large language model. Specifically, the present disclosure is applicable to the translation processing of knowledge-intensive texts or long texts, solves the problem of proper noun recognition and its translation accuracy, and improves the performance of existing machine translation systems in complex contexts.

Claims

1. A text processing method, characterized in that: include: Obtaining an input sequence of a text to be translated and domain information of the text to be translated; Processing the input sequence by a context-aware proper noun recognition model to recognize proper nouns in the text to be translated; Retrieving candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated from a dynamic vocabulary, and determining a target translation of the proper nouns in the text to be translated based on the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated; Processing the text to be translated by a translation model to obtain a preliminary translation result of the text to be translated, wherein the preliminary translation result includes preliminary translation suggestions for proper nouns in the text to be translated; A target translation result of the text to be translated is generated according to the preliminary translation result and the target translation of the proper noun in the text to be translated.

2. The method according to claim 1, characterized in that The context-aware proper noun recognition model includes a bidirectional neural recurrent network model and a transformer model based on an attention mechanism; the bidirectional neural recurrent network model includes a forward neural recurrent network model and a reverse neural recurrent network model; The step of processing the input sequence by using a context-aware proper noun recognition model to recognize proper nouns in the text to be translated includes: Using the forward recurrent neural network model to start processing from the starting position of the input sequence, obtain the forward hidden state of the input sequence at each time step; using the reverse recurrent neural network model to start processing from the end position of the input sequence, obtain the reverse hidden state of the input sequence at each time step; Merging the forward hidden state and the reverse hidden state of each time step to obtain the hidden state of the bidirectional neural network model at each time step; Using the attention-based transformer model, the hidden state of each time step is linearly transformed to obtain a query matrix, a key matrix, and a value matrix respectively; Obtain an attention weight matrix according to the query matrix and the key matrix; Processing the value matrix according to the attention weight matrix to obtain a feature vector for each position in the input sequence; The feature vectors at each position are processed to identify proper nouns in the text to be translated.

3. The method according to claim 1, characterized in that The dynamic word library includes the domain identifier and the proper nouns under the corresponding domain identifier and their translations and citation times; The method of retrieving candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated from the dynamic word library and determining the target translation of the proper nouns in the text to be translated according to the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated comprises: If a proper noun matching the proper noun in the text to be translated is found in the dynamic word library, the matching proper noun is used as the candidate proper noun; If no proper noun matching the proper noun in the text to be translated is found in the dynamic vocabulary, a similar proper noun whose semantic similarity with the proper noun in the text to be translated satisfies a condition is retrieved from the dynamic vocabulary as the candidate proper noun; The translation of the candidate proper noun and the field and number of citations corresponding to its field identifier are used as the candidate translation information; The translation of the candidate proper noun whose domain matches the domain information of the proper noun in the text to be translated and is cited the most times is determined as the target translation.

4. The method according to claim 3, characterized in that Generating a target translation result of the text to be translated according to the preliminary translation result and the target translation of the proper noun in the text to be translated includes: If the target translation is retrieved, the target translation is used to replace the preliminary translation suggestion of the proper noun in the text to be translated in the preliminary translation result to generate a target translation result of the text to be translated.

5. The method according to claim 4, characterized in that Also includes: Displaying the target translation result on the translation output interface, and displaying a hidden control at a location associated with the target translation of the proper noun in the target translation result; In response to a triggering operation on the hidden control, the number of citations of the target translation in various fields, the translations of other candidate proper nouns corresponding to the target translation and their fields and citation times, and a correction control are displayed in order according to the number of citations; In response to the triggering operation of the correction control, feedback data of a target translation of the proper noun in the text to be translated is obtained, wherein the feedback data includes a marked field and a feedback translation of the proper noun in the text to be translated; Updating the dynamic vocabulary according to the annotated fields and feedback translations of the proper nouns; The context-aware proper noun recognition model is trained using the proper nouns and their feedback translations.

6. The method according to claim 1, characterized in that Also includes: If the target translation is not found, the preliminary translation result is used as the target translation result, and a pending verification mark is added to the preliminary translation suggestion; Displaying the target translation result on the translation output interface, and displaying a hidden control at a location associated with a preliminary translation suggestion for a proper noun in the target translation result; In response to a triggering operation on the hidden control, displaying to-be-verified mark information of the preliminary translation suggestion and a correction control; In response to the triggering operation of the correction control, feedback data of preliminary translation suggestions for the proper nouns in the text to be translated is obtained, wherein the feedback data includes the marked fields and feedback translations of the proper nouns in the text to be translated; Updating the dynamic vocabulary according to the annotated fields and feedback translations of the proper nouns; The context-aware proper noun recognition model is trained using the proper nouns and their feedback translations.

7. A text processing device, characterized in that: include: A receiving unit, used to obtain an input sequence of a text to be translated and domain information of the text to be translated; A processing unit, configured to process the input sequence through a context-aware proper noun recognition model to recognize proper nouns in the text to be translated; A retrieval unit, configured to retrieve candidate translation information of candidate proper nouns matching the proper nouns in the text to be translated from a dynamic word library, and determine a target translation of the proper nouns in the text to be translated based on the candidate translation information matching the semantics and domain information of the proper nouns in the text to be translated; The processing unit is further used to process the text to be translated through a translation model to obtain a preliminary translation result of the text to be translated, wherein the preliminary translation result includes preliminary translation suggestions for proper nouns in the text to be translated; A generating unit is used to generate a target translation result of the text to be translated according to the preliminary translation result and the target translation of the proper noun in the text to be translated.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: one or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images