Vocabulary consistency evaluation method, electronic device and readable storage medium

Abstract

The present disclosure provides a vocabulary consistency evaluation method, including: S1: selecting a test paragraph from a test sample for annotation, and establishing a vocabulary consistency evaluation test set; S2: inputting the source language paragraph to be tested into a model to generate a real translation; S3: for each consistent sample, finding the corresponding translation sentence sentence by sentence according to the annotated sentence index and performing word form restoration to obtain a translation set to be tested; S4: recording the sentence index of the word in the translation candidate set of the annotated sample that appears in the translation set to be tested; S5: judging whether there is a word in the annotated translation candidate set with the same sentence index in the translation set to be tested as the annotated sentence index, and if so, it is considered that the translation of the consistent sample is correct. The present disclosure also provides a vocabulary consistency evaluation device, an electronic device, and a readable storage medium.

Description

Technical Field

[0001] The present disclosure belongs to the field of language processing technology, and in particular relates to a vocabulary consistency evaluation method in neural machine translation, an electronic device, and a readable storage medium. Background Art

[0002] Lexical consistency refers to the tendency of repeated source words in a text to be translated into the same target word. Lexical translation inconsistency is the most serious type of error in neural machine translation. Figure 1 An example of translation with inconsistent vocabulary is shown in . In this example, the words "CSRC" and "Guidelines" are repeated in two sentences of the same document, and are translated into the same target words "CSRC" and "Guidelines" in the reference translation. However, the standard sentence-level neural machine translation model SentNmt, which cannot refer to contextual information, translates "CSRC" into different words "commission" and "CSRC" in the two sentences, and "Guidelines" also corresponds to two translations "guidelines" and "standards". Obviously, this inconsistency in vocabulary translation seriously affects the cohesion of the translation.

[0003] Studies have shown that the general translation evaluation indicator BLEU is not sensitive enough for the translation of discourse phenomena. In recent years, some research works have constructed comparative test sets to evaluate specific discourse phenomena in translation. In this type of evaluation method, each test sample usually contains a correct reference translation and a set of incorrect reference translations. The incorrect translations differ from the correct translations only in the discourse phenomenon to be evaluated. During the evaluation, the model performs forced decoding on the correct reference translation and the incorrect reference translation to obtain the sequence generation probability of each translation. Finally, the proportion of test samples with a higher generation probability of correct translations than incorrect translations in the total test samples is used as the model evaluation score.

[0004] The comparative test evaluation method can control variables to focus on the phenomenon to be tested. However, this method has certain limitations and cannot evaluate the system performance of the model in real scenarios: first, the target-side context sentences in this method are given in advance by the designer, and in reality the target-side context sentences also need to be generated by the model, so this method cannot evaluate the translation of the context sentences; secondly, the current decoding selection is affected by the generated sequence, and the generated sequence used for forced decoding is not the same as the sequence actually generated by the model. Summary of the invention

[0005] In order to solve at least one of the above technical problems, the present disclosure provides a vocabulary consistency evaluation method, an electronic device, and a readable storage medium.

[0006] As one aspect of the present disclosure, a vocabulary consistency evaluation method is provided, comprising: S1: selecting a test paragraph from a test sample for annotation, and establishing a vocabulary consistency evaluation test set, wherein the vocabulary consistency evaluation test set comprises a trigger word, a trigger word type, a sentence index, and a translation candidate set; S2: inputting the source language paragraph to be tested into a model to generate a real translation; S3: for each consistent sample annotated in the paragraph, finding the corresponding translation sentence sentence by sentence according to the annotated sentence index and performing word form restoration to obtain a translation set to be tested; S4: recording the sentence index of each word in the translation candidate set of the annotated sample that appears in the translation set to be tested; S5: judging whether there is a word in the annotated translation candidate set whose sentence index in the translation set to be tested is the same as the annotated sentence index, and if so, it is considered that the translation of the consistent sample is correct.

[0007] According to the vocabulary consistency evaluation method of at least one embodiment of the present disclosure, each test sample corresponds to a source language paragraph and its reference translation, which contain different numbers of vocabulary consistency samples.

[0008] According to the vocabulary consistency evaluation method of at least one embodiment of the present disclosure, in step S3, translated sentences with duplicate indexes in the obtained translation set are removed from the translation set.

[0009] According to the vocabulary consistency evaluation method of at least one embodiment of the present disclosure, the model is a neural machine translation model.

[0010] According to the vocabulary consistency evaluation method of at least one embodiment of the present disclosure, in the step S1, a vocabulary consistency evaluation test set is established through the following steps: S11, searching for trigger words with vocabulary translation consistency in the source language paragraph; S12, for each group of trigger words, checking the target-side translation results corresponding to each word in the group in turn, if the corresponding target-side words remain consistent after morphological restoration, marking the group of trigger words as a consistent sample; S13, marking the sentence index and trigger word type of the trigger words in the consistent sample; S14, expanding the possible translation candidates of the trigger words in the consistent sample, selecting the top 20 translation results in the Moses vocabulary list for morphological restoration, and then including the words with correct meanings in the translation candidates for the trigger words.

[0011] As another aspect of the present disclosure, a vocabulary consistency evaluation device is provided, comprising: a vocabulary consistency evaluation test set establishment module, the vocabulary consistency evaluation test set establishment module selects test paragraphs from test samples for annotation, and establishes a vocabulary consistency evaluation test set, the vocabulary consistency evaluation test set including trigger words, trigger word types, sentence indexes and translation candidate sets; a translation generation module, the translation generation module inputs the source language paragraph to be tested into a model to generate a real translation; a to-be-checked translation set determination module, the to-be-checked translation set determination module finds the corresponding translation sentences one by one according to the annotated sentence indexes for each consistent sample annotated in the paragraph and performs word form restoration to obtain a to-be-checked translation set; a sentence index determination module, the sentence index determination module records the sentence index of each word in the annotated sample translation candidate set that appears in the to-be-checked translation set; and a consistent sample translation judgment module, the consistent sample translation judgment module determines that the consistent sample translation is correct when there is a word in the annotated translation candidate set with the same sentence index as the annotated sentence index in the to-be-checked translation set.

[0012] According to a vocabulary consistency evaluation device of at least one embodiment of the present disclosure, the vocabulary consistency evaluation test set establishment module includes: a trigger word determination submodule, the trigger word determination submodule searches for trigger words with vocabulary translation consistency in a source language paragraph; a consistency sample marking submodule, the consistency sample marking submodule checks the target end translation results corresponding to each word in the group in turn for each group of trigger words, and if the corresponding target end words remain consistent after word form restoration, the group of trigger words is marked as a consistency sample; a sentence index and trigger word type marking submodule, the sentence index and trigger word type marking submodule marks the sentence index and trigger word type of the trigger word in the consistency sample; a translation candidate set determination submodule, the translation candidate set determination submodule expands the possible translation candidates of the trigger word in the consistency sample, selects the top 20 translation results in the Moses vocabulary list for word form restoration, and then includes the words with correct meaning in the translation candidate set of the trigger word.

[0013] As another aspect of the present disclosure, an electronic device is provided, including:

[0014] a memory storing execution instructions; and

[0015] A processor executes the execution instruction stored in the memory, so that the processor executes the vocabulary consistency evaluation method as described in any one of the above.

[0016] As another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the execution instructions are used to implement the method for evaluating consistency of vocabulary enhancement as described in any one of the above items. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0018] Figure 1 is an example of a translation with inconsistent vocabulary in the prior art.

[0019] Figure 2 It is a flowchart of a vocabulary consistency evaluation method according to an embodiment of the present disclosure.

[0020] Figure 3 The flowchart of establishing a vocabulary consistency evaluation test set according to an embodiment of the present disclosure.

[0021] Figure 4 It is a schematic diagram of establishing a vocabulary consistency evaluation test set according to an embodiment of the present disclosure.

[0022] Figure 5 is a schematic structural diagram of an electronic device 1000 for implementing a vocabulary consistency evaluation method according to an embodiment of the present disclosure.

[0023] Figure 6 It is a structural diagram of a vocabulary consistency evaluation test set establishment module according to an embodiment of the present disclosure.

[0024] Reference numerals

[0025] 1001 Vocabulary consistency evaluation test set establishment module

[0026] 1002 Module for determining the set of translations to be checked

[0027] 1003 Sentence index determination module

[0028] 1004 Consistency Sample Translation Judgment Module

[0029] 2001 Trigger word determination submodule

[0030] 2002 Consistent sample labeling submodule

[0031] 2003 Sentence indexing and trigger word type tagging submodule

[0032] 2004 Translation candidate set determination submodule

[0033] 1000 Electronic device for implementing vocabulary consistency evaluation method

[0034] 1100 Bus

[0035] 1200 processor

[0036] 1300 Memory

[0037] 1400 Other circuits. DETAILED DESCRIPTION

[0038] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0039] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0041] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0042] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0043] For descriptive purposes, the present disclosure may use spatially relative terms such as "under," "beneath," "under," "down," "over," "upper," "above," "higher," and "side (e.g., as in "sidewall")," to describe the relationship of one component to another (other) component as shown in the accompanying drawings. The spatially relative terms are intended to encompass different orientations of the device in use, operation, and / or manufacture in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, components described as "under" or "beneath" other components or features would subsequently be positioned "over" the other components or features. Thus, the exemplary term "under" can encompass both the "above" and "below" orientations. Furthermore, the device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0044] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0045] Figure 2 is a flowchart of a vocabulary consistency evaluation method according to one embodiment of the present disclosure.

[0046] like Figure 2As shown, the vocabulary consistency evaluation method of the present embodiment includes: S1: selecting a test paragraph from a test sample for annotation, and establishing a vocabulary consistency evaluation test set, wherein the vocabulary consistency evaluation test set includes trigger words, trigger word types, sentence indexes, and translation candidate sets; S2: inputting the source language paragraph to be tested into the model to generate a real translation; S3: for each consistent sample annotated in the paragraph, finding the corresponding translation sentence sentence by sentence according to the annotated sentence index and performing word form restoration to obtain a translation set to be tested; S4: recording the sentence index of each word in the translation candidate set of the annotated sample that appears in the translation set to be tested; S5: judging whether there is a word in the annotated translation candidate set whose sentence index in the translation set to be tested is the same as the annotated sentence index, and if so, it is considered that the translation of the consistent sample is correct.

[0047] Each test sample corresponds to a source language paragraph and its reference translation, which contains different numbers of lexical consistency samples. Lexical consistency samples refer to repeated source words in the source language paragraph. Figure 3 The flowchart of establishing a vocabulary consistency evaluation test set according to one embodiment of the present disclosure.

[0048] like Figure 3 As shown, in step S1, a vocabulary consistency evaluation test set is established through the following steps:

[0049] S11. Searching for trigger words with consistent lexical translation in the source language paragraph; the trigger words with consistent lexical translation are repeated source words.

[0050] S12. For each group of trigger words, check the target translation results corresponding to each word in the group in turn. If the corresponding target words remain consistent after morphological restoration, mark the group of trigger words as a consistent sample. Lemmatic restoration refers to restoring words to their original words in the dictionary according to their parts of speech after word segmentation. If a consistent trigger word corresponds to multiple different translations in the target segment, it is not marked as a consistent sample.

[0051] S13. The sentence index and trigger word type of the trigger word in the labeled consistent sample; the trigger word type is marked, including "proper nouns" (abbreviated as "proper nouns") and "ordinary content words". Proper nouns refer to common named entity nouns, such as names of people, places, and organizational names.

[0052] S14, expand the possible translation candidates of the trigger word in the consistent sample, select the top 20 translation results in the Moses vocabulary, perform morphological restoration, and include the words with correct meanings in the translation candidate set of the trigger word. The correctness of the meaning can be determined by a translator or a translation machine.

[0053] Figure 4 This is an example of establishing a vocabulary consistency evaluation test set based on the above implementation. Figure 4 As shown, for the consistent sample trigger word "movie" in the source language, the vocabulary consistency evaluation test set is composed of "trigger word: movie, type: general, index: {S1, S3}, translation candidate set: {movie, film, cine}"; for the consistent sample trigger word "consciousness" in the source language, the vocabulary consistency evaluation test set is composed of "trigger word: consciousness, type: general, index: {S3, S6, S7, S7}, translation candidate set: {consciousness, mentality}". For the trigger word "research", there are two different translations "work" and "study" on the target end, so "research" is not marked as a consistent sample.

[0054] According to the embodiment of the present disclosure, after obtaining the vocabulary consistency evaluation test set, the machine translation vocabulary consistency evaluation test can be performed in steps S2-S5. The source language paragraph to be tested is input into the model to generate a real translation. The model can be a machine language translation model, such as a Transformer translation model. For each consistent sample annotated in the paragraph, the corresponding translation sentence is found sentence by sentence according to the annotated sentence index and word form restoration is performed to obtain a set of translations to be tested. Among them, the translation sentences with repeated indexes in the set are removed from the set of translations to be tested. For each word in the obtained record annotation sample translation candidate set that appears in the set of translations to be tested, it is recorded once it appears, and the index in the same sentence can be repeated. Then the sentence index in the set of translations to be tested is compared with the sentence index in the vocabulary consistency evaluation test. If the sentence index of a specific word in the translation candidate set is the same as the sentence index of the trigger word corresponding to the word in the vocabulary consistency evaluation test, it is considered that the consistency sample translation of the specific word is correct.

[0055] like Figure 4 For consistent sample 2 triggered by the word “consciousness”, the translation of sample 2 is considered correct only when the index of “consciousness” or “mentality” in the translation generated by the model is also {S3, S6, S7, S7} (S7 must appear twice).

[0056] The vocabulary consistency evaluation method of the embodiment of the present disclosure may be implemented by an electronic device that implements the vocabulary consistency evaluation method.

[0057] Figure 5The invention is a schematic diagram of the structure of an electronic device 1000 for implementing a vocabulary consistency evaluation method according to an embodiment of the present invention, comprising: a vocabulary consistency evaluation test set establishment module 1001, which selects test paragraphs from test samples for annotation and establishes a vocabulary consistency evaluation test set, wherein the vocabulary consistency evaluation test set includes trigger words, trigger word types, sentence indexes and translation candidate sets; a translation generation module, which inputs the source language paragraph to be tested into a model and generates a real translation; a to-be-tested translation set determination module 1002, which determines the to-be-tested translation set. The determination module 1002 finds the corresponding translation sentence sentence by sentence according to the marked sentence index for each consistent sample marked in the paragraph and performs word form restoration to obtain a set of translations to be checked; the sentence index determination module 1003 records the sentence index of each word in the marked sample translation candidate set that appears in the set of translations to be checked; the consistent sample translation judgment module 1004 determines that the translation of the consistent sample is correct when there is a word in the marked translation candidate set with the same sentence index as the marked sentence index in the set of translations to be checked.

[0058] Preferably, in the embodiments of the present disclosure, Figure 5 The structure of the vocabulary consistency evaluation test set building module 1001 is as follows Figure 6 shown. Figure 6 The present invention is a schematic diagram of the structure of a vocabulary consistency evaluation test set establishment module 1001 according to an embodiment of the present invention. The vocabulary consistency evaluation test set establishment module 1001 includes: a trigger word determination submodule 2001, which searches for trigger words with vocabulary translation consistency in a source language paragraph; a consistency sample marking submodule 2002, which checks the target end translation results corresponding to each word in the group in turn for each group of trigger words, and if the corresponding target end words remain consistent after word form restoration, marks the group of trigger words as a consistency sample; a sentence index and trigger word type marking submodule 2003, which marks the sentence index and trigger word type of the trigger word in the consistency sample; a translation candidate set determination submodule 2004, which expands the possible translation candidates of the trigger word in the consistency sample, selects the top 20 translation results in the Moses vocabulary table for word form restoration, and then includes the words with correct meaning in the translation candidate set of the trigger word.

[0059] Figure 5The electronic device in the embodiment may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by the corresponding module, and the device may include one or more modules of these modules. The module may be one or more hardware modules specially configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0060] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0061] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0062] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0063] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0064] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.

[0065] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0066] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0067] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0068] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0069] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0070] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A vocabulary consistency evaluation method, comprising: S1: Select test paragraphs from the test samples for annotation, and establish a vocabulary consistency evaluation test set, wherein the vocabulary consistency evaluation test set includes trigger words, trigger word types, sentence indexes, and translation candidate sets; S2: Input the source language paragraph to the model to generate a real translation; S3: For each consistent sample annotated in the paragraph, find the corresponding translation sentence sentence by sentence according to the annotated sentence index and perform word form restoration to obtain the translation set to be checked; S4: Record the sentence index of each word in the translation candidate set of the annotated sample that appears in the translation set to be checked; S5: Determine whether there is a word in the annotated translation candidate set whose sentence index in the translation set to be checked is the same as the annotated sentence index. If so, it is considered that the consistent sample is translated correctly; In the step S1, a vocabulary consistency evaluation test set is established through the following steps: S11, searching for trigger words with vocabulary translation consistency in the source language paragraph; S12, for each group of trigger words, checking the target-side translation results corresponding to each word in the group in turn, and if the corresponding target-side words remain consistent after morphological restoration, marking the group of trigger words as a consistent sample; S13, marking the sentence index and trigger word type of the trigger words in the consistent sample; S14, expanding the possible translation candidates of the trigger words in the consistent sample, selecting the top 20 translation results in the Moses vocabulary list for morphological restoration, and including the words with correct meanings in the translation candidates of the trigger words.

2. The vocabulary consistency evaluation method according to claim 1, characterized in that: Each test sample corresponds to a source language paragraph and its reference translation, which contains varying numbers of lexical consistency samples.

3. The vocabulary consistency evaluation method according to claim 1, characterized in that: In step S3, the translation sentences with duplicate indexes in the obtained translation set are removed.

4. The vocabulary consistency evaluation method according to claim 1, characterized in that: The model is a neural machine translation model.

5. The vocabulary consistency evaluation method according to claim 4, characterized in that: The neural machine translation model is a Transformer model.

6. A vocabulary consistency evaluation device, comprising: A vocabulary consistency evaluation test set establishment module, wherein the vocabulary consistency evaluation test set establishment module selects test paragraphs from the test samples for annotation and establishes a vocabulary consistency evaluation test set, wherein the vocabulary consistency evaluation test set includes trigger words, trigger word types, sentence indexes, and translation candidate sets; A translation generation module, wherein the translation generation module inputs the source language paragraph to be tested into the model to generate a true translation; A module for determining a set of translations to be checked, wherein the module finds the corresponding translation sentences one by one according to the marked sentence indexes for each consistent sample in the paragraph and performs morphological restoration to obtain a set of translations to be checked; A sentence index determination module, which records the sentence index of each word in the labeled sample translation candidate set that appears in the to-be-checked translation set; a consistent sample translation judgment module, which determines that the consistent sample translation is correct when there is a word in the labeled translation candidate set with the same sentence index as the labeled sentence index in the to-be-checked translation set; The vocabulary consistency evaluation test set establishment module includes: a trigger word determination submodule, which searches for trigger words with vocabulary translation consistency in a source language paragraph; a consistency sample marking submodule, which checks the target end translation results corresponding to each word in each group of trigger words in turn, and if the corresponding target end words remain consistent after word form restoration, marks the group of trigger words as a consistency sample; a sentence index and trigger word type marking submodule, which marks the sentence index and trigger word type of the trigger word in the consistency sample; and a translation candidate set determination submodule, which expands possible translation candidates for the trigger word in the consistency sample, selects the top 20 translation results in the Moses vocabulary list for word form restoration, and then includes the words with correct meanings in the translation candidate set of the trigger word.

7. An electronic device, characterized in that: include: A memory storing execution instructions; and a processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the vocabulary consistency evaluation method according to any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, and when the execution instructions are executed by the processor, they are used to implement the vocabulary consistency evaluation method according to any one of claims 1 to 5.

Images