Method for constructing Chinese-English bilingual corpus and related equipment

By building a bilingual entity word network and comparable sentence network, the problem of low training accuracy of Chinese and English bilingual corpus is solved, and the recognition accuracy of the dialogue system is improved.

CN111695361BActive Publication Date: 2024-11-26PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010356769.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-04-29
Publication Date
2024-11-26
Estimated Expiration
2040-04-29

AI Technical Summary

Technical Problem

The current training methods for bilingual Chinese and English corpus are not very accurate, which leads to semantic recognition errors in the dialogue system when using bilingual Chinese and English corpus for recognition, affecting the accuracy of the dialogue system.

Method used

By obtaining Chinese and English entities and their mapping relationships from the preset entity library, a bilingual entity word network is constructed, and a single-language and cross-language representation valuation is calculated. The comparable statement network is used to calculate the cross-language statement valuation and perform weighted summing. Generate target valuations, and if preset conditions are met, they will be combined into a bilingual Chinese-English corpus.

Benefits of technology

It improves the correlation and accuracy between different corpuses in the Chinese and English bilingual corpus, thereby improving the recognition accuracy of the dialogue system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111695361B_ABST
    Figure CN111695361B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of computer technology, and provides a method for constructing a Chinese-English bilingual corpus and related equipment thereof, wherein the method for constructing the Chinese-English bilingual corpus comprises: obtaining Chinese entities, English entities, and mapping relationships and mutual translation relationships between Chinese entities and English entities, and constructing a bilingual entity word network according to preset requirements; calculating the single language representation valuation and cross-language entity valuation of the bilingual entity word network according to Chinese entities, English entities, context words, preset hyperlink sets and preset sentence sets; using training sentences to calculate the cross-language sentence valuation corresponding to the obtained comparable sentence network; calculating the target valuation according to the three valuations; and according to the target valuation, using the bilingual entity word network and the comparable sentence network, combining into a Chinese-English bilingual corpus, and storing it on a blockchain. The present invention improves the accuracy of the corpus in the Chinese-English bilingual corpus by utilizing the correlation between the two networks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for constructing a Chinese-English bilingual corpus and related equipment. Background Art

[0002] At present, the traditional training methods of Chinese-English bilingual corpora based on bilingual dialogue systems are mainly divided into two categories: one method is to use the corpus of the corresponding field to match the Chinese-English bilingual corpus to be tested, without the need for parallel corpora, but the training process is unstable and complex, and the method can only be limited to small-scale data, and the accuracy is not high; the other method is to use existing multilingual resources to automatically generate "pseudo-bilingual documents". Although this method is stable, due to the large amount of training data and uncertainty, the training is time-consuming and inaccurate. As a result, when the dialogue system uses the Chinese-English bilingual corpus for recognition, there are semantic recognition errors, which further affects the accuracy of the dialogue system. Summary of the invention

[0003] The embodiment of the present invention provides a method for constructing a Chinese-English bilingual corpus and related devices thereof to solve the problem that the accuracy of traditional Chinese-English bilingual corpus training is low, which further affects the accuracy of a dialogue system using the Chinese-English bilingual corpus.

[0004] A method for constructing a Chinese-English bilingual corpus, comprising:

[0005] Obtain Chinese entities, English entities, and mapping relationships and mutual translation relationships between Chinese entities and English entities from a preset entity library;

[0006] According to the Chinese entity, the English entity, the mapping relationship and the mutual translation relationship, a bilingual entity word network is constructed according to preset requirements;

[0007] Acquire context words corresponding to each of the Chinese entities and each of the English entities from a preset database;

[0008] Based on the Chinese entity, the English entity, the context words, the preset hyperlink set and the preset sentence set, calculating the monolingual representation valuation and the cross-lingual entity valuation of the bilingual entity word network;

[0009] Obtaining a comparable sentence network and training sentences, and using the training sentences to calculate a cross-language sentence estimation value corresponding to the comparable sentence network;

[0010] Performing a weighted summation on the single language representation estimation, the cross-language entity estimation, and the cross-language sentence estimation to obtain a target estimation;

[0011] The target estimate is compared with a preset threshold, and if the preset condition is met, the bilingual entity word network and the comparable sentence network are used to combine into a Chinese-English bilingual corpus.

[0012] A device for constructing a Chinese-English bilingual corpus, comprising:

[0013] A first acquisition module is used to acquire Chinese entities, English entities, and mapping relationships and mutual translation relationships between Chinese entities and English entities from a preset entity library;

[0014] A construction module, used to construct a bilingual entity word network according to the Chinese entity, the English entity, the mapping relationship and the mutual translation relationship according to preset requirements;

[0015] A second acquisition module is used to acquire context words corresponding to each of the Chinese entities and each of the English entities from a preset database;

[0016] A first calculation module, configured to calculate a single language representation valuation and a cross-language entity valuation of the bilingual entity word network based on the Chinese entity, the English entity, the context words, a preset hyperlink set, and a preset sentence set;

[0017] A second calculation module, used to obtain a comparable sentence network and a training sentence, and use the training sentence to calculate a cross-language sentence estimation value corresponding to the comparable sentence network;

[0018] A summing module, configured to perform a weighted summation of the single language representation estimation, the cross-language entity estimation and the cross-language sentence estimation to obtain a target estimation;

[0019] The combination module is used to compare the target estimate with a preset threshold. If the preset condition is met, the bilingual entity word network and the comparable sentence network are used to combine them into a Chinese-English bilingual corpus.

[0020] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned Chinese and English entity information method when executing the computer program.

[0021] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the Chinese-English entity information method are implemented.

[0022] The above-mentioned Chinese-English bilingual corpus construction method and related equipment construct a bilingual entity word network based on mapping relationships and mutual translation relationships, which can strengthen the association between Chinese entities and English entities. By calculating the single language representation valuation, cross-language entity valuation and cross-language sentence valuation, it can accurately determine whether the bilingual entity word network and the comparable sentence network meet the set requirements. Finally, when the target valuation meets the preset conditions, the bilingual entity word network and the comparable sentence network are used to construct the Chinese-English bilingual corpus. Since the Chinese-English bilingual corpus is composed of two networks, it can improve the correlation between different corpora in the Chinese-English bilingual corpus, further improve the accuracy of the corpus in the Chinese-English bilingual corpus, and further improve the accuracy of the dialogue system using the Chinese-English bilingual corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0024] Figure 1 is a flow chart of a method for constructing a Chinese-English bilingual corpus provided by an embodiment of the present invention;

[0025] Figure 2 is a flowchart of step S2 in the method for constructing a Chinese-English bilingual corpus provided in an embodiment of the present invention;

[0026] Figure 3 is a flowchart of step S4 in the method for constructing a Chinese-English bilingual corpus provided in an embodiment of the present invention;

[0027] Figure 4 is a flowchart of step S5 in the method for constructing a Chinese-English bilingual corpus provided in an embodiment of the present invention;

[0028] Figure 5 is a flowchart of step S53 in the method for constructing a Chinese-English bilingual corpus provided in an embodiment of the present invention;

[0029] Figure 6 is a flowchart of step S7 in the method for constructing a Chinese-English bilingual corpus provided in an embodiment of the present invention;

[0030] Figure 7 is a schematic diagram of a Chinese-English bilingual corpus construction device provided by an embodiment of the present invention;

[0031] Figure 8 It is a basic structural block diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0033] The method for constructing a Chinese-English bilingual corpus provided by this application is applied to the server side, and the server side can specifically be implemented by an independent server or a server cluster composed of multiple servers. In one embodiment, as Figure 1 shown, a method for constructing a Chinese-English bilingual corpus is provided, including the following steps:

[0034] S1: Obtain Chinese entities, English entities, and the mapping relationship and mutual translation relationship between Chinese entities and English entities from a preset entity library.

[0035] In the embodiments of the present invention, the mapping relationship between Chinese entities and English entities refers to setting a connection relationship between Chinese entities and English entities according to the actual needs of users. For example, there is a mapping relationship between the Chinese entity "apple" and the English entity "fruit".

[0036] It should be noted that the mutual translation relationship means that Chinese entities and English entities are in a mutual translation relationship. For example, if the Chinese entity is "apple" and the English entity is "apple", since the English corresponding to "apple" is "apple", then the Chinese entity and the English entity "apple" belong to the mutual translation relationship.

[0037] Specifically, obtain Chinese entities, English entities, and the mapping relationship and mutual translation relationship between Chinese entities and English entities from a preset entity library. Among them, the preset entity library refers to a database specifically used to store Chinese entities, English entities, and the mapping relationship and mutual translation relationship between Chinese entities and English entities.

[0038] S2: Construct a bilingual entity word network according to Chinese entities, English entities, mapping relationships, and mutual translation relationships according to preset requirements.

[0039] In the embodiments of the present invention, construct a bilingual entity word network according to the Chinese entities, English entities, mapping relationship, and mutual translation relationship between Chinese entities and English entities obtained in step S1 according to preset requirements.

[0040] Among them, the preset requirements refer to the requirements for constructing a bilingual entity word network set according to the actual needs of users.

[0041] S3: Obtain context words corresponding to each Chinese entity and each English entity from a preset database.

[0042] In the embodiment of the present invention, context words refer to words that are associated with Chinese entities or English entities in a dialogue scenario. By matching the Chinese entity and the English entity with each legal word in a preset database, when the Chinese entity and the legal word are matched, the context word corresponding to the legal word is used as the context word corresponding to the Chinese entity. Similarly, the context word corresponding to the English entity is obtained.

[0043] The preset database refers to a database specifically used to store legal words and context words corresponding to the legal words.

[0044] For example, the preset database contains legal words "apple" and "pear", among which the context words corresponding to "apple" are "apple" and "fruit", and the context words corresponding to "pear" are "pear" and "fruit"; if the Chinese entity is "apple", the Chinese entity is matched with the legal words "apple" and "pear" respectively, and the legal word "apple" is the same as the Chinese entity, then the context words "apple" and "fruit" are used as the context words of the Chinese entity.

[0045] S4: Based on Chinese entities, English entities, context words, preset hyperlink sets and preset sentence sets, calculate the monolingual representation valuation and cross-lingual entity valuation of the bilingual entity word network.

[0046] In the embodiment of the present invention, based on Chinese entities, English entities, context words, preset hyperlink sets and preset sentence sets, the monolingual representation valuation and cross-lingual entity valuation corresponding to the bilingual entity word network are calculated according to a preset calculation formula. The preset calculation formula refers to the formula set by the user for calculating the monolingual representation valuation and the cross-lingual entity valuation.

[0047] The preset hyperlink set refers to a set of hyperlinks selected by the user.

[0048] The preset sentence set refers to a set of sentences pre-selected by users from Baidu Encyclopedia.

[0049] S5: Obtain a comparable sentence network and training sentences, and use the training sentences to calculate cross-language sentence estimates corresponding to the comparable sentence network.

[0050] Specifically, a comparable sentence network and training sentences are obtained from a preset initial library, and a cross-language sentence valuation corresponding to the comparable sentence network is calculated using a preset calculation method and training sentences.

[0051] The preset initial library refers to a database specifically used to store comparable sentence networks and training sentences.

[0052] The preset calculation method refers to a pre-set calculation method for calculating the cross-language sentence valuation corresponding to the comparable sentence network based on the training sentences.

[0053] It should be noted that the comparable sentence network refers to the network composed of Chinese sentences, English sentences, and the relationship between Chinese sentences and English sentences.

[0054] S6: Perform weighted summation of the single language representation valuation, cross-language entity valuation, and cross-language sentence valuation to obtain the target valuation.

[0055] Specifically, the single language representation valuation, the cross-language entity valuation, and the cross-language sentence valuation are multiplied by their corresponding preset weights, and the multiplication results are summed up to obtain the calculated result as the target valuation.

[0056] The preset weight refers to a ratio value preset by the user, and its specific value may be 0.3, or may be set according to the actual needs of the user, and is not limited here.

[0057] S7: Compare the target valuation with the preset threshold. If the preset condition is met, the bilingual entity word network and the comparable sentence network are used to combine them into a Chinese-English bilingual corpus.

[0058] In the embodiment of the present invention, the preset condition is a condition set by the user according to actual needs, and can be specifically set according to a comparison result obtained by comparing the target valuation with a preset threshold.

[0059] Specifically, the target valuation is compared with a preset threshold. If the comparison result is that the target valuation is less than or equal to the preset threshold, the current bilingual entity word network and the comparable sentence network are combined into a Chinese-English bilingual corpus; if the comparison result is that the target valuation is greater than the preset threshold, the current bilingual entity word network and the comparable sentence network are iteratively updated until the target valuation is less than or equal to the threshold.

[0060] The preset threshold refers to a value set according to the actual needs of the user and is not limited here.

[0061] In this embodiment, a bilingual entity word network is constructed based on mapping relationships and mutual translation relationships, which can strengthen the association relationship between Chinese entities and English entities. By calculating the single language representation valuation, cross-language entity valuation and cross-language sentence valuation, it is possible to accurately determine whether the bilingual entity word network and the comparable sentence network meet the set requirements. Finally, when the target valuation meets the preset conditions, the bilingual entity word network and the comparable sentence network are used to form a Chinese-English bilingual corpus. Since the Chinese-English bilingual corpus is composed of two networks, the correlation between different corpora in the Chinese-English bilingual corpus can be improved, and the accuracy of the corpus in the Chinese-English bilingual corpus can be further improved, and the accuracy of the dialogue system using the Chinese-English bilingual corpus can be further improved.

[0062] In one embodiment, if Figure 2 As shown, in step S2, according to Chinese entities, English entities, mapping relationships and mutual translation relationships, a bilingual entity word network is constructed according to preset requirements, including the following steps:

[0063] S21: Obtain all Chinese entities as the first set and all English entities as the second set.

[0064] Specifically, all Chinese entities acquired in step S1 are taken as the first set, and all English entities acquired are taken as the second set.

[0065] S22: Acquire from a mapping database a first mapping entity that has a mapping relationship with a Chinese entity as a third set, and a second mapping entity that has a mapping relationship with an English entity as a fourth set, wherein the mapping database includes the first mapping entity and the second mapping entity.

[0066] In an embodiment of the present invention, the mapping database includes different first entities and second entities, each first entity has its corresponding first mapping entity, each second entity has its corresponding second mapping entity, and there is a mapping relationship between the first mapping entity and the first entity, and there is a mapping relationship between the second mapping entity and the second entity.

[0067] Match the Chinese entity with the first entity in the mapping database. If the Chinese entity is the same as the first entity, obtain the first mapping entity corresponding to the first entity, and take all the first mapping entities as the third set; similarly, match the English entity with the second entity. If the English entity is the same as the second entity, obtain the second mapping entity corresponding to the second entity, and take all the second mapping entities as the fourth set.

[0068] S23: Acquire Chinese entities and English entities that have a mutual translation relationship as mutual translation entities, and combine all the mutual translation entities into a fifth set.

[0069] In the embodiment of the present invention, Chinese entities and English entities that have a mutual translation relationship are obtained from a preset entity library as mutual translation entities, and all the mutual translation entities are combined into a fifth set.

[0070] S24: Based on the first set, the second set, the third set, the fourth set and the fifth set, a bilingual entity word network is constructed according to formula (1): Formula (1)

[0071] Among them, E is the bilingual entity word network, For the first set, For the second set, For The third set, is the fourth set, For the fifth set .

[0072] Specifically, according to the first set, the second set, the third set, the fourth set and the fifth set, these five sets are combined into a new set according to formula (1), and the new set is used as the bilingual entity word network.

[0073] In this embodiment, by respectively obtaining the first set, the second set, the third set, the fourth set and the fifth set, a bilingual entity word network can be quickly and accurately constructed according to formula (1), thereby ensuring the accuracy of the bilingual entity word network construction and further improving the accuracy of the subsequent construction of the Chinese-English bilingual corpus using the bilingual entity word network.

[0074] In one embodiment, if Figure 3 As shown, the context words include Chinese context words and English context words. In step S4, based on the Chinese entities, English entities, context words, preset hyperlink sets and preset sentence sets, calculating the single language representation valuation and cross-language entity valuation of the bilingual entity word network includes the following steps:

[0075] S41: Import the Chinese entity and the English entity into the preset processing port for vector feature conversion to obtain training entities.

[0076] In an embodiment of the present invention, the preset processing port refers to a processing port specifically used to convert Chinese entities or English entities into word vector features, which specifically uses the word2vec model to perform vector feature conversion processing.

[0077] Specifically, by directly importing Chinese entities and English entities into a preset processing port for vector feature conversion, word vector features after conversion are obtained, and the word vector features are used as training entities.

[0078] It should be noted that when converting Chinese entities and English entities into word vector features, some word vector features contain hyperlinks.

[0079] S42: Based on the training entities, context words, preset hyperlink set and preset sentence set, calculate the single language representation valuation according to formula (2): Formula (2)

[0080] Among them, L is the single language representation estimation, zh is the Chinese entity, en is the English entity, is the training entity, D is the preset sentence set, A is the preset hyperlink set, G is the context word, It means: (i) if , whether it is a contextual term; (ii) if , link to (iii) If q exists in A, then Whether it is the context word of q, where q is an element in D.

[0081] S43: Based on the context words, calculate the cross-language entity valuation according to formula (3): Formula (3)

[0082] Among them, I is the cross-language entity valuation, For the current entity, Indicates context words. If the current entity is a Chinese entity, it indicates the English context words corresponding to the Chinese entity; if the current entity is an English entity, it indicates the Chinese context words corresponding to the English entity; that is, it indicates Connected entities in other languages.

[0083] In this embodiment, according to formula (2) and formula (3), the monolingual representation valuation and cross-lingual entity valuation corresponding to the bilingual entity word network can be calculated quickly and accurately, thereby ensuring the accuracy of the target valuation determined based on the monolingual representation valuation and cross-lingual entity valuation.

[0084] In one embodiment, if Figure 4 As shown, in step S5, obtaining the comparable sentence network and the training sentence, and using the training sentence to calculate the cross-language sentence estimation corresponding to the comparable sentence network includes the following steps:

[0085] S51: Obtain a comparable sentence network from a preset initial library, wherein the comparable sentence network includes Chinese sentences and English sentences.

[0086] In the embodiment of the present invention, the comparable sentence network is directly obtained from a preset initial library.

[0087] It should be noted that the comparable statement network is composed of Chinese statements and English statements, and there is a pre-set associated relationship between them. However, due to the inaccurate associated relationship with the increase in the amount of data, it is necessary to optimize the comparable statement network.

[0088] S52: Obtain Chinese statements and English statements containing 2 identical entities as training statements, where the training statements include Chinese sentence vectors corresponding to the Chinese statements.

[0089] In the embodiments of the present invention, the identical entities between the Chinese statements and the English statements refer to Chinese entities and English entities with a mutual translation relationship. For example, the Chinese entity "apple" and the English entity "apple" belong to identical entities.

[0090] Specifically, the training statements are obtained directly from a preset training library, where the preset training library refers to a database specifically used to store training statements.

[0091] S53: Convert the training statements into comprehensive vectors according to a preset vector conversion method.

[0092] Specifically, the training statements are converted into comprehensive vectors according to a preset vector conversion method.

[0093] Among them, the preset vector conversion method can specifically be converting the training statements into comprehensive vectors through a word2vec model.

[0094] S54: Calculate the cross-language statement valuation according to the comprehensive vectors and the Chinese sentence vectors according to formula (4): Formula (4)

[0095] Among them, J is the cross-language statement valuation, is the comprehensive vector, is the Chinese sentence vector, and K is the comparable statement network.

[0096] Specifically, the comprehensive vectors and the Chinese sentence vectors are substituted into formula (4), and the cross-language statement valuation is calculated using formula (4).

[0097] In this example, by obtaining the comparable statement network and the training statements, the cross-language statement valuation corresponding to the comparable statement network can be quickly and accurately calculated using formula (4), ensuring the accuracy of determining the target valuation using the cross-language statement valuation subsequently.

[0098] In one embodiment, as Figure 5 shown, in S53, that is, converting the training statements into comprehensive vectors according to a preset vector conversion method includes the following steps:

[0099] S531: Perform semantic accuracy judgment on the training sentence, determine a first weight value of the training sentence according to the judgment result, and use the training sentence with the determined first weight value as the first target sentence.

[0100] In an embodiment of the present invention, a training sentence is imported into a preset semantic port for semantic accuracy judgment, the accuracy is output, and a weight value corresponding to the accuracy is obtained from a preset weight table as a first weight value, and finally the training sentence with the first weight value is used as the first target sentence.

[0101] The preset semantic port refers to a processing port that is pre-trained to make semantic accuracy judgments on training sentences and output the accuracy based on the judgment results.

[0102] The preset weight table refers to a data table used to store different accuracies and weight values ​​corresponding to the accuracies.

[0103] For example, if the two sentences "Vancouver is the capital of Canada" and "Vancouver is an important city in Canada" are imported into the preset semantic port, the preset semantic port will judge the semantic accuracy and find that the relationship expressed by the former is wrong, while the information expressed by the latter is correct. Then the accuracy of the output of the former is 0%, and the accuracy of the latter is 100%.

[0104] S532: Perform sentence vector conversion on the first target sentence to obtain a first vector.

[0105] Specifically, the first target sentence is imported into a preset vector conversion port for sentence vector conversion processing to obtain a first vector after conversion processing. The preset vector conversion port refers to a processing port specifically used for sentence vector conversion processing.

[0106] S533: Determine the second weight value of each vocabulary word in the Chinese sentence and the English sentence in the training sentence by matching the training sentence meaning, and use the training sentence with the determined second weight value as the second target sentence.

[0107] In an embodiment of the present invention, sentence meaning matching refers to a method of judging whether the translation corresponding to the English sentence in the training sentence matches the Chinese sentence based on the Chinese sentence in the training sentence. Specifically, the training sentence is imported into a preset matching port for sentence meaning matching processing, and after the sentence meaning matching processing, each group of words in the English sentence is assigned a second weight value, and the training sentence with the second weight value is used as the second target sentence.

[0108] The preset matching port refers to a processing port that is pre-trained for matching the meaning of a training sentence and assigns a second weight value corresponding to each group of words in the English sentence included in the training sentence according to the matching result.

[0109] The specific processing process of the preset matching port is: convert the Chinese sentences and English sentences in the training sentences into Chinese sentence vectors and English sentence vectors through the word2vec model, and compare the vector of each dimension in the Chinese sentence vector with the vector of each dimension in the English sentence vector. If the vector of the Chinese sentence vector is different from the English sentence vector in the same dimension, then assign the second weight value corresponding to the vector in the English sentence vector according to the second weight value corresponding to the pre-trained vector ratio.

[0110] It should be noted that each vector has its corresponding vocabulary, and the vectors of Chinese vocabulary and English vocabulary that have a mutual translation relationship are the same.

[0111] For example, the training sentences include the Chinese sentence: "Xiaoming was once a student at Peking University", and the English sentence: "Xiaoming spent 7 years in Peking University." The Chinese sentence is converted into a Chinese sentence vector through the word2vec model: (1,2,3,4,5), and the English sentence is converted into an English sentence vector through the word2vec model: (1,0,3,4,9), where the second dimension vector 0 in the English sentence vector corresponds to the word "spent", and the fifth dimension vector 9 corresponds to the word "7 years".

[0112] By comparing the vectors of each dimension in the Chinese sentence vector and the English sentence vector, we can find that the second dimension 2 is different from 0, and the fifth dimension 5 is different from 9. If the second weight value corresponding to the pre-trained vector ratio 2:0 is 50%, the second weight value corresponding to the vector ratio 5:9 is 1%, and the second weight value of the same ratio is 100%; then the second weight value corresponding to the vector of the second dimension in the English sentence vector is 50%, that is, the second weight value corresponding to spent is 50%; the second weight value corresponding to the vector of the fifth dimension is 1%, that is, the second weight value of the 7 years vocabulary is 1%, and the second weight values ​​corresponding to other vectors with the same ratio are 100%.

[0113] S534: Perform sentence vector conversion on the English sentence in the second target sentence to obtain a second vector.

[0114] Specifically, the second target sentence is imported into a preset vector conversion port for sentence vector conversion processing to obtain a second vector after conversion processing.

[0115] S535: Based on the first vector and the second vector, calculate the comprehensive vector corresponding to the training sentence.

[0116] In the embodiment of the present invention, the comprehensive vector corresponding to the training sentence is calculated according to the first vector and the second vector according to formula (5): Formula (5)

[0117] in, is the comprehensive vector, is the first vector, is the second vector, For training sentences, is the preset sentence vector.

[0118] In this embodiment, by judging the semantic accuracy of the training sentences, it is possible to identify whether the training sentences have semantic errors, and determine the first weight value of the training sentences based on the judgment to determine the first target sentence, thereby improving the semantic accuracy of the first target sentence. Then, based on the sentence meaning matching method, it is possible to identify the similarity between the Chinese sentences and the English sentences in the training sentences, and then determine the second weight value of each word in the English sentence based on the similarity to determine the second target sentence, thereby improving the accuracy of the second target sentence. Finally, the comprehensive vector corresponding to the training sentences is calculated based on the first target sentence and the second target sentence, thereby ensuring the accuracy of the comprehensive vector.

[0119] In one embodiment, if Figure 6 As shown, in step S7, the target valuation is compared with a preset threshold. If the preset condition is met, the bilingual entity word network and the comparable sentence network are used to combine into a Chinese-English bilingual corpus, which includes the following steps:

[0120] S71: Compare the target estimate with a preset threshold.

[0121] Specifically, the target estimate is compared with a preset threshold.

[0122] S72: If the target estimate is less than or equal to a preset threshold, the bilingual entity word network and the comparable sentence network are combined into a Chinese-English bilingual corpus.

[0123] Specifically, according to the comparison method of step S71, if the target valuation is less than or equal to the preset threshold, the bilingual entity word network and the comparable sentence network corresponding to the target valuation are combined into a Chinese-English bilingual corpus.

[0124] S73: If the target valuation is greater than a preset threshold, the bilingual entity word network and the comparable sentence network are iteratively updated according to a preset parameter update method until the target valuation is less than or equal to the preset threshold, and the iteratively updated bilingual entity word network and the comparable sentence network are combined into a Chinese-English bilingual corpus.

[0125] Specifically, according to the comparison method of step S71, if the target valuation is greater than a preset threshold, the bilingual entity word network and the comparable sentence network are iteratively updated according to a preset parameter update method until the target valuation is less than or equal to the preset threshold, and the iteratively updated bilingual entity word network and the comparable sentence network are combined into a Chinese-English bilingual corpus.

[0126] The preset parameter updating method refers to a method for updating parameters in the bilingual entity word network and the comparable sentence network according to actual needs of users.

[0127] In this embodiment, by comparing the target valuation with a preset threshold, the Chinese-English bilingual corpus is determined when the target valuation is less than or equal to the preset threshold, and the bilingual entity word network and the comparable sentence network are iteratively updated again when the target valuation is greater than the preset threshold, and the Chinese-English bilingual corpus is determined when the target valuation is less than or equal to the preset threshold. The method of comparing the target valuation with the preset threshold to determine the Chinese-English bilingual corpus can ensure that the Chinese-English bilingual corpus is determined when the set conditions are met, thereby improving the accuracy of the Chinese-English bilingual corpus.

[0128] In one embodiment, after step S7, the method for constructing the Chinese-English bilingual corpus further includes: storing the Chinese-English bilingual corpus in a blockchain.

[0129] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned Chinese-English bilingual corpus, the above-mentioned Chinese-English bilingual corpus can also be stored in a node of a blockchain.

[0130] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0131] In one embodiment, a device for constructing a Chinese-English bilingual corpus is provided, and the device for constructing a Chinese-English bilingual corpus corresponds one-to-one to the method for constructing a Chinese-English bilingual corpus in the above embodiment. Figure 7 As shown, the Chinese-English bilingual corpus construction device includes a first acquisition module 71, a construction module 72, a second acquisition module 73, a first calculation module 74, a second calculation module 75, a summation module 76 and a combination module 77. The detailed description of each functional module is as follows:

[0132] A first acquisition module 71 is used to acquire Chinese entities, English entities, and mapping relationships and mutual translation relationships between Chinese entities and English entities from a preset entity library;

[0133] A construction module 72 is used to construct a bilingual entity word network according to preset requirements based on Chinese entities, English entities, mapping relationships and mutual translation relationships;

[0134] A second acquisition module 73, used to acquire context words corresponding to each Chinese entity and each English entity from a preset database;

[0135] A first calculation module 74, for calculating a single language representation valuation and a cross-language entity valuation of a bilingual entity word network based on Chinese entities, English entities, context words, a preset hyperlink set, and a preset sentence set;

[0136] A second calculation module 75 is used to obtain a comparable sentence network and a training sentence, and use the training sentence to calculate a cross-language sentence estimate corresponding to the comparable sentence network;

[0137] A summing module 76, configured to perform a weighted summation of the single language representation estimation, the cross-language entity estimation, and the cross-language sentence estimation to obtain a target estimation;

[0138] The combination module 77 is used to compare the target estimate with a preset threshold. If the preset condition is met, the bilingual entity word network and the comparable sentence network are used to combine them into a Chinese-English bilingual corpus.

[0139] Furthermore, the building block 72 includes:

[0140] A third acquisition submodule is used to acquire all Chinese entities as a first set and all English entities as a second set;

[0141] A fourth acquisition submodule is used to acquire, from a mapping database, first mapping entities that have a mapping relationship with Chinese entities as a third set, and second mapping entities that have a mapping relationship with English entities as a fourth set, wherein the mapping database includes the first mapping entity and the second mapping entity;

[0142] A fifth acquisition submodule is used to acquire Chinese entities and English entities that have a mutual translation relationship as mutual translation entities, and combine all the mutual translation entities into a fifth set;

[0143] The network construction submodule is used to construct a bilingual entity word network based on the first set, the second set, the third set, the fourth set and the fifth set according to formula (1): Formula (1)

[0144] Among them, E is the bilingual entity word network, For the first set, For the second set, For The third set, is the fourth set, For the fifth set .

[0145] Furthermore, the first calculation module 74 includes:

[0146] A conversion submodule, used to import Chinese entities and English entities into a preset processing port for vector feature conversion to obtain training entities;

[0147] The third calculation submodule is used to calculate the single language representation estimation according to formula (2) based on the training entity, context words, preset hyperlink set and preset sentence set: Formula (2)

[0148] Among them, L is the single language representation estimation, zh is the Chinese entity, en is the English entity, is the training entity, D is the preset sentence set, A is the preset hyperlink set, G is the context word, It means: (i) if , whether it is a contextual term; (ii) if , link to (iii) If q exists in A, then Is it a context word of q, where q is an element in D;

[0149] The fourth calculation submodule is used to calculate the cross-language entity valuation based on the context words according to formula (3): Formula (3)

[0150] Among them, I is the cross-language entity valuation, For the current entity, Indicates context words. If the current entity is a Chinese entity, it indicates the English context words corresponding to the Chinese entity; if the current entity is an English entity, it indicates the Chinese context words corresponding to the English entity; that is, it indicates Connected entities in other languages.

[0151] Furthermore, the second calculation module 75 includes:

[0152] A sixth acquisition submodule is used to acquire a comparable sentence network from a preset initial library, wherein the comparable sentence network includes Chinese sentences and English sentences;

[0153] A seventh acquisition submodule is used to acquire a Chinese sentence and an English sentence containing two identical entities as training sentences, wherein the training sentence includes a Chinese sentence vector corresponding to the Chinese sentence;

[0154] A comprehensive vector conversion submodule, used to convert the training sentence into a comprehensive vector according to a preset vector conversion method;

[0155] The fifth calculation submodule is used to calculate the cross-language sentence estimation according to the comprehensive vector and the Chinese sentence vector according to formula (4): Formula (4)

[0156] Among them, J is the cross-language sentence estimation, is the comprehensive vector, is the Chinese sentence vector, and K is the comparable sentence network.

[0157] Furthermore, the comprehensive vector conversion submodule includes:

[0158] A judgment unit, configured to judge the semantic accuracy of the training sentence, determine a first weight value of the training sentence according to the judgment result, and use the training sentence with the determined first weight value as the first target sentence;

[0159] A first vector acquisition unit, configured to perform sentence vector conversion on the first target sentence to obtain a first vector;

[0160] A matching unit, configured to determine a second weight value of each vocabulary word in the Chinese sentence and the English sentence in the training sentence by matching the training sentence meaning, and use the training sentence with the determined second weight value as the second target sentence;

[0161] A second vector acquisition unit, used to perform sentence vector conversion on the English sentence in the second target sentence to obtain a second vector;

[0162] The comprehensive vector calculation unit is used to calculate the comprehensive vector corresponding to the training sentence based on the first vector and the second vector.

[0163] Further, the combination module 77 includes:

[0164] A comparison submodule, for comparing the target estimate with a preset threshold;

[0165] A first comparison submodule is used to combine the bilingual entity word network and the comparable sentence network into a Chinese-English bilingual corpus if the target estimate is less than or equal to a preset threshold;

[0166] The second comparison submodule is used to iteratively update the bilingual entity word network and the comparable sentence network according to a preset parameter update method if the target valuation is greater than a preset threshold, until the target valuation is less than or equal to the preset threshold, and combine the iteratively updated bilingual entity word network and the comparable sentence network into a Chinese-English bilingual corpus.

[0167] Some embodiments of the present application disclose computer devices. Figure 8, which is a basic structural block diagram of a computer device 80 in one embodiment of the present application.

[0168] like Figure 8 As shown in FIG, the computer device 80 includes a memory 81, a processor 82, and a network interface 83 that are interconnected and communicated via a system bus. It should be noted that Figure 8 Only a computer device 80 having components 81-83 is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0169] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0170] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 80, such as a hard disk or memory of the computer device 80. In other embodiments, the memory 81 can also be an external storage device of the computer device 80, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 80. Of course, the memory 81 can also include both the internal storage unit of the computer device 80 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 80, such as the program code of the method for constructing the Chinese-English bilingual corpus, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0171] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 80. In this embodiment, the processor 82 is used to run the program code stored in the memory 81 or process data, such as running the program code of the method for constructing the Chinese-English bilingual corpus.

[0172] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 80 and other electronic devices.

[0173] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores a Chinese and English entity information entry program, and the Chinese and English entity information entry program can be executed by at least one processor so that the at least one processor performs any step of the above-mentioned Chinese-English bilingual corpus construction method.

[0174] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned Chinese-English bilingual corpus, the above-mentioned Chinese-English bilingual corpus can also be stored in a node of a blockchain.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a computer device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0176] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0177] Finally, it should be noted that the embodiments described above are obviously only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A method for constructing a Chinese-English bilingual corpus, characterized in that: The method for constructing the Chinese-English bilingual corpus includes: Obtain Chinese entities, English entities, and mapping relationships and mutual translation relationships between Chinese entities and English entities from a preset entity library; According to the Chinese entity, the English entity, the mapping relationship and the mutual translation relationship, a bilingual entity word network is constructed according to preset requirements; Acquire context words corresponding to each of the Chinese entities and each of the English entities from a preset database; Based on the Chinese entity, the English entity, the context words, the preset hyperlink set and the preset sentence set, calculating the monolingual representation valuation and the cross-lingual entity valuation of the bilingual entity word network; Obtaining a comparable sentence network and training sentences, and using the training sentences to calculate a cross-language sentence estimation value corresponding to the comparable sentence network; Performing a weighted summation on the single language representation estimation, the cross-language entity estimation, and the cross-language sentence estimation to obtain a target estimation; Comparing the target estimate with a preset threshold, and if the preset condition is met, combining the bilingual entity word network and the comparable sentence network into a Chinese-English bilingual corpus; The step of constructing a bilingual entity word network according to the Chinese entity, the English entity, the mapping relationship and the mutual translation relationship according to preset requirements comprises: Obtain all the Chinese entities as a first set, and all the English entities as a second set; Acquire from a mapping database a first mapping entity having the mapping relationship with the Chinese entity as a third set, and a second mapping entity having the mapping relationship with the English entity as a fourth set, wherein the mapping database includes the first mapping entity and the second mapping entity; Acquire the Chinese entities and the English entities that have the mutual translation relationship as mutual translation entities, and combine all the mutual translation entities into a fifth set; Based on the first set, the second set, the third set, the fourth set and the fifth set, the bilingual entity word network is constructed according to the following formula: ; Wherein, E is the bilingual entity word network, For the first set, For the second set, is the third set, is the fourth set, This is the fifth set.

2. The method for constructing a Chinese-English bilingual corpus according to claim 1, characterized in that: The context words include Chinese context words and English context words. Based on the Chinese entities, the English entities, the context words, a preset hyperlink set and a preset sentence set, the steps of calculating the monolingual representation valuation and the cross-lingual entity valuation of the bilingual entity word network include: Importing the Chinese entity and the English entity into a preset processing port for vector feature conversion to obtain a training entity; Based on the training entity, the context words, the preset hyperlink set and the preset sentence set, the single language representation estimate is calculated according to the following formula: ; Wherein, L is the single language representation estimate, zh is the Chinese entity, en is the English entity, is the training entity, D is a preset sentence set, A is a preset hyperlink set, G is the context word, It means: (i) if , whether it is the context term; (ii) if , link to (iii) If q exists in A, then Is it a context word of q, where q is an element in D; Based on the context words, the cross-language entity valuation is calculated according to the following formula: ; Where I is the cross-language entity valuation, For the current entity, represents the context words. If the current entity is the Chinese entity, it represents the English context words corresponding to the Chinese entity. If the current entity is the English entity, it represents the Chinese context words corresponding to the English entity. Connected entities in other languages.

3. The method for constructing a Chinese-English bilingual corpus according to claim 1, characterized in that: The step of obtaining a comparable sentence network and a training sentence, and using the training sentence to calculate a cross-language sentence valuation corresponding to the comparable sentence network includes: Acquire the comparable sentence network from a preset initial library, wherein the comparable sentence network includes Chinese sentences and English sentences; Obtaining the Chinese sentence and the English sentence containing two identical entities as the training sentence, wherein the training sentence contains the Chinese sentence vector corresponding to the Chinese sentence; According to a preset vector conversion method, the training sentence is converted into a comprehensive vector; According to the comprehensive vector and the Chinese sentence vector, the cross-language sentence estimation is calculated according to the following formula: ; Where J is the cross-language sentence estimation, is the comprehensive vector, is the Chinese sentence vector, and K is the comparable sentence network.

4. The method for constructing a Chinese-English bilingual corpus as claimed in claim 3, characterized in that: The step of converting the training sentence into a comprehensive vector according to a preset vector conversion method comprises: Performing semantic accuracy judgment on the training sentence, determining a first weight value of the training sentence according to the judgment result, and using the training sentence with the determined first weight value as a first target sentence; Performing sentence vector conversion on the first target sentence to obtain a first vector; Determine a second weight value of each vocabulary word in the Chinese sentence and the English sentence in the training sentence by performing sentence meaning matching on the training sentence, and use the training sentence with the determined second weight value as the second target sentence; Performing sentence vector conversion on the English sentence in the second target sentence to obtain a second vector; Based on the first vector and the second vector, a comprehensive vector corresponding to the training sentence is calculated.

5. The method for constructing a Chinese-English bilingual corpus according to claim 1, characterized in that: The step of comparing the target estimate with a preset threshold and, if the preset condition is met, combining the bilingual entity word network and the comparable sentence network into a Chinese-English bilingual corpus comprises: Compare the target estimate to a preset threshold; If the target estimate is less than or equal to a preset threshold, combining the bilingual entity word network and the comparable sentence network into the Chinese-English bilingual corpus; If the target valuation is greater than a preset threshold, the bilingual entity word network and the comparable sentence network are iteratively updated according to a preset parameter update method until the target valuation is less than or equal to the preset threshold, and the iteratively updated bilingual entity word network and comparable sentence network are combined into the Chinese-English bilingual corpus.

6. The method for constructing a Chinese-English bilingual corpus according to claim 1, characterized in that: After the combining into the Chinese-English bilingual corpus, the method further includes: storing the Chinese-English bilingual corpus in a blockchain.

7. A device for constructing a Chinese-English bilingual corpus, characterized in that: The Chinese-English bilingual corpus construction device performs the steps of the Chinese-English bilingual corpus construction method according to any one of claims 1 to 6, and the Chinese-English bilingual corpus construction device comprises: A first acquisition module is used to acquire Chinese entities, English entities, and mapping relationships and mutual translation relationships between Chinese entities and English entities from a preset entity library; A construction module, used to construct a bilingual entity word network according to the Chinese entity, the English entity, the mapping relationship and the mutual translation relationship according to preset requirements; A second acquisition module is used to acquire context words corresponding to each of the Chinese entities and each of the English entities from a preset database; A first calculation module, configured to calculate a single language representation valuation and a cross-language entity valuation of the bilingual entity word network based on the Chinese entity, the English entity, the context words, a preset hyperlink set, and a preset sentence set; A second calculation module, used to obtain a comparable sentence network and a training sentence, and use the training sentence to calculate a cross-language sentence estimation value corresponding to the comparable sentence network; A summing module, configured to perform a weighted summation of the single language representation estimation, the cross-language entity estimation and the cross-language sentence estimation to obtain a target estimation; The combination module is used to compare the target estimate with a preset threshold. If the preset condition is met, the bilingual entity word network and the comparable sentence network are used to combine them into a Chinese-English bilingual corpus.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for constructing a Chinese-English bilingual corpus as claimed in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a Chinese-English bilingual corpus as claimed in any one of claims 1 to 6 are implemented.