Chapter alignment method, device, computer equipment and medium
By calculating the difference between the target similarity and the pre-aligned chapter similarity in the chapter alignment method, the problem of low chapter alignment accuracy in the existing technology is solved, and the accuracy of chapter alignment is improved.
Patent Information
- Application Number
- CN202111411704.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-11-25
AI Technical Summary
When determining whether the two chapters are aligned, the existing chapter alignment method only considers the cosine similarity of the corresponding chapter vectors of the two chapters, resulting in the partial loss or inaccuracy of the multilingual embedded information, which reduces the accuracy of the chapter's discrimination.
By determining the cosine similarity of the vectors of each chapter in the first and second chapters, and combining the similarity of the pre-aligned chapters, we calculate the difference between the target similarity and multiple similarities to determine whether the two chapters are aligned and ensure that the chapters are correct High accuracy.
The accuracy of chapter judgment is improved, and the accuracy decrease caused by partial loss or inaccuracy of multilingual embedded information is offset, achieving higher chapter alignment accuracy.
Smart Images

Figure CN114564932B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a method, apparatus, computer device and medium for aligning a chapter. Background Art
[0002] High-quality and large-scale bilingual parallel corpora are the key to training machine translation models. Bilingual parallel corpora come from many sources, including not only United Nations corpora, subtitle corpora, and dictionary example corpora, but also Internet corpora, which have become more and more common with the development of the Internet in recent years.
[0003] Extracting bilingual parallel corpora from Internet corpora requires the mining of mutually translated text pairs (i.e., the step of text alignment needs to be performed). Taking Apple as an example, the company has localized official websites in almost all countries, including Chinese, English, Japanese, and Korean. These localized official websites each provide Internet corpora describing iPhone 13 in the local language. To extract bilingual parallel corpora from these Internet corpora, it is first necessary to mine mutually translated text pairs describing iPhone 13 from the two localized official websites. After obtaining the text pairs, the required bilingual parallel corpora can be obtained by mining mutually translated sentence pairs in the text pairs.
[0004] At present, the chapter alignment method only considers the cosine similarity of the chapter vectors corresponding to the two chapters when determining whether two chapters are aligned. This method will reduce the accuracy of chapter discrimination due to partial loss or inaccuracy of multilingual embedding information. Summary of the invention
[0005] In view of this, the present disclosure aims to improve the chapter alignment method so that the determined chapter pairs have higher alignment accuracy.
[0006] According to a first aspect of the present disclosure, a chapter alignment method is provided, for determining whether a first chapter included in a first text and a second chapter included in a second text are aligned, the first text including a plurality of pre-aligned chapters of the second chapter and the second text including a plurality of pre-aligned chapters of the first chapter, the method comprising:
[0007] Determine the cosine similarity of the first chapter and the second chapter's respective chapter vectors to obtain a target similarity;
[0008] Determine the cosine similarity of the chapter vectors of the first chapter and the pre-aligned chapters to obtain a plurality of first similarities;
[0009] Determine the cosine similarity of the chapter vectors of the second chapter and the pre-aligned chapter, and obtain multiple second similarities;
[0010] Whether the first chapter is aligned with the second chapter is determined according to the difference between the target similarity and the plurality of first similarities and the plurality of second similarities.
[0011] Optionally, determining whether the first chapter and the second chapter are aligned according to the gap between the target similarity and the plurality of first similarities and the plurality of second similarities includes:
[0012] determining whether the first chapter and the second chapter are aligned according to the first similarity information;
[0013] The first similarity information is a quotient of the target similarity and an average similarity value, and the average similarity value is an average value of the multiple first similarities and the multiple second similarities.
[0014] Optionally, each chapter included in the first text and the second text is text content in a webpage, and determining whether the first chapter and the second chapter are aligned according to the first similarity information includes:
[0015] determining whether the first chapter and the second chapter are aligned according to a weighted sum of the first similarity information and the second similarity information;
[0016] The second similarity information is information for determining whether the first document information and the second document information are similar, the first document information is meta information of the web page corresponding to the first chapter, and the second document information is meta information of the web page corresponding to the second chapter.
[0017] Optionally, the chapter alignment method further includes determining chapter vectors of pre-aligned chapters of the first chapter and the second chapter, and the determining steps are as follows:
[0018] Obtain the chapter vector of each chapter in the current text to obtain multiple candidate chapter vectors;
[0019] Get the chapter vector of the current chapter and obtain a current chapter vector;
[0020] Retrieving a preset number of candidate chapter vectors that are closer to the current chapter vector from the multiple candidate chapter vectors, and determining each of the retrieved candidate chapter vectors as a chapter vector of a pre-aligned chapter of the current chapter;
[0021] The current chapter is one of the first chapter and the second chapter. When the current chapter is the first chapter, the current text is the second text; when the current chapter is the second chapter, the current text is the first text.
[0022] Optionally, obtaining the chapter vector of each chapter in the current text includes: obtaining the sentence vector of each sentence in the chapter included in the current text in the multilingual vector space, and performing pooling operations in various dimensions on the multiple sentence vectors obtained for the chapter to obtain the chapter vector of the chapter.
[0023] Optionally, obtaining the chapter vector of each chapter in the current text also includes:
[0024] Perform any of the following adjustments on the vector obtained by the pooling operation through the vector adjustment model: increase the vector dimension, reduce the vector dimension, and adjust the value of the vector element;
[0025] And, the adjusted vector is determined as the chapter vector of the chapter.
[0026] Optionally, the vector adjustment model is pre-trained by the following steps:
[0027] Constructing a sample set consisting of passage vector samples, wherein the passage vector samples are associated with passage expectation vectors;
[0028] Inputting each chapter vector sample in the sample set into the vector adjustment model, and obtaining a chapter prediction vector of each chapter vector sample from the vector adjustment model;
[0029] The expected chapter vector and the predicted chapter vector of each chapter vector sample in the sample set are compared, and if the consistent ratio does not exceed the predetermined ratio, the weight of the vector adjustment model is adjusted until the consistent ratio exceeds the predetermined ratio.
[0030] According to a second aspect of the present disclosure, a chapter alignment device is provided, for determining whether a first chapter included in a first text and a second chapter included in a second text are aligned, the first text including a plurality of pre-aligned chapters of the second chapter and the second text including a plurality of pre-aligned chapters of the first chapter, the device comprising:
[0031] A first determining unit is used to determine the cosine similarity of the first chapter and the second chapter's respective chapter vectors to obtain a target similarity;
[0032] A second determining unit is used to determine the cosine similarity of the chapter vectors of the first chapter and the pre-aligned chapter, to obtain a plurality of first similarities;
[0033] A third determining unit is used to determine the cosine similarity of the chapter vectors of the second chapter and the pre-aligned chapter, to obtain a plurality of second similarities;
[0034] The fourth determining unit is configured to determine whether the first chapter and the second chapter are aligned according to the target similarity, the plurality of first similarities, and the plurality of second similarities.
[0035] According to a third aspect of the present disclosure, a computer device is provided, comprising:
[0036] A memory for storing computer executable code;
[0037] A processor is used to execute the computer executable code to implement any one of the methods described in the first aspect.
[0038] According to a fourth aspect of the present disclosure, a computer-readable medium is provided, comprising a computer executable code, wherein when the computer executable code is executed by a processor, any one of the methods described in the first aspect is implemented.
[0039] In the disclosed embodiment, the first text includes multiple pre-aligned chapters of the second chapter, and the second text includes multiple pre-aligned chapters of the first chapter. Whether the first chapter included in the first text and the second chapter included in the second text are aligned is determined based on the gap between the target similarity and the multiple first similarities and the multiple second similarities. The larger the gap, the greater the possibility that the first chapter and the second chapter are not aligned. Among them, the target similarity is the cosine similarity of the chapter vectors of the first chapter and the second chapter, the first similarity is the cosine similarity of the chapter vectors of the first chapter and its pre-aligned chapter, and the second similarity is the cosine similarity of the chapter vectors of the second chapter and its pre-aligned chapter. Therefore, when the first chapter and the second chapter are aligned, not only the first chapter and the second chapter need to be sufficiently similar, but also the similarity between the first chapter and its pre-aligned chapter and the similarity between the second chapter and its pre-aligned chapter need to be much smaller than the similarity between the first chapter and the second chapter. This method of determining the alignment of two passages is more in line with the requirements of passage pairs, and can therefore to a certain extent offset the decrease in the accuracy of passage pair identification caused by the partial loss or inaccuracy of multilingual embedded information, thereby achieving the technical effect of improving the accuracy of passage pair identification. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:
[0041] Figure 1 A system architecture diagram of the chapter alignment method of the embodiment of the present disclosure is shown;
[0042] Figures 2A-2C An interface state diagram is shown when the chapter alignment method of the embodiment of the present disclosure is applied;
[0043] Figure 3 A flowchart of a chapter alignment method according to an embodiment of the present disclosure is shown;
[0044] Figure 4A flow chart of a method for determining a chapter vector of a pre-aligned chapter of a current chapter in a current text is shown;
[0045] Figure 5 A vector space diagram of multilingual embeddings is shown;
[0046] Figure 6 A schematic structural diagram of a chapter alignment device according to an embodiment of the present disclosure is shown;
[0047] Figure 7 A structural diagram of a computer device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0048] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.
[0049] Scenarios and system architecture of this public application
[0050] Chapter alignment is common in the scenario of extracting bilingual parallel corpora from Internet corpora. To extract bilingual parallel corpora from Internet corpora, the following steps need to be performed: download text from the Internet → mine mutually translated chapter pairs from text → mine mutually translated sentence pairs from mutually translated chapter pairs.
[0051] Internet corpus, that is, text from the Internet. Many multinational companies provide parallel websites in multiple languages. Taking Apple as an example, the company has localized official websites in almost all countries, including Chinese, English, Japanese, and Korean. These localized official websites each describe the same product (such as iPhone 13) or the same event (such as new product launches) in the local language, and thus can provide Internet corpus with mutual translation relationships. Usually, the domain names of different localized official websites of the same multinational company will contain the same company representation part and different regional representation parts. Therefore, downloading text from the Internet can be to download texts with mutual translation relationships from different localized official websites of the same multinational company based on the domain name. The website content provides information on the same topic in units of web pages. In the embodiment of the present disclosure, the text downloaded from any localized official website of a company is called a text, and the content from a web page in a text is called a chapter, so a text will include a large number of chapters. The embodiment of the present disclosure aims to mine chapter pairs with mutual translation relationships from two texts that each include a large number of chapters.
[0052] Bilingual parallel corpus refers to corpus expressed in two languages with a mutual translation relationship. The above-mentioned mutually translated sentence pairs are bilingual parallel corpus. For example, the mutually translated chapter pair consists of an English chapter and a Chinese chapter. The English chapter includes the sentence "Discover the innovative world of Apple", and the Chinese chapter includes the sentence "Understand the innovative world of Apple". Finally, the sentence pairs "Discover the innovative world of Apple" and "Understand the innovative world of Apple" mined from the chapter pair are a bilingual parallel corpus.
[0053] High-quality and large-scale bilingual parallel corpora are the key to training machine translation models. When bilingual parallel corpora are used to train machine translation models, the chapter alignment method of the embodiment of the present disclosure can be applied in Figure 1 In the architecture shown in Figure 1. Figure 1 The system architecture includes a corpus platform 110 and a plurality of user terminals 120. The corpus platform 110 includes a corpus database 112, a front-end server 111 and a sentence pair extraction device 113 respectively connected to the corpus database 112, and a chapter alignment device 114 connected to the sentence pair extraction device 113. The chapter alignment device 114 receives a first text and a second text and extracts chapter pairs from the first text and the second text. The first text and the second text are, for example, texts downloaded from two different localized official websites of the same multinational company. Then, the chapter alignment device 114 inputs the extracted chapter pairs to the sentence pair extraction device 113. The sentence pair extraction device extracts sentence pairs from the chapter pairs and outputs the extracted sentence pairs to the corpus database 112 for storage. Next, if a user terminal 120 connected to the front-end server 111 sends a corpus acquisition request to the front-end server 111, the front-end server 111 can retrieve a certain number of sentence pairs from the corpus database 112 as bilingual parallel corpora and push them to the user terminal 120.
[0054] The chapter alignment device 114 is a device for executing the chapter alignment method described in the embodiment of the present disclosure. The chapter alignment device 114 can be a single computer or multiple computers working together. Figure 2A -C, briefly describe the chapter alignment method of the embodiment of the present disclosure applied in Figure 1 The interface state of the chapter alignment device 114 changes during the system architecture shown.
[0055] like Figure 2A As shown, the chapter alignment device 114 receives the first text and the second text input by the administrator. Since the first text and the second text each include a large amount of Internet content from a website, the administrator can import the first text and the second text into the document by importing files based on domain names. Figure 2Ain the text box. Figure 2A In the example, the first text is represented by language X, and the second text is represented by language Y which is different from language X. After the first and second texts are imported, the administrator can use Figure 2A The administrator can use the slider shown to check whether the imported text is the text of the required website or whether garbled characters are present. If the administrator confirms that the imported first text and second text are incorrect, the administrator clicks the "Return" button to re-import the first text and second text; if the administrator confirms that the first text and second text are correct, the administrator clicks the "Continue" button to proceed to the next step.
[0056] like Figure 2B As shown, after receiving the correct first text and second text, the chapter alignment device 114 executes the chapter alignment method, and then the display screen displays the execution result of the chapter alignment method, that is, displays the chapter pairs extracted from the first text and the second text. Since a localized official website of any multinational company contains many web pages, there are often more than one chapter pair extracted. Figure 2B , the administrator can browse the extracted multiple chapter pairs (including the first chapter pair, the second chapter pair, etc.) through the slider on the far right. A selection button marked with a check mark and a delete button marked with a cross can also be set in the title box of each chapter pair. The selection button is used for the administrator to select the corresponding chapter pair, and the delete button is used for the administrator to delete the corresponding chapter pair.
[0057] In the administrator Figure 2B After selecting or deleting the several chapters shown, Figure 2C As shown, the display screen of the chapter alignment device 114 displays a request message of "Do you want to output the selected chapter pair?" If the administrator clicks the button "Yes" on the display interface, the chapter alignment device 114 inputs the chapter pair selected by the administrator to the sentence pair extraction device 113; if the administrator clicks the button "No" on the display interface, the chapter alignment device 114 does not input the chapter pair selected by the administrator to the sentence pair extraction device 113, in which case the chapter alignment device 114 can store the chapter pair selected by the administrator locally.
[0058] A method for aligning a chapter according to an embodiment of the present disclosure
[0059] The existing article alignment method only considers the cosine similarity of the article vectors corresponding to the two articles when determining whether two articles are aligned. In this way, determining the mutually translated article of a certain article in the first text in the second text (the mutually translated article is also referred to as the aligned article in the embodiment disclosed in this disclosure) is to calculate the cosine similarity of the article vectors of each article in the second text and the article in the first text, and then find the maximum cosine similarity from the calculated cosine similarities, and determine the article in the second text corresponding to the maximum cosine similarity as the aligned article of the article in the first text. This method will reduce the accuracy of article discrimination due to the partial loss or inaccuracy of multi-language embedded information in the involved article vectors.
[0060] In view of this, an embodiment of the present disclosure provides a chapter alignment method, which mainly differs from the prior art in that when determining whether a first chapter included in a first text and a second chapter included in a second text are aligned, not only the cosine similarity of the chapter vectors corresponding to the first chapter and the second chapter is considered, but also the similarity between the first chapter and its pre-aligned chapter in the second text, and the similarity between the second chapter and its pre-aligned chapter in the first text are considered.
[0061] Figure 3 FIG. 1 is a flow chart of a method for aligning a chapter according to an embodiment of the present disclosure. Figure 3 , the chapter alignment method includes:
[0062] Step S110, determining the cosine similarity of the chapter vectors of the first chapter and the second chapter to obtain the target similarity;
[0063] Step S120, determining the cosine similarity of the chapter vectors of the first chapter and its pre-aligned chapters, and obtaining a plurality of first similarities;
[0064] Step S130, determining the cosine similarity of the chapter vectors of the second chapter and its pre-aligned chapter, and obtaining a plurality of second similarities;
[0065] Step S140 , determining whether the first chapter is aligned with the second chapter according to the difference between the target similarity and the plurality of first similarities and the plurality of second similarities.
[0066] The above steps are described in detail below.
[0067] The above steps S120 to S130 involve the chapter vectors of the pre-aligned chapters of the first chapter and the second chapter, so it is necessary to first determine the chapter vector of the pre-aligned chapter of the first chapter in the second text and determine the chapter vector of the pre-aligned chapter of the second chapter in the first text.
[0068] If one of the first chapter and the second chapter whose chapter vectors are currently determined to be pre-aligned is recorded as the current chapter, and the text where the pre-aligned chapter of the current chapter is located is recorded as the current text (when the current chapter is the first chapter, the current text is the second text, and when the current chapter is the second chapter, the current text is the first text), then in some optional embodiments, the above-mentioned determination of the chapter vector of the pre-aligned chapter of the first chapter / the second chapter is converted to: determining the chapter vector of the pre-aligned chapter of the current chapter in the current text, specifically as follows Figure 4 The steps shown include:
[0069] Step S101, obtaining the chapter vector of each chapter in the current text to obtain multiple candidate chapter vectors;
[0070] Step S102, obtaining the chapter vector of the current chapter to obtain a current chapter vector;
[0071] Step S103 : searching for a preset number of candidate chapter vectors that are closer to the current chapter vector from a plurality of candidate chapter vectors, and determining each of the searched candidate chapter vectors as a chapter vector of a pre-aligned chapter of the current chapter.
[0072] In the disclosed embodiment, a preset number of candidate chapter vectors that are closer to the current chapter vector among multiple candidate chapter vectors can be reasonably used as the chapter vectors of the pre-aligned chapter of the current chapter, thus not affecting the accuracy of the chapter alignment method in distinguishing the chapter pair. Based on this, the first text is used as the potential text of the pre-aligned chapter of the second chapter, and only part of the chapters included in the first text participate in the calculation of the above step S130 as the pre-aligned chapters of the second chapter. Similarly, only part of the chapters included in the second text participate in the calculation of the above step S120 as the pre-aligned chapters of the first chapter, thus reducing the computational complexity of steps S120 and S130 and improving the computational speed of steps S120 and S130; and, in retrieving a preset number of candidate chapter vectors that are closer to the current chapter vector among multiple candidate chapter vectors, the existing more mature vector retrieval technology can be used, and the retrieval can reach the required speed, and the execution speed of the entire chapter alignment process can also be improved.
[0073] In the above step S101, obtaining the chapter vector of each chapter in the current text, the following steps need to be performed:
[0074] Web pages are downloaded from the Internet and text content is extracted from each web page to generate a current text including multiple chapters.
[0075] Identify the language used in the current text, and segment the chapters of the current text according to the identified language. Usually a period or even a semicolon is used as a sign of the end of a sentence. Since different languages use different forms of semicolons and periods, it is necessary to identify the language and segment the chapters according to the form of the period or semicolon used by the identified language.
[0076] For example, if the recognized language is Chinese, semicolon ";" and period "." are used as punctuation marks for sentence segmentation; if the recognized language is English, semicolon ";" and period "." are used as punctuation marks for sentence segmentation.
[0077] for Figure 2B As shown, the first text includes the chapter "Learn about Apple's innovative world; buy various products such as mobile phones, tablets and watches. Explore accessories, entertainment and professional device support." After sentence segmentation, the following three sentences are obtained: "Learn about Apple's innovative world", "Buy various products such as mobile phones, tablets and watches", and "Explore accessories, entertainment and professional device support"; the second text includes the chapter "Discover the innovative world of Apple; shop everything such as iPhone, iPad and Apple Watch. Explore accessories, entertainment, and expert device support." After sentence segmentation, the following three sentences are obtained: "Discover the innovative world of Apple", "shop everything such as iPhone, iPad and Apple Watch", and "Explore accessories, entertainment, and expert device support". It should be understood that a chapter generated based on a web page usually contains a lot of information, and this is just a brief example for understanding.
[0078] Each text data obtained after this step of processing can be stored in files as text units. For example, the file is named "source_lang_doc", where "source" identifies the company and "lang" identifies the language. For example, the first text and the second text from two localized official websites of Apple Inc. are processed in this step to obtain files "apple_zn_doc" and "apple_en_doc", where the file "apple_zn_doc" corresponds to simplified Chinese and the file "apple_en_doc" corresponds to English.
[0079] For any chapter in the current text, each sentence is obtained by sentence segmentation, and then word segmentation is performed to obtain multiple minimum semantic units (tokens); then the sentence vector of the sentence in the multilingual vector space is obtained according to the mapping position of each minimum semantic unit in a sentence in the multilingual vector space.
[0080] In the above sentence segmentation process, if the sentence is a Chinese sentence, the jieba segmentation can be used. The jieba segmentation is mainly performed through a dictionary, so it can well divide the Chinese sentence into multiple minimum semantic units. If the sentence is a sentence in other languages, the space segmentation can be directly used. For example, the sentence "Discover theinnovative world of Apple" can obtain multiple words "Discover", "the", "innovative", "world", "of", and "Apple" after using the space segmentation.
[0081] In some examples, after obtaining the minimum semantic unit, the minimum semantic unit can be first processed by BPE to obtain the semantics of the minimum semantic unit through BPE processing and map the minimum semantic unit to the vector space according to the obtained semantics. BPE, the full name of "Byte Pair Encoding", is a data compression method used to replace the original continuous character data in a string with continuous character data that does not exist in the string, and then reconstruct the original continuous character data through a vocabulary. For example, the three words "loved", "loving", and "loves" all have the meaning of "love". BPE processes the above three words, that is, replaces the three words with "lov" + "ed", "lov" + "ing", and "lov" + "es" respectively, so that the meaning of the word itself and the tense can be separated. The last three words correspond to the meaning of the same word "lov" with the help of the tense suffixes "ed", "ing", and "es", so that the vocabulary that determines the meaning of the word does not need to include the meaning of all words, that is, the purpose of effectively reducing the size of the vocabulary is achieved.
[0082] Regardless of whether BPE processing is performed, the mapping positions of the above-mentioned minimum semantic units in the multilingual vector space are obtained based on the multilingual vector embedding technology. The multilingual vector embedding technology enables the text included in each language to be embedded in the same multilingual vector space, and texts with similar meanings in different languages are closely clustered in the multilingual vector space. For example, the text included in multiple languages is a single word, such as Figure 5As shown, "futbol" in Turkish, "soccer" in English and "football" in Chinese are very close in the three-dimensional multilingual vector space, and "top" in Turkish, "ball" in English and "ball" in Chinese are very close in the three-dimensional multilingual vector space. Of course, the texts included in multiple languages can also be sentences. The meaning of a sentence is determined by the words that make up the sentence. Therefore, the sentence vector of a sentence in the multilingual vector space can be determined according to the mapping position of each minimum semantic unit in a sentence in the multilingual vector space, and sentences with similar meanings are closely clustered in the multilingual vector space.
[0083] It should be noted that the multilingual vector space is a vector space, so texts with similar meanings in different languages correspond to the same vector in the multilingual vector space, and sentences with the same meaning also correspond to the same vector (i.e., the same sentence vector) in the multilingual vector space. Once the dimension of the multilingual vector space is determined, the dimension of the sentence vector is uniquely determined. Obtaining the sentence vector of a sentence in the multilingual vector space based on the mapping position of each minimum semantic unit in a sentence in the multilingual vector space belongs to the prior art and will not be described in detail here.
[0084] (d) After obtaining the sentence vectors of each sentence in the chapter included in the current text in the multilingual vector space, a pooling operation of each dimension is performed on the multiple sentence vectors obtained for the chapter to obtain the chapter vector of the chapter.
[0085] Pooling is essentially sampling, that is, reducing the dimension of input data through sampling. In the embodiment of the present disclosure, the pooling operation is to reduce the dimension of multiple sentence vectors into one paragraph vector. The two most commonly used pooling processes are Max Pooling and Average Pooling.
[0086] The multiple sentence vectors obtained for the article are pooled in each dimension, that is, the maximum value of each dimension in the multiple sentence vectors is selected, and then a vector is formed by the selected maximum values to generate the article vector. For example, the sentence vectors of multiple sentences in an article are (1, 2, 4, 5) and (3, 4, 6, 3) respectively. After the maximum pooling of each dimension of the multiple sentence vectors, the article vector of the article is (3, 4, 6, 5).
[0087] The multiple sentence vectors obtained for the chapter are averaged in each dimension, that is, the average values of each dimension in the multiple sentence vectors are calculated, and then a vector is formed from the calculated average values, that is, the chapter vector is generated. Similarly, the sentence vectors of multiple sentences in a chapter are (1,2,4,5) and (3,4,6,3) respectively. After the average pooling of each dimension of the multiple sentence vectors, the chapter vector of the chapter is (2,3,5,4).
[0088] The dimension of the passage vector obtained by the pooling operation in this step is the same as that of the sentence vector. The pooling operation allows the passage vector to sample the feature information of multiple sentence vectors, and the passage vector does not need to include all the features of multiple sentence vectors. The dimension of the passage vector is reduced, so the subsequent computational complexity based on the passage vector can also be effectively reduced, and the computational speed is improved.
[0089] Further, step S101, obtaining the chapter vector of each chapter in the current text, also includes: performing any of the following adjustments on the vector obtained by the pooling operation through the vector adjustment model: increasing the vector dimension, reducing the vector dimension, adjusting the value of the vector element; and determining the adjusted vector as the chapter vector of the chapter.
[0090] It should be noted that increasing the vector dimension and adjusting the vector element values are both for improving the accuracy of the chapter vector, so that the adjusted chapter vector can more accurately represent the chapter characteristics. Increasing the vector dimension can be to add one or more vector elements at the end of the vector obtained by the pooling operation. These added vector elements can be derived based on the arrangement rules or value rules of the existing vector elements in the chapter vector, which play a role in enhancing the characteristics of the adjusted chapter vector in a certain aspect. Adjusting the vector elements is to make the adjusted vector element configuration have a value that is more in line with practical significance, for example, assigning the vector element with a value greater than a threshold as the threshold.
[0091] In some cases, the passage vector obtained based on the above multilingual vector space still has a large dimension, and the current computing power of the passage alignment device 114 cannot bear the relevant calculations of this dimension, so it is necessary to reduce the vector dimension of the passage vector. To reduce the vector dimension of the passage vector, it is necessary to ensure that the accuracy of the passage vector is not affected as much as possible. For example, the vector elements that are more likely to represent function words are deleted from the vector obtained by the pooling operation.
[0092] The above-mentioned vector adjustment model can use deep neural networks (DNN for short), which are pre-trained through the following steps: constructing a sample set consisting of chapter vector samples, and the chapter vector samples are associated with chapter expectation vectors; inputting each chapter vector sample in the sample set into the vector adjustment model, and obtaining the chapter prediction vector of each chapter vector sample from the vector adjustment model; comparing the chapter expectation vector and the chapter prediction vector of each chapter vector sample in the sample set, and if the consistent ratio does not exceed the predetermined ratio (for example, 95%), adjusting the weight of the vector adjustment model until the consistent ratio exceeds the predetermined ratio.
[0093] After the above step S101 is executed, the chapter vectors of each chapter in the first text and the chapter vectors of each chapter in the second text are obtained. Therefore, the chapter vector of the current chapter can be obtained according to the correspondence between the chapter and the chapter vector, so that step S102 can be implemented. The chapter vectors of each chapter in the first text and the chapter vectors of each chapter in the second text can be stored in files in units of text. Exemplarily, the file storing the chapter vectors of the first text is named "src_doc_embeddings", and the file storing the chapter vectors of the second text is named "tgt_doc_embeddings".
[0094] In step S103, a preset number of candidate chapter vectors that are closer to the current chapter vector are retrieved from the multiple candidate chapter vectors, and an existing vector search tool can be used to search for k nearest neighbors of the current chapter vector from the multiple candidate chapter vectors.
[0095] Assume that the chapter vector of the first chapter is the vector x stored in the file "src_doc_embeddings", and the chapter vector of the second chapter is the vector y stored in the file "tgt_doc_embeddings". In the case that the current chapter is the first chapter, step S103 is to find the k nearest neighbors of vector x in the file "tgt_doc_embeddings" (i.e., find the k nearest neighbors of vector x). The k nearest neighbors found here can be recorded as NN k (x); In the case where the current chapter is the second chapter, step S103 is to find the k nearest neighbors of vector y in the file "src_doc_embeddings". The k nearest neighbors found here can be recorded as NN k (y).
[0096] The above-mentioned search for the k nearest neighbors of a vector is achieved by calculating the cosine similarity of two vectors, that is, measuring the distance between the two vectors by the cosine value of the angle between the two vectors in the vector space (the closer the cosine value is to 1, the closer the angle between the two vectors is to 0°, that is, the more similar the two vectors are, which is the so-called "cosine similarity"). Specifically, you can use Alibaba's proxima vector search tool or Facebook's faiss vector search tool. Both of these vector search tools can find the k nearest neighbors of a vector in a very short time.
[0097] After obtaining the chapter vector of the first chapter, the chapter vector of the second chapter, the chapter vector of the pre-aligned chapter of the first chapter, and the chapter vector of the pre-aligned chapter of the second chapter through the above method, the target similarity, the first similarity, and the second similarity can be obtained through steps S110-S130. At this time, the above step S140 can be executed.
[0098] Step S140 , determining whether the first chapter is aligned with the second chapter according to the difference between the target similarity and the plurality of first similarities and the plurality of second similarities.
[0099] The difference between the target similarity and the multiple first similarities and the multiple second similarities can be the result obtained by performing a difference operation between the target similarity and the multiple first similarities and the multiple second similarities. For example, the target similarity is subjected to a difference operation with the average similarity value, and the result of the difference operation represents the difference between the target similarity and the multiple first similarities and the multiple second similarities, wherein the average similarity value is the average value of the multiple first similarities and the multiple second similarities. In this case, if the result of the difference operation is greater than zero, the greater the result of the difference operation, the greater the difference between the target similarity and the multiple first similarities and the multiple second similarities; if the result of the difference operation is not greater than zero, the first chapter and the second chapter cannot be used as a chapter pair.
[0100] The gap between the target similarity and the plurality of first similarities and the plurality of second similarities may also be a result obtained by performing a quotient operation between the target similarity and the plurality of first similarities and the plurality of second similarities. For example, the target similarity is quotiented with the average similarity value and the result of the quotient operation represents the gap between the target similarity and the plurality of first similarities and the plurality of second similarities.
[0101] In the embodiment of the present disclosure, the quotient of the target similarity and the average similarity is also referred to as the first information. In this case, step S140 is to determine whether the first chapter and the second chapter are aligned according to the first information, and the first information can be determined according to formula (1).
[0102]
[0103] Among them, margin_score(x,y) represents the first similarity information, cos(x,y) represents the target similarity, cos(x,z) represents a first similarity, cos(y,z) represents a second similarity, k represents the number of first similarities and also represents the number of second similarities, and the rest is as described above.
[0104] In the case of determining whether the first chapter and the second chapter are aligned based on the first information, if the first information is greater than 1, the larger the first information is, the greater the gap between the target similarity and multiple first similarities and multiple second similarities is, and the greater the possibility that the first chapter and the second chapter are aligned is. Here, the gap between the target similarity and multiple first similarities and multiple second similarities is reflected by how many times the target similarity is the average value of the similarities, and the reflected gap is more meaningful for reference; if the first information is less than 1, the similarity between the first chapter and the second chapter is very small, and the first chapter and the second chapter cannot be used as a chapter pair.
[0105] In an optional embodiment, the above-mentioned determining whether the first chapter and the second chapter are aligned based on the first information includes: determining whether the first chapter and the second chapter are aligned based on the weighted sum of the first similarity information and the second similarity information, specifically, determining whether the first chapter and the second chapter are aligned based on the parameter f(x, y) represented by formula (2).
[0106]
[0107] In formula (2), meta_sim(x, y) represents the second similarity information, which is information for determining whether the first document information and the second document information are similar. The first document information is the meta information of the web page corresponding to the first chapter on the website where the first text is downloaded, and the second document information is the meta information of the web page corresponding to the second chapter on the website where the second text is downloaded. The process of determining the second similarity information meta_sim(x, y) here belongs to the prior art and will not be described in detail.
[0108] The meta-information of a web page is provided by the meta element, which is information about the information and is used to describe the structure, semantics, purpose and usage of the information on the web page. The meta element is the element stored in the meta tag, which is an auxiliary tag in the header area of Hypertext Markup Language (English abbreviation: HTML). The meta-information of the same web page is the same, and the web page meta-information of two localized official websites of the same multinational company has a certain degree of similarity (for example, the meta-information describing the purpose and usage of the information on the web page is the same, but the meta-information describing the structure of the information on the web page may differ due to different languages).
[0109] In formula (2), W1 represents the first weight, which is the weight of the first similar information margin_score, and W2 represents the second weight, which is the weight of the second similar information meta_sim(x, y), and W1+W2=1. In practice, W1 and W2 can be adjusted according to the similarity of the meta information of two localized official websites of the same multinational company, and the more similar they are, the larger W2 is.
[0110] In practice, for a first text and a second text that have a mutual translation relationship, a chapter in the first text may eventually match a single chapter in the second text as a chapter pair, or may match multiple chapters in the second text as a chapter pair. The former requires a higher precision for the chapter pair, while the latter requires a lower precision for the chapter pair. For the case where the first chapter included in the first text matches a single chapter in the second text as a chapter pair, the above scheme for determining whether the first chapter and the second chapter are aligned may be used to k The chapter corresponding to each chapter vector in (y) is used as the second chapter to calculate f(x,y), and the chapter corresponding to the calculated maximum f(x,y) in the second text is used as the chapter matched by the first chapter in the second text. In the case where the first chapter included in the first text matches multiple chapters in the second text as a chapter pair, the above scheme for determining whether the first chapter and the second chapter are aligned can be NN k The chapter corresponding to each chapter vector in (y) is used as the second chapter to calculate f(x, y), and the chapter corresponding to the calculated f(x, y) greater than the preset value in the second text is used as the chapter matched by the first chapter in the second text.
[0111] In the disclosed embodiment, the first similarity information is similarity information determined based on the chapter vector, and the second similarity information is similarity information determined based on the web page meta information. Whether the first chapter and the second chapter are aligned is determined based on the weighted sum of the first similarity information and the second similarity information, thereby improving the accuracy of chapter pair identification.
[0112] Commercial Uses of the Disclosure
[0113] When determining whether the first chapter and the second chapter are aligned, the disclosed embodiment not only considers whether the first chapter and the second chapter are similar enough, but also considers whether the similarity between the first chapter and its pre-aligned chapter and the similarity between the second chapter and its pre-aligned chapter are much smaller than the similarity between the first chapter and the second chapter. This chapter pair determination method is more in line with the requirements of the chapter pair, and thus can offset the decrease in the accuracy of chapter pair determination caused by the partial loss or inaccuracy of multilingual embedded information to a certain extent, thereby improving the accuracy of chapter pair determination.
[0114] Furthermore, the pre-aligned chapters of the first chapter are some of the chapters retrieved from all the chapters included in the second text using vector retrieval technology. Similarly, the pre-aligned chapters of the second chapter are some of the chapters retrieved from all the chapters included in the first text using vector retrieval technology. Therefore, the implementation efficiency of the chapter alignment process is fully improved, and ultimately it can ensure that hundreds of millions of web page alignment tasks are processed every day with limited machine resources (including central processing units, graphics processing units, memory, etc.), which meets the needs of extracting a large amount of bilingual parallel corpus from the Internet.
[0115] In addition, since the paragraph alignment process provided by the embodiment of the present disclosure is implemented based on sentence vectors mapped in a multilingual vector space, and the existing laser model can support the mapping of texts in 93 languages in a multilingual vector space with 4278 language directions (i.e., dimensions), the vector alignment solution provided by the embodiment of the present disclosure is suitable for a large number of languages and has good scalability.
[0116] In summary, the chapter alignment solution provided by the embodiments of the present disclosure can meet the needs of industrial large-scale chapter alignment tasks based on its excellent performance in accuracy, efficiency and scalability, and has a good market prospect in the extraction of bilingual parallel corpus.
[0117] The chapter alignment device of the disclosed embodiment
[0118] According to an embodiment of the present disclosure, a chapter alignment device is also provided, the chapter alignment device is used to determine whether a first chapter included in a first text and a second chapter included in a second text are aligned, the first text includes a plurality of pre-aligned chapters of the second chapter and the second text includes a plurality of pre-aligned chapters of the first chapter. Figure 6 , the chapter alignment device 600 comprises:
[0119] A first determining unit 610 is used to determine the cosine similarity of the first chapter and the second chapter's respective chapter vectors to obtain a target similarity;
[0120] A second determining unit 620 is used to determine the cosine similarity of the chapter vectors of the first chapter and the pre-aligned chapters to obtain a plurality of first similarities;
[0121] The third determining unit 630 is used to determine the cosine similarity of the chapter vectors of the second chapter and the pre-aligned chapter, and obtain multiple second similarities;
[0122] The fourth determining unit 640 is configured to determine whether the first chapter and the second chapter are aligned according to the target similarity, the plurality of first similarities, and the plurality of second similarities.
[0123] Since the implementation details of the above device have been described in the detailed introduction of the method embodiment above, they will not be repeated here for the sake of saving space.
[0124] The chapter alignment method according to an embodiment of the present disclosure may be performed by Figure 7 The computer device 800 shown is implemented. Figure 7 800 according to an embodiment of the present disclosure is described below. Figure 7 The computer device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0125] like Figure 7 As shown, the computer device 800 is in the form of a general computing device. The components of the computer device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).
[0126] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps of various exemplary embodiments of the present disclosure described in the description of the exemplary method described above in this specification. For example, the processing unit 810 can perform the steps shown in FIG. 2.
[0127] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 8201 and / or a cache storage unit 8202 , and may further include a read-only storage unit (ROM) 8203 .
[0128] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0129] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0130] The computer device 800 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 800, and / or may communicate with any device that enables the computer device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface 850. In addition, the computer device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 860. As shown, the network adapter 860 communicates with other modules of the computer device 800 through a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0131] It should be understood that the above is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, there are many variations of the embodiments of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
[0132] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0133] It should be understood that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0134] It should be understood that an element described in singular form herein or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be split into multiple modules or elements.
[0135] It should also be understood that the terms and expressions used herein are for description only, and one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or parts thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, changes and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.
Claims
1. A chapter alignment method, for determining whether a first chapter included in a first text and a second chapter included in a second text are aligned, the first text including a plurality of pre-aligned chapters of the second chapter and the second text including a plurality of pre-aligned chapters of the first chapter, the method comprising: Determine the cosine similarity of the first chapter and the second chapter's respective chapter vectors to obtain a target similarity; Determine the cosine similarity of the chapter vectors of the first chapter and the pre-aligned chapters to obtain a plurality of first similarities; Determine the cosine similarity of the chapter vectors of the second chapter and the pre-aligned chapter, and obtain multiple second similarities; Whether the first chapter and the second chapter are aligned is determined according to a gap between the target similarity and an average similarity value, wherein the average similarity value is used to represent an average value of the plurality of first similarities and the plurality of second similarities.
2. The method for aligning a paragraph according to claim 1, wherein: Determining whether the first chapter and the second chapter are aligned according to the gap between the target similarity and the plurality of first similarities and the plurality of second similarities includes: determining whether the first chapter and the second chapter are aligned according to the first similarity information; The first similarity information is a quotient of the target similarity and an average similarity value, and the average similarity value is an average value of the multiple first similarities and the multiple second similarities.
3. The method for aligning a chapter according to claim 2, wherein: Each chapter included in the first text and the second text is text content in a webpage, and determining whether the first chapter and the second chapter are aligned according to the first similarity information includes: determining whether the first chapter and the second chapter are aligned according to a weighted sum of the first similarity information and the second similarity information; The second similarity information is information for determining whether the first document information and the second document information are similar, the first document information is meta information of the web page corresponding to the first chapter, and the second document information is meta information of the web page corresponding to the second chapter.
4. The chapter alignment method according to claim 1, further comprising determining the chapter vectors of the first chapter and the second chapter as pre-aligned chapters, and the determining steps are as follows: Obtain the chapter vector of each chapter in the current text to obtain multiple candidate chapter vectors; Get the chapter vector of the current chapter and obtain a current chapter vector; Retrieving a preset number of candidate chapter vectors that are closer to the current chapter vector from the multiple candidate chapter vectors, and determining each of the retrieved candidate chapter vectors as a chapter vector of a pre-aligned chapter of the current chapter; in, The current chapter is one of the first chapter and the second chapter. When the current chapter is the first chapter, the current text is the second text. When the current chapter is the second chapter, the current text is the first text.
5. The method for aligning a paragraph according to claim 4, wherein: Obtaining the chapter vector of each chapter in the current text includes: obtaining the sentence vector of each sentence in the chapter included in the current text in the multilingual vector space, and performing pooling operations in various dimensions on the multiple sentence vectors obtained for the chapter to obtain the chapter vector of the chapter.
6. The method for aligning a paragraph according to claim 5, wherein: Get the chapter vector of each chapter in the current text, including: Perform any of the following adjustments on the vector obtained by the pooling operation through the vector adjustment model: increase the vector dimension, reduce the vector dimension, and adjust the value of the vector element; And, the adjusted vector is determined as the chapter vector of the chapter.
7. The method for aligning a chapter according to claim 6, wherein: The vector adjustment model is pre-trained by the following steps: Constructing a sample set consisting of passage vector samples, wherein the passage vector samples are associated with passage expectation vectors; Inputting each chapter vector sample in the sample set into the vector adjustment model, and obtaining a chapter prediction vector of each chapter vector sample from the vector adjustment model; The expected chapter vector and the predicted chapter vector of each chapter vector sample in the sample set are compared, and if the consistent ratio does not exceed the predetermined ratio, the weight of the vector adjustment model is adjusted until the consistent ratio exceeds the predetermined ratio.
8. A chapter alignment device, for determining whether a first chapter included in a first text and a second chapter included in a second text are aligned, the first text including a plurality of pre-aligned chapters of the second chapter and the second text including a plurality of pre-aligned chapters of the first chapter, the device comprising: A first determining unit is used to determine the cosine similarity of the first chapter and the second chapter's respective chapter vectors to obtain a target similarity; A second determining unit is used to determine the cosine similarity of the chapter vectors of the first chapter and the pre-aligned chapter, to obtain a plurality of first similarities; A third determining unit is used to determine the cosine similarity of the chapter vectors of the second chapter and the pre-aligned chapter, to obtain a plurality of second similarities; The fourth determination unit is used to determine whether the first chapter and the second chapter are aligned according to the difference between the target similarity and the similarity average value, wherein the similarity average value is used to represent the average value of the multiple first similarities and the multiple second similarities.
9. A computer device comprising: A memory for storing computer executable code; A processor, configured to execute the computer executable code to implement the method according to any one of claims 1 to 7.
10. A computer-readable medium comprising a computer-executable code, wherein the computer-executable code implements the method according to any one of claims 1 to 7 when executed by a processor.
Citation Information
Patent Citations
Sentence-level bilingual alignment method and device and computer readable storage medium
CN109670178A
Lao-Chinese bilingual parallel sentence extraction method based on Bi-LSTM algorithm
CN110362820A
