Corpus matching method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a corpus matching method, apparatus, device and storage medium, the method comprising: obtaining a corpus to be matched and a required matching type of the corpus to be matched; obtaining a corpus that matches the required matching type according to the required matching type of the corpus to be matched; each sentence pair contained in the corpus has multiple attribute settings; and obtaining a target corpus that matches the corpus to be matched according to the multiple attribute settings of each sentence pair contained in the corpus. Based on the multiple attribute settings for each sentence pair in the corpus, while achieving corpus matching, the corpus in the corpus can also be divided into fine-grained parts, and the accuracy of corpus reuse can be maximized based on the minimum granularity corpus matching. At the same time, the corpus reuse can be maximized based on multi-dimensional reuse, which is convenient for improving the efficiency of corpus matching.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a corpus matching method, a corresponding corpus matching device, a corresponding electronic device, and a computer storage medium. Background Art

[0002] At present, corpora can usually be used to establish knowledge systems, such as corpora, etc., and when applied in various fields, the established corpora can be used to provide support for corpus analysis, corpus matching and other operations. Among them, the application field can include the translation field, that is, in the translation field, corpora can be used to match corpora to maximize the accuracy of corpus matching and corpus reuse.

[0003] However, when performing corpus matching in the translation field, there are usually multiple matches for one sentence (i.e., the same source language sentence corresponds to sentences in different target languages. For example, in Chinese-English translation, there are different English sentences corresponding to the same Chinese sentence), because the same sentence needs to be translated into different sentences in the same language in documents of different products and different document types of the same product. In this case, the related mixed corpus matching method in the corpus will lead to inaccurate corpus matching and is not conducive to improving translation efficiency. Summary of the invention

[0004] In view of the above problems, embodiments of the present application are proposed to provide a corpus matching method, a corresponding corpus matching device, a corresponding electronic device and a computer storage medium that overcome the above problems or at least partially solve the above problems.

[0005] The present application discloses a corpus matching method, which includes:

[0006] Obtaining a corpus to be matched and a required matching type of the corpus to be matched;

[0007] According to the required matching type of the corpus to be matched, a corpus that matches the required matching type is acquired; each sentence pair included in the corpus has multiple attribute settings;

[0008] According to the multiple attribute settings of each sentence pair included in the corpus, a target corpus matching the corpus to be matched is obtained.

[0009] Optionally, the method further comprises:

[0010] Acquire multiple corpora and sentence pairs contained in each corpus, and determine at least one application scenario involved in each sentence pair;

[0011] Based on the information of the application scenario involved in each sentence pair, multiple attributes are set for each sentence pair; the multiple attributes set include the product code involved in the sentence pair, the document type involved in the sentence pair, and the document subject identifier involved in the sentence pair; wherein, there are documents of multiple document types under the same product code, and documents of each document type are composed of documents with multiple document subject identifiers.

[0012] Optionally, the acquiring, according to the required matching type of the corpus to be matched, a corpus that matches the required matching type includes:

[0013] Acquire a main corpus and a reference corpus that are multi-layer hierarchically arranged based on the required matching type;

[0014] The step of obtaining a main corpus and a reference corpus that are hierarchically arranged in multiple layers based on the requirement matching type includes:

[0015] Acquire a corpus that matches the required matching type, where the required matching type includes a matching attribute range;

[0016] The matching attribute range is used to set a corpus that matches the required matching type, and the corpus after the matching attribute range is set is divided into a main corpus and a reference corpus according to the priority of each corpus set.

[0017] Optionally, obtaining a target corpus matching the corpus to be matched according to multiple attribute settings for each sentence pair included in the corpus includes:

[0018] Based on the multiple attributes of each sentence pair contained in the acquired main corpus and reference corpus, a target corpus that meets the matching attribute range is obtained from the main corpus and the reference corpus.

[0019] Optionally, it also includes:

[0020] During the matching process, if the acquired main corpus and / or reference corpus is provided with an automatic import function for new sentence pairs, the new sentence pairs are automatically imported into the corresponding database provided with the automatic import function for new sentence pairs, and attribute settings are added for the imported new sentence pairs; wherein the imported new sentence pairs are the corpus obtained in the historical matching process and saved after deduplication.

[0021] Optionally, the sentence pairs contained in the acquired corpus include historically imported sentence pairs in the corpus and newly imported sentence pairs;

[0022] The method of matching the target corpus that meets the matching attribute range from the main corpus and the reference corpus based on the multiple attributes of each sentence pair contained in the obtained main corpus and the reference corpus includes:

[0023] Obtaining historically imported sentence pairs and / or imported newly added sentence pairs that meet the matching attribute range in the main corpus and the reference corpus;

[0024] Calculating the similarity between the historical imported sentence pairs and / or the imported newly added sentence pairs that meet the matching attribute range and the corpus to be matched;

[0025] If there are multiple sentence pairs with the same similarity to the corpus to be matched, then based on the multiple attributes set for the multiple sentence pairs with the same similarity, the similarity of the multiple sentence pairs with the same similarity is weighted respectively;

[0026] The weighted similarity is used to sort the multiple sentence pairs with the same similarity from high to low, and the sentence pair with the highest similarity in the sorting is determined as the target corpus to be matched with the corpus to be matched.

[0027] Optionally, the calculating the similarity with the corpus to be matched includes:

[0028] Acquire the original text information of the newly imported sentence pairs and / or the historically imported sentence pairs that meet the matching attribute range, and the sentences to be retrieved where the corpus to be matched is located;

[0029] Determine the text similarity between the original text information and the sentence to be retrieved where the corpus to be matched is located;

[0030] When the calculated text similarity reaches a preset degree, the similarity between the corresponding sentence pair and the corpus to be matched is increased based on the number of identical sentences existing in the original text information and the sentences to be retrieved.

[0031] The present application also discloses a corpus matching device, which includes:

[0032] A to-be-matched corpus acquisition module is used to acquire the to-be-matched corpus and the required matching type of the to-be-matched corpus;

[0033] A corpus acquisition module, used for acquiring a corpus that matches the required matching type according to the required matching type of the corpus to be matched; each sentence pair included in the corpus has multiple attribute settings;

[0034] The target corpus matching module is used to obtain a target corpus that matches the corpus to be matched according to the multiple attribute settings of each sentence pair contained in the corpus.

[0035] Optionally, the device further comprises:

[0036] A sentence pair acquisition module, used to acquire multiple corpora and sentence pairs contained in each corpus, and determine at least one application scenario involved in each sentence pair;

[0037] The attribute setting module is used to set multiple attributes for each sentence pair based on the information of the application scenario involved in each sentence pair; the multiple attributes set include the product code involved in the sentence pair, the document type involved in the sentence pair and the document subject identifier involved in the sentence pair; wherein, there are documents of multiple document types under the same product code, and each document type is composed of documents with multiple document subject identifiers.

[0038] Optionally, the corpus acquisition module includes:

[0039] A hierarchical corpus acquisition submodule is used to acquire a main corpus and a reference corpus that are hierarchically arranged in multiple layers based on the required matching type;

[0040] Among them, the hierarchical corpus acquisition submodule may include the following units:

[0041] A corpus acquisition unit, configured to acquire a corpus that matches the required matching type, wherein the required matching type includes a matching attribute range;

[0042] The corpus grading unit is used to set the corpus that matches the required matching type by using the matching attribute range, and divide the corpus after the matching attribute range is set into a main corpus and a reference corpus according to the priority of each set corpus.

[0043] Optionally, the target corpus matching module includes:

[0044] The target corpus screening submodule is used to match the target corpus that meets the matching attribute range from the main corpus and the reference corpus based on the multiple attributes of each sentence pair contained in the acquired main corpus and the reference corpus.

[0045] Optionally, the target corpus matching module further includes:

[0046] The newly added sentence pair importing submodule is used to automatically import the newly added sentence pairs into the corresponding database with the newly added sentence pair automatic importing function during the matching process, if the acquired main corpus and / or reference corpus is provided with the newly added sentence pair automatic importing function for itself, and add attribute settings for the imported newly added sentence pairs; wherein the imported newly added sentence pairs are the corpus obtained in the historical matching process and saved after deduplication.

[0047] Optionally, the sentence pairs contained in the acquired corpus include historically imported sentence pairs in the corpus and newly imported sentence pairs; and the target corpus screening submodule includes:

[0048] A newly added sentence pair acquisition unit is used to acquire historically imported sentence pairs and / or imported newly added sentence pairs that meet the matching attribute range in the main corpus and the reference corpus;

[0049] A similarity calculation unit, used to calculate the similarity between the historical imported sentence pairs and / or the imported newly added sentence pairs that meet the matching attribute range and the corpus to be matched;

[0050] A weighting unit, configured to weight the similarity of the plurality of sentence pairs with the same similarity, respectively, based on multiple attributes set for the plurality of sentence pairs with the same similarity, when there are a plurality of sentence pairs with the same similarity.

[0051] The sorting unit is used to sort the plurality of sentence pairs with the same similarity from high to low using the weighted similarity, and determine the sentence pair with the highest similarity in the sorting as the target corpus to be matched with the corpus to be matched.

[0052] An embodiment of the present application also discloses an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements any step of the corpus matching method when executed by the processor.

[0053] The embodiment of the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the corpus matching methods are implemented.

[0054] The embodiments of the present application include the following advantages:

[0055] In an embodiment of the present application, a corpus that matches the required matching type of the corpus to be matched is obtained, and each sentence pair contained in the obtained corpus has multiple attribute settings, so that a target corpus that matches the corpus to be matched is obtained based on the set multiple attribute matching. Based on the multiple attribute settings for each sentence pair in the corpus, while achieving corpus matching, the corpus in the corpus can also be divided into fine-grained parts, and the accuracy of corpus reuse can be maximized based on the corpus matching with the minimum granularity. At the same time, the corpus reuse can be maximized based on multi-dimensional reuse, which is convenient for improving the efficiency of corpus matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flowchart of the steps of an embodiment of a corpus matching method of the present application;

[0057] Figure 2 is a flowchart of another embodiment of a corpus matching method of the present application;

[0058] Figure 3 It is a schematic diagram of the relationship between multiple attributes set in an embodiment of the present application;

[0059] Figure 4 is a schematic diagram of corpus matching provided in an embodiment of the present application;

[0060] Figure 5 This is an application scenario diagram of corpus matching provided in an embodiment of the present application;

[0061] Figure 6 It is a structural block diagram of an embodiment of a corpus matching device of the present application. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0063] When corpus matching is performed in the translation field, there are usually multiple matches for one sentence (i.e., the same source language sentence corresponds to different target languages. Taking Chinese to English as an example, there are different English sentences corresponding to the same Chinese sentence), because the same sentence needs to be translated into different sentences in the same language in documents of different products and different document types of the same product. For this situation, in order to maximize the accuracy of corpus matching and the rate of corpus reuse, a multi-dimensional corpus matching logic can be proposed. Based on the multiple attribute settings of each sentence pair in the corpus, a fine-grained division of the corpus contained in the corpus is achieved. Since different sentence pairs may have multiple different attribute settings, at this time, while ensuring the accuracy of matching based on fine-grained attribute division, the corpus matching based on the minimum granularity can be achieved, that is, based on sentence pairs with multiple different attribute settings to maximize the accuracy of corpus reuse. At the same time, the corpus reuse can be maximized based on multi-dimensional reuse, which is convenient for improving the efficiency of corpus matching.

[0064] Reference Figure 1 , shows a flowchart of a corpus matching method embodiment of the present application, which may specifically include the following steps:

[0065] Step 101, obtaining the corpus to be matched and the required matching type of the corpus to be matched;

[0066] The matching of corpus can be expressed as the translation of a certain corpus. In this case, the corpus to be matched can be the corpus to be translated. When obtaining the corpus to be matched, the requirement matching type corresponding to the corpus to be matched can also be determined, so as to determine the target corpus when matching the corpus based on the determined requirement matching type.

[0067] Among them, since the required matching corpus may have multiple different translation styles in different types of documents or different products, for example, the same corpus needs to be translated into different words in the same language in documents of different products or different document types, the demand matching type can be used to express the requirements of this corpus in the translation process. It can refer to the document types and product codes that may be involved in the application scenarios of this corpus in the process of matching the corpus. This demand matching type can facilitate the subsequent acquisition of the corpus used for corpus matching.

[0068] Step 102, acquiring a corpus that matches the required matching type according to the required matching type of the corpus to be matched;

[0069] Based on the determined requirement matching type, the target corpus is determined when matching the corpus. First, based on the requirement matching type of the corpus to be matched, a corpus that matches this type needs to be acquired so that the corpus that meets the requirement matching type can be matched with the matching corpus, thereby improving the efficiency of corpus matching while ensuring the accuracy of corpus matching.

[0070] The corpus that matches the required matching type is obtained, and each sentence pair it contains has multiple attribute settings. The fine-grained division of the corpus contained in the corpus realizes the full life cycle management and scenario-based application of the corpus quality. Specifically, different sentence pairs may have multiple different attribute settings. At this time, while ensuring the matching accuracy based on fine-grained attribute division, the corpus matching based on the minimum granularity, that is, based on sentence pairs with multiple different attribute settings, can achieve the maximum accuracy of corpus reuse, and at the same time, the corpus reuse can be maximized based on multi-dimensional reuse.

[0071] Among them, corpus can refer to a large-scale electronic text library that has been scientifically sampled and processed. As the basic resource for corpus linguistic research and the main resource for empirical language research methods, it needs to be processed to become a useful resource. Corpus has the dynamics of corpus and the circulation of corpus. The dynamics of corpus means that corpus is constantly supplemented dynamically. The circulation of corpus can refer to the "circulation" attribute of corpus, which can be an attribute with quantitative attribute value.

[0072] Corpus can be a pair of sentences in the source language and the target language, which is the basic unit of the corpus. For a corpus in a certain language, there may be a phrase or sentence in the source language corresponding to different phrases or sentences in the target language. Assume that the corpus to be translated needs to translate the Chinese corpus into the English corpus. As an example, for the Chinese word "configuration", it may have different English translations such as "Parameter", "Configurations", and "Remarks" in different application scenarios. At this time, it is necessary to set specific attributes for the sentence pairs.

[0073] Step 103: According to the multiple attribute settings of each sentence pair included in the corpus, a target corpus matching the corpus to be matched is obtained.

[0074] The target corpus for the corpus to be matched is determined based on the determined requirement matching type. After a corpus matching this type is obtained based on the requirement matching type of the corpus to be matched, the matching corpus can be used to match the corpus that meets the requirement matching type. Specifically, it can be manifested in that each sentence pair contained in the corpus has multiple attribute settings, and a target corpus matching the corpus to be matched is obtained, thereby completing the matching of the corpus.

[0075] In an embodiment of the present application, a corpus that matches the required matching type of the corpus to be matched is obtained, and each sentence pair contained in the obtained corpus has multiple attribute settings, so that a target corpus that matches the corpus to be matched is obtained based on the set multiple attribute matching. Based on the multiple attribute settings for each sentence pair in the corpus, while achieving corpus matching, the corpus in the corpus can also be divided into fine-grained parts, and the accuracy of corpus reuse can be maximized based on the corpus matching with the minimum granularity. At the same time, the corpus reuse can be maximized based on multi-dimensional reuse, which is convenient for improving the efficiency of corpus matching.

[0076] Reference Figure 2 , shows a flowchart of another embodiment of a corpus matching method of the present application, which may specifically include the following steps:

[0077] Step 201, setting multiple attributes for each sentence pair included in multiple corpora;

[0078] The required matching corpus may have multiple different translation styles in different types of documents or different products. For example, the same corpus needs to be translated into different words in the same language in different products or different documents. In this case, in order to maximize the corpus matching accuracy and corpus reuse rate, a multi-dimensional corpus matching logic can be proposed. Based on the multiple attribute settings for each sentence pair in the corpus, a fine-grained division of the corpus contained in the corpus can be achieved.

[0079] Specifically, since the required matching corpus may have different translation styles in different types of documents or different products, multiple attributes can be set based on the application scenarios and fields to which the sentence pairs are applicable. While making more fine-grained distinctions among the corpora in the same corpus, full life cycle management of corpus quality and scenario-based application can be achieved.

[0080] In a specific implementation, multiple corpora and sentence pairs contained in each corpus may be obtained to determine at least one application scenario involved in each sentence pair. At this time, multiple attributes may be set for each sentence pair based on information about the application scenario involved in each sentence pair. The multiple attributes set may include the product code Code involved in the sentence pair, the document type involved in the sentence pair, and the document topic ID Topic ID involved in the sentence pair.

[0081] In practical applications, refer to Figure 3 , which shows a schematic diagram of the relationship between the multiple attributes set in an embodiment of the present application. The same product code Code may have documents of multiple document types, and each document type may be composed of documents with multiple document subject identifiers Topic IDs.

[0082] Among them, the document subject identifier Topic ID can refer to the identifier of the unique Topic to which each sentence belongs, such as Topic ID: 1; the document type can refer to the type of product document, such as document type: product introduction, etc., the product code Code can refer to other tags for each sentence, which can contain multiple, such as code: sls, code: ecs, and the memory weight can refer to each TM (Translation Memory) group containing multiple memories, each memory has a certain weight, such as weight: 2. It should be noted that the document type can include user guides, acceptance guides, product introductions, quick start, security white papers, development guides, API (Application Programming Interface, which is a pre-defined interface (such as function, HTTP interface) or refers to the agreement for connecting different components of a software system) reference, SDK (Software Development Kit, Software Development Kit), technical white papers and operation and maintenance guides, etc., and this embodiment of the application is not limited to this.

[0083] Step 202: Based on the multiple attributes of each sentence pair configured according to each matching requirement type, a target corpus that meets the matching attribute range is matched.

[0084] When obtaining the corpus to be matched, the required matching type corresponding to the corpus to be matched can also be determined, so as to obtain a corpus that matches this type based on the required matching type of the corpus to be matched. After obtaining the corpus that matches this type based on the required matching type of the corpus to be matched, the matching corpus can be used to match the corpus that meets the required matching type. Specifically, it can be manifested as having multiple attribute settings according to each sentence pair contained in the corpus, obtaining a target corpus that matches the corpus to be matched, and completing the matching of the corpus.

[0085] Specifically, when obtaining a corpus that matches the required matching type, a main corpus and a reference corpus that are multi-layered and hierarchically set based on the required matching type can be obtained. When obtaining a corpus that matches the required matching type, the obtained corpus can be a corpus that is set up in a multi-layer hierarchical manner. These corpora can include not only corpora that are exclusive to different teams, but also other cross-team corpora that are shared in real time. For the determination of the main corpus and the reference corpus, the main corpus is not necessarily the exclusive corpus of a certain team. It can also use other cross-team corpora as the main corpus, and the exclusive corpus may not be in the corpus required for the current corpus matching. The selection of the main corpus and the reference corpus needs to be determined based on the required matching type of the corpus to be matched.

[0086] In actual applications, a corpus that matches the required matching type can be obtained, wherein the required matching type of the corpus to be matched may include a matching attribute range. At this time, the matching attribute range can be used to set the corpus that matches the required matching type, and the corpus after the matching attribute range is set can be divided into a main corpus and a reference corpus according to the priority of each corpus set.

[0087] In the specific implementation, refer to Figure 4 , showing a schematic diagram of corpus matching provided in an embodiment of the present application, which mainly performs automatic matching of the corpus to be matched from the corpus it has according to the corpus, priority, and attribute range configured according to each required matching type, to obtain the target corpus. Specifically, it can be manifested as matching the target corpus that meets the matching attribute range from the main corpus and the reference corpus based on the multiple attributes of each sentence pair contained in the acquired main corpus and the reference corpus, wherein the acquisition of the multiple attributes of each sentence pair contained in the main corpus and the reference corpus can be achieved through the configuration of the TM group, and the configured TM group can include the main TM group (i.e., the main corpus) and the reference TM group (i.e., the reference corpus).

[0088] like Figure 4As shown, when configuring the TM group, it can configure whether to use the TM group, select the main TM group, select the reference TM group, and configure the relevant weights, matching attributes, and the automatic import function of new sentence pairs.

[0089] During the matching process, if the acquired main corpus and / or reference corpus is provided with an automatic import function for its own newly added sentence pairs, the newly added sentence pairs can be automatically imported into the corresponding database provided with the automatic import function for newly added sentence pairs, and attribute settings can be added for the imported newly added sentence pairs.

[0090] Among them, with the delivery of new translation requirements, the newly generated corpus will be promptly recycled into the corpus, so that the corpus can promptly recycle the corpus generated in the translation process, and can match the matching corpus (which may refer to the content to be translated) according to the corpus in the library based on the new translation requirements, that is, the newly imported sentence pairs can be the corpus obtained in the historical matching process after deduplication.

[0091] The acquired sentence pairs include historically imported sentence pairs and newly imported sentence pairs in the corpus. The determination of the target corpus can be expressed as calculating the similarity between the historically imported sentence pairs and / or newly imported sentence pairs in the corpus and the corpus to be matched, so as to determine the target corpus based on the similarity.

[0092] Specifically, historically imported sentence pairs and / or newly imported sentence pairs that meet the matching attribute range in the main corpus and the reference corpus can be obtained, and the similarity between the historically imported sentence pairs and / or newly imported sentence pairs that meet the matching attribute range and the corpus to be matched can be calculated; at this time, if there are multiple sentence pairs with the same similarity to the corpus to be matched, then the similarity of the multiple sentence pairs with the same similarity can be weighted based on the multiple attributes set for the multiple sentence pairs with the same similarity. Finally, the weighted similarity can be used to sort the multiple sentence pairs with the same similarity from high to low, and the sentence pair with the highest similarity in the sorting can be determined as the target corpus to match the corpus to be matched.

[0093] Among them, when calculating the degree of similarity with the corpus to be matched, the original text information of the imported newly added sentence pairs and / or historically imported sentence pairs that meet the matching attribute range, as well as the sentences to be retrieved where the corpus to be matched is located, can be obtained to determine the text similarity between the original text information and the sentences to be retrieved where the corpus to be matched is located. When the calculated text similarity reaches a preset degree, the similarity between the corresponding sentence pairs and the corpus to be matched can also be increased based on the number of identical sentences in the original text information and the sentences to be retrieved.

[0094] In practical applications, the determined similarity with the corpus to be matched can be reflected based on the similarity score, that is, the higher the calculated similarity score is, the higher the similarity of the corpus is. Then, when determining the text similarity between the original information and the sentence to be retrieved where the corpus to be matched is located, it can also be determined by text similarity. Specifically, when calculating text similarity, an edit distance algorithm can be used, such as NormalizedTagSimilarity.score(sourceFragment, SearchUnitResult.getSourceFragment()), where NormalizedTagSimilarity.score() can refer to the determination of the similarity based on the distance algorithm, sourceFragment can refer to the sentence to be retrieved where the corpus to be matched is located, and SearchUnitResult.getSourceFragment() can refer to the original information of the newly imported sentence pairs and / or historical imported sentence pairs that meet the matching attribute range. It should be noted that the tags will be extracted in the pre-processing, and only the similarity (score) of the text will be calculated. If the tags are not the same, the similarity score calculated will be reduced.

[0095] When the calculated text similarity reaches a preset degree, it is assumed that the preset degree is determined based on a preset threshold, that is, when the text similarity score reaches the preset threshold, for example, 100, the context matching situation can be further calculated, and the similarity between the corresponding sentence pair and the corpus to be matched is increased based on the number of identical sentences existing in the original text information and the sentence to be retrieved. This can be expressed as if there is an identical sentence, then the text similarity score calculated corresponding to this sentence pair can be added points, for example, if there is an identical sentence, 1 point is added, and this is not limited in this embodiment of the present application.

[0096] If there are multiple sentence pairs with the same similarity to the corpus to be matched, then the similarity of the multiple sentence pairs with the same similarity can be weighted based on the multiple attributes set for the multiple sentence pairs with the same similarity, that is, through similarity calculation, if the scores are the same, the final ranking is further determined by weighted attributes. It is mainly based on the product code Code involved in the set sentence pair, the document type involved in the sentence pair, and the document topic ID Topic ID involved in the sentence pair. The specific weighting strategy is not limited in the embodiment of this application.

[0097] In a preferred embodiment, each sentence pair may have metadata information, which may record the detailed records of the update. The updated detailed records may include TM description, person who performs the update, update time, Topic ID, and product code, etc., to facilitate translators' reference during translation and maintenance of sentence pair quality.

[0098] As an example, suppose that after the corpus is updated, its record can be expressed as "The log audit service inherits all the functions of the existing log service, and also supports real-time automation and centralized collection of cloud product logs and auditing under multiple accounts, as well as storage, query and information aggregation required for auditing. For log audit-005, the log audit service covers basic (ActionTrail, container service Kubernetes version), storage (OSS (Object Storage Service, an object storage service), NAS (Network Attached Storage), network (SLB (Server Load Balancer, load balancing, refers to the distribution of access traffic to multiple backend cloud servers according to forwarding policies), API (Application Programming Interface) gateway, VPC (Virtual Private Cloud, a private cloud that exists in a shared or public cloud)), database (relational database RDS (Relational Database Service), cloud native distributed database;

[0099] You can use Log Audit Service to audit the logs that are collected from the following Cloud services:ActionTrail Container Service forKubernetes(ACK),Object Storage Service(OSS),Apsara File Storage NAS(NAS),Server Load Balancer(SL B),API Gateway Virtual Private Cloud(VPC),ApsaraDBRDS,

[0100] TM:customer.JCMS_Log Service_XX(Log Recorder).Tue Dec 0717:35:05CST2021_12658_Other

[0101] _1_1_67500409_sign_off

[0102] Updatedby.XX (refers to the person who performed the update)

[0103] At:2022-02-0714:33 (refers to the update time)

[0104] Activity.translation.proofreading.revision

[0105] Atribute: tenant_id 270001project_id 121050930user_id 53111808topic_nameLog Audit Service OverviewProduct code sls task_id 609821561topic_id 1885782order_id67500409client_name xxx”

[0106] Reference Figure 5 , shows an application scenario diagram of corpus matching provided in an embodiment of the present application, which can be applied to any TMS (translation management platform) or translation system CAT (computeraided translation).

[0107] Assuming that the corpus of the embodiment of the present application is matched on a certain translation management platform 501, the matching of the corpus can be expressed as the translation of a certain corpus. At this time, the corpus to be matched obtained can be the corpus to be translated. The corpus to be translated needs to translate the Chinese corpus into English corpus. Since the documents of each product have different translation styles and each type of document has a different translation style, a sentence of Chinese needs to be translated into different English in different products. Even in the documents of the same product, a sentence of Chinese needs to be translated into different English sentences in different documents. That is, at this time, there may be multiple English sentences corresponding to a certain Chinese sentence in the corpus.

[0108] For this situation, on the translation management platform 501, the translation management platform 501 may include a TM matching platform 11 and a self-made machine translation engine 12. The corpus to be translated may be the original manuscript to be translated. At this time, the file may be parsed through the TM matching platform 11 and the parsed file may be preprocessed. The preprocessing may be specifically manifested as TM matching. At this time, a higher file translation ratio may be achieved, such as 75%-92% similar automatic filling of existing translations. After TM matching, for sentences in the file that have no TM match, the HTER score of each sentence can be calculated and the workload can be converted through the self-made machine translation engine 12, so that the files pre-processed by the TM matching platform 11 can be MTPE processed based on the HTER score and the converted workload. MTPE can refer to the machine translation process and the post-editing process. Specifically, the files pre-processed by the TM matching platform 11 can be distributed to the clients of translation (Translation, which can be the "T" in the "MTPE" process), editing (Edit, which can be the "E" in the "MTPE" process) and proofreading (Proofreading, which can be the "P" in the "MTPE" process) personnel, so that the translation (T), editing (E) and proofreading (P) personnel can translate, edit and proofread the sentences that have no TM match after pre-processing, and then deliver the translated files to complete the translation.

[0109] The TM matching performed by the TM matching platform 11 can be specifically manifested as determining the required matching type of the parsed file, after which a corpus that matches the required matching type can be obtained so that the corpus that matches the required matching type can be used to match the corpus that matches the required matching type, thereby improving the efficiency of corpus matching while ensuring the accuracy of corpus matching. Specifically, in the matching process, this embodiment is manifested in that during the translation process, the TM matching platform 11 can automatically match according to the corpus, priority, and attribute range configured for each required type, such as Figure 4 As shown, it is specifically manifested as matching the target corpus that meets the matching attribute range from the main corpus and the reference corpus based on the multiple attributes of each sentence pair contained in the acquired main corpus and the reference corpus, wherein the acquired corpus that matches the required matching type has multiple attribute settings for each sentence pair contained therein, and the fine-grained division of the corpus contained in the corpus realizes the full life cycle management of corpus quality and application in different scenarios. Different sentence pairs may have multiple different attribute settings. At this time, while ensuring the matching accuracy based on fine-grained attribute division, the corpus matching based on the minimum granularity, that is, based on sentence pairs with multiple different attribute settings, can achieve the maximum accuracy of corpus reuse, and at the same time, the corpus reuse can be maximized based on multi-dimensional reuse.

[0110] In an embodiment of the present application, a corpus that matches the required matching type of the corpus to be matched is obtained, and each sentence pair contained in the obtained corpus has multiple attribute settings, so that a target corpus that matches the corpus to be matched is obtained based on the set multiple attribute matching. Based on the multiple attribute settings for each sentence pair in the corpus, while achieving corpus matching, the corpus in the corpus can also be divided into fine-grained parts, and the accuracy of corpus reuse can be maximized based on the corpus matching with the minimum granularity. At the same time, the corpus reuse can be maximized based on multi-dimensional reuse, which is convenient for improving the efficiency of corpus matching.

[0111] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0112] Reference Figure 6 , shows a structural block diagram of an embodiment of a corpus matching device of the present application, which may specifically include the following modules:

[0113] The to-be-matched corpus acquisition module 601 is used to acquire the to-be-matched corpus and the required matching type of the to-be-matched corpus;

[0114] The corpus acquisition module 602 is used to acquire a corpus that matches the required matching type according to the required matching type of the corpus to be matched; each sentence pair included in the corpus has multiple attribute settings;

[0115] The target corpus matching module 603 is used to obtain a target corpus that matches the corpus to be matched according to the multiple attribute settings of each sentence pair included in the corpus.

[0116] In one embodiment of the present application, the device further includes:

[0117] A sentence pair acquisition module, used to acquire multiple corpora and sentence pairs contained in each corpus, and determine at least one application scenario involved in each sentence pair;

[0118] The attribute setting module is used to set multiple attributes for each sentence pair based on the information of the application scenario involved in each sentence pair; the multiple attributes set include the product code involved in the sentence pair, the document type involved in the sentence pair and the document subject identifier involved in the sentence pair; wherein, there are documents of multiple document types under the same product code, and each document type is composed of documents with multiple document subject identifiers.

[0119] In one embodiment of the present application, the corpus acquisition module 602 may include the following submodules:

[0120] A hierarchical corpus acquisition submodule is used to acquire a main corpus and a reference corpus that are hierarchically arranged in multiple layers based on the required matching type;

[0121] Among them, the hierarchical corpus acquisition submodule may include the following units:

[0122] A corpus acquisition unit, configured to acquire a corpus that matches the required matching type, wherein the required matching type includes a matching attribute range;

[0123] The corpus grading unit is used to set the corpus that matches the required matching type by using the matching attribute range, and divide the corpus after the matching attribute range is set into a main corpus and a reference corpus according to the priority of each set corpus.

[0124] In one embodiment of the present application, the target corpus matching module 603 may include the following submodules:

[0125] The target corpus screening submodule is used to match the target corpus that meets the matching attribute range from the main corpus and the reference corpus based on the multiple attributes of each sentence pair contained in the acquired main corpus and the reference corpus.

[0126] In one embodiment of the present application, the target corpus matching module 603 may further include the following submodules:

[0127] The newly added sentence pair importing submodule is used to automatically import the newly added sentence pairs into the corresponding database with the newly added sentence pair automatic importing function during the matching process, if the acquired main corpus and / or reference corpus is provided with the newly added sentence pair automatic importing function for itself, and add attribute settings for the imported newly added sentence pairs; wherein the imported newly added sentence pairs are the corpus obtained in the historical matching process and saved after deduplication.

[0128] In one embodiment of the present application, the sentence pairs contained in the acquired corpus include historically imported sentence pairs and newly imported sentence pairs in the corpus; the target corpus screening submodule may include the following units:

[0129] A newly added sentence pair acquisition unit is used to acquire historically imported sentence pairs and / or imported newly added sentence pairs that meet the matching attribute range in the main corpus and the reference corpus;

[0130] A similarity calculation unit, used to calculate the similarity between the historical imported sentence pairs and / or the imported newly added sentence pairs that meet the matching attribute range and the corpus to be matched;

[0131] A weighting unit, configured to weight the similarity of the plurality of sentence pairs with the same similarity, respectively, based on multiple attributes set for the plurality of sentence pairs with the same similarity, when there are a plurality of sentence pairs with the same similarity.

[0132] The sorting unit is used to sort the plurality of sentence pairs with the same similarity from high to low using the weighted similarity, and determine the sentence pair with the highest similarity in the sorting as the target corpus to be matched with the corpus to be matched.

[0133] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0134] The present application also provides an electronic device, including:

[0135] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, each process of the above-mentioned corpus matching method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0136] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned corpus matching method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0137] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0138] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0142] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0143] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0144] The above is a detailed introduction to a corpus matching method, a corresponding corpus matching device, a corresponding electronic device and a computer storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A corpus matching method, characterized in that: The method comprises: Obtaining a corpus to be matched and a required matching type of the corpus to be matched; According to the required matching type of the corpus to be matched, a corpus matching the required matching type is obtained; the required matching type includes a matching attribute range, the corpus includes a main corpus and a reference corpus obtained by dividing the corpus after setting the matching attribute range according to the priority of each corpus set, and each sentence pair contained in the corpus has multiple attribute settings; According to the multiple attribute settings for each sentence pair contained in the corpus, a target corpus matching the corpus to be matched is obtained; the multiple attributes set include the product code involved in the sentence pair, the document type involved in the sentence pair, and the document subject identifier involved in the sentence pair, wherein there are documents of multiple document types under the same product code, and documents of each document type are composed of documents with multiple document subject identifiers.

2. The method according to claim 1, characterized in that The method further comprises: Acquire multiple corpora and sentence pairs contained in each corpus, and determine at least one application scenario involved in each sentence pair; Based on the information of the application scenario involved in each sentence pair, multiple attributes are set for each sentence pair.

3. The method according to claim 1, characterized in that The step of acquiring a corpus that matches the required matching type according to the required matching type of the corpus to be matched includes: Acquire a main corpus and a reference corpus that are multi-layer hierarchically arranged based on the required matching type; The step of obtaining a main corpus and a reference corpus that are hierarchically arranged in multiple layers based on the requirement matching type includes: Acquire a corpus that matches the required matching type; The matching attribute range is used to set a corpus that matches the required matching type, and the corpus after the matching attribute range is set is divided into a main corpus and a reference corpus according to the priority of each corpus set.

4. The method according to claim 3, characterized in that The step of obtaining a target corpus matching the corpus to be matched according to multiple attribute settings for each sentence pair included in the corpus includes: Based on the multiple attributes of each sentence pair contained in the acquired main corpus and reference corpus, a target corpus that meets the matching attribute range is obtained from the main corpus and the reference corpus.

5. The method according to claim 4, characterized in that Also includes: During the matching process, if the acquired main corpus and / or reference corpus is provided with an automatic import function for new sentence pairs, the new sentence pairs are automatically imported into the corresponding database provided with the automatic import function for new sentence pairs, and attribute settings are added for the imported new sentence pairs; wherein the imported new sentence pairs are the corpus obtained in the historical matching process and saved after deduplication.

6. The method according to claim 4 or 5, characterized in that: The sentence pairs contained in the acquired corpus include historically introduced sentence pairs in the corpus and the introduced newly added sentence pairs; The method of matching the target corpus that meets the matching attribute range from the main corpus and the reference corpus based on the multiple attributes of each sentence pair contained in the obtained main corpus and the reference corpus includes: Obtaining historically imported sentence pairs and / or imported newly added sentence pairs that meet the matching attribute range in the main corpus and the reference corpus; Calculating the similarity between the historical imported sentence pairs and / or the imported newly added sentence pairs that meet the matching attribute range and the corpus to be matched; If there are multiple sentence pairs with the same similarity to the corpus to be matched, then based on the multiple attributes set for the multiple sentence pairs with the same similarity, the similarity of the multiple sentence pairs with the same similarity is weighted respectively; The weighted similarity is used to sort the multiple sentence pairs with the same similarity from high to low, and the sentence pair with the highest similarity in the sorting is determined as the target corpus to be matched with the corpus to be matched.

7. The method according to claim 6, characterized in that The calculating the similarity between the text and the corpus to be matched includes: Acquire the original text information of the newly imported sentence pairs and / or the historically imported sentence pairs that meet the matching attribute range, and the sentences to be retrieved where the corpus to be matched is located; Determine the text similarity between the original text information and the sentence to be retrieved where the corpus to be matched is located; When the calculated text similarity reaches a preset degree, the similarity between the corresponding sentence pair and the corpus to be matched is increased based on the number of identical sentences existing in the original text information and the sentences to be retrieved.

8. A corpus matching device, characterized in that: The device comprises: A to-be-matched corpus acquisition module is used to acquire the to-be-matched corpus and the required matching type of the to-be-matched corpus; A corpus acquisition module, used for acquiring a corpus that matches the required matching type according to the required matching type of the corpus to be matched; the required matching type includes a matching attribute range, the corpus includes a main corpus and a reference corpus obtained by dividing the corpus after setting the matching attribute range according to the priority of each corpus, and each sentence pair contained in the corpus has multiple attribute settings; The target corpus matching module is used to obtain a target corpus that matches the corpus to be matched according to multiple attribute settings for each sentence pair contained in the corpus; the multiple attributes set include the product code involved in the sentence pair, the document type involved in the sentence pair, and the document subject identifier involved in the sentence pair, wherein there are documents of multiple document types under the same product code, and each document type of document is composed of documents with multiple document subject identifiers.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the corpus matching method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the corpus matching method according to any one of claims 1 to 7 are implemented.

Images