A text processing method and device

Abstract

This application provides a text processing method and apparatus. When a large amount of text belonging to the same domain is needed, technicians are no longer required to manually sift through a text library containing texts from various domains. Instead, technicians can manually collect a small number of texts from that domain and submit them as seed texts to an electronic device. The electronic device can then select texts from the text library as the text to be recognized. It then automatically obtains a first text block from the text to be recognized based on an N-gram language model. The first text block includes N adjacent words, where N is an integer greater than 1. The frequency of the first text block in the seed text is then calculated, and the frequency is used to determine whether the text to be recognized and the seed text belong to the same domain. This method automatically filters a large amount of text from the text library, reducing manual workload and thus labor costs.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a text processing method and device. BACKGROUND

[0002] Machine translation and other artificial intelligence algorithms based on neural networks have been widely used in natural language translation and other related fields due to their strong fitting ability and end-to-end global optimization ability.

[0003] With the deepening of trade internationalization, machine translation is needed in many fields, such as e-commerce, medical, news, automotive, education, and transportation, and machine translation plays an increasingly important role.

[0004] In order to improve the accuracy of machine translation, it is often necessary to train a translation model corresponding to each field, so that when a text needs to be translated, the field to which the text belongs can be determined first, and then the text can be translated based on the translation model corresponding to the field.

[0005] Among them, for any field, the generalization ability of the translation model corresponding to the field depends on the amount of bilingual parallel corpus data prepared for the field. The larger the amount of bilingual parallel corpus data collected for the field, the higher the generalization ability of the translation model corresponding to the field.

[0006] Therefore, before training the translation model corresponding to the field, a large amount of bilingual parallel corpus for the field needs to be prepared, but in the prior art, the staff needs to manually screen the bilingual parallel corpus for the field from the massive bilingual parallel corpus.

[0007] However, manually screening the bilingual parallel corpus for the field from the massive bilingual parallel corpus is low in efficiency and requires a large amount of manual work, resulting in high labor costs. SUMMARY

[0008] In order to reduce labor costs, the present application shows a text processing method and device.

[0009] In a first aspect, the present application shows a text processing method, which comprises:

[0010] Obtaining a text to be recognized;

[0011] Obtaining a first text block in the text to be recognized based on an N-gram language model, the first text block comprising N adjacent words, N being an integer greater than 1;

[0012] Obtaining the number of occurrences of the first text block in a seed text;

[0013] determining whether the to-be-identified text and the seed text belong to the same field according to the occurrence number.

[0014] In an optional implementation, the obtaining the to-be-identified text comprises:

[0015] selecting any one text in the first text library as the to-be-identified text;

[0016] or,

[0017] obtaining a text related to the semantic of the seed text in the first text library as the to-be-identified text.

[0018] In an optional implementation, the obtaining a text related to the semantic of the seed text in the first text library comprises:

[0019] generating a search query word according to the semantic of the seed text;

[0020] searching in the first text library according to the search query word;

[0021] obtaining a text related to the semantic of the seed text.

[0022] In an optional implementation, the obtaining the occurrence number of the first text block in the seed text comprises:

[0023] querying a known frequency dictionary to obtain the occurrence number of the first text block in the seed text, wherein the frequency dictionary stores a corresponding relationship between a text block and an occurrence number of the text block in the seed text.

[0024] In an optional implementation, the frequency dictionary is obtained in the following manner:

[0025] obtaining a second text block in the seed text based on an N-gram language model, the second text block comprising N adjacent words;

[0026] obtaining an occurrence number of the second text block in the seed text;

[0027] establishing a corresponding relationship between the second text block and the occurrence number of the second text block in the seed text in the frequency dictionary.

[0028] In an optional implementation, the determining whether the to-be-identified text and the seed text belong to the same field according to the occurrence number comprises:

[0029] summing occurrence numbers of different first text blocks in the seed text to obtain a total number;

[0030] determining whether the text to be identified and the seed text belong to the same field according to the total number.

[0031] In an optional implementation, the determining whether the text to be identified and the seed text belong to the same field according to the total number includes:

[0032] In a case where the total number is greater than or equal to a first preset threshold, it is determined that the seed text and the text to be identified belong to the same field.

[0033] In a case where the total number is less than the first preset threshold, it is determined that the seed text and the text to be identified do not belong to the same field.

[0034] In an optional implementation, the determining whether the text to be identified and the seed text belong to the same field according to the total number includes:

[0035] obtaining a number of words included in the text to be identified;

[0036] calculating a ratio between the total number and the number;

[0037] In a case where the ratio is greater than or equal to a second preset threshold, it is determined that the seed text and the text to be identified belong to the same field.

[0038] In a case where the ratio is less than the second preset threshold, it is determined that the seed text and the text to be identified do not belong to the same field.

[0039] In an optional implementation, the method further includes:

[0040] in a case where the seed text and the text to be identified belong to the same field, obtaining a translated text corresponding to the text to be identified;

[0041] storing the text to be identified and the translated text in a second text library corresponding to the field.

[0042] In an optional implementation, the method further includes:

[0043] training a field classification model corresponding to the field according to texts in the second text library;

[0044] screening texts belonging to the same field as the seed text from a first text library based on the field classification model;

[0045] obtaining a translated text corresponding to the screened texts;

[0046] storing the screened texts and the translated text corresponding to the screened texts in the second text library.

[0047] In an optional implementation, the method further includes:

[0048] A text translation model corresponding to the domain is trained based on texts from the second text library.

[0049] Secondly, this application discloses a text processing apparatus, the apparatus comprising:

[0050] The first acquisition module is used to acquire the text to be recognized;

[0051] The second acquisition module is used to obtain a first text block in the text to be identified based on the N-gram language model. The first text block includes N adjacent words, where N is an integer greater than 1.

[0052] The third acquisition module is used to acquire the number of times the first text block appears in the seed text;

[0053] The determination module is used to determine whether the text to be identified and the seed text belong to the same domain based on the number of occurrences.

[0054] In an optional implementation, the first acquisition module includes:

[0055] The selection unit is used to select any text from the first text library and use it as the text to be recognized.

[0056] or,

[0057] The search unit is used to obtain texts that are semantically related to the seed text in the first text library and use them as the text to be identified.

[0058] In one optional implementation, the search unit includes:

[0059] A subunit is generated to generate search query terms based on the semantics of the seed text;

[0060] A retrieval subunit is used to perform a retrieval in the first text database based on the retrieval query terms;

[0061] Obtain a subunit for obtaining text that is semantically related to the seed text.

[0062] In one optional implementation, the third acquisition module includes:

[0063] The lookup unit is used to query a known frequency dictionary to obtain the number of times the first text block appears in the seed text, wherein the frequency dictionary stores the following correspondence: text block and the number of times the text block appears in the seed text.

[0064] In an optional implementation, the third acquisition module further includes:

[0065] The first acquisition unit is used to obtain a second text block from the seed text based on the N-gram language model, the second text block comprising N adjacent words;

[0066] The second acquisition unit is used to acquire the number of times the second text block appears in the seed text;

[0067] A storage unit is used in the frequency dictionary to establish a correspondence between the second text block and the number of times the second text block appears in the seed text.

[0068] In one optional implementation, the determining module includes:

[0069] The summation unit is used to sum the occurrence counts of different first text blocks in the seed text to obtain the total count;

[0070] The determining unit is used to determine whether the text to be identified and the seed text belong to the same domain based on the total number of times.

[0071] In one optional implementation, the determining unit includes:

[0072] The first determining subunit is used to determine that the seed text and the text to be identified belong to the same domain when the total number of times is greater than or equal to a first preset threshold.

[0073] The second determining subunit is used to determine that the seed text and the text to be identified do not belong to the same domain when the total number of times is less than a first preset threshold.

[0074] In one optional implementation, the determining unit includes:

[0075] A sub-unit is used to obtain the number of words included in the text to be identified;

[0076] A calculation subunit is used to calculate the ratio between the total number of times and the quantity;

[0077] The third determining subunit is used to determine that the seed text and the text to be identified belong to the same domain when the ratio is greater than or equal to the second preset threshold.

[0078] The fourth determining subunit is used to determine that the seed text and the text to be identified do not belong to the same domain when the ratio is less than the second preset threshold.

[0079] In an alternative implementation, the apparatus further includes:

[0080] The fourth acquisition module is used to acquire the translated text corresponding to the text to be identified when the seed text and the text to be identified belong to the same domain.

[0081] The first storage module is used to store the text to be identified and the translated text in a second text library corresponding to the domain.

[0082] In an alternative implementation, the apparatus further includes:

[0083] The first training module is used to train a domain classification model corresponding to the domain based on the text in the second text library;

[0084] The filtering module is used to filter texts belonging to the same domain as the seed text in the first text library based on the domain classification model.

[0085] The fifth module is used to obtain the translated text corresponding to the filtered text;

[0086] The second storage module is used to store the filtered text and its corresponding translation text in the second text library.

[0087] In an alternative implementation, the apparatus further includes:

[0088] The second training module is used to train a text translation model corresponding to the domain based on the text in the second text library.

[0089] Thirdly, this application discloses an electronic device comprising:

[0090] processor; and

[0091] A memory having executable code stored thereon, which, when executed, causes the processor to perform the text processing method as described in the first aspect.

[0092] Fourthly, this application discloses one or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the text processing method as described in the first aspect.

[0093] Compared with the prior art, the embodiments of this application have the following advantages:

[0094] In situations where a large amount of text belonging to the same domain is needed, such as when training a text translation model for that domain, technicians do not need to manually sift through a large number of texts from various domains. Instead, technicians can manually collect a small number of texts from that domain and submit them as seed texts to the electronic device. This allows the electronic device to select texts from the text library as the text to be recognized. Then, it automatically obtains the first text block from the text to be recognized based on the N-gram language model. The first text block includes N adjacent words. The frequency of the first text block in the seed text is obtained, and the frequency of occurrence is used to determine whether the text to be recognized and the seed text belong to the same domain.

[0095] Using the above method, a large number of texts in this field can be automatically filtered out from the text library. Since technicians do not need to manually filter a large number of texts in this field from the text library which includes texts from various fields, technicians can manually collect a small number of texts in this field, thereby reducing the workload and thus reducing labor costs. Attached Figure Description

[0096] Figure 1 This is a flowchart illustrating a text processing method according to an exemplary embodiment of this application.

[0097] Figure 2 This is a structural block diagram of a text processing apparatus illustrated in an exemplary embodiment of this application.

[0098] Figure 3 This is a schematic diagram of the structure of an apparatus provided in an exemplary embodiment of this application. Detailed Implementation

[0099] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0100] Reference Figure 1 The diagram illustrates a text processing method according to this application. This method is applied in an electronic device and may include:

[0101] In step S101, the text to be recognized is obtained;

[0102] In this application, for any given field, at least one text in a certain language of that field can be collected and used as a seed text. The seed text can be collected manually by a technician or collected by an electronic device through other means.

[0103] The electronic device can then filter texts from a massive amount of text that belong to the same field as the seed text, and the language of the filtered texts can be the same as that of the seed text.

[0104] Massive amounts of text can include text from the internet, such as text on web pages, or subtitles, bullet comments, and reviews in online videos.

[0105] For example, in one instance, an electronic device can automatically collect at least a portion of the text from the network beforehand, and then store the collected text in a first text library.

[0106] In this way, electronic devices can filter texts from the first text library that belong to the same domain as the seed text.

[0107] In this application, any text can be selected from the first text library as the text to be identified. Then, following steps S102 to S104, it is determined whether the text to be identified and the seed text belong to the same domain. Next, an unselected text is selected from the first text library as the text to be identified, and the process of following steps S102 to S104 continues to determine whether the newly selected text to be identified and the seed text belong to the same domain. This continues until it is determined whether each text in the first text library belongs to the same domain as the seed text, thereby achieving the goal of filtering out texts from the first text library that belong to the same domain as the seed text.

[0108] In step S102, a first text block is obtained from the text to be identified based on the N-gram language model. The first text block includes N adjacent words, where N is an integer greater than 1.

[0109] In this application, the text to be identified can be segmented into words to obtain multiple words in the text.

[0110] In one example, when the language of the text to be identified is a Latin-based language such as English, each word in the text to be identified can be treated as a word, and then based on the N-gram language model, N adjacent words in the text to be identified can be combined into a text block, and the resulting text block is used as the first text block.

[0111] In one example, if the language of the text to be identified is Chinese, single characters that do not form words can be treated as words, or words composed of at least two adjacent characters can be treated as words.

[0112] For example, suppose the text to be recognized includes: "Car engine is an important parameter", which includes the words: "car", "engine", "is", "important" and "parameter".

[0113] The first text block obtained based on the 2-gram language model for "car engine is an important parameter" includes two adjacent words in the text to be identified, such as "car engine", "engine is", "is important" and "important parameter".

[0114] The first text block obtained based on the 3-gram language model for "the car engine is an important parameter" includes three adjacent words in the text to be identified, such as "the car engine is", "the engine is important", and "is an important parameter".

[0115] In step S103, the number of times the first text block appears in the seed text is obtained;

[0116] In one embodiment of this application, for any first text block, the first text block can be searched for in real time in the seed text, and the number of the first text blocks found can be counted and used as the number of times the first text block appears in the seed text. The same applies to each other first text block.

[0117] Of course, other methods can also be used to obtain the number of times the first text block appears in the seed text, as detailed in the following examples, which will not be elaborated here.

[0118] In step S104, the frequency of occurrence is used to determine whether the text to be identified and the seed text belong to the same domain.

[0119] For details of this step, please refer to the following embodiments, which will not be described in detail here.

[0120] In situations where a large amount of text belonging to the same domain is needed, such as when training a text translation model for that domain, technicians do not need to manually sift through a large number of texts from various domains. Instead, technicians can manually collect a small number of texts from that domain and submit them as seed texts to the electronic device. This allows the electronic device to select texts from the text library as the text to be recognized. Then, it automatically obtains the first text block from the text to be recognized based on the N-gram language model. The first text block includes N adjacent words, where N is an integer greater than 1. The frequency of the first text block in the seed text is obtained, and the frequency of occurrence is used to determine whether the text to be recognized and the seed text belong to the same domain.

[0121] Using the above method, a large number of texts in this field can be automatically filtered out from the text library. Since technicians do not need to manually filter a large number of texts in this field from the text library which includes texts from various fields, technicians can manually collect a small number of texts in this field, thereby reducing the workload and thus reducing labor costs.

[0122] In step S101, a text can be selected from the first text library and used as the text to be identified. For example, a text can be randomly selected from the first text library (any text can be selected), or a text can be selected from the first text library in a specific order and used as the text to be identified. Then, the number of times the first text block corresponding to the text to be identified appears in the seed text determines whether the text to be identified and the seed text belong to the same domain.

[0123] However, the inventors discovered through research that, in reality, when two texts belong to the same domain, their semantics are often related; when two texts belong to different domains, their semantics are often unrelated.

[0124] Thus, if the semantics of two texts are unrelated, the two texts often do not belong to the same domain; only if the semantics of two texts are related can the two texts belong to the same domain.

[0125] Therefore, in order to further improve the accuracy of determining whether the text to be identified and the seed text belong to the same domain, in addition to considering the number of times the first text block corresponding to the text to be identified appears in the seed text, the semantics of the text to be identified and the semantics of the seed text can also be combined.

[0126] For example, in this application, texts that may belong to the same domain as the seed text can be initially screened in the first text library based on the semantics of the seed text, and then the process of steps S102 to S104 is used to determine whether the initially screened texts belong to the same domain as the seed text.

[0127] In the process of initially filtering texts that may belong to the same domain as the seed text in the first text library based on the semantics of the seed text, texts whose semantics are related to the seed text can be obtained in the first text library and used as texts to be identified.

[0128] Specifically, search query terms can be generated based on the semantics of the seed text. For example, keywords that best reflect the semantics of the seed text can be selected based on semantic analysis technology and used as search query terms. Then, the first text database can be searched based on the search query terms to obtain texts that are semantically related to the seed text, such as searching for texts that are semantically related to the search query terms.

[0129] For example, for any text in the first text library, the keywords that best represent the semantics of the text can be selected based on semantic analysis technology. Then, it can be determined whether the semantics of the keywords are related to the semantics of the search query. If they are related, it means that the semantics of the text are related to the semantics of the seed text.

[0130] In determining whether two words are semantically related, the semantic relevance between the two words can be obtained. If the semantic relevance is greater than or equal to the preset relevance, the two words can be determined to be semantically related. If the semantic relevance is less than the preset relevance, the two words can be determined to be semantically unrelated.

[0131] This application does not limit the specific method for obtaining the semantic relevance between two words; any method for obtaining semantic relevance in the prior art can be used.

[0132] Technicians can set the preset correlation degree according to the actual situation. This application does not limit the specific value of the preset correlation degree. For example, it can be 0.6, 0.7 or 0.8, etc.

[0133] In one approach, the semantic relevance between two words can be obtained based on DSSM (Deep Structured Semantic Models).

[0134] By combining the frequency of occurrence of the first text block corresponding to the text to be identified in the seed text, the semantics of the text to be identified, and the semantics of the seed text, we can determine whether the text to be identified and the seed text belong to the same domain, which can improve the accuracy of the determination results.

[0135] In one embodiment of this application, the electronic device begins to execute step S101, then step S102, and then step S103, etc., after the seed text has been collected.

[0136] However, when performing step S103: obtaining the occurrence count of the first text block in the seed text, in one approach, the electronic device can obtain the second text block in the seed text in real time based on the N-gram language model. The second text block includes N adjacent words, where N is an integer greater than 1. Then, for any second text block, the occurrence count of that second text block in the seed text can be obtained. Subsequently, a correspondence between the second text block and its occurrence count in the seed text can be established in a frequency dictionary. The same operation is performed for each other second text block. Then, the occurrence count of each first text block in the seed text can be obtained by querying a known frequency dictionary, where the frequency dictionary stores the following correspondence: text block and the occurrence count of the text block in the seed text.

[0137] However, the frequency dictionary creation process described above, which involves "obtaining a second text block from the seed text based on an N-gram language model, where the second text block includes N adjacent words (N being an integer greater than 1), then obtaining the frequency count of any second text block in the seed text, and subsequently establishing a correspondence between the second text block and its frequency count in the seed text using a frequency dictionary. The same operation is performed for each other second text block," takes a considerable amount of time. Furthermore, because the frequency dictionary creation process must be completed before the known frequency dictionary can be queried to obtain the frequency count of each first text block in the seed text, step S103 becomes time-consuming, thus reducing the efficiency of determining whether the text to be identified and the seed text belong to the same domain.

[0138] Therefore, to improve the efficiency of determining whether the text to be identified and the seed text belong to the same domain, it is necessary to reduce the time required for step S103. To reduce the time required for step S103, in another embodiment of this application, after collecting the seed text, the electronic device can begin executing step S101, and in parallel, the electronic device can also execute the frequency dictionary creation process. This ensures that when the electronic device begins executing step S101, it has also simultaneously begun executing the frequency dictionary creation process.

[0139] Thus, when the electronic device executes step S103, it has already completed at least part of the frequency dictionary creation process. The electronic device only needs to wait for the other parts of the frequency dictionary creation process to complete before it can query the known frequency dictionary to obtain the occurrence count of each first text block in the seed text. Compared to waiting for the frequency dictionary creation process in the aforementioned embodiment to complete before querying the known frequency dictionary to obtain the occurrence count of each first text block in the seed text, this embodiment can wait less time after completing step S102 before querying the known frequency dictionary to obtain the occurrence count of each first text block in the seed text. This reduces the time required for step S103 and improves the efficiency of determining whether the text to be identified and the seed text belong to the same domain.

[0140] In one embodiment of this application, in step S104, the number of occurrences of different first text blocks in the seed text can be summed to obtain the total number of occurrences, and then the text to be identified and the seed text can be determined to belong to the same domain based on the total number of occurrences.

[0141] For example, in one embodiment, if each first text block in the text to be identified appears frequently in the seed text, the text to be identified and the seed text often belong to the same domain.

[0142] When the first text blocks in the text to be identified appear less frequently in the seed text, the text to be identified and the seed text often do not belong to the same domain.

[0143] Furthermore, the more frequently each first text block in the text to be identified appears in the seed text, the greater the likelihood that the text to be identified belongs to the same domain as the seed text. Conversely, the fewer frequently each first text block in the text to be identified appears in the seed text, the less likely the text to be in the same domain as the seed text.

[0144] Technicians can set a first preset threshold in advance based on the actual situation and experience. In this embodiment, the electronic device can determine whether the total number of times is greater than or equal to the first preset threshold. If the total number of times is greater than or equal to the first preset threshold, it can be determined that the seed text and the text to be identified belong to the same domain. If the total number of times is less than the first preset threshold, it can be determined that the seed text and the text to be identified do not belong to the same domain.

[0145] However, sometimes the first preset threshold might be at a low level. Consider this scenario: the text to be identified and the seed text don't actually belong to the same domain, but the text to be identified contains many words. In this case, the N-gram language model will obtain a large number of first text blocks in the text to be identified. Thus, the total number of occurrences of different first text blocks in the seed text might exceed the first preset threshold. Following the above method, it would be determined that the seed text and the text to be identified belong to the same domain. However, in reality, the text to be identified and the seed text do not belong to the same domain, leading to an incorrect determination.

[0146] Of course, sometimes it is possible that the text to be identified and the seed text actually belong to the same domain, but it is determined that the text to be identified and the seed text do not belong to the same domain.

[0147] To avoid the two aforementioned incorrect determination results, in another embodiment of this application, when determining whether the text to be identified and the seed text belong to the same domain based on the total number of occurrences, the number of words included in the text to be identified can be obtained, and then the ratio between the total number of occurrences and the number of occurrences can be calculated. This normalizes the total number of occurrences obtained by summing the occurrences of different first text blocks in the seed text. In this way, even if different texts to be identified include different numbers of words, or even if the number of words included in different texts to be identified varies greatly, the total number of occurrences corresponding to each text to be identified can be normalized. Then, it is determined whether the ratio is greater than or equal to a second preset threshold. If the ratio is greater than or equal to the second preset threshold, it can be determined that the seed text and the text to be identified belong to the same domain; if the ratio is less than the second preset threshold, it can be determined that the seed text and the text to be identified do not belong to the same domain, thereby avoiding determination errors.

[0148] The second preset threshold can be set in advance by technicians based on actual conditions and experience.

[0149] One objective of this application is to train a text translation model corresponding to a given domain based on multiple texts belonging to the same domain. However, bilingual corpora are required when training a text translation model. Therefore, in another embodiment of this application, when the seed text and the text to be identified belong to the same domain, the translated text corresponding to the text to be identified can be obtained. Then, the text to be identified and the translated text are stored in a second text library corresponding to that domain to classify the texts belonging to that domain. After that, if the number of texts in the second text library meets the requirements, the text translation model corresponding to that domain can be trained based on the texts in the second text library.

[0150] The text translation model in this application is for translating text in one language into text in another language. The seed text and the text to be identified both belong to the first language, while the translated text corresponding to the text to be identified can belong to the second language.

[0151] In this application, the text to be identified is a text that belongs to the same domain as the seed text, which is selected by the electronic device from a first text library based on the seed text, and the language of the text to be identified is the same as that of the seed text.

[0152] In addition to the text to be identified, the first text library may also include the translation text corresponding to the text to be identified. Thus, when obtaining the translation text corresponding to the text to be identified, the translation text corresponding to the text to be identified can be obtained from the first text library.

[0153] The first text library includes at least one correspondence between text in one language and its translated text in another language.

[0154] For any entry in the correspondence table, the text in one language and its corresponding translation in another language can be collected in advance from the internet. For example, some online platforms provide bilingual news, bilingual novels, bilingual essays, and bilingual product introductions. Therefore, the text in one language and its corresponding translation in another language can be collected directly without the need for manual translation by technical personnel, thus reducing the burden on technical personnel.

[0155] There are many kinds of languages ​​in reality, such as Chinese, English, Russian, French, Japanese, German and Korean. When translating text, it is usually to translate text in one language into text in another language.

[0156] Therefore, for any two different languages, the first text database can include the correspondence between the text in one language and its translated text in the other language. The same applies to every other pair of different languages. Thus, the first text database can contain the translated text in another language corresponding to the text in any other language.

[0157] If the number of texts in the second text library does not reach the first preset number, the generalization level of the text translation model trained using the texts in the second text library may not meet the requirements.

[0158] Therefore, in order for the generalization of the trained text translation model in this domain to meet the requirements, the number of texts in the second text library needs to be greater than or equal to the first preset number. That is, it is necessary to continue to filter the texts in this domain and their corresponding translation texts from the first text library.

[0159] In one embodiment, based on the identified texts belonging to the same domain as the seed text, texts belonging to the same domain as the text to be identified can be filtered from the first text library. For details of the filtering process, please refer to... Figure 1 The embodiments shown are not described in detail here.

[0160] Alternatively, in another embodiment, texts belonging to the same domain as the text to be identified can be filtered from a first text library based on a text classification model.

[0161] Specifically, although the number of texts in the second text library is less than the first preset number, if the number of texts in the second text library is greater than the second preset number, a domain classification model corresponding to that domain can be trained based on the texts in the second text library. For example, an initial model can be trained based on the texts in the second text library until the parameters in the initial model converge, thus obtaining a text classification model. The initial model includes models such as Support Vector Machines, k-Nearest Neighbor models, and Multiple-Bernoulli models.

[0162] Then, based on a domain classification model, texts belonging to the same domain as the seed text can be selected from the first text library. The translated texts corresponding to the selected texts are then obtained, and the selected texts and their corresponding translated texts are stored in the second text library. This process continues until the number of texts in the second text library is greater than or equal to a first preset number. At this point, a text translation model for that domain can be trained based on the texts in the second text library.

[0163] In one possible implementation scenario, the text in the domain can be translated based on the trained text translation model corresponding to that domain, and the translation result can be obtained.

[0164] For example, in one possible embodiment, the translation results can be provided to various types of entities, such as patent agencies, the State Intellectual Property Office, enterprises, or translation companies. Enterprises include patent applicants and providers of various types of databases, such as patent databases. After receiving the translation results, these entities can use them to engage in normal activities.

[0165] For example, in one scenario, a company has application documents in English and needs to apply for a patent at the State Intellectual Property Office. However, applying for a patent at the State Intellectual Property Office requires application documents in Chinese. Therefore, it is necessary to translate the English application documents into Chinese and then apply for a patent at the State Intellectual Property Office based on the Chinese application documents.

[0166] In one example, a company can outsource the translation of its English application documents to a translation company. The translation company can determine the subject matter of the English application documents and then translate them into Chinese based on a text translation model specific to that subject. The translation company then provides the Chinese application documents to the company for review. After the company confirms that there are no errors, it can submit the Chinese application documents to a patent agency to handle the patent application process. The patent agency can then submit the Chinese application documents to the State Intellectual Property Office (SIPO) to apply for a patent. Upon receiving the patent application request and the Chinese application documents, the SIPO can begin the patent application process.

[0167] Alternatively, the application documents in Chinese can be submitted to the database provider so that the database provider can provide data services based on the Chinese application documents.

[0168] Reference Figure 2 The diagram shows a structural block diagram of an embodiment of a text processing apparatus according to this application, which may specifically include the following modules:

[0169] The first acquisition module 11 is used to acquire the text to be recognized;

[0170] The second acquisition module 12 is used to obtain a first text block in the text to be identified based on the N-gram language model. The first text block includes N adjacent words, where N is an integer greater than 1.

[0171] The third acquisition module 13 is used to acquire the number of times the first text block appears in the seed text;

[0172] The determination module 14 is used to determine whether the text to be identified and the seed text belong to the same domain based on the number of occurrences.

[0173] In an optional implementation, the first acquisition module includes:

[0174] The selection unit is used to select any text from the first text library and use it as the text to be recognized.

[0175] or,

[0176] The search unit is used to obtain texts that are semantically related to the seed text in the first text library and use them as the text to be identified.

[0177] In one optional implementation, the search unit includes:

[0178] A subunit is generated to generate search query terms based on the semantics of the seed text;

[0179] A retrieval subunit is used to perform a retrieval in the first text database based on the retrieval query terms;

[0180] Obtain a subunit for obtaining text that is semantically related to the seed text.

[0181] In one optional implementation, the third acquisition module includes:

[0182] The lookup unit is used to query a known frequency dictionary to obtain the number of times the first text block appears in the seed text, wherein the frequency dictionary stores the following correspondence: text block and the number of times the text block appears in the seed text.

[0183] In an optional implementation, the third acquisition module further includes:

[0184] The first acquisition unit is used to obtain a second text block from the seed text based on the N-gram language model, the second text block comprising N adjacent words;

[0185] The second acquisition unit is used to acquire the number of times the second text block appears in the seed text;

[0186] A storage unit is used in the frequency dictionary to establish a correspondence between the second text block and the number of times the second text block appears in the seed text.

[0187] In one optional implementation, the determining module includes:

[0188] The summation unit is used to sum the occurrence counts of different first text blocks in the seed text to obtain the total count;

[0189] The determining unit is used to determine whether the text to be identified and the seed text belong to the same domain based on the total number of times.

[0190] In one optional implementation, the determining unit includes:

[0191] The first determining subunit is used to determine that the seed text and the text to be identified belong to the same domain when the total number of times is greater than or equal to a first preset threshold.

[0192] The second determining subunit is used to determine that the seed text and the text to be identified do not belong to the same domain when the total number of times is less than a first preset threshold.

[0193] In one optional implementation, the determining unit includes:

[0194] A sub-unit is used to obtain the number of words included in the text to be identified;

[0195] A calculation subunit is used to calculate the ratio between the total number of times and the quantity;

[0196] The third determining subunit is used to determine that the seed text and the text to be identified belong to the same domain when the ratio is greater than or equal to the second preset threshold.

[0197] The fourth determining subunit is used to determine that the seed text and the text to be identified do not belong to the same domain when the ratio is less than the second preset threshold.

[0198] In an alternative implementation, the apparatus further includes:

[0199] The fourth acquisition module is used to acquire the translated text corresponding to the text to be identified when the seed text and the text to be identified belong to the same domain.

[0200] The first storage module is used to store the text to be identified and the translated text in a second text library corresponding to the domain.

[0201] In an alternative implementation, the apparatus further includes:

[0202] The first training module is used to train a domain classification model corresponding to the domain based on the text in the second text library;

[0203] The filtering module is used to filter texts belonging to the same domain as the seed text in the first text library based on the domain classification model.

[0204] The fifth module is used to obtain the translated text corresponding to the filtered text;

[0205] The second storage module is used to store the filtered text and its corresponding translation text in the second text library.

[0206] In an alternative implementation, the apparatus further includes:

[0207] The second training module is used to train a text translation model corresponding to the domain based on the text in the second text library.

[0208] In situations where a large amount of text belonging to the same domain is needed, such as when training a text translation model for that domain, technicians do not need to manually sift through a large number of texts from various domains. Instead, technicians can manually collect a small number of texts from that domain and submit them as seed texts to the electronic device. This allows the electronic device to select texts from the text library as the text to be recognized. Then, it automatically obtains the first text block from the text to be recognized based on the N-gram language model. The first text block includes N adjacent words. The frequency of the first text block in the seed text is obtained, and the frequency of occurrence is used to determine whether the text to be recognized and the seed text belong to the same domain.

[0209] Using the above method, a large number of texts in this field can be automatically filtered out from the text library. Since technicians do not need to manually filter a large number of texts in this field from the text library which includes texts from various fields, technicians can manually collect a small number of texts in this field, thereby reducing the workload and thus reducing labor costs.

[0210] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0211] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes servers, gateways, sub-devices, etc., with the sub-devices being devices such as Internet of Things (IoT) devices.

[0212] Embodiments of this disclosure can be implemented as an apparatus with any suitable hardware, firmware, software, or any combination thereof, configured as desired. This apparatus may include electronic devices such as servers (clusters) and terminal devices such as IoT devices.

[0213] Figure 3 An exemplary apparatus 1300 is schematically shown that can be used to implement the various embodiments described in this application.

[0214] In one embodiment, Figure 3An exemplary device 1300 is shown, which includes one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1306.

[0215] Processor 1302 may include one or more single-core or multi-core processors, and processor 1302 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1300 can serve as a server device such as a gateway as described in the embodiments of this application.

[0216] In some embodiments, apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 that are combined with the one or more computer-readable media and configured to execute instructions 1314 to implement modules and thus perform the actions described in this disclosure.

[0217] In one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1302 and / or any suitable device or component communicating with the control module 1304.

[0218] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0219] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for device 1300. In one embodiment, memory 1306 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1306 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0220] In one embodiment, the control module 1304 may include one or more input / output controllers to provide interfaces to the NVM / storage device 1308 and (one or more) input / output devices 1310.

[0221] For example, NVM / storage device 1308 may be used to store data and / or instructions 1314. NVM / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0222] NVM / storage device 1308 may include storage resources that are physically part of a device on which device 1300 is mounted, or that can be accessed by the device without needing to be part of the device. For example, NVM / storage device 1308 may be accessed via a network via one or more input / output devices 1310.

[0223] One or more input / output devices 1310 may provide an interface for device 1300 to communicate with any other suitable device. Input / output devices 1310 may include communication components, pinyin components, sensor components, etc. Network interface 1312 may provide an interface for device 1300 to communicate via one or more networks. Device 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0224] In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1304. In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers of the control module 1304 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die. In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die to form a system-on-a-chip (SoC).

[0225] In various embodiments, device 1300 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1300 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0226] This application provides an electronic device, including: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform one or more text processing methods as described in this application.

[0227] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0228] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0229] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable text processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable text processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0230] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable text processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0231] These computer program instructions can also be loaded onto a computer or other programmable text processing terminal device, causing a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0232] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0233] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0234] The foregoing has provided a detailed description of a text processing method and apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A text processing method, characterized in that, The method includes: Obtain the text to be recognized; The first text block is obtained from the text to be identified based on the N-gram language model. The first text block includes N adjacent words, where N is an integer greater than 1. Get the number of times the first text block appears in the seed text; The frequency of occurrence determines whether the text to be identified and the seed text belong to the same domain. The step of obtaining the number of occurrences of the first text block in the seed text includes: The frequency dictionary is used to obtain the number of times the first text block appears in the seed text. The frequency dictionary stores the following correspondence: the text block and the number of times the text block appears in the seed text.

2. The method according to claim 1, characterized in that, The process of obtaining the text to be recognized includes: Select any text from the first text library and use it as the text to be recognized; or, Obtain semantically related texts from the first text library and use them as the text to be identified.

3. The method according to claim 2, characterized in that, Obtaining semantically related text to the seed text in the first text library includes: Generate search query terms based on the semantics of the seed text; The search is performed in the first text database based on the search query terms; Obtain the text that is semantically related to the seed text.

4. The method according to claim 1, characterized in that, The frequency dictionary is obtained in the following way: The second text block is obtained from the seed text based on the N-gram language model. The second text block includes N adjacent words. Obtain the number of times the second text block appears in the seed text; In the frequency dictionary, a correspondence is established between the second text block and the number of times the second text block appears in the seed text.

5. The method according to claim 1, characterized in that, The step of determining whether the text to be identified and the seed text belong to the same domain based on the frequency of occurrence includes: The total number of occurrences of different first text blocks in the seed text is obtained by summing the occurrence counts of each first text block. The total number of occurrences determines whether the text to be identified and the seed text belong to the same domain.

6. The method according to claim 5, characterized in that, The step of determining whether the text to be identified and the seed text belong to the same domain based on the total number of times includes: If the total number of occurrences is greater than or equal to a first preset threshold, it is determined that the seed text and the text to be identified belong to the same domain. If the total number of occurrences is less than a first preset threshold, it is determined that the seed text and the text to be identified do not belong to the same domain.

7. The method according to claim 5, characterized in that, The step of determining whether the text to be identified and the seed text belong to the same domain based on the total number of times includes: Obtain the number of words included in the text to be identified; Calculate the ratio between the total number of times and the quantity; If the ratio is greater than or equal to a second preset threshold, it is determined that the seed text and the text to be identified belong to the same domain. If the ratio is less than a second preset threshold, it is determined that the seed text and the text to be identified do not belong to the same domain.

8. The method according to claim 1, characterized in that, The method further includes: If the seed text and the text to be identified belong to the same domain, obtain the translated text corresponding to the text to be identified; The text to be identified and the translated text are stored in a second text library corresponding to the domain.

9. The method according to claim 8, characterized in that, The method further includes: Train a domain classification model corresponding to the domain based on the text in the second text library; Based on the domain classification model, texts belonging to the same domain as the seed text are selected from the first text library; Get the translated text corresponding to the filtered text; The selected text and its corresponding translation text are stored in the second text library.

10. The method according to claim 8 or 9, characterized in that, The method further includes: A text translation model corresponding to the domain is trained based on texts from the second text library.

11. A text processing device, characterized in that, The device includes: The first acquisition module is used to acquire the text to be recognized; The second acquisition module is used to obtain a first text block in the text to be identified based on the N-gram language model. The first text block includes N adjacent words, where N is an integer greater than 1. The third acquisition module is used to acquire the number of times the first text block appears in the seed text; The determination module is used to determine whether the text to be identified and the seed text belong to the same domain based on the number of occurrences. The third acquisition module includes: The lookup unit is used to query a known frequency dictionary to obtain the number of times the first text block appears in the seed text, wherein the frequency dictionary stores the following correspondence: text block and the number of times the text block appears in the seed text.

12. The apparatus according to claim 11, characterized in that, The first acquisition module includes: The selection unit is used to select any text from the first text library and use it as the text to be recognized. or, The search unit is used to obtain texts that are semantically related to the seed text in the first text library and use them as the text to be identified.

13. The apparatus according to claim 12, characterized in that, The search unit includes: A subunit is generated to generate search query terms based on the semantics of the seed text; A retrieval subunit is used to perform a retrieval in the first text database based on the retrieval query terms; Obtain a subunit for obtaining text that is semantically related to the seed text.

14. The apparatus according to claim 11, characterized in that, The third acquisition module further includes: The first acquisition unit is used to obtain a second text block from the seed text based on the N-gram language model, the second text block comprising N adjacent words; The second acquisition unit is used to acquire the number of times the second text block appears in the seed text; A storage unit is used in the frequency dictionary to establish a correspondence between the second text block and the number of times the second text block appears in the seed text.

15. The apparatus according to claim 11, characterized in that, The determining module includes: The summation unit is used to sum the occurrence counts of different first text blocks in the seed text to obtain the total count; The determining unit is used to determine whether the text to be identified and the seed text belong to the same domain based on the total number of times.

16. The apparatus according to claim 15, characterized in that, The determining unit includes: The first determining subunit is used to determine that the seed text and the text to be identified belong to the same domain when the total number of times is greater than or equal to a first preset threshold. The second determining subunit is used to determine that the seed text and the text to be identified do not belong to the same domain when the total number of times is less than a first preset threshold.

17. The apparatus according to claim 15, characterized in that, The determining unit includes: A sub-unit is used to obtain the number of words included in the text to be identified; A calculation subunit is used to calculate the ratio between the total number of times and the quantity; The third determining subunit is used to determine that the seed text and the text to be identified belong to the same domain when the ratio is greater than or equal to the second preset threshold. The fourth determining subunit is used to determine that the seed text and the text to be identified do not belong to the same domain when the ratio is less than the second preset threshold.

18. The apparatus according to claim 11, characterized in that, The device further includes: The fourth acquisition module is used to acquire the translated text corresponding to the text to be identified when the seed text and the text to be identified belong to the same domain. The first storage module is used to store the text to be identified and the translated text in a second text library corresponding to the domain.

19. The apparatus according to claim 18, characterized in that, The device further includes: The first training module is used to train a domain classification model corresponding to the domain based on the text in the second text library; The filtering module is used to filter texts belonging to the same domain as the seed text in the first text library based on the domain classification model. The fifth module is used to obtain the translated text corresponding to the filtered text; The second storage module is used to store the filtered text and its corresponding translation text in the second text library.

20. The apparatus according to claim 18 or 19, characterized in that, The device further includes: The second training module is used to train a text translation model corresponding to the domain based on the text in the second text library.

21. An electronic device, characterized in that, The electronic device includes: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform one or more of the text processing methods as described in claims 1-10.

22. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform a text processing method as described in one or more of claims 1-10.

Images