Text processing method, device, computer equipment and readable storage medium

Abstract

The embodiment of the present application discloses a text processing method, apparatus, computer equipment and readable storage medium, the method comprising: obtaining a text to be processed, determining the character proportion of each first language in the text to be processed; if there is a first language with a character proportion greater than or equal to a first set threshold, determining the first language with a character proportion greater than or equal to the first set threshold as the target language corresponding to the text to be processed; if there is no first language with a character proportion greater than or equal to the first set threshold, determining the high-frequency word proportion of each second language in the text to be processed; if the maximum high-frequency word proportion among the high-frequency word proportions of each second language is greater than or equal to a second set threshold, determining the second language corresponding to the maximum high-frequency word proportion as the target language corresponding to the text to be processed. Through the embodiment of the present application, the main language of the text to be processed can be accurately determined.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text processing method, apparatus, computer device, and readable storage medium. Background Art

[0002] With the advancement of globalization and the rapid development of Internet information services, more and more products have services targeting different countries. At the same time, product service providers are also faced with processing large amounts of text data in different languages, such as consultation emails from overseas users. Among these text data, the vast majority are mixed in multiple languages. In this scenario, it is necessary to identify the main language of the text data so that more targeted services can be provided based on the identified language.

[0003] Currently, language recognition for text is typically performed based on the unique characters of a particular language. Specifically, if a special character in a text belongs to a particular language, that language is considered the text's language. Alternatively, a set of high-frequency words from a particular language is collected, and if a word in the text belongs to a high-frequency word in that language, that language is determined as the text's language. While these language recognition methods are reliable for identifying language components, they are less reliable for identifying the primary language of multilingual text, affecting the accuracy of the recognition results. Summary of the Invention

[0004] The embodiments of the present application provide a text processing method, apparatus, computer device, and readable storage medium, which can accurately determine the main language of the text to be processed.

[0005] An embodiment of the present application provides a text processing method, including:

[0006] Obtaining a text to be processed, and determining a percentage of characters in each first language in the text to be processed;

[0007] If there is a first language whose character ratio is greater than or equal to the first set threshold, determining the first language whose character ratio is greater than or equal to the first set threshold as the target language corresponding to the text to be processed;

[0008] If there is no first language with a character ratio greater than or equal to the first set threshold, determining the ratio of high-frequency words in each second language in the text to be processed;

[0009] If the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to a second set threshold, the second language corresponding to the maximum high-frequency word proportion is determined as the target language corresponding to the text to be processed.

[0010] In one aspect, an embodiment of the present application provides a text processing device, including:

[0011] An acquisition module, configured to acquire a text to be processed and determine a percentage of characters in each first language in the text to be processed;

[0012] a determination module configured to, if there is a first language whose character ratio is greater than or equal to a first set threshold, determine the first language whose character ratio is greater than or equal to the first set threshold as the target language corresponding to the text to be processed;

[0013] The determination module is further configured to determine the proportion of high-frequency words in each second language in the text to be processed if there is no first language with a character proportion greater than or equal to the first set threshold;

[0014] The determination module is further configured to determine the second language corresponding to the maximum high-frequency word proportion as the target language corresponding to the text to be processed if the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to a second set threshold.

[0015] In one aspect, an embodiment of the present application provides a computer device, including: a processor and a memory;

[0016] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the text processing method in the embodiment of the present application.

[0017] On one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the text processing method in the embodiment of the present application is executed.

[0018] Accordingly, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text processing method provided in one aspect of the embodiments of the present application.

[0019] In an embodiment of the present application, by comparing the character proportions of each first language with a first set threshold, the first language corresponding to the character proportion greater than the first set threshold is determined as the main language corresponding to the text to be processed (i.e., the target language, which is the language with a larger proportion in the text to be processed). In this way, for the text to be processed that is a mixture of different languages, the accuracy of language recognition can be enhanced. If the above-mentioned method of using characters cannot identify the language of the text to be processed, it is necessary to upgrade the character recognition of the text to be processed to the word recognition of the text to be processed. By comparing the high-frequency word proportions of each second language in the text to be processed with a second set threshold, the second language corresponding to the maximum high-frequency word proportion greater than the second set threshold can be determined as the main language corresponding to the text to be processed. This not only solves the problem of multi-language mixing in real application scenarios, but also improves the credibility and accuracy of the entire language recognition model in distinguishing the main language of the text to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a schematic diagram of the architecture of a text processing system provided in an embodiment of the present application;

[0022] Figure 2 This is a flowchart of an email processing method provided by an embodiment of the present application;

[0023] Figure 3 This is a flowchart of a text language recognition process provided by an embodiment of the present application;

[0024] Figure 4 This is a flowchart of a text processing method provided in an embodiment of the present application;

[0025] Figure 5 This is a flowchart of a text processing method provided by an embodiment of the present application;

[0026] Figure 6 is a structural diagram of a text processing device provided in an embodiment of the present application;

[0027] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] The core idea of ​​the embodiments of the present application is to determine the proportion of characters in the text to be processed by character encoding or the proportion of words in the text to be processed by the word segmentation information obtained from the word segmentation of the text to be processed, and determine the main language to which the text to be processed belongs based on the comparison result of the proportion and the corresponding threshold value. In this way, the detection method of the characteristics of different languages ​​is differentiated, which improves the efficiency of identifying the main language of the text to be processed. At the same time, the confidence level (i.e., threshold) is introduced to determine the language only when the proportion of the corresponding language reaches the threshold, thereby increasing the accuracy of the recognition result. Among them, there is usually only one main language to which the text to be processed belongs, which can be the language with the largest character proportion or the largest high-frequency word proportion calculated according to the characters or high-frequency words in the text to be processed, also known as the target language. Therefore, the main language is equivalent to the target language, and either of them can be replaced in the following content. The text processing method provided in the embodiments of the present application is based on the natural language processing (NLP) technology under the field of artificial intelligence.

[0030] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0031] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0032] Among them, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies. For example, in the embodiment of the present application, character encoding is used to obtain the character ratio of the text to be processed or the high-frequency word ratio is obtained by word segmentation of the text to be processed.

[0033] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of a text processing system provided by an embodiment of the present application. Figure 1 As shown, the architecture of the text processing system may include a text processing device 100 and a terminal device 101. The terminal device 101 may include multiple terminal devices. Each terminal device 101 may be connected to the above-mentioned text processing device 100 through a network. The network connection may include a wired connection or a wireless connection, so that each terminal device can interact with the text processing 100 through the network connection, and the text processing device 100 can receive business data from each terminal device 101. The specific functions of the text processing device 100 and the terminal device 101 are introduced in detail below.

[0034] The text processing device 100 can be one or more of a single server, a server cluster, and a cloud server, and there is no limitation on this. The main function of the text processing device 100 is to perform language recognition on the text to be processed. The text to be processed here can be pre-stored text data obtained from a text database, or it can be real-time text data directly uploaded to the text processing device 101 by the terminal device 101. During the language recognition process, the text processing device 100 first processes each character in the text to be processed and generates a character code. Then, based on the character code, it counts the number of characters in each first language and the total number of text characters. In combination with the character code interval corresponding to the character set pre-stored in the text database, it detects the character ratio of each first language in the text to be processed. Then, based on the character ratio and a first set threshold, it determines the main language of the text to be processed. If the main language of the text to be processed cannot be determined based on the character ratio, the text processing device 100 will perform word segmentation on the text to be processed, obtain word segmentation information, and in combination with the target high-frequency word list pre-stored in the text database, detect the ratio of high-frequency words in each second language in the text to be processed. The main language of the text to be processed is determined based on the high-frequency word ratio and the first set threshold. If the main language of the text to be processed cannot be determined based on the high-frequency word ratio, the default language will be determined as the main language of the text to be processed. After identifying the language of the text to be processed, the text processing device 100 can store the recognition result in a text database or directly send it to the terminal device 101 for subsequent processing. For example, if the text to be processed is an email, the recognition result of the email can be distributed to the terminal device 101 of the corresponding service department for a specialist to respond. In addition, the text processing device 100 can also serve as a text database to store all business data, which can be text data or recognition data, such as unrecognized text to be processed or recognized text to be processed.

[0035] The terminal device 101 can be one or more of a smartphone, a video camera, a desktop computer, a tablet computer, an MP4 player, and a laptop computer. The terminal device 101 can install and run a target application for managing text data. Within this target application, the terminal device 101 can monitor the amount of unrecognized text to be processed, the amount of text to be processed recognized by the text processing device 100, the amount of distributed recognized text to be processed, and view corresponding data reports. Furthermore, the terminal device 101 can also upload text to be processed to the text processing device 100 or download successfully recognized text to be processed from a text database or the text processing device 100 for subsequent processing.

[0036] For specific application scenarios of the above text processing system architecture diagram, please refer to Figure 2 , Figure 2This is a flowchart of an email processing process provided by an embodiment of this application. The language recognition capability of emails is mainly used in a customer service platform, specifically in a service scenario of replying to emails from overseas users. The entire process includes the following steps:

[0037] Step 1: Email Receiving. Specifically, the server retrieves emails sent by users from a dedicated email database, such as the database containing emails from a product's official email account. Typically, emails contain product information and user assistance, primarily in a single language. Therefore, emails may contain text in one or more languages, such as primarily Korean with a few English abbreviations.

[0038] Step 2: Language Identification. The language of the received email is intelligently detected in a language identification model. The model then processes the email using appropriate detection methods and ultimately determines the target language for the email, achieving language identification. For example, in the example above, for English and Korean, language identification determines that Korean is the target language for the email.

[0039] Step 3: Email Dispatch. Target emails with identified results are dispatched to the corresponding language service skill group. This automatic dispatch is based on the email recognition results, eliminating the need to aggregate emails into a public service group and then manually distribute them. This improves the efficiency of each language service skill group's ability to connect with the appropriate emails and simplifies the processing process.

[0040] Step 4: Manual processing. When an email is assigned to the corresponding language service skill group, manual processing is still required. Specifically, the language service skill group will assign a specialist to handle it. For example, the Korean email mentioned above is assigned to the Korean service skill group, who will assign a specialist to reply to the email.

[0041] This completes the entire email reply process. Throughout the email dispatch process, the machine automatically identifies the email language and dispatches it to different skill groups, eliminating the need for human intervention, reducing process complexity and saving manpower.

[0042] Figure 2 For the specific technical implementation of the language recognition step in the email processing process, please refer to Figure 3 , Figure 3This is a flow chart of text language recognition provided by an embodiment of the present application. Since overseas email replies are often based on multiple languages, efficient language detection is a prerequisite for implementing multilingual email replies. Therefore, this solution designs a detection service that can recognize multiple languages ​​and can be easily expanded to other languages. Currently, Chinese (ch), English (en), Arabic (ar), Russian (ru), Turkish (tr), Spanish (es) and Italian (it) are implemented. For the specific language recognition process, see Figure 3 The core of this process is character set detection and stop word detection, which uses different detection methods based on the characteristics of different languages. First, the text is preprocessed to filter out punctuation, hyperlinks, numbers, etc. The text is then tested using the character set to see if it contains special characters. If not, the stop word ratio is checked. If the test fails again, the default language, English, is returned. The corresponding content in the flowchart can be specifically explained as follows:

[0043] Step 1: Input text. The input text can be in any language, including languages ​​that can be processed by current language recognition, such as Chinese (ch), English (en), Arabic (ar), Russian (ru), Turkish (tr), Spanish (es), and Italian (it).

[0044] Step 2, quick preprocessing. This mainly removes symbols other than special characters in the text, such as punctuation marks, hyperlinks, numbers, etc. The text obtained after removal includes the required text characters, which can save subsequent processing time and speed up the processing speed.

[0045] Step 3: Character set detection. This is used to detect languages ​​with special characters, including Chinese (ch), Arabic (ar), and Russian (ru). If the character set detection succeeds, the recall step in step 5 is executed, returning languages ​​that meet the criteria to step 5. If the character set detection fails, the stop word ratio detection step in step 4 is executed.

[0046] Step 4: Stop word ratio detection. This step is performed when the character set detection fails. It is mainly used to detect Turkish (tr), Spanish (es), and Italian (it). If the stop word ratio detection is successful, the recall process in step 5 is also performed, and the results of the stop word detection are recalled. If the stop word ratio detection fails, the recall process is also performed. In this case, the default language English (en) is recalled. The recalled language is then used as the primary language of the input text, and the text recognition ends.

[0047] Step 5: Recall. If the character set detection or stop word ratio detection is successful, the language obtained by the character set detection or the language obtained by the stop word ratio detection is recalled, and then the recalled language is used as the target language of the input text, and text recognition ends. If the stop word ratio detection fails, the default language English is recalled as the target language of the input text. Normally, only one language will be returned if the recall is successful, and from the perspective of characters or words, the sum of the proportions of multiple languages ​​is 1. By calculating the proportion of each language and setting a specific threshold, the main language of the input text can be determined.

[0048] The above-mentioned text language recognition solution uses character set detection and stop word ratio detection to specifically detect the primary language of a text based on the characteristics of different languages. This improves the accuracy and reliability of the model's discrimination. This overall text language recognition solution efficiently identifies the primary language of the input text, facilitating the accurate assignment of processing personnel with the corresponding language to the text being processed. In particular, the intelligent recognition of email languages ​​further accelerates the delivery of targeted responses by customer service personnel in the corresponding language, improving email processing efficiency. Experiments have demonstrated that the language recognition accuracy of this solution for emails exceeds 90%. After its launch, the solution has simplified the multilingual email response process, reduced system complexity, and saved manpower.

[0049] It is understood that the text language recognition method provided in the embodiments of the present application can be executed by a computer device, including but not limited to the terminal device 101 and the text processing device 100. The computer device can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0050] For further understanding, please see Figure 4 , Figure 4 This is a flowchart of a text processing method provided by an embodiment of the present application. The execution subject in this embodiment can be a computer device or a cluster of multiple computer devices. The computer device can be a terminal device or a server. Here, the execution subject in this embodiment is a server as an example. The text processing method may include at least the following steps S401-S404:

[0051] S401: Obtain a text to be processed, and determine the character ratio of each first language in the text to be processed.

[0052] In a possible embodiment, the text to be processed includes one or more characters, which may be one or more of Roman numerals, Latin letters, punctuation marks, geometric symbols, and text characters in multiple languages. Based on these characters, a content expressing specific semantics can be formed, such as a hyperlink composed of Latin letters and punctuation marks, or a sentence composed of multiple words composed of English characters, or a content composed of other languages. Of course, it can also be content without any semantics composed of random characters. The content of the text to be processed and the languages ​​involved are not limited in this application. A server with a text processing function can obtain the text to be processed from a text database, which can belong to the server. After obtaining the text to be processed, it is also necessary to preprocess the text to be processed, filter out punctuation marks, hyperlinks, numbers, etc. in the text, and then use the characters included in the preprocessed text to determine the character proportion of each first language.

[0053] In a possible embodiment, the method for determining the proportion of characters of each first language in the text to be processed may include the following steps: obtaining the character code corresponding to each character in the text to be processed. The character code here may be a Unicode code (i.e., a global unified text code) generated for each character of the text to be processed. Since the Unicode code assigns a unique code to each character of various texts in the world, it can be used for cross-language and cross-platform applications. Therefore, the Unicode code can be used to determine the code range of the characters in the text to be processed and the language to which the characters belong. Therefore, the Unicode code can be used to identify the language of the text well. Of course, the character code can also be other codes that can uniquely represent texts in various languages, which is not limited here. The characters included in the text to be processed mentioned above can be multilingual characters, and the character code and character code interval of each language are different. Therefore, according to the character code interval of each first language, the character code of each character in the text to be processed, and the total number of characters in the text to be processed, the proportion of characters of each first language in the text to be processed can be determined. Correspondingly, the character encoding range of each first language here corresponds to the Unicode encoding range with special characters. The character encoding range of each language corresponds to a character set composed of characters corresponding to different character encodings. The character ratio refers to the ratio of special characters in the first language. Special characters are characters that only appear in this language (or language) and do not appear in other languages ​​(or languages). Languages ​​with special characters are also collectively referred to as first languages. For example, Latin characters are not only used in English, but also in Italian, French, etc. Therefore, Latin characters are not special characters, and English, Italian, French, etc. are not first languages. Even if each Latin character has a Unicode encoding, words composed of Latin characters in different languages ​​may be repeated and have no unique representation. On the contrary, for example, Chinese, Arabic, Russian, Korean and Japanese, these languages ​​each have unique characters and unique Unicode encoding ranges. As shown in Table 1, the Unicode encoding ranges of different languages.

[0054] Table 1 Unicode encoding ranges for different languages

[0055]

[0056] As can be seen from the table above, each code range corresponds to a character set. For example, 4E00-9FCC is a partial set of commonly used Chinese characters, 3400-4DB5 is the character set of Chinese Extended A, which includes 6582 characters, and 3040-309F in Japanese is the character set of Japanese Hiragana, and 30A0-30FF is the character set of Japanese Katakana. Since the languages ​​in Table 1 all have unique characters (i.e., special characters) corresponding to the Unicode code range, they can all be used as first languages. If the first language is to be expanded, the specific expansion standard can be that all languages ​​with special characters can be used as first languages. For example, in languages ​​such as Thai and Mongolian, characters that do not appear in other languages ​​can be considered special characters. Languages ​​with special characters such as Thai and Mongolian can be used as first languages ​​together with the languages ​​in Table 1.

[0057] In summary, since different languages ​​may share the same characters, and the encodings for these characters are the same, special characters can be identified by finding the encoding ranges for different languages. Then, based on the character encoding of the text to be processed, it can be determined whether it falls within the character encoding range for the first language with special characters, as shown in Table 1. If it falls within this range, the character is considered a special character of the corresponding language. Other characters are also determined as special characters using the same principle. The ratio of the number of special characters in the text to the total number of characters in the text to be processed is then taken as the special character percentage. The special character percentages need to be calculated separately for different first languages, so the character percentage values ​​may be the same or different. Corresponding to the first languages ​​in Table 1, the character percentages for each first language in the text to be processed are finally obtained, including Chinese, Arabic, Russian, Korean, and Japanese. The character percentages for each language can be denoted as {x1, x2, x3, x4, x5}, respectively. The character percentage for each first language is typically expressed as a percentage, such as 10%. If the text to be processed does not include a first language, such as Chinese, the corresponding character percentage is recorded as 0.

[0058] For example, if the text to be processed is "Help me unblock my PUBG ID as soon as possible", the characters involved include Chinese and English, and the special characters are only Chinese. Therefore, the number of special characters is 8, the number of English characters is 6, and the total number of characters in the text is 14. The calculated proportion of Chinese characters is 8 / 14 (about 57.14%), and the proportion of characters of other first languages ​​(including Arabic, Russian, Korean and Japanese) is 0. Since English is not a special character, it is not included in the calculation of the character proportion of the first language.

[0059] S402: If there is a first language whose character ratio is greater than or equal to a first set threshold, determine the first language whose character ratio is greater than or equal to the first set threshold as the target language corresponding to the text to be processed.

[0060] In one possible embodiment, ideally, as long as the Unicode code range of the characters in the input text contains the code range of a specific language, the text can be determined to be in that language. However, in actual application scenarios, texts often contain a mixture of different languages, and the above method cannot determine the primary language of the text. For example, in the example of a mixed Chinese and English text: "Help me unblock my PUBG ID as soon as possible, thank you." This text contains both Chinese and English. According to the Unicode code, the language of the text is determined to be Chinese, but in fact, English is the primary language. Only by correctly identifying English can processing of the corresponding language be provided. Therefore, to enhance the accuracy of the determination, this solution introduces the special character ratio (i.e., character ratio) and confidence level (i.e., a first predetermined threshold). The special character ratio is calculated as the ratio of the number of special characters contained in the text to the total number of characters in the text. Only when this ratio reaches a specific threshold (i.e., the first predetermined threshold) can the text be determined to be in the corresponding language. It should be noted that each first language shares the same first predetermined threshold.

[0061] By comparing the character proportions of each first language with the first set threshold, the first language corresponding to the character proportion greater than or equal to the first set threshold can be determined, and used as the target language corresponding to the text to be processed, that is, the main language of the text to be processed. As an optional implementation method, the maximum character proportion can also be determined from the character proportions of each first language, and the maximum character proportion can be compared with the first set threshold. If the maximum character proportion is greater than or equal to the first set threshold, then the main language of the text to be processed must exist in the first language, and the first language corresponding to the maximum character proportion can be used as the main language corresponding to the text to be processed without performing subsequent processing steps. Normally, the value of the first set threshold is set to be greater than 50%. In this way, for the same language, it can be ensured that more than half of the characters in the text to be processed are special characters of the language, and it is more credible to use it as the main language of the text to be processed.

[0062] S403: If there is no first language with a character ratio greater than or equal to the first set threshold, determine the ratio of high-frequency words in each second language in the text to be processed.

[0063] In one possible embodiment, if the maximum character ratio of each first language in the text to be processed is less than the first set threshold, then it means that there is no first language in each first language whose character ratio is greater than or equal to the first set threshold, and the main language of the text to be processed cannot be identified. This may be because the text to be processed includes a large number of Latin letters. Therefore, it is necessary to upgrade the recognition method from using characters to using words, and determine the best corresponding language by establishing a high-frequency word distribution in multiple languages ​​for the input text to be processed. This recognition method first needs to determine the high-frequency word ratio of each second language, where high-frequency words refer to words or single characters that are used more frequently in the second language. The determination of the high-frequency words included in the text to be processed needs to be combined with a stop word list that has been collected and organized in advance. In this solution, stop word lists for Turkish, Spanish and Italian are collected and organized, where Turkish, Spanish and Italian can be called second languages. The concept of stop words is often used in retrieval systems. They are called stop words because when these words are encountered during text processing, processing is stopped immediately and they are discarded to reduce the index volume, increase retrieval efficiency, and thus improve retrieval results. The same principle also applies in the embodiments of the present application. According to stop words (high-frequency words are used as stop words in this solution), the languages ​​included in the text to be processed can be determined more quickly and effectively, and then the main language (i.e., the target language) can be determined from the languages ​​included in the text to be processed based on the proportion of high-frequency words. After the input text to be processed is segmented, the proportion of stop words of each second language in the text to be processed is calculated. Specifically, the ratio of the number of high-frequency words in the second language to the total number of words in the text can be used as the corresponding high-frequency word proportion, and each second language has a corresponding high-frequency word proportion, which usually appears in the form of a percentage.

[0064] S404: If the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to a second set threshold, the second language corresponding to the maximum high-frequency word proportion is determined as the target language corresponding to the text to be processed.

[0065] In a possible embodiment, similar to the character ratio, this step also introduces the high-frequency word ratio and confidence (here referred to as the second set threshold), and the second language corresponding to the high-frequency word ratio reaching the second set threshold is determined as the main language (i.e., the target language) to which the text to be processed belongs. Specifically, the maximum high-frequency word ratio can be first determined from the high-frequency word ratios of each second language, and then the maximum high-frequency word ratio is compared with the second set threshold. If the maximum high-frequency word ratio is greater than or equal to the second set threshold, then the target language to which the text to be processed belongs is determined to be the second language corresponding to the maximum high-frequency word ratio. In other words, if the high-frequency word ratio of a certain language is both the highest value and reaches the set threshold, then the text to be processed is determined to be of that language. On the contrary, if the maximum high-frequency word ratio is less than the second set threshold, this means that there is no high-frequency word ratio greater than or equal to the second set threshold among the high-frequency word ratios of each second language, and it is impossible to determine the main language corresponding to the text to be processed from the second language. In this case, the default language can be used as the main language corresponding to the text to be processed. The default language here can be English.

[0066] Optionally, the high-frequency word proportions of each second language can be compared with the second set threshold respectively to determine the maximum high-frequency word greater than or equal to the second set threshold. If there are two or more high-frequency word proportions greater than or equal to the second set threshold, the maximum high-frequency word proportion is selected from these two or more high-frequency word proportions greater than or equal to the second set threshold, and the second language corresponding to the maximum high-frequency word proportion is determined as the target language corresponding to the text to be processed.

[0067] It should be noted that the second set threshold and the first set threshold are set differently in numerical value, but similarly, the second set threshold is also shared by multiple languages, that is, the proportion of high-frequency words in each second language is compared based on the second set threshold as the judgment standard.

[0068] In summary, the embodiments of the present application have at least the following advantages:

[0069] The text to be processed, which includes multiple languages, is detected by using a character set composed of Unicode encoding or a high-frequency word list composed of high-frequency words to determine the main language of the text to be processed. Specifically, in order to ensure the accuracy of the main language recognition of the multilingual text to be processed, from the character aspect, the special character ratio and the first set threshold are introduced as reference standards for the main language recognition. The character set is used to detect the special character ratio of the text to be processed to determine the main language of the text to be processed; when the character set detection cannot determine the main language corresponding to the text to be processed, it is necessary to consider the word aspect. By introducing the high-frequency word ratio of each second language and the second set threshold, the high-frequency word list is used to detect the high-frequency word ratio of the text to be processed to determine the main language. In the processing process of the above two methods, character set detection is preferred, and high-frequency word list detection is used as an auxiliary. The introduction of ratios and thresholds in both methods ensures that the recognition results obtained in the case of multiple languages ​​for the text to be processed are more convincing, and improves the accuracy and credibility of the main language recognition of the text to be processed.

[0070] See Figure 5 , Figure 5 This is a flowchart of a text processing method provided by an embodiment of the present application. The execution subject in this embodiment can be a computer device or a cluster of multiple computer devices. The computer device can be a terminal device or a server. Here, the execution subject in this embodiment is a server as an example. The text processing method may include at least the following steps S501-S505:

[0071] S501: Obtain a text to be processed, and determine the character ratio of each first language in the text to be processed.

[0072] S502: If there is a first language whose character ratio is greater than or equal to a first set threshold, determine the first language whose character ratio is greater than or equal to the first set threshold as the target language corresponding to the text to be processed.

[0073] The specific implementation of the above steps can be found in the Figure 4 S401-S402 in the corresponding embodiment will not be described again here.

[0074] S503: If there is no first language with a character ratio greater than or equal to the first set threshold, determine a target high-frequency word list for each second language, and obtain word segmentation information of the text to be processed.

[0075] In one possible embodiment, if the character proportions of each first language are less than a first set threshold, the method for detecting the text to be processed using a character set cannot effectively determine the primary language. In this case, the language detection can be performed using the words in the text to be processed. In this process, the key is to pre-establish an accurate target high-frequency word list for each second language. The target high-frequency word list here is the stop word list mentioned in the above embodiment. The process of determining the target high-frequency word list for each second language can include the following: for any second language, obtain a candidate high-frequency word list consisting of one or more high-frequency words in the second language. The candidate high-frequency word list here is determined based on the statistical number of occurrences of words in the second language and a set threshold. For example, words with a number of occurrences greater than the threshold are considered high-frequency words. It should be noted that the candidate high-frequency word lists for each second language are independently counted. High-frequency words that are repeated between the second languages ​​are also included in the candidate high-frequency word list. For example, Spanish and Italian share many high-frequency words, and candidate high-frequency word A exists in the candidate high-frequency word lists of both Spanish and Italian. It can be seen from this that there may be intersections between the candidate high-frequency word lists of each second language, and too much intersection will affect the distribution of high-frequency words. The distribution of stop words calculated in a text will be very misleading, which will lead to a relatively low language recognition accuracy. In order to solve this problem, the candidate high-frequency word list is pre-processed in the embodiment of the present application. Specifically, the common candidate high-frequency words in the candidate high-frequency word list are eliminated, and then the target high-frequency word list of the second language is obtained. The common high-frequency words here refer to candidate high-frequency words that appear in the candidate high-frequency word list of each second language more than or equal to the set number of times. Simply put, they are stop words shared by multiple second languages. For example, assuming that there are 4 second languages, the corresponding candidate high-frequency word lists are {T1, T2, T3, T4,}, and the set number of times each candidate high-frequency word appears is 2 times. Among them, the candidate high-frequency word A appears 3 times in these four candidate high-frequency word lists, which is greater than the set number, so it can be eliminated. The screening criteria for common high-frequency words can be expressed as follows (1):

[0076] S≥ceil(N / 2) (1)

[0077] Here, S represents the number of times a candidate high-frequency word appears in the candidate stop word list, ceil represents rounding up, and N represents the number of languages. Based on this screening criterion, candidate high-frequency words that appear in at least half of the candidate high-frequency word list are deleted.

[0078] In addition, single-character candidate high-frequency words in the candidate high-frequency word list can also be eliminated, where the single-character candidate high-frequency word can be a single Latin letter AZ (or az). Both single-character candidate high-frequency words and common candidate high-frequency words in the candidate high-frequency word list are eliminated. The specific operation can be performed by traversing the candidate high-frequency word list corresponding to each second language, eliminating the single-character candidate high-frequency words in the candidate high-frequency word list, and then putting all candidate high-frequency words of the language that needs to be detected using the candidate high-frequency word list (part of the second language or all the second language) in a set, counting and sorting the words in the set, and recording all words that are filtered out more than or equal to the set number of times in the common high-frequency word list filtered_stopwords, and then traversing each candidate high-frequency word list, eliminating the candidate high-frequency words that appear in filtered_stopwords, and finally using the eliminated candidate high-frequency word list as the target high-frequency word list of the second language, such as the stop word list shown in Table 2 (i.e., the target high-frequency word list of each second language):

[0079] Table 2 Stop word list

[0080]

[0081] The Turkish, Spanish, and Italian languages ​​in the above-mentioned stop word lists are collectively referred to as second languages, and the corresponding stop words (i.e., high-frequency words) are displayed as part of the content. The second language can be expanded based on whether a corresponding high-frequency word list can be established.

[0082] In addition, some frequently occurring high-frequency business words can be added to the target high-frequency word list based on business characteristics, and high-frequency business words in different languages ​​can be added to the target high-frequency word list. Specifically, the business type indicated by the text to be processed can be determined first, such as games, beauty, entertainment, etc. This can be determined according to the semantic information of the text to be processed or in other ways, which are not limited here. Then, high-frequency words related to the business type can be obtained. For example, for overseas game business, game-related words such as account suspension, unblocking, login, and freeze can be collected from the text to be processed. The second language corresponding to these high-frequency words can be added to the target high-frequency word list. For example, the Turkish, Spanish, and Italian corresponding to "account suspension" can be added to the target high-frequency word list respectively. This allows the user to quickly determine the language included in the text to be processed when encountering the same business type of text to be processed, thereby improving the efficiency of identifying the main language of the text to be processed. Alternatively, all business-related terms can be pre-collected as high-frequency business terms, added to a corresponding target high-frequency word list, and then partitioned by business type. Then, based on the business type indicated by the document to be processed, the high-frequency business terms in the corresponding business type partition can be used to determine the high-frequency business terms for the corresponding language. Furthermore, high-frequency terms unique to certain languages ​​can be added to improve the differentiation of each second language, allowing for quick identification of the second language corresponding to the high-frequency terms in the document to be processed.

[0083] The above scheme preprocesses the candidate high-frequency word list, including deleting all stop words represented by single Latin letters, deleting some stop words shared by multiple languages, and adding some high-frequency words that are more language-specific. During the entire process, different thresholds need to be set for different operations for judgment, and finally the target high-frequency word list is obtained. The high-frequency word distribution it reflects is more instructive, and the appropriate introduction of more targeted business-related stop words increases business relevance and improves the distinctiveness of the target candidate high-frequency word list, thereby achieving efficient recognition of the main language of the text to be processed.

[0084] While determining the target candidate high-frequency word lists for each second language, it is also necessary to obtain word segmentation information for the text to be processed. Specifically, this can be done by segmenting the text to be processed based on word separators or using a word segmentation model to segment the text to be processed, obtaining one or more words and using them as the word segmentation information. The word separator can be a space. Since there is a natural space between words as a segmentation criterion, using spaces to segment Latin letters is not only accurate, but also fast and convenient. However, for languages ​​that do not have spaces as a basis for word segmentation, such as Chinese and Japanese, word segmentation using word separators can have certain errors, resulting in poor accuracy. Therefore, it is necessary to consider adopting a universal deep word segmentation method, such as using a universal word segmentation model to process the text to be processed. This method can not only quickly split words based on spaces, but also split closely connected characters based on semantics, thereby improving the accuracy of multilingual text segmentation and thus improving the accuracy of identifying the primary language of the text to be processed based on the target high-frequency word lists for each second language. The universal word segmentation model can be a deep neural network trained in various languages, or other network models, without limitation here.

[0085] S504 : Determine the proportion of high-frequency words in each second language in the text to be processed based on the target high-frequency word list of each second language and the word segmentation information of the text to be processed.

[0086] In one possible embodiment, since the word segmentation information of the text to be processed includes one or more text words, the high-frequency words and the total number of text words included in the text to be processed can be determined based on these one or more text words. The determination of high-frequency words requires combining a target high-frequency stop word list for each second language. For each second language, the text words in the word segmentation information are matched with the high-frequency words in the target high-frequency stop word list. Text words that are identical to those in the target high-frequency stop word list are then determined as high-frequency words. The high-frequency words and the number of high-frequency words in each second language are then determined. The ratio of the number of high-frequency words to the total number of text words can then be used as the high-frequency word ratio for the second language.

[0087] For example, "Ayúdame a desbloquear mi ID de PUBG lo antes posible" involves English abbreviations and Spanish. It can be segmented according to spaces, and the total number of words in the text is 10. The high-frequency words include Ayúdame, desbloquear, mi, lo, and de. Since English is also common in other languages, there is no corresponding high-frequency word list, and no corresponding high-frequency words. Therefore, among the high-frequency word proportions of each second language, the high-frequency proportion of Spanish is 50%, while the high-frequency word proportions of Turkish and Italian are both 0.

[0088] S505: If the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to a second set threshold, the second language corresponding to the maximum high-frequency word proportion is determined as the target language corresponding to the text to be processed.

[0089] The specific implementation of this step can be found in the above Figure 4 S404 in the corresponding embodiment will not be described again here.

[0090] In summary, the embodiments of the present application have at least the following advantages:

[0091] In the process of using the high-frequency word proportion and the second set threshold to determine the target language of the text to be processed, the target high-frequency word list of each second language and the word segmentation information of the text to be processed are both key to text language recognition. It is necessary to obtain a sufficiently accurate high-frequency word distribution and text segmentation to ensure that the high-frequency word proportion and text segmentation are accurate, thereby ensuring a more accurate final recognition result. Among them, since high-frequency words between different second languages ​​may be cross-shared, which will affect the effectiveness of the high-frequency word distribution, the specific measure taken is to delete high-frequency words shared by multiple languages. In addition, various methods can be used to assist in improving the accuracy of the high-frequency word distribution, such as removing single-character high-frequency words, adding language-specific high-frequency words, and introducing high-frequency words specific to business types, thereby increasing the discrimination of the target high-frequency word list of each second language. In this way, when using the target high-frequency words to match the word segmentation information of the text to be processed, the high-frequency word proportion can be quickly and accurately determined, thereby determining the primary language corresponding to the text to be processed in each second language. In addition, for the processing of word segmentation information of the text to be processed, a universal word segmentation model is used to process the text to be processed in multiple languages, which can greatly reduce the error of word segmentation and efficiently and accurately identify the main language of the text to be processed.

[0092] See Figure 6 , is a structural diagram of a text processing device provided in an embodiment of the present application, the text processing device may be running on Figure 1 A computer program (including program code) in the text processing device 100 shown, such as a text processing device, is an application software; the device can be used to perform the corresponding steps of the method provided in the embodiment of the present application. The text processing device 60 includes: an acquisition module 601 and a determination module 602, wherein:

[0093] An acquisition module 601 is configured to acquire a text to be processed and determine a percentage of characters in each first language in the text to be processed;

[0094] A determination module 602 is configured to, if there is a first language with a character ratio greater than or equal to a first set threshold, determine the first language with a character ratio greater than or equal to the first set threshold as the target language corresponding to the text to be processed;

[0095] The determination module 602 is further configured to determine the proportion of high-frequency words in each second language in the text to be processed if there is no first language with a character proportion greater than or equal to the first set threshold;

[0096] The determination module 602 is further configured to determine the second language corresponding to the maximum high-frequency word proportion as the target language corresponding to the text to be processed if the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to a second set threshold.

[0097] In one embodiment, the determination module 602 is specifically used to: obtain the character code corresponding to each character in the text to be processed; determine the character ratio of each first language in the text to be processed based on the character code range of each first language, the character code corresponding to each character in the text to be processed, and the total number of characters in the text to be processed.

[0098] In one embodiment, the determination module 602 is specifically used to: determine a target high-frequency word list for each second language and obtain word segmentation information of the text to be processed; and determine the proportion of high-frequency words in each second language in the text to be processed based on the target high-frequency word list for each second language and the word segmentation information of the text to be processed.

[0099] In one embodiment, the determination module 602 is specifically used to: obtain a candidate high-frequency word list of the second language for any second language, where the candidate high-frequency word list includes one or more candidate high-frequency words; eliminate one or more single-character candidate high-frequency words and common candidate high-frequency words in the candidate high-frequency word list to obtain a target high-frequency word list of the second language, where the common candidate high-frequency words refer to candidate high-frequency words that appear in the candidate high-frequency word lists of each second language a number of times greater than or equal to a set number of times.

[0100] In one embodiment, the text processing apparatus 60 further includes an adding module 603, wherein:

[0101] Determination module 602, used to determine the business type indicated by the text to be processed;

[0102] An acquisition module 601 is used to acquire high-frequency words related to the business type;

[0103] The adding module 603 is used to add high-frequency words related to the business type to the target high-frequency word list.

[0104] In one embodiment, the acquisition module 601 is specifically used to: perform word segmentation processing on the text to be processed according to word separators, and use the obtained one or more words as the word segmentation information of the text to be processed; or perform word segmentation processing on the text to be processed using a word segmentation model, and use the obtained one or more words as the word segmentation information of the text to be processed.

[0105] In one embodiment, the determination module 602 is further configured to: if the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is less than a second set threshold, determine the default language as the target language corresponding to the text to be processed.

[0106] It is understood that the functions of the various functional modules of the text processing device described in the embodiments of the present application can be specifically implemented according to the relevant methods in the above-mentioned text processing method embodiments. The specific implementation process can be referred to the relevant description of the above-mentioned method embodiments and will not be repeated here. In addition, the description of the beneficial effects of adopting the same text processing method will not be repeated here.

[0107] See Figure 7 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 70 may include a processor 701, a memory 702, a network interface 703, and at least one communication bus 704. Processor 701 is used to schedule computer programs and may include a central processing unit, a controller, or a microprocessor; memory 702 is used to store computer programs and may include high-speed random access memory, non-volatile memory such as a disk storage device or a flash memory device; network interface 703 provides data communication functions, and communication bus 704 is responsible for connecting various communication components. This computer device 70 corresponds to the text processing device 100 described above.

[0108] The processor 701 may be configured to call a computer program in the memory to perform the following operations:

[0109] Obtaining the text to be processed and determining the character ratio of each first language in the text to be processed;

[0110] If there is a first language whose character ratio is greater than or equal to the first set threshold, determining the first language whose character ratio is greater than or equal to the first set threshold as the target language corresponding to the text to be processed;

[0111] If there is no first language with a character ratio greater than or equal to the first set threshold, determining the ratio of high-frequency words in each second language in the text to be processed;

[0112] If the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to the second set threshold, the second language corresponding to the maximum high-frequency word proportion is determined as the target language corresponding to the text to be processed.

[0113] In one embodiment, the processor 701 is specifically used to: obtain the character code corresponding to each character in the text to be processed; determine the character ratio of each first language in the text to be processed based on the character code range of each first language, the character code corresponding to each character in the text to be processed, and the total number of characters in the text to be processed.

[0114] In one embodiment, the processor 701 is specifically used to: determine a target high-frequency word list for each second language, and obtain word segmentation information of the text to be processed; and determine the proportion of high-frequency words in each second language in the text to be processed based on the target high-frequency word list for each second language and the word segmentation information of the text to be processed.

[0115] In one embodiment, the processor 701 is specifically used to: obtain a candidate high-frequency word list of the second language for any second language, the candidate high-frequency word list including one or more candidate high-frequency words; eliminate one or more single-character candidate high-frequency words and common candidate high-frequency words in the candidate high-frequency word list to obtain a target high-frequency word list of the second language, the common candidate high-frequency words refer to candidate high-frequency words that appear in the candidate high-frequency word list of each second language a number of times greater than or equal to a set number of times.

[0116] In one embodiment, the processor 701 is further configured to: determine the business type indicated by the text to be processed; obtain high-frequency words related to the business type; and add the high-frequency words related to the business type to a target high-frequency word list.

[0117] In one embodiment, the processor 701 is specifically used to: perform word segmentation processing on the text to be processed according to word separators, and use the obtained one or more words as the word segmentation information of the text to be processed; or perform word segmentation processing on the text to be processed using a word segmentation model, and use the obtained one or more words as the word segmentation information of the text to be processed.

[0118] In one embodiment, the processor 701 is further configured to: if the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is less than a second set threshold, determine the default language as the target language corresponding to the text to be processed.

[0119] In a specific implementation, the processor 701, memory 702, and network interface 703 described in the embodiment of the present application can execute the implementation of the computer device described in a text processing method provided in an embodiment of the present application, and can also execute the implementation and beneficial effects described in a text processing device provided in an embodiment of the present application, which will not be repeated here.

[0120] The present application also provides a computer-readable storage medium having instructions stored therein. When executed on a computer, the computer executes the text processing method described in the present application. The specific implementation method can be found in the previous description and will not be further elaborated here.

[0121] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text processing method according to the present application. The specific implementation method is described above and will not be repeated here.

[0122] It should be noted that for the aforementioned various method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0123] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0124] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A text processing method, characterized in that: include: Obtaining a text to be processed, and determining a ratio of characters in each first language in the text to be processed; for each first language, the characters contained in the first language do not belong to any language other than the first language; If there is a first language whose character ratio is greater than or equal to the first set threshold, determining the first language whose character ratio is greater than or equal to the first set threshold as the target language corresponding to the text to be processed; If there is no first language whose character ratio is greater than or equal to the first set threshold, determining the ratio of high-frequency words of each second language in the text to be processed; for each second language, some or all of the characters contained in the second language also belong to a language other than the second language; If the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to a second set threshold, the second language corresponding to the maximum high-frequency word proportion is determined as the target language corresponding to the text to be processed.

2. The method according to claim 1, wherein Determining the proportion of characters in each first language in the text to be processed includes: Obtaining the character code corresponding to each character in the text to be processed; The proportion of characters of each first language in the text to be processed is determined according to the character encoding range of each first language, the character encoding corresponding to each character in the text to be processed, and the total number of characters in the text to be processed.

3. The method according to claim 1, wherein Determining the proportion of high-frequency words in each second language in the text to be processed includes: Determining a target high-frequency word list for each second language, and obtaining word segmentation information of the text to be processed; The proportion of high-frequency words in each second language in the text to be processed is determined according to the target high-frequency word list of each second language and the word segmentation information of the text to be processed.

4. The method according to claim 3, wherein Determining the target high-frequency word list for each second language includes: For any second language, obtaining a candidate high-frequency word list of the second language, wherein the candidate high-frequency word list includes one or more candidate high-frequency words; One or more of the single-character candidate high-frequency words and the common candidate high-frequency words in the candidate high-frequency word list are eliminated to obtain a target high-frequency word list of the second language. The common candidate high-frequency words refer to candidate high-frequency words that appear in the candidate high-frequency word list of each second language a number of times greater than or equal to a set number of times.

5. The method according to claim 4, wherein The method further comprises: Determining the business type indicated by the text to be processed; Obtain high-frequency words related to the business type; The high-frequency words related to the business type are added to the target high-frequency word list.

6. The method according to claim 3, wherein The step of obtaining word segmentation information of the text to be processed includes: Performing word segmentation on the text to be processed according to word separators, and using the obtained one or more words as word segmentation information of the text to be processed; or The word segmentation model is used to perform word segmentation processing on the text to be processed, and the obtained one or more words are used as word segmentation information of the text to be processed.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: If the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is less than the second set threshold, the default language is determined as the target language corresponding to the text to be processed.

8. A text processing device, characterized in that: include: an acquisition module, configured to acquire a text to be processed and determine a proportion of characters in each first language in the text to be processed; for each first language, the characters contained in the first language do not belong to any language other than the first language; a determination module configured to, if there is a first language whose character ratio is greater than or equal to a first set threshold, determine the first language whose character ratio is greater than or equal to the first set threshold as the target language corresponding to the text to be processed; The determination module is further configured to determine, if there is no first language with a character ratio greater than or equal to the first set threshold, a ratio of high-frequency words in each second language in the to-be-processed text; for each second language, some or all of the characters contained in the second language also belong to a language other than the second language; The determination module is further configured to determine the second language corresponding to the maximum high-frequency word proportion as the target language corresponding to the text to be processed if the maximum high-frequency word proportion among the high-frequency word proportions of the second languages ​​is greater than or equal to a second set threshold.

9. A computer device, characterized in that: include: processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code to execute the text processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the text processing method according to any one of claims 1 to 7 is executed.

11. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the text processing method according to any one of claims 1 to 7.

Images