Text processing method and device, electronic equipment and storage medium

Abstract

The invention provides a text processing method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing and deep learning. According to the specific implementation scheme, for each original character in a to-be-processed text, an associated character set of the original characters is determined according to the original characters and at least one associated character generated based on the original characters; according to a co-occurrence relationship among a plurality of elements from the plurality of associated character sets, determining respective replacement characters of a plurality of original characters in the to-be-processed text; and generating a replacement text according to the respective replacement characters of the plurality of original characters.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of natural language processing and deep learning technology. More specifically, the present disclosure provides a text processing method, device, electronic device, storage medium and computer program product. Background Art

[0002] With the continuous development of computer and Internet technology, there is a large amount of information on the Internet. Some information often contains text with variant forms. Identifying information containing variant text to determine whether it contains sensitive information such as contact information is an important means to control advertising diversion and cheating and maintain the network environment. Summary of the invention

[0003] The present disclosure provides a text processing method, device, electronic device, storage medium and computer program product.

[0004] According to a first aspect, a text processing method is provided, the method comprising: for each original character in a text to be processed, determining a set of associated characters of the original character according to the original character and at least one associated character generated based on the original character; determining replacement characters for each of the multiple original characters in the text to be processed according to a co-occurrence relationship between multiple elements respectively from the multiple associated character sets; and generating replacement text according to the replacement characters for each of the multiple original characters.

[0005] According to a second aspect, a text processing device is provided, which includes: an associated character set determination module, which is used to determine, for each original character in a text to be processed, an associated character set of the original character based on the original character and at least one associated character generated based on the original character; a replacement character determination module, which is used to determine, based on the co-occurrence relationship between multiple elements respectively from multiple associated character sets, replacement characters for each of multiple original characters in the text to be processed; and a replacement text determination module, which is used to generate replacement texts based on the replacement characters for each of the multiple original characters.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a text processing method and apparatus can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of a text processing method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of an associated character linked list according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of a text processing method according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of a text processing method according to another embodiment of the present disclosure;

[0016] Figure 6 is a block diagram of a text processing apparatus according to an embodiment of the present disclosure; and

[0017] Figure 7 The present invention is a block diagram of an electronic device of a text processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] Identifying contact information in text is a key technology for ad traffic management. Contact information includes, for example, phone numbers, email addresses, social accounts, etc. However, text containing contact information often avoids being identified by using variant forms, which makes identification difficult.

[0020] Currently, variant texts can be identified by training a deep learning model to learn the features of variant texts. For example, a contact information recognition model is obtained by training with samples containing variant contact information. However, the context of the variant contact information is normal text, and the model cannot learn the variant features well during learning, resulting in poor model recognition accuracy.

[0021] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0023] Figure 1 is a schematic diagram of an exemplary system architecture to which a text processing method and apparatus can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0024] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0025] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0026] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results to the terminal device.

[0027] The text processing method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the text processing device provided by the embodiments of the present disclosure can generally be set in the server 105. The text processing method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103, and / or the server 105. Correspondingly, the text processing device provided by the embodiments of the present disclosure can also be set in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103, and / or the server 105.

[0028] Figure 2 is a flowchart of a text processing method according to an embodiment of the present disclosure.

[0029] As Figure 2 shown, the text processing method 200 includes operations S210 to S230.

[0030] In operation S210, for each original character in the text to be processed, an associated character set of the original character is determined according to the original character and at least one associated character generated based on the original character.

[0031] The text to be processed can be text obtained from information such as published information, articles, comments, and push messages on the network. The text to be processed may contain characters in variant forms.

[0032] Each character in the text to be processed can be used as an original character first, and then for each original character, multiple variant forms of the original character are expanded to obtain multiple associated characters of the original character. The original character can be a normal character without drainage cheating behavior or a variant character with drainage cheating behavior. For example, if the text to be processed is "clip skirt chat", the original characters "clip" and "skirt" are variant characters, and "chat" is a normal character.

[0033] For the original character "clip", multiple associated characters can be generated through various variant forms such as homophonic characters, homographic characters, and homophonic symbols. For example, associated characters such as "add", "home", "+", and an emoji of a house graphic (representing home) can be generated.

[0034] The original character "clip" and the associated characters "add", "home", "+", and an emoji of a house graphic (representing home) can form the associated character set of the original character "clip".

[0035] In operation S220, according to the co-occurrence relationship between multiple elements from multiple associated character sets, replacement characters for each of the multiple original characters in the text to be processed are determined.

[0036] For example, multiple associated character sets may include an associated character set of the original character "clip", an associated character set of the original character "skirt", and an associated character set of the original character "chat".

[0037] Any combination can be made for elements respectively from different associated character sets to obtain combined phrases. For example, combined phrases may include "clip skirt chat", "add skirt chat", "add group chat", etc. For each combined phrase, the co-occurrence frequency of multiple characters (such as 3 characters) in the phrase in the corpus can be calculated, and the multiple characters in the phrase with the highest co-occurrence frequency are respectively determined as replacement characters for the multiple original characters in the text to be processed.

[0038] For example, the three characters "add", "group", and "chat" in the phrase "add group chat" have the highest co-occurrence frequency in the corpus. Therefore, the three characters "add", "group", and "chat" can be respectively used as replacement characters for the three characters "clip", "skirt", and "chat" in the text to be processed.

[0039] In operation S230, replacement text is generated according to the replacement characters of the multiple original characters respectively.

[0040] For example, the original characters "clip skirt chat" in the text to be processed can be replaced with "add group chat", and "add group chat" is the replacement text. The replacement text can restore the true information of the text to be processed, facilitating subsequent processing such as sensitive information recognition and contact information recognition for anti-advertisement drainage and cheating.

[0041] According to an embodiment of the present disclosure, for each original character in the text to be processed, at least one associated character of each original character is generated, the original character and the at least one associated character form an associated character set, and according to the co-occurrence relationship of multiple elements respectively from multiple associated character sets in the corpus, the multiple elements with the highest co-occurrence frequency are determined as replacement characters for the multiple original characters in the text to be processed, the original characters are replaced with the replacement characters to obtain replacement text, and the replacement text can restore the true information of the text to be processed and improve the accuracy of subsequent text recognition.

[0042] Figure 3 It is a schematic diagram of an associated character linked list according to an embodiment of the present disclosure.

[0043] According to an embodiment of the present disclosure, for each original character, at least one associated character of the original character is generated according to at least one variant type, where the variant type includes at least one of homophonic characters, homophonic characters, homographic characters, homophonic numbers, and homophonic symbols; and an associated character set is determined according to the original character and the at least one associated character.

[0044] Such as Figure 3As shown, taking the original character "夹" as an example, according to various variant types such as homophonic characters, homonymic characters, homomorphic characters, homophonic numbers, homophonic symbols, etc., multiple associated characters such as "家", "加", "+", "jia", and the graphic or expression of "家" can be generated. The original character "夹" and each associated character can form an associated character set 310.

[0045] According to an embodiment of the present disclosure, for each associated character set, determine the frequency of occurrence of each associated character in the corpus for at least one associated character in the associated character set; and taking the original character in the associated character set as the starting node and each associated character in the associated character set as the associated nodes, sort the associated nodes according to the frequency of occurrence of each associated character in the corpus to obtain an associated character linked list.

[0046] For example, taking the original character "夹" as the starting node and each associated character as the associated nodes, calculate the frequency of occurrence of each associated character in the corpus, and sort the associated nodes in descending order according to the frequency of occurrence of the associated characters in the corpus to obtain an associated character linked list 320.

[0047] For example, the frequencies of occurrence of associated characters such as "加", "家", …, "+" in the corpus decrease in turn. Therefore, they can be arranged in turn behind the original character "夹" to obtain an associated character linked list 320.

[0048] According to an embodiment of the present disclosure, arranging the associated characters in descending order of the frequency of occurrence in the corpus facilitates quickly combining the characters with the highest co-occurrence frequency during subsequent character combination, thereby improving the efficiency of determining replacement characters.

[0049] The following combines Figures 4 and 5 to illustrate the determination of replacement characters in the embodiments of the present disclosure.

[0050] Figure 4 is a schematic diagram of a text processing method according to an embodiment of the present disclosure.

[0051] As Figure 4 shown, the associated character linked lists 410, 420, and 430 are the associated character linked lists of the original characters "夹", "裙", and "聊" respectively. The associated characters in each associated character linked list are arranged in descending order according to the frequency of occurrence in the corpus.

[0052] According to an embodiment of the present disclosure, combine multiple elements from multiple associated character sets respectively to obtain multiple first phrases; for each first phrase, calculate the co-occurrence frequency of the multiple elements in the corpus; and determine the multiple elements in the first phrase with the highest co-occurrence frequency as the replacement characters for the multiple original characters in the text to be processed respectively.

[0053] For example, elements from the associated character linked list 410, elements from the associated character linked list 420, and elements from the associated character linked list 430 can be combined arbitrarily to obtain multiple first phrases. The first phrases include, for example, "pin group chat", "pin group", "join group chat", etc. For each first phrase, the co-occurrence frequency of multiple characters in the group in the corpus can be calculated, and the element in the first phrase with the highest co-occurrence frequency is determined as the replacement character for the original character.

[0054] For example, to improve the efficiency of determining the replacement character, the associated character linked list can be divided into multiple character blocks according to the number of tasks that can be executed in parallel each time, and the combination is performed in sequence according to the order of the character blocks. During the combination of the characters in each character block, the co-occurrence frequency of the characters in the first phrase combined in real time can be calculated.

[0055] For example, if the number of tasks that can be executed in parallel each time is 3, then "pin", "join", "home" in the associated character linked list 410, "skirt", "group", "qun" in the associated character linked list 420, and "chat", "le", "liao" in the associated character linked list 430 can be divided into the first character block. Use "pin", "join", "home" in the associated character linked list 410 to be combined with "skirt", "group", "qun" in the associated character linked list 420 and "chat", "le", "liao" in the associated character linked list 430 in parallel to obtain multiple first phrases.

[0056] During the process of combining the characters in the first character block, the co-occurrence frequency of multiple elements in the first phrase combined in real time in the corpus can be calculated. A threshold for the co-occurrence frequency can be set (for example, 70%). Among all the character combination results of the first character block, the first phrase with the co-occurrence frequency reaching the threshold can be used as a candidate phrase, and then the first phrase with the highest co-occurrence frequency among the candidate phrases can be selected as the target first phrase. After obtaining the target first phrase, the character combination of the subsequent character blocks can be stopped, and the elements in the target first phrase are used as the replacement characters for the original characters. Since the associated characters in the associated character linked list are arranged in descending order of the frequency of appearance in the corpus, the first phrase with the highest co-occurrence frequency is likely to be generated in the character combination of the character block with a higher ranking. Therefore, the first phrase that has exceeded the threshold and has the highest co-occurrence frequency can be used as the first target phrase, which can improve the efficiency of determining the first target phrase. The elements in the first target phrase are the replacement characters for the original characters, so the efficiency of determining the replacement characters can be improved.

[0057] For example, "join group chat" is the combination result of characters in the first character block, which was generated earlier, has a co-occurrence frequency exceeding the threshold, and has the highest co-occurrence frequency among all the character combination results in the first character block. Therefore, "join group chat" can be determined as the replacement text for the text to be processed "clip skirt chat".

[0058] In an embodiment of the present disclosure, the associated characters in the associated character linked list are arranged in descending order of the frequency of occurrence in the corpus. By combining according to the associated character linked list, a plurality of first phrases are obtained, and the co-occurrence frequency of a plurality of elements in the first phrase in the corpus is calculated. The plurality of elements with the highest co-occurrence frequency are determined as the replacement characters of the original characters, which can improve the determination efficiency of the replacement characters and the replacement text.

[0059] Figure 5 It is a schematic diagram of a text processing method according to another embodiment of the present disclosure.

[0060] According to an example of the present disclosure, for two adjacent associated character sets respectively corresponding to two adjacent original characters, according to the co-occurrence relationship between two elements respectively from the two adjacent associated character sets, the replacement characters of the two adjacent original characters are determined respectively.

[0061] As Figure 5 shown, in this embodiment, for every two adjacent original characters, the implementation manner of character combination and determination of replacement characters is carried out according to two adjacent associated character linked lists. For the text to be processed with a relatively large number of characters, the co-occurrence frequency of multiple characters may be relatively low. Therefore, for every two adjacent original characters, the replacement characters can be determined in turn according to the co-occurrence frequency.

[0062] According to an embodiment of the present disclosure, each element in the first associated character set is combined with each element in the second associated character set respectively to obtain a plurality of second phrases; for each second phrase, the co-occurrence frequency of the two elements in the second phrase in the corpus is calculated; and the two elements in the second phrase with the highest co-occurrence frequency are respectively determined as the replacement characters of the two adjacent original characters.

[0063] For example, the first associated character set may be the associated character linked list 510, and the second associated character set may be the associated character linked list 520. The second phrase "join group" can be determined as the target second phrase in a manner similar to the character combination and determination of replacement characters with the above three associated character linked lists, that is, the element "join" is the replacement character of the original character "clip", and the element "group" is the replacement character of the original character "skirt".

[0064] Next, the first set of associated characters can be determined as the linked list of associated characters 520, and the second set of associated characters can be determined as the linked list of associated characters 530. Since it has been determined that "group" in the linked list of associated characters 520 is the replacement character for the original character "skirt", therefore, "group" in the linked list of associated characters 520 can be directly combined with each element in the linked list of associated characters 530, and the replacement character for the original character "chat" can be determined according to the co-occurrence frequency.

[0065] Alternatively, each element in the linked list of associated characters 520 can be combined with each element in the linked list of associated characters 530 to obtain multiple second word groups. At the same time, the co-occurrence frequency of the second word groups containing "group" is weighted, and then the second word group with the highest co-occurrence frequency is determined from all the second word groups as the second target word group. The two elements in the second target word group are used as the replacement characters for the character "skirt" and the character "chat" respectively.

[0066] For example, the target second word group "group chat" is the replacement character for the original character "skirt chat". Combining with the replacement character "add" for the original character "clip", the replacement text "add group chat" can be obtained.

[0067] The embodiments of the present disclosure can restore the real text information for long texts and improve the determination efficiency of replacement characters and replacement texts by performing character combination and determination of replacement characters through the linked list of associated characters for every two adjacent original characters.

[0068] According to the embodiments of the present disclosure, sensitive information recognition is performed on the replacement text to obtain a recognition result, including: using a deep learning model to perform contact information recognition on the replacement text to obtain a recognition result indicating whether the replacement text contains contact information.

[0069] After converting the text to be processed into a replacement text, the replacement text restores the real information of the text to be processed. Therefore, the replacement text can be recognized to identify whether it contains sensitive or illegal information.

[0070] In one example, a deep learning model can be used to perform contact information recognition on the replacement text to identify whether the replacement text contains contact information. The contact information can include phone numbers, email addresses, social accounts, etc. If it is recognized that the replacement text contains contact information, it can be determined that the text to be processed belongs to text for advertising drainage or cheating.

[0071] The above-mentioned deep learning model can be trained using contact sample text containing normal text. Compared with the model trained using contact sample text containing variant text in the related art, which is difficult to learn variant features and leads to poor text recognition accuracy, the embodiment of the present disclosure uses a deep learning model trained using contact sample text containing normal text to recognize replacement text containing normal text, which can improve the text recognition accuracy.

[0072] According to an embodiment of the present disclosure, the present disclosure also provides a text processing device.

[0073] Figure 6 is a block diagram of a text processing apparatus according to an embodiment of the present disclosure.

[0074] like Figure 6 As shown, the text processing device 600 includes an associated character set determination module 610 , a replacement character determination module 620 and a replacement text determination module 630 .

[0075] The associated character set determination module 610 is used to determine, for each original character in the text to be processed, an associated character set of the original character according to the original character and at least one associated character generated based on the original character.

[0076] The replacement character determination module 620 is used to determine replacement characters for each of the multiple original characters in the to-be-processed text according to the co-occurrence relationship between the multiple elements respectively from the multiple associated character sets.

[0077] The replacement text determination module 630 is used to generate replacement text according to the replacement characters of each of the multiple original characters.

[0078] The replacement character determination module 620 includes a first combining unit, a first co-occurrence frequency determination unit, and a first replacement character determination unit.

[0079] The first combining unit is used to combine multiple elements from multiple associated character sets to obtain multiple first phrases.

[0080] The first co-occurrence frequency determination unit is used to calculate, for each first phrase, the co-occurrence frequency of multiple elements in the first phrase in the corpus.

[0081] The first replacement character determination unit is used to determine replacement characters for each of the plurality of original characters in the to-be-processed text according to the first phrase with the highest co-occurrence frequency.

[0082] The first replacement character determination unit is used to determine the multiple elements in the first phrase with the highest co-occurrence frequency as replacement characters for each of the multiple original characters in the text to be processed.

[0083] The replacement character determination module 620 is further configured to determine, for each of the two adjacent associated character sets corresponding to the two adjacent original characters, a replacement character according to a co-occurrence relationship between two elements from the two adjacent associated character sets.

[0084] The two adjacent associated character sets include a first associated character set and a second associated character set. The replacement character determination module 620 includes a second combination unit, a second co-occurrence frequency determination unit and a second replacement character determination unit.

[0085] The second combining unit is used to combine each element in the first associated character set with each element in the second associated character set to obtain a plurality of second phrases.

[0086] The second co-occurrence frequency determination unit is used to calculate, for each second phrase, the co-occurrence frequency of two elements in the second phrase in the corpus.

[0087] The second replacement character determination unit is used to determine replacement characters for each of two adjacent original characters according to the second phrase with the highest co-occurrence frequency.

[0088] The second replacement character determination unit is used to determine two elements in the second phrase with the highest co-occurrence frequency as replacement characters for two adjacent original characters respectively.

[0089] The associated character set determination module 610 includes an associated character generation unit and an associated character set determination unit.

[0090] The associated character generating unit is used to generate at least one associated character of each original character according to at least one variant type, wherein the variant type includes at least one of homophone characters, homophone characters, homographic characters, homophone numbers, and homophone symbols.

[0091] The associated character set determining unit is used to determine an associated character set according to the original character and at least one associated character.

[0092] According to an embodiment of the present disclosure, the text processing device 600 further includes a frequency calculation module and an associated character list determination module.

[0093] The frequency calculation module is used to determine, for each associated character set, a frequency of occurrence of each associated character in at least one associated character in the associated character set in the corpus.

[0094] The associated character list determination module is used to take the original character in the associated character set as the starting node, take each associated character in the associated character set as an associated node, sort the associated nodes according to the frequency of each associated character appearing in the corpus, and obtain an associated character list.

[0095] The text processing device 600 further includes a recognition module.

[0096] The recognition module is used to identify sensitive information in the replacement text and obtain a recognition result.

[0097] The recognition module is also used to use a deep learning model to identify contact information for the replacement text, and obtain a recognition result indicating whether the replacement text contains contact information.

[0098] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0099] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0100] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0101] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a text processing method. For example, in some embodiments, the text processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the text processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the text processing method in any other appropriate manner (e.g., by means of firmware).

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0108] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A text processing method, comprising: For each original character in the text to be processed, determine a set of associated characters of the original character according to the original character and at least one associated character generated based on the original character; Determining replacement characters for respective original characters in the to-be-processed text according to co-occurrence relationships between multiple elements from multiple associated character sets; as well as A replacement text is generated according to the replacement characters of the respective original characters.

2. The method according to claim 1, wherein: The step of determining, based on the co-occurrence relationship between the multiple elements from the multiple associated character sets, the replacement characters for the multiple original characters in the to-be-processed text comprises: Combining multiple elements from the multiple associated character sets to obtain multiple first phrases; For each first word group, calculating the co-occurrence frequency of multiple elements in the first word group in the corpus; and According to the first phrase with the highest co-occurrence frequency, replacement characters for each of the plurality of original characters in the to-be-processed text are determined.

3. The method according to claim 2, wherein: The step of determining, based on the first phrase with the highest co-occurrence frequency, replacement characters for each of the plurality of original characters in the to-be-processed text comprises: The multiple elements in the first phrase with the highest co-occurrence frequency are respectively determined as replacement characters for the multiple original characters in the text to be processed.

4. The method according to claim 1, wherein: The step of determining, based on the co-occurrence relationship between the multiple elements from the multiple associated character sets, the replacement characters for the multiple original characters in the to-be-processed text comprises: For two adjacent associated character sets corresponding to two adjacent original characters respectively, replacement characters for the two adjacent original characters are determined according to a co-occurrence relationship between two elements from the two adjacent associated character sets respectively.

5. The method according to claim 4, wherein: The two adjacent associated character sets include a first associated character set and a second associated character set; and determining the replacement characters of the two adjacent original characters according to the co-occurrence relationship between two elements from the two adjacent associated character sets includes: Combining each element in the first associated character set with each element in the second associated character set to obtain a plurality of second phrases; For each second word group, calculating the co-occurrence frequency of two elements in the second word group in the corpus; and According to the second phrase with the highest co-occurrence frequency, replacement characters for each of the two adjacent original characters are determined.

6. The method according to claim 5, wherein: The step of determining the replacement characters of the two adjacent original characters according to the second phrase with the highest co-occurrence frequency comprises: The two elements in the second phrase with the highest co-occurrence frequency are respectively determined as replacement characters for the two adjacent original characters.

7. The method according to claim 1, wherein: The step of determining, for each original character in the to-be-processed text, a set of associated characters of the original character according to the original character and at least one associated character generated based on the original character comprises: For each original character, generating at least one associated character of the original character according to at least one variant type, wherein the variant type includes at least one of homophone characters, homophonic characters, homographic characters, homophone numbers, and homophone symbols; and The associated character set is determined according to the original character and the at least one associated character.

8. The method according to claim 1, further comprising: For each associated character set, determining a frequency of occurrence of each associated character in the at least one associated character in the associated character set in the corpus; as well as The original character in the associated character set is used as the starting node, each associated character in the associated character set is used as an associated node, and the associated nodes are sorted according to the frequency of each associated character appearing in the corpus to obtain an associated character linked list.

9. The method according to claim 1, further comprising: Sensitive information is identified on the replacement text to obtain an identification result.

10. The method according to claim 9, wherein: The performing of sensitive information identification on the replacement text to obtain an identification result includes: The contact information of the replacement text is identified using a deep learning model to obtain an identification result indicating whether the replacement text contains the contact information.

11. A text processing device, comprising: An associated character set determination module, configured to determine, for each original character in the text to be processed, an associated character set of the original character according to the original character and at least one associated character generated based on the original character; A replacement character determination module, used for determining replacement characters for each of the plurality of original characters in the to-be-processed text according to a co-occurrence relationship between a plurality of elements respectively from a plurality of associated character sets; as well as The replacement text determination module is used to generate replacement text according to the replacement characters of the multiple original characters.

12. The device according to claim 11, wherein The replacement character determination module comprises: A first combining unit, used for combining a plurality of elements respectively from the plurality of associated character sets to obtain a plurality of first phrases; A first co-occurrence frequency determination unit, configured to calculate, for each first phrase, a co-occurrence frequency of a plurality of elements in the first phrase in a corpus; and The first replacement character determination unit is used to determine replacement characters for each of the plurality of original characters in the to-be-processed text according to the first phrase with the highest co-occurrence frequency.

13. The device according to claim 12, wherein: The first replacement character determination unit is used to determine the multiple elements in the first phrase with the highest co-occurrence frequency as replacement characters for each of the multiple original characters in the text to be processed.

14. The device according to claim 11, wherein: The replacement character determination module is further used to determine, for two adjacent associated character sets corresponding to two adjacent original characters respectively, replacement characters for the two adjacent original characters according to a co-occurrence relationship between two elements from the two adjacent associated character sets respectively.

15. The device according to claim 14, wherein: The two adjacent associated character sets include a first associated character set and a second associated character set; and the replacement character determination module includes: A second combining unit, used to combine each element in the first associated character set with each element in the second associated character set to obtain a plurality of second phrases; A second co-occurrence frequency determination unit, configured to calculate, for each second phrase, a co-occurrence frequency of two elements in the second phrase in the corpus; and The second replacement character determination unit is used to determine the replacement characters of the two adjacent original characters according to the second phrase with the highest co-occurrence frequency.

16. The device according to claim 15, wherein: The second replacement character determination unit is used to determine two elements in the second phrase with the highest co-occurrence frequency as replacement characters for each of the two adjacent original characters.

17. The device according to claim 11, wherein: The associated character set determination module comprises: an associated character generating unit, configured to generate, for each original character, at least one associated character of the original character according to at least one variant type, wherein the variant type includes at least one of homophone characters, homophonic characters, homographic characters, homophone numbers, and homophone symbols; and The associated character set determining unit is used to determine the associated character set according to the original character and the at least one associated character.

18. The apparatus according to claim 11, further comprising: A frequency calculation module, for determining, for each associated character set, a frequency of occurrence of each associated character in the at least one associated character in the associated character set in the corpus; as well as The associated character list determination module is used to take the original character in the associated character set as the starting node, take each associated character in the associated character set as an associated node, sort the associated nodes according to the frequency of each associated character appearing in the corpus, and obtain an associated character list.

19. The apparatus according to claim 11, further comprising: The recognition module is used to perform sensitive information recognition on the replacement text to obtain a recognition result.

20. The device according to claim 19, wherein The recognition module is used to use a deep learning model to identify the contact information of the replacement text, and obtain a recognition result indicating whether the replacement text contains the contact information.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images