Variant text recognition method and device, computer equipment and readable storage medium
By combining multi-text reconstruction chains and scoring rules, the problem of low accuracy in variant text recognition is solved, enabling efficient recognition and risk assessment of variant texts and improving the recognition accuracy of texts related to minors.
Patent Information
- Application Number
- CN202410941570.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
AI Technical Summary
Existing text recognition methods struggle to accurately identify transformed text, especially in online scenarios such as live streaming, resulting in low recognition accuracy for minors.
By acquiring multiple text reconstruction chains, the text to be identified is reconstructed according to the order of different text reconstruction nodes. Combining preset scoring rules and text recognition models, the density of preset phrases contained in variant texts is scored and identified to determine whether they are risky texts.
It improves the accuracy of identifying variant text, enhances recognition efficiency, and avoids the recognition risks caused by variant text.
Smart Images

Figure CN118917273A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of text recognition, and in particular to a variant text recognition method, apparatus, computer equipment, computer-readable storage medium and computer program product. Background Art
[0002] With the continuous development of science and technology, more and more minors are participating in the online world. Strengthening the identification and control of minors by various online platforms has become a top priority. For example, in the field of live broadcasting, it is necessary to identify risky texts.
[0003] However, in order to circumvent traditional identification methods, some accounts will transform risky texts to obtain variants or variants of the text, and then use the transformed text in online scenarios such as live broadcasts, making traditional identification methods for risky texts difficult to apply to variant identification for minors. Therefore, the current identification methods for risky texts have the problem of low recognition accuracy. Summary of the invention
[0004] Based on this, it is necessary to provide a variant text recognition method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the recognition accuracy of variant texts in response to the above technical problems.
[0005] In a first aspect, an embodiment of the present application provides a method for identifying variant text. The method comprises:
[0006] Get the text to be recognized;
[0007] Obtain at least two different text restoration chains, and restore the text to be recognized in sequence according to the text restoration rules corresponding to the text restoration nodes in the text restoration chains and in the execution order of the text restoration nodes to obtain at least two restored texts;
[0008] Scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each of the restored texts; the scoring value represents the density of a preset phrase contained in each of the restored texts, where the preset phrase is a phrase associated with minors;
[0009] According to the order of the score values corresponding to each restored text, each restored text is input into the text recognition model in turn; the text recognition model is used to output the recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text, and the risky text is a variant text obtained after the text associated with minors is transformed.
[0010] In one embodiment, before the step of restoring the text to be recognized in sequence, the method further includes:
[0011] According to a preset character conversion set, a first character in the text to be recognized is replaced with a second character corresponding to the first character to obtain the replaced text to be recognized;
[0012] The character conversion set includes a correspondence between the first character and the second character, and a usage frequency of the first character is less than a usage frequency of the second character.
[0013] In one embodiment, scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each restored text includes:
[0014] Identifying each entity phrase contained in each restored text;
[0015] In the case where the entity phrase is the preset phrase, increasing the score value of the restored text by the unit score value corresponding to the preset phrase;
[0016] The unit score values corresponding to each of the restored texts are accumulated to obtain the score values corresponding to each of the restored texts.
[0017] In one embodiment, obtaining at least two different text restoration chains includes:
[0018] Obtaining each of the pre-constructed text restoration nodes; the text restoration nodes are used to restore the text according to the corresponding text restoration rules;
[0019] According to the execution order of each text restoration node, each text restoration node is assembled to obtain at least two different assembled text restoration chains.
[0020] In one embodiment, assembling the text restoration nodes to obtain at least two different assembled text restoration chains includes:
[0021] Obtain the assembled text restoration chain, and determine the node to be added from each of the text restoration nodes;
[0022] The assembled text restoration chain and the node to be added are reassembled to obtain a composite text restoration chain.
[0023] In one embodiment, the text restoration node includes a text restoration node of a removal type, and the obtaining of each pre-constructed text restoration node includes:
[0024] Identify the target to be removed; the target to be removed includes: preset characters, preset punctuation marks, symbols at set positions, and continuous characters;
[0025] The text restoration rule corresponding to the text restoration node is set to delete the target to be removed in the text, so as to construct a text restoration node of the removal type according to the text restoration rule.
[0026] In one embodiment, the text restoration node includes a compressed text restoration node, and the obtaining of each pre-built text restoration node includes:
[0027] Identifying a target to be compressed; the target to be compressed includes continuous characters;
[0028] The text restoration rule corresponding to the text restoration node is set to compress the length of the characters to be compressed in the text to a target length, so as to construct a text restoration node of the compression type according to the text restoration rule.
[0029] In one embodiment, the text restoration node includes a mobile type text restoration node, and the obtaining of each pre-constructed text restoration node includes:
[0030] Identifying a target to be moved; the target to be moved includes a character belonging to a target character type; the target character type includes a numeral type;
[0031] The text restoration rule corresponding to the text restoration node is set to transfer the characters to be moved in the text to the beginning or the end of the text, so as to construct a text restoration node of the moving type according to the text restoration rule.
[0032] In a second aspect, the present application also provides a variant text recognition device. The device comprises:
[0033] A text acquisition module is used to acquire the text to be recognized;
[0034] A text restoration module is used to obtain at least two different text restoration chains, and restore the text to be recognized in sequence according to the text restoration rules corresponding to each text restoration node in the text restoration chain and the execution order of each text restoration node to obtain at least two restored texts;
[0035] A text scoring module, used for scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each of the restored texts; the scoring value represents the density of a preset phrase contained in each of the restored texts, wherein the preset phrase is a phrase associated with minors;
[0036] A model recognition module is used to input each restored text into a text recognition model in order of the size of the score values corresponding to each restored text; the text recognition model is used to output a recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text, and the risky text is a variant text obtained after the text associated with minors is transformed.
[0037] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0038] Get the text to be recognized;
[0039] Obtain at least two different text restoration chains, and restore the text to be recognized in sequence according to the text restoration rules corresponding to the text restoration nodes in the text restoration chains and in the execution order of the text restoration nodes to obtain at least two restored texts;
[0040] Scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each of the restored texts; the scoring value represents the density of a preset phrase contained in each of the restored texts, where the preset phrase is a phrase associated with minors;
[0041] According to the order of the score values corresponding to each restored text, each restored text is input into the text recognition model in turn; the text recognition model is used to output the recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text, and the risky text is a variant text obtained after the text associated with minors is transformed.
[0042] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0043] Get the text to be recognized;
[0044] Obtain at least two different text restoration chains, and restore the text to be recognized in sequence according to the text restoration rules corresponding to the text restoration nodes in the text restoration chains and in the execution order of the text restoration nodes to obtain at least two restored texts;
[0045] Scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each of the restored texts; the scoring value represents the density of a preset phrase contained in each of the restored texts, where the preset phrase is a phrase associated with minors;
[0046] According to the order of the score values corresponding to each restored text, each restored text is input into the text recognition model in turn; the text recognition model is used to output the recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text, and the risky text is a variant text obtained after the text associated with minors is transformed.
[0047] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0048] Get the text to be recognized;
[0049] Obtain at least two different text restoration chains, and restore the text to be recognized in sequence according to the text restoration rules corresponding to the text restoration nodes in the text restoration chains and in the execution order of the text restoration nodes to obtain at least two restored texts;
[0050] Scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each of the restored texts; the scoring value represents the density of a preset phrase contained in each of the restored texts, where the preset phrase is a phrase associated with minors;
[0051] According to the order of the score values corresponding to each restored text, each restored text is input into the text recognition model in turn; the text recognition model is used to output the recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text, and the risky text is a variant text obtained after the text associated with minors is transformed.
[0052] The above-mentioned variant text recognition method, device, computer equipment, storage medium and computer program product first obtain the text to be recognized, then obtain at least two different text restoration chains, and restore the text to be recognized in turn according to the text restoration rules corresponding to each text restoration node in the text restoration chain and the execution order of each text restoration node to obtain at least two restored texts, and then score the at least two restored texts according to the preset scoring rules to obtain the scoring value corresponding to each restored text, and finally input each restored text into the text recognition model in order of the size of the corresponding scoring value of each restored text to analyze whether the text to be recognized corresponding to the original text is a risky text; on the basis of the existing text recognition model, a pre-process for restoring the variant text is added, which is improved in a convenient way, thereby improving the recognition efficiency, while avoiding the risks brought by the use of variant texts by some accounts, and improving the recognition accuracy of variant texts. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0054] FIG1 is a diagram showing an application environment of a variant text recognition method according to an embodiment;
[0055] Figure 2 A schematic diagram of a process for identifying variant text in one embodiment;
[0056] Figure 3 is a flow chart of a method for identifying variant text in another embodiment;
[0057] Figure 4 A schematic diagram of a text restoration chain of a variant text recognition method in one embodiment;
[0058] Figure 5 is a structural block diagram of a variant text recognition device in one embodiment;
[0059] FIG6 is a diagram showing the internal structure of a computer device in one embodiment;
[0060] FIG. 7 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0063] The variant text recognition method provided in the embodiment of the present application can be applied in the application environment shown in FIG1 . The terminal 102 communicates with the server 104 through a network.
[0064] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.
[0065] In one embodiment, Figure 2 As shown, a variant text recognition method is provided, which is described by taking the method applied to the terminal 102 in FIG. 1 as an example, and includes the following steps:
[0066] S201, obtaining text to be recognized.
[0067] The text to be identified is the text to be identified, and the goal of the identification is to detect whether there is risky text in the text, such as text related to minors. Specifically, in scenarios such as live broadcasting, risky text can be the name of the live broadcast room, the names of the members in the live broadcast room, the speeches of the members in the live broadcast room, and other texts.
[0068] Generally, the text to be identified may contain variant texts containing information related to minors.
[0069] S202, obtaining at least two different text restoration chains, and restoring the texts to be recognized in sequence according to the text restoration rules corresponding to the text restoration nodes in the text restoration chains and the execution order of the text restoration nodes to obtain at least two restored texts.
[0070] The number of text restoration chains is at least two. Generally, for a recognition task corresponding to the same text to be recognized, the corresponding text restoration chains are different.
[0071] Among them, the text restoration chain includes at least one text restoration node, which is also called a text restoration module. Each text restoration node corresponds to a text restoration rule, which is used to explain the rules for restoring the text. In addition, in each text restoration chain, different text restoration nodes have different execution orders, which are used to indicate the sequence between different restoration processes.
[0072] Exemplarily, after restoration, each text restoration chain corresponds to an output restored text.
[0073] S203, scoring at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each restored text.
[0074] The scoring rules include the scores corresponding to the preset phrases.
[0075] The score value represents the density of preset phrases contained in each restored text, and the preset phrases are phrases associated with minors.
[0076] For example, the preset phrases include isolated English / Chinese numbers or words related to minors with values ranging from 8 to 17. If it is detected that the restored result contains the preset phrase, it is scored as 1 point; if multiple hits are found, the scores are added up.
[0077] S204, inputting the restored texts into the text recognition model in sequence according to the order of the corresponding score values of the restored texts.
[0078] The text recognition model is used to output the recognition results corresponding to each restored text, and the recognition results are used to indicate whether the text to be recognized corresponding to the restored text is a risky text. The risky text is a variant text obtained after the text associated with minors is transformed.
[0079] Exemplarily, the text recognition model can be any algorithm that lacks processing for variant text and can directly implement recognition of minor-related vocabulary for text. For example, it can be a model that automatically learns the conversion relationship between variants and original texts by training a large model.
[0080] In the method for identifying the above variant text, first, the text to be identified is obtained, then at least two different text restoration chains are obtained. According to the text restoration rules corresponding to each text restoration node in the text restoration chain, and in the execution order of each text restoration node, the text to be identified is restored sequentially to obtain at least two restored texts. Furthermore, according to a preset scoring rule, the at least two restored texts are scored to obtain the scoring values corresponding to each restored text. Finally, in the order of the magnitudes of the scoring values corresponding to each restored text, each restored text is input into the text recognition model in turn to analyze whether the text to be identified corresponding to the restored text is a text with risks; on the basis of the existing text recognition model, a preprocess for restoring variant text is added, which is improved in a convenient way, improving the recognition efficiency, and at the same time avoiding the risks brought by some accounts using variant text, and improving the recognition accuracy for variant text.
[0081] In one embodiment, before the step of sequentially restoring the text to be identified, the method further includes: according to a preset character conversion set, replacing the first character in the text to be identified with the second character corresponding to the first character to obtain the text to be identified after replacement;
[0082] Wherein, the character conversion set includes the corresponding relationship between the first character and the second character, and the usage frequency of the first character is less than that of the second character.
[0083] Exemplarily, the corresponding relationship between the first character and the second character included in the character conversion set can be that the first character "①" corresponds to the second character "1", and the first character "戀" corresponds to the second character "恋". Generally, the character conversion set is used to convert all special characters into common characters.
[0084] In this embodiment, according to the preset character conversion set, the first character in the text to be identified is replaced with the second character corresponding to the first character to obtain the text to be identified after replacement, which realizes replacing the character with a lower usage frequency with the character with a higher usage frequency, that is, converting all special characters into common characters, avoiding the problem of characters being unrecognizable during the restoration process, improving the algorithm's ability to handle special characters, and thus improving the text recognition efficiency.
[0085] In one embodiment, according to the preset scoring rule, scoring the at least two restored texts to obtain the scoring values corresponding to each restored text includes: identifying each entity phrase included in each restored text; in the case where the entity phrase is a preset phrase, increasing the scoring value of the restored text by the unit scoring value corresponding to the preset phrase; accumulating the various unit scoring values corresponding to each restored text to obtain the scoring value corresponding to each restored text.
[0086] Among them, the entity phrases can be phrases with practical meanings identified by entity recognition technology, the preset phrases can be isolated English / Chinese numbers with values of 8 to 17 or words related to minors, and the unit rating value is the score corresponding to different phrases, for example, the unit rating value is 1.
[0087] For example, if it is detected that the restored result contains isolated English / Chinese numbers with a value of 8 to 17 or words related to minors, it is scored as 1 point; if multiple hits are found, the scores are added up.
[0088] In this embodiment, each entity phrase contained in each restored text is first identified, and then, when the entity phrase is a preset phrase, the scoring value of the restored text is increased by the unit scoring value corresponding to the preset phrase, and finally, the unit scoring values corresponding to each restored text are accumulated to obtain the scoring value corresponding to each restored text. By setting a unit scoring value for each preset phrase and scoring each text, the restoration effect of different text restoration chains is evaluated, which facilitates the subsequent sorting of texts according to the evaluation results, so that the text recognition model can timely identify risky texts, thereby improving the timeliness of variant text recognition.
[0089] In one embodiment, obtaining at least two different text restoration chains includes: obtaining pre-constructed text restoration nodes; and assembling the text restoration nodes according to the execution order of the text restoration nodes to obtain at least two different assembled text restoration chains.
[0090] Among them, the text restoration node is also called a text restoration module, and the text restoration node is used to restore the text according to the corresponding text restoration rule.
[0091] Exemplarily, text restoration nodes are used to assemble a text restoration chain, such as: remove all punctuation marks -> replace string; remove spaces and control characters -> compress continuous punctuation marks -> replace string.
[0092] In this embodiment, first, pre-constructed text restoration nodes are obtained, and then the text restoration nodes are assembled according to the execution order of the text restoration nodes to obtain at least two different assembled text restoration chains. The chains are assembled by setting the node order, so that the text restoration chains have different functions, thereby improving the accuracy of text restoration of the text restoration chains.
[0093] In one embodiment, the text restoration nodes are assembled to obtain at least two different assembled text restoration chains, including: obtaining the assembled text restoration chain, determining the node to be added from each text restoration node; and assembling the assembled text restoration chain and the node to be added again to obtain a composite text restoration chain.
[0094] Generally, there are two ways to construct a text restoration chain. One is to directly use text restoration nodes to assemble the chain, and the other is to use an already assembled text restoration chain and add some text restoration nodes to reassemble the chain.
[0095] Exemplarily, on the one hand, multiple restoration modules are assembled to form a restoration chain, such as: remove all punctuation marks -> replace string; remove spaces and control characters -> compress continuous punctuation marks -> replace string; on the other hand, based on a certain chain, new restoration modules are continuously added to obtain a composite chain, such as: (remove all punctuation marks -> replace string) -> move numbers to one side.
[0096] In this embodiment, an assembled text restoration chain is first obtained, and the nodes to be added are determined from each text restoration node. Then, the assembled text restoration chain and the nodes to be added are reassembled to obtain a composite text restoration chain. This provides a variety of construction methods for the text restoration chain, improves the diversity of text restoration chain construction, and further improves the richness of the restored text, making it easier to screen and use the rich restoration results.
[0097] In one embodiment, the text restoration node includes a removal type text restoration node, and each pre-constructed text restoration node is obtained, including: identifying a target to be removed; setting a text restoration rule corresponding to the text restoration node to delete the target to be removed in the text, so as to construct a removal type text restoration node according to the text restoration rule.
[0098] The targets to be removed include: preset characters, preset punctuation marks, symbols at set positions, and continuous characters.
[0099] Exemplarily, the targets to be removed may include: removing spaces and control characters (\n, \t, \f, etc.); removing all punctuation marks (custom punctuation marks may be skipped); removing punctuation marks at the end; removing consecutive Chinese numbers, etc.
[0100] In this embodiment, the target to be removed is first identified, and then the text restoration rule corresponding to the text restoration node is set to delete the target to be removed in the text, so as to construct a text restoration node of the removal type according to the text restoration rule, and clearly provide a basis for the removal task, thereby improving the accuracy of removing the corresponding target in the text, and further improving the accuracy of text restoration by each text restoration node.
[0101] In one embodiment, the text restoration node includes a compression type text restoration node, and each pre-constructed text restoration node is obtained, including: identifying a target to be compressed; setting a text restoration rule corresponding to the text restoration node to compress the length of the characters to be compressed in the text to the target length, so as to construct a compression type text restoration node according to the text restoration rule.
[0102] The target to be compressed includes continuous characters. For example, the target to be compressed may be compressed continuous punctuation marks or symbols. The target length may be 1, indicating that multiple symbols are compressed into one.
[0103] In this embodiment, the target to be compressed is first identified, and then the text restoration rule corresponding to the text restoration node is set to compress the length of the characters to be compressed in the text to the target length, so as to construct a text restoration node of the compression type according to the text restoration rule, and clearly define the basis for the compression task, thereby improving the accuracy of the corresponding target in the compressed text, and further improving the accuracy of text restoration by each text restoration node.
[0104] In one embodiment, the text restoration node includes a moving type text restoration node, and each pre-constructed text restoration node is obtained, including: identifying a target to be moved; setting a text restoration rule corresponding to the text restoration node to transfer the characters to be moved in the text to the beginning or end of the text, so as to construct a moving type text restoration node according to the text restoration rule.
[0105] The target to be moved includes characters belonging to a target character type; the target character type includes a number type.
[0106] Exemplarily, the target to be moved may be to move a number to one side.
[0107] In this embodiment, the target to be moved is first identified, and then the text restoration rule corresponding to the text restoration node is set to transfer the characters to be moved in the text to the beginning or end of the text, so as to construct a moving type text restoration node according to the text restoration rule, thereby realizing the construction of a moving type text restoration node, facilitating the guidance of the restoration task of moving part of the text, and improving the accuracy of text restoration by each text restoration node.
[0108] In another embodiment, Figure 3 As shown, a method for identifying variant text is provided, comprising the following steps:
[0109] S301, according to a preset character conversion set, replacing a first character in a text to be recognized with a second character corresponding to the first character, to obtain a replaced text to be recognized.
[0110] S302, obtaining each pre-constructed text restoration node.
[0111] S303, assembling the text restoration nodes according to the execution order of the text restoration nodes to obtain at least two different assembled text restoration chains.
[0112] S304, restoring the texts to be recognized in sequence to obtain at least two restored texts.
[0113] S305, identifying each entity phrase contained in each restored text.
[0114] S306: When the entity phrase is a preset phrase, the score value of the restored text is increased by the unit score value corresponding to the preset phrase.
[0115] S307, accumulating the unit score values corresponding to the restored texts to obtain the score values corresponding to the restored texts.
[0116] S308, inputting the restored texts into the text recognition model in sequence according to the order of the corresponding score values of the restored texts.
[0117] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a variant text recognition method mentioned above, which will not be repeated here.
[0118] In one embodiment, the method and system for identifying text related to minors has the disadvantage that, in the face of various variants, existing rules are difficult to identify minors, or difficult to apply to the identification of minor variants.
[0119] Based on this, the embodiment of the present application provides a method for identifying variant texts. To facilitate the understanding of those skilled in the art, Figure 4 A text restoration chain diagram of a variant text recognition method is provided, as shown below. Figure 4 , a method for identifying variant text is described in detail with a specific embodiment. It is worth noting that the following description is only an exemplary description and not a specific limitation of the application.
[0120] The variant text recognition method provided by the present application is also called a method for identifying minor variants of room names. Specifically, multiple restoration chains can be sent to restore the text according to the room name variants, and the original meaning of the user is guessed as much as possible, and then the traditional rules or models are used for recognition. The overall idea of the variant text recognition method provided by the present application is to simplify the complex, and an additional variant restoration module can be added to make the recognition process simple and efficient.
[0121] The variant text recognition method provided in this application has the following technical steps:
[0122] 1. Restoration of a single special character
[0123] In some scenarios, in order to circumvent the review measures for underage keywords, some users will represent some Chinese, English, numbers, etc. with special characters. To address the above problems, the embodiments of this application need to create a character conversion set to convert all these special characters into common characters. The examples are as follows: Replace "①" with "1", replace "戀" with "恋", replace " " with "禁", replace " " with "五", etc.
[0124] 2. Construction of a restoration module
[0125] The following shows some variants of room names:
[0126]
[0127] To restore variant texts involving minors such as those in the above table, the embodiments of this application construct some restoration modules to facilitate restoring the original meaning of the variants. The conventional restoration modules are as follows: Remove spaces and control characters (\n, \t, \f, etc.); Remove all punctuation marks (custom punctuation marks can be skipped); Compress consecutive punctuation marks or symbols; Remove trailing punctuation marks; Remove consecutive Chinese numbers; Move numbers to one side; Replace strings; Customized advanced restoration modules (regular extraction, homophonic restoration, etc.).
[0128] 3. Construction of a restoration chain / composite chain
[0129] Multiple restoration modules are assembled to form a restoration chain. For example: Remove all punctuation marks -> Replace strings; Remove spaces and control characters -> Compress consecutive punctuation marks -> Replace strings.
[0130] A composite chain means adding new restoration modules based on a certain chain. For example: (Remove all punctuation marks -> Replace strings) -> Move numbers to one side. The advantage of a composite chain is that it can cache the input and output results of a certain chain to speed up the restoration of the composite chain. The structure of the restoration chain is as Figure 4 shown.
[0131] The following are several examples of using the restoration chain to restore text:
[0132] For the text "1..4", the fastest chain is: Remove all punctuation marks. After restoration, the text is "14".
[0133] For the text "石,三,,,,", the fastest chain is: Remove all punctuation marks -> Replace strings. Specifically, after removing all punctuation marks, the text "石三" is obtained, and then after replacing the string, the text "13" is obtained.
[0134] For the text "|() 5, find a partner for my best friend", the fastest chain is: remove spaces and control characters -> remove all punctuation marks (retain the | character) -> replace the string. Specifically, after removing spaces and control characters, the text "|() 5, find a partner for my best friend" is obtained. After removing all punctuation marks and retaining the "|" character, the text "|5 find a partner for my best friend" is obtained. After replacing the string, the text "15 find a partner for my best friend" is obtained.
[0135] For the text " / / / / -4,, welcome to the room, if you have a microphone, please come on stage,," the fastest chain is: remove trailing punctuation marks -> compress consecutive characters -> remove all punctuation marks (retain the / and, characters) -> replace the string. Specifically, after removing trailing punctuation marks, the text " / / / / -4,, welcome to the room, if you have a microphone, please come on stage" is obtained. After compressing consecutive characters, the text " / -4, welcome to the room, if you have a microphone, please come on stage" is obtained. After removing all punctuation marks and retaining the " / " and "," characters, the text " / 4, welcome to the room, if you have a microphone, please come on stage" is obtained. After replacing the string, the text "14, welcome to the room, if you have a microphone, please come on stage" is obtained.
[0136] For the text " / baby dd 3", the fastest chain is: remove spaces and control characters -> custom restoration module (replace the one starting with / with 1) -> move the number to the end. Specifically, after removing spaces and control characters, the text " / baby dd3" is obtained. After the custom restoration module, the text "1baby dd3" is obtained. After moving the number to the end, the text "baby dd13" is obtained.
[0137] The above examples only show the fastest chains. In the actual restoration process, there may be more than a dozen chains, that is, more than a dozen output results. Since it is not known which chain to go through to restore the original meaning of the text, it is necessary to traverse all the chains to guess all possible meanings of the user. There will always be a chain that can be guessed correctly, that is, it can be hit by the existing rules. For example, for the text " / baby dd 3", the possible guessed results are texts such as " / baby dd3", "baby dd3", "1baby dd3", "baby dd13", "13baby dd", etc.
[0138] 4. Multi-chain sorting
[0139] After collecting the results of all restoration chains, duplicate removal is performed, and then the restoration chains are sorted. The reason for sorting is that originally one variant result needed to be detected, but after multi-chain restoration, perhaps a dozen results need to be detected. This processing method is time-consuming. Therefore, among all the restoration results, it is necessary to guess the most likely result in advance and place it at the front for matching to save matching time.
[0140] Sorting principle: If the restored result contains isolated English / Chinese numbers with values between 8 and 17 or words related to minors, it will be scored as 1 point; if multiple hits are found, the scores will be added up. Finally, the scores of all restored results will be sorted from high to low.
[0141] For example: if multiple restoration results are: " / baby dd3, baby dd3, 1 baby dd3, baby dd13, 13 baby dd", then the sorting result can be: "baby dd13, 13 baby dd, / baby dd3, baby dd3, 1 baby dd3".
[0142] 5. Traditional Identification of Minors
[0143] A series of sorted restoration results can be sent to the existing algorithm model for identifying minors. No special operation is required, and high-accuracy hits can be achieved. Subsequent restoration result recognition can be terminated in advance.
[0144] Among them, the existing algorithm model for identifying minors can be any text that lacks processing for variant texts, and can directly implement an algorithm for identifying minor-related vocabulary in the text.
[0145] For example, an existing algorithm model for identifying minors may be a method for identifying text for minors, and specific steps may include: obtaining a text to be identified that includes several sentences, and for each sentence in the text to be identified, identifying the sentence in turn through each recognition combination in a pre-established sentence recognition module in descending order of priority of each recognition combination, obtaining an intermediate marked sentence with a keyword mark, analyzing the keyword mark carried by the intermediate marked sentence, marking the intermediate marked sentence with a minor discrimination mark, and obtaining a target sentence with the minor discrimination mark, wherein the sentence recognition module includes multiple recognition combinations of different priorities, each recognition combination includes a sentence matching algorithm and a keyword stamp, counting the number of target sentences with the minor discrimination mark in the text to be identified as a first number, and calculating a first score based on the first number, counting the number of target sentences with the minor discrimination mark in the text to be identified as a highly suspected minor mark, and determining it as a second number, and calculating a second score based on the second number, and if the sum of the first score and the second score is greater than a preset score threshold, determining that the text to be identified is a text for minors.
[0146] The variant text recognition method provided in this application can borrow the traditional scheme for identifying minors. It only needs to pre-place a variant restoration module. It is simple and efficient. It does not need to set too many variant recognition rules for numerous variants. It only needs to be restored, which is very consistent with the human thinking process.
[0147] In addition, the variant text recognition method provided in the present application can also be applied in other model scenarios, for example, by training a large model to automatically learn the conversion relationship between variants and original texts, the large model will automatically perform text calibration and error correction, which belongs to the understanding at the semantic level, but the disadvantage is that the speed is too slow. After using the variant text recognition method provided in the embodiment of the present application, all common conversion methods can be displayed and expressed, and the variant text can be restored before recognition, thereby improving the recognition efficiency of the original model.
[0148] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0149] Based on the same inventive concept, the embodiment of the present application also provides a variant text recognition device for implementing the variant text recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of one or more variant text recognition devices provided below can refer to the limitations of the variant text recognition method above, and will not be repeated here.
[0150] In one embodiment, Figure 5 As shown, a variant text recognition device is provided, comprising: a text acquisition module 501, a text restoration module 502, a text scoring module 503, and a model recognition module 504, wherein:
[0151] The text acquisition module 501 is used to acquire the text to be recognized;
[0152] The text restoration module 502 is used to obtain at least two different text restoration chains, and restore the texts to be recognized in sequence according to the text restoration rules corresponding to the text restoration nodes in the text restoration chains and the execution order of the text restoration nodes to obtain at least two restored texts;
[0153] The text scoring module 503 is used to score at least two restored texts according to a preset scoring rule to obtain a score value corresponding to each restored text; the score value represents the density of the preset phrases contained in each restored text, and the preset phrases are phrases associated with minors;
[0154] The model recognition module 504 is used to input each restored text into the text recognition model in order of the size of the corresponding score value of each restored text; the text recognition model is used to output the recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text. The risky text is a variant text obtained after the text associated with minors is transformed.
[0155] In one embodiment, the device is used to: replace the first character in the text to be recognized with the second character corresponding to the first character according to a preset character conversion set, so as to obtain the replaced text to be recognized; wherein the character conversion set includes the correspondence between the first character and the second character, and the usage frequency of the first character is less than the usage frequency of the second character.
[0156] In one embodiment, the device is used to: identify each entity phrase contained in each restored text; when the entity phrase is a preset phrase, increase the score value of the restored text by the unit score value corresponding to the preset phrase; accumulate the unit score values corresponding to each restored text to obtain the score value corresponding to each restored text.
[0157] In one embodiment, the device is used to: obtain pre-constructed text restoration nodes; the text restoration nodes are used to restore the text according to the corresponding text restoration rules; according to the execution order of each text restoration node, the text restoration nodes are assembled to obtain at least two different text restoration chains after assembly.
[0158] In one embodiment, the device is used to: obtain an assembled text restoration chain, determine the node to be added from each text restoration node; and reassemble the assembled text restoration chain and the node to be added to obtain a composite text restoration chain.
[0159] In one embodiment, the device is used to: identify targets to be removed; the targets to be removed include: preset characters, preset punctuation marks, symbols at set positions, and continuous characters; set the text restoration rules corresponding to the text restoration nodes to delete the targets to be removed in the text, so as to construct a text restoration node of the removal type according to the text restoration rules.
[0160] In one embodiment, the device is used to: identify a target to be compressed; the target to be compressed includes continuous characters; set the text restoration rule corresponding to the text restoration node to compress the length of the characters to be compressed in the text to the target length, so as to construct a text restoration node of the compression type according to the text restoration rule.
[0161] In one embodiment, the device is used to: identify a target to be moved; the target to be moved includes characters belonging to a target character type; the target character type includes a numeric type; and set a text restoration rule corresponding to a text restoration node to transfer the characters to be moved in the text to the beginning or end of the text, so as to construct a text restoration node of the moving type according to the text restoration rule.
[0162] Each module in the variant text recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0163] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for recognizing variant text is implemented.
[0164] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a variant text recognition method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.
[0165] Those skilled in the art will understand that the structures shown in FIGS. 6 and 7 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.
[0166] In one embodiment, a computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.
[0167] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0168] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0171] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for recognizing variant text, characterized in that: The method comprises: Get the text to be recognized; Obtain at least two different text restoration chains, and restore the text to be recognized in sequence according to the text restoration rules corresponding to the text restoration nodes in the text restoration chains and the execution order of the text restoration nodes to obtain at least two restored texts; Scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each of the restored texts; the scoring value represents the density of a preset phrase contained in each of the restored texts, wherein the preset phrase is a phrase associated with minors; According to the order of the score values corresponding to each restored text, each restored text is input into the text recognition model in turn; the text recognition model is used to output the recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text, and the risky text is a variant text obtained after the text associated with minors is transformed.
2. The method according to claim 1, characterized in that Before the step of restoring the text to be recognized in sequence, the method further includes: According to a preset character conversion set, a first character in the text to be recognized is replaced with a second character corresponding to the first character to obtain the replaced text to be recognized; The character conversion set includes a correspondence between the first character and the second character, and a usage frequency of the first character is less than a usage frequency of the second character.
3. The method according to claim 1, characterized in that Scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each restored text includes: Identifying each entity phrase contained in each restored text; In the case where the entity phrase is the preset phrase, increasing the score value of the restored text by the unit score value corresponding to the preset phrase; The unit score values corresponding to each of the restored texts are accumulated to obtain the score values corresponding to each of the restored texts.
4. The method according to claim 1, characterized in that The obtaining of at least two different text restoration chains includes: Obtaining each of the pre-constructed text restoration nodes; the text restoration nodes are used to restore the text according to the corresponding text restoration rules; According to the execution order of each text restoration node, each text restoration node is assembled to obtain at least two different assembled text restoration chains.
5. The method according to claim 4, characterized in that The step of assembling the text restoration nodes to obtain at least two different assembled text restoration chains includes: Obtain the assembled text restoration chain, and determine the node to be added from each of the text restoration nodes; The assembled text restoration chain and the node to be added are reassembled to obtain a composite text restoration chain.
6. The method according to claim 4, characterized in that The text restoration node includes a text restoration node of a removal type, and the obtaining of each pre-constructed text restoration node includes: Identify the target to be removed; the target to be removed includes: preset characters, preset punctuation marks, symbols at set positions, and continuous characters; The text restoration rule corresponding to the text restoration node is set to delete the target to be removed in the text, so as to construct a text restoration node of the removal type according to the text restoration rule.
7. The method according to claim 4, characterized in that The text restoration node includes a compressed type text restoration node, and the obtaining of each pre-built text restoration node includes: Identifying a target to be compressed; the target to be compressed includes continuous characters; The text restoration rule corresponding to the text restoration node is set to compress the length of the characters to be compressed in the text to a target length, so as to construct a text restoration node of the compression type according to the text restoration rule.
8. The method according to claim 4, characterized in that The text restoration node includes a mobile type text restoration node, and the obtaining of each pre-constructed text restoration node includes: Identifying a target to be moved; the target to be moved includes a character belonging to a target character type; the target character type includes a numeral type; The text restoration rule corresponding to the text restoration node is set to transfer the characters to be moved in the text to the beginning or the end of the text, so as to construct a text restoration node of the moving type according to the text restoration rule.
9. A variant text recognition device, characterized in that: The device comprises: A text acquisition module is used to acquire the text to be recognized; A text restoration module is used to obtain at least two different text restoration chains, and restore the text to be recognized in sequence according to the text restoration rules corresponding to each text restoration node in the text restoration chain and the execution order of each text restoration node to obtain at least two restored texts; A text scoring module, used for scoring the at least two restored texts according to a preset scoring rule to obtain a scoring value corresponding to each of the restored texts; the scoring value represents the density of a preset phrase contained in each of the restored texts, wherein the preset phrase is a phrase associated with minors; A model recognition module is used to input each restored text into a text recognition model in order of the size of the score values corresponding to each restored text; the text recognition model is used to output a recognition result corresponding to each restored text, and the recognition result is used to characterize whether the text to be recognized corresponding to the restored text is a risky text, and the risky text is a variant text obtained after the text associated with minors is transformed.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
