Method, device and equipment for processing document garbled content and storage medium

Abstract

The present disclosure provides a document garbled content processing method and device, equipment and storage medium, and relates to the technical field of computers, specifically to the field of natural language processing, and can be applied to the scene of garbled code processing. The implementation scheme is: dividing the content in the to-be-processed document into at least one paragraph, identifying rare characters in each paragraph; calculating the frequency of occurrence of the rare characters in each paragraph; determining a target paragraph from the at least one paragraph based on the frequency of occurrence of the rare characters corresponding to each paragraph; dividing the target paragraph into at least one sentence, calculating the perplexity of each sentence; determining a candidate sentence potentially having garbled content from the at least one sentence based on the perplexity corresponding to each sentence; determining a target sentence actually containing garbled content from the candidate sentence, and deleting the garbled content. The above method can avoid the effective content of the document being mistakenly deleted or the actual garbled content being missed, and ensure that the document has high usability.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to the field of natural language processing, and can be applied to the scene of messy code processing. BACKGROUND

[0002] In some application scenarios related to natural language understanding, the content in some documents needs to be parsed before it can be normally used. Due to the limited accuracy of the parsing technology, this may cause the parsed new document to have messy code content, affecting the usability of the new document. In the related technology, the messy code content is identified in the parsed new document, and the messy code content is deleted, thereby ensuring the usability of the new document. However, the existing way of identifying messy code content is relatively single, and the accuracy of the messy code content identification result is low. SUMMARY

[0003] The present disclosure provides a document messy code content processing method, device, equipment and storage medium.

[0004] According to a first aspect of the present disclosure, a document messy code content processing method is provided, comprising:

[0005] dividing the content in the document to be processed into at least one paragraph, and identifying rare characters in each paragraph;

[0006] calculating the occurrence frequency of the rare characters in each paragraph;

[0007] determining a target paragraph from the at least one paragraph based on the occurrence frequency corresponding to the rare characters in each paragraph;

[0008] dividing the target paragraph into at least one sentence, and calculating the perplexity of each sentence, wherein the perplexity is used to indicate the rationality degree of the sentence;

[0009] determining a candidate sentence potentially having messy code content from the at least one sentence based on the perplexity corresponding to each sentence, wherein the messy code content includes at least one rare character;

[0010] determining a target sentence actually containing messy code content from the candidate sentence, and deleting the messy code content in the target sentence.

[0011] According to a second aspect of the present disclosure, a document messy code content processing device is provided, comprising:

[0012] a rare character identification module, configured to divide the content in the document to be processed into at least one paragraph, and identify rare characters in each paragraph;

[0013] an occurrence frequency calculation module, configured to calculate the occurrence frequency of the rare characters in each paragraph;

[0014] a target paragraph determination module configured to determine a target paragraph from the at least one paragraph based on a frequency of occurrence of the rare character in each paragraph;

[0015] a confusion degree calculation module configured to divide the target paragraph into at least one sentence and calculate a confusion degree of each sentence, wherein the confusion degree is used to indicate a rationality degree of the sentence;

[0016] a candidate sentence determination module configured to determine a candidate sentence potentially containing the garbled content from the at least one sentence based on the confusion degree of each sentence, wherein the garbled content includes at least one rare character;

[0017] a garbled content deletion module configured to determine a target sentence actually containing the garbled content from the candidate sentence and delete the garbled content in the target sentence.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0019] at least one processor; and a memory connected with the at least one processor in communication;

[0020] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the document garbled content processing method provided in the first aspect.

[0021] According to a fourth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the document garbled content processing method provided in the first aspect.

[0022] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the document garbled content processing method provided in the first aspect.

[0023] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

[0024] The technical solutions provided by the present disclosure have the following beneficial effects:

[0025] In the technical solution of the present disclosure, first, the target paragraph containing the code content with a larger probability is identified through the frequency of appearance of the rare characters, and then the target sentence actually containing the code content is identified through the perplexity of each sentence in the target paragraph. Through the information of the two dimensions of the frequency of appearance of the rare characters and the perplexity of the sentence, the code content in the document can be accurately and directly located, avoiding the effective content of the document being mistakenly deleted or the actual code content being missed, so as to ensure that the document after the code content has higher usability. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation on the present disclosure. Among them:

[0027] Figure 1 A flowchart of a document code content processing method provided by an embodiment of the present disclosure is shown;

[0028] Figure 2 A flowchart of another document code content processing method provided by an embodiment of the present disclosure is shown;

[0029] Figure 3 A schematic diagram of a document code content processing device provided by an embodiment of the present disclosure is shown;

[0030] Figure 4 A schematic block diagram of an example electronic device that can be used to implement the document code content processing method provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0032] In some application scenarios involving natural language understanding, the content in some documents needs to be parsed before it can be used normally. Due to the limited accuracy of parsing technology, this may result in the existence of code content in the parsed new document, affecting the usability of the new document. In related technologies, the code content is identified in the parsed new document, and the code content is deleted, thereby ensuring the usability of the new document. However, the existing way of identifying code content is relatively single, and the accuracy of the code content identification result is low.

[0033] The document code content processing method, device, equipment and storage medium provided by the embodiments of the present disclosure aim to solve at least one of the above technical problems in the prior art.

[0034] Figure 1 The flowchart shows a method for processing garbled content in a document provided by an embodiment of the present disclosure. As shown, the method mainly may include the following steps: Figure 1 as shown, the method mainly may include the following steps:

[0035] S110: Divide the content in the document to be processed into at least one paragraph, and identify the rare characters in each paragraph.

[0036] The document to be processed is a new document obtained after parsing the original document. Due to the limited accuracy of the parsing technology, there may be differences between the content in the new document and the content in the original document. Specifically, in the new document obtained after parsing, there may be some characters that do not exist in the original document, and these characters may include rare characters.

[0037] In the embodiment of the present disclosure, the number of paragraphs into which the content in the document to be processed needs to be divided may be determined based on actual design requirements. When the number of paragraphs into which the content in the document to be processed needs to be divided is not less than 2, the number of characters in each paragraph may be approximately equal. For example, the content in the document to be processed may be equally divided into 10 paragraphs.

[0038] In the embodiment of the present disclosure, rare characters may be characters, symbols (such as question marks, full stops, and commas, etc.), and numbers, etc. The types of rare characters include at least one of ancient Chinese codes, mouth codes, symbol codes, question mark codes, and stick copy codes.

[0039] For example, the original document includes the content "spring in 2022". After parsing the above content of the original document, the new document obtained may include the content " spring in 2022". Here, " " can be regarded as a rare character.

[0040] Optionally, in the embodiment of the present disclosure, when identifying the rare characters in each paragraph, a common character set corresponding to the document may be determined; for each paragraph, the rare characters that do not belong to the common character set are identified from the paragraph. Here, the type of the common character set is related to the application field of the document.

[0041] Optionally, in the embodiment of the present disclosure, when identifying the rare characters in each paragraph, a rare character set corresponding to the document may be determined; for each paragraph, the rare characters that belong to the rare character set are identified from the paragraph. Here, the type of the rare character set is related to the application field of the document.

[0042] S120: Calculate the occurrence frequency of the rare characters in each paragraph.

[0043] In the embodiments of the present disclosure, the occurrence frequency of the rare character in the paragraph refers to at least one of the proportion of the rare character in the paragraph and the proportion of the rare character in the document. In order to facilitate understanding and description, the present disclosure defines the proportion of the rare character in the paragraph as the first occurrence frequency, and the proportion of the rare character in the document as the second occurrence frequency. Optionally, in the calculation of the occurrence frequency of the rare character in each paragraph, the present disclosure can calculate the first occurrence frequency of the rare character in the paragraph and the second occurrence frequency of the rare character in the document for each paragraph.

[0044] S130: determining a target paragraph from the at least one paragraph based on the occurrence frequency of the rare character corresponding to each paragraph.

[0045] As described above, the occurrence frequency of the rare character corresponding to each paragraph includes the first occurrence frequency and the second occurrence frequency. The present disclosure can determine a target paragraph from the at least one paragraph based on at least one of the first occurrence frequency and the second occurrence frequency of the rare character corresponding to each paragraph.

[0046] Optionally, the present disclosure calculates, for each paragraph, the occurrence frequency ratio of the first occurrence frequency and the second occurrence frequency of the rare character corresponding to each paragraph; and determines a target paragraph from the at least one paragraph based on the occurrence frequency ratio of the rare character corresponding to each paragraph.

[0047] Optionally, the present disclosure determines, from the at least one paragraph, a target paragraph corresponding to the first occurrence frequency greater than the first preset occurrence frequency based on the first occurrence frequency of the rare character corresponding to each paragraph.

[0048] Optionally, the present disclosure determines, from the at least one paragraph, a target paragraph corresponding to the second occurrence frequency greater than the second preset occurrence frequency based on the second occurrence frequency of the rare character corresponding to each paragraph.

[0049] S140: dividing the target paragraph into at least one sentence and calculating the perplexity of each sentence.

[0050] In the embodiments of the present disclosure, the at least one sentence can be divided based on the punctuation marks in the target paragraph, which can include comma, period and question mark, etc. Of course, the present disclosure can also pre-configure the number of sentences for the target paragraph, and divide the at least one sentence according to the number of sentences.

[0051] In the embodiments of the present disclosure, the perplexity is used to indicate the rationality of the sentence. Specifically, the perplexity can indicate whether the sentence is fluent, whether the collocation and the use of grammar of the sentence are consistent with the normal use of people, etc.

[0052] The disclosure embodiments can regard the comprehensive degree of the unreasonableness of the adjacent or associated word pairs in a sentence in grammar and semantics as the perplexity of the whole sentence. The disclosure embodiments can perform word segmentation on the sentence to obtain a plurality of segmented words corresponding to the sentence, calculate the context appearance probability of each segmented word in the plurality of segmented words, and multiply the context appearance probability of each segmented word in the plurality of segmented words corresponding to the sentence to obtain the perplexity of the sentence. Of course, the disclosure embodiments can also calculate the perplexity of the sentence in other ways, which are not listed one by one here.

[0053] S150: Determine a candidate sentence potentially having the garbled content from the at least one sentence based on the perplexity corresponding to each sentence.

[0054] It can be understood that the document appears garbled usually because it contains rare characters, and therefore the garbled content includes at least one rare character. Step S150 determines whether the sentence is a candidate sentence potentially having the garbled content based on the perplexity corresponding to the sentence. Here, the candidate sentence refers to a sentence that has a higher possibility of having the garbled content. Generally, the more the number of rare characters in the sentence, the greater the perplexity corresponding to the sentence, and the higher the probability of the sentence being determined as the candidate sentence.

[0055] Optionally, the disclosure embodiments can determine a perplexity threshold corresponding to the document when determining the candidate sentence potentially having the garbled content, and determine the candidate sentence having the perplexity greater than the perplexity threshold from the at least one sentence. That is, when the perplexity corresponding to a sentence is greater than the perplexity threshold, the sentence can be determined as the candidate sentence potentially having the garbled content.

[0056] Optionally, the disclosure embodiments can determine the candidate sentence having the greatest perplexity from the at least one sentence when determining the candidate sentence potentially having the garbled content. That is, the disclosure embodiments can determine the sentence having the greatest perplexity as the candidate sentence potentially having the garbled content.

[0057] S160: Determine a target sentence actually containing the garbled content from the candidate sentence, and delete the garbled content in the target sentence.

[0058] As described above, the candidate sentence refers to a sentence that has a higher possibility of having the garbled content. Step S160 can continue to determine whether the candidate sentence is the target sentence actually containing the garbled content, and delete the garbled content in the target sentence when the candidate sentence is determined as the target sentence actually containing the garbled content. Here, the candidate sentence can be determined as the target sentence actually containing the garbled content by the related data of the rare characters in the candidate sentence. For example, the number, position characteristics (such as whether continuous) and other information of the rare characters in the candidate sentence determine the candidate sentence as the target sentence actually containing the garbled content.

[0059] Optionally, the disclosure embodiment can determine the candidate sentence as the target sentence actually having the garbled content by the following steps: identifying a rare character sequence from the candidate sentence; and determining the candidate sentence as the target sentence actually having the garbled content and the rare character sequence as the garbled content when it is determined that the number of rare characters in the rare character sequence is greater than a preset number. Here, the rare character sequence includes consecutive rare characters.

[0060] The document garbled content processing method provided by the disclosure embodiment first identifies the target paragraph that has a greater probability of containing garbled content through the occurrence frequency of rare characters, and then identifies the target sentence that actually contains garbled content through the perplexity of each sentence in the target paragraph. Through the information of two dimensions of the occurrence frequency of rare characters and the perplexity of the sentence, the garbled content in the document can be accurately and directly located, avoiding that the effective content of the document is mistakenly deleted or the actual garbled content is missed, so as to ensure that the document after the garbled content has higher usability.

[0061] Figure 2 A flowchart of a document garbled content processing method provided by the disclosure embodiment is shown, as shown in Figure 2 The method can mainly include the following steps:

[0062] S201: dividing the content in the document to be processed into at least one paragraph.

[0063] The document to be processed is a new document obtained after the original document is parsed. Due to the limited accuracy of the parsing technology, there may be differences between the content in the new document and the content of the original document. Specifically, the new document obtained after parsing may contain some characters that do not exist in the original document, and these characters may include rare characters.

[0064] In the disclosure embodiment, the number of paragraphs in which the content in the document to be processed needs to be divided can be determined based on actual design needs. When the number of paragraphs in which the content in the document to be processed needs to be divided is not less than 2, the number of characters in each paragraph can be approximately equal. For example, the content in the document to be processed can be equally divided into 10 paragraphs.

[0065] S202: determining a commonly used character set corresponding to the document.

[0066] S203: identifying rare characters that do not belong to the commonly used character set from each paragraph.

[0067] In this embodiment of the disclosure, uncommon characters can be text, symbols (such as question marks, periods, and commas), and numbers. The types of uncommon characters include at least one of the following: ancient script codes, square root codes, symbol codes, question mark codes, and comma codes. For example, if the original document includes the sentence "Spring of 2022," after parsing the above content of the original document, the resulting new document might include " The phrase "springtime in 2022" is mentioned here. " can be considered a rare character."

[0068] In this embodiment, the type of commonly used character set is related to the application domain of the document, and the commonly used character set includes commonly used characters used in the corresponding domain. It is understood that the commonly used characters used in different domains may differ slightly; therefore, different commonly used character sets can be configured for different domains. For example, corresponding commonly used character sets can be configured for the medical, computer, mechanical, and chemical domains, respectively. Using a commonly used character set based on the application domain of the document allows for more objective identification of uncommon characters in each paragraph of the document, ensuring the accuracy of the identified uncommon characters.

[0069] S204: For each paragraph, calculate the first frequency of occurrence of the rare character in that paragraph and the second frequency of occurrence of the rare character in that paragraph in the document.

[0070] In this embodiment of the disclosure, the frequency of occurrence of rare characters in a paragraph refers to at least one of the proportion of rare characters in the paragraph and the proportion of rare characters in the paragraph in the document. For ease of understanding and description, this embodiment of the disclosure defines the proportion of rare characters in a paragraph as the first frequency of occurrence, and the proportion of rare characters in a paragraph in the document as the second frequency of occurrence.

[0071] For example, for a paragraph 'a', we can count the number of uncommon characters in paragraph 'a', the total number of characters in paragraph 'a', and the total number of characters in the document. The ratio of the number of uncommon characters in paragraph 'a' to the total number of characters in paragraph 'a' is taken as the first frequency of occurrence of the uncommon character in paragraph 'a' within that paragraph; the ratio of the number of uncommon characters in paragraph 'a' to the total number of characters in the document is taken as the second frequency of occurrence of the uncommon character in paragraph 'a' within the document.

[0072] S205: For each paragraph, calculate the ratio of the first frequency of occurrence to the second frequency of occurrence of the uncommon characters in that paragraph. This can be understood as defining the ratio of the first frequency of occurrence to the second frequency of occurrence of the uncommon characters in a paragraph as the frequency ratio.

[0073] S206: Determine the target paragraph from the at least one paragraph based on the frequency ratio of the rare character corresponding to each paragraph.

[0074] The greater the frequency ratio of the rare character corresponding to each paragraph, the greater the difference between the first frequency and the second frequency of the rare character corresponding to each paragraph, and the smaller the frequency of the rare character in the paragraph and the smaller the frequency of the rare character in other paragraphs. This can indicate that the rare character in the paragraph is more concentrated, and the paragraph is more likely to have garbled code. Therefore, based on the frequency ratio of the rare character corresponding to each paragraph, the target paragraph containing actual garbled code content can be more accurately determined.

[0075] Optionally, in the process of determining the target paragraph, the standard ratio corresponding to the document can be determined. It can be understood that the standard ratio can be preconfigured, and the specific value of the standard ratio is determined according to the actual design needs. After determining the standard ratio corresponding to the document, the target paragraph in which the frequency ratio of the rare character corresponding to each paragraph is greater than the standard ratio can be determined from the at least one paragraph. The present disclosure determines the concentration degree of the rare character in the paragraph by comparing the frequency ratio of the rare character corresponding to each paragraph and the standard ratio preconfigured for the document to be processed, objectively presents the possibility of the paragraph having garbled code, and accurately identifies the target paragraph containing actual garbled code content.

[0076] S207: Divide the target paragraph into at least one sentence and calculate the perplexity of each sentence.

[0077] In the present disclosure, at least one sentence can be divided based on the punctuation marks in the target paragraph, which can include commas, periods, and question marks. Of course, the present disclosure can also preconfigure the number of sentences for the target paragraph, and divide at least one sentence according to the number of sentences.

[0078] In the present disclosure, perplexity is used to indicate the degree of rationality of a sentence. Specifically, perplexity can indicate whether a sentence is smooth, whether the collocation and use of grammar of a sentence are consistent with normal use, and the like.

[0079] The present disclosure can consider the comprehensive measure of the degree of irrationality of adjacent or associated words in a sentence in terms of grammar and semantics as the perplexity of the entire sentence. The present disclosure can perform word segmentation processing on the sentence to obtain a plurality of segmented words corresponding to the sentence, calculate the context appearance probability of each segmented word in the plurality of segmented words, and multiply the context appearance probability of each segmented word in the plurality of segmented words corresponding to the sentence to obtain the perplexity of the sentence. Of course, the present disclosure can also calculate the perplexity of the sentence in other ways, which are not listed here.

[0080] S208: determining the perplexity threshold corresponding to the document.

[0081] S209: determining, from the at least one sentence, a candidate sentence corresponding to a perplexity greater than the perplexity threshold.

[0082] It can be understood that when a sentence corresponds to a perplexity greater than the perplexity threshold, the sentence can be determined as a candidate sentence potentially having the garbled content. The document appears garbled usually because it contains rare characters, and thus the garbled content includes at least one rare character.

[0083] Step S209 determines whether a sentence is a candidate sentence potentially having the garbled content by comparing the perplexity corresponding to the sentence with the perplexity threshold. It can be understood that the candidate sentence refers to a sentence having a higher possibility of having the garbled content. Generally, the more the number of rare characters in a sentence, the greater the perplexity corresponding to the sentence, and the higher the probability of the sentence being determined as the candidate sentence. Here, when the perplexity of a sentence is greater than the perplexity threshold of the candidate sentence, it indicates that the perplexity of the sentence has exceeded a reasonable range, indicating that the sentence has a greater possibility of containing the garbled content including the rare characters.

[0084] S210: identifying a rare character sequence from the candidate sentence.

[0085] In the embodiments of the present disclosure, the rare character sequence refers to a sequence including consecutive rare characters. It can be understood that the rare character sequence should include at least two consecutive rare characters.

[0086] S211: determining the candidate sentence as a target sentence actually having the garbled content and the rare character sequence as the garbled content when it is determined that the number of rare characters in the rare character sequence is greater than a preset number.

[0087] Step S211 can continue to determine whether the candidate sentence is the target sentence actually containing the garbled content, and delete the garbled content in the target sentence when it is determined that the candidate sentence is the target sentence actually containing the garbled content. Specifically, step S211 can count the number of rare characters in the rare character sequence of the candidate sentence, and if the number of rare characters in the rare character sequence identified by step S210 is greater than the preset number, it can be determined that the candidate sentence is the target sentence actually having the garbled content, and it can be determined that the rare character sequence identified by step S210 is the garbled content in the candidate sentence.

[0088] It should be noted that it is found through research that when the garbled content appears in the sentence, a plurality of continuous rare characters usually appear in the sentence. The rare characters of the sentence can be pre-configured with a preset number. When the number of rare characters in the rare character sequence of the sentence is greater than the preset number, the candidate sentence is the target sentence actually having the garbled content, so as to avoid deleting the valid content in the sentence as the garbled content, thereby ensuring the integrity of the valid content of the sentence.

[0089] S212: deleting the garbled content in the target sentence.

[0090] For example, the document includes the sentence of "2022 years of spring".The rare character sequence of " in the sentence is determined as the garbled content, and then the garbled content can be deleted in the sentence.

[0091] Based on the same principle as the above-mentioned document garbled content processing method, the embodiment of the disclosure provides a document garbled content processing device, Figure 3 The schematic diagram of the document garbled content processing device provided by the embodiment of the disclosure is shown. As shown in Figure 3 The document garbled content processing device 300 includes a rare character identification module 310, an occurrence frequency calculation module 320, a target paragraph determination module 330, a perplexity calculation module 340, a candidate sentence determination module 350, and a garbled content deletion module 360.

[0092] The rare character identification module 310 is used to divide the content in the document to be processed into at least one paragraph, and identify the rare characters in each paragraph.

[0093] The occurrence frequency calculation module 320 is used to calculate the occurrence frequency of the rare characters in each paragraph.

[0094] The target paragraph determination module 330 is used to determine the target paragraph from the at least one paragraph based on the occurrence frequency corresponding to the rare characters in each paragraph.

[0095] The perplexity calculation module 340 is used to divide the target paragraph into at least one sentence, and calculate the perplexity of each sentence, wherein the perplexity is used to indicate the rationality degree of the sentence.

[0096] The candidate sentence determination module 350 is used to determine the candidate sentence potentially having the garbled content from the at least one sentence based on the perplexity corresponding to each sentence, wherein the garbled content includes at least one rare character.

[0097] The garbled content deletion module 360 is used to determine the target sentence actually containing the garbled content from the candidate sentence, and delete the garbled content in the target sentence.​

[0098] The document garbled content processing apparatus provided by the embodiments of the present disclosure first identifies a target paragraph that has a greater probability of containing garbled content through the occurrence frequency of rare characters, and then identifies a target sentence that actually contains garbled content through the perplexity of each sentence in the target paragraph. Through the information of the two dimensions of the occurrence frequency of rare characters and the perplexity of a sentence, the garbled content in the document can be accurately and directly located, the effective content of the document is avoided from being mistakenly deleted, or the actual garbled content is avoided from being missed, so that the document after the garbled content has higher usability.

[0099] In the embodiments of the present disclosure, the rare character identification module 310, when identifying the rare characters in each paragraph, is specifically configured to:

[0100] determine a commonly used character set corresponding to the document, wherein the type of the commonly used character set is related to the application field of the document;

[0101] for each paragraph, identify rare characters in the paragraph that do not belong to the commonly used character set.

[0102] In the embodiments of the present disclosure, the occurrence frequency calculation module 320, when calculating the occurrence frequency of the rare characters in each paragraph, is specifically configured to:

[0103] for each paragraph, calculate a first occurrence frequency of the rare characters in the paragraph in the paragraph, and a second occurrence frequency of the rare characters in the paragraph in the document.

[0104] In the embodiments of the present disclosure, the target paragraph determination module 330, when determining the target paragraph from at least one paragraph based on the occurrence frequency of the rare characters in each paragraph, is specifically configured to:

[0105] for each paragraph, calculate an occurrence frequency ratio of the first occurrence frequency and the second occurrence frequency of the rare characters corresponding to the paragraph, and determine the target paragraph from at least one paragraph based on the occurrence frequency ratio of the rare characters corresponding to each paragraph.

[0106] In the embodiments of the present disclosure, the target paragraph determination module 330, when determining the target paragraph from at least one paragraph based on the occurrence frequency ratio of the rare characters corresponding to each paragraph, is specifically configured to:

[0107] determine a standard ratio corresponding to the document, and determine the target paragraph from at least one paragraph, wherein the target paragraph is a paragraph in which the occurrence frequency ratio of the rare characters corresponding to the paragraph is greater than the standard ratio.

[0108] In the embodiments of the present disclosure, the candidate sentence that potentially has garbled content is determined from at least one sentence based on the perplexity corresponding to each sentence, including:

[0109] determine a confusion threshold corresponding to the document, wherein a size of the confusion threshold is related to an application field of the document; and determine, from the at least one sentence, a candidate sentence corresponding to a confusion degree greater than the confusion threshold.

[0110] In the embodiment of the present disclosure, the garbled content deletion module 360 is configured to determine the candidate sentence as the target sentence actually having the garbled content by the following steps:

[0111] identify a rare character sequence from the candidate sentence, wherein the rare character sequence comprises continuous rare characters; and determine the candidate sentence as the target sentence actually having the garbled content and the rare character sequence as the garbled content when a number of the rare characters in the rare character sequence is greater than a preset number.

[0112] In the embodiment of the present disclosure, the type of the rare character comprises at least one of the following: an ancient Chinese code, a Kouheng code, a symbol code, a question mark code, and a stick code.

[0113] It can be understood that the above modules of the document garbled content processing apparatus in the embodiment of the present disclosure have the functions of corresponding steps of the above-mentioned document garbled content processing method. The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software comprises one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented individually or integratedly. The function description of each module of the document garbled content processing apparatus can be referred to the corresponding description of the above-mentioned document garbled content processing method, which will not be described here.

[0114] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations, and do not violate public order and good customs.

[0115] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0116] Figure 4 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0117] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0118] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] The computing unit 401 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the method for processing garbled document content. For example, in some embodiments, the method for processing garbled document content may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the method for processing garbled document content described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform a method for processing garbled document content by any other suitable means (e.g., by means of firmware).

[0120] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0121] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0122] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0124] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0126] It should be understood that various forms of flow shown above can be used, with steps reordered, added, or removed. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology disclosed in the present disclosure are achieved.

[0127] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalent substitutions, improvements, and the like, either alone or in some combination or sub-combination, are intended to be included within the scope of the present disclosure.

Claims

1. A method for handling garbled text in a document, comprising: The content of the document to be processed is divided into at least one paragraph, and uncommon characters in each paragraph are identified. Calculate the frequency of occurrence of rare characters in each paragraph; the frequency of occurrence refers to at least one of the proportion of the rare characters in the paragraph and the proportion of the rare characters in the paragraph in the document to be processed; Based on the frequency of occurrence of rare characters in each paragraph, a target paragraph is determined from the at least one paragraph; The target paragraph is divided into at least one sentence, and the perplexity of each sentence is calculated, wherein the perplexity is used to indicate the degree of reasonableness of the sentence; Based on the perplexity level corresponding to each sentence, candidate sentences with potential garbled content are identified from at least one of the sentences, wherein the garbled content includes at least one uncommon character; Identify the target sentence that actually contains the garbled content from the candidate sentences, and delete the garbled content from the target sentence; The candidate sentences are determined to be the target sentences that actually contain garbled text through the following steps: Identify rare character sequences from the candidate sentences, wherein the rare character sequences consist of consecutive rare characters; When the number of rare characters in the rare character sequence is greater than a preset number, the candidate sentence is determined to be the target sentence with actual garbled content, and the rare character sequence is determined to be garbled content.

2. The method of claim 1, wherein, The identification of rare characters in each paragraph includes: Determine the common character set corresponding to the document, wherein the type of the common character set is related to the application field of the document; For each paragraph, identify rare characters that do not belong to the commonly used character set.

3. The method of claim 1, wherein, The calculation of the frequency of rare characters in each paragraph includes: For each paragraph, calculate the first frequency of occurrence of the uncommon character in that paragraph and the second frequency of occurrence of the uncommon character in that paragraph in the document.

4. The method according to claim 3, wherein, The step of determining the target paragraph from the at least one paragraph based on the frequency of occurrence of rare characters in each paragraph includes: Based on the first occurrence frequency corresponding to the rare characters in each paragraph, a target paragraph whose first occurrence frequency is greater than a first preset occurrence frequency is determined from at least one paragraph; And / or, based on the second occurrence frequency corresponding to the rare characters in each paragraph, determine the target paragraph from at least one paragraph whose second occurrence frequency is greater than a second preset occurrence frequency.

5. The method according to claim 1, wherein, The step of determining candidate sentences with potential garbled content from at least one sentence based on the perplexity level corresponding to each sentence includes: Determine the perplexity threshold corresponding to the document; From the at least one sentence, determine the corresponding candidate sentences whose perplexity is greater than the perplexity threshold.

6. The method according to claim 1, wherein the type of the rare character includes at least one of the following: ancient script code, mouth character code, symbol code, question mark code, and stick code.

7. A device for processing garbled text in documents, comprising: The uncommon character recognition module is used to divide the content of the document to be processed into at least one paragraph and recognize uncommon characters in each paragraph. The frequency of occurrence calculation module is used to calculate the frequency of occurrence of rare characters in each paragraph; the frequency of occurrence refers to at least one of the proportion of the rare characters in the paragraph and the proportion of the rare characters in the paragraph in the document to be processed; The target paragraph determination module is used to determine a target paragraph from the at least one paragraph based on the frequency of occurrence of rare characters in each paragraph; A perplexity calculation module is used to divide the target paragraph into at least one sentence and calculate the perplexity of each sentence, wherein the perplexity is used to indicate the degree of reasonableness of the sentence; A candidate sentence determination module is used to determine, based on the perplexity of each sentence, a candidate sentence with potential garbled content from at least one of the sentences, wherein the garbled content includes at least one uncommon character; The garbled content deletion module is used to determine the target sentence that actually contains the garbled content from the candidate sentences, and delete the garbled content in the target sentence; The garbled text removal module is used to determine the candidate sentence as the target sentence that actually contains garbled text through the following steps: Identify rare character sequences from the candidate sentences, wherein the rare character sequences consist of consecutive rare characters; When the number of rare characters in the rare character sequence is greater than a preset number, the candidate sentence is determined to be the target sentence with actual garbled content, and the rare character sequence is determined to be garbled content.

8. The apparatus according to claim 7, wherein, The uncommon character recognition module, when used to recognize uncommon characters in each paragraph, is specifically used for: Determine the common character set corresponding to the document, wherein the type of the common character set is related to the application field of the document; For each paragraph, identify rare characters that do not belong to the commonly used character set.

9. The apparatus according to claim 7, wherein, The frequency calculation module, when calculating the frequency of rare characters in each paragraph, is specifically used for: For each paragraph, calculate the first frequency of occurrence of the uncommon character in that paragraph and the second frequency of occurrence of the uncommon character in that paragraph in the document.

10. The apparatus according to claim 9, wherein, When the target paragraph determination module determines a target paragraph from at least one paragraph based on the frequency of occurrence of rare characters in each paragraph, it is specifically used for: Based on the first occurrence frequency corresponding to the rare characters in each paragraph, a target paragraph whose first occurrence frequency is greater than a first preset occurrence frequency is determined from at least one paragraph; And / or, based on the second occurrence frequency corresponding to the rare characters in each paragraph, determine the target paragraph from at least one paragraph whose second occurrence frequency is greater than a second preset occurrence frequency.

11. The apparatus according to claim 7, wherein, Based on the perplexity level corresponding to each of the sentences, candidate sentences with potential garbled content are identified from at least one of the sentences, including: Determine the perplexity threshold corresponding to the document; From the at least one sentence, determine the corresponding candidate sentences whose perplexity is greater than the perplexity threshold.

12. The apparatus according to claim 7, wherein the type of the rare character includes at least one of the following: ancient script code, mouth character code, symbol code, question mark code, and stick code.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images