Document processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a document processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed text, processing the to-be-processed text, and obtaining an ordered word source set containing a plurality of to-be-processed word sources; the to-be-processed word source is processed through the combined recognition model, abnormal words corresponding to the to-be-processed text are obtained, and the abnormal words are words which are not interpreted in the to-be-processed text and are not easy to understand; corresponding annotation information is generated based on the abnormal word, and the annotation information is at least used for indicating the position of the abnormal word in the to-be-processed text. The to-be-processed file is decomposed into the to-be-processed word sources, the to-be-processed word sources are detected and recognized through the combined recognition model, the abnormal words in the to-be-processed text are determined, the corresponding annotation information is generated, efficient detection and recognition of the abnormal words in the to-be-processed text are achieved, the abnormal word detection efficiency is improved, and the user experience is improved. Therefore, the efficiency and the quality of editing and processing the document are improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology, and in particular, to a document processing method, device, electronic device, and storage medium. Background Art

[0002] Currently, document applications are widely used in all walks of life. Based on traditional offline documents, online documents and corresponding functions have been further developed, realizing functions such as multi-person simultaneous editing and cloud storage of document content, which greatly improves the efficiency of document editing.

[0003] For the application scenario of online document editing, the prior art provides an errata function for document content, that is, for example, functions such as identifying typos and incorrect punctuation in documents. However, abnormal words in the document content that are difficult to understand cannot be effectively identified and need to be manually checked and modified by users, which affects the editing and processing efficiency of online documents. Summary of the invention

[0004] The embodiments of the present disclosure provide a document processing method, device, electronic device and storage medium to overcome the problem of being unable to recognize abnormal words that are difficult to understand in a document.

[0005] In a first aspect, an embodiment of the present disclosure provides a document processing method, including:

[0006] Acquire a text to be processed, and process the text to be processed to obtain an ordered word source set including multiple word sources to be processed; process the word sources to be processed by a combined recognition model to obtain abnormal words corresponding to the text to be processed, wherein the abnormal words are difficult to understand words that are not explained in the text to be processed; based on the abnormal words, generate corresponding annotation information, wherein the annotation information is at least used to indicate the position of the abnormal words in the text to be processed.

[0007] In a second aspect, an embodiment of the present disclosure provides a document processing device, including:

[0008] A preprocessing module, used for acquiring a text to be processed, and processing the text to be processed to obtain an ordered word source set including a plurality of word sources to be processed;

[0009] A recognition module, used for processing the word source to be processed by combining recognition models to obtain abnormal words corresponding to the text to be processed, wherein the abnormal words are difficult to understand words that are not explained in the text to be processed;

[0010] A generating module is used to generate corresponding annotation information based on the abnormal word, wherein the annotation information is at least used to indicate the position of the abnormal word in the text to be processed.

[0011] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0012] The memory stores computer-executable instructions;

[0013] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the document processing method as described in the first aspect and various possible designs of the first aspect.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the document processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0015] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the document processing method as described in the first aspect and various possible designs of the first aspect.

[0016] The document processing method, device, electronic device and storage medium provided in this embodiment obtain an ordered word source set containing multiple word sources to be processed by acquiring a text to be processed and processing the text to be processed; the word sources to be processed are processed by a combined recognition model to obtain abnormal words corresponding to the text to be processed, wherein the abnormal words are difficult to understand words that are not explained in the text to be processed; based on the abnormal words, corresponding annotation information is generated, and the annotation information is at least used to indicate the position of the abnormal words in the text to be processed. By decomposing the file to be processed into word sources to be processed, and detecting and identifying the word sources to be processed by a combined recognition model, the abnormal words in the text to be processed are determined, and the corresponding annotation information is generated, so as to achieve efficient detection and identification of abnormal words in the text to be processed, improve the efficiency of abnormal word detection, and then improve the efficiency and quality of editing and processing documents, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 An application scenario diagram of the document processing method provided by the embodiment of the present disclosure;

[0019] Figure 2Schematic diagram of the document processing method provided in the embodiment of the present disclosure Figure 1 ;

[0020] Figure 3 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;

[0021] Figure 4 A schematic diagram of a combined recognition model provided by an embodiment of the present disclosure;

[0022] Figure 5 for Figure 2 A flowchart of a specific implementation method of step S103 in the illustrated embodiment;

[0023] Figure 6 Schematic diagram of the document processing method provided in the embodiment of the present disclosure Figure 2 ;

[0024] Figure 7 for Figure 6 A flowchart of a specific implementation method of step S202 in the illustrated embodiment;

[0025] Figure 8 for Figure 6 A flowchart of a specific implementation method of step S203 in the illustrated embodiment;

[0026] Fig. 9 for Figure 6 A flowchart of a specific implementation method of step S204 in the illustrated embodiment;

[0027] Fig.10 A schematic diagram of a process for generating an abnormal word list provided by an embodiment of the present disclosure;

[0028] Fig.11 A structural block diagram of a document processing device provided by an embodiment of the present disclosure;

[0029] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0030] Fig.13 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] The application scenarios of the embodiments of the present disclosure are explained below:

[0034] Figure 1 An application scenario diagram of the document processing method provided in the embodiment of the present disclosure. The document processing method provided in the embodiment of the present disclosure can be applied to an application (APP, Application) with a document editing function. More specifically, for example, it can be applied to an application scenario of online text editing. The execution subject of this embodiment can be a terminal device that runs the above-mentioned application with a document editing function, or a server that deploys the server corresponding to the above-mentioned application, or other electronic devices that perform similar functions. Among them, when the execution subject is a terminal device, the terminal device executes the method provided in this embodiment by running the above-mentioned application; when the execution subject is a server, the server of the above-mentioned application with data and business management functions can be partially or completely run on the server, and the method provided in this embodiment is executed on the server side, and the terminal device runs the client of the application. The communication between the server and the terminal device is based on the server-client, so that the terminal device can obtain the execution result of the method provided in this embodiment and display it as needed.

[0035] Among them, in some embodiments, the terminal device or server can implement the document editing method provided by the embodiment of the present disclosure by running various computer executable instructions or computer programs. For example, the computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program that runs based on a browser environment. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Furthermore, in the process of implementing the document editing method provided by the embodiment of the present disclosure, the terminal device can execute the method by running a computer executable instruction or computer program set locally, or it can execute the method by calling a computer executable instruction or computer program set in an external server. In some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communications, cloud databases, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms. Among them, cloud services can be interactive processing services for terminal devices to call.

[0036] refer to Figure 1 As shown in, taking a terminal device as an example, the terminal device displays an online document by running the above-mentioned application with document editing function and edits the online document DOC_ by responding to user operations. In this process, the terminal device can detect the document content and identify abnormal content therein. For example, as shown in the figure, the normal content in the document is represented by "X", and the content of the above-mentioned document contains abnormal content "AAA", which is marked by underlining, etc., and the above-mentioned abnormal content is displayed in the "Detection Result" area, and the relevant abnormal reasons are explained to remind the user to confirm or modify the above-mentioned abnormal content, thereby achieving the purpose of detecting abnormal content in the document content.

[0037] In the prior art, the detection of document content is usually for punctuation errors, spelling errors and other erroneous contents with significant regularities, while abnormal words in the document that are difficult to understand cannot be effectively identified. The above abnormal words are not errors in essence, but are caused by differences in fields and other reasons. They can be understood by editing users but may not be understood by reading users, such as abbreviations and abbreviations of names. Therefore, on the one hand, since there are no spelling errors, it is difficult for terminal devices and applications to automatically identify based on preset rules. On the other hand, since editing users have prior knowledge, they can understand the "abnormal words" when they see them, resulting in the problem that even if manual inspection is implemented, the above abnormal words cannot be identified, resulting in low content quality of the document and low stable editing efficiency.

[0038] The embodiment of the present disclosure provides a document processing method to solve the above problems.

[0039] refer to Figure 2 , Figure 2 Schematic diagram of the document processing method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a terminal device. The document processing method includes:

[0040] Step S101: obtaining a text to be processed, and processing the text to be processed to obtain an ordered word source set including a plurality of word sources to be processed.

[0041] For example, reference Figure 1 As shown in the schematic diagram of the application scenario, in this embodiment, the document processing method provided is introduced with the terminal device as the execution subject. First, the terminal device loads the text to be processed by running a target application with a document editing function, wherein the above-mentioned target application can be an application dedicated to realizing the document editing function, such as an electronic document program; it can also be other comprehensive applications, such as an instant messaging program with an electronic document function; it can also be a browser program, which obtains the text to be processed by accessing the corresponding system platform and executes subsequent processing steps. It will not be described one by one here and can be set as needed.

[0042] Furthermore, after the terminal device loads the text to be processed through the target application, the text to be processed is processed to obtain an ordered word source set containing multiple word sources to be processed, wherein the process can be performed locally on the terminal device, that is, the terminal device splits the text to be processed to obtain multiple word sources, and then obtains the above-mentioned ordered word source set; it can also be performed outside the terminal device, for example, the text to be processed is sent to an external server, and after the server splits it, an ordered word source set consisting of multiple word sources is generated, and then the ordered word source set is sent back to the terminal device. Among them, the word source (Token) is the basic unit of text preprocessing, which usually refers to a part of the text separated by spaces, punctuation marks, etc., such as words, phrases, symbols, etc. The process of generating the word source to be processed, that is, the process of segmenting the text to be processed, is to segment the text to be processed into multiple word sources, and decompose the text into parts that are easier to understand and process, so that the subsequent model can understand and analyze the content of the text to be processed through the word source. Among them, in the process of processing the text to be processed and generating the word sources to be processed, it is necessary to generate the corresponding word sources based on appropriate granularity, such as word level granularity, subword level granularity, character level granularity, etc. It can be determined according to the implementation method and training method of the model. In this embodiment, the granularity of the generated word sources to be processed is, for example, word level granularity.

[0043] Step S102: Processing the word sources to be processed by combining the recognition models to obtain abnormal words corresponding to the text to be processed, where the abnormal words are difficult to understand words that are not explained in the text to be processed.

[0044] Exemplarily, after obtaining an ordered word source set containing multiple word sources to be processed, the terminal device calls a combined recognition model to process the word sources to be processed, wherein the combined recognition model can be deployed locally on the terminal device, or can be fully or partially deployed outside the terminal device. Specifically, the combined recognition model is composed of multiple models, each model is used to implement a corresponding recognition function, and the above-mentioned recognition models can be deployed in different locations respectively. For example, the combined recognition model is composed of a first model and a second model, wherein the first model is deployed locally on the terminal device and the second model is deployed on a cloud server; for another example, the first model is deployed on a first cloud server and the second model is deployed on a second cloud server.

[0045] Furthermore, the combined recognition model is at least used to identify the words in the text to be processed that are difficult to understand and have not been explained, and determine them as abnormal words. The above process is implemented by at least two models. In a possible implementation, the combined recognition model includes a word recognition model and a word interpretation model. The word recognition model is used to detect the difficult-to-understand words in the text to be processed, and the word interpretation model is used to detect the original text interpretation information corresponding to the difficult-to-understand words from the text to be processed; Figure 3 As shown, the specific implementation of step S102 includes:

[0046] Step S1021: Process the ordered word source set through the word recognition model to obtain the first difficult-to-understand word.

[0047] Step S1022: Process the first difficult-to-understand word and the ordered word source set through the word interpretation model to obtain abnormal words in the text to be processed.

[0048] Exemplarily, the combined recognition model includes a word recognition model and a word interpretation model. After obtaining the ordered word source set, the ordered word source set is first processed by the word recognition model, and the difficult-to-understand words in the text to be processed, such as English abbreviations, abbreviations, etc., are detected and processed, and determined as the first difficult-to-understand words. After that, the first difficult-to-understand words are processed by the word interpretation model to determine the difficult-to-understand and unexplained words in the text to be processed, that is, abnormal words.

[0049] Specifically, the first incomprehensible word obtained in the above steps is, for example, an abbreviation. For such abbreviations, there are abbreviations that have been explained in the text to be processed (the text to be processed contains original text explanation information), and abbreviations that have not been explained in the text to be processed (the text to be processed does not contain original text explanation information). For abbreviations that have been explained in the text to be processed, combined with the corresponding original text explanation information, the corresponding meaning can be accurately expressed. Therefore, it is necessary to exclude them from the scope of abnormal words. For abbreviations that have not been explained in the text to be processed, it is impossible to determine the specific meaning they represent. Therefore, in a possible implementation method, they can be determined as abnormal words and subsequent processing steps can be performed. The word interpretation model is a model that detects whether the text to be processed contains the original text explanation information corresponding to the difficult-to-understand words. By combining the word interpretation model with the text to be processed, the obtained difficult-to-understand words can be filtered, thereby obtaining difficult-to-understand words that have not been explained in the text to be processed, that is, abnormal words.

[0050] Figure 4 A schematic diagram of a combined recognition model provided by an embodiment of the present disclosure is shown below in combination with Figure 4 The above process is described in more detail, such as Figure 4As shown, exemplarily, first, after obtaining an ordered word source set according to the text to be processed, the ordered word source set is input into the combined recognition model, which includes a word recognition model and a word interpretation model that are arranged in an ordered manner. After the ordered word source set is first input into the word recognition model, the first difficult-to-understand word is obtained, for example, including [AA], [BB], and [CC] shown in the figure. Afterwards, the first difficult-to-understand word is input into the word interpretation model, and the word interpretation model is combined with the ordered word source set corresponding to the text to be processed input previously, and the first difficult-to-understand word is filtered out, and [BB] is filtered out (that is, [BB] has corresponding original text interpretation information in the text to be processed), and the abnormal words are [AA] and [CC]. Subsequent processing steps are then performed based on the abnormal words.

[0051] In the steps of this embodiment, by setting a combined recognition model composed of multiple models to detect the text to be processed, accurate detection of abnormal words in the text to be processed is achieved, false detections and missed detections are reduced, and detection efficiency and accuracy are improved.

[0052] Step S103: Generate corresponding annotation information based on the abnormal word, where the annotation information is at least used to indicate the position of the abnormal word in the text to be processed.

[0053] Exemplarily, after generating the abnormal word, further generate annotation information corresponding to the abnormal word, the annotation information is used to at least indicate the position of the abnormal word in the text to be processed, for example, the position of the abnormal word in the text to be processed, more specifically, the annotation information contains information such as the break number, line number and offset corresponding to the abnormal word, that is, positioning information, through which the positioning information representing the position of the abnormal word in the text to be processed is used to locate the abnormal word. Afterwards, based on the annotation information, the abnormal word can be marked, for example, in the text to be processed, the abnormal word can be highlighted by annotation, color, shadow, etc., so as to realize the user reminder for the abnormal word determined above.

[0054] In a possible implementation, the annotation information also includes semantic interpretation information corresponding to the abnormal words, and the semantic interpretation information is used to characterize the semantics of the abnormal words in the text scenario corresponding to the text to be processed. That is, information that explains and describes the abnormal words that are difficult to understand. Among them, there are many implementation methods for generating corresponding annotation information based on the abnormal words. In a possible implementation, the specific implementation method of step S103 includes:

[0055] Step S103A: Obtain entry data.

[0056] Step S103B: Generate semantic interpretation information corresponding to abnormal words through the entry data.

[0057] Exemplarily, the entry data may be pre-generated information such as text, images, web page links, etc., used to explain the preset vocabulary. The entry data may be pre-stored on the server side and updated in real time, and the user may edit the entry data as needed. Specifically, the entry data is retrieved through abnormal words, and when the corresponding preset vocabulary in the entry data is hit, the entry interpretation corresponding to the preset vocabulary in the entry data is used to generate the corresponding semantic interpretation information.

[0058] In another possible implementation, a specific implementation of step S103 includes:

[0059] Step S103C: Process the ordered word source set and abnormal words through the pre-trained language model to generate semantic interpretation information corresponding to the abnormal words.

[0060] Exemplarily, in this embodiment, abnormal words and ordered word source sets are processed by a pre-trained language model. Specifically, a prompt word matching the language model is first generated based on the ordered word source set, and the text scene is expressed by the prompt word. Then, the prompt word and the abnormal word are input into the language model, thereby utilizing the text generation capability of the language model to generate semantic text, i.e., semantic interpretation information, of the abnormal word in the text scene corresponding to the text to be processed.

[0061] Furthermore, in another possible implementation, the semantic introduction information can be generated by combining the above two methods. Specifically, Figure 5 As shown, the specific implementation of step S103 includes:

[0062] Step S1031: Obtain entry data.

[0063] Step S1032: obtaining the entry explanation of the preset vocabulary that the abnormal word hits by searching the entry data;

[0064] Step S1033: Process the ordered word source set and word entry explanations through the pre-trained language model to generate semantic explanation information corresponding to the abnormal words.

[0065] In the steps of this embodiment, first, the entry data is obtained, and the entry data contains several preset words and corresponding entry explanations. After that, the entry data is retrieved through the abnormal words to obtain the preset words that hit, and the entry explanations corresponding to the preset words. After that, the first prompt word representing the general explanation information of the abnormal words is generated based on the entry explanation, and the second prompt word of the text scene is generated through the ordered word source set. Finally, the first prompt word and the second prompt word are input into the pre-trained language model for processing, and the text generation ability of the language model is used to generate semantic interpretation information with both text scene features and entry explanation features. The generated semantic interpretation information of the abnormal words can take into account the preset entry data and text scenes, so that the interpretation of the abnormal words is more accurate.

[0066] Furthermore, after obtaining the semantic interpretation information, optionally, the method further includes:

[0067] Persist the annotation information in the text to be processed to generate an annotation text, wherein the annotation information includes at least one of the following: explanation text, web page link, and pre-generated entry text. And / or, display the text to be processed and the corresponding annotation information, and update the annotation information in real time in response to the modification instruction for the text to be processed.

[0068] The annotation text is the text to be processed that contains the persistent annotation information. The annotation text can be stored on the server side. After the annotation text is generated, it can be opened and edited here, including the modification of the text content in the annotation text (i.e. the text to be processed in the previous step) and the annotation information. There are many ways to implement the annotation information, such as: one or more of the explanation text, web page link, and pre-generated entry text, which can be set as needed.

[0069] In a possible implementation, the annotation text can be understood as a text data with a specific data format, which is an improved data relative to traditional text data. When the text to be processed is traditional text data without the above-mentioned processing steps, after the text to be processed is opened, the text to be processed can be automatically converted into the annotation text, and then the text to be processed is automatically processed based on the above-mentioned embodiment steps and the corresponding annotation information is generated, and then the annotation information is written into the annotation text. Alternatively, after the text to be processed is automatically processed based on the above-mentioned embodiment steps and the corresponding annotation information is generated, the text to be processed is converted into the annotation text, and the annotation information is written into the annotation text. The above two methods for realizing the persistence of annotation information can be set as needed. At the same time, optionally, when the scheme provided in this embodiment is executed on a terminal device with a display unit, the above-mentioned text to be processed and the annotation information can be displayed at the same time, and the specific display method will not be repeated here.

[0070] In this embodiment, by obtaining the text to be processed and processing the text to be processed, an ordered word source set containing multiple word sources to be processed is obtained; by processing the word sources to be processed by the combined recognition model, abnormal words corresponding to the text to be processed are obtained, and the abnormal words are difficult to understand words that are not explained in the text to be processed; based on the abnormal words, corresponding annotation information is generated, and the annotation information is at least used to indicate the position of the abnormal words in the text to be processed. By decomposing the file to be processed into word sources to be processed, and detecting and identifying the word sources to be processed by the combined recognition model, the abnormal words in the text to be processed are determined, and the corresponding annotation information is generated, so as to achieve efficient detection and identification of abnormal words in the text to be processed, improve the efficiency of abnormal word detection, and then improve the efficiency and quality of editing and processing documents, and improve the user experience.

[0071] refer to Figure 6 , Figure 6 Schematic diagram of the document processing method provided in the embodiment of the present disclosure Figure 2 In this embodiment Figure 2 Based on the illustrated embodiment, step S102 is further refined, and the document processing method includes:

[0072] Step S201: obtaining a text to be processed, and processing the text to be processed to obtain an ordered word source set including a plurality of word sources to be processed.

[0073] Step S202: Process the ordered word source set through a word recognition model to obtain a first difficult-to-understand word.

[0074] For example, in combination Figure 2 In the introduction of the illustrated embodiment, the combined recognition model includes a word recognition model and a word interpretation model. The word recognition model is used to detect the difficult-to-understand words in the text to be processed, and the word interpretation model is used to detect the original text interpretation information corresponding to the difficult-to-understand words from the text to be processed. After obtaining the ordered word source set, the ordered word source set is first processed by the word recognition model, wherein the word recognition model includes a word list recognition model and a word list-free recognition model. The word list recognition model is used to determine the difficult-to-understand words based on a preset difficult-to-understand word list; the word list-free recognition model is used to determine the difficult-to-understand words based on the semantic features of the ordered word source set. In short, the word list recognition model is a model that determines the difficult-to-understand words through a preset difficult-to-understand word list, while the word list-free recognition model is a pre-trained recognition model that determines the difficult-to-understand words by recognizing the semantic features of the ordered word source set.

[0075] For example, Figure 7 As shown, the specific implementation of step S202 includes:

[0076] Step S2021: Process the ordered word source set based on the word list recognition model to obtain the first word source.

[0077] Step S2022: Process the ordered word source set based on the word list-free recognition model to obtain a second word source.

[0078] Step S2023: Merge the first word source and the second word source to obtain a first difficult-to-understand word.

[0079] Exemplarily, the ordered word source set is processed by the word list recognition model and the word list-free recognition model respectively, wherein the word list recognition model retrieves the ordered word source set based on the difficult-to-understand word list to obtain the corresponding difficult-to-understand words, i.e., the first word source; and the word list-free recognition model obtains the corresponding difficult-to-understand words, i.e., the second word source, based on feature recognition of each word source in the ordered word source set. After merging and removing duplicates of the difficult-to-understand words obtained by the above two models, the first difficult-to-understand word can be obtained.

[0080] Step S203: filtering the first difficult-to-understand words based on prior knowledge to obtain second difficult-to-understand words.

[0081] The first difficult-to-understand word obtained in the above steps is equivalent to predicting difficult-to-understand words from an ordered word source set based on a "blacklist" method. Through the above steps in this embodiment, the accuracy of the model regression can be effectively improved. In the steps of this embodiment, in order to further improve the accuracy of predicting abnormal words, based on the steps of the above embodiment, the first difficult-to-understand word obtained in the above steps is screened in a "white list" screening method based on prior knowledge, and the words that are mistakenly judged as difficult-to-understand words are filtered out, so as to obtain a more accurate difficult-to-understand word, that is, the second difficult-to-understand word. For example, Figure 8 As shown, the specific implementation of step S203 includes:

[0082] Step S2031: determining the word origin that matches the prior knowledge in the first difficult to understand word as an explainable word origin;

[0083] Step S2032: Filter out the explainable word sources in the first difficult-to-understand word to obtain a second difficult-to-understand word.

[0084] Exemplarily, the prior knowledge includes at least one of the following: Internet information, a manually configured vocabulary whitelist, and preset punctuation marks. For example, the first difficult-to-understand word is screened through Internet information, and the word sources (such as network words) that are judged to be difficult to understand but can be explained through Internet information are selected, that is, the explainable word sources. For another example, by accessing the vocabulary whitelist manually configured by the user, the word sources in the first difficult-to-understand word that are judged to be difficult to understand but can be explained through the vocabulary whitelist are selected; for another example, through a specific punctuation mark combination, such as book title marks (i.e., "《》"), the word sources (such as book names) in the first difficult-to-understand word that are judged to be difficult to understand but can be introduced through the above punctuation mark combination are selected. After the word sources hit in the above whitelist are determined as explainable word sources, the above explainable word sources are deleted from the first difficult-to-understand word, and a filtered second difficult-to-understand word with higher accuracy can be obtained.

[0085] Step S204: Detect the text to be processed based on the second difficult-to-understand word through the word interpretation model to determine abnormal words.

[0086] Exemplarily, after obtaining the second difficult to understand word, the second difficult to understand word refers to the word source that meets the "difficult to understand" feature in appearance, form or semantics in the text to be processed. However, in some possible cases, the text to be processed will record the relevant explanations for the above difficult to understand words, that is, the difficult to understand word can be explained through the original text explanation information recorded in the text to be processed. For example, there is an English abbreviation "AB" in the text to be processed, and the English abbreviation "AB" formally meets the judgment conditions of the difficult to understand word. Therefore, after the above steps, the English abbreviation "AB" will be judged as the second difficult to understand word. After that, in this embodiment, by calling the explanation model, the text to be processed is further retrieved. If the text to be processed does not contain an explanation for the English abbreviation "AB", the English abbreviation "AB" is determined as an abnormal word; and if the text to be processed contains an explanation for the English abbreviation "AB", the English abbreviation "AB" is not determined as an abnormal word.

[0087] For example, Fig. 9 As shown, the specific implementation of step S204 includes:

[0088] Step S2041: using the word interpretation model, locate the relevant text content corresponding to the second difficult-to-understand word in the text to be processed.

[0089] Step S2042: Determine whether the relevant text content contains original text explanation information for the second difficult-to-understand word, and generate an abnormal word list based on the determination result.

[0090] Exemplarily, the word interpretation model can be a pre-trained feature recognition model. By inputting the ordered word source set and the second difficult-to-understand word corresponding to the text to be processed into the model, the word interpretation model determines the target word source in the ordered word source set that is close to the feature distance of the second difficult-to-understand word through feature comparison, and the word source within a certain range before and after the target word source, which are collectively used as the relevant text content. For example, the second difficult-to-understand word is the English abbreviation "AB". The word interpretation model can determine the location of the English abbreviation "AB" and the N characters before and after the location, that is, the relevant text content. Afterwards, it is determined whether there is original text interpretation information for the second difficult-to-understand word in the relevant text content. If so, the second difficult-to-understand word is an interpretable word; if not, the second difficult-to-understand word is an uninterpretable word. Finally, the set of all uninterpretable words is used as an abnormal word list, and each uninterpretable word in the abnormal word list is an abnormal word.

[0091] Of course, in another possible implementation, the step of filtering the first difficult-to-understand word can be omitted, that is, the first difficult-to-understand word is directly processed through the interpretation model to generate an abnormal word list. Specifically, the word interpretation model is used to locate the relevant text content corresponding to the first difficult-to-understand word in the text to be processed; it is determined whether the relevant text content contains the original text interpretation information for the first difficult-to-understand word, and the abnormal word list is generated according to the determination result. The specific implementation process is similar and will not be repeated here.

[0092] Fig.10 A schematic diagram of a process for generating an abnormal word list provided in an embodiment of the present disclosure is shown below. Fig.10 The above process is further introduced as follows: Fig.10 As shown, exemplarily, first, an ordered word source set is generated by processing the text to be processed, and the combined recognition model is input, and a word recognition model and a word interpretation model are sequentially set on the model link of the combined recognition model, wherein the word recognition model is composed of a word list recognition model and a word list-free recognition model, and the ordered word source set is first processed by the word list recognition model and the word list-free recognition model, and the results are merged to obtain a first difficult-to-understand word; then, the first difficult-to-understand word is filtered through prior knowledge to obtain a second difficult-to-understand word; finally, the second difficult-to-understand word model is processed through the word interpretation model to determine the abnormal word.

[0093] Step S205: Generate corresponding annotation information based on the abnormal word, where the annotation information is at least used to indicate the position of the abnormal word in the text to be processed and the semantics of the abnormal word in the text scenario corresponding to the text to be processed.

[0094] Step S206: displaying the text to be processed and the corresponding annotation information, and updating the annotation information in real time in response to the modification instruction for the text to be processed.

[0095] Further, exemplarily, after the abnormal word is generated, the abnormal word can be further processed through the engineering link to generate annotation information representing the position of the abnormal word in the text to be processed and the semantics of the abnormal word in the text scenario corresponding to the text to be processed, and displayed in the application. At the same time, in the application scenario of real-time editing, when the terminal device receives a modification instruction for the text to be processed, on the one hand, the text to be processed is modified, and on the other hand, the corresponding annotation information is synchronously updated for the modified part of the text to be processed. For example, after the English abbreviation "AB" determined to be an abnormal word in the text to be processed is changed to "ABC", the annotation information for the abnormal word "ABC" will be updated to the content of the English abbreviation "ABC"; for another example, after the English abbreviation "AB" is added to the text to be processed, the terminal device will generate annotation information for the abnormal word "AB".

[0096] In this embodiment, the implementation method of step S201 is the same as that of the present disclosure. Figure 2 The implementation method of step S101 in the illustrated embodiment is the same and will not be described in detail here.

[0097] Corresponding to the document processing method of the above embodiment, Fig.11 This is a structural block diagram of a document processing device provided in an embodiment of the present disclosure. The methods described in the above embodiments can be executed by the document processing device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. The electronic device can include but is not limited to a mobile terminal with big data processing capabilities, and a fixed terminal with big data processing capabilities such as a desktop computer and a supercomputer.

[0098] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig.11 , the document processing device 3 comprises:

[0099] A preprocessing module 31 is used to obtain a text to be processed and process the text to be processed to obtain an ordered word source set including a plurality of word sources to be processed;

[0100] The recognition module 32 is used to process the word source to be processed by combining the recognition model to obtain abnormal words corresponding to the text to be processed, where the abnormal words are difficult to understand words that are not explained in the text to be processed;

[0101] The generating module 33 is used to generate corresponding annotation information based on the abnormal word, and the annotation information is at least used to indicate the position of the abnormal word in the text to be processed.

[0102] According to one or more embodiments of the present disclosure, the combined recognition model includes a word recognition model and a word interpretation model, the word recognition model is used to detect difficult-to-understand words in the text to be processed, and the word interpretation model is used to detect original text interpretation information corresponding to the difficult-to-understand words from the text to be processed; the recognition module 32 is specifically used to: process the ordered word source set through the word recognition model to obtain the first difficult-to-understand word; process the first difficult-to-understand word and the ordered word source set through the word interpretation model to obtain abnormal words in the text to be processed.

[0103] According to one or more embodiments of the present disclosure, the word recognition model includes a word list recognition model and a word list-free recognition model. The word list recognition model is used to determine the difficult-to-understand words based on a preset difficult-to-understand word list; the word list-free recognition model is used to determine the difficult-to-understand words based on the semantic features of the ordered word source set; when the recognition module 32 processes the ordered word source set through the word recognition model to obtain the first difficult-to-understand word, it is specifically used to: process the ordered word source set based on the word list recognition model to obtain the first word source; process the ordered word source set based on the word list-free recognition model to obtain the second word source; merge the first word source and the second word source to obtain the first difficult-to-understand word.

[0104] According to one or more embodiments of the present disclosure, the recognition module 32 is also used to: filter the first difficult-to-understand word based on prior knowledge to obtain the second difficult-to-understand word; when the recognition module 32 processes the first difficult-to-understand word and the ordered word source set through the word interpretation model to obtain the abnormal word in the text to be processed, it is specifically used to: detect the text to be processed based on the second difficult-to-understand word through the word interpretation model to determine the abnormal word.

[0105] According to one or more embodiments of the present disclosure, the prior knowledge includes at least one of the following: Internet information, a manually configured vocabulary whitelist, and preset punctuation marks; when the recognition module 32 filters the first difficult-to-understand word based on the prior knowledge to obtain the second difficult-to-understand word, it is specifically used to: determine the word source in the first difficult-to-understand word that matches the prior knowledge as an explainable word source; filter out the explainable word source in the first difficult-to-understand word to obtain the second difficult-to-understand word.

[0106] According to one or more embodiments of the present disclosure, when the recognition module 32 processes the first difficult-to-understand word and the ordered word source set through the word interpretation model to obtain abnormal words in the text to be processed, it is specifically used to: locate the relevant text content corresponding to the first difficult-to-understand word in the text to be processed through the word interpretation model; determine whether the relevant text content contains original text interpretation information for the first difficult-to-understand word, and generate a list of abnormal words based on the determination result.

[0107] According to one or more embodiments of the present disclosure, the annotation information also includes semantic interpretation information corresponding to the abnormal words, and the semantic interpretation information is used to characterize the semantics of the abnormal words in the text scenario corresponding to the text to be processed.

[0108] According to one or more embodiments of the present disclosure, when the generation module generates corresponding annotation information based on abnormal words, it is specifically used to: obtain entry data, and generate semantic interpretation information corresponding to the abnormal words through the entry data, and / or, through a pre-trained language model, process the ordered word source set and the abnormal words to generate semantic interpretation information corresponding to the abnormal words, and / or; obtain the entry interpretation of the preset vocabulary hit by the abnormal word by retrieving the entry data, and through the pre-trained language model, process the ordered word source set and the abnormal words to generate semantic interpretation information corresponding to the abnormal words.

[0109] According to one or more embodiments of the present disclosure, the generation module 33 is also used to perform at least one of the following: persisting annotation information in the text to be processed and generating annotated text, wherein the annotation information includes at least one of the following: explanation text, web page link, and pre-generated entry text; displaying the text to be processed and the corresponding annotation information, and updating the annotation information in real time in response to modification instructions for the text to be processed.

[0110] The preprocessing module 31, the identification module 32 and the generation module 33 are connected in sequence. The document processing device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0111] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.12 As shown, the electronic device 4 includes:

[0112] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0113] The memory 42 stores computer executable instructions;

[0114] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 9 The document processing method in the illustrated embodiment.

[0115] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0116] For related instructions, please refer to Figure 2-Figure 9 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0117] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 9 The document processing method provided by any one of the corresponding embodiments.

[0118] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 9 The document processing method provided by any one of the corresponding embodiments.

[0119] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0120] refer to Fig.13 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0121] like Fig.13 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0122] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.13The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0123] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0124] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0125] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0126] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0127] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.

[0130] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0131] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] In a first aspect, according to one or more embodiments of the present disclosure, a document processing method is provided, comprising:

[0133] Acquire a text to be processed, and process the text to be processed to obtain an ordered word source set including multiple word sources to be processed; process the word sources to be processed by a combined recognition model to obtain abnormal words corresponding to the text to be processed, wherein the abnormal words are difficult to understand words that are not explained in the text to be processed; based on the abnormal words, generate corresponding annotation information, wherein the annotation information is at least used to indicate the position of the abnormal words in the text to be processed.

[0134] According to one or more embodiments of the present disclosure, the combined recognition model includes a word recognition model and a word interpretation model, the word recognition model is used to detect the difficult-to-understand words in the text to be processed, and the word interpretation model is used to detect the original text interpretation information corresponding to the difficult-to-understand words from the text to be processed; the word sources to be processed are processed by the combined recognition model to obtain abnormal words corresponding to the text to be processed, including: processing the ordered word source set by the word recognition model to obtain a first difficult-to-understand word; processing the first difficult-to-understand word and the ordered word source set by the word interpretation model to obtain abnormal words in the text to be processed.

[0135] According to one or more embodiments of the present disclosure, the word recognition model includes a word list recognition model and a word list-free recognition model, the word list recognition model is used to determine the difficult-to-understand word based on a preset difficult-to-understand word list; the word list-free recognition model is used to determine the difficult-to-understand word based on the semantic features of the ordered word source set; the word recognition model processes the ordered word source set to obtain a first word source, including: processing the ordered word source set based on the word list recognition model to obtain a first word source; processing the ordered word source set based on the word list-free recognition model to obtain a second word source; merging the first word source and the second word source to obtain the first difficult-to-understand word.

[0136] According to one or more embodiments of the present disclosure, it also includes: filtering the first difficult-to-understand words based on prior knowledge to obtain second difficult-to-understand words; processing the first difficult-to-understand words and the ordered word source set through the word interpretation model to obtain abnormal words in the text to be processed, including: detecting the text to be processed based on the second difficult-to-understand words through the word interpretation model to determine the abnormal words.

[0137] According to one or more embodiments of the present disclosure, the prior knowledge includes at least one of the following: Internet information, a manually configured vocabulary whitelist, and preset punctuation marks; the filtering of the first difficult-to-understand word based on the prior knowledge to obtain the second difficult-to-understand word includes: determining the word source in the first difficult-to-understand word that matches the prior knowledge as an explainable word source; filtering out the explainable word source in the first difficult-to-understand word to obtain the second difficult-to-understand word.

[0138] According to one or more embodiments of the present disclosure, the first difficult-to-understand word and the ordered word source set are processed by the word interpretation model to obtain abnormal words in the text to be processed, including: locating relevant text content corresponding to the first difficult-to-understand word in the text to be processed through the word interpretation model; judging whether the relevant text content contains original text interpretation information for the first difficult-to-understand word, and generating a list of abnormal words based on the judgment result.

[0139] According to one or more embodiments of the present disclosure, the annotation information further includes semantic interpretation information corresponding to the abnormal word, and the semantic interpretation information is used to characterize the semantics of the abnormal word in a text scenario corresponding to the text to be processed.

[0140] According to one or more embodiments of the present disclosure, the generating corresponding annotation information based on the abnormal word includes: acquiring entry data, and generating semantic interpretation information corresponding to the abnormal word through the entry data, and / or, processing the ordered word source set and the abnormal word through a pre-trained language model, and generating semantic interpretation information corresponding to the abnormal word, and / or; obtaining the entry interpretation of the preset vocabulary hit by the abnormal word by retrieving the entry data, and processing the ordered word source set and the abnormal word through a pre-trained language model, and generating semantic interpretation information corresponding to the abnormal word.

[0141] According to one or more embodiments of the present disclosure, it also includes at least one of the following: persisting the annotation information in the text to be processed to generate annotated text, wherein the annotation information includes at least one of the following: explanation text, web page link, pre-generated entry text; displaying the text to be processed and the corresponding annotation information, and updating the annotation information in real time in response to modification instructions for the text to be processed.

[0142] In a second aspect, according to one or more embodiments of the present disclosure, there is provided a document processing device, comprising:

[0143] A preprocessing module, used for acquiring a text to be processed, and processing the text to be processed to obtain an ordered word source set including a plurality of word sources to be processed;

[0144] A recognition module, used for processing the word source to be processed by combining recognition models to obtain abnormal words corresponding to the text to be processed, wherein the abnormal words are difficult to understand words that are not explained in the text to be processed;

[0145] A generating module is used to generate corresponding annotation information based on the abnormal word, wherein the annotation information is at least used to indicate the position of the abnormal word in the text to be processed.

[0146] According to one or more embodiments of the present disclosure, the combined recognition model includes a word recognition model and a word interpretation model, the word recognition model is used to detect the difficult-to-understand words in the text to be processed, and the word interpretation model is used to detect the original text interpretation information corresponding to the difficult-to-understand words from the text to be processed; the recognition module is specifically used to: process the ordered word source set through the word recognition model to obtain a first difficult-to-understand word; process the first difficult-to-understand word and the ordered word source set through the word interpretation model to obtain abnormal words in the text to be processed.

[0147] According to one or more embodiments of the present disclosure, the word recognition model includes a recognition model with a vocabulary list and a recognition model without a vocabulary list, the recognition model with a vocabulary list is used to determine the difficult-to-understand word based on a preset difficult-to-understand vocabulary list; the recognition model without a vocabulary list is used to determine the difficult-to-understand word based on the semantic features of the ordered word source set; when the recognition module processes the ordered word source set through the word recognition model to obtain a first difficult-to-understand word, the recognition module is specifically used to: process the ordered word source set based on the recognition model with a vocabulary list to obtain a first word source; process the ordered word source set based on the recognition model without a vocabulary list to obtain a second word source; merge the first word source and the second word source to obtain the first difficult-to-understand word.

[0148] According to one or more embodiments of the present disclosure, the recognition module is also used to: filter the first difficult-to-understand word based on prior knowledge to obtain a second difficult-to-understand word; when the recognition module processes the first difficult-to-understand word and the ordered word source set through the word interpretation model to obtain abnormal words in the text to be processed, it is specifically used to: detect the text to be processed based on the second difficult-to-understand word through the word interpretation model to determine the abnormal word.

[0149] According to one or more embodiments of the present disclosure, the prior knowledge includes at least one of the following: Internet information, a manually configured vocabulary whitelist, and preset punctuation marks; when the recognition module filters the first difficult-to-understand word based on the prior knowledge to obtain the second difficult-to-understand word, it is specifically used to: determine the word source in the first difficult-to-understand word that matches the prior knowledge as an explainable word source; filter out the explainable word source in the first difficult-to-understand word to obtain the second difficult-to-understand word.

[0150] According to one or more embodiments of the present disclosure, when the recognition module processes the first difficult-to-understand word and the ordered word source set through the word interpretation model to obtain abnormal words in the text to be processed, it is specifically used to: locate the relevant text content corresponding to the first difficult-to-understand word in the text to be processed through the word interpretation model; determine whether the relevant text content contains original text interpretation information for the first difficult-to-understand word, and generate a list of abnormal words based on the determination result.

[0151] According to one or more embodiments of the present disclosure, the annotation information further includes semantic interpretation information corresponding to the abnormal word, and the semantic interpretation information is used to characterize the semantics of the abnormal word in a text scenario corresponding to the text to be processed.

[0152] According to one or more embodiments of the present disclosure, when the generation module generates corresponding annotation information based on the abnormal word, it is specifically used to: obtain entry data, and generate semantic interpretation information corresponding to the abnormal word through the entry data, and / or, through a pre-trained language model, process the ordered word source set and the abnormal word to generate semantic interpretation information corresponding to the abnormal word, and / or; obtain the entry interpretation of the preset vocabulary hit by the abnormal word by retrieving the entry data, and process the ordered word source set and the abnormal word through a pre-trained language model to generate semantic interpretation information corresponding to the abnormal word.

[0153] According to one or more embodiments of the present disclosure, the generation module is also used to perform at least one of the following: persist the annotation information in the text to be processed to generate annotated text, wherein the annotation information includes at least one of the following: explanation text, web page link, pre-generated entry text; display the text to be processed and the corresponding annotation information, and update the annotation information in real time in response to modification instructions for the text to be processed.

[0154] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0155] The memory stores computer-executable instructions;

[0156] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the document processing method as described in the first aspect and various possible designs of the first aspect.

[0157] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the document processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0158] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the document processing method as described in the first aspect and various possible designs of the first aspect.

[0159] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0160] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0161] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A document processing method, characterized in that: include: Acquire a text to be processed, and process the text to be processed to obtain an ordered word source set including a plurality of word sources to be processed; Processing the word source to be processed by combining recognition models to obtain abnormal words corresponding to the text to be processed, wherein the abnormal words are difficult to understand words that are not explained in the text to be processed; Based on the abnormal word, corresponding annotation information is generated, where the annotation information is at least used to indicate the position of the abnormal word in the text to be processed.

2. The document processing method according to claim 1, characterized in that: The combined recognition model includes a word recognition model and a word interpretation model, wherein the word recognition model is used to detect the difficult-to-understand words in the text to be processed, and the word interpretation model is used to detect the original text interpretation information corresponding to the difficult-to-understand words from the text to be processed; Processing the word source to be processed by combining the recognition model to obtain abnormal words corresponding to the text to be processed includes: Processing the ordered word source set through a word recognition model to obtain a first difficult-to-understand word; The first incomprehensible word and the ordered word source set are processed by the word interpretation model to obtain abnormal words in the text to be processed.

3. The document processing method according to claim 2, characterized in that: The word recognition model includes a word list recognition model and a word list-free recognition model, wherein the word list recognition model is used to determine the difficult-to-understand word based on a preset difficult-to-understand word list; the word list-free recognition model is used to determine the difficult-to-understand word based on the semantic features of the ordered word source set; The step of processing the ordered word source set through a word recognition model to obtain a first difficult-to-understand word includes: Processing the ordered word source set based on the word list recognition model to obtain a first word source; Processing the ordered word source set based on the word list-free recognition model to obtain a second word source; The first word source and the second word source are combined to obtain the first difficult-to-understand word.

4. The document processing method according to claim 2, characterized in that: Also includes: Filtering the first difficult-to-understand words based on prior knowledge to obtain second difficult-to-understand words; Processing the first difficult-to-understand word and the ordered word source set by the word interpretation model to obtain abnormal words in the text to be processed includes: The text to be processed is detected based on the second difficult-to-understand word through the word interpretation model to determine the abnormal word.

5. The document processing method according to claim 4, characterized in that: The prior knowledge includes at least one of the following: Internet information, manually configured vocabulary whitelist, preset punctuation; The filtering the first difficult-to-understand words based on prior knowledge to obtain second difficult-to-understand words includes: Determine the word origin that matches the prior knowledge in the first difficult to understand word as an explainable word origin; The explainable word origins in the first difficult to understand word are filtered out to obtain the second difficult to understand word.

6. The document processing method according to claim 2, characterized in that: Processing the first difficult-to-understand word and the ordered word source set by the word interpretation model to obtain abnormal words in the text to be processed includes: Using the word interpretation model, locating relevant text content corresponding to the first difficult-to-understand word in the text to be processed; It is determined whether the relevant text content contains original text explanation information for the first difficult-to-understand word, and a list of abnormal words is generated according to the determination result.

7. The document processing method according to claim 1, characterized in that: The annotation information also includes semantic interpretation information corresponding to the abnormal word, and the semantic interpretation information is used to represent the semantics of the abnormal word in the text scenario corresponding to the text to be processed.

8. The document processing method according to claim 7, characterized in that: The generating corresponding annotation information based on the abnormal word includes: Acquire entry data, and generate semantic interpretation information corresponding to the abnormal word through the entry data, and / or, Processing the ordered word source set and the abnormal word through a pre-trained language model to generate semantic interpretation information corresponding to the abnormal word, and / or; By retrieving the entry data, the entry explanation of the preset vocabulary hit by the abnormal word is obtained, and the ordered word source set and the abnormal word are processed through a pre-trained language model to generate semantic explanation information corresponding to the abnormal word.

9. The document processing method according to claim 1, characterized in that: Also includes at least one of the following: Persisting the annotation information in the text to be processed to generate an annotation text, wherein the annotation information includes at least one of the following: an explanation text, a webpage link, and a pre-generated entry text; The text to be processed and the corresponding annotation information are displayed, and in response to a modification instruction for the text to be processed, the annotation information is updated in real time.

10. A document processing device, characterized in that: include: A preprocessing module, used for acquiring a text to be processed, and processing the text to be processed to obtain an ordered word source set including a plurality of word sources to be processed; A recognition module, used for processing the word source to be processed by combining recognition models to obtain abnormal words corresponding to the text to be processed, wherein the abnormal words are difficult to understand words that are not explained in the text to be processed; A generating module is used to generate corresponding annotation information based on the abnormal word, wherein the annotation information is at least used to indicate the position of the abnormal word in the text to be processed.

11. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the document processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the document processing method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the document processing method according to any one of claims 1 to 9 is implemented.

Images