Text anaphora resolution method and device, storage medium and electronic equipment
By identifying and dividing documents text and using natural language processing model to perform reference and digesting text blocks, the problem of digesting reference pronouns in documents is solved, and the accuracy and efficiency of information retrieval is improved.
Patent Information
- Application Number
- CN202412000190.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-05-02
AI Technical Summary
The prior art is difficult to effectively deal with the problem of decomposition of pronouns in documents, resulting in a decrease in the accuracy and efficiency of information retrieval.
By text recognition and division of the obtained documents, the text data is divided into multiple text blocks according to the preset text block size, and the natural language processing model is used to perform reference digestion processing for each text block, and replace the reference pronoun as its reference object.
It realizes the accurate dissolution of reference pronouns in the document, improves the accuracy and efficiency of information retrieval, and can better understand and deal with complex reference relationships.
Smart Images
Figure CN119918528A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of natural language processing, and more specifically, to a method, device, storage medium and electronic device for deconstructing text. Background Art
[0002] In the field of information technology and natural language processing, with the development of big data and artificial intelligence technology, in-depth analysis and understanding of document content has become increasingly important. Especially when dealing with a large number of documents, how to accurately extract useful information from the documents and understand the relationship between this information has become a key technical challenge. Among them, the problem of reference resolution is particularly prominent, which involves identifying and parsing the relationship between pronouns in the text and the nouns or noun phrases they refer to. Traditional document management systems often rely on keyword retrieval and cannot understand complex reference relationships. As a result, the accuracy and efficiency of information retrieval are greatly reduced.
[0003] Therefore, in the related art, there is a problem of how to resolve pronouns in documents.
[0004] Regarding the problem of how to resolve pronouns in documents in the related art, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a method, device, storage medium and electronic device for deconstructing a text, so as to at least solve the problem of how to handle deconstructing pronouns in a document in the related art.
[0006] According to an embodiment of the present application, a method for reference resolution of text is provided, the method comprising: performing text recognition on an acquired document to obtain text data; dividing the text data into multiple text blocks according to a preset text block size, wherein there is an association order between the multiple text blocks, and the association order is determined according to the order of texts corresponding to the text blocks in the document; for each of the multiple text blocks, performing reference resolution processing on the each text block through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution processing represents replacing the reference word in the text corresponding to the text block with the reference object of the reference word.
[0007] In an exemplary embodiment, for each of the multiple text blocks, a natural language processing model is used to perform a reference resolution process on each of the text blocks. Before obtaining the reference resolved text block, the method further includes: configuring a reference resolution prompt word for the natural language processing model, wherein the reference resolution prompt word includes a task description and a task example; the task description includes outputting a text after reference resolution process is performed on a second text according to a first text, wherein the first text is a preceding text of the second text, the task example includes an example text of the first text, an example text of the second text and an example text of the output text, and the type of the task example includes a first task example type and a second task example type, and in the In the first task example type, the example text of the first text is the first example text, the example text of the second text is the second example text, the example text of the output text is the second example text, and there is no pronoun in the second example text. In the second task example, the example text of the first text is the third example text, the example text of the second text is the fourth example text, and the example text of the output text is the fifth example text. There is a pronoun in the fourth example text, and the referent object corresponding to the pronoun in the fourth example text exists in the third example text. The fifth example text represents the text after the pronoun in the fourth example text is replaced with the corresponding referent object in the third example text.
[0008] In an exemplary embodiment, for each of the multiple text blocks, a reference resolution process is performed on each text block through a natural language processing model to obtain a text block after reference resolution, including: determining a first text block among the multiple text blocks as the first text input into the natural language processing model; determining a second text block among the multiple text blocks as the second text input into the natural language processing model, wherein; the first text block is a preceding text block of the second text block; obtaining a first output text output by the natural language processing model based on the first text block and the second text block; and when it is determined that there is no reference word in the first output text, determining the first output text as the text block after reference resolution.
[0009] In an exemplary embodiment, the method further includes: when it is determined that a pronoun exists in the first output text, determining a preset number of text blocks before the second text block as the first text input into the natural language processing model according to the association order between the multiple text blocks, and determining the second text block as the second text input into the natural language processing model; obtaining a second output text output by the natural language processing model based on the preset number of text blocks and the second text block; and determining the second output text as the text block after the reference resolution.
[0010] In an exemplary embodiment, after the multiple text blocks are subjected to reference resolution processing through a natural language processing model to obtain the reference resolved text blocks, the method further includes: performing vector encoding on the reference resolved text blocks to obtain text vectors, wherein the text vectors are used to represent the semantic features of the reference resolved text blocks; and saving the text vectors to a database.
[0011] In an exemplary embodiment, after the text vector is saved in a database, the method further includes: receiving a query request from a target object, wherein the query request is used to query text data in the document; vector encoding the query text corresponding to the query request to obtain a query text vector; calculating the similarity between the query text vector and the text vector in the database, and sending a text block corresponding to a text vector having a similarity greater than a preset similarity to the target object.
[0012] In an exemplary embodiment, text recognition is performed on an acquired document to obtain text data, including: parsing the document format of the document, extracting all text content of the document according to a text extraction method corresponding to the document format; denoising the entire text content to obtain denoised text content; saving the denoised text content as a preset text format to obtain the text data.
[0013] According to another embodiment of the present application, a device for reference resolution of text is provided, comprising: a recognition module, used to perform text recognition on an acquired document to obtain text data; a division module, used to divide the text data into multiple text blocks according to a preset text block size, wherein there is an association order between the multiple text blocks, and the association order is determined according to the order of texts corresponding to the text blocks in the document; a processing module, used to perform reference resolution processing on each of the multiple text blocks through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution processing represents replacing the reference word in the text corresponding to the text block with the reference object of the reference word.
[0014] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0015] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented.
[0016] Through this application, the text content of the document to be processed can be recognized, and then after the text data is extracted, it can be divided into text blocks of preset size according to the order of the text in the document, and multiple text blocks can be subjected to reference resolution processing through a natural language model, and the pronouns in the text blocks can be replaced with the referent objects of the pronouns, and finally the text blocks after reference resolution can be obtained. This can solve the problem of how to resolve pronouns in documents in the related art. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a hardware structure block diagram of a server device of a method for dereferencing text in an embodiment of the present application;
[0018] Figure 2 is a flow chart of a method for dereferencing a text according to an embodiment of the present application;
[0019] Figure 3 is a schematic diagram of a metrology equipment calibration management system according to an embodiment of the present application;
[0020] Figure 4 It is a structural block diagram of a device for deconstructing text according to an embodiment of the present application. DETAILED DESCRIPTION
[0021] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0023] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a text reference resolution method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0024] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for reference resolution of text in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the method for reference resolution of the above text is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0025] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0026] In this embodiment, a method for dereferential resolution of text is provided, which is applied to a baseboard management controller. The baseboard management controller includes a video transmission port, which is used to transmit image data to a display terminal via Ethernet, and the display terminal is used to display images. Figure 2 is a flowchart of a method for deconstructing text according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:
[0027] Step S202, performing text recognition on the acquired document to obtain text data;
[0028] Step S204, dividing the text data into a plurality of text blocks according to a preset text block size, wherein there is an association order between the plurality of text blocks, and the association order is determined according to the order of texts corresponding to the text blocks in the document;
[0029] Step S206, for each of the multiple text blocks, perform reference resolution processing on the each text block through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution processing means replacing the pronoun in the text corresponding to the text block with the referent object of the pronoun.
[0030] Through the above steps, the text content of the document to be processed can be recognized, and then after the text data is extracted, it can be divided into text blocks of preset size according to the order of the text in the document, and multiple text blocks can be subjected to reference resolution processing through a natural language model, and the pronouns in the text blocks can be replaced with the referent objects of the pronouns, and finally the text blocks after reference resolution can be obtained. This can solve the problem of how to process the resolution of pronouns in documents in the related art.
[0031] In an exemplary embodiment, for each of the multiple text blocks, a natural language processing model is used to perform a reference resolution process on each of the text blocks. Before obtaining the reference resolved text block, the method further includes: configuring a reference resolution prompt word for the natural language processing model, wherein the reference resolution prompt word includes a task description and a task example; the task description includes outputting a text after reference resolution process is performed on a second text according to a first text, wherein the first text is a preceding text of the second text, the task example includes an example text of the first text, an example text of the second text and an example text of the output text, and the type of the task example includes a first task example type and a second task example type, and in the In the first task example type, the example text of the first text is the first example text, the example text of the second text is the second example text, the example text of the output text is the second example text, and there is no pronoun in the second example text. In the second task example, the example text of the first text is the third example text, the example text of the second text is the fourth example text, and the example text of the output text is the fifth example text. There is a pronoun in the fourth example text, and the referent object corresponding to the pronoun in the fourth example text exists in the third example text. The fifth example text represents the text after the pronoun in the fourth example text is replaced with the corresponding referent object in the third example text.
[0032] Optionally, in the above embodiment, for example, Chunk is defined as the current text block, Context is defined as the preceding content of the current text block, Output is defined as the output text, and the task description and task examples are as follows:
[0033] Task description: "Resolve Chunk references based on Context".
[0034] Task example 1: Context: "How are you doing recently?", Chunk: "I'm fine", Output: "I'm fine."
[0035] Task example 2: Context: "The first snow in 2002 came a little later than usual", Chunk: "Japan and South Korea hosted the Football World Cup this year", Output: "Japan and South Korea hosted the Football World Cup in 2002".
[0036] Through the above embodiments, a precise text processing method is realized, which provides detailed reference resolution prompt words for the natural language processing model, so that the model can better understand and process the reference relationship in the text and output the processing results. This improves the accuracy and efficiency of text understanding and information retrieval, and can provide more accurate and coherent text processing results when processing complex references.
[0037] In an exemplary embodiment, for each of the multiple text blocks, a reference resolution process is performed on each text block through a natural language processing model to obtain a text block after reference resolution, including: determining a first text block among the multiple text blocks as the first text input into the natural language processing model; determining a second text block among the multiple text blocks as the second text input into the natural language processing model, wherein; the first text block is a preceding text block of the second text block; obtaining a first output text output by the natural language processing model based on the first text block and the second text block; and when it is determined that there is no reference word in the first output text, determining the first output text as the text block after reference resolution.
[0038] In an exemplary embodiment, the method further includes: when it is determined that a pronoun exists in the first output text, determining a preset number of text blocks before the second text block as the first text input into the natural language processing model according to the association order between the multiple text blocks, and determining the second text block as the second text input into the natural language processing model; obtaining a second output text output by the natural language processing model based on the preset number of text blocks and the second text block; and determining the second output text as the text block after the reference resolution.
[0039] Optionally, in the above embodiment, the text data in the document includes multiple text blocks, and there is a pronoun in the current text block. However, if the referent of the pronoun does not exist in the previous text block of the current text block, the pronoun cannot be replaced with the referent, and the output text still contains the pronoun. In this case, more contextual content is needed to determine the referent of the pronoun in the current text block, for example, searching for the referent of the pronoun in the current text block from the 5 preceding text blocks before the current text block, and after finding the referent, replacing the current text block with the text after the reference is resolved for output.
[0040] Optionally, through the above embodiment, the text search scope can be expanded to find the referent of the pronoun, not only limited to the adjacent text blocks, but also by digging deep into the preceding text blocks to accurately identify and replace the pronoun. This ensures that even when the referential relationship is not obvious or spans a large text distance, the system can correctly understand and process the text content, thereby providing more accurate and reliable results in information retrieval and content understanding.
[0041] In an exemplary embodiment, after the multiple text blocks are subjected to reference resolution processing through a natural language processing model to obtain the reference resolved text blocks, the method further includes: performing vector encoding on the reference resolved text blocks to obtain text vectors, wherein the text vectors are used to represent the semantic features of the reference resolved text blocks; and saving the text vectors to a database.
[0042] Optionally, in the above embodiment, for example, a research report on artificial intelligence mentions that "it has a wide range of applications", but it is not clear what "it" refers to. The natural language processing model identifies that "it" refers to the "deep learning" mentioned in the previous text. After processing, the text block "deep learning has a wide range of applications" after the reference resolution can be vector encoded to obtain a vector that can represent the semantic features of this text. This vector is then saved in the database for efficient retrieval and matching in the future.
[0043] In an exemplary embodiment, after the text vector is saved in a database, the method further includes: receiving a query request from a target object, wherein the query request is used to query text data in the document; vector encoding the query text corresponding to the query request to obtain a query text vector; calculating the similarity between the query text vector and the text vector in the database, and sending a text block corresponding to a text vector having a similarity greater than a preset similarity to the target object.
[0044] Optionally, in the above embodiment, for example, many text blocks that have been reference resolved and vector encoded are stored in the database. The user submits a query request, and the query text is "Application of deep learning in the medical field". After receiving the query request, the system first vector encodes the query text "Application of deep learning in the medical field" to obtain the query text vector. Then calculate the similarity between this query vector and the existing text vectors in the database, and find text blocks whose similarity is greater than a preset threshold, such as "Application of deep learning in medical image analysis", and finally send these related text contents to the user.
[0045] In an exemplary embodiment, text recognition is performed on an acquired document to obtain text data, including: parsing the document format of the document, extracting all text content of the document according to a text extraction method corresponding to the document format; denoising the entire text content to obtain denoised text content; saving the denoised text content as a preset text format to obtain the text data.
[0046] Optionally, in the above embodiment, for example, when processing a market analysis report PDF file containing charts and mixed format content, the PDF format is first parsed, and then the extracted text is subjected to noise reduction processing to remove some non-text elements, such as headers, footers, chart descriptions, etc., as well as unnecessary format information, and only retain the pure text content. After processing, the noise-reduced text is saved as a preset text format.
[0047] Obviously, the embodiments described above are only some embodiments of the present application, not all embodiments. In order to better understand the verification method of the above test case, the above process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present application, specifically:
[0048] In an optional embodiment, the reference resolution method of the text in the embodiment of the present application is described in combination with the RAG (Retrieval-Augmented Generation) system. The RAG system mainly includes three parts: knowledge vectorization, knowledge retrieval, and knowledge recall. This method solves the reference resolution problem in knowledge vectorization and knowledge retrieval, thereby improving the accuracy of knowledge recall. The text vectorization process is a complex processing process, such as Figure 3 As shown, including:
[0049] 1. Content extraction: Extract the plain text content from the original file and remove irrelevant information such as formatting.
[0050] 2. Block segmentation: According to the configured text block size, the plain text in the previous step is segmented into multiple text blocks. In the text segmentation process, it is usually decomposed by length or natural paragraphs, so it is easy to separate the pronoun from the object being referred to.
[0051] 3. Co-reference resolution: Use prompts and chain thinking (CoT) methods to identify and replace pronouns through the natural language processing model (LLM), ensure that each pronoun can be correctly mapped to its corresponding entity, and obtain the co-reference resolution result of the text block.
[0052] 4. Vectorization: Convert the reference-eliminating text output by the natural language processing model into a text vector.
[0053] 5. Storage: Store the vectorized text blocks in the database for subsequent retrieval.
[0054] Optionally, in the above-mentioned step of reference resolution, the process of chain thinking CoT includes:
[0055] 1. Identify pronouns: First, identify the pronouns in the text (such as "he", "she", "it").
[0056] 2. Contextual Analysis: Analyze the context surrounding the pronoun to determine possible antecedents.
[0057] 3. Relationship evaluation: Evaluate entity relationships in context to determine the most appropriate antecedent.
[0058] 4. Decision making process: Based on the above analysis, make an educated decision to map the pronoun to the correct entity.
[0059] 5. Example guidance: Provide examples with detailed reasoning steps to help the model learn how to apply CoT for reference resolution. For example: The input preceding text block Context is: The first snow in 2002 was a little later than in previous years. The input current text block Chunk is: That year, Japan and South Korea hosted the Football World Cup. CoT process: Identify "that year" as a pronoun, analyze the context, determine that "that year" refers to "2002", replace "that year" with "2002", and generate the final output result.
[0060] In an optional embodiment, the following is an example of a prompt word input to a natural language processing model:
[0061] ###You are a text processing expert, please process the text content according to the following rules;
[0062] 1. The input consists of two parts, Context and Chunk:
[0063] Context contains the preceding content of the current text block;
[0064] Chunk contains the current text block;
[0065] 2. Perform COREFERENCE RESOLUTION on Chunk based on Context:
[0066] ###Please imitate the thinking process in the following example to process text:
[0067] Context:[];
[0068] Chunk: Hello, how are you doing?
[0069] Is reference resolution needed: No=>Thinking: There is no pronoun that needs to be replaced, output according to the original question=>Output result: Hello, how are you doing recently?
[0070] ------------------------
[0071] Context: ["The first snow in 2002 came a little later than usual"];
[0072] Chunk: "Japan and South Korea hosted the World Cup this year";
[0073] Is reference resolution required: Yes=>Thinking: I need to replace "this year" in Chunk. According to the Context content, I need to replace "this year" with "2002"=>Output result: In 2002, Japan and South Korea hosted the Football World Cup;
[0074] ------------------------
[0075] Context: ["2024 company year-end summary report
[0076] Dear colleagues:
[0077] As 2024 comes to an end, we are delighted to welcome the harvest season of the company's operating results. In the past year, all employees of the company have worked hard in a complex and changing market environment and achieved remarkable achievements. The following is a summary of this year's operating conditions and related data display:
[0078] 1. Overall business overview
[0079] Operating Revenue: This year the company achieved total revenue of RMB 500 million, a year-on-year increase of 20%, setting a historical record since the company was founded.
[0080] Net profit: The full-year net profit reached RMB 120 million, an increase of 18% year-on-year, and the profit margin remained above 24%, showing the healthy growth of the company's business.
[0081] Market share: The company's domestic market share increased to 15%, consolidating its competitive position in the industry, especially smartphones, which, as a new product, provided the company with 30% of its revenue. ″]
[0082] Chunk: "R&D investment This year, the company invested RMB 80 million in R&D, accounting for 16% of its total revenue, an increase of 20% year-on-year. Technology upgrades and new product development have gradually become important drivers of the company's development."
[0083] Is reference resolution required: Yes=>Thinking: I need to replace "this year" and "new products" in Chunk. According to the Context content, "this year" needs to be replaced with "2024", and "new products" needs to be replaced with "smartphones"=>Output result: "R&D investment: In 2024, the total R&D investment was RMB 80 million, accounting for 16% of the company's total revenue, a year-on-year increase of 20%. Technology upgrades and smartphone R&D have gradually become important development drivers for the company. "
[0084] ------------------------
[0085] Context: ${Context}.
[0086] Chunk: ${Chunk}.
[0087] By combining the above-mentioned Prompt with the natural language processing model LLM, the problem of reference resolution in text blocks can be significantly solved.
[0088] Optionally, the above method can be used not only to handle the problem of document text reference resolution, but also to handle the reference resolution problem that occurs during the user's questioning process when the user conducts multiple rounds of question and answering to the RAG system. For example, the current question content of the user is subjected to reference resolution processing through a natural language processing model, specifically including: configuring reference resolution prompt words for the natural language processing model, wherein the reference resolution prompt words include a task description and a task example; the task description includes outputting reference resolution processing for the current question content based on historical question and answer content, and the historical question and answer content includes the user's historical question content and the model's historical answer content.
[0089] Optionally, in the above embodiment, during the knowledge retrieval process of RAG, it is necessary to semantically understand the user's questions, then search in the vector library, and input the retrieved knowledge and user questions into the natural language processing model to obtain the desired answers. When users conduct multiple rounds of question and answer sessions with the RAG system, the problem of reference resolution usually arises. Therefore, according to the idea of reference resolution of text content, it is also necessary to reference resolve the user's questions when the user searches. Also by using the Prompt+CoT+LLM method, the following is an example of a Prompt prompt word for reference resolution of user questions:
[0090] ###You are a text processing expert, please process the text content according to the following rules;
[0091] 1. The input consists of two parts, History and New Question:
[0092] -History contains the previous rounds of questions and answers for the current question;
[0093] -New Question contains the current question;
[0094] 2. Perform reference resolution on New Question based on History;
[0095] ###Please imitate the thinking process in the following example to process text:
[0096] History: [];
[0097] New Question: Hello, how are you doing recently?
[0098] Is reference resolution needed: No=>Thinking: There is no pronoun that needs to be replaced, output according to the original question=>Output result: Hello, how are you doing recently?
[0099] ------------------------
[0100] History: [“Q: Where is the capital of China?”
[0101] "A: The capital of China is Beijing."
[0102] “Q: Where is the capital of Hainan?”
[0103] "A: The capital of Hainan is Haikou."
[0104] New Question: Which of these two cities is tropical?
[0105] Is reference resolution needed: yes=>Thinking: I need to replace "these two cities" in New Question. According to the History content, I need to replace "these two cities" with "Beijing and Haikou"=>Output result: Which one of Beijing and Haikou belongs to the tropics?
[0106] ------------------------
[0107] Context: ${Context}.
[0108] Chunk: ${Chunk}.
[0109] By combining the above-mentioned Prompt with the natural language processing model LLM, the problem of reference resolution in user questions can be significantly solved.
[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0111] In this embodiment, a device for dereferencing a text is also provided, and the system is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0112] Figure 4 is a structural block diagram of a device for deconstructing text according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:
[0113] The recognition module 42 is used to perform text recognition on the acquired document to obtain text data;
[0114] A division module 44, configured to divide the text data into a plurality of text blocks according to a preset text block size, wherein there is an association order between the plurality of text blocks, and the association order is determined according to the order of texts corresponding to the text blocks in the document;
[0115] The processing module 46 is used to perform a reference resolution process on each of the multiple text blocks through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution process means replacing the pronoun in the text corresponding to the text block with the referent object of the pronoun.
[0116] Through the above device, the text content of the document to be processed can be recognized, and then after the text data is extracted, it can be divided into text blocks of preset size according to the order of the text in the document, and multiple text blocks can be subjected to reference resolution processing through a natural language model, and the pronouns in the text blocks can be replaced with the referent objects of the pronouns, and finally the text blocks after reference resolution can be obtained. This can solve the problem of how to resolve pronouns in documents in the related art.
[0117] In an exemplary embodiment, the processing module 46 is further used to configure a reference resolution prompt word for the natural language processing model, wherein the reference resolution prompt word includes a task description and a task example; the task description includes outputting a text after reference resolution processing is performed on the second text according to the first text, wherein the first text is the preceding text of the second text, the task example includes an example text of the first text, an example text of the second text and an example text of the output text, and the type of the task example includes a first task example type and a second task example type. In the first task example type, the example text of the first text is the first example text, the example text of the second text is the second example text, the example text of the output text is the second example text, and there is no reference word in the second example text. In the second task example, the example text of the first text is the third example text, the example text of the second text is the fourth example text, and the example text of the output text is the fifth example text. There is a reference word in the fourth example text, and the reference object corresponding to the reference word in the fourth example text exists in the third example text, and the fifth example text represents the text after the reference word in the fourth example text is replaced with the corresponding reference object in the third example text.
[0118] In an exemplary embodiment, the processing module 46 is also used to determine the first text block among the multiple text blocks as the first text input into the natural language processing model; determine the second text block among the multiple text blocks as the second text input into the natural language processing model, wherein; the first text block is a preceding text block of the second text block; obtain the first output text output by the natural language processing model based on the first text block and the second text block; and when it is determined that there is no pronoun in the first output text, determine the first output text as the text block after the pronoun resolution.
[0119] In an exemplary embodiment, the processing module 46 is also used to, when it is determined that there is a pronoun in the first output text, determine a preset number of text blocks before the second text block as the first text input into the natural language processing model according to the association order between the multiple text blocks, and determine the second text block as the second text input into the natural language processing model; obtain the second output text output by the natural language processing model based on the preset number of text blocks and the second text block; and determine the second output text as the text block after the reference resolution.
[0120] In an exemplary embodiment, the processing module 46 is further used to perform vector encoding on the text block after the reference resolution to obtain a text vector, wherein the text vector is used to represent the semantic features of the text block after the reference resolution; and save the text vector to a database.
[0121] In an exemplary embodiment, the processing module 46 is also used to receive a query request for a target object, wherein the query request is used to query text data in the document; vector encoding is performed on the query text corresponding to the query request to obtain a query text vector; the similarity between the query text vector and the text vector in the database is calculated, and the text block corresponding to the text vector having a similarity greater than a preset similarity is sent to the target object.
[0122] In an exemplary embodiment, the recognition module 42 is also used to parse the document format of the document, extract all the text content of the document according to the text extraction method corresponding to the document format; perform denoising on the all the text content to obtain the denoised text content; save the denoised text content as a preset text format to obtain the text data.
[0123] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0124] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0125] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:
[0126] S1, performing text recognition on the acquired document to obtain text data;
[0127] S2, dividing the text data into a plurality of text blocks according to a preset text block size, wherein there is an association order between the plurality of text blocks, and the association order is determined according to the order of texts corresponding to the text blocks in the document;
[0128] S3, for each of the multiple text blocks, perform reference resolution processing on the each text block through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution processing means replacing the pronoun in the text corresponding to the text block with the referent object of the pronoun.
[0129] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0130] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0131] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0132] S1, performing text recognition on the acquired document to obtain text data;
[0133] S2, dividing the text data into a plurality of text blocks according to a preset text block size, wherein there is an association order between the plurality of text blocks, and the association order is determined according to the order of texts corresponding to the text blocks in the document;
[0134] S3, for each of the multiple text blocks, perform reference resolution processing on the each text block through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution processing means replacing the pronoun in the text corresponding to the text block with the referent object of the pronoun.
[0135] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.
[0136] Optionally, in this embodiment, the computer program may be configured to perform the following steps by means of the computer program:
[0137] S1, performing text recognition on the acquired document to obtain text data;
[0138] S2, dividing the text data into a plurality of text blocks according to a preset text block size, wherein there is an association order between the plurality of text blocks, and the association order is determined according to the order of texts corresponding to the text blocks in the document;
[0139] S3, for each of the multiple text blocks, perform reference resolution processing on the each text block through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution processing means replacing the pronoun in the text corresponding to the text block with the referent object of the pronoun.
[0140] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0141] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.
[0142] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.
Claims
1. A method for dereferential resolution of text, characterized in that: The method comprises: Perform text recognition on the acquired document to obtain text data; The text data is divided into a plurality of text blocks according to a preset text block size, wherein there is an associated order between the plurality of text blocks, and the associated order is determined according to the order of texts corresponding to the text blocks in the document; for each of the plurality of text blocks, a reference resolution process is performed on the each text block through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution process represents replacing a reference word in the text corresponding to the text block with a reference object of the reference word.
2. The method according to claim 1, characterized in that: For each of the multiple text blocks, performing a reference resolution process on each of the text blocks by using a natural language processing model to obtain a text block after reference resolution, the method further includes: The natural language processing model is configured with reference resolution prompt words, wherein the reference resolution prompt words include a task description and a task example; the task description includes outputting a text after reference resolution processing is performed on a second text according to a first text, wherein the first text is a preceding text of the second text, the task example includes an example text of the first text, an example text of the second text, and an example text of an output text, and the types of the task example include a first task example type and a second task example type. In the first task example type, the example text of the first text is a first example text, the example text of the second text is a second example text, the example text of the output text is the second example text, and there is no reference word in the second example text. In the second task example, the example text of the first text is a third example text, the example text of the second text is a fourth example text, and the example text of the output text is the fifth example text. There is a reference word in the fourth example text, and the reference object corresponding to the reference word in the fourth example text exists in the third example text, and the fifth example text represents the text after the reference word in the fourth example text is replaced with the corresponding reference object in the third example text.
3. The method according to claim 2, characterized in that For each of the multiple text blocks, performing a reference resolution process on each of the text blocks through a natural language processing model to obtain a reference resolved text block, including: Determine a first text block among the multiple text blocks as the first text and input it into the natural language processing model; Determine a second text block among the multiple text blocks as the second text and input it into the natural language processing model, wherein the first text block is a preceding text block of the second text block; Obtain a first output text output by the natural language processing model according to the first text block and the second text block; When it is determined that no pronoun exists in the first output text, the first output text is determined as the pronoun resolved text block.
4. The method according to claim 3, characterized in that The method further comprises: In the case where it is determined that there is a anaphora in the first output text, a preset number of text blocks before the second text block are determined as the first texts to be input into the natural language processing model according to the association order between the multiple text blocks, and the second text block is determined as the second text to be input into the natural language processing model; Obtain a second output text output by the natural language processing model according to the preset number of text blocks and the second text block; The second output text is determined as the text block after the reference resolution.
5. The method according to claim 1, characterized in that After performing a reference resolution process on the multiple text blocks through a natural language processing model to obtain the reference resolved text blocks, the method further includes: Performing vector encoding on the text block after the coreference resolution to obtain a text vector, wherein the text vector is used to represent the semantic features of the text block after the coreference resolution; The text vector is saved in a database.
6. The method according to claim 5, characterized in that After the text vector is saved in the database, the method further includes: Receiving a query request of a target object, wherein the query request is used to query text data in the document; Performing vector encoding on the query text corresponding to the query request to obtain a query text vector; The similarity between the query text vector and the text vector in the database is calculated, and the text blocks corresponding to the text vectors having the similarity greater than a preset similarity are sent to the target object.
7. The method according to claim 1, characterized in that Perform text recognition on the acquired documents to obtain text data, including: Parsing the document format of the document, and extracting all text contents of the document according to a text extraction method corresponding to the document format; Denoising all the text content to obtain denoised text content; The denoised text content is saved in a preset text format to obtain the text data.
8. A device for deconstructing text, characterized in that: The device comprises: A recognition module is used to perform text recognition on the acquired document to obtain text data; A division module is used to divide the text data into multiple text blocks according to a preset text block size, wherein there is an association order between the multiple text blocks, and the association order is determined according to the order of the texts corresponding to the text blocks in the document; a processing module is used to perform a reference resolution process on each of the multiple text blocks through a natural language processing model to obtain a text block after reference resolution, wherein the reference resolution process means replacing the reference word in the text corresponding to the text block with the reference object of the reference word.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.