Document analysis method and device, electronic equipment and storage medium
By summarizing and identifying the to-process documents, determining the type of questions of users, solving the problem that automatic document summary tools in the prior art cannot give accurate answers, and achieving efficient and accurate document analysis and answers.
Patent Information
- Application Number
- CN202411872430.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-11
AI Technical Summary
Existing automatic document summary tools cannot provide accurate answers to user specific questions, lack interactivity and provide too broad summary.
By obtaining the pending documents and summarizing them to generate a document summary, identifying the user's target questioning intention, determining the type of questioning is document summary or document question-and-answer, and thus adopting corresponding analysis strategies to generate target answers.
It realizes efficient and accurate generation of target answers based on the specific needs of users, and improves the interactivity and accuracy of document analysis.
Smart Images
Figure CN119807406A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a document analysis method, device, electronic device and storage medium. Background Art
[0002] In the era of information explosion, electronic documents have become the main way to acquire and store knowledge. The challenge that follows is how to quickly and effectively extract key information from a large number of electronic documents, and then achieve in-depth understanding and utilization of electronic documents.
[0003] In the prior art, there are already some automatic document summarization tools that can help users quickly understand the general content of electronic documents to a certain extent. However, these automatic document summarization tools often lack interactivity and cannot meet the personalized needs of users for understanding electronic documents. Moreover, they can often only provide relatively broad document summaries and lack the ability to accurately answer specific questions.
[0004] Therefore, current automatic document summarization tools are unable to provide accurate answers to users' specific questions. Summary of the invention
[0005] To solve the above technical problems, the present application shows a document analysis method, device, electronic device and storage medium, so as to at least solve the problem that the automatic document summarization tool in the related art cannot give accurate answers to the user's specific questions. The technical solution of the present disclosure is as follows:
[0006] In a first aspect, the present application provides a document analysis method, the method comprising:
[0007] Obtaining a document to be processed, summarizing the document to be processed, and generating and storing a document summary;
[0008] After obtaining the target question for the document to be processed, performing intent recognition on the target question to determine the request type corresponding to the target question; the request type includes document summarization or document question and answer;
[0009] When the request type is document summary, obtaining a target answer corresponding to the target question based on the document summary;
[0010] When the request type is document question and answer, a target answer corresponding to the target question is obtained based on the document to be processed.
[0011] Optionally, summarizing the document to be processed to generate and store a document summary includes:
[0012] According to a preset segment length, the document to be processed is divided into a plurality of segments, and the length of each segment is not greater than the segment length;
[0013] Inputting the multiple segments into the first large language model respectively for summarization to obtain multiple segment summaries;
[0014] Inputting the plurality of segment summaries into the first language model for summarization to obtain a summary;
[0015] The segment summary, the mapping relationship between the segment summary and the segment, and the summary summary are used as a document summary, and the document summary is stored.
[0016] Optionally, the step of inputting the multiple segments into the first large language model for summarization to obtain multiple segment summaries includes:
[0017] Creating a plurality of threads, and determining the thread corresponding to each of the segments;
[0018] In each thread, the corresponding segment is input into the first largest language model for summarization to obtain a segment summary of the corresponding segment.
[0019] Optionally, dividing the to-be-processed document into a plurality of segments according to a preset segment length includes:
[0020] The document to be processed is divided into a plurality of segments according to a preset segment length and redundant length; wherein adjacent segments have overlapping parts and the length of the overlapping parts is the redundant length.
[0021] Optionally, after obtaining the document to be processed, the method further includes:
[0022] The document to be processed is input into a feature extraction model to obtain a feature vector of the document to be processed, and the feature vector is stored in a corresponding vector library.
[0023] Optionally, obtaining a target answer corresponding to the target question based on the document summary includes:
[0024] A text feature matching retrieval is performed according to the target question, and a target answer corresponding to the target question is determined from the document summary.
[0025] Optionally, obtaining a target answer corresponding to the target question based on the document to be processed includes:
[0026] Inputting the target question into a feature extraction model to obtain a question vector;
[0027] Searching in a vector library based on the question vector to obtain a feature vector related to the target question as a context vector;
[0028] The context vector and the target question are input into a second language model to obtain a target answer corresponding to the target question.
[0029] Optionally, after obtaining the document to be processed, the method further includes:
[0030] Determining the language used by the document to be processed as the target language;
[0031] The step of inputting the target question into the feature extraction model to obtain a question vector includes:
[0032] determining whether the target question is in the target language;
[0033] If yes, the target question is input into the feature extraction model to obtain a question vector;
[0034] If not, the target question is converted into the target language, and the converted target question is input into the feature extraction model to obtain a question vector.
[0035] Optionally, the performing intent recognition on the target question to determine the request type corresponding to the target question includes:
[0036] Based on the preset prompt word project, the target question is input into the question-answering model for intent recognition to determine the request type corresponding to the target question.
[0037] In a second aspect, an embodiment of the present invention provides a document analysis device, including:
[0038] An acquisition module is used to acquire a document to be processed, summarize the document to be processed, and generate and store a document summary;
[0039] An identification module, configured to obtain a target question about the document to be processed, perform intent identification on the target question, and determine a request type corresponding to the target question; the request type includes document summarization or document question and answer;
[0040] The answering module is used to obtain the target answer corresponding to the target question based on the document summary when the request type is document summarization; and to obtain the target answer corresponding to the target question based on the document to be processed when the request type is document question and answer.
[0041] In a third aspect, an embodiment of the present invention provides an electronic device, including:
[0042] processor;
[0043] a memory for storing instructions executable by the processor;
[0044] The processor is configured to execute the instructions to implement any of the document analysis methods described above.
[0045] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of a document analysis electronic device, the document analysis electronic device can execute any of the document analysis methods described above.
[0046] Compared with the prior art, this application has the following advantages:
[0047] Obtain the document to be processed, summarize the document to be processed, generate and store the document summary; after obtaining the target question of the document to be processed, perform intent recognition on the target question and determine the request type corresponding to the target question; the request type includes document summary or document question and answer; when the request type is document summary, obtain the target answer corresponding to the target question based on the document summary; when the request type is document question and answer, obtain the target answer corresponding to the target question based on the document to be processed.
[0048] In this way, after obtaining the document to be processed, the document to be processed can be summarized first, and the document summary can be generated and stored. Subsequently, after obtaining the target question of the document to be processed, the request type of the target question can be determined to be document summary or document question and answer based on the intention recognition of the target question, that is, to determine whether the user's demand is to summarize the document to be processed or to ask and answer questions based on the content of the document to be processed. According to different request types, flexible strategies can be adopted: if the request type is document summary, the corresponding target answer is directly determined using the pre-generated document summary; if the request type is document question and answer, further analysis is performed based on the document to be processed to determine the corresponding target answer. Therefore, it is possible to efficiently and accurately generate corresponding target answers according to different request types, thereby achieving the effect of giving accurate answers to specific user needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a flowchart of the steps of a document analysis method of the present application;
[0050] Figure 2 It is a logical schematic diagram of segmenting the document to be processed in the present application;
[0051] Figure 3 It is a logic diagram of language conversion for a target question in the present application;
[0052] Figure 4 It is a logical schematic diagram of a document analysis method of the present application;
[0053] Figure 5It is a structural block diagram of a document analysis device of the present application;
[0054] Figure 6 is a schematic diagram of an electronic device of the present application;
[0055] Figure 7 It is a block diagram of a device for document analysis of the present application. DETAILED DESCRIPTION
[0056] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0058] At present, there are some automatic document summarization tools that can help users quickly understand the general content of electronic documents to a certain extent. However, these automatic document summarization tools often lack interactivity and cannot meet the personalized needs of users for understanding electronic documents. Moreover, they can often only provide relatively broad document summaries and lack the ability to accurately answer specific questions. Therefore, they are unable to give accurate answers to users' specific questions.
[0059] Based on this, the present application proposes a document analysis method, which determines the request type of the target question by identifying the intention of the target question for the document to be processed. Then, different strategies can be adopted to analyze the document to be processed according to different request types to solve the above problems.
[0060] Specifically, the present application can be applied to a server or a terminal, and the server can be a local server, other servers in the network, etc. The above-mentioned terminal can be a terminal device in various forms such as a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc., and the embodiments of the present application are not limited to this.
[0061] The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.
[0062] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.
[0063] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system. Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.
[0064] Reference Figure 1 , shows a flowchart of the steps of a document analysis method of the present application, which may specifically include the following steps:
[0065] In step S11, the document to be processed is obtained, the document to be processed is summarized, and a document summary is generated and stored.
[0066] In this step, you first need to obtain the document to be processed, where the document to be processed can be obtained from various sources, such as file systems, databases, networks, etc. The document to be processed may exist in different formats, such as text files, PDFs, Word documents, etc. Therefore, in actual operations, format conversion or preprocessing may be required to ensure the smooth progress of subsequent steps. This application does not specifically limit this.
[0067] It is understandable that in some cases, the user's requirement is to summarize the document to be processed. Therefore, after obtaining the document to be processed, the document to be processed can be summarized to generate and store the document summary. The document summary can be used for subsequent question-answering of the target question of the document summary type, and the answer efficiency can be improved through fast matching based on the document summary.
[0068] In one implementation, obtaining a document to be processed includes:
[0069] Obtain the original document; perform initialization and formatting on the original document to obtain the document to be processed.
[0070] In other words, the original document can be initialized and formatted to obtain a high-quality document to be processed, where the original document is the source of the document to be processed and may come from various channels, such as databases, file systems, web crawlers, etc. By removing irrelevant information and unifying the format, the complexity and resource consumption of subsequent processing steps are reduced. The standardized format and unified structure help to more accurately extract and analyze the information in the document to be processed, thereby increasing the value of the data.
[0071] Initialization processing is the preliminary processing of the original document to remove irrelevant information and provide clean data for subsequent processing steps, including but not limited to: denoising, that is, removing noise information in the original document, such as advertisements, watermarks, irrelevant text, etc.; encoding conversion, that is, converting the original document into a unified encoding format to avoid garbled characters in subsequent processing; content extraction, that is, extracting key content in the original document, such as title, text, pictures, etc. according to needs.
[0072] Formatting is the process of converting the original document after initialization into a unified and standardized format so that subsequent processing steps can be carried out efficiently, including but not limited to: structural standardization, that is, converting the original document into a unified structure, such as paragraphs, sentences, words, etc., to facilitate subsequent processing and analysis; style unification, that is, unifying the style of the original document, such as font, font size, color, etc., to ensure the consistency of the processing results; data format conversion, that is, converting the original document into a data format suitable for subsequent processing, etc.
[0073] In one implementation, summarizing the document to be processed, generating and storing a document summary, includes:
[0074] According to the preset segment length, the document to be processed is divided into multiple segments, and the length of each segment is not greater than the segment length;
[0075] Inputting the multiple segments into the first language model for summarization respectively to obtain multiple segment summaries;
[0076] Input the multiple segment summaries into the first language model for summarization to obtain a summary summary;
[0077] The segment summary, the mapping relationship between the segment summary and the segment, and the summary summary are used as the document summary, and the document summary is stored.
[0078] It is understandable that the length and word count of the document to be processed are very long, and the input of the first language model is also limited, which will greatly affect the summarization effect and speed. In order to solve this problem, after uploading the document to be processed, asynchronous processing can be performed to summarize the full text of the document to be processed, and the document to be processed can be divided into multiple segments, thereby splitting the document to be processed into smaller and easier to process parts, solving the limitation of the first language model on the number of input words, reducing the processing difficulty, and improving the processing efficiency.
[0079] The length of a segment is no greater than a preset segment length, and the preset segment length can be adjusted according to actual conditions to ensure that each segment contains enough information for analysis while not being too lengthy or complex. For example, the segment length can be set to 20,000 words.
[0080] Then, each segment can be input into the first language model for summarization. The first language model can refer to a trained neural network model with strong text understanding and generation capabilities. Inputting each segment into the first language model for summarization can be regarded as a natural language processing task that the first language model needs to perform.
[0081] Specifically, first, the document to be processed is converted into a format that can be processed by the first language model, including pre-processing steps such as word segmentation, stop word removal, and stem extraction. Then, the processed document to be processed will be input into the first language model. Then, the first language model will understand the input document to be processed, including encoding the sentences, paragraphs or entire documents in the document to be processed to generate one or more vectors or embeddings representing the main content of the document to be processed. Next, the first language model will generate a segmented summary based on these vectors or embeddings, which may be one or more sentences. In addition, the generated segmented summary may need to go through some post-processing steps, such as removing duplicate content, adjusting sentence structure, etc., to ensure its accuracy and readability.
[0082] After obtaining the generalized segment summaries of each segment, these segment summaries need to be input into the first language model again to further extract the key information in each segment summary and generate a generalized sentence or paragraph as a summary summary. The first language model's summarization process for segment summaries is similar to that for each segment, so it will not be repeated here. The summary summary is a further refinement and integration of each segment summary, and a comprehensive summary of the entire document to be processed.
[0083] Furthermore, the segment summary, the mapping relationship between the segment summary and the segment, and the summary summary can be used as a document summary and stored for subsequent use. By storing the document summary, subsequent retrieval, query and other operations can be easily performed, improving the flexibility and efficiency of document processing. The first language model has a good global perspective, which can achieve a global understanding of the document to be processed, extract the most important or most representative part of the document to be processed as a document summary, and avoid missing key information.
[0084] For example, if Figure 2 As shown, the document to be processed can be divided into N segments, and corresponding N segment summaries can be obtained, and then the N segment summaries can be summarized to obtain a summary of the full text of the document to be processed.
[0085] In one implementation, multiple segments are respectively input into the first language model for summarization to obtain multiple segment summaries, including:
[0086] Create multiple threads and determine the thread corresponding to each segment;
[0087] In each thread, the corresponding segment is input into the first largest language model for summarization to obtain a segment summary of the corresponding segment.
[0088] That is to say, when using the first language model for document summarization, in order to improve processing speed and efficiency, a parallel processing method is usually adopted, that is, multiple threads process multiple segments at the same time, thereby improving the summarization speed.
[0089] Among them, a thread is the smallest unit that the operating system can perform computation scheduling. By creating multiple threads, multiple tasks can be executed simultaneously. In this embodiment, it is necessary to create an independent thread for each segment to ensure that each segment can be input into the first language model at the same time, thereby improving the overall processing speed.
[0090] Specifically, it can be implemented through a simple mapping relationship, such as using a list or dictionary to store the correspondence between segments and threads, ensuring that each segment has only one corresponding thread, and each thread only processes one segment, avoiding data confusion and errors.
[0091] Then, the corresponding processing task can be performed in each thread, and the corresponding segment can be input into the first large language model for summarization. After all threads have completed the segment processing, the segment summary output by each thread can be collected.
[0092] In one implementation, the document to be processed is divided into a plurality of segments according to a preset segment length, including:
[0093] The document to be processed is divided into a plurality of segments according to a preset segment length and redundant length; wherein adjacent segments have overlapping parts and the length of the overlapping parts is the redundant length.
[0094] That is to say, the redundant length of the segmentation can be pre-set, where the redundant length refers to defining a certain overlap range between each segment. Assuming the segment length is n and the redundant length is k, there are k sentences or words repeated between every two adjacent segments. Then, the first segment includes text of length n starting from the beginning of the document to be processed; the second segment includes moving nk units backward from the end of the first segment, and continuing to take n units of text backward, and so on until the end of the document to be processed.
[0095] For example, suppose n is 100 words and k is 20 words, then:
[0096] The first segment: words 1 to 100 of the document to be processed;
[0097] Second segment: words 81 to 180 of the document to be processed;
[0098] The third segment: words 161 to 260 of the document to be processed; and so on.
[0099] Among them, the redundancy length can also adjust the division strategy according to different application scenarios and requirements. In this way, since there are overlapping parts between adjacent segments, the key information in the document to be processed can be more effectively extracted and retained.
[0100] In one implementation, after obtaining the document to be processed, the process further includes:
[0101] The document to be processed is input into the feature extraction model to obtain the feature vector of the document to be processed, and the feature vector is stored in the corresponding vector library.
[0102] That is to say, after obtaining the document to be processed, in addition to summarizing and generating document summaries, another important step is to input the document to be processed into a feature extraction model. The feature extraction model can extract representative and discriminative feature vectors from the document to be processed. The feature vector can represent the key information and content structure in the document to be processed, including but not limited to word vectors, sentence vectors, etc.
[0103] Subsequently, the feature vectors of the documents to be processed can be stored in the corresponding vector library, which can be a simple list, array or hash table, or a more complex database system, for subsequent retrieval, matching and reasoning tasks, providing strong support for document processing and analysis.
[0104] In step S12, after obtaining the target question of the document to be processed, the intent of the target question is identified to determine the request type corresponding to the target question; the request type includes document summarization or document question and answer.
[0105] In this step, after obtaining the target question of the document to be processed, the intent of the target question can be identified to understand the user's real needs. Specifically, the user may ask various questions or requests in the form of natural language, such as "Please briefly summarize the main content of this document" or "What are the latest developments about XXX mentioned in this document?" Through intent recognition based on natural language processing, machine learning and other technologies, the user's questions can be analyzed and the corresponding request type can be determined.
[0106] Among them, the request types are usually divided into two types: "document summarization" and "document Q&A". Document summarization refers to extracting key information from the document to be processed to help users quickly understand the main content of the document to be processed. Document Q&A, on the other hand, is to find relevant information from the document to be processed and give accurate answers to the user's specific questions.
[0107] In one implementation, the intent of the target question is identified to determine the request type corresponding to the target question, including:
[0108] Based on the preset prompt word project, the target question is input into the question-answering model for intent recognition to determine the request type corresponding to the target question.
[0109] In other words, the intention recognition of the target question can be realized based on the preset prompt word engineering and question-answering model, wherein the preset prompt word engineering sets different prompt words (Prompt) for different types of questions by summarizing and summarizing various types of user questions. Furthermore, the question-answering model can accurately identify and classify the input target question based on the prompt.
[0110] For example, the prompt can be "Please determine whether the user's query content requires a summary of the document or a question about a specific content in the document. Please answer in JSON format. The key is "summary" or "question and answer" or "re-summary", and the value is the queried entity. The query content needs to be translated into the corresponding language."
[0111] In this way, the accuracy and scalability of document analysis can be improved, and strong support can be provided for its application in multiple scenarios.
[0112] In step S13, when the request type is document summarization, a target answer corresponding to the target question is obtained based on the document summary.
[0113] In this application, different strategies can be used to generate a target answer corresponding to the target question according to the request type determined above. For example, for a document summary request, the target answer corresponding to the target question can be determined based on the document summary.
[0114] In one implementation, obtaining a target answer corresponding to a target question based on a document summary includes:
[0115] Perform text feature matching retrieval based on the target question and determine the target answer corresponding to the target question from the document summary.
[0116] That is to say, firstly, text feature extraction can be performed on the target question, including identifying keywords, phrases, synonyms, antonyms, etc. in the target question, and understanding the semantic content and context of the target question.
[0117] Then, text feature matching retrieval can be performed in the document summary, comparing the text features of the target question with the document summary to find the most similar parts. Among them, text feature matching retrieval can be based on traditional text matching algorithms such as vector space model and cosine similarity, or based on deep learning methods such as attention mechanism and semantic matching network, which can evaluate the semantic similarity between the target question and the document summary, so as to find the most relevant target answer.
[0118] In step S14, when the request type is document question and answer, a target answer corresponding to the target question is obtained based on the document to be processed.
[0119] In this step, for the document question-answering request type, the answer to the target question can be found and generated based on the document to be processed as the target answer. In this way, by continuously optimizing and improving the intention recognition of the target question and the analysis strategy adopted, more accurate, useful and personalized document analysis services can be provided to users.
[0120] In one implementation, obtaining a target answer corresponding to a target question based on a document to be processed includes:
[0121] Input the target question into the feature extraction model to obtain the question vector;
[0122] Based on the question vector, search in the vector library to obtain the feature vector related to the target question as the context vector;
[0123] The context vector and the target question are input into the second largest language model to obtain the target answer corresponding to the target question.
[0124] That is to say, when the request type is document question answering, the target question can be input into the feature extraction model to obtain a question vector, which can represent the key information and intention in the target question.
[0125] Then, the question vector can be used to search in the vector library to find the feature vector related to the target question as the context vector. The context vector may come from different parts of the document to be processed and have a certain semantic similarity with the target question. The retrieval process can use different algorithms and strategies, such as cosine similarity calculation, K nearest neighbor search, etc., to find the most relevant context vector.
[0126] The found context vector and the target question are input into the second largest language model, which generates the target answer based on the context vector and the content of the target question. The second largest language model may be a pre-trained deep learning model.
[0127] After being processed by the second language model, the target answer corresponding to the target question can be obtained. The target answer may be a concise sentence or a paragraph containing multiple sentences. In addition, some post-processing steps can be performed on the target answer, such as removing duplicate content and adjusting sentence structure, to ensure the accuracy and readability of the target answer.
[0128] Among them, the feature extraction model can be an Embedding vector model. The Embedding vector model represents semantic similarity by converting text information into points in the vector space. It is easy to capture fragmented information, has strong local semantics, and is better at processing local similarities. Therefore, the Embedding vector model can help locate the context vector related to the target question in the document to be processed and generate an accurate and targeted target answer.
[0129] In one implementation, after obtaining the document to be processed, the process further includes:
[0130] Determine the language of the document to be processed as the target language;
[0131] Input the target question into the feature extraction model to obtain the question vector, including:
[0132] Determine whether the target question is in the target language;
[0133] If yes, then the target question is input into the feature extraction model to obtain the question vector;
[0134] If not, the target question is converted into the target language, and the converted target question is input into the feature extraction model to obtain a question vector.
[0135] That is, the target language used by the document to be processed can be obtained, for example, the target language can be Chinese, English or other languages. According to the target language, a more accurate feature extraction model can be selected. For example, the bge-large-zh-v1.5 model has a better understanding of Chinese, while the bge-large-en-v1.5 model has a better understanding of English.
[0136] Then, when a target question is received, it is necessary to determine whether the target question is in the same target language as the document to be processed. If the target question is in the same target language as the target document, then the question can be directly input into the feature extraction model. If the target question is not in the same target language as the target document, then the target question needs to be converted into the target language before being input into the feature extraction model.
[0137] For example, if Figure 3 As shown, if the document to be processed is an English paper, and the target question input by the user is in Chinese: "How are the relevant issues of this paper implemented?", then the target question needs to be converted into English first, and then the features of the target question in English are extracted to obtain the question vector. Based on the question vector, a search is performed in the vector library to obtain the context vector related to the target question, and then the context vector and the target question are input into the second largest language model to obtain the target answer corresponding to the target question.
[0138] In this way, by using feature extraction models with better effects corresponding to each target language, the accuracy of the context vector retrieved by the question vector can be further improved.
[0139] like Figure 4As shown, it is a logical diagram of the document analysis method of the present application, which mainly includes two parts: storage and retrieval. Among them, in the storage stage, the documents to be processed uploaded by the user can be obtained, and the documents to be processed can be segmented. The target language used by the documents to be processed can also be determined, and then the corresponding feature extraction model is determined to perform feature analysis on the segmented documents to be processed, and the obtained vector data is stored in the vector library. In the retrieval stage, after obtaining the user's target question, the target question can be intentionally identified and the request type of the target question can be determined. At the same time, the target question can also be converted into the target language corresponding to the document to be processed, and then the corresponding question vector is generated. Based on the question vector, it can be retrieved in the vector library to obtain the context vector. Then, the second largest language model is analyzed based on the target question and the context vector to obtain the target answer corresponding to the target question.
[0140] As can be seen from the above, the technical solution provided by the embodiment of the present disclosure can first summarize the document to be processed, generate and store the document summary after obtaining the document to be processed. Subsequently, after obtaining the target question of the document to be processed, it can be determined based on the intention recognition of the target question whether the request type of the target question is a document summary or a document question and answer, that is, it can be determined whether the user's demand is to summarize the document to be processed or to ask and answer questions based on the content of the document to be processed. Depending on the type of request, a flexible strategy can be adopted: if the request type is a document summary, the corresponding target answer is directly determined using the pre-generated document summary; if the request type is a document question and answer, further analysis is performed based on the document to be processed to determine the corresponding target answer. Therefore, it is possible to efficiently and accurately generate corresponding target answers according to different request types, thereby achieving the effect of giving accurate answers to the specific needs of users.
[0141] Reference Figure 5 , shows a schematic diagram of the structure of a document analysis device of the present application, which may specifically include:
[0142] The acquisition module 201 is used to acquire the document to be processed, summarize the document to be processed, and generate and store the document summary;
[0143] The identification module 202 is used to obtain the target question of the document to be processed, perform intent recognition on the target question, and determine the request type corresponding to the target question; the request type includes document summarization or document question answering;
[0144] The answer module 203 is used to obtain the target answer corresponding to the target question based on the document summary when the request type is document summarization; and to obtain the target answer corresponding to the target question based on the document to be processed when the request type is document question and answer.
[0145] As can be seen from the above, the technical solution provided by the embodiment of the present disclosure can first summarize the document to be processed, generate and store the document summary after obtaining the document to be processed. Subsequently, after obtaining the target question of the document to be processed, it can be determined based on the intention recognition of the target question whether the request type of the target question is a document summary or a document question and answer, that is, it can be determined whether the user's demand is to summarize the document to be processed or to ask and answer questions based on the content of the document to be processed. Depending on the type of request, a flexible strategy can be adopted: if the request type is a document summary, the corresponding target answer is directly determined using the pre-generated document summary; if the request type is a document question and answer, further analysis is performed based on the document to be processed to determine the corresponding target answer. Therefore, it is possible to efficiently and accurately generate corresponding target answers according to different request types, thereby achieving the effect of giving accurate answers to the specific needs of users.
[0146] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0147] Figure 6 It is a block diagram of an electronic device for document analysis according to an exemplary embodiment, including a processor and a memory, wherein the memory is used to store computer programs; and the processor is used to execute the programs stored in the memory.
[0148] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0149] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0150] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to perform the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0151] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed on a computer, the computer implements the above-mentioned review document analysis method.
[0152] As can be seen from the above, the technical solution provided by the embodiment of the embodiment of the present invention can first summarize the document to be processed, generate and store a document summary after obtaining the document to be processed. Subsequently, after obtaining the target question of the document to be processed, it can be determined based on the intention recognition of the target question whether the request type of the target question is a document summary or a document question and answer, that is, it can be determined whether the user's demand is to summarize the document to be processed or to ask and answer questions based on the content of the document to be processed. Depending on the type of request, a flexible strategy can be adopted: if the request type is a document summary, the corresponding target answer is directly determined using the pre-generated document summary; if the request type is a document question and answer, further analysis is performed based on the document to be processed to determine the corresponding target answer. Therefore, it is possible to efficiently and accurately generate corresponding target answers according to different request types, thereby achieving the effect of giving accurate answers to the specific needs of users.
[0153] Figure 7 It is a block diagram of a device 800 for document analysis according to an exemplary embodiment.
[0154] For example, apparatus 800 may be a mobile phone, a computer, a digital broadcast electronic device, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0155] Reference Figure 7 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0156] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0157] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0158] The power supply component 807 provides power to the various components of the device 800. The power supply component 807 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0159] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the account. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the account. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data to be processed. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0160] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0161] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0162] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0163] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0164] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the methods described in the first and second aspects.
[0165] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to complete the above method. Optionally, for example, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0166] In an exemplary embodiment, a computer program product including instructions is also provided. When the computer program product is run on a computer, the computer is enabled to execute the document analysis method described in any one of the above embodiments.
[0167] As can be seen from the above, the technical solution provided by the embodiment of the embodiment of the present invention can first summarize the document to be processed, generate and store a document summary after obtaining the document to be processed. Subsequently, after obtaining the target question of the document to be processed, it can be determined based on the intention recognition of the target question whether the request type of the target question is a document summary or a document question and answer, that is, it can be determined whether the user's demand is to summarize the document to be processed or to ask and answer questions based on the content of the document to be processed. Depending on the type of request, a flexible strategy can be adopted: if the request type is a document summary, the corresponding target answer is directly determined using the pre-generated document summary; if the request type is a document question and answer, further analysis is performed based on the document to be processed to determine the corresponding target answer. Therefore, it is possible to efficiently and accurately generate corresponding target answers according to different request types, thereby achieving the effect of giving accurate answers to the specific needs of users.
[0168] Those skilled in the art will readily appreciate other implementations of the embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The embodiments of the present invention are intended to cover any variations, uses, or adaptations of the embodiments of the present invention that follow the general principles of the embodiments of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the embodiments of the present invention. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the embodiments of the present invention are indicated by the following claims.
[0169] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.
Claims
1. A document analysis method, characterized in that: include: Obtaining a document to be processed, summarizing the document to be processed, and generating and storing a document summary; After obtaining the target question for the document to be processed, performing intent recognition on the target question to determine the request type corresponding to the target question; the request type includes document summarization or document question and answer; When the request type is document summary, obtaining a target answer corresponding to the target question based on the document summary; When the request type is document question and answer, a target answer corresponding to the target question is obtained based on the document to be processed.
2. The method according to claim 1, characterized in that The step of summarizing the document to be processed and generating and storing a document summary includes: According to a preset segment length, the document to be processed is divided into a plurality of segments, and the length of each segment is not greater than the segment length; Inputting the multiple segments into the first large language model respectively for summarization to obtain multiple segment summaries; Inputting the plurality of segment summaries into the first language model for summarization to obtain a summary; The segment summary, the mapping relationship between the segment summary and the segment, and the summary summary are used as a document summary, and the document summary is stored.
3. The method according to claim 2, characterized in that The multiple segments are respectively input into the first large language model for summarization to obtain multiple segment summaries, including: Creating a plurality of threads, and determining the thread corresponding to each of the segments; In each thread, the corresponding segment is input into the first largest language model for summarization to obtain a segment summary of the corresponding segment.
4. The method according to claim 2, characterized in that: The step of dividing the to-be-processed document into a plurality of segments according to a preset segment length includes: The document to be processed is divided into a plurality of segments according to a preset segment length and redundant length; wherein adjacent segments have overlapping parts and the length of the overlapping parts is the redundant length.
5. The method according to claim 1, characterized in that After obtaining the document to be processed, the method further includes: The document to be processed is input into a feature extraction model to obtain a feature vector of the document to be processed, and the feature vector is stored in a corresponding vector library.
6. The method according to claim 1, characterized in that The step of obtaining a target answer corresponding to the target question based on the document summary includes: A text feature matching retrieval is performed according to the target question, and a target answer corresponding to the target question is determined from the document summary.
7. The method according to claim 1, characterized in that The step of obtaining a target answer corresponding to the target question based on the document to be processed includes: Inputting the target question into a feature extraction model to obtain a question vector; Searching in a vector library based on the question vector to obtain a feature vector related to the target question as a context vector; The context vector and the target question are input into a second language model to obtain a target answer corresponding to the target question.
8. The method according to claim 7, characterized in that After obtaining the document to be processed, the method further includes: Determining the language used by the document to be processed as the target language; The step of inputting the target question into the feature extraction model to obtain a question vector includes: determining whether the target question is in the target language; If yes, the target question is input into the feature extraction model to obtain a question vector; If not, the target question is converted into the target language, and the converted target question is input into the feature extraction model to obtain a question vector.
9. The method according to claim 1, characterized in that: The performing intention recognition on the target question and determining the request type corresponding to the target question includes: Based on the preset prompt word project, the target question is input into the question-answering model for intent recognition to determine the request type corresponding to the target question.
10. A document analysis device, characterized in that: include: An acquisition module is used to acquire a document to be processed, summarize the document to be processed, and generate and store a document summary; An identification module, configured to obtain a target question about the document to be processed, perform intent identification on the target question, and determine a request type corresponding to the target question; the request type includes document summarization or document question and answer; An answering module, configured to obtain a target answer corresponding to the target question based on the document summary when the request type is a document summary; When the request type is document question and answer, a target answer corresponding to the target question is obtained based on the document to be processed.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the document analysis method according to any one of claims 1 to 9 are implemented.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the document analysis method according to any one of claims 1 to 9 are implemented.