Document analysis method, device and equipment

Abstract

The invention discloses a document analysis method, device and equipment, which can efficiently and accurately realize document analysis. The method comprises the following steps: receiving a to-be-analyzed document uploaded by a user and a task input by the user; executing the task on the document according to a pre-trained large language model; and outputting an execution result of the task to the user.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, apparatus and device for document parsing. Background Art

[0002] With the rapid development of the Internet and digital technology, the amount of data generated and processed by people is increasing, including a large amount of document data. However, traditional document parsing methods usually require manual complex rule matching and pattern recognition, which is inefficient and prone to errors. Therefore, how to improve the efficiency and accuracy of document parsing has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus and device for document parsing, which can efficiently and accurately implement document parsing.

[0004] In a first aspect, a method for document parsing is provided, comprising: receiving a document to be parsed uploaded by a user and a task input by the user; executing the task on the document according to a pre-trained large language model; and outputting the execution result of the task to the user.

[0005] In the embodiment of the present application, based on the document, a pre-trained large language model is used to execute the task input by the user for the document uploaded by the user, and the execution result of the task is output to the user. Since the large language model is used and combined with the corresponding document processing model to perform the parsing task on the document uploaded by the user, the efficiency of document parsing is greatly improved, the time and cost of manual intervention are reduced, and the use of deep learning technology to model large-scale text can improve the accuracy and precision of document parsing.

[0006] The large language model may be trained for a specific field of technology, so that the large language model can understand the content of documents uploaded by users related to the field and complete the tasks input by the user from the documents. For example, the large language model may be trained based on battery-related technology.

[0007] In a possible implementation, the task includes a question input by a user, and executing the task on the document according to a pre-trained large language model includes: obtaining an answer to the question in the document according to the large language model; and outputting the execution result of the task to the user includes: outputting the answer to the user.

[0008] In this embodiment, the user can ask questions and hope to find answers from the documents he uploaded. After processing the documents accordingly, the large language model can obtain content that can answer the questions from the documents based on the questions input by the user, and output it to the user, thereby efficiently and accurately meeting the needs of the user and improving the user experience.

[0009] In a possible implementation, obtaining the answer to the question in the document according to the large language model includes: segmenting the document to obtain a plurality of segmented texts; searching for segmented texts whose similarity to the question satisfies a condition among the plurality of segmented texts; and obtaining the answer from the segmented texts whose similarity to the question satisfies a condition according to the large language model.

[0010] In this embodiment, in order to enable the large language model to answer questions input by users based on the content in the documents uploaded by users, it is necessary to segment the documents into multiple parts and find the segmented texts that are most relevant to the user's questions, so that the large language model can output corresponding answers to the users based on the segmented contents of these parts to respond to the questions input by users.

[0011] In a possible implementation, segmenting the document to obtain a plurality of segmented texts includes: obtaining the type of the document, wherein different types correspond to different segmentation algorithms; and segmenting the document according to the segmentation algorithm corresponding to the type of the document to obtain the plurality of segmented texts.

[0012] Document types may include various formats such as WORD, PDF or TXT, so different segmentation algorithms need to be set for different formats, and the documents of that type are segmented according to the corresponding segmentation algorithms to obtain accurate and complete segmented content.

[0013] In a possible implementation, the multiple segmented texts are multiple paragraphs of the document. By segmenting the document in paragraph units, each sentence has its context, which can improve the semantic coherence and the accuracy of document parsing, and will not have a significant impact on the difficulty of the large language model in finding the answer.

[0014] In a possible implementation, segmenting the document includes: cleaning the document to remove noise and unparseable parts in the document; and segmenting the cleaned document.

[0015] The unparseable parts may include garbled codes, pictures, etc., for example. Since the large language model cannot recognize the content of these parts, removing the noise and unparseable parts in the document will help the large language model understand the document content and improve the efficiency and accuracy of document parsing.

[0016] In a possible implementation, the retrieving, from the multiple segmented texts, segmented texts whose similarity with the question satisfies a condition, includes: calculating, based on a text embedding algorithm, text embedding values ​​corresponding to each of the multiple segmented texts; calculating, based on the text embedding algorithm, a question embedding value corresponding to the question; retrieving, from the text embedding values ​​corresponding to each of the multiple segmented texts, a target text embedding value whose similarity with the question embedding value satisfies a condition; and obtaining, based on the large language model, the answer from the segmented texts whose similarity with the question satisfies a condition, includes: obtaining, based on the large language model, the answer from the segmented texts corresponding to the target text embedding value.

[0017] In this embodiment, a text embedding algorithm can be used to calculate text embedding values ​​corresponding to multiple segmented texts in a document, and the text embedding values ​​are linked to their corresponding original texts through a retriever. Based on the text embedding algorithm, a question embedding value corresponding to the question input by the user is calculated to find a text embedding value that meets the similarity conditions with the question embedding value, so that the large language model obtains the answer to the question from the segmented texts corresponding to the text embedding values.

[0018] The segmented texts whose similarity to the user question satisfies the condition include, for example, at least one segmented text having the highest similarity to the question among the multiple segmented texts, or a segmented text having a similarity to the question greater than a similarity threshold among the multiple segmented texts.

[0019] In a possible implementation, obtaining the answer from the segmented text whose similarity to the question satisfies a condition according to the large language model includes: embedding the question into a prompt template, and based on the prompt template and according to the large language model, obtaining the answer from the segmented text whose similarity to the question satisfies a requirement.

[0020] In this embodiment, by designing a prompt template, the question is embedded in the prompt template, so that the large language model can obtain the answer to the question from the segmented texts that meet the similarity requirements based on the requirements of the prompt template. Due to the restrictions of the prompt template, it is beneficial for the large language model to understand and find the answer.

[0021] In a possible implementation, the task also includes at least one of the following parsing tasks: automatic summarization, relationship extraction, translation, text classification, named entity recognition, text polishing, and document parsing tasks combined with knowledge graphs. In the embodiment of the present application, the large language model can also perform parsing tasks required by users on documents uploaded by users, and has good scalability and versatility, so that the document parsing method has a wider range of application scenarios.

[0022] In a possible implementation, the method also includes: obtaining a web page user interface, the web page user interface including an operation area and a display area, the operation area including a first area and a second area; in response to the document uploaded by the user in the first area and the task input by the user in the second area, presenting the execution result of the task to the user in the display area.

[0023] In this embodiment, building a web user interface enables users to obtain document information through dialogue, thereby improving user experience.

[0024] In a second aspect, a device for document parsing is provided, comprising: an input port for receiving a document to be parsed uploaded by a user and a task input by the user; a processing unit for executing the task on the document according to a pre-trained large language model; and an output port for outputting the execution result of the task to the user.

[0025] In a possible implementation, the task includes a question input by a user, and the processing unit is specifically used to obtain an answer to the question in the document according to the large language model; and the output port is specifically used to output the answer to the user.

[0026] In a possible implementation, the processing unit is specifically used to segment the document to obtain multiple segmented texts; retrieve the segmented texts whose similarity with the question meets the conditions among the multiple segmented texts; and obtain the answer from the segmented texts whose similarity with the question meets the conditions based on the large language model.

[0027] In a possible implementation, the processing unit is specifically used to obtain the type of the document, wherein different types correspond to different segmentation algorithms; and segment the document according to the segmentation algorithm corresponding to the type of the document to obtain the multiple segmented texts.

[0028] In a possible implementation manner, the multiple segmented texts are respectively multiple paragraphs of the document.

[0029] In a possible implementation, the processing unit is specifically configured to clean the document to remove noise and unparseable parts in the document; and segment the cleaned document.

[0030] In a possible implementation, the processing unit is specifically used to, based on a text embedding algorithm, calculate a text embedding value corresponding to each of the multiple segmented texts; based on the text embedding algorithm, calculate a question embedding value corresponding to the question; retrieve a target text embedding value whose similarity with the question embedding value meets a condition from the text embedding values ​​corresponding to each of the multiple segmented texts; and obtain the answer from the segmented text corresponding to the target text embedding value according to the large language model.

[0031] In a possible implementation, the segmented text whose similarity with the question satisfies the condition includes: at least one segmented text among the multiple segmented texts whose similarity with the question is the highest; or a segmented text among the multiple segmented texts whose similarity with the question is greater than a similarity threshold.

[0032] In a possible implementation, the processing unit is specifically configured to embed the question into a prompt template, and based on the prompt template and according to the large language model, obtain the answer from the segmented text whose similarity with the question meets the requirements.

[0033] In a possible implementation, the task also includes at least one of the following parsing tasks: automatic summarization, relationship extraction, translation, text classification, named entity recognition, text polishing, and document parsing tasks combined with knowledge graphs.

[0034] In a possible implementation, the processing unit is also used to obtain a web page user interface, the web page user interface includes an operation area and a display area, the operation area includes a first area and a second area; in response to the document uploaded by the user in the first area and received by the input port, and the task input by the user in the second area and received by the input port, presenting the execution result of the task output by the output port to the user in the display area.

[0035] In a third aspect, a document parsing device is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method described in the first aspect or any possible implementation of the first aspect.

[0036] In a fourth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying creative work.

[0038] Figure 1 It is a schematic flowchart of the document parsing method of an embodiment of the present application.

[0039] Figure 2 It is a schematic flowchart of the process of document parsing based on a large language model in an embodiment of the present application.

[0040] Figure 3 It is a schematic diagram of a web page user interface of an embodiment of the present application.

[0041] Figure 4 yes Figure 1 A schematic flow chart of a possible specific implementation of the method shown.

[0042] Figure 5 It is a schematic block diagram of a document parsing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following detailed description and drawings of the embodiments of the present application are used to illustrate the principles of the present application, but cannot be used to limit the scope of the present application, that is, the present application is not limited to the described embodiments.

[0044] Traditional document parsing methods usually require professional technicians to perform complex rule matching and pattern recognition for different document contents, and the process of establishing rules takes a lot of time, especially for large and complex documents, which leads to low efficiency and increased costs. At the same time, due to the subjectivity and inaccuracy of manual processing, errors or omissions may occur, resulting in inaccurate parsing results. Moreover, manual document parsing is difficult to cope with large-scale and diverse data because it requires continuous training and adjustment of personnel, and it is difficult to ensure the consistency of parsing results. Therefore, traditional document parsing methods can no longer meet people's needs.

[0045] With the development of deep learning technology, language models based on neural networks can be applied to document parsing tasks. For example, the named entity recognition task can be implemented based on the iLSTM-CRF model and the BERT model; for another example, the relationship extraction task can be implemented based on the Transformer model combined with the attention mechanism; for another example, some document parsing tasks combined with knowledge graphs can be implemented based on the ELMo+BiLSTM model, RotatE+BERT model, etc.

[0046] In addition, the emergence of large language models such as the Chat Generative Pre-trained Transformer (ChatGPT) has accelerated the development of document parsing technology. By utilizing ChatGPT's powerful language understanding and generation capabilities, various document parsing tasks can be effectively completed.

[0047] For traditional document parsing methods that are not based on large language models, the current models have limited processing capabilities for long texts. Due to the limitations of the model architecture, their processing capabilities for long texts are very weak. In practical applications, they cannot adapt well to tasks in certain specific fields, and it is necessary to select appropriate models or perform model fine-tuning according to specific tasks.

[0048] As for the method of using ChatGPT for document parsing, since the entire process requires continuous calls to the Application Programming Interface (API), the security of confidential documents is always a difficult problem to solve, and continuous calls to the API will bring high fees, which may be a heavy burden for users. It can be seen that the use of ChatGPT for document parsing has problems in scalability and ease of use.

[0049] To this end, the embodiment of the present application provides a solution for document parsing, which uses a pre-trained Large Language Model (LLM) as a semantic understanding model and combines it with a document retrieval model to perform document parsing, which can efficiently and accurately implement document parsing. Furthermore, a web user interface can be built for users to obtain document information through dialogue.

[0050] Figure 1 1 shows a schematic flow chart of a document parsing method 100 according to an embodiment of the present application. The method 100 may be executed by a server, etc. Figure 1 As shown, method 100 includes some or all of the following steps.

[0051] In step 110, a document to be parsed uploaded by a user and a task input by the user are received.

[0052] In step 120, the task is performed on the document according to the pre-trained large language model.

[0053] In step 130, the execution result of the task is output to the user.

[0054] In the embodiment of the present application, a pre-trained large language model is used to execute the task input by the user for the document uploaded by the user, and the execution result of the task is output to the user. Since the large language model is used in combination with the corresponding document processing model to perform the parsing task on the document uploaded by the user, the efficiency of document parsing is greatly improved, the time and cost of manual intervention are reduced, and the use of deep learning to model large-scale text can improve the accuracy and precision of document parsing.

[0055] The large language model is a deep learning model trained with a large amount of text data. It has the functions of understanding the meaning of language text and generating natural language text, and can handle a variety of natural language tasks. The conventional large language model cannot have a good understanding of the content in a specific technical field, and the large language model in the embodiment of the present application can be obtained by, for example, technical training for a specific field, so that the large language model can understand the content of the document related to the field uploaded by the user, and complete the task input by the user from the document.

[0056] That is to say, the large language model of the embodiment of the present application can be obtained by fine-tuning the pre-trained large language model in combination with the technology in the battery-related field, that is, the fine-tuned pre-trained large language model.

[0057] The large language model in the embodiment of the present application is located on an independent server. For confidentiality and security, the working process of the large language model is executed on the server, and does not require Internet operation or frequent API interface calls.

[0058] The task input by the user may include, for example, a question input by the user, that is, the user may ask a question and hope to find the answer from the document uploaded by the user. In this case, in step 120, the answer to the question may be obtained from the document according to the large language model; in step 130, the answer may be output to the user.

[0059] After processing the document accordingly, the large language model can obtain the answer or solution to the question from the document based on the question input by the user, and output it to the user, thereby efficiently and accurately meeting the user's needs and improving the user experience.

[0060] For example, the large language model can be trained based on battery-related technologies. The trained large language model has basic knowledge in the battery field and can recognize battery terminology. If a user needs to find an answer to a question about batteries from a battery-related document uploaded by the user, after the server obtains the document and interprets the question, the large language model trained based on battery-related technologies can well understand the battery-related content in the document and obtain an answer that satisfies the user.

[0061] Figure 2 A possible specific implementation of step 120 is shown, in which the answer to the question is obtained from the document uploaded by the user based on the pre-trained large language model. Figure 2 As shown, step 120 includes steps 121 to 123.

[0062] In step 121, the document uploaded by the user is segmented to obtain a plurality of segmented texts.

[0063] In step 122, a segmented text whose similarity with the question satisfies a condition is searched among the plurality of segmented texts.

[0064] In step 123, based on the pre-trained large language model, the answer to the question is obtained from the segmented texts whose similarity with the question satisfies the condition.

[0065] In order to enable the large language model to answer questions input by users based on the content in the documents uploaded by users, it is necessary to segment the documents into multiple parts and find the segmented text that is most relevant to the user's questions. In this way, the large language model can output the corresponding answers to the users based on the segmented content of this part to respond to the questions input by users.

[0066] The document can be any type of file. The user can upload multiple or a large number of documents at the same time, and can upload documents of the same or different formats, such as documents in PDF format, WORD format, and TXT format. Different segmentation algorithms need to be designed for different document formats, and the documents of that type are segmented according to the corresponding segmentation algorithms to obtain accurate and complete segmented content. For example, in the process of segmenting the document in step 121, it is necessary to obtain the type of the document uploaded by the user, and segment the document according to the segmentation algorithm corresponding to the type of the document to obtain multiple segmented texts, wherein different types correspond to different segmentation algorithms.

[0067] In some embodiments, during the document segmentation process in step 121, the document content needs to be segmented based on a certain granularity. For example, the document content can be segmented in units of paragraphs, and the multiple segmented texts obtained are multiple paragraphs of the document.

[0068] Document segmentation is performed in paragraph units. Each sentence has its context, which can improve semantic coherence and the accuracy of document parsing without significantly affecting the difficulty of large language models in finding answers.

[0069] In some embodiments, during the document segmentation process in step 121, the document may be cleaned first to remove noise and unparseable parts in the document, and then the cleaned document may be segmented.

[0070] The unparseable parts may include garbled codes, pictures, etc., for example. Since the large language model cannot recognize the content of these parts, removing the noise and unparseable parts in the document will help the large language model understand the document content and improve the efficiency and accuracy of document parsing.

[0071] Specifically, for documents uploaded by users, we can first identify the text, images, tables and other elements in the document, remove the images and other elements that cannot be parsed, and classify and extract the text parts that can be parsed to facilitate natural language processing, such as word segmentation, named entity recognition, sentiment analysis, translation, polishing, etc., so that the large language model can better understand the content of the document.

[0072] In some embodiments, in step 122, searching for segmented texts whose similarity with the question satisfies a condition in multiple segmented texts may include: calculating text embedding values ​​corresponding to each of the multiple segmented texts based on a text embedding algorithm; calculating a question embedding value corresponding to the question based on the text embedding algorithm; and searching for a target text embedding value whose similarity with the question embedding value satisfies a condition in the text embedding values ​​corresponding to each of the multiple segmented texts. At this time, in step 123, the answer to the question is obtained in the segmented text corresponding to the target text embedding value according to the large language model.

[0073] The text embedding algorithm is used to map text data into a vector space of fixed length and retain certain semantic information in the original text. In this vector space, similar texts will have similar vector representations. In step 122, for example, a Text2Vec model, a Word2Vec model, or a Doc2Ve model can be used as a text embedding algorithm to process multiple segmented texts of the segmented document to obtain corresponding multiple text embedding values, and by generating a retriever, each text embedding value is linked to the original text in its corresponding segmented text. In addition, based on the text embedding algorithm, the question embedding value corresponding to the question input by the user can be calculated.

[0074] It can be understood that the process of document embedding is equivalent to feature extraction of text content, and the extracted features are represented by corresponding vectors, namely embedded values. Therefore, the embedded values ​​identify the corresponding text content.

[0075] After obtaining the text embedding value and the question embedding value, it is necessary to perform embedding retrieval to match the text embedding value with the question embedding value to obtain a matching result, which can be, for example, the similarity between the text embedding value and the question embedding value. For example, the Facebook Artificial Intelligence Similarity Search (Faiss) algorithm can be used to retrieve a target text embedding value that satisfies the similarity conditions with the question embedding value among multiple text embedding values, so that the large language model obtains the answer to the question from the segmented text corresponding to the target text embedding value. In other words, according to the user's question, the segmented text that is more related to the question is found among the multiple segmented texts formed after the document segmentation, which serves as the basis for the large language model to generate the answer.

[0076] The similarity can be represented by a similarity score, for example. The similarity score is calculated for the question embedding value and the text embedding value corresponding to each segmented text. The higher the similarity score, the higher the similarity between the question embedding value and the text embedding value.

[0077] For example, a similarity threshold may be set, wherein the segmented texts whose similarity to the question satisfies the condition may include the segmented texts whose similarity to the question is greater than the similarity threshold among the multiple segmented texts. The large language model needs to obtain the answer to the user's question from the segmented texts whose similarity to the question is greater than the similarity threshold among the multiple segmented texts.

[0078] For another example, a segmented text with the highest similarity to the question may be selected from among multiple segmented texts, and the large language model may select the segmented text with the highest similarity to the question.

[0079] For another example, a topk function can also be set, which is used to quickly find the top k largest or top k smallest data in a large data set, where the segmented texts that meet the similarity conditions with the question can include the k segmented texts with the highest similarity with the question among multiple segmented texts. By reasonably setting the parameters of the topk function to meet data processing requirements, the k segments of text closest to the question can be found quickly and accurately. The parameters of the topk function include, for example, the data set to be processed, the k value, the function for extracting similarity, the function for sorting similarity, and other parameters.

[0080] In some embodiments, in step 123, obtaining the answer from the segmented text whose similarity with the question meets the conditions according to the large language model may include: embedding the question into a prompt template, and based on the prompt template, obtaining the answer from the segmented text whose similarity with the question meets the requirements according to the large language model.

[0081] By designing a prompt template and embedding the question into the prompt template, the large language model can obtain the answer to the question from the segmented text that meets the similarity requirements based on the requirements of the prompt template. Due to the limitations of the prompt template, it is easier for the large language model to understand and find the answer.

[0082] The prompt template is a role and specification set for the large language model. The large language model needs to find the answer to the problem under the requirements of the prompt template. The prompt template may include, for example, maintaining the security of the text content, generating an answer to the question based on the text content, performing some specific analysis on the text content and outputting some form of results, outputting preset text content or outputting "unknown" when the answer cannot be obtained, and so on.

[0083] The essence of the prompt template is to use the template to convert the task input into the prompt format, that is, cloze, so as to encode the natural language instructions into the embedding of the model without the help of the semantic parser. For example, fill in the blanks in the prompt template and embed the question into the empty space of the prompt template. Based on the prompt template, the large language model processes the segmented text that meets the above similarity conditions and generates the corresponding answer.

[0084] A variety of prompt templates can be pre-designed, and according to the type of task input by the user, a matching prompt template can be used to complete the corresponding task.

[0085] In step 110, the task input by the user may include at least one of the following parsing tasks in addition to questions for obtaining information: automatic summarization, relationship extraction, translation, text classification, named entity recognition, text polishing, document parsing tasks combined with knowledge graphs, etc. According to the needs of the user, document structured information such as summaries, directories, texts, and charts may also be automatically generated to better organize and manage documents.

[0086] By expanding the large language model, other tasks that require complex reasoning and logical judgment of user input can be performed on documents uploaded by users. It has good scalability and versatility, which enables it to have a wider range of application scenarios, attempting to provide users with more text services and more convenience.

[0087] In the embodiment of the present application, the significance of using a large language model to parse documents is that it can automatically process a large number of documents, thereby improving work efficiency and accuracy. In addition, users can upload documents as a knowledge base, and by building a private knowledge base, reduce manual intervention and duplication of work, and provide users with a simple way to interact with the knowledge base, and can switch models and parameters according to usage scenarios and user habits. In this way, the difficulty of knowledge acquisition is reduced, work efficiency is improved, and reliable data support can be provided to users, as well as a large amount of information can be stored and organized, so that users can quickly find the answers or solutions they need, help managers make more informed decisions, and promote collaboration and sharing, and improve the performance and creativity of the team.

[0088] In some embodiments, method 100 also includes: obtaining a web user interface (Web UI) 200, the web user interface 200 includes an operation area 210 and a display area 220, the operation area 210 includes a first area 211 and a second area 212; in response to a document uploaded by a user in the first area 211 and a task input by a user in the second area 212, presenting the execution result of the task to the user in the display area 220.

[0089] The web user interface 200 is a user interface that can be used in a web browser. It is a user interface written using web technology. These interfaces can build a beautiful, interactive, and smooth user experience, allowing website users to more easily focus on the website's data content. By building a web user interface, users can obtain document information through dialogue, which improves the user experience.

[0090] For example, Figure 3 As shown, a web user interface 200 may be presented to a user, and the user clicks to upload a document to be parsed in a first area 211 of an operation area 210, and inputs a task in a second area 212 of the operation area 210. The server transfers the document, interprets the question input by the user, and performs the above processing on the document, and then executes the task input by the user for the document through a large language model to obtain the execution result of the task. The execution result will be presented to the user in a display area 220.

[0091] For ease of deployment, the large language model can be optionally quantized, for example, using the Accurate Post-Training Quantization for Generative Pre-trained Transformers model to quantize the large language model, thereby reducing resource usage on the server.

[0092] Figure 4 Shows Figure 1 A possible specific implementation of the method shown is as follows: Figure 4 The document parsing process shown includes the following steps, the order of which can be adjusted based on actual applications.

[0093] In step 301, a document uploaded by a user is received.

[0094] A file receiving interface can be designed in the front-end UI part, and users can upload documents to the server through the interface, and the server transfers the document.

[0095] In step 302, a question input by a user is received.

[0096] In step 303, a document parser is generated.

[0097] The document parser is used to process the document in subsequent steps, such as segmentation, text embedding, text retrieval, etc.

[0098] In step 304, the document is segmented to obtain a plurality of segmented texts.

[0099] The server can use different document segmentation methods according to the document type to segment the document. In step 304, first, the document is cleaned to remove noise and unparseable parts, including garbled parts, pictures, etc.; then, the remaining text parts are divided into paragraphs to obtain the final segmentation result as the object to be matched.

[0100] In step 305, the segmented text is encoded based on a text embedding algorithm to obtain a text embedding value.

[0101] For example, the Text2Vec model can be used to calculate the text embedding values ​​corresponding to each of the multiple segmented texts in the document.

[0102] In step 306, the question input by the user is encoded based on the text embedding algorithm to obtain a question embedding value.

[0103] For example, the Text2Vec model can be used to calculate the question embedding value corresponding to the question entered by the user.

[0104] In step 307, a text retriever is generated.

[0105] The text retriever is used to link the text embedding value with the original text, and in the subsequent step, perform embedding retrieval based on the text embedding value and the question embedding value, such as using the Faiss algorithm for embedding retrieval.

[0106] In step 308, a target text embedding value whose similarity with the question embedding value satisfies a condition is retrieved from the text embedding values ​​corresponding to each of the plurality of segmented texts.

[0107] The target text embedding value, for example, includes a text embedding value whose similarity with the question embedding value is greater than a similarity threshold, or includes at least one text embedding value whose similarity with the question embedding value is the highest.

[0108] In step 309, the segmented text corresponding to the target text embedding value is processed based on the large language model to obtain the answer to the question.

[0109] Here, a prompt template can be designed and the question can be embedded into the prompt template. The large language model can answer the user's question after understanding it.

[0110] In step 310, the answer to the question is output to the user.

[0111] The interaction with users throughout the entire process occurs on the front-end UI.

[0112] In the above embodiment, if a target text embedding value whose similarity with the question embedding value meets the conditions cannot be found, that is, the similarity between each paragraph of the document and the question is low, optionally, the question input by the user can be directly selected and input into other language models for answering, such as the Vicuna-13B model or the ChatGLM model.

[0113] It is understandable that when training the large language model of the embodiment of the present application, a large number of samples are required, such as texts related to battery technology, so that the large language model can better understand the battery-related content in the documents uploaded by users. Optionally, the large language model can be trained using battery-related sample data based on, for example, the Vicuna-13B model or the ChatGLM model.

[0114] The document parsing method 100 of the embodiment of the present application can analyze the meaning of paragraphs in a document and present it to the user in a dialogue manner. This function will be more powerful than traditional reading comprehension questions and answers. It can deeply interpret the relevant information in the document through translation, interpretation, query, etc., allowing users to quickly sort out the knowledge viewpoints, data context, etc. in the document.

[0115] The present application also provides a document parsing device 400, such as Figure 5As shown, the device 400 includes an input port 410, a processing unit 420 and an output port 430. The input port 410 is used to receive the document to be parsed uploaded by the user and the task input by the user; the processing unit 420 is used to execute the task on the document according to the pre-trained large language model; and the output port 430 is used to output the execution result of the task to the user.

[0116] In some embodiments, the task includes a question input by a user, and the processing unit 420 is specifically used to obtain an answer to the question in the document according to the large language model; the output port 430 is specifically used to output the answer to the user.

[0117] In some embodiments, the processing unit 420 is specifically used to segment the document to obtain multiple segmented texts; retrieve the segmented texts whose similarity with the question meets the conditions among the multiple segmented texts; and obtain the answer from the segmented texts whose similarity with the question meets the conditions according to the large language model.

[0118] In some embodiments, the processing unit 420 is specifically used to obtain the type of the document, wherein different types correspond to different segmentation algorithms; and segment the document according to the segmentation algorithm corresponding to the type of the document to obtain the multiple segmented texts.

[0119] In some embodiments, the multiple segmented texts are respectively multiple paragraphs of the document.

[0120] In some embodiments, the processing unit 420 is specifically used to clean the document to remove noise and unparseable parts in the document; and segment the cleaned document.

[0121] In some embodiments, the processing unit 420 is specifically used to calculate, based on a text embedding algorithm, a text embedding value corresponding to each of the multiple segmented texts; calculate a question embedding value corresponding to the question based on the text embedding algorithm; retrieve a target text embedding value whose similarity with the question embedding value meets a condition from the text embedding values ​​corresponding to each of the multiple segmented texts; and obtain the answer from the segmented text corresponding to the target text embedding value according to the large language model.

[0122] In some embodiments, the segmented text whose similarity with the question meets the conditions includes: at least one segmented text among the multiple segmented texts whose similarity with the question is the highest; or a segmented text among the multiple segmented texts whose similarity with the question is greater than a similarity threshold.

[0123] In some embodiments, the processing unit 420 is specifically configured to embed the question into a prompt template, and based on the prompt template and according to the large language model, obtain the answer from the segmented text whose similarity with the question meets the requirements.

[0124] In some embodiments, the task also includes at least one of the following parsing tasks: automatic summarization, relationship extraction, translation, text classification, named entity recognition, text polishing, and document parsing tasks combined with knowledge graphs.

[0125] In some embodiments, the processing unit 420 is also used to obtain a web page user interface, which includes an operation area and a display area, and the operation area includes a first area and a second area; in response to the document uploaded by the user in the first area and received by the input port, and the task input by the user in the second area and received by the input port, the execution result of the task output by the output port 430 is presented to the user in the display area.

[0126] It can be understood that the document parsing device 400 is used to execute the above-mentioned method 100. The specific details of the device 400 can refer to the above-mentioned description of the method 100, and for the sake of brevity, they are not repeated here.

[0127] The present application also provides a document parsing device, which may be, for example, a server, comprising a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the file parsing method 100 described in any of the above embodiments.

[0128] The present application also provides a computer-readable storage medium for storing a computer program, which enables a computer to execute the file parsing method 100 described in any of the above embodiments.

[0129] It should be noted that, under the premise of no conflict, the various embodiments described in this application and / or the technical features in each embodiment can be arbitrarily combined with each other, and the technical solution obtained after the combination should also fall within the protection scope of this application.

[0130] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A document parsing method, characterized in that: include: Receive documents to be parsed uploaded by users and tasks entered by users; Performing the task on the document according to a pre-trained large language model; The execution result of the task is output to the user.

2. The method according to claim 1, characterized in that: The task includes a question input by a user, and performing the task on the document according to a pre-trained large language model includes: Obtaining an answer to the question in the document according to the large language model; The outputting the execution result of the task to the user includes: The answer is output to the user.

3. The method according to claim 2, characterized in that The step of obtaining an answer to the question in the document according to the large language model includes: Segmenting the document to obtain multiple segmented texts; Retrieving a segmented text whose similarity with the question satisfies a condition from among the multiple segmented texts; The answer is obtained from the segmented texts whose similarity to the question satisfies a condition according to the large language model.

4. The method according to claim 3, characterized in that The document is segmented to obtain a plurality of segmented texts, including: Obtaining the type of the document, wherein different types correspond to different segmentation algorithms; The document is segmented according to a segmentation algorithm corresponding to the type of the document to obtain the multiple segmented texts.

5. The method according to claim 4, characterized in that The multiple segmented texts are respectively multiple paragraphs of the document.

6. The method according to claim 4 or 5, characterized in that: The step of segmenting the document comprises: Cleaning the document to remove noise and unparseable parts in the document; The cleaned document is segmented.

7. The method according to any one of claims 3 to 6, characterized in that The step of searching the plurality of segmented texts for segmented texts whose similarity to the question satisfies a condition includes: Based on a text embedding algorithm, calculating text embedding values ​​corresponding to each of the plurality of segmented texts; Based on the text embedding algorithm, calculating the question embedding value corresponding to the question; Retrieving a target text embedding value whose similarity with the question embedding value satisfies a condition from the text embedding values ​​corresponding to each of the plurality of segmented texts; The step of obtaining the answer from the segmented texts whose similarity to the question satisfies a condition according to the large language model includes: According to the large language model, the answer is obtained from the segmented text corresponding to the target text embedding value.

8. The method according to any one of claims 3 to 7, characterized in that The segmented texts whose similarity with the question meets the conditions include: At least one segmented text among the multiple segmented texts has the highest similarity to the question; or The segmented texts among the multiple segmented texts have a similarity with the question that is greater than a similarity threshold.

9. The method according to any one of claims 3 to 8, characterized in that The step of obtaining the answer from the segmented texts whose similarity to the question satisfies a condition according to the large language model includes: The question is embedded into a prompt template, and based on the prompt template and the large language model, the answer is obtained from the segmented texts whose similarity with the question meets the requirement.

10. The method according to any one of claims 2 to 9, characterized in that The task also includes at least one of the following parsing tasks: automatic summarization, relationship extraction, translation, text classification, named entity recognition, text polishing, and document parsing tasks combined with knowledge graphs.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Acquire a webpage user interface, wherein the webpage user interface includes an operation area and a display area, and the operation area includes a first area and a second area; In response to the document uploaded by the user in the first area and the task input by the user in the second area, the execution result of the task is presented to the user in the display area.

12. The method according to any one of claims 1 to 11, characterized in that The large language model is trained based on battery-related technologies.

13. A document parsing device, characterized in that: include: Input port, used to receive documents to be parsed uploaded by users and tasks entered by users; a processing unit, configured to perform the task on the document according to a pre-trained large language model; The output port is used to output the execution result of the task to the user.

14. The device according to claim 13, characterized in that The task includes a question input by the user, The processing unit is specifically configured to obtain an answer to the question in the document according to the large language model; The output port is specifically used to output the answer to the user.

15. The device according to claim 14, characterized in that The processing unit is specifically used for: Segmenting the document to obtain multiple segmented texts; Retrieving a segmented text whose similarity with the question satisfies a condition from among the multiple segmented texts; The answer is obtained from the segmented texts whose similarity to the question satisfies a condition according to the large language model.

16. The device according to claim 15, characterized in that The processing unit is specifically used for: Obtaining the type of the document, wherein different types correspond to different segmentation algorithms; The document is segmented according to a segmentation algorithm corresponding to the type of the document to obtain the multiple segmented texts.

17. The device according to claim 16, characterized in that The multiple segmented texts are respectively multiple paragraphs of the document.

18. The device according to claim 16 or 17, characterized in that The processing unit is specifically used for: Cleaning the document to remove noise and unparseable parts in the document; The cleaned document is segmented.

19. The device according to any one of claims 15 to 18, characterized in that The processing unit is specifically used for: Based on a text embedding algorithm, calculating text embedding values ​​corresponding to each of the plurality of segmented texts; Based on the text embedding algorithm, calculating the question embedding value corresponding to the question; Retrieving a target text embedding value whose similarity with the question embedding value satisfies a condition from the text embedding values ​​corresponding to each of the plurality of segmented texts; According to the large language model, the answer is obtained from the segmented text corresponding to the target text embedding value.

20. The device according to any one of claims 15 to 19, characterized in that The segmented texts whose similarity with the question meets the conditions include: At least one segmented text among the multiple segmented texts has the highest similarity to the question; or The segmented texts among the multiple segmented texts have a similarity with the question that is greater than a similarity threshold.

21. The device according to any one of claims 15 to 20, characterized in that The processing unit is specifically used for: The question is embedded into a prompt template, and based on the prompt template and the large language model, the answer is obtained from the segmented texts whose similarity with the question meets the requirement.

22. The device according to any one of claims 13 to 21, characterized in that The task also includes at least one of the following parsing tasks: automatic summarization, relationship extraction, translation, text classification, named entity recognition, text polishing, and document parsing tasks combined with knowledge graphs.

23. The device according to any one of claims 13 to 22, characterized in that The processing unit is further configured to: Acquire a webpage user interface, wherein the webpage user interface includes an operation area and a display area, and the operation area includes a first area and a second area; In response to the document uploaded by the user in the first area and received by the input port, and the task input by the user in the second area and received by the input port, the execution result of the task output by the output port is presented to the user in the display area.

24. The device according to any one of claims 13 to 23, characterized in that The large language model is trained based on battery-related technologies.

25. A text parsing device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the text parsing method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the text parsing method according to any one of claims 1 to 12.

Images