Document processing method and related device

Abstract

Disclosed are a document processing method and a related device, relating to the field of artificial intelligence. The method comprises: acquiring a document processing request; and acquiring a first fragment set of a first document according to the document processing request, and processing the first fragment set to obtain a summary result of the first document. The first fragment set is from a part of a fragment graph network constructed from a plurality of fragments of the first document, and the fragment graph network is configured for indicating an association relationship between the plurality of fragments of the first document. Therefore, performing document summarization by selecting from the fragment graph network a small number of representative fragments capable of expressing content of the first document ensures document summary quality, reduces the fragments input into a large language model, shortens the duration for processing long documents, and improves processing efficiency.

Description

Document processing methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202410918381.5, filed with the State Intellectual Property Office of China on July 9, 2024, entitled "A Long Document Processing Method and Related Apparatus", and also claims priority to Chinese Patent Application No. 202411303327.6, filed with the State Intellectual Property Office of China on September 18, 2024, entitled "Document Processing Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and more particularly to a document processing method, a computing device, and a cluster of computing devices. Background Technology

[0003] Currently, Large Language Models (LLMs) are applied to long document processing, such as summarizing the content of long documents. However, this involves dividing the long document into multiple segments and using a large language model to summarize each segment, which leads to lower processing speed and efficiency, and longer document processing time. Summary of the Invention

[0004] This application provides a document processing method, a computing device, and a cluster of computing devices, thereby shortening the processing time of long documents and improving processing efficiency.

[0005] Firstly, a document processing method is provided, comprising: obtaining a document processing request; obtaining a first set of fragments of a first document according to the document processing request; processing the first set of fragments to obtain a summary result of the first document. The first set of fragments is a portion of a fragment graph network constructed from multiple fragments of the first document, and the fragment graph network is used to indicate the relationships between the multiple fragments of the first document.

[0006] The document processing method provided in this application constructs a fragment graph network by associating discrete fragments in a first document based on the relationships between multiple segments. Since the fragment graph network indicates the relationships between multiple segments of the first document, a small number of representative fragments are obtained from the fragment graph network to form a first fragment set. This first fragment set can express the content of the first document. Document summarization is then performed on the first fragment set to obtain a summary result of the first document. Therefore, summarizing the document using a small number of representative fragments selected from the fragment graph network ensures the quality of the document summary, reduces the number of fragments input to the model, shortens the processing time for long documents, and improves processing efficiency.

[0007] In one possible implementation, a fragment graph network is used to indicate the structure of a first document and the semantic relationships between multiple fragments of the first document, which are identified in graph form.

[0008] Since the fragment graph network indicates the semantic relationships between multiple fragments of the first document, a small number of representative fragments can be obtained from the fragment graph network to avoid selecting duplicate fragments, reduce the impact of information differences between different fragments, and ensure the quality of the document summary.

[0009] In another possible implementation, before obtaining the first set of fragments of the first document according to the document processing request, the method further includes: obtaining multiple fragments of the first document; and constructing a fragment graph network based on the features of the multiple fragments of the first document. The nodes in the fragment graph network indicate fragments of the first document, and the edges connecting the nodes in the fragment graph network indicate semantic relationships between the fragments.

[0010] A fragment graph network was constructed by associating discrete fragments in the first document based on a graph structure. This fragment graph network can intuitively display the structure of the document and the semantic relationships between the fragments of the document, providing strong support for subsequent fragment selection, retrieval and document summarization.

[0011] In another possible implementation, multiple fragments of the first document are obtained, including: dividing the first document according to a partitioning rule to obtain multiple fragments, wherein the partitioning rule is used to indicate the partitioning of the document based on document content, document type, or splitting window.

[0012] In another possible implementation, obtaining multiple fragments of the first document includes: performing a segmentation operation on the first document according to the segmentation window to obtain multiple fragments of the first document; constructing a fragment graph network based on the features of the multiple fragments of the first document includes: constructing a fragment graph network based on the similarity between the multiple fragments of the first document, where the similarity is used to indicate the degree of semantic relevance between the fragments.

[0013] In another possible implementation, the features of multiple fragments include fragment semantic relationships; constructing a fragment graph network based on the features of multiple fragments of the first document includes: establishing connections between fragments with fragment semantic relationships among the multiple fragments to construct a fragment graph network.

[0014] In another possible implementation, connecting segments with semantic relationships among multiple segments includes: connecting text segments with semantic relationships among multiple segments, wherein the segment graph network includes text segments with semantic relationships.

[0015] In another possible implementation, fragments with fragment semantic relationships also include analytical data fragments.

[0016] In another possible implementation, the features of the multiple fragments also include keywords; constructing a fragment graph network based on the features of the multiple fragments of the first document includes: connecting fragments with the same keywords among the multiple fragments to construct a fragment graph network.

[0017] In another possible implementation, connecting multiple fragments with the same key includes connecting multiple table fragments with the same field, where the fragment graph network includes table fragments with related relationships.

[0018] Fragment graph networks can be constructed by defining the relationships between fragments in a document, such as based on the semantic similarity of fragments or the relationships between entities in the fragments, thereby improving the flexibility of fragment graph network construction.

[0019] Furthermore, by combining the document segmentation method based on document content and document type provided in this application, and by establishing connections based on segments with semantic relationships or segments with the same keywords, a segment graph network is constructed. This improves the accuracy of segment graph network construction, thereby enabling the accurate selection of a small number of representative segments from the segment graph network for document summarization. This ensures the quality of document summarization, reduces the number of segments input to the model, shortens the processing time for long documents, and improves processing efficiency.

[0020] In another possible implementation, after constructing a fragment graph network based on the features of multiple fragments of the first document, the method further includes: dividing the fragment graph network into multiple sub-networks based on the hierarchical relationship of the document content, wherein the sub-networks contain content with the same features in the first document; and identifying the fragment with the most correlation in each sub-network as the key fragment of the sub-network.

[0021] The graph structure modeling method was used to mine the relationships between fragments in the first document, and implicit structural relationships were discovered in the document, laying the foundation for fragment selection, retrieval and document summarization based on fragment graph networks.

[0022] In another possible implementation, the first set of fragments includes a key fragment and at least one fragment associated with the key fragment.

[0023] In another possible implementation, the fragment graph network includes multiple subnetworks and key fragments of each subnetwork; obtaining a first fragment set of a first document according to a document processing request includes: obtaining key fragments in multiple subnetworks and at least one fragment associated with the key fragments to obtain a first fragment set.

[0024] In another possible implementation, key segments from multiple sub-networks and at least one segment associated with the key segments are obtained to form a first segment set, including: obtaining key segments from multiple sub-networks and at least one segment associated with the key segments based on the performance of the large language model to form a first segment set.

[0025] In this way, by adding the selection of document fragments, document summarization based on all fragments of the document is avoided, shortening the processing time of long documents and improving processing efficiency and document summary quality. In addition, by customizing the selection of multi-hop fragments associated with key fragments based on the performance of the large language model, the length of fragments input to the large language model can be flexibly controlled, further shortening the processing time of long documents and improving processing efficiency and document summary quality.

[0026] In another possible implementation, the fragment graph network includes multiple subnetworks and key fragments in each subnetwork; the document processing request includes a user question; obtaining a first fragment set of a first document according to the document processing request includes: obtaining key fragments in at least one subnetwork similar to the user question; obtaining at least one fragment associated with the key fragment in at least one subnetwork to obtain a first fragment set, the first fragment set including the key fragment in at least one subnetwork and at least one fragment associated with the key fragment.

[0027] By extracting key fragments relevant to user questions and summarizing the document based on these fragments, we can obtain summary results highly relevant to user intent, thus enabling us to handle summary tasks that satisfy specific user intentions. This eliminates the need to traverse all document fragments, filtering out irrelevant ones, shortening the processing time for long documents, and improving processing efficiency and the quality of document summaries.

[0028] In another possible implementation, the first set of fragments is processed to obtain a summary result of the first document, including: using a large language model to process the first set of fragments to obtain a summary result of at least two fragments in the first set of fragments; and concatenating the summary results of at least two fragments in the first set of fragments to obtain a summary result of the first document.

[0029] By summarizing a small number of representative fragments obtained from the fragment graph network, the quality of the summary results is high. By concatenating the summary results of at least two fragments in the first fragment set, it is not necessary to perform document summarization on the summary results of the fragments through a large language model, thus reducing the amount of computation.

[0030] In a second aspect, a document processing apparatus is provided, comprising modules for performing the methods of the first aspect or any possible design of the first aspect. For example, the document processing apparatus includes a communication module, a document selection module, and a document processing module.

[0031] The communication module is used to obtain document processing requests; the document selection module is used to obtain a first set of fragments of the first document according to the document processing request. The first set of fragments comes from a part of a fragment graph network constructed from multiple fragments of the first document. The fragment graph network is used to indicate the relationship between multiple fragments of the first document; the document processing module is used to process the first set of fragments to obtain a summary result of the first document.

[0032] In one possible implementation, a fragment graph network is used to indicate the structure of a first document and the semantic relationships between multiple fragments of the first document, which are identified in graph form.

[0033] In another possible implementation, the apparatus further includes a graph construction module; a communication module for acquiring multiple fragments of the first document; and a graph construction module for constructing a fragment graph network based on the features of the multiple fragments of the first document, wherein the nodes in the fragment graph network indicate fragments of the first document, and the edges connecting the nodes in the fragment graph network indicate semantic relationships between the fragments.

[0034] In another possible implementation, the features of multiple fragments include fragment semantic relationships; when the graph construction module constructs a fragment graph network based on the features of multiple fragments of the first document, it is specifically used to: establish connections between fragments with fragment semantic relationships among multiple fragments to construct a fragment graph network.

[0035] In another possible implementation, when the graph building module connects multiple fragments that have semantic relationships, it is specifically used to: connect multiple text fragments that have semantic relationships, and the fragment graph network includes text fragments with semantic relationships.

[0036] In another possible implementation, fragments with fragment semantic relationships also include analytical data fragments.

[0037] In another possible implementation, the features of multiple fragments also include keywords; when the graph construction module constructs a fragment graph network based on the features of multiple fragments of the first document, it is specifically used to: establish connections between fragments with the same keywords in multiple fragments to construct a fragment graph network.

[0038] In another possible implementation, when the graph building module connects multiple fragments with the same key, it is specifically used to connect multiple table fragments with the same field, and the fragment graph network includes table fragments with related relationships.

[0039] In another possible implementation, the graph building module is also used to: divide the fragment graph network into multiple subnetworks based on the hierarchical relationship of the document content, with each subnetwork containing content with the same features in the first document; and identify the fragment with the most relationships in each subnetwork as the key fragment of the subnetwork.

[0040] In another possible implementation, the first set of fragments includes a key fragment and at least one fragment associated with the key fragment.

[0041] In another possible implementation, the fragment graph network includes multiple sub-networks and key fragments of each sub-network; when the document selection module obtains the first fragment set of the first document according to the document processing request, it is specifically used to: obtain key fragments in multiple sub-networks and at least one fragment associated with the key fragments to obtain the first fragment set.

[0042] In another possible implementation, when the document selection module obtains key segments from multiple sub-networks and at least one segment associated with the key segments to obtain a first segment set, it is specifically used to: obtain key segments from multiple sub-networks and at least one segment associated with the key segments based on the performance of the large language model to obtain a first segment set.

[0043] In another possible implementation, the fragment graph network includes multiple subnetworks and key fragments in each subnetwork; the document processing request includes a user question; when the document selection module obtains the first fragment set of the first document according to the document processing request, it is specifically used to: obtain key fragments in at least one subnetwork similar to the user question; obtain at least one fragment associated with the key fragment in at least one subnetwork to obtain the first fragment set, the first fragment set including the key fragment in at least one subnetwork and at least one fragment associated with the key fragment.

[0044] In another possible implementation, when the document processing module uses a large language model to process the first set of fragments and obtain the summary result of the first document, it specifically performs the following steps: using the large language model to process the first set of fragments and obtain the summary results of at least two fragments in the first set of fragments; merging the summary results of at least two fragments in the first set of fragments to obtain the summary result of the first document.

[0045] Thirdly, a data processing system is provided, comprising various means for performing the methods of the first aspect or any possible design of the first aspect. For example, the data processing system includes an execution means and a training means. The execution means is used to obtain a first set of fragments of a first document according to a document processing request, and to process the first set of fragments using a large language model to obtain a summary result of the first document. The first set of fragments is derived from a portion of a fragment graph network constructed from multiple fragments of the first document, the fragment graph network being used to indicate the relationships between the multiple fragments of the first document. The training means is also used to train the large language model.

[0046] Fourthly, a chip system is provided, the chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the operational steps of the method in the first aspect or any possible implementation of the first aspect.

[0047] Fifthly, a computing device is provided, the computing device including a memory and a plurality of processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the computing device is caused to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0048] In a sixth aspect, a computing device is provided, comprising a general-purpose processor, a special-purpose processor, and a memory; the general-purpose processor is configured to execute instructions stored in the memory such that the general-purpose processor acquires a first set of fragments of a first document, and the special-purpose processor processes the first set of fragments to obtain a summary result of the first document, thereby causing the computing device to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0049] In a seventh aspect, a computing device cluster is provided, comprising at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the device to cause the computing device cluster to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0050] Eighthly, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a processor, causing the processor to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0051] Ninthly, a computer program product is provided that, when run on a computer, causes the computer to perform the operational steps of the method as described in the first aspect or any possible implementation thereof.

[0052] The technical effects of any of the design methods in aspects two through nine can be found in aspect one or in different design methods in aspect one, and will not be repeated here.

[0053] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0054] Figure 1 is a schematic diagram of the architecture of a data processing system provided in this application;

[0055] Figure 2 is a schematic diagram of the deployment of a document summary function provided in this application;

[0056] Figure 3 is a schematic diagram of the process of constructing a fragment graph network according to this application;

[0057] Figure 4 is a schematic diagram of the process for constructing a sub-network and key segments according to this application;

[0058] Figure 5 is a flowchart illustrating a document processing method provided in this application;

[0059] Figure 6 is a schematic diagram of a document summary scenario provided in this application;

[0060] Figure 7 is a schematic diagram of a document summary scenario provided in this application;

[0061] Figure 8 is a structural schematic diagram of a document processing device provided in this application;

[0062] Figure 9 is a schematic diagram of the structure of a computing device provided in this application;

[0063] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0064] Figure 11 is a schematic diagram of the structure of a computing device cluster provided in this application. Detailed Implementation

[0065] To facilitate understanding, the main terms used in this application will be explained first.

[0066] Large models refer to ultra-large-scale artificial intelligence (AI) models. They are widely used in natural language processing (NLP) and are transforming the landscape of NLP tasks, giving rise to more powerful and intelligent language technologies. Large models are a key direction in AI development. They also excel in various NLP tasks, such as document summarization, document classification, sentiment analysis, summary generation, and translation. Large models are used in multiple application areas, including automatic writing, chatbots, virtual assistants, voice assistants, and automatic translation. Examples include Large Language Models (LLMs), Bidirectional Encoder Representations from Transformers (BERT), Generative Pre-Trained Transformers (GPT), GPT3, GPT4, and Mixture of Experts (MoE).

[0067] Large language models refer to deep learning models trained on large amounts of text data to generate natural language text or understand the meaning of language text. Large language models can directly process input objects (such as documents of different formats) to meet the processing needs of different tasks such as intelligent question answering, sentiment analysis, chat, and coding.

[0068] Because large language models have excellent text processing capabilities, they can be applied to document summarization (or abstract generation). A long text is input into a large language model, which then produces a summary (or abstract). In practical applications, if the length of the long text exceeds the maximum context length that a large language model can handle, the long document can be divided into smaller segments, and each segment can be input into the large language model for summarization to obtain a summary of the long text. However, document summarization is time-consuming and the quality of the summaries is not high.

[0069] To address the issues of time-consuming and inefficient document summarization for long documents, this application provides a document processing method that involves obtaining a document processing request; obtaining a first set of fragments from a first document based on the document processing request; processing the first set of fragments to obtain a summary result of the first document. The first set of fragments is derived from a portion of a fragment graph network constructed from multiple fragments of the first document, and the fragment graph network is used to indicate the relationships between the multiple fragments of the first document.

[0070] The document processing method provided in this application constructs a fragment graph network by associating discrete fragments in a first document based on the relationships between multiple segments. Since the fragment graph network indicates the relationships between multiple segments of the first document, a small number of representative fragments are obtained from the fragment graph network to form a first fragment set. This first fragment set can express the content of the first document. Document summarization is then performed on the first fragment set to obtain a summary result of the first document. Therefore, summarizing the document using a small number of representative fragments selected from the fragment graph network ensures the quality of the document summary, reduces the number of fragments input to the model, shortens the processing time for long documents, and improves processing efficiency.

[0071] The implementation methods of the document processing method provided in this application will now be described in detail with reference to the accompanying drawings.

[0072] The solution provided in this application can be applied to network systems that include clients and servers. For ease of understanding, the example given is a client / server (C / S) architecture network system using such an architecture. It is worth noting that the communication method between the client and server does not limit the applicable scenarios of this application.

[0073] Specifically, a client / server (C / S) architecture network can include a single server or a server cluster. This application does not limit the type and function of the servers described. For example, server types include blade servers, tower servers, rack servers, and platform servers. Furthermore, servers may include storage servers with storage capabilities or computing servers with computing capabilities; computing servers may also have storage capabilities.

[0074] For example, Figure 1 is a schematic diagram of the architecture of a data processing system provided in this application. As shown in Figure 1, the data processing system 100 includes a client 110 and a control node 120.

[0075] For example, control node 120 is a computer device, such as a server, desktop computer, or storage array controller.

[0076] Optionally, the control node 120 is a heterogeneous computing architecture to provide high-performance computing. For example, the control node 120 includes computing units with computing capabilities such as a central processing unit (CPU), a graphics processing unit (GPU), a data processing unit (DPU), a neural processing unit (NPU), and an embedded neural-network processing unit (NPU) to provide high-performance computing.

[0077] As one possible implementation, the control node 120 can also be implemented in software, such as through a virtual machine, container, or software model. This application does not limit this implementation. For ease of understanding, the following embodiments use the computing device shown in Figure 1 as an example to illustrate the control node 120.

[0078] Control node 120 is used to construct a fragment graph network, obtain the first fragment set of the first document according to the document processing request, process the first fragment set, and obtain the summary result of the first document.

[0079] The control node 120 also includes memory, including volatile memory such as random access memory (RAM). The memory is used to store the fragment graph network constructed according to the document processing method provided in this application.

[0080] Optionally, the data processing system 100 includes multiple control nodes 120, which communicate with each other. For example, the multiple control nodes 120 execute the document processing method provided in this application.

[0081] In some embodiments, client 110 communicates with control node 120 via network 130. For example, client 110 sends a document processing request to control node 120 via network 130, requesting control node 120 to obtain a first set of fragments of a first document, process the first set of fragments, and obtain a summary result of the first document. For example, network 130 refers to an enterprise intranet (e.g., a local area network, LAN) or the Internet. Client 110 refers to a computer connected to network 130, also known as a workstation. Different clients share network resources (e.g., computing resources, storage resources).

[0082] Client 110 has client program 111 installed. Client 110 runs client program 111 and displays a user interface (UI). User 140 uses the user interface to submit document processing requests. For example, user 140 uses the user interface to submit multiple document processing requests. After receiving a document processing request, control node 120 obtains the first set of fragments of the first document, processes the first set of fragments, and obtains the summary result of the first document.

[0083] In other embodiments, control node 120 performs document processing based on a heterogeneous computing architecture. For example, the heterogeneous computing architecture includes a general-purpose processor and a dedicated processor. For instance, the general-purpose processor is a CPU, and the dedicated processor is an NPU. The general-purpose processor is used to acquire multiple fragments of the first document and construct a fragment graph network based on the features of the multiple fragments of the first document. The dedicated processor is used to process the first set of fragments to obtain a summary result of the first document.

[0084] Optionally, the control node 120 includes multiple dedicated processors. For example, a first dedicated processor uses a large language model to process a first number of segments in the first segment set to obtain a summary result of the first number of segments. A second dedicated processor uses the large language model to process a second number of segments in the first segment set to obtain a summary result of the second number of segments. A general-purpose processor concatenates the summary results of at least two segments in the first segment set to obtain a summary result of the first document.

[0085] Optionally, the system administrator 150 may configure system information, such as the algorithm used to construct the graph network, the community mining algorithm, and the number of hops for segment selection, by calling the application platform interface (API) 112 or the command-line interface (CLI) 113 through the client 110.

[0086] It is worth noting that Figure 1 is merely a schematic diagram of a data processing system to which the technical solution to be protected in this application applies. This application does not limit the connection method or the number of devices in the data processing system. For example, the data processing system includes multiple clients. One client connects to multiple control nodes. Different clients establish connections with different control nodes.

[0087] The document processing function provided in this application is implemented by a software system, a hardware device, or a combination of both. In one possible implementation, as shown in Figure 2, the cloud service provider abstracts the document summarization function into a cloud service and deploys this service in a cloud environment. Users consult and purchase this cloud service through a cloud platform. After purchasing the cloud service, the cloud platform provides the service to the user. For example, a document processing request is submitted to a data center via a terminal device, and the data center runs a document processing module to implement the document processing function provided in this application.

[0088] In another possible implementation, the document processing module is packaged into a software package by a software provider. The user purchases this package and deploys it on their own server, or on a cloud server. For example, the document processing module is deployed by a tenant on computing resources (e.g., virtual machines) in a rented data center. The tenant purchases cloud computing resources from a cloud service provider through a cloud service platform and runs the document processing module on these resources, enabling it to perform the document processing functions provided in this application. Optionally, the document processing module also encrypts the user-uploaded data and its file paths to avoid direct access to the user-uploaded data without hindering the implementation effect, thus ensuring information security.

[0089] Next, the document processing process will be explained in detail with reference to the accompanying drawings.

[0090] First, the construction process of a fragment graph network is explained. Figure 3 is a schematic diagram of a fragment graph network construction process provided in this application. Here, we will use the control device shown in Figure 1 or the cloud environment shown in Figure 2 as an example to illustrate the process. As shown in Figure 3, the method includes the following steps.

[0091] Step 310: Obtain multiple fragments from the first document.

[0092] The control node divides the first document according to the partitioning rules, resulting in multiple fragments. These multiple fragments constitute the first document; that is, the multiple fragments are all the fragments in the first document; or, the multiple fragments are some of the fragments that make up the first document. The first document is either a long document or an extremely long document.

[0093] It is worth noting that when using large language models to analyze documents (e.g., extracting summaries), due to the limitations of document size and the processing power of large language models, it is often impossible to obtain analysis results directly from a complete document. Usually, it is necessary to divide the long or very long document into multiple segments, and then perform analysis processing based on multiple segments. Such documents are also referred to in the industry as long documents or very long documents.

[0094] The partitioning rules are used to indicate how documents are partitioned. This application provides the following document partitioning methods.

[0095] In the first possible implementation, the partitioning rules are used to indicate how to partition documents based on their content.

[0096] For example, the first document can be divided into multiple segments according to its document structure. A segment is a paragraph contained in the first document. For instance, the first document can be divided into multiple segments according to its "general-specific," "general-specific-general," or "specific-general" structure.

[0097] For example, the first document can be divided into multiple segments according to the time sequence of its content, changes in location, or the development of events.

[0098] For example, the first document can be divided into multiple segments based on the multiple keywords it contains. A segment is a segment that contains one type of keyword. This segment contains one or more paragraphs from the first document. Optionally, the first document may contain tables containing keywords, and the segment may also contain tables.

[0099] In the second possible implementation, the partitioning rules are used to indicate how documents are partitioned based on document type.

[0100] In some embodiments, the document type indicates the type of content contained in the document. Document types include text types, table types, and analytical data types, etc. For example, a first document contains text and tables. The text in the first document is divided into one segment, and the tables in the first document are divided into another segment. Optionally, the first document contains multiple tables, and one table is considered as one segment. As another example, the first document contains bar charts, line charts, etc., and the bar charts and line charts are considered as one segment.

[0101] In the third possible implementation, the partitioning rules are used to indicate how to partition the document based on the splitting window.

[0102] A segmentation window is used to indicate the size of a segment. The first document is segmented into multiple segments based on the segmentation window. This application does not limit the size of the segmentation window; its size can be flexibly configured according to different needs. For example, the size of the segmentation window can be 1024 bytes.

[0103] In some embodiments, the size of the segmentation window can be flexibly configured based on the performance of the large language model, the language, and the use case. For example, the stronger the performance of the large language model, the larger the segmentation window, and the longer the resulting segments. The performance of the large language model is measured by computing power and memory. Strong performance of the large language model refers to characteristics such as large computing power and large memory. Poor performance of the large language model refers to characteristics such as low computing power and / or small memory. Weak performance of the large language model results in a smaller segmentation window and a shorter resulting segments.

[0104] For example, since the byte length of characters differs in different languages, different window sizes should be set for documents in different languages.

[0105] For example, the content of a document summary varies depending on the usage scenario. Financial documents, for instance, often contain numerous tables. Since financial documents contain relatively fragmented information, smaller segmentation windows are used to accurately summarize the meaning of the document, thus improving the quality of the summary.

[0106] Optionally, split the text by character count or delimiter (such as period, comma, etc.). Use CharacterTextSplitter to split the text in a fixed-length order, or use TokenTextSplitter to split the document based on the length of the token.

[0107] Optionally, you can divide the first document into multiple segments by creating new paragraphs through simple operations such as pressing the Enter key, or by using the Shift+Enter key combination to insert hard or soft returns.

[0108] Choosing the appropriate partitioning method depends on the specific needs and purpose of the document. For example, in semantic search, when indexing a document corpus, it is necessary to ensure that the search results accurately capture the essence of the user's query; therefore, choosing a suitable partitioning method is crucial.

[0109] The above describes several different document segmentation rules. This application does not limit which document segmentation rule to use; it should be set according to actual application needs. For example, one or more of the above segmentation methods can be combined. For instance, the first document can be divided into multiple segments according to its document structure. If a segment is long, it can be further divided into multiple segments based on delimiters (such as periods, commas, etc.). For instance, the first document can be divided into multiple segments according to multiple keywords it contains. Then, each segment can be further divided into multiple segments based on the keywords it contains.

[0110] Step 320: Construct a fragment graph network based on the features of multiple fragments of the first document.

[0111] The multiple fragments of the first document are related, and the features of the multiple fragments described in this application are, for example, related features. For instance, different fragments have similar semantics. Or, different fragments contain the same keywords. The features of the multiple fragments described in this application include fragment semantic relationships and / or keywords.

[0112] Based on the characteristics of multiple fragments of a first document, related fragments are connected to construct a fragment graph network. A fragment graph network is a network structure that uses a graph to represent the structure of a first document and the semantic relationships between its multiple fragments.

[0113] In the first possible implementation, segments with semantic relationships among multiple segments are connected to construct a segment graph network.

[0114] Segment semantics indicates the meaning of a segment, that is, the meaning that the segment intends to express. Different segments express similar meanings, and therefore, the different segments have segment semantic relationships.

[0115] In some embodiments, a first document contains text, and multiple segments dividing the first document contain text segments. Text segments with semantic relationships are connected within the multiple segments. The segment graph network comprises text segments with semantic relationships.

[0116] Different segments may belong to a single paragraph in a first document. A paragraph may contain multiple statements, and these statements may have similar meanings. A paragraph may be divided into multiple segments, each segment containing one or more statements from the paragraph, and these segments may have similar meanings.

[0117] For example, segments with semantic relationships are identified based on their semantic similarity. These segments are then connected to form a segment graph network. Assume the first document is divided into 5 segments, namely segment 1 to segment 5.

[0118] First, semantic analysis is performed on each of fragments 1 to 5 to obtain semantic features. Optionally, the semantic features are vectorized numerical values. For example, natural language processing such as part-of-speech tagging, named entity recognition, and dependency parsing is performed on the fragments to identify the meaning they are meant to convey, ensuring accurate capture of the semantic content of the fragments.

[0119] Next, calculate the similarity between any two segments based on the semantic features of each segment from segment 1 to segment 5. For example, calculate the similarity between segment 1 and segment 2 based on the semantic features of segment 1 and segment 2. Calculate the similarity between segment 1 and segment 3 based on the semantic features of segment 1 and segment 3. And so on, calculate the similarity between segment 1 and segment 4, segment 1 and segment 5, segment 2 and segment 3, segment 2 and segment 4, segment 2 and segment 5, segment 3 and segment 4, segment 3 and segment 5, and segment 4 and segment 5.

[0120] Similarity is used to indicate the degree of relevance between multiple segments. Similarity algorithms include, but are not limited to, cosine similarity, Jaccard similarity, or other custom similarity metrics to assess the strength of semantic association between segments. For example, similarity values ​​range from 0 to 1. The closer the similarity is to 1, the stronger the relevance between the two segments, and the more semantically similar they are; the closer the similarity is to 0, the weaker the relevance between the two segments, and the greater the semantic difference between them.

[0121] Finally, segments with semantic relationships are identified based on their similarity. For each segment, segments with semantic relationships are connected based on their similarity to construct a segment graph network. The similarity between two segments is checked against a threshold. If the similarity is greater than the threshold, the two segments are determined to have a semantic relationship, and the segments with the semantic relationship are connected. If the similarity is less than the threshold, the two segments are determined not to have a semantic relationship. Optionally, if the similarity is equal to the threshold, the two segments are either determined to have a semantic relationship or not.

[0122] A larger threshold indicates a more stringent requirement for establishing edges between two segments, resulting in fewer edges; a smaller threshold indicates a more lenient requirement for establishing edges between two segments, resulting in more edges. This application does not limit the value of the threshold. For example, the threshold can be 0.8 or 0.9. In this way, two segments will only be connected if they are semantically related, thus ensuring the quality and relevance of the segment graph network.

[0123] If the semantics of segment 1 are similar to those of segment 3 and segment 4, then segment 1, segment 3, and segment 4 are connected. If the semantics of segment 2 are similar to those of segment 3, then segment 2 and segment 3 are connected. If the semantics of segment 4 are similar to those of segment 5, then segment 4 and segment 5 are connected.

[0124] In this fragment graph network, the nodes indicate fragments of the first document, and the edges connecting the nodes indicate the semantic relationships between the fragments. Within a document, fragments with relationships may be geographically distant. The fragment graph network provided in this application can intuitively display the document's structure and the semantic connections between its fragments, providing strong support for subsequent fragment selection, retrieval, and document summarization.

[0125] In other embodiments, the first document also includes analytical data. For example, the analytical data may include bar charts, line graphs, or other data capable of expressing semantic information. Multiple segments dividing the first document contain analytical data segments. For example, connections between analytical data segments are established using the method described above for establishing connections between text segments. Semantic analysis is performed on the analytical data segments to obtain semantic features. The similarity between any two analytical data segments is calculated based on their semantic features. Analytical data segments with semantic relationships are determined based on their similarity, and these segments are connected to construct a segment graph network.

[0126] Optionally, if there is a semantic relationship between a text fragment and an analysis data fragment, a connection will also be established between the text fragment and the analysis data fragment. The fragment graph network includes text fragments with semantic relationships, analysis data fragments with semantic relationships, analysis data fragments with semantic relationships, and text fragments.

[0127] In the second possible implementation, segments with the same keyword from multiple segments are connected to construct a segment graph network.

[0128] For example, connecting segments containing the same keyword from multiple segments of the first document.

[0129] For example, suppose the first document is divided into 5 segments, namely segments 1 to 5. Segments 1, 3, and 4 all contain the first keyword, so segments 1, 3, and 4 are joined together. Segments 2 and 3 both contain the second keyword, so segments 2 and 3 are joined together. Segments 4 and 5 both contain the third keyword, so segments 4 and 5 are joined together.

[0130] For example, connecting segments with keyword hierarchical relationships within multiple segments of the first document.

[0131] For example, if both fragment 1 and fragment 3 contain the first keyword, then fragment 1 and fragment 3 are connected. If both fragment 2 and fragment 4 contain the second keyword, and the second keyword belongs to the category of the first keyword, then fragment 2 and fragment 4 are both connected to fragment 1 and fragment 3. If fragment 5 contains the third keyword, and the third keyword belongs to the category of the first keyword, then fragment 5 is connected to both fragment 1 and fragment 3.

[0132] In some embodiments, the first document contains a table. The table header includes multiple fields. Keywords are, for example, the fields of the table. Table fragments with the same fields are linked together, and the fragment graph network includes table fragments with related relationships.

[0133] For example, comparing the number of identical fields in two tables: if the number of identical fields in both tables is greater than a threshold, the two tables are determined to be related, and the related tables are joined; if the number of identical fields in both tables is less than the threshold, the two tables are determined not to be related. Optionally, if the number of identical fields in both tables is equal to the threshold, the two tables are determined to either be related or not related.

[0134] This application does not limit the document type of the first document, and may use one or more of the aforementioned document types in combination. For example, the first document may contain at least one of text, tables, or analytical data. Different segment connection methods are used to establish connections between segments of different document types. For example, for text segments and analytical data segments, semantic relationships between segments are used to establish connections. For table segments, keywords are used to establish connections between segments. Optionally, keywords may also be used to establish connections between text segments and analytical data segments.

[0135] The order in which the fragment graph network is constructed is not restricted. Each fragment contained in the first document is traversed, and a fragment graph network is constructed according to the fragment connection method described above.

[0136] In other embodiments, after constructing a fragment graph network based on the features of multiple fragments of the first document, the implicit structural relationships in the fragment graph network are mined to lay the foundation for fragment selection, retrieval, and document summarization based on the fragment graph network, so as to adapt to diverse application scenarios and document needs of different sizes. As shown in Figure 4, the embodiments of this application also include steps 330 and 340.

[0137] Step 330: Divide the fragment graph network into multiple sub-networks according to the hierarchical relationship of the document content.

[0138] In some possible implementations, the hierarchical relationship of document content refers to the semantic hierarchical relationship between the content contained in the document. For example, the document structure determines the semantic hierarchical relationship between the content contained in the document. Optionally, the hierarchical relationship of document content refers to the semantic hierarchical relationship of content related to the document processing purpose. For example, the semantic hierarchical relationship extracted from the document based on the document processing purpose.

[0139] In some embodiments, the hierarchical relationship of the document content is obtained based on the document processing purpose. The document processing purpose is obtained based on the document processing request. Optionally, the document processing purpose is obtained by analyzing the document processing request using a large language model.

[0140] For example, a document processing request instructs the extraction of a document summary. The purpose of document processing is to extract a document summary. Semantic analysis is performed on the document to obtain the semantic hierarchy relationships between the content contained within it.

[0141] For example, a document processing request might contain user questions related to the document. The purpose of document processing is to provide feedback on these user questions. Semantic analysis is performed on the document to obtain the semantic hierarchy of the content related to the user questions. Optionally, the semantic hierarchy between the content within the document can be obtained first, and then the semantic hierarchy of the content related to the user questions can be obtained based on this hierarchy. For instance, semantic analysis can be performed on the user questions to obtain their keywords, and then the semantic hierarchy of the content related to the user questions can be obtained based on these keywords.

[0142] For example, the document contains multiple keywords indicating business processes, with a hierarchical relationship between these keywords. For instance, tax types include income tax, value-added tax (VAT), and vehicle and vessel tax. Income tax includes personal income tax and corporate income tax. Multiple keywords include income tax, personal income tax, corporate income tax, VAT, and vehicle and vessel tax. For example, segments related to income tax are divided into a sub-network. Income tax has a hierarchical relationship with personal income tax and corporate income tax. For example, segments related to personal income tax are also divided into a sub-network, and segments related to corporate income tax are divided into another sub-network.

[0143] For example, the content described in the document, including chronological order, location changes, or event progression, may have a hierarchical relationship. For instance, segments related to income tax in year 1 could be divided into one subnetwork, and segments related to income tax in year 2 into another.

[0144] After obtaining the hierarchical relationship of the document content, the network consisting of fragments belonging to the same level in the fragment graph network is divided into a sub-network.

[0145] In some embodiments, segments with hierarchical relationships in the segment graph network are mined using a community mining algorithm. Segments with the same characteristics in the segment graph network are then divided into subnetworks, resulting in multiple subnetworks. Each subnetwork contains content with the same characteristics from the first document. For example, a subnetwork may contain at least one segment with the same keyword. Alternatively, a subnetwork may include at least two segments with segment semantic relationships.

[0146] For example, a subnetwork refers to a community. A subnetwork contains a hierarchy of document content. A fragment graph network contains multiple related fragments about different types of taxes. A subnetwork contains at least one fragment of a single type of tax. A first subnetwork contains at least one fragment related to income tax. A second subnetwork contains at least one fragment related to personal income tax. A third subnetwork contains at least one fragment related to corporate income tax. A first subnetwork contains both a second and a third subnetwork.

[0147] Community mining algorithms are used to calculate and analyze the relationships between nodes and edges in a network to discover the community structure within the network. The goal of these algorithms is to reveal hidden relationships and patterns in the network and identify key nodes.

[0148] In a fragment graph network, interconnected nodes represent nodes with semantic relationships or nodes with weak semantic relationships, while unconnected nodes represent nodes without semantic relationships. The greater the distance between nodes, the weaker the semantic relationship. Nodes with more connections are grouped into a subnetwork.

[0149] For example, the similarity of each node in a fragment graph network to other nodes is used to assess the tightness of the community. For instance, node similarity can be determined based on the semantic similarity of the fragments. It is determined whether the similarity between two fragments is greater than a threshold. If the similarity is greater than the threshold, the two nodes are considered similar and are grouped into a subnetwork; if the similarity is less than the threshold, the two nodes are considered dissimilar. The semantic similarity of fragments can be referred to the explanation in step 320 above, and will not be repeated here.

[0150] Optionally, the boundaries of subnetworks can be defined based on the number of similar nodes connected to a node. For example, if the number of similar nodes connected to a node is less than a threshold, then that node is determined to be the boundary of the subnetwork.

[0151] Step 340: Identify the segment with the most connections in each subnetwork as the key segment of the subnetwork.

[0152] After dividing the fragment graph network into multiple subnetworks, key fragment identification is performed on each subnetwork to accurately identify the most influential or bridging fragments within each subnetwork. Key fragments in the subnetworks are determined using the following key fragment identification method.

[0153] Method 1: Determine key nodes based on their degree, which indicates the number of connected nodes. Nodes in each subnetwork are sorted according to the number of other nodes they are connected to, i.e., their degree. The node with the highest degree is marked as a key node, and key nodes are connected to the most extensive group of nodes within the subnetwork.

[0154] Method two involves identifying key nodes based on closeness centrality. This method focuses on the mediation efficiency of nodes within a subnetwork. It assesses a node's importance by calculating the average shortest path length from the node to all other nodes in the subnetwork. The closeness centrality of each node in the subnetwork is calculated, and the node with the highest closeness centrality is identified as a key node. Key nodes can quickly connect to other nodes and are crucial for interaction and information flow within the subnetwork.

[0155] The formula for near-centrality is: Among them, c (u) Let d(u,v) represent the proximity centrality of node u, N represent the total number of nodes in the network, d(u,v) represent the shortest path length between node u and node v, and v≠u represents the summation of all nodes v that are different from node u.

[0156] For example, as shown in Figure 4, the fragment graph network is divided into multiple subnetworks, where each subnetwork includes nodes with identical padding and identifies key nodes. Key nodes in subnetwork 1 are connected to all other nodes in subnetwork 1. Key nodes indicate key fragments.

[0157] The fragment graph network shown in this application is merely an illustration and is not intended to limit the scope of the application. In practical applications, fragment graph networks can be graph networks of arbitrary structure. For example, a fragment graph network may also include fragments that are not connected to any other fragment.

[0158] After constructing the fragment graph network and identifying its subnetworks and key segments, fragments are selected from the network for document summarization. This process involves choosing a small number of representative fragments that effectively represent the content of the first document. This ensures the quality of the document summarization, reduces the number of fragments input to the model, shortens the processing time for long documents, and improves processing efficiency.

[0159] Figure 5 is a flowchart illustrating a document processing method provided in this application. Here, we will use the computing device shown in Figure 1 or the cloud environment shown in Figure 2 as an example to illustrate the document summarization process. As shown in Figure 5, the method includes the following steps.

[0160] Step 510: Obtain the document processing request.

[0161] The client sends document processing requests to the computer device via a network. This network can be an internal corporate network (such as a local area network) or the Internet. The computer device then receives the document processing requests from the client.

[0162] A document processing request is used to instruct a document to be summarized. The document processing request contains relevant information about the document so that a computer device can obtain the document that needs to be processed. For example, a document processing request may include at least one of the following: document name, folder name containing the document, document storage location, etc. Optionally, the document processing request may include the document to be processed.

[0163] The embodiments of this application do not limit the object of the document summary. For example, the object of the document summary includes text, charts, tables, etc. In addition, the first document described in this application is a long text, for example, the first document includes at least one document.

[0164] Here, we take the first document as an example and assume that the document processing request is used to instruct a document summary of the first document.

[0165] Step 520: Obtain the first set of fragments of the first document according to the document processing request.

[0166] Fragment graph networks are used to indicate the relationships between multiple fragments of a first document. The nodes in a fragment graph network represent fragments of the first document, and the edges connecting these nodes indicate the semantic relationships between the fragments. That is, fragments with high similarity are connected, while fragments with low similarity are not connected.

[0167] In some embodiments, a fragment graph network includes multiple subnetworks and key fragments in each subnetwork. Subnetworks contain fragments with high similarity.

[0168] When summarizing documents using fragments from a subnetwork, the summaries of each fragment show minimal variation. Therefore, in document summarization, highly representative fragments are selected from the subnetwork, while some are discarded. These highly representative fragments contain the main information of the subnetwork. Summarizing these representative fragments ensures the quality of the document summary and reduces the number of fragments input to the model, shortening the processing time for long documents and improving efficiency. Furthermore, grouping similar fragments together for summarization avoids the possibility of slight information differences in individual fragments causing the large language model to deviate from the overall theme of the fragment when each fragment is processed independently.

[0169] The first set of fragments is a portion of a fragment graph network constructed from multiple fragments of the first document. The first set of fragments includes key fragments and at least one fragment associated with the key fragments.

[0170] In a first possible implementation, key segments from multiple sub-networks and at least one segment associated with the key segments are obtained to obtain a first segment set.

[0171] Obtain the key segments in each sub-network and at least one segment associated with the key segment to obtain the first segment set.

[0172] In some embodiments, as shown in Figure 6, key segments and segments within k-hop relationships with the key segments are flexibly selected to obtain a first segment set. Optionally, the k-hops are determined by the performance of the large language model. The first segment set is then processed using the large language model to obtain a summary result of the first document.

[0173] Larger language models have stronger performance, and k can be set larger, selecting key segments in each subnetwork, as well as k-hop segments that are frequently associated with the key segments. Larger language models have weaker performance, and k can be set smaller, selecting key segments in each subnetwork, as well as k-hop segments that are less frequently associated with the key segments.

[0174] In the second possible implementation, based on the user intent, the fragment graph network is quickly retrieved, the sub-networks in the fragment graph network related to the user intent are obtained, and the first set of fragments in the first document related to the user intent is obtained.

[0175] The document processing request includes a user question. Key segments from at least one sub-network similar to the user question are retrieved; at least one segment associated with the key segments in the at least one sub-network is retrieved, resulting in a first segment set. The first segment set includes the key segments from the at least one sub-network and at least one segment associated with the key segments. For example, summarizing the content in a first document related to a first topic.

[0176] In some embodiments, as shown in Figure 6, the similarity between the user question and key segments in multiple subnetworks is calculated. The user question and key segments are semantically vectorized, and their semantic similarity is calculated. Similarity algorithms include, but are not limited to, cosine similarity, Jaccard similarity, or other custom similarity metrics.

[0177] For example, 1. First, vectorize the key fragments and user questions, resulting in two vectors. 2. Calculate the similarity between the two vectors, for example, the similarity satisfies the following formula: Here, A·B represents the dot product of vectors AA and BB, ||A|| represents the magnitude of vector AA, and ||B|| represents the magnitude of vector BB. Furthermore, the similarity between the user's question and multiple key segments is ranked. The top k similarities are selected from highest to lowest similarity. The segments corresponding to these top k similarities are then input into the large language model for document summarization, yielding k summary results. These k summary results are then concatenated to obtain the final summary result relevant to the user's specified content.

[0178] For example, as shown in Figure 7, based on the user question, two sub-networks similar to the user question are selected from the fragment graph network. Using the large language model, the fragments contained in the two sub-networks are processed to obtain the summary result of the first document.

[0179] Optionally, after obtaining at least one subnetwork similar to the user's question, at least one segment associated with the key segment is also obtained from each subnetwork to obtain a first segment set. That is, it is not necessary to use all segments in the subnetwork for document summarization; instead, a selection of representative segments from the subnetwork is chosen for summarization.

[0180] In addition, after obtaining the document processing request and identifying the documents to be summarized, the process involves constructing a fragment graph network for the documents and identifying subnetworks and key segments within those subnetworks. Then, a segment selection operation is performed. That is, steps 310 to 340 are executed between steps 510 and 520.

[0181] Step 530: Process the first set of fragments to obtain the summary result of the first document.

[0182] Using a large language model, the first set of fragments is processed to obtain summary results for at least two fragments in the first set of fragments; the summary results for at least two fragments in the first set of fragments are then concatenated to obtain the summary result for the first document. For example, merging the summary results for at least two fragments in the first set of fragments yields the summary result for the first document.

[0183] Optionally, for different document types, the data input to the large language model can be vectorized. This involves preprocessing the data before inputting it to the large language model, such as vectorized data. For example, semantic analysis can be performed on text, analytical data, and tables to extract semantic features, which are then input to the large language model as vectorized semantic features.

[0184] The document processing method provided in this application constructs a fragment graph network by associating discrete fragments in a first document based on the relationships between multiple segments. Since the fragment graph network indicates the relationships between multiple segments of the first document, a small number of representative fragments are obtained from the fragment graph network to form a first fragment set. This first fragment set can express the content of the first document. Document summarization is then performed on the first fragment set to obtain a summary result of the first document. Therefore, summarizing the document using a small number of representative fragments selected from the fragment graph network ensures the quality of the document summary, reduces the number of fragments input to the model, shortens the processing time for long documents, and improves processing efficiency.

[0185] Optionally, the fragment graph network can be dynamically updated after construction. This new dynamic update mechanism allows the fragment graph network to update in real time as document content is updated, effectively tracking document version changes and historical evolution.

[0186] Knowledge discovery based on fragment graph networks: This involves using fragment graph networks for knowledge discovery, such as identifying key concepts, trends, and patterns in documents through graph mining techniques. This has significant value in fields such as academic research and market analysis.

[0187] This application also provides a document processing device, as shown in FIG8, which includes a communication module 810, a document selection module 820, and a document processing module 830.

[0188] The communication module 810 is used to acquire document processing requests. For example, the communication module 810 is used to execute step 510 in Figure 5 above.

[0189] The document selection module 820 is used to obtain a first set of fragments from a first document according to a document processing request. The first set of fragments comes from a portion of a fragment graph network constructed from multiple fragments of the first document. The fragment graph network is used to indicate the relationships between multiple fragments of the first document. For example, the document selection module 820 is used to perform step 520 in Figure 5 above.

[0190] The document processing module 830 is used to process the first set of fragments using a large language model to obtain a summary result of the first document. For example, the document processing module 830 is used to perform step 530 in Figure 5 above.

[0191] Optionally, the document processing apparatus 800 further includes a storage module 840 and a graph construction module 850. The graph construction module 850 is used to construct a fragment graph network based on the features of multiple fragments of the first document. For example, the communication module 810 is used to perform step 320 in FIG3 above.

[0192] Storage module 840 is used to store fragment graph networks, document fragments, summary results, etc., to facilitate document summarization.

[0193] The document selection module 820, document processing module 830, and graph construction module 850 can all be implemented in software or hardware. For example, the implementation of document selection module 820 will be described below. Similarly, the implementation of document processing module 830 and graph construction module 850 can refer to the implementation of document processing module 830.

[0194] As an example of a software functional unit, the document selection module 820 may include code running on a compute instance. The compute instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the compute instance may be one or more. For example, the document selection module 820 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0195] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0196] As an example of a hardware functional unit, the document selection module 820 may include at least one computing device, such as a server. Alternatively, the document selection module 820 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0197] The document selection module 820 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the document selection module 820 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the document selection module 820 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0198] It should be noted that, in other embodiments, the document selection module 820 can be used to execute any step in the document processing method, and the document processing module 830 can be used to execute any step in the document processing method. The steps implemented by the document processing module 830 and the document selection module 820 can be specified as needed. The document processing module 830 and the document selection module 820 respectively implement different steps in the document processing method to realize all the functions of the document processing device.

[0199] This application also provides a chip system including a processor and a power supply circuit. The power supply circuit supplies power to the processor, which executes the operation steps corresponding to the document processing method. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0200] It should be understood that the document processing apparatus according to this application can correspond to the computing node in the data processing system of the embodiments of this application, and can correspond to the corresponding subject executing the method shown in Figures 3-5 of this application. The above and other operations and / or functions of each module in the document processing apparatus are respectively for implementing the corresponding flow of each method in Figures 3-5. For the sake of brevity, they will not be described in detail here.

[0201] Optionally, the aforementioned document processing apparatus may be located on the same computing device or on different computing devices. This application does not limit the specific implementation method.

[0202] This application also provides a computing device 900. As shown in FIG9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0203] Bus 902 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in Figure 9, but this does not indicate that there is only one bus or one type of bus. Bus 902 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0204] The processor 904 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an accelerator card.

[0205] Memory 906 may include volatile memory, such as random access memory (RAM). Processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 906 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0206] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 906, or two or more types of storage media can be configured to realize the function of memory 906. This application does not limit this.

[0207] The memory 906 stores executable program code, and the processor 904 executes this executable program code to implement the functions of the aforementioned document selection module and document processing module, thereby implementing the document processing method. That is, the memory 906 stores instructions for executing the document processing method. Alternatively, the memory 906 stores executable code, and the processor 904 executes this executable code to implement the functions of the aforementioned execution device, thereby implementing the document processing method. That is, the memory 906 stores instructions for executing the document processing method.

[0208] The communication interface 908 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0209] It should be understood that the computing device 900 according to this application may correspond to the document processing device 800 in this application, and may correspond to the corresponding subject executing the methods shown in Figures 3-5 of this application according to the present application. The above and other operations and / or functions of each module in the document processing device 800 are respectively for implementing the corresponding processes of each method in Figures 3-5. For the sake of brevity, they will not be described in detail here.

[0210] As one possible implementation, the computing device may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the document processing method. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0211] As one possible implementation, the computing device may include multiple processes, including general-purpose processors and AI processors. The general-purpose processors include CPUs, and the AI ​​processors include GPUs, SoCs, DPUs, offloading cards, accelerator cards, and other processors used to perform artificial intelligence-related computing tasks. The multiple processes work together to complete the corresponding flow of the document processing method shown in Figures 3-5. For the sake of brevity, they will not be described in detail here.

[0212] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0213] As shown in Figure 10, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing document processing methods.

[0214] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing document processing methods. In other words, a combination of one or more computing devices 100 can jointly execute instructions for executing document processing methods.

[0215] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the training device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules in the document selection module and the document processing module.

[0216] The computing device 100 may also include a processor 104, a communication interface 108, and a bus 102. For a detailed explanation of the processor 104, the communication interface 108, and the bus 102, please refer to the description of the processor 904, the communication interface 908, and the bus 902 shown in Figure 9.

[0217] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation. As shown in Figure 11, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 106 in computing device 100A stores instructions for executing the functions of the graph construction module. Simultaneously, the memory 106 in computing device 100B stores instructions for executing the functions of the document selection module and the document processing module.

[0218] The connection method between the computing device clusters shown in Figure 11 can be based on the fact that the document processing method provided in this application requires a large amount of data storage and a large amount of computing power. Therefore, it is considered that the functions implemented by the document processing device are performed by the computing device 100B.

[0219] It should be understood that the functions of computing device 100A shown in Figure 11 can also be performed by multiple computing devices 100. Similarly, the functions of computing device 100B can also be performed by multiple computing devices 100.

[0220] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster shown in FIG11. The difference is that the memory 106 of one or more computing devices 100 in this computing device cluster can store the same instructions for executing document processing methods.

[0221] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing document processing methods. In other words, a combination of one or more computing devices 100 can jointly execute instructions for executing document processing methods.

[0222] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions for executing some functions of the data processing system. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more devices in the document processing apparatus.

[0223] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a document processing method.

[0224] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a document processing method.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A document processing method, characterized in that, include: Get the document processing request; According to the document processing request, a first set of fragments of the first document is obtained. The first set of fragments comes from a part of a fragment graph network constructed from multiple fragments of the first document. The fragment graph network is used to indicate the relationship between multiple fragments of the first document. The first set of fragments is processed to obtain the summary result of the first document.

2. The method according to claim 1, characterized in that, The fragment graph network is used to indicate the structure of the first document and the semantic relationships between multiple fragments of the first document in a graph form.

3. The method according to claim 1 or 2, characterized in that, Before obtaining the first set of fragments of the first document according to the document processing request, the method further includes: Obtain multiple fragments from the first document; The fragment graph network is constructed based on the features of multiple fragments of the first document. The nodes in the fragment graph network indicate fragments of the first document, and the edges connecting the nodes in the fragment graph network indicate semantic relationships between fragments.

4. The method according to claim 3, characterized in that, The features of the multiple segments include segment semantic relationships; Constructing the fragment graph network based on features of multiple fragments of the first document includes: Connect the segments that have semantic relationships among the multiple segments to construct the segment graph network.

5. The method according to claim 4, characterized in that, Connecting segments with semantic relationships among the multiple segments includes: The fragment graph network connects text fragments that have semantic relationships among the multiple fragments.

6. The method according to claim 5, characterized in that, The fragments with fragment semantic relationships also include analytical data fragments.

7. The method according to any one of claims 4-6, characterized in that, The features of the multiple fragments also include keywords; Constructing the fragment graph network based on features of multiple fragments of the first document includes: Connect segments with the same keyword from the multiple segments to construct the segment graph network.

8. The method according to claim 7, characterized in that, Connecting segments with the same keyword among the multiple segments includes: The fragment graph network connects table fragments with the same fields from the multiple fragments. The fragment graph network includes table fragments with related relationships.

9. The method according to any one of claims 3-8, characterized in that, After constructing the segment graph network based on the features of multiple segments of the first document, the method further includes: The fragment graph network is divided into multiple sub-networks based on the hierarchical relationship of the document content, and the sub-networks contain content with the same characteristics in the first document; The segment with the most connections in each sub-network is identified as the key segment of that sub-network.

10. The method according to any one of claims 3-8, characterized in that, Obtain multiple fragments of the first document, including: The first document is divided according to the division rules to obtain the multiple segments. The division rules are used to indicate the division of the document based on the document content, document type, or split window.

11. The method according to any one of claims 1-10, characterized in that, The first set of segments includes a key segment and at least one segment associated with the key segment.

12. The method according to claim 11, characterized in that, The fragment graph network includes multiple sub-networks and key fragments of each sub-network; According to the document processing request, the first set of fragments of the first document is obtained, including: The first set of segments is obtained by acquiring key segments from the plurality of sub-networks and at least one segment associated with the key segments.

13. The method according to claim 11, characterized in that, The fragment graph network includes multiple sub-networks and key fragments of each sub-network; the document processing request includes user questions; According to the document processing request, the first set of fragments of the first document is obtained, including: Obtain key segments from at least one subnetwork that are similar to the user's problem; Obtain at least one segment associated with a key segment in the at least one sub-network to obtain the first segment set, the first segment set including the key segment in the at least one sub-network and at least one segment associated with the key segment.

14. The method according to any one of claims 1-13, characterized in that, The first set of fragments is processed to obtain a summary result of the first document, including: The first set of segments is processed to obtain a summary result of at least two segments in the first set of segments; The summary results of at least two segments in the first segment set are combined to obtain the summary result of the first document.

15. A computing device, characterized in that, The computing device includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to perform the operation steps of the method as described in any one of claims 1-14.

16. A computing device cluster, characterized in that, The system includes at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1-14.

Images