Method and system for deep analysis of super large files based on dynamic segmentation and knowledge graph

Abstract

This application relates to a method and system for deep analysis of ultra-large files based on dynamic segmentation and knowledge graphs, belonging to the field of file processing technology. The method includes: receiving a user-triggered processing instruction; obtaining the file to be processed from the processing instruction; segmenting the file into several file segments; matching processing nodes for each file segment based on a preset distributed computing framework; controlling all processing nodes to simultaneously analyze and process the corresponding file segments and generate processing results; merging and outputting all processing results, and making the output content known to the user. This application improves the processing efficiency of ultra-large files.

Description

Technical Field

[0001] This application relates to the field of document processing technology, and in particular to a method and system for deep analysis of ultra-large documents based on dynamic segmentation and knowledge graphs. Background Technology

[0002] With the continuous improvement of informatization, the amount of data generated by various industries is growing exponentially, especially the demand for processing ultra-large files (such as documents, images, and videos exceeding GB in size). Currently, although big data processing technology has made some progress, it still faces many challenges in processing ultra-large files.

[0003] Currently, the main technical approach for processing very large files is based on serial processing, which involves processing the file segment by segment according to its content. Although this method is simple to implement, it is slow, especially when dealing with very large files, where its processing efficiency is low and cannot meet real-time requirements. Summary of the Invention

[0004] To improve the efficiency of processing ultra-large files, this application provides a method and system for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph.

[0005] Firstly, this application provides a method for deep analysis of ultra-large files based on dynamic segmentation and knowledge graphs, including:

[0006] Receive a processing instruction triggered by a user, obtain the file to be processed in the processing instruction, and divide the file to be processed into several file segments;

[0007] Based on a pre-defined distributed computing framework, a processing node is matched for each file segment, and all processing nodes are controlled to simultaneously analyze and process the corresponding file segment and generate processing results.

[0008] All processing results are combined and output, and the user is made aware of the output content.

[0009] By adopting the above technical solution, the ultra-large file is segmented, and then processing nodes are allocated to each file segment to achieve parallel processing, thereby making full use of computing resources to optimize and improve the efficiency of file processing.

[0010] Optionally, dividing the processed file into several file segments includes:

[0011] The file to be processed is divided into several file segments, the content of each file segment is analyzed, and the core content is extracted from the segment content to generate a summary of the corresponding file segment; wherein, the summary content is used for analysis and processing by the corresponding processing node.

[0012] By adopting the above technical solution, in order to avoid the problem of high computational cost caused by directly processing the entire file segment, this application proposes to extract the core content from the file segment and form a summary. In this way, while ensuring that the analysis and processing operations of the processing node can be realized, the amount of data for the analysis and processing of the corresponding file segment by the processing node is further reduced, and the efficiency of file processing is further improved.

[0013] Optionally, the method further includes:

[0014] Analyze the file type of the processed file and match it with the preset file type, then retrieve the entity library corresponding to the successfully matched file type;

[0015] Analyze the paragraph content of each file segment, identify and extract entities from the paragraph content; wherein, the extracted entities are included in the entity library corresponding to the file type that successfully matches the processed file;

[0016] The relationships between the extracted entities are analyzed, and a knowledge graph is constructed for the processed file based on these relationships.

[0017] By adopting the above technical solution, loose text is transformed into entity and relation structures by constructing a knowledge graph, realizing the automated conversion from unstructured text to domain knowledge graph, and upgrading the analysis and review of documents from shallow processing based on string matching to cognitive computing based on semantic networks.

[0018] Optionally, the step of analyzing the paragraph content of each file segment and extracting the core content from the paragraph content to generate a summary of the corresponding file segment includes:

[0019] The content of each file segment is analyzed separately, and each segment content is matched with each review point in the preset review point library. The successfully matched review points are used as the pre-judged review points of the corresponding file segment, and a review set with all corresponding pre-judged review points is generated for each file segment. The processing node is used to process and analyze the corresponding file segment according to the review points. The review points include at least entities and processing and analysis rules for entities.

[0020] Based on the review set of each file segment, a multi-level summary structure is constructed for each file segment. The multi-level summary structure contains review summaries that correspond one-to-one with the predicted review points, and the summary content of each review summary includes at least the entity contained in the corresponding predicted review point.

[0021] By adopting the above technical solution, this application uses the examination points to guide the abstract generation process in reverse. The abstract content extracted in this way is more suitable for the processing needs of the processing node. In addition, considering that the paragraph content of the same file segment may match multiple pre-judgment examination points, this application proposes to generate a corresponding abstract for each pre-judgment examination point, thereby forming a multi-level abstract structure. This is applicable to all possible examination scenarios of the file segment, and is suitable for scenarios where the processing node is a manual examination node. It has the function of supporting interactive switching of abstract levels and reduces the repetitive analysis of the paragraph content of the file segment during the examination.

[0022] Optionally, the method further includes:

[0023] If a pre-judged review point satisfies a preset association condition, and the association condition is that the entity contained in the pre-judged review point exists in more than one file segment, then an association relationship will be established between all review summaries corresponding to all pre-judged review points that satisfy the association condition.

[0024] The actual processing operation of each processing node is monitored in real time to determine the review summary corresponding to the actual processing operation. The priority of review summaries that are related to the review summary corresponding to the actual processing operation in the multi-level summary structure is adjusted, and the following condition is met: the higher the priority of the review summary, the more likely it is to be retrieved and known by the processing node.

[0025] By adopting the above technical solution, the implicit correlation between review summaries is analyzed across file segments to establish a relationship, and the effect of "reviewing one place, responding globally" is achieved based on this relationship. The review tendency is known in real time based on the actual processing operation, and the priority of review summaries is adjusted to improve the efficiency of the processing node in retrieving and knowing the review summaries that need to be processed.

[0026] Optionally, the process of matching processing nodes for each file segment based on a preset distributed computing framework further includes:

[0027] The overlap of the summary content of all review summaries contained in each multi-level summary structure is determined. If there is a target review summary with an overlap lower than a preset overlap, the co-occurrence probability between the target review summaries is determined. If the co-occurrence probability is higher than the preset co-occurrence probability, the corresponding file segment is split according to the target review summary, so that the split segment content corresponding to the target review summary is used as a new file segment; a correspondence is established between the target review summary and the split new file segment; wherein, the co-occurrence probability refers to the probability of being reviewed simultaneously.

[0028] By adopting the above technical solution, the segmentation is adjusted based on a dual dynamic segmentation strategy of overlap and co-occurrence probability, so that the segmentation strategy can be adapted to the needs of document review and processing. At the same time, the segments adjusted in the above way will be processed in parallel in the future. Therefore, the aforementioned adjustment scheme can also indirectly improve the processing and analysis efficiency of review points with high co-occurrence probability.

[0029] Optionally, the process of matching processing nodes for each file segment based on a preset distributed computing framework further includes:

[0030] Extract the differences between the content of each review summary and the content of other review summaries in the same multi-level summary structure;

[0031] Based on the distinguishing content, the corresponding review summary is segmented to generate a simplified information layer and a complete information layer. The simplified information layer contains the corresponding distinguishing content and the identifier of the predicted review point of the corresponding review summary; the complete information layer includes the complete summary content of the corresponding review summary; the simplified information layer is used as the specific content of the review summary within the multi-level summary structure.

[0032] The method further includes:

[0033] When a processing node analyzes and processes a corresponding file segment, it retrieves the complete information layer corresponding to the simplified information layer selected by the processing node for processing and analysis.

[0034] By adopting the above technical solution, the distinguishing content that differentiates the review summary from other review summaries is extracted, and the summary content of the review summary is further segmented. That is, only a simplified information layer is displayed in the multi-level summary structure, and the corresponding complete information layer is retrieved only when it is determined that it needs to be reviewed. This achieves dynamic folding display of the review summary and reduces the amount of information loaded.

[0035] Secondly, this application provides a deep analysis system for ultra-large files based on dynamic segmentation and knowledge graphs, including:

[0036] The file segmentation processing module is used to receive processing instructions triggered by the user, obtain the file to be processed in the processing instructions, and segment the file to be processed into several file segments.

[0037] The segmented parallel analysis module is used to match processing nodes for each file segment based on a preset distributed computing framework, control all processing nodes to analyze and process the corresponding file segment simultaneously, and generate processing results.

[0038] The analysis results output module is used to merge and output all the processing results, and to make the output content known to the user.

[0039] Thirdly, this application provides a device for deep analysis of ultra-large files based on dynamic segmentation and knowledge graphs, including a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any of the first aspects.

[0040] Fourthly, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described in any of the first aspects.

[0041] In summary, this application includes at least one of the following beneficial technical effects:

[0042] 1. In this application, the ultra-large file is segmented and all segments are analyzed in parallel using distributed computing resources to improve the efficiency of analysis and identification of ultra-large files.

[0043] 2. Furthermore, before performing parallel analysis on the segments, natural language processing technology will be applied to generate a summary for each segment. During parallel processing, only the generated summaries need to be analyzed, thereby reducing the amount of data that needs to be analyzed in parallel and further improving the efficiency of analysis and recognition of ultra-large files.

[0044] 3. Furthermore, this application automatically generates a description of the relationships between entities in the document content based on the document content, and further conducts in-depth association analysis through a knowledge graph, realizing the automated conversion from unstructured text to a domain knowledge graph, and upgrading the document analysis and review operation from shallow processing based on string matching to cognitive computing based on semantic networks. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the deep analysis method for ultra-large files based on dynamic segmentation and knowledge graphs disclosed in this application.

[0047] Figure 2 This is a structural block diagram of the ultra-large file deep analysis system based on dynamic segmentation and knowledge graph disclosed in the embodiments of this application.

[0048] Figure labeling: 201, File segmentation processing module; 202, Segmentation parallel analysis module; 203, Analysis result output module. Detailed Implementation

[0049] The following is in conjunction with the appendix Figure 1-2 This application will be described in further detail.

[0050] This application discloses a method for deep analysis of ultra-large files based on dynamic segmentation and knowledge graphs (hereinafter referred to as the analysis method), aiming to improve the efficiency of analyzing ultra-large files. The specific execution entity of the analysis method is an ultra-large file deep analysis system based on dynamic segmentation and knowledge graphs (hereinafter referred to as the analysis system). The following will describe the method in conjunction with the attached... Figure 1 The specific execution flow of the analysis system for the analysis method is described.

[0051] S101 receives a processing instruction triggered by the user, obtains the file to be processed in the processing instruction, and divides the file to be processed into several file segments.

[0052] In implementation, users can access the analysis system via a webpage and upload extremely large files (such as documents, images, and videos exceeding GB size) by clicking preset processing buttons on the system's access interface, thereby triggering processing instructions. The analysis system acquires these instructions and treats the extremely large files within them as the processed files. It then segments these files into several file segments. Specifically, the segmentation can be done by dividing the files into initial segments according to a fixed length (e.g., a fixed number of characters / lines). To avoid disrupting semantic logic, the segments are divided according to a fixed file structure (e.g., chapters / sections / paragraphs). Furthermore, an NLP model can be used to detect topic boundaries and calculate the similarity between the beginning and end of adjacent file segments. Content with similarity higher than a preset similarity score is then merged into the same file segment, thus adjusting the file segment size.

[0053] S102, based on a preset distributed computing framework, matches processing nodes for each file segment, controls all processing nodes to simultaneously analyze and process the corresponding file segment, and generates processing results;

[0054] S103, merge all processing results and output them, so that the user can know the output content.

[0055] In implementation, based on a pre-defined distributed computing framework (such as MapReduce or Spark), the segmented file segments are distributed to different processing nodes, allowing each node to process its corresponding file segment. Correspondingly, the analysis system pre-stores several file types and a pre-defined review list for each file type. Each file type includes filename keywords. The analysis system performs similarity matching between the filename of the file to be processed and the filename keywords contained in all pre-stored file types, selecting the file type with the highest similarity as the file type of the file to be processed, thereby determining the review list corresponding to the file to be reviewed.

[0056] The review list stores the specific review content (i.e., review points) to be followed when reviewing the content of files of the corresponding file type. The review points specifically include the review object (i.e., entity) and the processing and analysis rules for the corresponding review object; for example, if the entity is "date", the corresponding review content is "verify whether the date is within the specified period".

[0057] Specifically, the processing node can be a system that automatically reviews file segments according to the review points in the corresponding review list. Alternatively, the processing node can be a human node, where the analysis system sends the file segments and corresponding review points to an artificial intelligence terminal so that humans can review them according to the review points.

[0058] Finally, the analysis system combines the processing results generated by each processing node into output content, which is displayed on the access page for users to see. The specific processing result here can be the judgment result of whether the review is passed.

[0059] Furthermore, as mentioned above, each processing node will analyze and review the corresponding file segment based on the review points. In order to improve the processing efficiency of the processing nodes, this application will pre-determine all the review contents that may be reviewed for each file segment before the processing nodes perform analysis and review. That is, it establishes the correspondence between file segments and review points (hereinafter referred to as pre-judgment review points), so that subsequent processing nodes can directly analyze and review the corresponding file segment according to the pre-judgment review points, without the processing nodes needing to traverse all the review points in the corresponding review list to perform review and analysis on the corresponding file segment, thereby improving the review efficiency.

[0060] Accordingly, the step of "splitting the file to be processed into several file segments" in S101 specifically includes the following sub-steps:

[0061] The processing file is divided into several file segments, and the content of each file segment is analyzed. Each segment content is matched with each review point in the preset review point library. The successfully matched review points are used as the pre-judged review points of the corresponding file segment, and a review set with all corresponding pre-judged review points is generated for each file segment. The processing node is used to process and analyze the corresponding file segment according to the review points. The review points include at least the entity and the processing and analysis rules for the entity.

[0062] Based on the review set of each file segment, a multi-level summary structure is constructed for each file segment. The multi-level summary structure contains review summaries that correspond one-to-one with the predicted review points, and the summary content of each review summary includes at least the entities contained in the corresponding predicted review point.

[0063] The analysis method also includes the following steps:

[0064] Analyze the file type of the processed file and match it with the preset file types, then retrieve the entity library corresponding to the successfully matched file types;

[0065] Analyze the content of each file segment, identify and extract entities from the content; the extracted entities are included in the entity library corresponding to the file type that successfully matches the processed file.

[0066] The relationships between entities are analyzed and extracted, and a knowledge graph is constructed for the processed documents based on these relationships.

[0067] In implementation, the analysis system pre-stores an entity library for each file type. This library contains entities, which are the sets of entities included in the review points of the corresponding review list. Each entity also corresponds to a relationship graph, which shows the specific representations (such as names) of the same entity within files of the corresponding file type. For example, if the entity is "Date," it can have multiple representations: February 3, 2019, 2019.2.3, 2019-2-3, etc. The relationship graph is used to establish and store the relationships between all representations of the same entity.

[0068] After determining the file type of the file being processed, the analysis system identifies the corresponding entity library. Based on the entities contained in the entity library and the relationship graph corresponding to each entity, it uses a preset entity alignment technique to extract entities that match the entities in the entity library from the specific content (i.e., paragraph content) within each file segment.

[0069] Based on the previous information regarding the review list of the processed documents, the entities extracted from each document segment are compared with the entities contained in each review point in the review list to determine the pre-judgment review points corresponding to each document segment. The pre-judgment review points are the review points where the entities contained in the review list of the processed document and the entities extracted from the document segment have an intersection (i.e., there are common entities). Then, all the pre-judgment review points corresponding to the same document segment are combined into a review set.

[0070] Furthermore, the analysis system generates a multi-level summary structure for each file segment based on the review set corresponding to each file segment. The number of levels in the multi-level summary structure is related to the number of pre-judged review points contained in the review set. Specifically, the multi-level summary structure contains review summaries that correspond one-to-one with the pre-judged review points in the review set. The specific content of the review summary includes: the entity contained in the corresponding pre-judged review point, and the specific paragraph content in the file segment to which the aforementioned entity belongs, used to describe the entity.

[0071] Furthermore, the analysis system also constructs a knowledge graph for the processed files. This knowledge graph contains all file segments, all entities extracted from each segment, and the pre-judgment review points corresponding to each entity. The position information of a file segment within the processed file can be used as the representation of that segment in the knowledge graph. Additionally, each review point in the review list can be pre-numbered, and these numbers can be used as the representation of the pre-judgment review points in the knowledge graph. Furthermore, a relationship graph can be embedded within the knowledge graph, such as connecting entities within the same relationship graph to represent the relationships between entities.

[0072] Optionally, the analysis method may also include the following steps:

[0073] If there are pre-judged review points that meet the preset association conditions, and the association conditions are: the entity contained in the pre-judged review point exists in more than one file segment; then establish an association relationship between all review summaries corresponding to all pre-judged review points that meet the association conditions.

[0074] The system monitors the actual processing operations of each processing node in real time, determines the review summary corresponding to the actual processing operation, and adjusts the priority of review summaries that are related to the review summary corresponding to the actual processing operation in the multi-level summary structure, satisfying the following condition: the higher the priority of the review summary, the more likely it is to be retrieved and known by the processing node.

[0075] In implementation, if there are more than one review summary (hereinafter referred to as associated review summary) that satisfy the following conditions: all associated review summaries correspond to the same pre-judgment review point, and the entities contained in each associated review summary are only a subset of the entities contained in the corresponding pre-judgment review point, and the union of the entities contained in all associated review summaries is consistent with the entities contained in the corresponding pre-judgment review point, and the entities contained in any two associated review summaries have no intersection, then an association relationship is established between the associated review summaries, and the pre-judgment review point corresponding to the associated review summary is considered to satisfy the association condition. In other words, when processing pre-judgment review points that satisfy the association condition, all related associated review summaries need to be processed and analyzed across file segments.

[0076] The analysis system updates the actual processing operations of each processing node on the corresponding file segment in real time. The actual processing operations include the review digests actually selected from the corresponding multi-level summary structure. When any review digest included in the actual processing operation is a related review digest (i.e., it is related to other review digests), the priority of all related review digests in the multi-level summary structure is defined as the highest priority. It should be noted that all review digests in each multi-level summary structure have a preset priority. The priority determines the order of the review digests in the multi-level summary structure. The higher the priority, the earlier the order, and the more likely the review digest will be retrieved and viewed by the processing node when the review node selects a review digest.

[0077] Optionally, the following steps may be included before S102:

[0078] The overlap of the summary content of all review summaries contained in each multi-level summary structure is determined. If there is a target review summary with an overlap lower than the preset overlap, the co-occurrence probability between the target review summaries is determined. If the co-occurrence probability is higher than the preset co-occurrence probability, the corresponding file segment is split according to the target review summary, so that the content of the split segment corresponding to the target review summary is used as a new file segment; a correspondence is established between the target review summary and the split new file segment; where co-occurrence probability refers to the probability of being reviewed at the same time;

[0079] Extract the differences between the content of each review summary and the content of other review summaries in the same multi-level summary structure;

[0080] Based on the differences in content, the corresponding review summary is segmented to generate a simplified information layer and a complete information layer. The simplified information layer contains the corresponding differences in content and the identifier of the predicted review points of the corresponding review summary; the complete information layer contains the complete summary content of the corresponding review summary; the simplified information layer is regarded as the specific content of the review summary within the multi-level summary structure.

[0081] When a processing node analyzes and processes a corresponding file segment, it retrieves the complete information layer corresponding to the simplified information layer selected by the processing node for processing and analysis.

[0082] In implementation, the target review summary refers to the summary content of other review summaries in its multi-level summary structure whose similarity (i.e., overlap) is lower than the preset overlap (i.e., preset similarity). The similarity here can be specifically reflected in the number of overlapping entities contained in the review summary. The fewer the number of overlaps, the lower the corresponding similarity. The co-occurrence frequency refers to the fact that when reviewing the same document, multiple review points that are called form a co-occurrence relationship with each other, and the co-occurrence probability is 100%. The analysis system stores all the predicted review points corresponding to the processed document in the processing instruction each time a processing instruction is received, forming a review record. All predicted review points in the same review record have a co-occurrence relationship with each other. Each time a new review record is stored, the co-occurrence probability between each predicted review point in the historical storage and other review points is automatically updated. For example, the co-occurrence probability of review point A and review point B = (the number of review records stored in the historical period that contain both review point A and review point B) / [(the number of review records stored in the historical period that only contain review point A) + (the number of review records stored in the historical period that only contain review point B) + (the number of review records stored in the historical period that contain both review point A and review point B)].

[0083] The analysis system is used to further analyze the co-occurrence probability of target review summaries whose overlap is lower than a preset overlap. If the co-occurrence probability is higher than the preset co-occurrence probability, the file segment to which the target review summary belongs (hereinafter referred to as the target file segment) is further split. For example, the paragraph content containing the entity contained in the target review summary is separated from the target file segment and formed as a separate file segment, i.e., a new file segment is formed. The paragraph content in the original target file segment, except for the separated new file segment corresponding to the target review summary, is separated as a separate file segment (hereinafter referred to as the remaining file segment). If there is no entity in any entity library in the remaining file segment, the remaining file segment is discarded so that the remaining file segment does not participate in the subsequent processing node matching and processing node analysis processing operations.

[0084] The analysis system is also used to extract distinguishing content for each review summary in each multi-level summary structure. The distinguishing content is the summary content that is different from other review summaries in the same multi-level summary structure (such as entities or processing analysis rules). The specific summary content of each review summary is segmented to obtain a simplified information layer and a complete information layer. Only the simplified information layer of each level of review summary is displayed in the multi-level summary structure. That is, when the processing node retrieves and views the review summary, it only displays the simplified information layer of the corresponding review summary. The analysis system will only retrieve the complete information layer of the selected review summary when the processing node explicitly selects a specific review summary.

[0085] Reference Figure 2 This application discloses a deep analysis system for ultra-large files based on dynamic segmentation and knowledge graphs, comprising:

[0086] The file segmentation processing module 201 is used to receive processing instructions triggered by the user, obtain the file to be processed in the processing instructions, and segment the file into several file segments.

[0087] The segmented parallel analysis module 202 is used to match processing nodes for each file segment based on a preset distributed computing framework, control all processing nodes to analyze and process the corresponding file segments simultaneously, and generate processing results.

[0088] The analysis result output module 203 is used to merge and output all processing results, and to make the output content known to the user.

[0089] Optionally, the file segmentation processing module 201 is also used to divide the processed file into several file segments, analyze the paragraph content of each file segment, and extract the core content from the paragraph content to generate a summary of the corresponding file segment; wherein, the summary content is used for analysis and processing by the corresponding processing node.

[0090] Optionally, it also includes a knowledge graph construction module, which is used to analyze the file type of the processed file and match it with preset file types, and retrieve the entity library corresponding to the successfully matched file types; analyze the paragraph content of each file segment, and identify and extract entities from the paragraph content; wherein, the extracted entities are included in the entity library corresponding to the file type that is successfully matched with the processed file; and is also used to analyze the relationship between the extracted entities, and construct a knowledge graph for the processed file based on the relationship.

[0091] Optionally, the file segmentation processing module 201 is also used to analyze the paragraph content of each file segment separately, match each paragraph content with each review point in the preset review point library, and use the successfully matched review points as the pre-judged review points of the corresponding file segment, generating a review set with all corresponding pre-judged review points for each file segment; the processing node is used to process and analyze the corresponding file segment according to the review points, and the review points include at least entities and processing and analysis rules for entities; based on the review set of each file segment, a multi-level summary structure is constructed for each file segment, and the multi-level summary structure contains review summaries that correspond one-to-one with the pre-judged review points, and the summary content of each review summary includes at least the entities contained in the corresponding pre-judged review point.

[0092] Optionally, it also includes a review summary scheduling module, which is used to establish an association relationship between all review summaries corresponding to all pre-judged review points that meet the preset association conditions if there are pre-judged review points that meet the association conditions, and the association conditions are: the entities contained in the pre-judged review points exist in more than one file segment; then, it will monitor the actual processing operation of each processing node in real time, determine the review summary corresponding to the actual processing operation, adjust the priority of review summaries that are associated with the review summary corresponding to the actual processing operation in the multi-level summary structure, and satisfy: the higher the priority of the review summary, the more likely it is to be retrieved and known by the processing node.

[0093] Optionally, it also includes a file segment adjustment module, used to determine the overlap of the summary content of all review summaries contained in each multi-level summary structure. If there is a target review summary with an overlap lower than a preset overlap, the co-occurrence probability between the target review summaries is determined. If the co-occurrence probability is higher than the preset co-occurrence probability, the corresponding file segment is split according to the target review summary, so that the split paragraph content corresponding to the target review summary is used as a new file segment; a correspondence is established between the target review summary and the split new file segment; wherein, the co-occurrence probability is the probability of being reviewed simultaneously.

[0094] Optionally, the review summary scheduling module is also used to extract the differences between the corresponding summary content and the summary content of other review summaries in the multi-level summary structure for each review summary; based on the differences, the module performs segmentation processing on the corresponding review summary to generate a simplified information layer and a complete information layer. The simplified information layer contains the corresponding differences and the identifier of the predicted review point of the corresponding review summary; the complete information layer includes the complete summary content of the corresponding review summary; the simplified information layer is used as the specific content of the review summary within the multi-level summary structure; and when the processing node analyzes the corresponding file segment, it retrieves the complete information layer corresponding to the simplified information layer selected by the processing node for processing and analysis.

[0095] This application also discloses a device for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph. The device includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executed as described above for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph.

[0096] This application also discloses a computer-readable storage medium that stores a computer program that can be loaded by a processor and executed as described above for a deep analysis method of ultra-large files based on dynamic segmentation and knowledge graphs. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0098] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit the scope of protection of the application. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

Claims

1. A method for deep analysis of a super large file based on dynamic segmentation and a knowledge graph, characterized in that, include: Receive a processing instruction triggered by a user, obtain the file to be processed in the processing instruction, and divide the file to be processed into several file segments; Based on a pre-defined distributed computing framework, a processing node is matched for each file segment, and all processing nodes are controlled to simultaneously analyze and process the corresponding file segment and generate processing results. All processing results are combined and output, and the user is informed of the output content. The step of dividing the processed file into several file segments includes: The file to be processed is divided into several file segments, the content of each file segment is analyzed, and the core content is extracted from the content of the segment to generate a summary of the corresponding file segment; wherein, the summary content is used for analysis and processing by the corresponding processing node; The step of analyzing the paragraph content of each file segment and extracting the core content from the paragraph content to generate a summary of the corresponding file segment includes: The content of each file segment is analyzed separately, and each segment content is matched with each review point in the preset review point library. The successfully matched review points are used as the pre-judged review points of the corresponding file segment, and a review set with all corresponding pre-judged review points is generated for each file segment. The processing node is used to process and analyze the corresponding file segment according to the review points. The review points include at least entities and processing and analysis rules for entities. Based on the review set of each file segment, a multi-level summary structure is constructed for each file segment. The multi-level summary structure contains review summaries that correspond one-to-one with the predicted review points, and the summary content of each review summary includes at least the entity contained in the corresponding predicted review point.

2. The method for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph as described in claim 1, characterized in that, The method further includes: Analyze the file type of the processed file and match it with the preset file type, then retrieve the entity library corresponding to the successfully matched file type; Analyze the paragraph content of each file segment, identify and extract entities from the paragraph content; wherein, the extracted entities are included in the entity library corresponding to the file type that successfully matches the processed file; The relationships between the extracted entities are analyzed, and a knowledge graph is constructed for the processed file based on these relationships.

3. The method for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph as described in claim 1, characterized in that, The method further includes: If a pre-judged review point satisfies a preset association condition, and the association condition is that the entity contained in the pre-judged review point exists in more than one file segment, then an association relationship will be established between all review summaries corresponding to all pre-judged review points that satisfy the association condition. The actual processing operation of each processing node is monitored in real time to determine the review summary corresponding to the actual processing operation. The priority of review summaries that are related to the review summary corresponding to the actual processing operation in the multi-level summary structure is adjusted, and the following condition is met: the higher the priority of the review summary, the more likely it is to be retrieved and known by the processing node.

4. The method for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph as described in claim 3, characterized in that, The aforementioned distributed computing framework, which matches processing nodes for each file segment, also includes: The overlap of the summary content of all review summaries contained in each multi-level summary structure is determined. If there is a target review summary with an overlap lower than a preset overlap, the co-occurrence probability between the target review summaries is determined. If the co-occurrence probability is higher than the preset co-occurrence probability, the corresponding file segment is split according to the target review summary, so that the content of the split segment corresponding to the target review summary is used as a new file segment; a correspondence is established between the target review summary and the split new file segment; wherein, the co-occurrence probability refers to the probability of being reviewed at the same time.

5. The method for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph as described in claim 3, characterized in that, The aforementioned distributed computing framework, which matches processing nodes for each file segment, also includes: Extract the differences between the content of each review summary and the content of other review summaries in the same multi-level summary structure; Based on the distinguishing content, the corresponding review summary is segmented to generate a simplified information layer and a complete information layer. The simplified information layer contains the corresponding distinguishing content and the identifier of the predicted review point of the corresponding review summary; the complete information layer includes the complete summary content of the corresponding review summary; the simplified information layer is used as the specific content of the review summary within the multi-level summary structure. The method further includes: When a processing node analyzes and processes a corresponding file segment, it retrieves the complete information layer corresponding to the simplified information layer selected by the processing node for processing and analysis.

6. A system for deep analysis of ultra-large files based on dynamic segmentation and knowledge graphs, applied to the method for deep analysis of ultra-large files based on dynamic segmentation and knowledge graphs as described in claim 1, characterized in that, The system for deep analysis of ultra-large files based on dynamic segmentation and knowledge graphs includes: The file segmentation processing module is used to receive processing instructions triggered by the user, obtain the file to be processed in the processing instructions, and segment the file to be processed into several file segments. The segmented parallel analysis module is used to match processing nodes for each file segment based on a preset distributed computing framework, control all processing nodes to analyze and process the corresponding file segment simultaneously, and generate processing results. The analysis results output module is used to merge and output all the processing results, and to make the output content known to the user.

7. A device for deep analysis of ultra-large files based on dynamic segmentation and knowledge graph, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 5.

Images