A method and apparatus for segmenting an unstructured file

Abstract

This invention discloses a method and apparatus for segmenting unstructured files. The method includes: acquiring multiple unstructured files; classifying the unstructured files according to their file size to obtain classification results; and segmenting the unstructured files using different segmentation strategies based on the classification results and file types to obtain file segmentation results. By classifying unstructured files and applying different segmentation strategies based on the classification results and file types, unstructured files can be segmented, thereby improving the storage and retrieval efficiency of unstructured files, providing a foundation for subsequent unstructured file analysis and value mining, and reducing the management cost of unstructured files.

Description

Technical Field

[0001] This invention belongs to the technical field of file segmentation, specifically relating to a method and apparatus for segmenting unstructured files. Background Technology

[0002] Unstructured data refers to data with irregular or incomplete structures, lacking a predefined data model, and inconvenient to represent using two-dimensional logical tables in a database. It includes all formats of office documents, text, images, HTML, various reports, images, audio, and video information. Unstructured data has a wide variety of formats and standards, and technically, it is more difficult to standardize and understand than structured information. Therefore, the storage, retrieval, publication, and utilization of unstructured data require more intelligent technologies, such as massive storage, intelligent retrieval, knowledge mining, content protection, and value-added development and utilization of information.

[0003] Current technologies for segmenting large unstructured files do not consider file types within the file, resulting in low segmentation efficiency. They may also group subfiles of different file types into the same segment, increasing the probability of errors during final file reassembly. Patent CN119513057A describes a method and system for parallel synchronization of unstructured files. This method includes intelligently segmenting a target file using a pre-trained file segmentation model, generating a two-level cascaded index number and data hash value. Then, based on a data transmission priority model, the data sub-blocks are transmitted to the target database via an adaptive parallel transmission channel and asynchronously verified. Finally, using a pre-trained file reassembly model, the data sub-blocks are reassembled based on the two-level cascaded index number and data block association graph to generate a synchronized target file, and confirmation information is submitted to the blockchain network. The data sub-blocks are generated by inputting the initial partitioning scheme into a dynamic programming model, constructing a cost function with semantic integrity, data balance, and structural coherence as optimization objectives, inputting the cost function into the dynamic programming model, iteratively calculating the optimal state transition sequence through the dynamic programming model, obtaining the final partitioning scheme, determining the final partitioning boundary position based on the final partitioning scheme, and splitting the target unstructured file according to the final partitioning boundary position to generate multiple data sub-blocks.

[0004] Although dynamic programming models are used in the file segmentation process in related technologies, the segmentation accuracy of dynamic programming models depends on the training dataset and training effect. If there are large differences between the unstructured files in the training dataset and the target files, the final segmentation effect of the data sub-blocks cannot be guaranteed.

[0005] How to improve the efficiency of unstructured file segmentation while ensuring the accuracy of unstructured file segmentation is a problem that needs to be solved. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and apparatus for segmenting unstructured files. The method includes: acquiring multiple unstructured files; classifying the unstructured files according to their file size to obtain classification results; and segmenting the unstructured files based on the classification results and their file types, matching corresponding segmentation strategies to obtain file segmentation results. By classifying unstructured files and matching corresponding segmentation strategies based on the classification results and file types, unstructured files can be segmented, thereby improving the storage and retrieval efficiency of unstructured files, providing a foundation for subsequent unstructured file analysis and value mining, and reducing the management cost of unstructured files.

[0007] In a first aspect, the present invention provides a method for segmenting unstructured files, specifically comprising the following steps:

[0008] Retrieve multiple unstructured files;

[0009] Based on the file size of each unstructured file, the unstructured files are classified to obtain the classification results;

[0010] Based on the classification results, and combined with the file type of each unstructured file, the corresponding segmentation strategy is matched to segment the unstructured file and obtain the file segmentation result.

[0011] Furthermore, the classification results include at least one of the first file set, the second file set, and the third file set;

[0012] Unstructured files are classified based on their file size, resulting in the following classification results:

[0013] If the file size of an unstructured file is less than the first volume threshold, then the unstructured file is added to the first file set.

[0014] If the file size of an unstructured file is greater than or equal to the first volume threshold and less than the second volume threshold, then the unstructured file is added to the second file set.

[0015] If the file size of an unstructured file is greater than or equal to the second volume threshold, then the unstructured file is added to the third file set.

[0016] Furthermore, based on the classification results and the file types of each unstructured file, the corresponding segmentation strategy is matched to segment the unstructured files, resulting in file segmentation results, specifically including:

[0017] For the unstructured files in the first file set, a natural language processing strategy is used for segmentation to obtain the file segmentation results;

[0018] For the unstructured files in the second file set, a compression and segmentation strategy is used to segment them, resulting in file segmentation results;

[0019] If the unstructured files in the third file set are text files, traverse the unstructured files, split the unstructured files based on the punctuation marks in the unstructured files, combine them to obtain multiple sub-segments, and use a dynamic programming strategy to segment them to obtain the file segmentation result;

[0020] If the unstructured files in the third file set are non-text files, determine the number of split points for the unstructured files, and then split the unstructured files based on the number of split points to obtain the file segmentation results.

[0021] Furthermore, natural language processing strategies are employed for segmentation to obtain file segmentation results, specifically including:

[0022] Based on the pre-acquired natural language processing model, semantic analysis is performed on the unstructured files in the first file set to obtain the segmentation identifiers corresponding to the unstructured files;

[0023] Based on the segmentation identifier corresponding to the unstructured file, the unstructured files in the first file set are segmented to obtain the file segmentation result.

[0024] Furthermore, the unstructured file is split based on punctuation marks, resulting in multiple sub-segments. A dynamic programming strategy is then used for further segmentation to obtain the file segmentation result, which specifically includes:

[0025] Based on the punctuation marks in the unstructured file, the unstructured file is split into multiple sub-segments;

[0026] Based on the order of sub-segments, multiple consecutive sub-segments are combined to form candidate sub-texts. Each candidate sub-text consists of n consecutive sub-segments. The text volume of each candidate sub-text is less than a preset sub-text threshold, and the text volume of the candidate sub-text and the (n+1)th sub-segment is greater than the preset sub-text threshold, where n∈N. + ;

[0027] Construct a directed acyclic graph based on the order of candidate subtexts, where each node in the directed acyclic graph includes at least one candidate subtext, and the edges in the directed acyclic graph are the intersection degree of two candidate subtexts.

[0028] Based on the directed acyclic graph, we analyze the forward shortest path from the source to the sink in the directed acyclic graph and determine the forward split point corresponding to the forward shortest path.

[0029] Based on the directed acyclic graph, we analyze the reverse shortest path from the sink to the source in the directed acyclic graph and determine the reverse split point corresponding to the reverse shortest path.

[0030] By comparing the forward shortest path and the reverse shortest path, the target split point is determined from the forward split point and the reverse split point, and the file splitting result is obtained.

[0031] Furthermore, the degree of crossover is specifically expressed as:

[0032] JCD(i-1,i)=JCW(i-1,i) 2

[0033] Where JCD(i-1,i) is the intersection degree between the (i-1)th candidate subtext and the ith candidate subtext in the directed acyclic graph, and JCW(i-1,i) is the total number of characters corresponding to the intersection text between the (i-1)th candidate subtext and the ith candidate subtext in the directed acyclic graph.

[0034] Furthermore, based on the directed acyclic graph (DAG), the forward shortest path from the source to the sink is analyzed, and the forward split point corresponding to the forward shortest path is determined, specifically including:

[0035] Based on the source node of the directed acyclic graph, the next node is determined sequentially according to the intersection degree between the source node and other nodes until the sink node of the directed acyclic graph is reached, thus determining the forward shortest path. Each node in the forward shortest path corresponds to a target subtext.

[0036] The end of the intersection of any two target sub-texts in the forward shortest path is taken as the forward split point.

[0037] Furthermore, based on the directed acyclic graph (DAG), the reverse shortest path from the sink to the source is analyzed, and the reverse split point corresponding to the reverse shortest path is determined, specifically including:

[0038] Based on the sink of the directed acyclic graph, the next node is determined sequentially according to the intersection degree between the sink and other nodes until the source of the directed acyclic graph is reached, and the reverse shortest path is determined. Each node in the reverse shortest path corresponds to a target subtext.

[0039] The end of the intersection of any two target sub-texts in the reverse shortest path is taken as the reverse split point.

[0040] Furthermore, by comparing the forward shortest path and the reverse shortest path, the target split point is determined from the forward and reverse split points to obtain the file segmentation result, which specifically includes:

[0041] Compare the sum of the forward cross degrees corresponding to the forward shortest path with the sum of the reverse cross degrees corresponding to the reverse shortest path;

[0042] If the sum of forward cross-degrees is less than the sum of backward cross-degrees, the forward split point is taken as the target split point, and the unstructured file is split to obtain the file splitting result;

[0043] If the sum of forward cross-degrees is greater than the sum of backward cross-degrees, the backward split point is used as the target split point to split the unstructured file and obtain the file splitting result.

[0044] If the sum of forward cross-degrees equals the sum of backward cross-degrees, the target split point is determined based on the number of forward and backward split points, and the unstructured file is split to obtain the file splitting result.

[0045] Secondly, the present invention also provides an apparatus for segmenting unstructured files, employing the unstructured file segmentation method as described in any of the above-mentioned methods, comprising:

[0046] The data acquisition module is used to acquire multiple unstructured files;

[0047] The file classification module is used to classify unstructured files according to their file size and obtain the classification results.

[0048] The file segmentation module is used to segment unstructured files based on classification results, combined with the file type of each unstructured file, and match the corresponding segmentation strategy to obtain the file segmentation result.

[0049] The present invention provides a method and apparatus for segmenting unstructured files, which has at least the following beneficial effects:

[0050] (1) By classifying unstructured files and matching the classification results with the corresponding segmentation strategy based on file type, the unstructured files can be segmented, thereby improving the storage and retrieval efficiency of unstructured files, providing a foundation for subsequent unstructured file analysis and value mining, and reducing the management cost of unstructured files.

[0051] (2) By using a compression and segmentation strategy that first compresses and then segments unstructured files, the file size of unstructured files can be significantly reduced, the storage, transmission and processing efficiency of unstructured files can be improved, and segmentation errors and computational resource consumption can be reduced. Attached Figure Description

[0052] Figure 1 A flowchart of a method for segmenting unstructured files provided in an embodiment of the present invention;

[0053] Figure 2 A flowchart for determining the classification result provided in an embodiment of the present invention;

[0054] Figure 3 A flowchart illustrating the matching segmentation strategy provided in this embodiment of the invention;

[0055] Figure 4 A flowchart illustrating the segmentation using a dynamic programming strategy provided in this embodiment of the invention;

[0056] Figure 5 A schematic diagram illustrating the determination of candidate subtexts provided in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram illustrating the construction of a directed acyclic graph according to an embodiment of the present invention;

[0058] Figure 7 A flowchart for determining file segmentation results provided in an embodiment of the present invention;

[0059] Figure 8 This is a structural block diagram of an unstructured file segmentation device provided in an embodiment of the present invention.

[0060] Among them, 201 is the data acquisition module; 202 is the file classification module; and 203 is the file segmentation module. Detailed Implementation

[0061] To better understand the above technical solutions, a detailed description of the solutions will be provided below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0062] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0063] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0064] Unstructured files have a rich variety of data types, diverse formats, and no fixed format. At the same time, unstructured files are generated and changed rapidly, their content is complex, unpredictable, and involves massive amounts of data, making them difficult to manage.

[0065] In practical applications of unstructured files, due to the dispersed data storage of unstructured files, it is difficult to classify and manage them uniformly. Currently, the common practice for segmenting unstructured files is to segment them according to a fixed value, such as segmenting them according to a fixed file size or a fixed number of lines. This method has low segmentation efficiency and low segmentation accuracy.

[0066] With the acceleration of digitalization, unstructured data accounts for an increasingly larger proportion of enterprise data assets, typically exceeding 80% of the total data volume. Therefore, how to efficiently and accurately segment and manage unstructured data has become a key issue in data governance.

[0067] This invention provides a method for segmenting unstructured files. The method includes: acquiring multiple unstructured files; classifying the unstructured files according to their file size to obtain classification results; and segmenting the unstructured files based on the classification results and their file types, matching corresponding segmentation strategies to obtain file segmentation results. By classifying unstructured files and matching corresponding segmentation strategies based on the classification results and file types, the method achieves file segmentation, thereby improving the storage and retrieval efficiency of unstructured files, providing a foundation for subsequent unstructured file analysis and value mining, and reducing the management cost of unstructured files.

[0068] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for segmenting unstructured files, the specific steps of which are as follows:

[0069] S101: Obtain multiple unstructured files.

[0070] S102: Classify unstructured files according to their file size to obtain classification results.

[0071] Specifically, file size refers to the size of an unstructured file, that is, a measure of the storage space occupied by an unstructured file. Based on the determination of the file size of unstructured files, unstructured files of different sizes are classified into three categories: the first file set, the second file set, and the third file set.

[0072] Furthermore, referring to Figure 2 If the file size of an unstructured file is less than the first volume threshold, then the unstructured file is added to the first file set.

[0073] If the file size of an unstructured file is greater than or equal to the first volume threshold and less than the second volume threshold, then the unstructured file is added to the second file set.

[0074] If the file size of an unstructured file is greater than or equal to the second volume threshold, then the unstructured file is added to the third file set.

[0075] In one specific example, the first volume threshold is 2GB and the second volume threshold is 500GB. In other implementations, the first and second volume thresholds can be set according to the actual partitioning requirements, and there is no limitation on this.

[0076] By setting a first volume threshold and a second volume threshold, unstructured files are classified and added to different file sets. For unstructured files in different file sets, corresponding segmentation strategies are matched to improve the segmentation accuracy of unstructured files.

[0077] It is understandable that multiple unstructured files may all be smaller than the first volume threshold, in which case the first file set is a non-empty set, while the second and third file sets are empty sets. Similarly, it is also possible that only the second or third file set exists. Therefore, the first, second, and third file sets can exist individually or all of them. Unstructured files in the first, second, and third file sets can be processed simultaneously, or unstructured files in a single file set can be processed; there is no limitation on this.

[0078] S103: Based on the classification results, combined with the file types of each unstructured file, the corresponding segmentation strategy is matched to segment the unstructured file and obtain the file segmentation result.

[0079] Specifically, refer to Figure 3 For the unstructured files in the first file set, a natural language processing strategy is used for segmentation to obtain the file segmentation results;

[0080] For the unstructured files in the second file set, a compression and segmentation strategy is used to segment them, resulting in file segmentation results;

[0081] If the file type of the unstructured files in the third file set is non-text files, determine the number of split points for the unstructured files, and split the unstructured files based on the number of split points to obtain the file segmentation results;

[0082] If the unstructured files in the third file set are text files, traverse the unstructured files, split the unstructured files based on the punctuation marks in the unstructured files, combine them to obtain multiple sub-segments, and use a dynamic programming strategy to segment them to obtain the file segmentation result.

[0083] Based on the foregoing description of the first, second, and third file sets, each file set can exist individually or all at once. Regardless of whether all three file sets exist, the processing of unstructured files within each file set can be performed simultaneously or sequentially, or unstructured files within a single file set can be processed; there is no specific order of steps.

[0084] In the embodiments provided by this invention, unstructured files in the first file set are segmented using a natural language processing (NLP) strategy. Specifically, NLP techniques are used to train the model, obtaining segmentation identifiers for the unstructured files in the first file set, thereby completing the segmentation of the unstructured files in the first file set. During model training, the training text in the training dataset is first cleaned to remove noise data such as advertisements, headers, and footers, and to unify the encoding format, providing clean data for subsequent processing. Then, NLP libraries (such as SpaCy and NLTK) are used to segment the text into sentences, decomposing the text into individual sentences. Each sentence is then converted into an embedding vector, enabling the model to understand the meaning of the sentences. By calculating the semantic similarity between sentences, it is determined which sentences are semantically related, providing a basis for subsequent block segmentation. Semantically related sentences are grouped together to obtain text blocks, and the size of the segmented blocks is adjusted and controlled according to specific needs and model limitations to ensure that the block size is suitable for subsequent processing and analysis. By analyzing the text blocks or individual sentences, segmentation identifiers for each training text can be obtained. The segmentation of the training text is then completed using these segmentation identifiers. In a specific example, natural language processing (NLP) can be used to understand and analyze the unstructured files in the first document set to obtain segmentation identifiers. Then, the segmentation identifiers in the corresponding file content of the unstructured files in the first document set can be identified and segmented. For example, the segmentation identifiers can be "Thank you for watching," "Yours sincerely," or they can be delimiters in the file content.

[0085] Furthermore, a compression segmentation strategy is employed to segment the file, resulting in the following file segmentation results:

[0086] The unstructured files in the second file set are first compressed to obtain compressed files. Then, based on a preset compressed file size threshold, the file size of the compressed file is divided by the threshold to determine the number of compression split points for each unstructured file. After determining the number of compression split points, the compressed file is divided into equal-sized sub-files to obtain the file segmentation result. For example, if the compressed file size is 10GB and the preset compressed file size threshold is 2GB, then the number of compression split points is 10 / 2 = 5, and the compressed file is divided into 2GB sub-files to obtain the file segmentation result.

[0087] By employing a compression and segmentation strategy that first compresses and then segments unstructured files, the file size of unstructured files can be significantly reduced, improving the efficiency of storage, transmission, and processing of unstructured files, while also reducing segmentation errors and computational resource consumption.

[0088] Furthermore, the number of segmentation points in the unstructured file is determined, and the unstructured file is segmented based on the number of segmentation points to obtain the file segmentation result, specifically including:

[0089] First, the file sizes of non-text files in the third file set are obtained. Then, based on a preset non-text file size threshold, the file sizes of the non-text files are divided by the threshold to determine the number of segmentation points for the unstructured files. After determining the number of segmentation points, the non-text files are divided into equal-volume subfiles to obtain the file segmentation result. For example, if the size of a non-text file is 200GB and the preset non-text file size threshold is 20GB, then the number of segmentation points is 200 / 20 = 10. Therefore, the non-text file is divided into 20GB subfiles, resulting in the file segmentation result. In the embodiments provided by this invention, the segmentation of non-text files can be completed using a media segmenter. In other embodiments, the segmentation of non-text files can also be performed using the aforementioned natural language processing strategy, that is, by analyzing the non-text files to obtain corresponding segmentation identifiers, and then performing the segmentation of the non-text files based on these identifiers to obtain the file segmentation result.

[0090] Furthermore, referring to Figure 4 The file is segmented using a dynamic programming strategy, resulting in the following file segmentation results:

[0091] Based on the punctuation marks in the unstructured file, the unstructured file is split into multiple sub-segments;

[0092] Based on the order of sub-segments, multiple consecutive sub-segments are combined to form candidate sub-texts. Each candidate sub-text consists of n consecutive sub-segments. The text volume of each candidate sub-text is less than a preset sub-text threshold, and the text volume of the candidate sub-text and the (n+1)th sub-segment is greater than or equal to the preset sub-text threshold, where n∈N.+ ;

[0093] Construct a directed acyclic graph based on the order of candidate subtexts, where each node in the directed acyclic graph includes at least one candidate subtext, and the edges in the directed acyclic graph are the intersection degree of two candidate subtexts.

[0094] Based on the directed acyclic graph, we analyze the forward shortest path from the source to the sink in the directed acyclic graph and determine the forward split point corresponding to the forward shortest path.

[0095] Based on the directed acyclic graph, we analyze the reverse shortest path from the sink to the source in the directed acyclic graph and determine the reverse split point corresponding to the reverse shortest path.

[0096] By comparing the forward shortest path and the reverse shortest path, the target split point is determined from the forward split point and the reverse split point, and the file splitting result is obtained.

[0097] In one specific implementation, refer to Figure 5 Based on the punctuation marks in the unstructured file, the unstructured file is split into multiple sub-segments. These sub-segments are then numbered sequentially as a1, a2, a3, a4, a5, ..., am. Next, based on a preset sub-text threshold, the sub-segments are recombined to obtain multiple candidate sub-texts, which are then labeled sequentially as b1, b2, b3, b4, b5, ..., bq. In a specific example, for candidate sub-text b1, b1 is composed of a1, a2, and a3, and the text volume of a1, a2, and a3 is less than the preset sub-text threshold, while the text volume of a1, a2, a3, and a4 is greater than or equal to the preset sub-text threshold. That is, for candidate sub-text b1, n is 3. Candidate subtext b1 has only one possible combination. For other candidate subtexts, there may be multiple possible combinations. Taking candidate subtext b2 as an example, it can be composed of a2, a3, and a4; it can be composed of a3 and a4; or it can be composed of a4 and a5. It is understandable that in the above example, the combination methods of candidate subtext b2 all satisfy the constraints regarding the subtext threshold. Through the combination methods of candidate subtext b2, it can be understood that other candidate subtexts may also include multiple combination methods. After completing the combination of all candidate subtexts, a directed acyclic graph is constructed based on all candidate subtexts. (Refer to...) Figure 6In a directed acyclic graph (DAG), the source vertex is candidate sub-text b1, which is a node with an in-degree of zero. Nodes connected to candidate sub-text b1 are different candidate sub-texts b2. The edge between b1 and b2 is the intersection degree of b1 and b2, which is the square of the total number of characters in the intersection text between b1 and b2. For example, if b1 consists of a1, a2, and a3, and b2 consists of a2, a3, and a4, then the intersection texts of b1 and b2 are a2 and a3, and the square of the total number of characters in a2 and a3 is the intersection degree between b1 and the corresponding b2. Similarly, after constructing other candidate sub-texts, the DAG is completed, and the sink vertex in the DAG is candidate sub-text bq, which is a node with an out-degree of zero.

[0098] Furthermore, the degree of crossover is specifically expressed as:

[0099] JCD(i-1,i)=JCW(i-1,i) 2

[0100] Where JCD(i-1,i) is the intersection degree between the (i-1)th candidate subtext and the ith candidate subtext in the directed acyclic graph, and JCW(i-1,i) is the total number of characters corresponding to the intersection text between the (i-1)th candidate subtext and the ith candidate subtext in the directed acyclic graph.

[0101] Furthermore, the positive segmentation point is determined, specifically including:

[0102] Based on the source node of the directed acyclic graph, the next node is determined sequentially according to the intersection degree between the source node and other nodes until the sink node of the directed acyclic graph is reached, thus determining the forward shortest path. Each node in the forward shortest path corresponds to a target subtext.

[0103] The end of the intersection of any two target sub-texts in the forward shortest path is taken as the forward split point.

[0104] In one specific implementation, each layer of the directed acyclic graph contains multiple nodes, and the nodes in each layer represent multiple possibilities corresponding to a candidate subtext. The shortest path between candidate node u and candidate node v is calculated recursively, using the following formula:

[0105] F u,v =F u,v-1 +minJCD(v-1,v),u <v-1

[0106] Where u and v are the candidate sub-text numbers, F u,v F is the shortest path between target subtext u and target subtext v. u,v-1 JCD(v-1,v) represents the shortest path between target subtext u and target subtext v-1, and JCD(v-1,v) represents the minimum intersection between target subtext v-1 and candidate subtext v.

[0107] Similarly, the shortest path between candidate subtext b1 and candidate subtext bq is calculated, which is the forward shortest path, and the end of the intersection of any two target subtexts in the forward shortest path is taken as the forward split point.

[0108] In a specific example, to calculate F b1,bq Then it is necessary to calculate F. b1,b(q-1) Based on recursive calculation, F needs to be calculated first. b1,b2 It can be understood that there are multiple possibilities for candidate subtext b2. According to minJCD(b1,b2), the target subtext b2 with the minimum cross-degree can be determined from multiple candidate subtexts b2. Target subtexts b3, ..., target subtext bq are determined sequentially to obtain the forward shortest path. Based on the cross-text situation among each adjacent target subtext in the forward shortest path, the end of each cross-text is marked as a forward split point.

[0109] Further, the reverse segmentation point is determined, specifically including:

[0110] Based on the sink of the directed acyclic graph, the next node is determined sequentially according to the intersection degree between the sink and other nodes until the source of the directed acyclic graph is reached, and the reverse shortest path is determined. Each node in the reverse shortest path corresponds to a target subtext.

[0111] The end of the intersection of any two target sub-texts in the reverse shortest path is taken as the reverse split point.

[0112] In one specific implementation, a virtual node N is constructed after the node corresponding to the candidate sub-text bq. The distance between the virtual node N and the last candidate sub-text bq is calculated, and the candidate sub-text bq with the shortest distance to the virtual node N is taken as the target sub-text. Then, the shortest path between the virtual node N and the candidate sub-text b1 is calculated and taken as the reverse shortest path. At the same time, the end of the intersection text of any two target sub-texts in the reverse shortest path is taken as the reverse split point.

[0113] Understandably, the methods for determining the reverse shortest path and the forward shortest path are the same; the difference lies in the starting and ending points of the two candidate text paths.

[0114] In a specific example, to calculate F N,b1 Then it is necessary to calculate F. N,b2 Based on recursive calculation, F needs to be calculated first. N,bqIt can be understood that there are multiple possibilities for the candidate subtext bq. According to minJCD(N,bq), the target subtext bq with the minimum cross-degree can be determined from multiple candidate subtexts bq. The target subtexts bq-1, ..., target subtext b1 are determined in sequence to obtain the reverse shortest path. Based on the cross-text situation of each adjacent target subtext in the reverse shortest path, the end of each cross-text is marked as the reverse split point.

[0115] Furthermore, referring to Figure 7 Determine the file segmentation result, specifically including:

[0116] Compare the sum of the forward cross degrees corresponding to the forward shortest path with the sum of the reverse cross degrees corresponding to the reverse shortest path;

[0117] If the sum of forward cross-degrees is less than the sum of backward cross-degrees, the forward split point is taken as the target split point, and the unstructured file is split to obtain the file splitting result;

[0118] If the sum of forward cross-degrees is greater than the sum of backward cross-degrees, the backward split point is used as the target split point to split the unstructured file and obtain the file splitting result.

[0119] If the sum of forward cross-degrees equals the sum of backward cross-degrees, the target split point is determined based on the number of forward and backward split points, and the unstructured file is split to obtain the file splitting result.

[0120] In one possible implementation, the cross-degrees along the forward shortest path are summed to obtain the forward cross-degree sum, and the cross-degrees along the reverse shortest path are summed to obtain the reverse cross-degree sum. The forward cross-degree sum and the reverse cross-degree sum are compared. If the forward cross-degree sum is less than the reverse cross-degree sum, the forward segmentation point is used as the target segmentation point to segment the unstructured file, resulting in a file segmentation result. If the forward cross-degree sum is greater than the reverse cross-degree sum, the reverse segmentation point is used as the target segmentation point to segment the unstructured file, resulting in a file segmentation result. The smaller the cross-degree sum, the fewer the intersecting texts in the corresponding shortest path, resulting in less repetition in the file segmentation result. If the forward and reverse cross-degree sums are the same, the number of forward and reverse segmentation points is further compared, and the point with fewer points is used as the target segmentation point to segment the unstructured file, resulting in a file segmentation result. Selecting points with fewer segmentation points while maintaining segmentation effectiveness can improve the efficiency of unstructured text segmentation. If the number of forward and reverse segmentation points is the same, either one can be chosen as the target segmentation point to complete the segmentation of the unstructured text.

[0121] In a specific example, an empty dictionary DG is first created to represent the directed graph. The number of candidate subtexts is determined by obtaining the length N0 of the candidate subtext list. For each candidate subtext index k, directed edges are generated from k to other candidate subtexts. Specifically, for each k, an edge is created for all indices from k to k+1 to min(alls[k][-1]+1, N0). If there are no indices in this range, only an edge from k to k+1 is created. Here, alls[k][-1] is the index of the last segment of the k-th candidate subtext, alls[k][-1]+1 is the index of the next possible segment of the k-th candidate subtext, and min(alls[k][-1]+1, N0) is the smaller value selected from alls[k][-1]+1 and N0, ensuring that alls[k][-1]+1 does not exceed the total number of candidate subtexts N0.

[0122] Then, N is treated as a virtual node representing the destination. The path dictionary `routes` is initialized, with `routes[N]` set to (0, -1), indicating that the path length from the virtual node N to itself is 0 and there is no predecessor node. Starting from candidate subtext index N-1, the path is traversed in reverse order to index 0. The neighbors of each node are traversed: for the current node i, all its neighbor nodes j in the directed graph DG are traversed, and the path weight from node i to node j is calculated. JCD(i,j) is the intersection degree between the i-th and j-th candidate subtexts in the directed acyclic graph. The total path length is calculated and the shortest path is recorded: for each node i, among all possible neighbor nodes j, the node with the shortest path length is selected, and the path length and the corresponding predecessor node j are recorded in `routes[i]`. Finally, the dynamic programming table `routes` is obtained. Each key-value pair in the `routes` dictionary represents the shortest path length from that node to the destination and its corresponding predecessor node. By backtracking the path, we can obtain the shortest path from the starting point to the ending point by starting from the starting node and backtracking according to the predecessor nodes recorded in routes.

[0123] The forward split point can be obtained through the routes dictionary, and the reverse split point can be obtained by tracing back the path. The forward and reverse split points are then compared to determine the target split point and complete the split.

[0124] Methods for segmenting unstructured files also include:

[0125] Obtain the total time for splitting unstructured files and the file size of unstructured files;

[0126] Divide the file size of the unstructured file by the total segmentation time to obtain the segmentation rate;

[0127] The average segmentation rate is obtained by summing the segmentation rates of all unstructured texts and taking the average value.

[0128] Reference Figure 8 This invention provides an apparatus for segmenting unstructured files, comprising:

[0129] Data acquisition module 201 is used to acquire multiple unstructured files;

[0130] The file classification module 202 is used to classify unstructured files according to their file size and obtain classification results.

[0131] The file segmentation module 203 is used to segment unstructured files based on the classification results, combined with the file types of each unstructured file, and matching the corresponding segmentation strategy to obtain the file segmentation results.

[0132] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0133] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method of segmenting an unstructured file, characterized by, include: Retrieve multiple unstructured files; Based on the file size of each unstructured file, the unstructured files are classified to obtain the classification results. Specifically, if the file size of an unstructured file is less than a first volume threshold, the unstructured file is added to a first file set; if the file size of an unstructured file is greater than or equal to the first volume threshold and less than a second volume threshold, the unstructured file is added to a second file set; if the file size of an unstructured file is greater than or equal to the second volume threshold, the unstructured file is added to a third file set. The classification results include at least one of the first, second, and third file sets. Based on the classification results and the file types of each unstructured file, corresponding segmentation strategies are matched to segment the unstructured files, resulting in file segmentation results. Specifically, this includes: for unstructured files in the first file set, natural language processing strategies are used for segmentation; for unstructured files in the second file set, compression segmentation strategies are used for segmentation; if the unstructured files in the third file set are text files, the unstructured files are traversed, and the files are split based on punctuation marks. Multiple sub-segments are obtained, and dynamic programming strategies are used for segmentation to obtain file segmentation results; if the unstructured files in the third file set are non-text files, the number of segmentation points for each unstructured file is determined, and the files are segmented based on this number of segmentation points to obtain file segmentation results.

2. The method for segmenting unstructured files as described in claim 1, characterized in that, Natural language processing strategies are used for file segmentation to obtain the file segmentation results, which include: Based on the pre-acquired natural language processing model, semantic analysis is performed on the unstructured files in the first file set to obtain the segmentation identifiers corresponding to the unstructured files; Based on the segmentation identifier corresponding to the unstructured file, the unstructured files in the first file set are segmented to obtain the file segmentation result.

3. The method for segmenting unstructured files as described in claim 1, characterized in that, Unstructured files are split based on punctuation marks, resulting in multiple sub-segments. A dynamic programming strategy is then used for further segmentation to obtain the file segmentation results, which include: Based on the punctuation marks in the unstructured file, the unstructured file is split into multiple sub-segments; Based on the order of sub-segments, multiple consecutive sub-segments are combined to form candidate sub-texts. Each candidate sub-text consists of n consecutive sub-segments. The text volume of each candidate sub-text is less than a preset sub-text threshold, and the text volume of the candidate sub-text and the (n+1)th sub-segment is greater than the preset sub-text threshold. ; Construct a directed acyclic graph based on the order of candidate subtexts, where each node in the directed acyclic graph includes at least one candidate subtext, and the edges in the directed acyclic graph are the intersection degree of two candidate subtexts. Based on the directed acyclic graph, we analyze the forward shortest path from the source to the sink in the directed acyclic graph and determine the forward split point corresponding to the forward shortest path. Based on the directed acyclic graph, we analyze the reverse shortest path from the sink to the source in the directed acyclic graph and determine the reverse split point corresponding to the reverse shortest path. By comparing the forward shortest path and the reverse shortest path, the target split point is determined from the forward split point and the reverse split point, and the file splitting result is obtained.

4. The method for segmenting unstructured files as described in claim 3, characterized in that, Crossover is specifically expressed as: ； in, Let be the intersection degree between the (i-1)th candidate subtext and the ith candidate subtext in the directed acyclic graph. Let be the total number of characters corresponding to the intersection text of the (i-1)th candidate subtext and the ith candidate subtext in the directed acyclic graph.

5. The method for segmenting unstructured files as described in claim 3, characterized in that, Based on a directed acyclic graph (DAG), we analyze the forward shortest path from the source to the sink and determine the forward split point corresponding to the forward shortest path. Specifically, this includes: Based on the source node of the directed acyclic graph, the next node is determined sequentially according to the intersection degree between the source node and other nodes until the sink node of the directed acyclic graph is reached, thus determining the forward shortest path. Each node in the forward shortest path corresponds to a target subtext. The end of the intersection of any two target sub-texts in the forward shortest path is taken as the forward split point.

6. The method for segmenting unstructured files as described in claim 3, characterized in that, Based on a directed acyclic graph (DAG), we analyze the reverse shortest path from the sink to the source and determine the reverse split point corresponding to the reverse shortest path. Specifically, this includes: Based on the sink of the directed acyclic graph, the next node is determined sequentially according to the intersection degree between the sink and other nodes until the source of the directed acyclic graph is reached, and the reverse shortest path is determined. Each node in the reverse shortest path corresponds to a target subtext. The end of the intersection of any two target sub-texts in the reverse shortest path is taken as the reverse split point.

7. The method for segmenting unstructured files as described in claim 3, characterized in that, Compare the forward shortest path and the reverse shortest path, determine the target split point from the forward split point and the reverse split point, and obtain the file splitting result, which specifically includes: Compare the sum of the forward cross degrees corresponding to the forward shortest path with the sum of the reverse cross degrees corresponding to the reverse shortest path; If the sum of forward cross-degrees is less than the sum of backward cross-degrees, the forward split point is taken as the target split point, and the unstructured file is split to obtain the file splitting result; If the sum of forward cross-degrees is greater than the sum of backward cross-degrees, the backward split point is used as the target split point to split the unstructured file and obtain the file splitting result. If the sum of forward cross-degrees equals the sum of backward cross-degrees, the target split point is determined based on the number of forward and backward split points, and the unstructured file is split to obtain the file splitting result.

8. A device for segmenting unstructured documents, characterized in that, The method for segmenting unstructured files as described in any one of claims 1-7 includes: The data acquisition module is used to acquire multiple unstructured files; The file classification module is used to classify unstructured files according to their file size and obtain classification results. Specifically, if the file size of an unstructured file is less than a first volume threshold, the unstructured file is added to a first file set; if the file size of an unstructured file is greater than or equal to the first volume threshold and less than a second volume threshold, the unstructured file is added to a second file set; if the file size of an unstructured file is greater than or equal to the second volume threshold, the unstructured file is added to a third file set. The classification result includes at least one of the first, second, and third file sets. The file segmentation module is used to segment unstructured files based on classification results and the file type of each file, matching the corresponding segmentation strategy to obtain file segmentation results. Specifically, it includes: segmenting unstructured files in the first file set using a natural language processing strategy; segmenting unstructured files in the second file set using a compression segmentation strategy; if the unstructured files in the third file set are text files, it iterates through the unstructured files, splits them based on punctuation marks, combines the resulting sub-segments, and uses a dynamic programming strategy for segmentation; if the unstructured files in the third file set are non-text files, it determines the number of segmentation points for each unstructured file and segments them based on these points to obtain file segmentation results.

Images