Knowledge file segmentation processing method and device, storage medium and processor

Abstract

The invention discloses a knowledge file segmentation processing method and device, a storage medium and a processor. In the method, based on the data volume of a to-be-processed knowledge file, the to-be-processed knowledge file is segmented into a plurality of knowledge file blocks, and the plurality of knowledge file blocks obtained through segmentation are cached to a pre-created knowledge file block loading library; loading a plurality of knowledge file blocks from a knowledge file block loading library, and constructing file block segmentation tasks corresponding to the knowledge file blocks according to the segmentation sequence of the plurality of knowledge file blocks; and adopting a plurality of threads to asynchronously execute the plurality of file block segmentation tasks in batches to segment the plurality of knowledge file blocks until the segmentation tasks are executed to obtain a knowledge document set corresponding to the to-be-processed knowledge file. Repeated segmentation of the same to-be-processed knowledge file in the first stage is avoided in a mode of caching the knowledge file blocks; and a plurality of threads are adopted to asynchronously execute the file block segmentation task in batches, so that the segmentation speed and the segmentation efficiency of the knowledge file can be improved on the whole.

Description

Technical Field

[0001] The present application relates to the field of file segmentation technology, and in particular to a knowledge file segmentation processing method, device, storage medium and processor. Background Art

[0002] With the development of information technology, knowledge management has become the key to improving work efficiency, increasing productivity and promoting innovation in various enterprises, groups and organizations. When automatically executing knowledge management, it is often necessary to process knowledge files. However, since some knowledge files are relatively large, the complexity of a single processing task is increased, resulting in low processing speed and low processing efficiency. How to improve the processing efficiency of knowledge files and thus improve the effectiveness of knowledge management is a technical problem that urgently needs to be solved in this field. Summary of the invention

[0003] Based on the above problems, the present application provides a knowledge file segmentation processing method, device, storage medium and processor, the purpose of which is to improve the processing efficiency of knowledge files and facilitate knowledge management.

[0004] The embodiments of the present application disclose the following technical solutions:

[0005] The first aspect of the present application provides a knowledge file segmentation processing method, the method comprising:

[0006] Based on the data volume of the knowledge file to be processed, the knowledge file to be processed is divided into a plurality of knowledge file blocks, and the plurality of knowledge file blocks obtained by the division are cached in a pre-created knowledge file block loading library;

[0007] Loading the plurality of knowledge file blocks from the knowledge file block loading library, and constructing a plurality of file block segmentation tasks corresponding to the plurality of knowledge file blocks one by one according to the segmentation order of the plurality of knowledge file blocks;

[0008] The multiple knowledge file blocks are segmented by using multiple threads to asynchronously execute the multiple file block segmentation tasks in batches until the multiple file block segmentation tasks are completed, thereby obtaining a knowledge document set corresponding to the knowledge file to be processed; the knowledge document set contains multiple knowledge documents, and each knowledge document serves as the segmentation result of the corresponding knowledge file block.

[0009] In an optional implementation, the step of using multiple threads to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks includes:

[0010] Start multiple threads, and use the knowledge segmentation strategy corresponding to the knowledge file to be processed to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks; the knowledge segmentation strategy is configured according to the file type and / or business requirements of the knowledge file to be processed.

[0011] In an optional implementation, the knowledge file segmentation processing method further includes:

[0012] Reconfigure the knowledge segmentation strategy corresponding to the knowledge file to be processed in response to changes in the business requirements of the file to be processed;

[0013] Load the plurality of knowledge file blocks from the knowledge file block loading library;

[0014] Start multiple threads, and use the knowledge segmentation strategy newly configured for the knowledge file to be processed to asynchronously execute the multiple file block segmentation tasks in batches to re-segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining the latest knowledge document set corresponding to the knowledge file to be processed.

[0015] In an optional implementation, the knowledge file segmentation processing method further includes:

[0016] According to the division order of the plurality of knowledge file blocks, different identification information is assigned to the plurality of knowledge file blocks to distinguish different knowledge file blocks;

[0017] Constructing an association relationship between the identification information of each of the plurality of knowledge file blocks and the knowledge file to be processed;

[0018] The step of obtaining a knowledge document set corresponding to the knowledge file to be processed includes:

[0019] Associating the knowledge documents obtained by segmentation with the identification information of the corresponding knowledge file blocks;

[0020] According to the association between the identification information of each of the plurality of knowledge file blocks and the knowledge file to be processed, and the association between the knowledge document and the identification information, each knowledge document is integrated into the knowledge document set corresponding to the knowledge file to be processed in the segmentation order.

[0021] In an optional implementation, the thread uses the knowledge segmentation strategy corresponding to the knowledge file to be processed to perform the file block segmentation task to segment the knowledge file block, including:

[0022] The thread converts the knowledge file block corresponding to the executed file block segmentation task into text content;

[0023] The text content is segmented using the knowledge segmentation strategy corresponding to the knowledge file to be processed to obtain a segmentation result.

[0024] In an optional implementation, configuring a knowledge segmentation strategy corresponding to the knowledge file to be processed according to the file type and / or business requirements of the knowledge file to be processed includes:

[0025] Obtaining file type information and business requirement information of the knowledge file to be processed;

[0026] Based on the correspondence between the file type information and the recommended knowledge segmentation strategy, the recommended knowledge segmentation strategy corresponding to the file type information of the knowledge file to be processed is displayed; and based on the correspondence between the business requirement information and the recommended knowledge segmentation strategy, the recommended knowledge segmentation strategy corresponding to the business requirement information of the knowledge file to be processed is displayed;

[0027] In response to a triggering operation of the user on the displayed recommended knowledge segmentation strategy, the triggered recommended knowledge segmentation strategy is configured for the knowledge file to be processed as the knowledge segmentation strategy corresponding to the knowledge file to be processed.

[0028] In an optional implementation, the knowledge segmentation strategy configured for the knowledge file to be processed is one of the following strategies or a combination of multiple strategies in the following strategies:

[0029] Split by chapter, split by paragraph, split by sentence, split by separator, split by word count.

[0030] The second aspect of the present application provides a knowledge file segmentation processing device, the device comprising:

[0031] A first segmentation module, configured to segment the knowledge file to be processed into a plurality of knowledge file blocks based on the data volume of the knowledge file to be processed;

[0032] A cache module, used for caching the plurality of knowledge file blocks obtained by segmentation into a pre-created knowledge file block loading library;

[0033] A loading module, used for loading the plurality of knowledge file blocks from the knowledge file block loading library;

[0034] A task construction module, used to construct a plurality of file block segmentation tasks corresponding to the plurality of knowledge file blocks in a one-to-one manner according to the segmentation order of the plurality of knowledge file blocks;

[0035] The second segmentation module is used to use multiple threads to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining a knowledge document set corresponding to the knowledge file to be processed; the knowledge document set contains multiple knowledge documents, and each knowledge document serves as the segmentation result of the corresponding knowledge file block.

[0036] The third aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the knowledge file segmentation processing method introduced in any implementation method of the first aspect is implemented.

[0037] A fourth aspect of the present application provides a processor for running a computer program, which, when running, executes the knowledge file segmentation processing method introduced in any implementation manner of the first aspect.

[0038] Compared with the prior art, this application has the following beneficial effects:

[0039] In the present application, the knowledge file to be processed is segmented in two stages. In the first stage, the knowledge file to be processed is segmented into multiple knowledge file blocks based on the data volume of the knowledge file to be processed; in the second stage, the multiple knowledge file blocks obtained after segmentation are further segmented to obtain multiple knowledge documents. When executing the first stage of segmentation, this solution will segment the knowledge file to be processed into multiple knowledge file blocks, and then cache the multiple knowledge file blocks into a pre-created knowledge file block loading library, so that when the same knowledge file to be processed needs to be segmented again, it is not necessary to repeatedly segment it into knowledge file blocks, and the relevant knowledge file blocks of the knowledge file can be directly loaded from the knowledge file block loading library. The method of caching knowledge file blocks avoids repeated segmentation of the same knowledge file to be processed in the first stage, and improves the segmentation speed and efficiency of the overall file segmentation process. In addition, when executing the second stage of segmentation, this solution uses multiple threads to asynchronously execute file block segmentation tasks in batches, which effectively improves the segmentation efficiency of multiple knowledge file blocks, thereby also being able to improve the segmentation speed and efficiency of knowledge files as a whole. By improving the speed and efficiency of segmenting knowledge files, it is more convenient to perform knowledge management based on the knowledge document sets obtained by segmenting these knowledge files, thereby improving the efficiency of knowledge management. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 A schematic diagram of a segmentation processing stage of a knowledge file segmentation processing method provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of a knowledge file segmentation processing method provided in an embodiment of the present application;

[0043] Figure 3 A scenario architecture diagram of a knowledge file segmentation processing method provided in an embodiment of the present application;

[0044] Figure 4 An example flow chart of configuring a knowledge segmentation strategy provided in an embodiment of the present application;

[0045] Figure 5 A flowchart of another knowledge file segmentation processing method provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of a knowledge file segmentation and processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] As described above, current knowledge management is a key means to improve work efficiency, increase productivity and promote innovation and development. However, the processing of knowledge documents has the limitations of low processing efficiency and slow processing speed, which affects the effectiveness of knowledge management and the effective application of the processed knowledge documents in knowledge management.

[0048] In order to solve this problem, the present application provides a knowledge file segmentation processing method, device, storage medium and processor. The purpose is to improve the problem of low efficiency and slow processing speed of knowledge file processing through the technical means provided by this solution. In the technical solution of the present application, after the knowledge file to be processed is divided into multiple knowledge file blocks, these knowledge file blocks are cached in the knowledge file block loading library, so that these knowledge file blocks can be used for further segmentation when the segmentation is performed on the knowledge file to be processed this time, or these knowledge file blocks can be reused for further segmentation when the segmentation is performed on the knowledge file to be processed later. Thereby avoiding the need to repeatedly segment the knowledge file to be processed, which affects the processing speed and efficiency. In addition, when the knowledge file blocks are segmented, multi-threaded asynchronous batch execution of the file block segmentation task is adopted. The asynchronous batch execution method can also correspondingly improve the segmentation processing speed and efficiency of a large number of knowledge file blocks, thereby improving the processing speed and efficiency of knowledge files as a whole, and improving the effectiveness of knowledge management.

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] To facilitate understanding of this solution, first, combine Figure 1 The segmentation processing stage of a knowledge file segmentation processing method provided in an embodiment of the present application is introduced. Figure 1 As shown, the segmentation processing stages involved in the knowledge file segmentation processing method include: the first stage and the second stage. In the first stage, the knowledge file 101 to be processed needs to be segmented into multiple knowledge file blocks 102; in the second stage, the knowledge file block 102 needs to be further segmented into knowledge documents 103, each knowledge document 103 is used as the segmentation result of the corresponding knowledge file block 102, and finally the knowledge documents 103 of the same knowledge file 101 to be processed are collected to form a knowledge document set 104.

[0051] Figure 1 The segmentation processing method for a single knowledge file is illustrated in stages. The knowledge file segmentation processing method provided by the embodiment of the present application is introduced in more detail below in conjunction with the flow chart. Figure 2 A flow chart of a knowledge file segmentation processing method provided in an embodiment of the present application. Figure 2 As shown, the knowledge file segmentation processing method includes:

[0052] S201. Based on the data volume of the knowledge file to be processed, the knowledge file to be processed is divided into a plurality of knowledge file blocks, and the plurality of knowledge file blocks obtained by the division are cached in a pre-created knowledge file block loading library.

[0053] The knowledge files to be processed refer to the knowledge files that need to be processed for knowledge management. The knowledge files to be processed usually contain relatively rich knowledge content, for example, multiple chapters, multiple paragraphs, multiple sentences, and even multiple charts. The file format of the knowledge files to be processed can be txt, json, word, pdf, excel or html, etc. For the knowledge files to be processed in different file formats, the data structure of the content contained therein may also be different. For example, the knowledge files to be processed in the word file format contain more unstructured paragraphs, while the content contained in excel is more presented in the form of structured data. The embodiment of the present application does not limit the file format of the knowledge files to be processed and the data structure of the content contained therein.

[0054] In the embodiment of the present application, the knowledge file to be processed can be segmented according to the data volume. For example, for the knowledge file to be processed, every x data volume is segmented into a knowledge file block, where x is a value with a data volume unit. For example, if the total data volume of the knowledge file to be processed is 100x, the knowledge file to be processed can be segmented into 100 knowledge file blocks according to the above segmentation rules. The value of x can be set according to the segmentation requirements. As an example, x can be set to 100KB.

[0055] When segmenting in this step, the knowledge file to be processed can be segmented into multiple knowledge file blocks according to the order of the content from front to back and according to the amount of data. And in order to distinguish these segmented knowledge file blocks, identification information (ID) can be added to the knowledge file blocks. The added identification information may include numbers, characters, or a combination of numbers and characters. In the example with numbers as identification information, the identification information may be incremental, for example, the first knowledge file block obtained by segmentation is added with identification information 1, the second knowledge file block obtained by segmentation is added with identification information 2, the third knowledge file block obtained by segmentation is added with identification information 3, and so on. The specific form of the identification information of the knowledge file block is not limited in the present application.

[0056] In an optional implementation, the present application may further construct an association relationship between the identification information of each of the multiple knowledge file blocks obtained by segmentation in this step and the knowledge file to be processed. In one example, the identification information of each of the multiple knowledge file blocks is associated with the file name of the knowledge file to be processed. In another example, the identification information of each of the multiple knowledge file blocks is associated with the file identification of the knowledge file to be processed. Since it is necessary to obtain knowledge documents by segmenting knowledge file blocks in the following steps, by constructing an association relationship between the identification information of each of the multiple knowledge file blocks and the knowledge file to be processed, it is convenient to subsequently establish a corresponding relationship between each knowledge document obtained by segmentation and the knowledge file to be processed, and it is convenient to realize the orderly integration of each knowledge document segmented from the knowledge file to be processed based on the sequence of the identification information of the knowledge file blocks. Thereby, the accuracy of knowledge file segmentation is improved, and the complexity and difficulty of knowledge management are reduced.

[0057] After the initial segmentation to obtain multiple knowledge file blocks of the knowledge file to be processed, the embodiment of the present application further caches the knowledge file blocks obtained by segmentation into the knowledge file block loading library. When the segmentation is specifically performed, each knowledge file block obtained by segmentation can be cached into the knowledge file block loading library in sequence according to the segmentation order.

[0058] The knowledge file block loading library is a loading library created in advance before the current segmentation, which is dedicated to caching the knowledge file blocks obtained by segmenting each knowledge file. In the present application, the knowledge file block loading library supports querying the associated knowledge file blocks based on the identification information of the knowledge file, which facilitates efficient query and loading of the knowledge file blocks and facilitates the subsequent segmentation of the knowledge file blocks. In an optional implementation method, the file identification information, file name, segmentation sequence number and other related fields can be pre-defined for the knowledge file block loading library.

[0059] S202. Load the multiple knowledge file blocks from the knowledge file block loading library, and construct multiple file block segmentation tasks corresponding to the multiple knowledge file blocks one by one according to the segmentation order of the multiple knowledge file blocks.

[0060] In actual applications, after completing the first stage of segmentation processing in S201, the present application further executes S202 to load each knowledge file block stored in the knowledge file block loading library into the memory in order to carry out the second stage of segmentation processing. It should be noted that the loading action for multiple knowledge file blocks in the present application can be executed once or in multiple times. In one example, all knowledge file blocks of the knowledge file to be processed can be loaded into the memory, and then the segmentation processing of the knowledge file blocks is performed asynchronously in batches through S203. In another example, the number of knowledge file blocks that can be batch segmented in each batch of S203 is determined, and then based on this number, only the corresponding number of knowledge file blocks are loaded each time in S202. After the knowledge file blocks waiting to be loaded are batch segmented to obtain knowledge documents, the next batch of knowledge file blocks to be processed are loaded in the manner of S202.

[0061] In the specific implementation, the present application constructs multiple file block segmentation tasks corresponding to the multiple knowledge file blocks loaded. If the loading of multiple knowledge file blocks is performed in batches, then correspondingly, the construction of the file block segmentation tasks can also be performed in batches. For example, if N knowledge file blocks need to be loaded for the first time, the file block segmentation tasks corresponding to these N knowledge file blocks are first constructed.

[0062] S203, using multiple threads to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining a knowledge document set corresponding to the knowledge file to be processed.

[0063] In S203, thread pool technology and asynchronous batch task processing mechanism are used to segment the knowledge file blocks loaded in the previous step. In specific implementation, it is possible to analyze whether each thread in the thread pool is available, and assign file block segmentation tasks to idle threads, thereby improving the execution efficiency of file block segmentation tasks. In an embodiment of the present application, multiple threads execute file block segmentation tasks in batches. For example, a total of w threads are arranged to execute file block segmentation tasks, and a total of N knowledge file blocks are segmented out for the knowledge files to be processed. Then, each batch of these w threads can segment a total of w knowledge file blocks until these N knowledge file blocks are segmented into knowledge documents.

[0064] When a thread executes a file block segmentation task, the thread may use a pre-configured knowledge segmentation strategy to execute the file block segmentation task. That is, in this step, multiple threads may be started, and the multiple file block segmentation tasks may be asynchronously executed in batches using the knowledge segmentation strategy corresponding to the knowledge file to be processed, so as to segment the multiple knowledge file blocks.

[0065] For example, the knowledge file to be processed is file 1, and knowledge segmentation strategy 1 is configured for file 1; the knowledge file to be processed is file 2, and knowledge segmentation strategy 2 is configured for file 2. The thread can determine the corresponding knowledge file to be processed based on the identification information of the knowledge file block to be processed, the identification information of the knowledge file block and the file identification of the knowledge file to be processed. Since the knowledge segmentation strategy can be pre-configured for the knowledge file to be processed, the correspondence between the file identification of the knowledge file to be processed and the strategy identification of the configured knowledge segmentation strategy can be pre-constructed. In this way, as well as the above multiple correspondences, the thread can clearly identify the knowledge segmentation strategy that should be adopted, thereby completing the work of this step.

[0066] In a possible implementation, the knowledge segmentation strategy configured for the knowledge file to be processed is one of the following strategies or a combination of multiple strategies in the following strategies:

[0067] Split by chapter, split by paragraph, split by sentence, split by separator, split by word count.

[0068] In an optional implementation, the knowledge segmentation strategy is configured according to the file type and / or business requirements of the knowledge file to be processed. For example, if the file type of the knowledge file to be processed is a policy document, it generally contains many chapters, paragraphs, etc. Based on this, for the knowledge files to be processed of the policy document type, the configured knowledge segmentation strategy can be: segmentation by chapter and / or segmentation by paragraph. If the file type of the knowledge file to be processed is a script, the knowledge segmentation strategy configured for this type of file can be: segmentation by sentence. If the text type of the knowledge file to be processed is a data document containing many symbols, the knowledge segmentation strategy configured for this type of file can be: segmentation by delimiter. In addition, in knowledge management, the knowledge segmentation strategy of the knowledge file can also be configured in combination with the needs, because the granularity and precision of knowledge management of knowledge files in different businesses may be different. Configuring the knowledge segmentation strategy in combination with business needs can make the segmented knowledge documents more effective in knowledge management.

[0069] Combined with the introduction of the above embodiments, in the present application, the knowledge file to be processed is segmented in two stages. In the first stage, the knowledge file to be processed is segmented into multiple knowledge file blocks based on the data volume of the knowledge file to be processed; in the second stage, the multiple knowledge file blocks obtained after segmentation are further segmented to obtain multiple knowledge documents. When executing the first stage of segmentation, this scheme caches the multiple knowledge file blocks into the pre-created knowledge file block loading library after segmenting the knowledge file to be processed into multiple knowledge file blocks, so that when the same knowledge file to be processed needs to be segmented again, it is not necessary to repeatedly segment it into the form of knowledge file blocks, and the relevant knowledge file blocks of the knowledge file can be directly loaded from the knowledge file block loading library. The method of caching knowledge file blocks avoids repeated segmentation of the same knowledge file to be processed in the first stage, and improves the segmentation speed and efficiency of the overall file segmentation process. In addition, when executing the second stage of segmentation, this scheme uses multiple threads to asynchronously execute the file block segmentation task in batches, which effectively improves the segmentation efficiency of multiple knowledge file blocks, thereby also being able to improve the segmentation speed and efficiency of knowledge files as a whole. By improving the speed and efficiency of segmenting knowledge files, it is more convenient to perform knowledge management based on the knowledge document sets obtained by segmenting these knowledge files, thereby improving the efficiency of knowledge management.

[0070] Figure 3 A scenario architecture diagram of a knowledge file segmentation processing method provided in an embodiment of the present application. Figure 3 As shown, the scenario includes a user device 31, a knowledge file library 32, a knowledge file block loading library 33, a file block segmentation task (Task) 34 and a thread pool 35. In this scenario, the user can upload knowledge files to the knowledge file library 32 through the user device 31, and these knowledge files added to the knowledge file library 32 are used as knowledge files to be processed in subsequent links. The knowledge file library 32 is pre-created before executing this solution. In the stage of creating the knowledge file library 32, fields such as file identification, file name, and upload time are defined, thereby supporting file classification, file query and other operations based on these related fields in the knowledge file library 32. The knowledge file library 32 is provided with a knowledge file upload interface, so that the interface can be used to receive knowledge files uploaded by users. After receiving the knowledge file, the knowledge file library 32 associates and stores the file information (including file identification, file name, upload time, etc.) of the file with the file content. The file content can be stored in the knowledge file library 32 in binary form, thereby reducing the demand for storage space used by the knowledge file library 32.

[0071] When it is necessary to perform segmentation processing on the knowledge file 101 to be processed, the file identifier of the file can be used to query the knowledge file block loading library 33 whether there is a knowledge file block corresponding to the file identifier. If it exists, the next step of segmentation is performed; if it does not exist, it is necessary to use the file identifier of the knowledge file 101 to be processed, so as to obtain the knowledge file 101 to be processed from the knowledge file library 32, so as to segment the knowledge file 101 to be processed into multiple knowledge file blocks 102. These knowledge file blocks 102 can be easily distinguished after being identified, and the identification method can be incremented according to the segmentation order. And after the identification is completed, the knowledge file blocks with the identification are stored in the knowledge file block loading library 33 in the segmentation order (which can also be understood as the order of the identification from small to large), so as to be subsequently loaded into the memory for use and segmentation.

[0072] The knowledge file block 102 is loaded from the knowledge file block loading library 33, and the second stage of segmentation is performed. It can be understood that the loaded knowledge file block 102 needs to be further segmented. For this purpose, a file block segmentation task (Task) 34 is created, and the thread 351 in the thread pool 35 executes Task 34 in an asynchronous batch processing manner to segment the loaded knowledge file block 102 according to the knowledge segmentation strategy corresponding to the knowledge file 101 to be processed. Finally, knowledge documents 103 are obtained one by one. These knowledge documents 103 correspond to the segmented knowledge file blocks 102, and the knowledge documents 103 are the segmentation results of the corresponding knowledge file blocks 102.

[0073] In the present application, a knowledge document 103 may include a segmentation unit obtained by segmenting the corresponding knowledge file block 102. For example, if a knowledge file block 102 is segmented according to sentences, then a knowledge document 103 may include a sentence. In addition, in other possible implementations, a knowledge document 103 may also include some or all of the segmentation units obtained by segmenting the corresponding knowledge file block 102. For example, if a knowledge file block 102 is segmented according to sentences, then a knowledge document 103 may include some or all of the sentences obtained by segmenting the knowledge file block 102. These sentences are independently identified or separated in the knowledge document 103, that is, the knowledge document 103 is also different in form from the knowledge file block 102 before segmentation.

[0074] In the embodiment of the present application, when thread 351 executes the file block segmentation task 34, the knowledge file block 102 to be segmented can be converted into text content first. This is because the file format of the knowledge file may be txt, json, word, pdf, excel, html, etc., and the file format is not uniform, resulting in the data structure of different knowledge files is also not uniform. In order to achieve more accurate segmentation in the second stage, it is necessary to convert the knowledge file block 102 into text content with a relatively uniform data structure before further segmentation.

[0075] After the knowledge document 103 is obtained by segmentation, the knowledge document 103 obtained by segmentation can be further associated with the identification information of the corresponding knowledge file block 102. As mentioned above, the knowledge file block 102 obtained by segmentation can be associated with the knowledge file 101 to be processed after identification, and the knowledge document 103 will also be associated with the knowledge file block 102 later. On the premise that the first and second stage segmentation executions have constructed an association between the content before segmentation and the result after segmentation, each knowledge document 103 can finally be integrated into the knowledge document set 104 corresponding to the knowledge file 101 to be processed according to the segmentation order. Here, the segmentation order specifically refers to the order in which the knowledge file blocks 102 are obtained by segmentation, and can also be understood as the identification order of the knowledge file blocks 102. In actual applications, since the thread executes Task 34 asynchronously in batches, when integrating the knowledge document 103, it can be specifically waited until all the knowledge file blocks 102 of the knowledge file 101 to be processed are segmented into knowledge documents 103, and then all the knowledge documents of the knowledge file 101 to be processed are integrated and summarized to finally form the knowledge document set 104. By aggregating knowledge documents in the above manner, the obtained knowledge documents can be concatenated together according to the correct segmentation sequence, so that on the basis of segmenting the knowledge documents, the expected accurate segmentation results can be obtained, which assists in realizing efficient knowledge management work.

[0076] As introduced in the foregoing, in this application, the corresponding knowledge segmentation strategy can be configured according to the file type and / or business requirements of the knowledge file to be processed. The present application proposes that, through a technology for dynamically configuring the knowledge segmentation strategy, the configured strategy can take effect immediately after being saved. The following provides an implementation method for reconfiguring the knowledge segmentation strategy in combination with changes in business needs. After the knowledge segmentation strategy changes, it means that for the same knowledge file to be processed, the newly configured knowledge segmentation strategy needs to be enabled in the second stage to re-segment. Specifically, the process of reconfiguring the knowledge segmentation strategy and using the new strategy to implement segmentation includes the following steps:

[0077] Reconfigure the knowledge segmentation strategy corresponding to the knowledge file to be processed in response to changes in the business requirements of the file to be processed;

[0078] Load the plurality of knowledge file blocks from the knowledge file block loading library;

[0079] Start multiple threads, and use the knowledge segmentation strategy newly configured for the knowledge file to be processed to asynchronously execute the multiple file block segmentation tasks in batches to re-segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining the latest knowledge document set corresponding to the knowledge file to be processed.

[0080] Combined with the steps introduced above, it can be seen that when re-segmenting the knowledge file again, it is only necessary to load the segmented knowledge file blocks from the knowledge file block loading library, and there is no need to repeat the segmentation process of the first stage. In this way, even if it is necessary to re-segment the knowledge file to be processed, the segmentation processing time can be saved and the segmentation efficiency can be improved. Through dynamic configuration, the knowledge file is segmented according to different knowledge segmentation strategies to obtain a collection of knowledge documents with different segmentation effects, thereby meeting different needs and conditions. It can be seen that the method of dynamically adjusting the knowledge segmentation strategy significantly improves the bottleneck of the existing technology of insufficient flexibility in processing knowledge documents and difficulty in adapting to changing business needs. In addition, the asynchronous batch task processing mechanism and thread pool technology are used to improve the execution efficiency and responsiveness of the segmentation tasks.

[0081] The following is an example of an implementation method for configuring a knowledge segmentation strategy corresponding to the knowledge file to be processed according to the file type and / or business requirements of the knowledge file to be processed. Figure 4 The following is an example flowchart for configuring a knowledge segmentation strategy. Figure 4 As shown, this process includes:

[0082] S401: Obtain file type information and business requirement information of the knowledge file to be processed.

[0083] In order to recommend the knowledge segmentation strategy, it is first necessary to obtain the file type information and business requirement information about the knowledge file to be processed that needs to be configured this time. If one of the above two pieces of information is missing or unknown, the next step S402 can also be performed normally. This is because the knowledge segmentation strategy can independently establish a corresponding relationship with the file type information, and can also independently establish a corresponding relationship with the business requirement information.

[0084] S402. Based on the correspondence between file type information and recommended knowledge segmentation strategies, display the recommended knowledge segmentation strategies corresponding to the file type information of the knowledge file to be processed; and, based on the correspondence between business requirement information and recommended knowledge segmentation strategies, display the recommended knowledge segmentation strategies corresponding to the business requirement information of the knowledge file to be processed.

[0085] In this application, one or more specific knowledge segmentation strategies are recommended in advance for specific file type information, and one or more specific knowledge segmentation strategies are also recommended in advance for specific business demand information. This recommended knowledge segmentation strategy is called: recommended knowledge segmentation strategy. In some cases, the recommended knowledge segmentation strategy can be used as a default configuration, or it can be displayed for the user to make further selections and confirmations, thereby completing the configuration process of the strategy.

[0086] S403: In response to a trigger operation of the user on the displayed recommended knowledge segmentation strategy, the triggered recommended knowledge segmentation strategy is configured for the knowledge file to be processed as the knowledge segmentation strategy corresponding to the knowledge file to be processed.

[0087] For example, after executing S402, a total of 5 recommended knowledge segmentation strategies are displayed, and one or more of these 5 recommended knowledge segmentation strategies may hit the user's real segmentation needs. To this end, the user can express his or her confirmation intention by touching the relevant strategies displayed on the screen of the device. The system responds to the trigger operation, confirms the touched strategy among the 5 recommended knowledge segmentation strategies as the strategy that needs to be configured later, and performs the strategy configuration operation on the knowledge file to be processed.

[0088] Figure 5 A flowchart of another knowledge file segmentation processing method provided in an embodiment of the present application. Figure 5 As shown, in the process shown, first, the knowledge file to be processed uploaded to the knowledge file library is received. The knowledge files uploaded to the knowledge file library are stored in a unified manner for easy management and retrieval. The knowledge file library can facilitate the centralized management and maintenance of knowledge files, thereby better ensuring the consistency and security of file data. When it is necessary to process the knowledge file to be processed, it is necessary to determine whether the file has been split into knowledge file blocks. For those that have been split, it means that there is no need to repeat the splitting and they can be directly loaded from the knowledge file block loading library; for those that have not been split, they need to be extracted from the knowledge file library and split into knowledge file blocks before further segmentation processing is performed. Subsequently, similar to the method embodiment introduced above, multiple threads are used to asynchronously execute the segmentation task in batches until a knowledge document collection is obtained.

[0089] Based on the knowledge file segmentation method introduced in the above embodiment, the present application accordingly proposes a knowledge file segmentation processing device. Figure 6 Figure 1 is a schematic diagram of the structure of the device. Figure 6 As shown, the knowledge file segmentation processing device 60 includes:

[0090] A first segmentation module 601 is used to segment the knowledge file to be processed into a plurality of knowledge file blocks based on the data volume of the knowledge file to be processed;

[0091] A cache module 602, configured to cache the plurality of knowledge file blocks obtained by segmentation into a pre-created knowledge file block loading library;

[0092] A loading module 603, configured to load the plurality of knowledge file blocks from the knowledge file block loading library;

[0093] A task construction module 604 is used to construct a plurality of file block segmentation tasks corresponding to the plurality of knowledge file blocks in a one-to-one manner according to the segmentation order of the plurality of knowledge file blocks;

[0094] The second segmentation module 605 is used to use multiple threads to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining a knowledge document set corresponding to the knowledge file to be processed; the knowledge document set contains multiple knowledge documents, and each knowledge document serves as the segmentation result of the corresponding knowledge file block.

[0095] In the present application, the knowledge file to be processed is segmented in two stages. In the first stage, the knowledge file to be processed is segmented into multiple knowledge file blocks based on the data volume of the knowledge file to be processed; in the second stage, the multiple knowledge file blocks obtained after segmentation are further segmented to obtain multiple knowledge documents. When executing the first stage of segmentation, this solution will segment the knowledge file to be processed into multiple knowledge file blocks, and then cache the multiple knowledge file blocks into a pre-created knowledge file block loading library, so that when the same knowledge file to be processed needs to be segmented again, it is not necessary to repeatedly segment it into knowledge file blocks, and the relevant knowledge file blocks of the knowledge file can be directly loaded from the knowledge file block loading library. The method of caching knowledge file blocks avoids repeated segmentation of the same knowledge file to be processed in the first stage, and improves the segmentation speed and efficiency of the overall file segmentation process. In addition, when executing the second stage of segmentation, this solution uses multiple threads to asynchronously execute file block segmentation tasks in batches, which effectively improves the segmentation efficiency of multiple knowledge file blocks, thereby also being able to improve the segmentation speed and efficiency of knowledge files as a whole. By improving the speed and efficiency of segmenting knowledge files, it is more convenient to perform knowledge management based on the knowledge document sets obtained by segmenting these knowledge files, thereby improving the efficiency of knowledge management.

[0096] In an optional implementation, the second segmentation module 605 of the knowledge file segmentation processing device is specifically used to:

[0097] Start multiple threads, and use the knowledge segmentation strategy corresponding to the knowledge file to be processed to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks; the knowledge segmentation strategy is configured according to the file type and / or business requirements of the knowledge file to be processed.

[0098] In an optional implementation, the knowledge file segmentation processing device further includes:

[0099] A strategy configuration module 606, configured to reconfigure the knowledge segmentation strategy corresponding to the knowledge file to be processed in response to changes in the business requirements of the file to be processed;

[0100] The loading module 603 is further used to load the plurality of knowledge file blocks from the knowledge file block loading library after the strategy configuration module 606 reconfigures the knowledge segmentation strategy;

[0101] The second segmentation module 605 is also used to start multiple threads after the loading module loads the knowledge file block again, and use the knowledge segmentation strategy newly configured for the knowledge file to be processed to asynchronously execute the multiple file block segmentation tasks in batches to re-segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining the latest knowledge document set corresponding to the knowledge file to be processed.

[0102] In an optional implementation, the knowledge file segmentation processing device further includes:

[0103] an identification module, configured to assign different identification information to the plurality of knowledge file blocks according to the division order of the plurality of knowledge file blocks so as to distinguish different knowledge file blocks;

[0104] A relationship building module, used to build an association relationship between the identification information of each of the plurality of knowledge file blocks and the knowledge file to be processed;

[0105] The second segmentation module 605 is specifically used for:

[0106] Associating the knowledge documents obtained by segmentation with the identification information of the corresponding knowledge file blocks;

[0107] According to the association between the identification information of each of the plurality of knowledge file blocks and the knowledge file to be processed, and the association between the knowledge document and the identification information, each knowledge document is integrated into the knowledge document set corresponding to the knowledge file to be processed in the segmentation order.

[0108] In an optional implementation, the second segmentation module 605 is specifically configured to:

[0109] The thread converts the knowledge file block corresponding to the executed file block segmentation task into text content;

[0110] The text content is segmented using the knowledge segmentation strategy corresponding to the knowledge file to be processed to obtain a segmentation result.

[0111] In an optional implementation, the policy configuration module 606 is specifically configured to:

[0112] Obtaining file type information and business requirement information of the knowledge file to be processed;

[0113] Based on the correspondence between the file type information and the recommended knowledge segmentation strategy, the recommended knowledge segmentation strategy corresponding to the file type information of the knowledge file to be processed is displayed; and based on the correspondence between the business requirement information and the recommended knowledge segmentation strategy, the recommended knowledge segmentation strategy corresponding to the business requirement information of the knowledge file to be processed is displayed;

[0114] In response to a triggering operation of the user on the displayed recommended knowledge segmentation strategy, the triggered recommended knowledge segmentation strategy is configured for the knowledge file to be processed as the knowledge segmentation strategy corresponding to the knowledge file to be processed.

[0115] Based on the above content, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a knowledge file segmentation processing method as described in any of the implementation methods described in the method embodiment is implemented.

[0116] In addition, the present application also provides a processor for running a computer program, and when the computer program is running, the knowledge file segmentation processing method of any implementation method introduced in the method embodiment part is executed.

[0117] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0118] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A knowledge file segmentation processing method, characterized in that: include: Based on the data volume of the knowledge file to be processed, the knowledge file to be processed is divided into a plurality of knowledge file blocks, and the plurality of knowledge file blocks obtained by the division are cached in a pre-created knowledge file block loading library; Loading the plurality of knowledge file blocks from the knowledge file block loading library, and constructing a plurality of file block segmentation tasks corresponding to the plurality of knowledge file blocks one by one according to the segmentation order of the plurality of knowledge file blocks; Using multiple threads to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining a knowledge document set corresponding to the knowledge file to be processed; The knowledge document set includes a plurality of knowledge documents, each of which is a segmentation result of a corresponding knowledge file block.

2. The method according to claim 1, characterized in that The step of using multiple threads to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks includes: Start multiple threads, and use the knowledge segmentation strategy corresponding to the knowledge file to be processed to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks; the knowledge segmentation strategy is configured according to the file type and / or business requirements of the knowledge file to be processed.

3. The method according to claim 2, characterized in that Also includes: Reconfigure the knowledge segmentation strategy corresponding to the knowledge file to be processed in response to changes in the business requirements of the file to be processed; Load the plurality of knowledge file blocks from the knowledge file block loading library; Start multiple threads, and use the knowledge segmentation strategy newly configured for the knowledge file to be processed to asynchronously execute the multiple file block segmentation tasks in batches to re-segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining the latest knowledge document set corresponding to the knowledge file to be processed.

4. The method according to claim 1, characterized in that: Also includes: According to the division order of the plurality of knowledge file blocks, different identification information is assigned to the plurality of knowledge file blocks to distinguish different knowledge file blocks; Constructing an association relationship between the identification information of each of the plurality of knowledge file blocks and the knowledge file to be processed; The step of obtaining a knowledge document set corresponding to the knowledge file to be processed includes: Associating the knowledge documents obtained by segmentation with the identification information of the corresponding knowledge file blocks; According to the association between the identification information of each of the plurality of knowledge file blocks and the knowledge file to be processed, and the association between the knowledge document and the identification information, each knowledge document is integrated into the knowledge document set corresponding to the knowledge file to be processed in the segmentation order.

5. The method according to claim 2, characterized in that: The thread uses the knowledge segmentation strategy corresponding to the knowledge file to be processed to perform the file block segmentation task to segment the knowledge file block, including: The thread converts the knowledge file block corresponding to the executed file block segmentation task into text content; The text content is segmented using the knowledge segmentation strategy corresponding to the knowledge file to be processed to obtain a segmentation result.

6. The method according to claim 2, characterized in that Configuring a knowledge segmentation strategy corresponding to the knowledge file to be processed according to the file type and / or business requirements of the knowledge file to be processed includes: Obtaining file type information and business requirement information of the knowledge file to be processed; Based on the correspondence between the file type information and the recommended knowledge segmentation strategy, the recommended knowledge segmentation strategy corresponding to the file type information of the knowledge file to be processed is displayed; and based on the correspondence between the business requirement information and the recommended knowledge segmentation strategy, the recommended knowledge segmentation strategy corresponding to the business requirement information of the knowledge file to be processed is displayed; In response to a triggering operation of the user on the displayed recommended knowledge segmentation strategy, the triggered recommended knowledge segmentation strategy is configured for the knowledge file to be processed as the knowledge segmentation strategy corresponding to the knowledge file to be processed.

7. The method according to claim 2 or 6, characterized in that: The knowledge segmentation strategy configured for the knowledge file to be processed is one of the following strategies or a combination of multiple strategies in the following strategies: Split by chapter, split by paragraph, split by sentence, split by separator, split by word count.

8. A knowledge file segmentation processing device, characterized in that: include: A first segmentation module, configured to segment the knowledge file to be processed into a plurality of knowledge file blocks based on the data volume of the knowledge file to be processed; A cache module, used for caching the plurality of knowledge file blocks obtained by segmentation into a pre-created knowledge file block loading library; A loading module, used for loading the plurality of knowledge file blocks from the knowledge file block loading library; A task construction module, used to construct a plurality of file block segmentation tasks corresponding to the plurality of knowledge file blocks in a one-to-one manner according to the segmentation order of the plurality of knowledge file blocks; A second segmentation module is used to use multiple threads to asynchronously execute the multiple file block segmentation tasks in batches to segment the multiple knowledge file blocks until the multiple file block segmentation tasks are completed, thereby obtaining a knowledge document set corresponding to the knowledge file to be processed; The knowledge document set includes a plurality of knowledge documents, each of which is a segmentation result of a corresponding knowledge file block.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the knowledge file segmentation processing method as described in any one of claims 1 to 7 is implemented.

10. A processor, characterized in that: Used to run a computer program, which, when running, executes the knowledge file segmentation processing method as described in any one of claims 1 to 7.

Images