Data processing method for knowledge base, computer equipment and medium

Abstract

The invention provides a data processing method for a knowledge base, computer equipment and a medium. The method comprises the steps of obtaining a to-be-analyzed target knowledge file; performing layout recognition on the target knowledge file to determine a plurality of layout elements in the target knowledge file and a layout area where each layout element is located, the plurality of layout elements including at least one of picture elements and table elements and character elements; identifying the layout elements by utilizing a text identification tool corresponding to each layout element to obtain text data of the layout elements; and based on the text data of the plurality of layout elements and the located layout area, generating knowledge data of the target knowledge file so as to obtain a knowledge base comprising the knowledge data. According to the method, layout recognition is performed on the target knowledge file, and text recognition is performed by using corresponding text recognition tools for different layout elements, so that more complete knowledge data is generated, and the use effect of a knowledge base is improved.

Description

Technical Field

[0001] The present application relates to the technical field of knowledge bases, and in particular to a data processing method, computer equipment and medium for a knowledge base. Background Art

[0002] In building a knowledge base, it is necessary to parse the uploaded knowledge files to obtain the corresponding knowledge data. In some application scenarios, knowledge files may contain information in various forms, such as text, pictures, and tables, but usually only text information can be successfully parsed, while information in the form of pictures, tables, etc. cannot be parsed, making the generated knowledge data incomplete and the use effect of the knowledge base poor. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, computer device and medium for a knowledge base, aiming to generate more complete knowledge data and improve the use effect of the knowledge base.

[0004] In a first aspect, an embodiment of the present application provides a data processing method for a knowledge base, and the data processing method for a knowledge base includes:

[0005] Obtain the target knowledge file to be parsed;

[0006] Performing layout recognition on the target knowledge file to determine a plurality of layout elements in the target knowledge file and a layout area where each of the layout elements is located, wherein the plurality of layout elements include at least one of a picture element and a table element and a text element;

[0007] Using a text recognition tool corresponding to each of the layout elements, perform text recognition on the layout elements to obtain text data of the layout elements;

[0008] Based on the text data of the plurality of layout elements and the layout areas where they are located, the knowledge data of the target knowledge file is generated, thereby obtaining a knowledge base including the knowledge data.

[0009] In some embodiments, generating the knowledge data of the target knowledge file based on the text data of the plurality of layout elements and the layout areas in which they are located includes:

[0010] Segmenting the text data of each of the layout elements to obtain a plurality of text data segments in the layout element;

[0011] Determine the adjacent text data segments of each of the text data segments by using the layout area where the plurality of layout elements are located;

[0012] generating context information of each of the text data segments based on adjacent text data segments of the text data segment;

[0013] The knowledge data of the target knowledge file is generated by using the plurality of text data segments and context information of the text data segments.

[0014] In some embodiments, the step of generating the knowledge data of the target knowledge file by using the plurality of text data segments and the context information of the text data segments includes:

[0015] generating at least one of a text summary and a text label for each of the text data segments;

[0016] The knowledge data of the target knowledge document is generated by using a plurality of the text data segments, at least one of text summaries and text tags of the text data segments, and context information of the text data segments.

[0017] In some embodiments, using a text recognition tool corresponding to each of the layout elements to perform text recognition on the layout elements to obtain text data of the layout elements includes:

[0018] When the layout element is a text element, determining the number of layout columns of the text element;

[0019] If the number of layout columns is multiple, the text element is divided into multiple column sub-elements;

[0020] Rearrange the sub-elements in the multiple columns to obtain a column of target sub-elements;

[0021] Using a text recognition tool corresponding to the text element, text recognition is performed on the column of target sub-elements to obtain text data of the text element.

[0022] In some embodiments, after performing text recognition on the layout elements using the text recognition tool corresponding to each of the layout elements to obtain the text data of the layout elements, the method further includes:

[0023] Displaying first preview information of the layout of the target knowledge document, and displaying second preview information of text data including a plurality of the layout elements;

[0024] In response to a modification instruction for the second preview information, the second preview information and text data of at least one of the layout elements are modified.

[0025] In some embodiments, after obtaining the target knowledge file to be parsed, the method further includes:

[0026] Determine a target knowledge management template from among a plurality of preset knowledge management templates, wherein the target knowledge management template includes at least one of a specified layout recognition model, a text recognition tool, a slicer, a knowledge data storage method, an indexing method, and a vector database type;

[0027] Among them, the layout recognition model is used to perform layout recognition on the target knowledge file, the slicer is used to segment the text data of the layout elements, the knowledge data storage method includes associating and storing at least one of context information, text summary, and text label with the text data segment in the text data, and the indexing method and the vector database type are used to limit the generation rules of the knowledge base including the knowledge data.

[0028] In some embodiments, the preset knowledge governance template is generated by:

[0029] Get the preset knowledge file to be parsed;

[0030] Displaying a navigation bar including a configuration process of a knowledge governance template, wherein the navigation bar includes a configuration node for at least one of a layout recognition model, a text recognition tool, a slicer, a knowledge data storage method, an indexing method, and a vector database type;

[0031] Determine a current configuration node among the multiple configuration nodes;

[0032] Based on the preset knowledge file, generate preview information after the knowledge file processing corresponding to the configuration information in the current configuration node;

[0033] Displaying preview information of the knowledge file after processing, so as to modify the configuration information in the current configuration node;

[0034] In response to the knowledge governance template generation instruction, the preset knowledge governance template is generated using the configuration information in all the configuration nodes.

[0035] In some embodiments, determining a target knowledge governance template from a plurality of preset knowledge governance templates includes:

[0036] Acquire multiple parsed historical knowledge files and a historical knowledge governance template used by each of the historical knowledge files, each of the historical knowledge governance templates being a preset knowledge governance template;

[0037] Determining the similarity between the layout of the target knowledge document and the layout of each of the historical knowledge documents;

[0038] Based on the similarity, determining a target historical knowledge file from the plurality of historical knowledge files;

[0039] Based on the historical knowledge governance template adopted by the target historical knowledge file, the target knowledge governance template is determined.

[0040] In a second aspect, an embodiment of the present application provides a data processing device for a knowledge base, the data processing device for a knowledge base comprising:

[0041] A first acquisition module is used to acquire a target knowledge file to be parsed;

[0042] A first determination module is used to perform layout recognition on the target knowledge file to determine a plurality of layout elements in the target knowledge file and a layout area where each of the layout elements is located, wherein the plurality of layout elements include at least one of a picture element and a table element and a text element;

[0043] A second determination module is used to use a text recognition tool corresponding to each of the layout elements to perform text recognition on the layout elements to obtain text data of the layout elements;

[0044] The first generating module is used to generate the knowledge data of the target knowledge file based on the text data of the plurality of layout elements and the layout areas where they are located, so as to obtain a knowledge base including the knowledge data.

[0045] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is configured to be executed by the processor to implement a data processing method for a knowledge base as described in any one of the above items.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to be executed by a processor to implement a data processing method for a knowledge base as described in any one of the above items.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which are executed by a processor to implement a data processing method for a knowledge base as described in any of the above items.

[0048] Beneficial effects of the embodiments of the present application:

[0049] In an embodiment of the present application, layout recognition is performed on the target knowledge file to determine multiple layout elements in the target knowledge file and the layout area where each layout element is located. For different layout elements, text recognition is performed using corresponding text recognition tools to obtain corresponding text data. Based on the text data of multiple layout elements and the layout area where they are located, knowledge data of the target knowledge file is generated, thereby making the generated knowledge data more complete and improving the use effect of the knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 It is a schematic diagram of an embodiment of a data processing method for a knowledge base provided in an embodiment of the present application;

[0052] Figure 2 is a schematic diagram of a display interface including first preview information and second preview information provided by an embodiment of the present application;

[0053] Figure 3 It is a flowchart of another embodiment of a data processing method for a knowledge base provided in an embodiment of the present application;

[0054] Figure 4 It is a flowchart of another embodiment of a data processing method for a knowledge base provided in an embodiment of the present application;

[0055] Figure 5 It is a schematic diagram of a display interface including a navigation bar, configuration information in a current configuration node, and preview information after knowledge file processing provided by an embodiment of the present application;

[0056] Figure 6 is a schematic diagram of the structure of an embodiment of a data processing device for a knowledge base provided in an embodiment of the present application;

[0057] Figure 7 It is a schematic diagram of the structure of an embodiment of a computer device provided in an embodiment of the present application.

[0058] in, Figure 2 and Figure 5 The image is obtained by rotating the corresponding display interface 90 degrees counterclockwise. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0060] In the description of this application, the meaning of "plurality" is two or more, unless otherwise expressly and specifically defined. In addition, in the description of this application, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0061] In order to improve the use effect of the knowledge base, the embodiment of the present application provides a data processing method, computer device and medium for the knowledge base, which performs layout recognition on the target knowledge file to determine multiple layout elements in the target knowledge file and the layout area where each layout element is located, and uses corresponding text recognition tools to perform text recognition on different layout elements to obtain corresponding text data, and generates knowledge data of the target knowledge file based on the text data of multiple layout elements and the layout area where they are located, so that the generated knowledge data is more complete, so as to improve the use effect of the knowledge base. For the specific scheme, please refer to the following specific description.

[0062] In a first aspect, the embodiments of the present application provide a data processing method for a knowledge base. Specifically, referring to Figure 1 , Figure 1 The figure is a flow chart of an embodiment of a method for processing data of a knowledge base.

[0063] exist Figure 1 In the data processing method for the knowledge base, the data processing method may include:

[0064] 101. Obtain the target knowledge file to be parsed.

[0065] In an embodiment of the present application, the target knowledge file to be parsed can be uploaded by the user or synchronized from a designated data source. The target knowledge file can be, for example, a local unstructured document, a local table, a local question-and-answer pair uploaded by the user, and the designated data source can be, for example, a designated website, a designated data platform, a designated API (Application Programming Interface). Local unstructured documents can be, for example, local documents in formats such as TXT, HTML, MARKDOWN, PDF, DOC, DOCX, PNG, JPG, MP4, MP3, AVI, and WMV. The format of the local table can be, for example, XLS or XLSX. The format of the local question-and-answer pair can be, for example, CSV (Comma-Separated Values). In the step of obtaining the target knowledge file from the designated website, it can be specifically to crawl web page information from the designated website, and then convert the web page information into a file in MD (MARKDOWN) format, and use it as the target knowledge file. It can be seen that the data processing method for the knowledge base in the embodiment of the present application supports the processing of knowledge files of multiple formats and multiple data sources (such as invoices, operating manuals, financial statements, research reports, patent documents, PDF photocopies, public account articles, bidding forms, posters, etc. of the same type within the enterprise). In addition, there can be multiple target knowledge files to be parsed at the same time. For example, a user can upload multiple local unstructured documents at the same time, and then each of the multiple local unstructured documents can be used as a target knowledge file to be parsed in turn or in batches to perform subsequent steps.

[0066] In some embodiments of the present application, after obtaining the target knowledge file to be parsed, a similar file repetition check can also be performed on the file content in this knowledge base. Specifically, in the preset knowledge base, it is detected whether there is a knowledge file whose repetition rate with the text content in the target knowledge file is greater than the preset repetition rate; if there is a knowledge file whose repetition rate with the text content in the target knowledge file is greater than the preset repetition rate, the knowledge file and its file path are displayed for the user to choose whether to perform at least one of the following operations: replace the knowledge file with the target knowledge file, retain the knowledge file and use the target knowledge file as a new knowledge file to execute step 102 and its subsequent steps, or discard the target knowledge file. Among them, the preset repetition rate can be 80%, for example.

[0067] 102. Perform layout recognition on the target knowledge file to determine multiple layout elements in the target knowledge file and a layout area where each layout element is located, wherein the multiple layout elements include at least one of a picture element and a table element and a text element.

[0068] In an embodiment of the present application, layout recognition can be implemented using a corresponding layout recognition model. The layout recognition model can be, for example, a neural network model (such as a YOLO (You Only Look Once) model). When performing layout recognition, the target knowledge file can be output to the corresponding layout recognition model, and then the layout recognition result output by the layout recognition model is received. The layout recognition result can, for example, include multiple layout elements in the target knowledge file, and the layout area where each layout element is located. Among them, the layout area represents the position of the corresponding layout element in the layout of the target knowledge file.

[0069] In an embodiment of the present application, the plurality of layout elements include at least one of a picture element and a table element and a text element. Among them, the text element is a layout element displayed in a text format, the picture element is a layout element displayed in a picture format, and the table element is a layout element displayed in a table format. The text element can also be subdivided into a text title element, a text body element, etc.

[0070] 103. Use the text recognition tool corresponding to each layout element to perform text recognition on the layout element to obtain the text data of the layout element.

[0071] In the embodiments of the present application, different text recognition tools are applicable to different recognition objects. For example, some text recognition tools are applicable to text content recognition in pictures and generate corresponding text data, while another part of text recognition tools are applicable to text content recognition in tables and generate corresponding text data. Therefore, the embodiments of the present application are pre-set with corresponding text recognition tools for different layout elements.

[0072] In some embodiments of the present application, text recognition is performed on each layout element, and the data format of the obtained text data is generally in MARKDOWN format. MARKDOWN is a lightweight markup language used to generate a plain text format.

[0073] In some embodiments of the present application, the text content in the text element may be arranged in a single column or multiple columns (for example, some paper journals contain text content arranged in multiple columns). In order to more accurately identify the text content arranged in multiple columns in the text element, a text content rearrangement function may also be provided. Specifically, using the text recognition tool corresponding to each layout element, performing text recognition on the layout element to obtain the text data of the layout element may include: when the layout element is a text element, determining the number of layout columns of the text element, the number of layout columns may be, for example, a single column or multiple columns; if the number of layout columns is multiple columns, dividing the text element into multiple columns of sub-elements; rearranging the multiple columns of sub-elements to obtain a column of target sub-elements, taking the left and right columns of sub-elements as an example, the right sub-element may be merged as the next sub-element of the left sub-element to obtain a column of target sub-elements; using the text recognition tool corresponding to the text element, performing text recognition on the column of target sub-elements to obtain the text data of the text element. In this way, the text content in the text element with multiple columns can be converted into the text content in the text element with single column layout, so that the recognition of the text content in the text element with multiple columns layout is more accurate.

[0074] In some embodiments of the present application, since the recognition results of the text recognition tool may have certain errors, a text data correction function can also be provided. Specifically, using the text recognition tool corresponding to each layout element to perform text recognition on the layout element, after obtaining the text data of the layout element, it can also include: displaying first preview information of the layout of the target knowledge file, and displaying second preview information including the text data of multiple layout elements, for example Figure 2 As shown in , the first preview information of the layout of the target knowledge file is located on the left side of the display interface, and the second preview information of the text data including multiple layout elements is located on the right side of the display interface. In this way, the user can compare the first preview information displayed on the left side of the display interface with the second preview information displayed on the right side to detect whether there is an error in the recognition result of the text recognition tool (if the recognition result is detected to be an error, the user can trigger a modification instruction based on the MD (MARKDOWN) editor associated with the second preview information); in response to the modification instruction for the second preview information, the second preview information and the text data of at least one layout element are modified, thereby modifying the error in the text data. It can be seen that the embodiment of the present application can make the error identification and correction of text data more convenient by simultaneously displaying the first preview information of the layout of the target knowledge file and the second preview information of the text data including multiple layout elements.

[0075] In addition, the display interface may also include a drop-down radio button, which includes options for multiple different target knowledge files. Users can select different target knowledge files through the drop-down radio button to enable Figure 2The display interface displays the first preview information and the second preview information of the corresponding target knowledge file.

[0076] The display interface also has a location tracking function. Specifically, when the user clicks Figure 2 When you preview part of the first information, Figure 2 The corresponding content in the second preview information will be highlighted and flashed. Figure 2 When the second preview information is displayed, Figure 2 The corresponding content in the first preview information will be highlighted and flashed. When sliding the progress scroll bar of the first preview information, the second preview information will be located and scrolled synchronously due to position tracking.

[0077] 104. Generate knowledge data of the target knowledge file based on the text data of the plurality of layout elements and the layout areas where they are located, thereby obtaining a knowledge base including the knowledge data.

[0078] In the embodiment of the present application, after obtaining the text data of multiple layout elements, the layout area where the corresponding layout element is located can be used as the layout area where the text data of the layout element is located, thereby obtaining a knowledge file in a plain text format. By segmenting and extracting data from the knowledge file in a plain text format, the knowledge data of the target knowledge file can be obtained, and then by steps such as indexing, a knowledge base including the knowledge data can be generated.

[0079] It can be seen that in the above-mentioned embodiments of the present application, layout recognition is performed on the target knowledge file to determine multiple layout elements in the target knowledge file and the layout area where each layout element is located. For different layout elements, text recognition is performed using corresponding text recognition tools to obtain corresponding text data. Based on the text data of multiple layout elements and the layout area where they are located, knowledge data of the target knowledge file is generated, thereby making the generated knowledge data more complete and improving the use effect of the knowledge base.

[0080] In some embodiments of the present application, Figure 3 As shown, in Figure 1 On the basis of the illustrated embodiment, based on the text data of multiple layout elements and the layout areas where they are located, generating knowledge data of the target knowledge file may include:

[0081] 301. Segment the text data of each layout element to obtain multiple text data segments in the layout element.

[0082] In the embodiment of the present application, since the text data of the layout element may contain multiple sections, the text data of each layout element may be segmented to obtain multiple text data sections in the corresponding layout element. The segmentation process may be implemented using a corresponding slicer, such as a recursive character text segmenter, and corresponding slicing rules may be set for the slicer, such as the maximum length of the segment, the number of overlapping characters in the segment, and the like.

[0083] 302. Determine adjacent text data segments of each text data segment by using the layout area where multiple layout elements are located.

[0084] In an embodiment of the present application, because there may be a front and back logical association between different text data segments, therefore the layout area at which a plurality of layout elements are located can be utilized to determine the adjacent text data segments of each text data segment, for example, the layout area at which a plurality of layout elements are located can be utilized to determine two adjacent layout elements, and then in two adjacent layout elements, the last text data segment in the layout element at the front position is used as the adjacent text data segment of the first text data segment in the layout element at the back position. Like this, the adjacent text data segments of each text data segment can be determined. Wherein, the adjacent text data segments can include at least one of the previous text data segment and the next text data segment of the corresponding text data segment. In addition, the adjacent text data segments adopt different colors when displayed, so that the user can distinguish two adjacent text data segments.

[0085] 303. Generate context information of each text data segment based on adjacent text data segments of each text data segment.

[0086] In an embodiment of the present application, adjacent text data segments of a text data segment may be directly used as context information of the text data segment, so that when searching and recalling the knowledge base, the context information of the text data segment may be obtained.

[0087] 304. Generate knowledge data of a target knowledge file by using the multiple text data segments and context information of the text data segments.

[0088] In some embodiments of the present application, each text data segment is associated with the context information of the text data segment and stored to obtain metadata of the text data segment. The collection of metadata of multiple text data segments is the knowledge data of the target knowledge file.

[0089] It can be seen that in the above-mentioned embodiments of the present application, by utilizing the layout area where multiple layout elements are located to determine the adjacent text data segments of each text data segment, and then utilizing multiple text data segments and the context information of the text data segments to generate knowledge data, subsequent retrieval and recall can be made more convenient, thereby improving the use effect of the knowledge base.

[0090] In some embodiments of the present application, using multiple text data segments and context information of the text data segments to generate knowledge data of the target knowledge file may include: generating at least one of a text summary and a text label for each text data segment, and the text summary and the text label may be implemented using natural language processing (NLP) technology (text summary refers to compressing the text data segment into a concise and comprehensive summary, thereby improving the density of information, and the text label refers to labeling the text data segment with a feature label for association to adapt to the algorithm based on text precise matching, which is very suitable for keyword retrieval); using multiple text data segments, at least one of the text summary and the text label of the text data segment and the context information of the text data segment, to generate the knowledge data of the target knowledge file, for example, the knowledge data of the target knowledge file may include the text data segment, the context information of the text data segment, the text summary and the text label at the same time, and the same text data segment is associated with the context information, the text summary and the text label of the text data segment, so as to make the subsequent retrieval and recall of the knowledge base more convenient. In addition, the text summary and the text label can also be manually modified by the user based on demand to avoid errors caused by natural language processing technology.

[0091] In some embodiments of the present application, Figure 4 As shown, in Figures 1 to 3 Based on any of the embodiments shown, after obtaining the target knowledge file to be parsed, the following steps may also be included:

[0092] 401. Determine a target knowledge management template from among a plurality of preset knowledge management templates, wherein the target knowledge management template includes at least one of a specified layout recognition model, a text recognition tool, a slicer, a knowledge data storage method, an indexing method, and a vector database type.

[0093] In the embodiment of the present application, a plurality of preset knowledge management templates are pre-set, and the preset knowledge management templates are used to indicate how to perform data processing on the knowledge file to obtain the corresponding knowledge base. Therefore, each preset knowledge management template includes at least one of a preset layout recognition model, a text recognition tool, a slicer, a knowledge data storage method, an indexing method, and a vector database type.

[0094] Among them, the layout recognition model is used to perform layout recognition on the target knowledge file, for example, it can be a neural network model (such as a YOLO model), etc. The slicer is used to segment the text data of the layout elements, for example, it can be a recursive character text segmenter. The knowledge data storage method includes associating and storing at least one of context information, text summary, and text label with the text data segment in the text data. The index method and vector database type are used to limit the generation rules of the knowledge base including the knowledge data. The index method can be, for example, an inverted index, a full-text index, a hash index, etc., and the vector database type can be, for example, Milvus, Faiss, GraphDB, etc.

[0095] After determining the target knowledge governance template among multiple preset knowledge governance templates, the target knowledge governance template can be used to indicate how to perform data processing on the target knowledge file (ie, execute step 102 and subsequent steps).

[0096] In some embodiments of the present application, an example is given to illustrate the generation process of a preset knowledge governance template. Specifically, the preset knowledge governance template is generated by: obtaining a preset knowledge file to be parsed, wherein the method of obtaining the preset knowledge file is similar to that of the target knowledge file and is not limited here; displaying a navigation bar including the configuration process of the knowledge governance template, the navigation bar including at least one configuration node of the layout recognition model, text recognition tool, slicer, knowledge data storage method, indexing method, and vector database type (for example, Figure 5The navigation bar in the file includes five configuration nodes: the first configuration node, the second configuration node, the third configuration node, the fourth configuration node, and the fifth configuration node). The configuration process of the knowledge governance template may include, for example, the configuration of at least one of a layout recognition model, a text recognition tool, a slicer, a knowledge data storage method, an indexing method, and a vector database type; among multiple configuration nodes, the current configuration node is determined. For example, a user can select the current configuration node by triggering a switching instruction between different configuration nodes; based on a preset knowledge file, a preview information of the knowledge file after processing corresponding to the configuration information in the current configuration node is generated (taking the current configuration node as the configuration node where the layout recognition model is located as an example, the corresponding preview information of the knowledge file after processing may be a preview of the preset Preview information of the recognition result obtained by performing layout recognition on the knowledge file. Taking the current configuration node as a text recognition tool as an example, the preview information after the corresponding knowledge file is processed can be the preview information of the text data of multiple layout elements in the preset knowledge file); display the preview information after the knowledge file is processed, so that the user can check whether the processing result of the preset knowledge file meets the expectations. If it does not meet the expectations, the modification instruction can be triggered to modify the configuration information in the current configuration node (for example, if the segmentation of the preset knowledge file does not meet the expectations, the segmentation of at least one text data segment in the preset knowledge file can be adjusted or split or merged by triggering the modification instruction); in response to the knowledge governance template generation instruction, the configuration information in all configuration nodes is used to generate a preset knowledge governance template. It can be seen that the preset knowledge file is used to be parsed to reflect the actual effects of different configuration information in the preset knowledge governance template, so as to assist the user to complete the setting of the configuration information in the preset knowledge governance template more accurately and conveniently. In addition, by providing a clear navigation bar for the knowledge governance template configuration process, the user can configure the processing file in a streamlined manner, clearly understand the operation process of each step, and visually view the operation steps, thereby reducing the user's learning cost and difficulty of use.

[0097] In some embodiments of the present application, determining a target knowledge governance template from among multiple preset knowledge governance templates may include: obtaining multiple parsed historical knowledge files and the historical knowledge governance template adopted by each historical knowledge file, each historical knowledge governance template being a preset knowledge governance template; determining the similarity between the layout of the target knowledge file and the layout of each historical knowledge file; determining the target historical knowledge file from among multiple historical knowledge files based on the similarity, for example, the historical knowledge file with the highest similarity may be used as the target historical knowledge file, or all historical knowledge files with similarity greater than the preset similarity may be used as the target historical knowledge file; determining the target knowledge governance template based on the historical knowledge governance template adopted by the target historical knowledge file, for example, when there is only one target historical knowledge file, directly using the historical knowledge governance template adopted by the target historical knowledge file as the target knowledge governance template, if there are multiple target historical knowledge files at the same time, then among the historical knowledge governance templates adopted by multiple target historical knowledge files, the historical knowledge governance template adopted by the largest number of target historical knowledge files is used as the target knowledge governance template. It can be seen that the embodiment of the present application can automatically determine the knowledge governance template suitable for the target knowledge file through the matching between the target knowledge file and the historical knowledge file, thereby making the data processing for the knowledge base more convenient and accurate.

[0098] In some embodiments of the present application, determining the similarity between the layout of the target knowledge file and the layout of each historical knowledge file may include: performing layout recognition on each historical knowledge file to determine multiple layout elements in the historical knowledge file, and the layout area where each layout element is located, the multiple layout elements include image elements, at least one of table elements and text elements; determining the difference value between each historical knowledge file and the target knowledge file in at least one of the following dimensions: the number of image elements, the position and / or area of ​​the layout area where the image elements are located, the number of table elements, the position and / or area of ​​the layout area where the table elements are located, the number of text elements, the position and / or area of ​​the layout area where the text elements are located; based on the difference value, determining the similarity between the layout of the target knowledge file and the layout of each historical knowledge file, the similarity is negatively correlated with the difference value, that is, the smaller the difference value, the greater the similarity, so as to accurately match the knowledge governance template suitable for the target knowledge file.

[0099] In the second aspect, based on the data processing method for the knowledge base in the above embodiment, Figure 6 As shown, an embodiment of the present application provides a data processing device 600 for a knowledge base, and the data processing device 600 for a knowledge base is used to perform the steps in any embodiment of the above-mentioned data processing method for a knowledge base. Specifically, the data processing device 600 for a knowledge base may include:

[0100] The first acquisition module 601 is used to acquire the target knowledge file to be parsed;

[0101] A first determination module 602 is used to perform layout recognition on the target knowledge document to determine a plurality of layout elements in the target knowledge document and a layout area where each layout element is located, wherein the plurality of layout elements include at least one of a picture element and a table element and a text element;

[0102] The second determination module 603 is used to use the text recognition tool corresponding to each layout element to perform text recognition on the layout element to obtain text data of the layout element;

[0103] The first generating module 604 is used to generate knowledge data of the target knowledge file based on the text data of multiple layout elements and the layout areas where they are located, so as to obtain a knowledge base including the knowledge data.

[0104] In a third aspect, an embodiment of the present application provides a computer device that integrates any of the data processing devices for a knowledge base provided in the embodiments of the present application. The computer device includes a processor and a memory, and a computer program is stored in the memory. The computer program is configured to be executed by the processor to implement the data processing method for a knowledge base as described in any of the above embodiments, for example:

[0105] Acquire a target knowledge file to be parsed; perform layout recognition on the target knowledge file to determine multiple layout elements in the target knowledge file, and a layout area where each layout element is located, wherein the multiple layout elements include at least one of a picture element and a table element, and a text element; utilize a text recognition tool corresponding to each layout element to perform text recognition on the layout element to obtain text data of the layout element; generate knowledge data of the target knowledge file based on the text data of the multiple layout elements and the layout area where they are located, thereby obtaining a knowledge base including the knowledge data.

[0106] In a fourth aspect, an embodiment of the present application provides a computer device that integrates any one of the data processing devices for a knowledge base provided in an embodiment of the present application. Figure 7 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0107] The computer device may include one or more processing core processors 701, one or more computer readable storage medium storage units 702, a power supply 703, an input unit 704 and other components. Those skilled in the art will understand that Figure 7 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0108] The processor 701 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the storage unit 702 and calling data stored in the storage unit 702, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 701.

[0109] The storage unit 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the storage unit 702. The storage unit 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the storage unit 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the storage unit 702 may also include a memory controller to provide the processor 701 with access to the storage unit 702.

[0110] The computer device also includes a power supply 703 for supplying power to various components. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 703 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0111] The computer device may further include an input unit 704, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0112] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically, in the embodiment of the present application, the processor 701 in the computer device will load the executable file corresponding to the process of one or more application programs into the storage unit 702 according to the following instructions, and the processor 701 will run the application program stored in the storage unit 702, thereby realizing various functions, such as:

[0113] Acquire a target knowledge file to be parsed; perform layout recognition on the target knowledge file to determine multiple layout elements in the target knowledge file, and a layout area where each layout element is located, wherein the multiple layout elements include at least one of a picture element and a table element, and a text element; utilize a text recognition tool corresponding to each layout element to perform text recognition on the layout element to obtain text data of the layout element; generate knowledge data of the target knowledge file based on the text data of the multiple layout elements and the layout area where they are located, thereby obtaining a knowledge base including the knowledge data.

[0114] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. The computer-readable storage medium stores a computer program, and the computer program is configured to be executed by a processor to implement the data processing method for a knowledge base as described in any of the above items, for example:

[0115] Acquire a target knowledge file to be parsed; perform layout recognition on the target knowledge file to determine multiple layout elements in the target knowledge file, and a layout area where each layout element is located, wherein the multiple layout elements include at least one of a picture element and a table element, and a text element; utilize a text recognition tool corresponding to each layout element to perform text recognition on the layout element to obtain text data of the layout element; generate knowledge data of the target knowledge file based on the text data of the multiple layout elements and the layout area where they are located, thereby obtaining a knowledge base including the knowledge data.

[0116] In a sixth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the data processing method for a knowledge base as described in any one of the above items, for example:

[0117] Acquire a target knowledge file to be parsed; perform layout recognition on the target knowledge file to determine multiple layout elements in the target knowledge file, and a layout area where each layout element is located, wherein the multiple layout elements include at least one of a picture element and a table element, and a text element; utilize a text recognition tool corresponding to each layout element to perform text recognition on the layout element to obtain text data of the layout element; generate knowledge data of the target knowledge file based on the text data of the multiple layout elements and the layout area where they are located, thereby obtaining a knowledge base including the knowledge data.

[0118] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method for a knowledge base, characterized in that: The data processing method for the knowledge base includes: Obtain the target knowledge file to be parsed; Performing layout recognition on the target knowledge file to determine a plurality of layout elements in the target knowledge file and a layout area where each of the layout elements is located, wherein the plurality of layout elements include at least one of a picture element and a table element and a text element; Using a text recognition tool corresponding to each of the layout elements, perform text recognition on the layout elements to obtain text data of the layout elements; Based on the text data of the plurality of layout elements and the layout areas where they are located, the knowledge data of the target knowledge file is generated, thereby obtaining a knowledge base including the knowledge data.

2. The data processing method for a knowledge base according to claim 1, characterized in that: The step of generating the knowledge data of the target knowledge file based on the text data of the plurality of layout elements and the layout areas where the text data of the plurality of layout elements are located comprises: Segmenting the text data of each of the layout elements to obtain a plurality of text data segments in the layout element; Determine the adjacent text data segments of each of the text data segments by using the layout area where the plurality of layout elements are located; generating context information of each of the text data segments based on adjacent text data segments of the text data segment; The knowledge data of the target knowledge file is generated by using the plurality of text data segments and context information of the text data segments.

3. The data processing method for a knowledge base according to claim 2, characterized in that: The step of generating the knowledge data of the target knowledge file by using the plurality of text data segments and the context information of the text data segments comprises: generating at least one of a text summary and a text label for each of the text data segments; The knowledge data of the target knowledge document is generated by using a plurality of the text data segments, at least one of text summaries and text tags of the text data segments, and context information of the text data segments.

4. The data processing method for a knowledge base according to claim 1, characterized in that: The step of using a text recognition tool corresponding to each of the layout elements to perform text recognition on the layout elements to obtain text data of the layout elements includes: When the layout element is a text element, determining the number of layout columns of the text element; If the number of layout columns is multiple, the text element is divided into multiple column sub-elements; Rearrange the sub-elements in the multiple columns to obtain a column of target sub-elements; Using a text recognition tool corresponding to the text element, text recognition is performed on the column of target sub-elements to obtain text data of the text element.

5. The data processing method for a knowledge base according to claim 1, characterized in that: After using the text recognition tool corresponding to each of the layout elements to perform text recognition on the layout elements to obtain the text data of the layout elements, the method further includes: Displaying first preview information of the layout of the target knowledge document, and displaying second preview information of text data including a plurality of the layout elements; In response to a modification instruction for the second preview information, the second preview information and text data of at least one of the layout elements are modified.

6. The data processing method for a knowledge base according to claim 1, characterized in that: After obtaining the target knowledge file to be parsed, the method further includes: Determine a target knowledge management template from among a plurality of preset knowledge management templates, wherein the target knowledge management template includes at least one of a specified layout recognition model, a text recognition tool, a slicer, a knowledge data storage method, an indexing method, and a vector database type; Among them, the layout recognition model is used to perform layout recognition on the target knowledge file, the slicer is used to segment the text data of the layout elements, the knowledge data storage method includes associating and storing at least one of context information, text summary, and text label with the text data segment in the text data, and the indexing method and the vector database type are used to limit the generation rules of the knowledge base including the knowledge data.

7. The data processing method for a knowledge base according to claim 6, characterized in that: The preset knowledge governance template is generated in the following way: Get the preset knowledge file to be parsed; Displaying a navigation bar including a configuration process of a knowledge governance template, wherein the navigation bar includes a configuration node for at least one of a layout recognition model, a text recognition tool, a slicer, a knowledge data storage method, an indexing method, and a vector database type; Determine a current configuration node among the multiple configuration nodes; Based on the preset knowledge file, generate preview information after the knowledge file processing corresponding to the configuration information in the current configuration node; Displaying preview information of the knowledge file after processing, so as to modify the configuration information in the current configuration node; In response to the knowledge governance template generation instruction, the preset knowledge governance template is generated using the configuration information in all the configuration nodes.

8. The data processing method for a knowledge base according to claim 6, characterized in that: Determining a target knowledge governance template from among a plurality of preset knowledge governance templates includes: Acquire multiple parsed historical knowledge files and a historical knowledge governance template used by each of the historical knowledge files, each of the historical knowledge governance templates being a preset knowledge governance template; Determining the similarity between the layout of the target knowledge document and the layout of each of the historical knowledge documents; Based on the similarity, determining a target historical knowledge file from the plurality of historical knowledge files; Based on the historical knowledge governance template adopted by the target historical knowledge file, the target knowledge governance template is determined.

9. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is configured to be executed by the processor to implement the data processing method for a knowledge base according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to be executed by a processor to implement the data processing method for a knowledge base according to any one of claims 1 to 8.

Images