Method for analyzing multi-source heterogeneous data, computer device and storage medium

Abstract

The application discloses a multi-source heterogeneous data analysis method, computer equipment and a storage medium, and relates to the technical field of data analysis.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for parsing multi-source heterogeneous data, a computer device, and a storage medium. Background Technology

[0002] Knowledge base construction refers to using retrieval-enhanced generative techniques to store information from specific scenarios in a way that large models can understand. This allows users to query questions based on the knowledge base, and the large models then use the knowledge stored in the knowledge base to provide corresponding answers. Knowledge base construction mainly includes processes such as source data cleaning, knowledge extraction, knowledge storage, question-and-answer query, knowledge retrieval, and knowledge output.

[0003] Existing knowledge base systems based on large models rarely have detailed processing methods for document parsing in specific domains. After trial use, it was found that the existing knowledge base construction framework has the following shortcomings in multi-source document parsing: (1) It cannot meet the parsing of multi-source heterogeneous data in the rail transit industry; (2) When parsing different types of documents, the original file is easily lost, and special formats are prone to errors; (3) The block segmentation strategy is mostly based on a fixed number of characters, which easily cuts off the integrity of the original data; (4) When parsing content, there is no strict strategy to ensure that the table title or image title and content are divided into the same block when encountering tables and images; (5) When using a block segmentation method with a fixed number of characters and a sliding window, the final block result does not have the title information of the original document in each block; (6) When parsing question-and-answer pairs in question & answer format, the block segmentation method with a fixed number of characters and a sliding window will cause different types of questions to be divided into the same block, or even questions and answers to be divided into different blocks. That is, the existing technology is inaccurate in parsing multi-source heterogeneous data.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method, computer device, and storage medium for parsing multi-source heterogeneous data, aiming to solve the technical problem of inaccurate parsing of multi-source heterogeneous data in the prior art.

[0006] To achieve the above objectives, the present invention provides a method for parsing multi-source heterogeneous data, the method comprising: Get the file to be parsed; Based on the format type of the file to be parsed, the file to be parsed is converted to a different format file. The converted format file is read using a document loader to obtain structured content; The structured content is divided into blocks according to its content type.

[0007] Preferably, in the method for parsing multi-source heterogeneous data, the format types include Word documents, Excel documents, PDF documents, TXT documents, and CSV documents; Accordingly, the step of converting the format of the file to be parsed according to its format type to obtain a converted format file includes: When the format of the file to be parsed is a Word document, Excel document, or PDF document, the file to be parsed will be converted into a Markdown format file; When the format of the file to be parsed is a TXT document or a CSV document, the file to be parsed is directly used as the format file.

[0008] Preferably, in the method for parsing multi-source heterogeneous data, when the format type of the file to be parsed is a Word document, an Excel document, or a PDF document, converting the file to be parsed into a Markdown format file includes: When the format of the file to be parsed is a Word document, an Excel document, or a PDF document, the file to be parsed is preprocessed. The preprocessing includes filtering the header and footer content of the file to be parsed, extracting the title information and paragraph structure of the file to be parsed, and extracting the images and tables of the file to be parsed. Convert the preprocessed file to be parsed into a Markdown format file.

[0009] Preferably, in the method for parsing multi-source heterogeneous data, when an image is extracted from the file to be parsed, the step of converting the preprocessed file to be parsed into a Markdown format file includes: Obtain the location of the image and the link address of the image; The preprocessed file to be parsed is converted into a Markdown format file, and the link address of the image is inserted into the format file according to the position of the obtained image.

[0010] Preferably, in the method for parsing multi-source heterogeneous data, when a table is extracted from the file to be parsed, the step of converting the preprocessed file to be parsed into a Markdown format file includes: Convert the table into a two-dimensional list; Based on the generated two-dimensional list, determine whether the table has multi-level headers, spans multiple pages, or merged cells; When multiple levels of headers exist, merge the multiple levels of headers; When there are tables spanning multiple pages, merge the tables that span those pages. When merged cells exist, split the merged cells and fill the empty cells that appear after the split; The preprocessed file to be parsed is converted into a Markdown format file, and the processed table is inserted into a preset position in the format file.

[0011] Preferably, in the method for parsing multi-source heterogeneous data, the content types of the structured content include structured text, images, tables, and QA pairs; Accordingly, the step of dividing the structured content into blocks according to the content type of the structured content includes: When the structured content is an image, the content of the image is identified, the text corresponding to the image is output and embedded in the image storage location, and the output text is used as structured text; When the structured content is structured text, it is divided into blocks according to the title and hierarchical relationship of the structured text; When the structured content is a table, the table content is reshaped, each piece of information in the table is output, and all the information output from each table is formed into a table block. When the structured content is a QA pair, each QA pair is treated as a block.

[0012] Preferably, in the method for parsing multi-source heterogeneous data, when the structured content is a table, the table content is reshaped, each piece of information in the table is output, and all the information output from each table is formed into a table block, including: When the structured content is a table, the content of the first row in the table is used as the table header, and "table header + meaning of each column" is output, forming a table block with all the information output from each table.

[0013] Preferably, in the method for parsing multi-source heterogeneous data, when the structured content is structured text, the step of segmenting the data according to the title and hierarchical relationship of the structured text includes: When the structured content is structured text, it is initially divided into blocks according to the hierarchical relationship of the headings; Based on the combination of Chinese and English punctuation marks and line breaks, the text under the same title is divided into two blocks.

[0014] To achieve the above objectives, the present invention provides a computer device, characterized in that it comprises: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the above-described method for parsing multi-source heterogeneous data.

[0015] To achieve the above objectives, the present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the above-described method for parsing multi-source heterogeneous data.

[0016] The present invention has at least the following beneficial effects: The present invention provides a method for parsing multi-source heterogeneous data, which involves obtaining a file to be parsed; converting the format of the file to be parsed according to its format type to obtain a converted format file; reading the converted format file using a document loader to obtain structured content; and dividing the structured content into blocks according to its content type, thereby enabling more accurate parsing of multi-source heterogeneous data.

[0017] Furthermore, the multi-source heterogeneous data parsing method provided by this invention supports the parsing of different types of original documents in the rail transit industry; ensures 100% retention of original document information, accurately restores format information, and does not lose useful information; ensures that complete sentences or paragraphs are not split into different blocks by designing a method that combines punctuation and line break judgment; parses the original document content, uses regular expressions to match table or image titles, ensuring that table content or image content is divided into separate blocks, while also ensuring that table titles and image titles are in table blocks and image blocks respectively; uses regular expressions to match different levels of headings in the document, ensuring that each block of content, whether it is plain text, tables, or images, can be linked to the heading information of its content in the original document.

[0018] Furthermore, compared to existing solutions that segment text content using fixed characters and sliding windows when parsing CSV files, this invention loads the text using a text loader and designs a text reconstruction strategy to reconstruct the original text into the format of "Question: Answer". Then, it designs regular expressions to match questions and answers, grouping each line of question-answer pairs into a block, thus avoiding different questions being grouped into the same block or questions and answers being grouped into different blocks.

[0019] Furthermore, the multi-source heterogeneous data parsing method provided by this invention can effectively identify and parse content from various formats, including Word documents ending in .doc, Excel documents ending in .xls, Word documents ending in .docx, Excel documents ending in .xlsx, PDF documents, TXT documents, and CSV documents. Compared to existing frameworks, many frameworks do not support older document content. This invention accurately parses and restores the original content of documents in various formats, preserving the original document format, including heading structure, plain text paragraphs, table styles, and image positions. When segmenting data, this invention uses a combination of punctuation and line breaks. Compared to existing frameworks that use fixed characters combined with sliding windows for segmentation, this invention ensures the sentence and paragraph integrity of the original information. When segmenting Markdown format files, this invention designs… The invention employs separate detection strategies for table and image areas to ensure that table or image content is segmented separately, preventing it from being mixed with ordinary text and thus diluting the information. This ensures the semantic independence of the segmented information. For Markdown format files, the invention designs a directory-enhanced segmentation strategy and a title linking strategy, ensuring that when ordinary text, table areas, and image areas are segmented, they can all be linked to the corresponding title content, guaranteeing the integrity of semantic information. For question-and-answer pairs generated in specific industry domains or scenarios, the invention designs a strategy to divide each question-and-answer pair into a separate block. Compared to fixed character and sliding window segmentation methods, this ensures the independence between each question and the integrity of the same question. This approach facilitates question-only vectorization when designing vectorization strategies, reducing information noise caused by excessively long answers, ensuring the accuracy of data entry, and improving the hit rate during knowledge retrieval. Attached Figure Description

[0020] Figure 1 A schematic diagram of an embodiment of the method for parsing multi-source heterogeneous data provided by the present invention; Figure 2 A schematic diagram of yet another embodiment of the method for parsing multi-source heterogeneous data provided by the present invention; Figure 3 A schematic diagram of the computer device provided by the present invention.

[0021] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0023] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] In this embodiment of the invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0026] In this invention, unless otherwise stated, directional terms such as "upper," "lower," "top," and "bottom" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction of the component itself; similarly, for ease of understanding and description, "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not intended to limit this invention.

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the embodiments of the present invention to facilitate a better understanding of the invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0028] The existing knowledge base construction framework has the following shortcomings in multi-source document parsing: (1) It cannot meet the parsing of multi-source heterogeneous data in the rail transit industry; (2) When parsing different types of documents, the original file is easily lost and special formats are prone to errors; (3) The block segmentation strategy is mostly fixed character count, which easily cuts off the integrity of the original data; (4) When parsing content, when encountering tables and pictures, there is no strict strategy to ensure that the table title or picture title and content are divided into the same block; (5) When using the block segmentation method with fixed character count and sliding window, the final block result does not have the title information of the original document in each block; (6) When parsing question and answer pair files in question & answer format, the block segmentation method with fixed character count and sliding window will cause different types of questions to be divided into the same block, or even questions and answers to be divided into different blocks.

[0029] The aforementioned six shortcomings result in several problems: First, the constructed knowledge base lacks source data, leading to incomplete question-and-answer functionality and an inability to provide answers. Second, the knowledge base content contains errors, resulting in inaccurate question-and-answer responses that are irrelevant to the question, impacting the industry application of large-scale models. Third, table or image information is mixed with plain text, diluting semantic information and reducing retrieval accuracy. Fourth, content under the same heading level is divided into different blocks, especially in intelligent operations and maintenance scenarios. Since common fault symptoms occur under the same heading level, this leads to incomplete search results, failing to output all information. Fifth, when searching a knowledge base containing question-and-answer pairs, the system cannot accurately answer user questions.

[0030] To address the aforementioned issues, this embodiment relates to a method for parsing multi-source heterogeneous data, which can be applied to computer devices. These computer devices can be desktop computers, tablet computers, laptops, or other electronic devices with data processing capabilities. In other embodiments, they can also be other electronic devices with data processing capabilities, and no specific limitations are imposed here. Figure 1 and Figure 2 The flowchart illustrates the method for parsing multi-source heterogeneous data provided by the present invention.

[0031] In step S100, the file to be parsed is obtained. The file to be parsed can be, but is not limited to, a Word document, an Excel document, a PDF document, a TXT document, or a CSV document. Taking Word documents and Excel documents as examples, they include docx and doc formats, and Excel documents include xlsx and xls formats.

[0032] In step S200, the file to be parsed is converted according to its format type to obtain a converted format file. It should be noted that different formats are converted to create different format files.

[0033] For example, document formats include Word, Excel, PDF, TXT, and CSV.

[0034] When the format of the file to be parsed is a Word document, Excel document, or PDF document, the file to be parsed will be converted into a Markdown format file.

[0035] When the file to be parsed is a TXT or CSV document, it is used directly as the format file. Since TXT and CSV documents contain plain text and have no special formatting, there is no need to convert them to Markdown format; instead, the file to be parsed is used directly as the format file, avoiding the fragmentation of the document content.

[0036] In some implementations, the Word document can be converted to a PDF document first, and then converted to the format file.

[0037] Markdown is a lightweight markup language designed to allow people to write documents in an easy-to-read and easy-to-write plain text format and easily convert them into valid HTML documents. Common Markdown formatting includes elements such as headings, text styles, lists, links, images, and tables. For example, headings include level 1, level 2, level 3, and so on. Lists include item 1, item 2, item 3, and so on; item 2 includes sub-item 1, sub-item 2, and so on.

[0038] In addition, to avoid interference from redundant information, preprocessing is required during the format conversion of the file to be parsed. Specifically, when the format of the file to be parsed is a Word document, Excel document, or PDF document, converting the file to be parsed into a Markdown format file includes steps S210 and S220.

[0039] In step S210, when the format of the file to be parsed is a Word document, Excel document, or PDF document, the file to be parsed is preprocessed. This preprocessing includes filtering the header and footer content of the file to be parsed, extracting title information and paragraph structure from the file to be parsed, and extracting images and tables from the file to be parsed. Filtering the header and footer content of the file to be parsed can be done, but is not limited to, using fixed-height cropping and regular expression matching to precisely filter the header and footer content.

[0040] Extracting the title information from the file to be parsed can be done, but is not limited to, identifying the title based on font size; in other implementations, it can also be done by parsing the original document's directory information to restore the document title.

[0041] Extracting the paragraph structure from the file to be parsed can be achieved by using, but is not limited to, layout detection algorithms to accurately identify the original document paragraph structure.

[0042] In step S220, the preprocessed file to be parsed is converted into a Markdown format file. It should be noted that when extracting images from the file to be parsed, step S220 includes steps S221 and S222.

[0043] In step S221, the location of the image and the link address of the image are obtained.

[0044] In step S222, the preprocessed file to be parsed is converted into a Markdown format file, and the link address of the image is inserted into the format file according to the position of the obtained image.

[0045] Traditional image processing methods can lead to image loss. This invention avoids this by embedding the image's link address into a Markdown format file. Furthermore, the image can be retrieved later by uploading it to a knowledge base. In some implementations, image recognition can be performed directly while the file to be parsed is being converted to Markdown format. The specific recognition method is as follows: detect and record the image position; use a large visual model to recognize the image content; output the recognized content as Markdown text and add it to the Markdown format file according to the image's position in the original file. Additionally, prompts can be used during image content recognition to ensure the completeness of the output content and more accurate format restoration.

[0046] For example, if the image contains a table, by combining a visual model and a cue word project, the cue words are adjusted multiple times to ensure that the model can accurately understand the content and format of the original image, and finally output the content in table form according to the original format.

[0047] For example, if the image is a diagram of a fault in the rail transit industry, then prompts can be used to allow the large model to describe the fault shown in the image in text form.

[0048] In other implementations, the visual big data model and prompt word engineering can be used to identify the image content when the structured content is segmented. The specific step in which the identification is performed is not limited here.

[0049] When a table is extracted from the file to be parsed, step S220 includes steps S223 to S228.

[0050] In step S223, the table is converted into a two-dimensional list. Some related technologies directly convert tables into text, which leads to the loss of the table's structural information. This invention avoids this loss by converting the table into a two-dimensional list.

[0051] In step S224, based on the formed two-dimensional list, it is determined whether the table has multi-level headers, spans multiple pages, or merged cells.

[0052] In step S225, when multiple levels of headers exist, the headers are merged. Merging multiple levels of headers ensures that the table information remains complete and the format is well-organized when the table content is divided into blocks.

[0053] In step S226, if there are cross-page tables, merge the cross-page tables.

[0054] In step S227, if there are merged cells, the merged cells are split and the empty cells that appear after the split are filled.

[0055] In step S228, the preprocessed file to be parsed is converted into a Markdown format file, and the processed table is inserted into a preset position in the format file.

[0056] The table processing, including merging tables across pages, splitting merged cells, and filling empty cells after splitting, ensures the accuracy of the table format and the completeness of the content when converting the file to be parsed into a Markdown format file.

[0057] In step S300, the document loader is used to read the converted format file to obtain structured content.

[0058] In step S400, the structured content is divided into blocks based on its content type. It should be noted that due to the special nature of Markdown format files, document titles can be identified using "#", and the block-splitting strategy is designed sequentially according to the file's top-to-bottom order. A directory-enhanced block-splitting strategy is adopted, where each level of heading and its content are grouped into the same block until a heading line starting with "#" is encountered, at which point block-splitting restarts. This directory-enhanced strategy ensures that each block retains complete content from all levels of headings, guaranteeing the integrity of the information's semantics. Within headings at the same level, table detection logic is added, using regular expressions to detect table titles and content, and a reconstruction strategy is designed to reconstruct the table content into the format of "[Header: Content]", enhancing the table information while ensuring that the final generated table block contains the table title. Within headings at the same level, image detection logic is added, using regular expressions to detect image titles and content, ensuring that the final generated image block contains the image title. When designing a block strategy for plain text under the same heading, a combination of Chinese and English punctuation and line breaks is used to ensure that sentences within the same paragraph are not split, and that text within the same sentence is not separated. In the Markdown document block strategy, a heading linking strategy is also added, meaning that regardless of whether it's a plain text block, table block, or image block, the final block contains the content of its respective heading levels.

[0059] For TXT format files, a combination of Chinese and English punctuation marks and paragraph marks is used for segmentation. This segmentation method ensures the integrity of paragraph data and the integrity of the same sentence.

[0060] Regarding the processing of CSV files, the CSV files involved in this invention can be, but are not limited to, records of questions and answers generated during production and maintenance in a specific industry. For the intelligent operation and maintenance profession in the rail transit industry, this refers to question-and-answer pairs of equipment fault phenomena and solutions. For this type of CSV file, the chunking strategy can be, but is not limited to: when loading via a text loader, designing a text reconstruction strategy to reconstruct the original text into a "question: answer" format; then designing a text chunking strategy, using regular expressions to match questions and answers, grouping each line of question-and-answer pairs into a chunk, avoiding different questions being grouped into the same chunk or questions and answers being grouped into different chunks. Simultaneously, in subsequent vectorization operations, designing a vectorization scheme only for questions, avoiding excessive text noise introduced by overly long answers, thereby improving the hit rate of question queries.

[0061] The content types of structured content may include, but are not limited to, structured text, images, tables, and QA pairs; correspondingly, step S400 includes steps S410 to S440.

[0062] In step S410, when the structured content is an image, the content of the image is identified, the text corresponding to the image is output and embedded in the image storage location, and the output text is used as structured text.

[0063] In step S420, when the structured content is structured text, it is divided into blocks according to the headings and hierarchical relationships of the structured text. Specifically, step S420 includes steps S421 and S422.

[0064] In step S421, when the structured content is structured text, it is initially divided into blocks according to the hierarchical relationship of the headings; In step S422, the text under the same title is divided into blocks based on the combination of Chinese and English punctuation marks and line breaks.

[0065] In step S430, when the structured content is a table, the table content is reshaped, each piece of information in the table is output, and all the information output from each table is combined into a table block. Specifically, when the structured content is a table, the content of the first row in the table content is used as the table header, and "table header + meaning of each column" is output, and all the information output from each table is combined into a table block.

[0066] In step S440, when the structured content is a QA pair, each QA pair is treated as a block.

[0067] To achieve the above objectives, the present invention also provides a computer device, such as... Figure 3 As shown, the computer device includes at least one processor 301; and a memory 302 communicatively connected to at least one processor 301; wherein the memory 302 stores instructions executable by at least one processor 301, the instructions being executed by at least one processor 301 to enable at least one processor 301 to perform the above-described method for parsing multi-source heterogeneous data.

[0068] The memory 302 and processor 301 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 301 and memory 302 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 301 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 301.

[0069] Processor 301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 302 can be used to store data used by processor 301 during operation.

[0070] To achieve the above objectives, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for parsing multi-source heterogeneous data.

[0071] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0072] Obviously, the embodiments described above are merely some, not all, embodiments of the present invention. Based on the embodiments of the present invention, those skilled in the art can make other variations or modifications without creative effort, and all such variations or modifications should fall within the scope of protection of the present invention.

Claims

1. A method for parsing multi-source heterogeneous data, characterized in that, include: Get the file to be parsed; Based on the format type of the file to be parsed, the file to be parsed is converted to a different format file. The converted format file is read using a document loader to obtain structured content; The structured content is divided into blocks according to its content type.

2. The method for parsing multi-source heterogeneous data as described in claim 1, characterized in that, The format types include Word documents, Excel documents, PDF documents, TXT documents, and CSV documents; Accordingly, the step of converting the format of the file to be parsed according to its format type to obtain a converted format file includes: When the format of the file to be parsed is a Word document, Excel document, or PDF document, the file to be parsed will be converted into a Markdown format file; When the format of the file to be parsed is a TXT document or a CSV document, the file to be parsed is directly used as the format file.

3. The method for parsing multi-source heterogeneous data as described in claim 2, characterized in that, When the format of the file to be parsed is a Word document, Excel document, or PDF document, the process of converting the file to be parsed into a Markdown format file includes: When the format of the file to be parsed is a Word document, an Excel document, or a PDF document, the file to be parsed is preprocessed. The preprocessing includes filtering the header and footer content of the file to be parsed, extracting the title information and paragraph structure of the file to be parsed, and extracting the images and tables of the file to be parsed. Convert the preprocessed file to be parsed into a Markdown format file.

4. The method for parsing multi-source heterogeneous data as described in claim 3, characterized in that, When an image is extracted from the file to be parsed, the step of converting the preprocessed file to be parsed into a Markdown format file includes: Obtain the location of the image and the link address of the image; The preprocessed file to be parsed is converted into a Markdown format file, and the link address of the image is inserted into the format file according to the position of the obtained image.

5. The method for parsing multi-source heterogeneous data as described in claim 3, characterized in that, When a table is extracted from the file to be parsed, the step of converting the preprocessed file to be parsed into a Markdown format file includes: Convert the table into a two-dimensional list; Based on the generated two-dimensional list, determine whether the table has multi-level headers, spans multiple pages, or merged cells; When multiple levels of headers exist, merge the multiple levels of headers; When there are tables spanning multiple pages, merge the tables that span those pages. When merged cells exist, split the merged cells and fill the empty cells that appear after the split. The preprocessed file to be parsed is converted into a Markdown format file, and the processed table is inserted into a preset position in the format file.

6. The method for parsing multi-source heterogeneous data as described in claim 1, characterized in that, The content types of the structured content include structured text, images, tables, and Q&A pairs; Accordingly, the step of dividing the structured content into blocks according to the content type of the structured content includes: When the structured content is an image, the content of the image is identified, the text corresponding to the image is output and embedded in the image storage location, and the output text is used as structured text; When the structured content is structured text, it is divided into blocks according to the title and hierarchical relationship of the structured text; When the structured content is a table, the table content is reshaped, each piece of information in the table is output, and all the information output from each table is formed into a table block. When the structured content is a QA pair, each QA pair is treated as a block.

7. The method for parsing multi-source heterogeneous data as described in claim 6, characterized in that, When the structured content is a table, the table content is reshaped, each piece of information in the table is output, and all the output information from each table is combined into a table block, including: When the structured content is a table, the content of the first row in the table is used as the table header, and "table header + meaning of each column" is output, forming a table block with all the information output from each table.

8. The method for parsing multi-source heterogeneous data as described in claim 6, characterized in that, When the structured content is structured text, it is divided into blocks according to the headings and hierarchical relationships of the structured text, including: When the structured content is structured text, it is initially divided into blocks according to the hierarchical relationship of the headings; Based on the combination of Chinese and English punctuation marks and line breaks, the text under the same title is divided into two blocks.

9. A computer device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method for parsing multi-source heterogeneous data as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for parsing multi-source heterogeneous data as described in any one of claims 1 to 8.

Images