Text analysis method and device based on corpus cleaning, medium and product

Abstract

The embodiment of the invention relates to the field of natural language processing, and discloses a text analysis method and device based on corpus cleaning, a medium and a product. Comprising the following steps: acquiring text data, and preprocessing the text data to obtain first corpus data; performing text cleaning on the first corpus data, including performing character cleaning through a natural language processing model and performing semantic cleaning through a machine learning algorithm to obtain second corpus data; performing information extraction on the second corpus data, including extracting text information through a text analysis engineering package; identifying and segmenting a picture and a table in the second corpus data, and extracting picture information and table information; and reconstructing the text information, the picture information and the table information according to a logic structure of the text data to obtain third corpus data. The problem that the document data analysis capability is insufficient can be at least solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a text parsing method, device, medium and product based on corpus cleaning. Background Art

[0002] With the rapid development of information technology, the types and quantity of digital documents have exploded. Electronic documents in formats such as PDF and EPUB are widely used in academic research, business activities, and daily affairs. These documents usually contain complex structural information, such as text, pictures, tables, formulas, etc. However, existing technologies have many shortcomings in processing these complex documents.

[0003] The existing technology has poor compatibility with documents in various formats, and the internal structures of documents in different formats vary greatly, which makes it difficult for existing text parsing methods to effectively identify and extract text, pictures, tables and other elements when processing multi-format documents. This poses a great obstacle to subsequent text analysis, data extraction and reuse. In many scenarios, noise characters, blank lines and format characters in documents affect the accuracy of text parsing. In addition, it is difficult for existing technologies to accurately identify and process contents such as tables, pictures and formulas in documents, resulting in insufficient integrity and accuracy of extracted information. There is a lack of targeted information reconstruction methods. Even if text, tables, pictures and other information can be successfully extracted, existing technologies usually lack the ability to reconstruct this information according to the original logical structure, resulting in the output results being difficult to use directly in subsequent operations. For example, the results generated by traditional parsing methods are not in a uniform format and cannot support secondary processing and system integration. Summary of the invention

[0004] One purpose of the present application is to provide a text parsing method, device, medium and product based on corpus cleaning, at least to solve the problem of insufficient document data parsing capability.

[0005] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0006] In a first aspect, some embodiments of the present application further provide a text parsing method based on corpus cleaning, comprising obtaining text data, preprocessing the text data, including formatting the text data and converting it into a unified internal representation format; performing segmentation processing to separate corpus elements to obtain first corpus data;

[0007] Performing text cleaning on the first corpus data, including character cleaning through a natural language processing model and semantic cleaning through a machine learning algorithm, to obtain second corpus data;

[0008] Extracting information from the second corpus data, including extracting text information through a text parsing engineering package; extracting image information and table information by identifying and segmenting images and tables in the second corpus data;

[0009] The text information, picture information and table information are reconstructed according to the logical structure of the text data to obtain third corpus data.

[0010] In a second aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0011] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.

[0012] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the method described above when executed by a processor.

[0013] Compared with related technologies, the solution provided by the embodiment of the present application can clean invalid information in complex format documents and specifically extract and reconstruct text, pictures, tables, formulas, etc. This will greatly improve the efficiency of document parsing and data utilization, and provide stronger technical support for information management, data mining and automated processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0015] Figure 1 A flowchart of a text parsing method based on corpus cleaning provided according to an embodiment of the present application;

[0016] Figure 2 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] First embodiment

[0019] The first embodiment of the present application relates to a text parsing method based on corpus cleaning. Figure 1 As shown, the method may include the following steps:

[0020] S101, obtaining text data, preprocessing the text data, including formatting the text data and converting it into a unified internal representation format; performing segmentation processing to separate the corpus elements to obtain first corpus data. By preprocessing the text data, documents of different formats are converted into a unified internal representation format, ensuring that subsequent processing steps can be operated based on a consistent format. At the same time, segmentation processing can effectively separate corpus elements such as text, pictures, and tables to obtain clear first corpus data. This process improves the compatibility with different document formats and ensures the accuracy and efficiency of subsequent data processing.

[0021] S102, performing text cleaning on the first corpus data, including character cleaning through a natural language processing model and semantic cleaning through a machine learning algorithm, to obtain second corpus data. Character cleaning of the text is performed through a natural language processing model to effectively remove noise characters, blank lines and format characters, thereby improving the purity of the text data. Semantic cleaning is performed in combination with a machine learning algorithm to further improve the semantic consistency of the text and ensure that the extracted text has a high accuracy. The cleaned second corpus data lays a high-quality foundation for subsequent information extraction and effectively reduces the interference of irrelevant information on the parsing process.

[0022] S103, extracting information from the second corpus data, including extracting text information through a text parsing engineering package; extracting image information and table information by identifying and segmenting images and tables in the second corpus data. The text parsing engineering package can accurately extract structured text information from the second corpus data, including paragraphs, titles, lists, etc. In addition, by identifying and segmenting images and tables, image information and table information are accurately extracted. This step ensures that multiple types of content are extracted from complex documents, effectively addresses the limitation of traditional methods that are difficult to handle non-text elements, thereby improving the integrity and diversity of the parsing results.

[0023] S104, reconstructing the text information, image information and table information according to the logical structure of the text data to obtain the third corpus data. The extracted text, image and table information are reconstructed according to the logical structure of the original document, and the generated third corpus data can ensure that the data is organized in the same way as the original document. This process makes the final output Markdown format and JSON format files not only easy to edit and store, but also convenient for secondary processing and use in other systems. Through logical structure reconstruction, this method improves the operability and application value of information.

[0024] It is not difficult to find that compared with the related art, the solution provided in the embodiment of the present application utilizes a text parsing engineering package combined with a custom algorithm to achieve rapid parsing and information extraction of documents such as PDF and EPUB; through multi-level text cleaning and semantic parsing, the accuracy and high quality of the extracted information are ensured; it is applicable to a variety of document formats, and can handle complex document structures, especially the parsing ability of tables and pictures is significantly improved.

[0025] Second embodiment

[0026] The second embodiment of the present application relates to a text parsing method based on corpus cleaning. The second implementation is an improvement on the first embodiment, and the specific improvements are:

[0027] The text data includes PDF format files and EPUB format files. In order to ensure the accuracy and consistency of subsequent text cleaning and information extraction, the text data in these two formats need to be preprocessed first. The preprocessing step adopts corresponding parsing methods for documents in different formats, and finally converts the document content into a unified Markdown (md) and JSON format, which is convenient for subsequent structured processing and reuse.

[0028] Get text data in PDF format and preprocess it to ensure the accuracy and completeness of subsequent information extraction and reconstruction. Get text data in PDF format from the specified file path. PDF files usually contain complex layouts and multiple elements, including text, pictures, tables, and formulas. In order for these elements to be accurately parsed, they must be preprocessed first.

[0029] After obtaining the PDF file, the bbox (bounding box) technology is used to identify and divide the text area in the document. Bbox recognition is to determine the position of the text on the page and its area boundaries by detecting the boundaries of each text block (bounding box) on the page. The bbox can accurately divide each independent text block on the page, ensuring that the boundaries and relative positions of each text block can be identified during subsequent extraction. Different types of content such as text, tables, and pictures on the page are initially distinguished to provide a basis for subsequent processing.

[0030] Although the text in a PDF file is visually presented as ordered paragraphs or lists, due to the typesetting characteristics of PDF, its text content is not always stored in a natural reading order within the file. In order to ensure that the extracted text conforms to the original logical structure of the document, text order detection is performed. The relative position of the text blocks that have been divided by bbox is detected to determine the arrangement order between the text blocks; based on the typesetting order of the text blocks and the direction of the text (from left to right, from top to bottom, etc.), the reading order of the text is reconstructed to ensure that the extracted text content is consistent with the original order of the document.

[0031] Mathematical formulas in PDF documents are usually stored in the form of images or complex vector graphics, and traditional text parsing methods have difficulty in accurately extracting these contents. To solve this problem, a formula detection model is used to identify and extract mathematical formulas in documents. The formula detection model can analyze specific symbol combinations and typesetting features in the document to locate and identify the formula area; once the formula area is detected, the system uses the formula recognition algorithm to extract these formulas into editable text format or LaTeX format to ensure that the content of the mathematical formula can be accurately presented during subsequent reconstruction.

[0032] Table detection and structured parsing, table parsing in PDF is a difficult point in document processing. Tables are often composed of multiple rows and columns and contain rich structured data. The table detection module is used to automatically identify the table area in the PDF document, identify the outer border of the table and the internal rows, columns and cell boundaries; extract the content in each cell, and perform structured parsing on the row and column positions of the cell to generate structured data consistent with the original table; the parsed table data can be output in a structured format (such as CSV or JSON) to ensure that the table content can be accurately reproduced in subsequent processing.

[0033] Image detection and segmentation,The image information contained in PDF documents is very important for many application scenarios.In order to accurately extract the image content from PDF, perform image detection and segmentation,use image processing algorithms to scan the image area in the document,identify the image boundaries and positions; extract and segment the image,save it separately as an independent image file,and record its position in the document for subsequent reconstruction.

[0034] After completing the recognition and extraction of various elements such as text, formulas, tables, and pictures, different text partitions are merged. The purpose of this step is to integrate the previously separated text blocks according to their positions and logical order to generate a complete corpus file. The text areas identified by bbox are reorganized according to the text order detection results; non-text information such as formulas, tables, and pictures are embedded in the corresponding text positions to ensure that all elements in the document are arranged in the original logical order to form a complete corpus structure. Through this series of preprocessing steps, complex elements in PDF documents (such as text, formulas, tables, pictures, etc.) are accurately extracted and organized into structured data, ensuring that subsequent text cleaning, information extraction, and reconstruction can be carried out efficiently.

[0035] Obtain text data in EPUB format and preprocess it to ensure the accuracy and completeness of subsequent information extraction and reconstruction. An EPUB file is a compressed package that follows the ZIP format, and its internal structure contains multiple directories and files. After decompressing the EPUB file, you will get a folder structure that mainly includes the following parts: the OEBPS directory stores the main content of the document, including HTML files, CSS style sheets, and image resources, providing the actual content and format of the document; the META-I NF directory contains the metadata of the EPUB file, such as the container.xml file, which indicates the storage location of the main content and other related metadata. During the decompression process, you can use appropriate tools (such as the unzip command or Python's zipfile module) to decompress the EPUB file into a complete folder structure containing these subdirectories and files for further processing and use.

[0036] Parse the content.opf file in the EPUB file, which contains a<man i fest> section, which lists all resource items and their file names. By reading the content.opf file <item>Tags can be used to obtain the file names and order information of each HTML file. Then, based on this information, read the HTML files one by one in order from the decompressed OEBPS directory. When reading files, make sure to follow <item>Each HTML file is processed in the order defined in the tag to maintain the logical structure and reading order of the document. This process can be implemented programmatically, for example, using a Python script to read and process the content.opf file, and then load the HTML files in the file system in the specified order, thereby ensuring the accurate extraction of the document content and the integrity of subsequent processing.

[0037] When cleaning HTML files, first use HTML parsing tools (such as Beautiful Soup) to read the HTML file content. These tools can effectively parse HTML tags and extract text content. Next, perform tag cleaning, which includes two main steps: first, remove all redundant whitespace characters and formatting characters to ensure the clarity of the text content; second, remove irrelevant tags, such as<scr i pt> and<sty l e> , these tags usually contain content that is irrelevant to the final text and should be removed from the document. Finally, the original content structure, including titles and paragraphs, is preserved during the cleaning process. <h1> and< / h1> <h2>) should be converted to the corresponding Markdown heading format to maintain the document's structural information and reading order.

[0038] In the process of saving the cleaned content in Markdown and JSON format, generating a Markdown file includes organizing the text content. According to the cleaned text content and title information, the content is organized into chapters and paragraphs. The Markdown format uses a simple syntax to mark the structure of the document, including titles, paragraphs, and lists. For example, use # to represent a first-level title, ## to represent a second-level title, and - or * to represent an unordered list.

[0039] Write to a Markdown file, create a new Markdown file, and write the organized text content to the file. Make sure to write the content into the Markdown file paragraph by paragraph in the cleaned order and structure to maintain the logical structure of the original document. Each title and paragraph should be formatted according to the Markdown syntax so that the document has a clear hierarchy and structure.

[0040] Generating a JSON file involves extracting structured information from the cleaned content, such as titles, chapter contents, and chapter order. Organize this information into an appropriate structure, such as using dictionaries or lists to represent chapters and contents. Convert the organized structured information into JSON format. JSON format is a lightweight data exchange format suitable for storing metadata and structured information of documents. Use a JSON library (such as Python's json module) to write data into a JSON file. Make sure the structure of the JSON file is clear and can accurately reflect the chapters and content order of the document.

[0041] Corpus elements include text, pictures, tables, and formulas; character cleaning is performed through natural language processing models, including the removal of noise characters, blank lines, and format characters. All major corpus elements, including text, pictures, tables, and formulas, are extracted from text data. The extracted text content is separated from pictures, tables, and formulas to ensure that each element is processed in an independent structure.

[0042] Image processing: extract and save image files to ensure that they are correctly referenced in the final document. Image processing tools can be used to optimize image quality or perform format conversion. Table processing: perform structured processing on the extracted table data and convert the table data into a suitable format, such as CSV or JSON. Use table parsing tools to clean and format the table to ensure the accuracy and consistency of the data. If the formula is in LaTeX format, use a LaTeX parser to convert it into a readable format to ensure that the formula is presented correctly and clearly in the final document.

[0043] Use appropriate natural language processing models (such as pre-trained BERT or GPT models) to process and clean text content. These models are able to identify and process noise characters and irregular formats in text. Use NLP models to identify and remove noise characters in text, such as garbled characters, special symbols, or other unnecessary characters. For example, the model can detect and replace non-printing characters or incorrectly encoded characters in text.

[0044] The model analyzes the line structure in the text and automatically removes blank lines and unnecessary separators. The model can identify which lines do not actually contain useful information and clean them up; it cleans up format characters in the text, such as extra spaces, tabs (Tab) or line breaks (\n). The model can automatically remove these extra format characters and maintain the coherence of the text by learning the general pattern of text formatting.

[0045] Save the cleaned text content in a standard format, such as Markdown or plain text files. Ensure that the text content is clean, noise-free, and in a uniform format. Update the cleaned data into structured files, such as document metadata in JSON format, to maintain the integrity and consistency of the information.

[0046] Semantic cleaning by machine learning algorithm includes: using OCR model combination, formula model and table model for semantic cleaning; the OCR model combination includes Chinese PPOCR model and English marker OCR model.

[0047] In the document processing process, semantic cleaning through machine learning algorithms can significantly improve the accuracy and clarity of text. The following is a detailed specific example showing how to use OCR model combination, formula model and table model for semantic cleaning.

[0048] The text content in the image is identified and cleaned through the OCR model to ensure accurate text extraction and semantic clarity, including the Chinese PPOCR model and the English MarkerOCR model to process texts in different languages. The Chinese PPOCR model is suitable for extracting Chinese text, including processing Chinese characters, punctuation marks, etc.; the English MarkerOCR model is suitable for extracting English text, processing English characters, numbers and punctuation marks.

[0049] The formula model identifies and cleans the formulas in the document to ensure accurate expression and readability of the formulas. Formula recognition models (such as MathPix or LaTeX parsers) are used to process formulas and extract formulas from images or text. The formula model can recognize the structure and symbols of the formula and convert it into LaTeX format or other parseable formats. The extracted formulas are standardized into consistent LaTeX code to ensure accurate display of the formulas. Mathematical libraries (such as SymPy) are used to verify the correctness of the formulas and correct possible recognition errors. Formula parsing stores the content of the table in md text format and saves the original formula image at the same time, so that the parsing model can be repeatedly parsed to improve accuracy after continuous upgrades.

[0050] Table model, extract and clean table data to ensure data structure and accuracy. Use table recognition models (such as Tabu la or DeepDeSRT) to process tables, extract table areas from documents, and identify the row and column structure of tables. When parsing the table, store the table content in md text format and save the original table image at the same time, so that the parsing model can be repeatedly parsed to improve accuracy after continuous upgrade.

[0051] Integrate the cleaned OCR text, formulas, and table data and organize them according to the document structure. Generate Markdown and JSON files: Save the organized content in Markdown format to retain the structure and content of the document, and save the structured information in JSON format to facilitate subsequent data processing and application integration. Through the above steps, combined with the OCR model combination, formula model, and table model, efficient semantic cleaning of documents can be achieved to ensure the accuracy and clarity of text content.

[0052] The third corpus data is in Markdown format and is imported into a database or application system through an API interface for secondary processing and utilization. In the process of processing documents, the cleaned corpus data is saved in Markdown format and then imported into a database or application system through an API interface for further processing and utilization. Convert the semantically cleaned and formatted document content into Markdown format. Markdown files include structured content such as document titles, paragraphs, lists, and pictures to ensure the logic and readability of the document. Design an API interface that supports receiving text data in Markdown format. The API interface should include necessary parameters, such as document title, content, creation time, etc., so that they can be correctly stored and processed in a database or application system. Define API protocols and data formats, such as data transmission protocols in JSON format, to ensure that Markdown data can be correctly parsed and processed.

[0053] Use programming languages ​​(such as Python, JavaScript) to write scripts or applications, and send the data of the Markdown file to the designed API interface through HTTP requests. The request should contain the content of the Markdown file and related metadata. After receiving the data, the API interface stores the Markdown content and metadata in the specified database or application system. The database can be a relational database (such as MySQL, PostgreSQL) or a non-relational database (such as MongoDB), and the specific choice depends on the application requirements.

[0054] In the database or application system, the imported Markdown data is processed again, and the Markdown content is further analyzed and processed, such as sentiment analysis and topic modeling. The Markdown content is converted into other formats, such as HTML or PDF, to meet different display requirements. The Markdown data is integrated with the data in other systems to provide a more comprehensive view of information. The processed data is used in application systems, such as content management systems, online editors, report generation tools, etc., to realize actual application functions.

[0055] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0056] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 2 An exemplary structural diagram of the electronic device is disclosed. Figure 2 As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0057] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0058] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0059] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0060] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0061] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0062] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0063] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0064] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0065] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0066] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0067] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive so lid stated isk (SSD)), etc.

[0068] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0069] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first" and "second" are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0070] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.< / h2> < / item> < / item>

Claims

1. A text parsing method based on corpus cleaning, characterized in that: The method comprises: Acquire text data, pre-process the text data, including formatting the text data and converting it into a unified internal representation format; perform segmentation processing to separate corpus elements to obtain first corpus data; Performing text cleaning on the first corpus data, including character cleaning through a natural language processing model and semantic cleaning through a machine learning algorithm, to obtain second corpus data; Extracting information from the second corpus data, including extracting text information through a text parsing engineering package; extracting image information and table information by identifying and segmenting images and tables in the second corpus data; The text information, picture information and table information are reconstructed according to the logical structure of the text data to obtain third corpus data.

2. The method according to claim 1, characterized in that The text data includes PDF format files and EPUB format files.

3. The method according to claim 2, characterized in that The preprocessing of the text data comprises: The text data of the PDF file is subjected to bbox recognition and division, sequence detection, formula detection, table detection, image detection and merging of text partitions, and then converted into md+json format after combination.

4. The method according to claim 2, characterized in that: The preprocessing of the text data comprises: The text data of the EPUB format file is parsed to obtain an html combination file; the corresponding html file is read according to the html combination file and the sequence file; the tag of the html file is removed and the content sequence and title attributes are retained, and the html file is assembled and packaged into the md+json format.

5. The method according to any one of claims 1 to 4, characterized in that: The corpus elements include text, pictures, tables and formulas; The character cleaning is performed by a natural language processing model, including removing noise characters, blank lines, and format characters.

6. The method according to claim 5, characterized in that The semantic cleaning by machine learning algorithm includes: using OCR model combination, formula model and table model for semantic cleaning; the OCR model combination includes Chinese PPOCR model and English marker OCR model.

7. The method according to claim 6, characterized in that The method further comprises: The third corpus data is in Markdown format and is imported into a database or application system through an API interface for secondary processing and utilization.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 7.

9. A computer readable medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images