Method, apparatus, device and computer readable medium for processing document data

Abstract

The application discloses a method, device and equipment for processing document data and a computer readable medium, and relates to the technical field of big data. The specific implementation of the method comprises the following steps: obtaining unstructured data based on the identification of the unstructured data in a document database, and dividing the unstructured data into structured document data and text document data according to a file structure and a document content type; constructing a structured information format file by using the identification of the structured document data and the content of the structured document data, and establishing a text format file by using the identification of the text document data and the content of the text document data; embedding the structured information format file into the text format file, and then establishing a text format file of the unstructured data by using a document template; and storing the text format file in a data warehouse according to the version identification of the text format file, and storing structured data in the data warehouse, so that the data in the data warehouse can be searched. The implementation can improve the speed of searching unstructured data.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a method, apparatus, device, and computer-readable medium for processing document data. Background Technology

[0002] Currently, enterprise-level digital assets are mainly divided into two types: structured data and unstructured data. Structured data is typically stored in databases and queried and maintained through IT management systems. Unstructured data is usually stored in document libraries in document format.

[0003] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems: the data structure of unstructured data is irregular, making it difficult to quickly retrieve unstructured data. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and computer-readable medium for processing document data, which can improve the speed of retrieving unstructured data.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for processing document data is provided, comprising: Unstructured data is obtained from the document database based on the identifier of unstructured data, and the unstructured data is divided into structured document data and text document data according to the file structure and document content type. A structured information format file is constructed using the identifier and content of the structured document data, and a text format file is established using the identifier and content of the textual document data. After embedding the structured information format file into the text format file, a text format file of the unstructured data is created using a document template, and the text format file is stored in the data warehouse according to the version identifier of the text format file. The data warehouse stores structured data for data retrieval.

[0006] The process of obtaining unstructured data from a document database based on its identifier, and then classifying the unstructured data into structured document data and text-based document data according to file structure and document content type, includes: After obtaining unstructured data based on the identifier of unstructured data in the document database, the unstructured data is parsed to obtain the document structure of the unstructured data, and the unstructured data is split into multiple document structure data according to the document structure of the unstructured data. Based on the document content type corresponding to the unstructured data, the structured document data and the textual document data are obtained from the multiple document structure data.

[0007] The multiple document structure data include content titles, paragraphs, lists, image media, attribute groups, and attribute group lists; Constructing a structured information format file using the identifier and content of the structured document data, and establishing a text format file using the identifier and content of the textual document data, includes: The structured information format file is established using the identifiers of the attribute groups and the list of attribute groups, along with the contents of the attribute groups and the list of attribute groups. The text format file is constructed using the identifiers of the content title, the paragraphs, the list, and the image media, along with the content of the content title, the paragraphs, the list, and the image media.

[0008] The step of embedding the structured information format file into the text format file and then creating the text format file of the unstructured data using a document template includes: The text format file is parsed using a file parser to obtain a parsed text format file, and the structure format file is parsed using a structure parser to obtain a parsed structured information format file. The parsed structured information format file is embedded into the text format file using a document renderer to obtain a rendered file; The rendered file is populated according to the document template to create a text format file of the unstructured data.

[0009] The step of filling the rendered file according to the document template and creating a text format file of the unstructured data includes: Fill the parsed structured information format file in the rendered file according to the structured data identifier in the document template; Based on the unstructured data identifiers in the document template, populate the parsed text format file in the rendered file; The document template file after filling the file is used as the text format file of the unstructured data.

[0010] After storing the text format file in the data warehouse according to the version identifier of the text format file, the method further includes: The corresponding text format file is obtained based on multiple version identifiers of the text format file; Compare the text format files corresponding to different version identifiers to display the changes in the file formats corresponding to the different version identifiers.

[0011] The structured information format file includes a text format with structural attribute descriptions; the text format file includes a plain text file format.

[0012] According to a second aspect of the present invention, an apparatus for processing document data is provided, comprising: The segmentation module is used to retrieve unstructured data from the document database based on the identifier of unstructured data, and to segment the unstructured data into structured document data and text document data according to the file structure and document content type. A module is established to construct a structured information format file using the identifier and content of the structured document data, and to establish a text format file using the identifier and content of the textual document data. The processing module is used to embed the structured information format file into the text format file, create a text format file of the unstructured data using a document template, and store the text format file in a data warehouse according to the version identifier of the text format file. The data warehouse stores structured data for data retrieval.

[0013] According to a third aspect of the present invention, an electronic device for processing document data is provided, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0014] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0015] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method as described above in the embodiments of the present invention.

[0016] One embodiment of the above invention has the following advantages or beneficial effects: Unstructured data is obtained from a document database based on its identifier, and the unstructured data is divided into structured document data and textual document data according to file structure and document content type; a structured information format file is constructed using the identifier and content of the structured document data, and a text format file is established using the identifier and content of the textual document data; after embedding the structured information format file into the text format file, a text format file of the unstructured data is established using a document template, and the text format file is stored in a data warehouse according to its version identifier. The data warehouse stores structured data for data retrieval. Storing unstructured data according to version identifiers in the data warehouse improves the speed of unstructured data retrieval.

[0017] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0018] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main flow of a method for processing document data according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the process of dividing document data into structured document data and text document data according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of constructing structured information format files and text format files according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the process of creating a text format file of unstructured data using a document template according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the process for creating a text format file of unstructured data according to an embodiment of the present invention; Figure 6 This is a flowchart illustrating the changes in file formats corresponding to different version identifiers according to an embodiment of the present invention. Figure 7 This is a schematic diagram of the main structure of a device for processing document data according to an embodiment of the present invention; Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0019] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant national laws and regulations.

[0020] To address the difficulty in quickly retrieving unstructured data, the following technical solutions from the embodiments of the present invention can be adopted.

[0021] See Figure 1 , Figure 1 This is a schematic diagram of the main flow of a method for processing document data according to an embodiment of the present invention, in which unstructured data is stored in a data warehouse using version identifiers. For example... Figure 1 As shown in 100, the specific steps include:

[0022] S101. Obtain unstructured data from the document database based on the identifier of unstructured data, and classify the unstructured data into structured document data and textual document data according to the file structure and document content type.

[0023] In embodiments of the present invention, considering the irregular data structure of unstructured data, it is impossible to store unstructured data in a relational database. Typically, unstructured data is stored in a document database in a preset document format. To improve the speed of retrieving unstructured data, it can be divided into structured document data and text-based document data. The division is based on file structure and document content type.

[0024] See Figure 2 That is, 200. Figure 2 This is a flowchart illustrating the process of dividing document data into structured document data and text document data according to an embodiment of the present invention. Specifically, it includes the following steps.

[0025] S201. After obtaining unstructured data based on the identifier of unstructured data in the document database, parse the unstructured data to obtain the document structure of the unstructured data, and split the unstructured data into multiple document structure data according to the document structure of the unstructured data.

[0026] In an embodiment of the present invention, after obtaining unstructured data based on the identifier of unstructured data in the document database, the unstructured data is parsed to obtain a preset document structure of the unstructured data, and the unstructured data is split into multiple document structure data based on the preset document structure.

[0027] As an example, the pre-defined document structure of unstructured data is obtained by parsing the identifiers of unstructured data. For example, the identifiers of unstructured data include the identifiers of unstructured data.

[0028] As another example, unstructured data is parsed to obtain its document structure data according to the document structure identifiers of a preset document format. For example, document structure identifiers include content titles, paragraphs, tables, and images. The document structure data of the unstructured data includes: content title data, paragraph data, table data, and image data. The content title data includes multiple hierarchical content title data.

[0029] S202. Based on the document content type corresponding to the unstructured data, structured document data and textual document data are obtained from multiple document structure data.

[0030] Document structure data is a subset of unstructured data, and it has corresponding document content types. Specifically, the document content types in unstructured data are used as the document content types in document structure data. As an example, document content types include: text, list, image, and attributes.

[0031] Specifically, based on the document content type corresponding to the unstructured data, structured document data and textual document data are obtained from multiple document structure data.

[0032] As an example, in multiple file structure data sets, unstructured data corresponds to document content types including text, lists, or images, while structured file data is classified as text-based document data. Unstructured data corresponds to document content types including structure, while structured file data is classified as structured document data.

[0033] exist Figure 2 In this embodiment, after parsing the unstructured data, the unstructured data is further divided into structured document data and textual document data, and then targeted processing is performed on different types of data.

[0034] S102. Construct a structured information format file using the identifiers and content of structured document data, and establish a text format file using the identifiers and content of textual document data.

[0035] In embodiments of the present invention, corresponding processing methods are adopted for different document data. Specifically, a structured information format file is constructed using the identifier of structured document data and the content of textual document data, and a text format file is established using the identifier of textual document data and the content of textual document data.

[0036] See Figure 3 That is, 300. Figure 3 This is a schematic diagram illustrating the process of constructing structured information format files and text format files according to an embodiment of the present invention. Specifically, it includes the following steps.

[0037] S301. Establish a structured information format file using the identifiers of attribute groups and attribute group lists, along with the contents of attribute groups and attribute group lists.

[0038] In one embodiment of the present invention, multiple document structure data include content titles, paragraphs, lists, image media, attribute groups, and attribute group lists.

[0039] Content headings are used to identify sections of a document. As an example, content headings may include multi-level headings. Paragraphs are used to carry text information within a document's paragraphs. Lists include ordered lists and unordered lists. Lists are used to carry entry-style records. Image media are used to carry images or other media information.

[0040] Attribute groups are used to record multiple attribute values ​​that describe unstructured data. As an example, an attribute group includes: system basic attribute group, system name, system code, system level, and system function description.

[0041] An attribute group list consists of a list of multiple attribute groups used to hold table information within a document. As an example, an attribute group list might include a system context table and a list of attribute groups.

[0042] In embodiments of the present invention, content titles, paragraphs, lists, and image media are treated as structured document data; attribute groups and attribute group lists are treated as text format files.

[0043] Specifically, a structured information format file is created using attribute groups and a list of attribute groups. As an example, the identifiers of the attribute groups and the attribute group list are used as the identifiers of the structured information format file; the content of the attribute groups and the content of the attribute group list are used as the content of the structured information format file. That is, the structured information format file is created using the identifiers of the attribute groups and the attribute group list, along with their respective contents.

[0044] S302. Construct a text format file using the identifiers of content titles, paragraphs, lists, and image media, along with the content of the content titles, paragraphs, lists, and image media.

[0045] In embodiments of the present invention, a text format file is constructed using content titles, paragraphs, lists, and image media. Specifically, the identifiers of the text format file are established using the identifiers of the content titles, paragraphs, lists, and image media. Then, the text format file is constructed using the content content of the content titles, paragraphs, lists, and image media. That is, the text format file is constructed using the identifiers of the content titles, paragraphs, lists, and image media, along with their respective content.

[0046] exist Figure 3 In one embodiment, a corresponding format file is created based on the document data to achieve targeted processing of different types of document data.

[0047] In one embodiment of the present invention, the structured information format file includes a text format with structural attribute descriptions. As an example, the structured information format file includes one or more of the following: YAML format, JSON format, and XML format. YAML format is a readable data serialization format. Its design goal is to provide an easy-to-read and write data representation method, suitable for configuration files, data storage, and data exchange.

[0048] Text file formats include plain text file formats. As an example, text file formats include Markdown format. Markdown is a lightweight markup language designed to write documents in plain text and simplify typesetting and formatting.

[0049] S103. After embedding the structured information format file into the text format file, a text format file for unstructured data is created using a document template. The text format file is then stored in the data warehouse according to its version identifier. The data warehouse stores structured data so that data can be retrieved within the data warehouse.

[0050] In embodiments of the present invention, after the structured information format file and text format file of the unstructured data are determined, the structured information format file needs to be embedded into the text format file, and then the text format file of the unstructured data is created using a document template. The text format file of the unstructured data is marked with a version identifier, and thus the text format file is stored in the data warehouse according to its version identifier. In addition, structured data is stored in the data warehouse. Both structured and unstructured data are stored in the data warehouse, enabling the retrieval of both structured and unstructured data within the same data warehouse, i.e., data can be retrieved within the data warehouse.

[0051] See Figure 4 That is, 400. Figure 4This is a flowchart illustrating the process of creating a text-formatted file of unstructured data using a document template, according to an embodiment of the present invention. Specifically, it includes the following steps.

[0052] S401. Use a file parser to parse a text format file to obtain a parsed text format file, and use a structure parser to parse a structured format file to obtain a parsed structured information format file.

[0053] In embodiments of the present invention, a parser is used to parse the file to obtain the parsed document. Specifically, a file parser is used to parse a text format file to obtain a parsed text format file, and a structure parser is used to parse a structured format file to obtain a parsed structured information format file.

[0054] S402. Using a document renderer, the parsed structured information format file is embedded into a text format file to obtain a rendered file.

[0055] In embodiments of the present invention, to display unstructured data, a document renderer can be used to embed the parsed structured information format file into a text format file to obtain a rendered file. The rendered file includes not only the structured information format file but also the text format file.

[0056] S403. Fill the rendering file according to the document template to create a text format file for unstructured data.

[0057] In embodiments of the present invention, to improve the speed of retrieving unstructured data, the rendered file needs to be stored according to a document template. Specifically, the rendered file is filled according to the document template to create a text format file of the unstructured data.

[0058] See Figure 5 That is, 500. Figure 5 This is a schematic diagram illustrating the process of creating a text format file of unstructured data according to an embodiment of the present invention. Specifically, it includes the following steps.

[0059] S501. Fill the parsed structured information format file in the rendering file according to the structured data identifier in the document template.

[0060] Pre-set structured data identifiers in the document template, and populate the parsed structured information format file in the rendering file according to the structured data identifiers in the document template.

[0061] The document template is pre-set. The same template can be used to create text files for different types of unstructured data, thus achieving a unified format for text files.

[0062] S502. Based on the unstructured data identifier in the document template, populate the parsed text format file in the rendered file.

[0063] Populate the parsed text format file in the rendered file according to the unstructured data identifier in the document template.

[0064] S503. Use the document template file after filling the file as a text format file of unstructured data.

[0065] Both the parsed structured information format file and the parsed text format file are used to fill a document template file to obtain a document template file. This document template file is then used as a text format file for unstructured data.

[0066] exist Figure 5 In one embodiment, a text format file of unstructured data is created using a document template to achieve the structuring of unstructured data.

[0067] The following content is a diagram of a document template file. Here, `yaml property-group` refers to the property group, and `yamlproperty-group-list` refers to the list of property groups.

[0068] # Basic Information ```yaml property-group #[Basic Information about the Physical Subsystem] Number: XXX Full Chinese name: XXX Full English name: XXX ``` # Overall Design ## Design Goals - Scalability: 1. XXX 2. XXX - Security: 1. XXX 2. XXX # Application Architecture ## Contextual Relationships  1. Explanation 1 2. Explanation 2 ```yaml property-group-list #[Context Relationship Table] - Subsystem Name: xxx Relationship type: xxx Relationship description: xxx - Subsystem Name: xxx Relationship type: xxx Relationship description: xxx ```.

[0069] In embodiments of the present invention, the format of unstructured data in text format files is constrained by document templates. Furthermore, setting document templates enables the structuring of unstructured data.

[0070] See Figure 6 That is, 600. Figure 6 This is a flowchart illustrating changes in file formats corresponding to different version identifiers according to an embodiment of the present invention. Specifically, it includes the following steps.

[0071] S601. Obtain the corresponding text format file based on multiple version identifiers of the text format file.

[0072] In embodiments of the present invention, version identifiers are used to represent different versions of the same unstructured data in text format files.

[0073] The corresponding text format file is obtained based on its version identifier. Therefore, multiple text format files can be obtained based on their version identifiers.

[0074] As an example, the version identifiers for text format files include 1.0 and 2.0. Text format file 1 is obtained based on 1.0, and text format file 2 is obtained based on 2.0.

[0075] S602. Compare the text format files corresponding to different version identifiers to display the changes in the file format corresponding to different version identifiers.

[0076] All text files are built on the same document template. By comparing each item in the document template, different version identifiers for the corresponding text files are compared. Then, the changes in the file format corresponding to different version identifiers are displayed.

[0077] exist Figure 6 In one embodiment, version identifiers are used to compare text format files, enabling historical comparison of unstructured data.

[0078] In the above embodiments, unstructured data is obtained from the document database based on its identifier, and then divided into structured document data and text document data according to file structure and document content type. A structured information format file is constructed using the identifier and content of the structured document data, and a text format file is created using the identifier and content of the text document data. After embedding the structured information format file into the text format file, a text format file for the unstructured data is created using a document template, and the text format file is stored in a data warehouse according to its version identifier. The data warehouse stores structured data for data retrieval. Storing unstructured data according to version identifiers in the data warehouse improves the speed of unstructured data retrieval.

[0079] Text-formatted documents can be used to replace unstructured data, and a directory structure can also be used to store text-formatted documents in the data warehouse.

[0080] See Figure 7 , Figure 7 This is a schematic diagram of the main structure of a document data processing apparatus according to an embodiment of the present invention. The document data processing apparatus can implement a method for processing document data, such as... Figure 7 As shown in 700, the apparatus for processing document data specifically includes: The segmentation module 701 is used to obtain unstructured data in the document database based on the identifier of unstructured data, and to segment the unstructured data into structured document data and text document data according to the file structure and document content type. A module 702 is used to construct a structured information format file using the identifier and content of the structured document data, and to construct a text format file using the identifier and content of the textual document data. The processing module 703 is used to embed the structured information format file into the text format file, then create a text format file of the unstructured data using a document template, and store the text format file in a data warehouse according to the version identifier of the text format file. The data warehouse stores structured data for data retrieval.

[0081] In one embodiment of the present invention, the partitioning module 701 is specifically used to obtain unstructured data based on the identifier of unstructured data in the document database, parse the unstructured data to obtain the document structure of the unstructured data, and split the unstructured data into multiple document structure data according to the document structure of the unstructured data. Based on the document content type corresponding to the unstructured data, the structured document data and the textual document data are obtained from the multiple document structure data.

[0082] In one embodiment of the present invention, the plurality of document structure data includes content titles, paragraphs, lists, image media, attribute groups, and attribute group lists; The module 702 is specifically used to establish the structured information format file by using the identifiers of the attribute group and the attribute group list, along with the contents of the attribute group and the attribute group list. The text format file is constructed using the identifiers of the content title, the paragraphs, the list, and the image media, along with the content of the content title, the paragraphs, the list, and the image media.

[0083] In one embodiment of the present invention, the processing module 703 is specifically used to parse the text format file with a file parser to obtain a parsed text format file, and to parse the structured format file with a structure parser to obtain a parsed structured information format file; The parsed structured information format file is embedded into the text format file using a document renderer to obtain a rendered file; The rendered file is populated according to the document template to create a text format file of the unstructured data.

[0084] In one embodiment of the present invention, the processing module 703 is specifically used to fill the parsed structured information format file in the rendered file according to the structured data identifier in the document template; Based on the unstructured data identifiers in the document template, populate the parsed text format file in the rendered file; The document template file after filling the file is used as the text format file of the unstructured data.

[0085] In one embodiment of the present invention, the processing module 703 is further configured to obtain the corresponding text format file based on multiple version identifiers of the text format file; Compare the text format files corresponding to different version identifiers to display the changes in the file formats corresponding to the different version identifiers.

[0086] In one embodiment of the present invention, the format of the structured information format file includes a text format with structural attribute descriptions; the format of the text format file includes a plain text file format.

[0087] Figure 8 An exemplary system architecture 800 is shown, which can be applied to a method or apparatus for processing document data according to embodiments of the present invention.

[0088] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, and 803, a network 804, and a server 805. Network 804 serves as the medium for providing communication links between terminal devices 801, 802, and 803 and server 805. Network 804 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0089] Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0090] Terminal devices 801, 802, and 803 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0091] Server 805 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 801, 802, and 803 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information and product information - for example only) to the terminal devices.

[0092] It should be noted that the method for processing document data provided in the embodiments of the present invention is generally executed by server 805, and correspondingly, the device for processing document data is generally located in server 805.

[0093] It should be understood that Figure 8 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0094] A computing program product according to an embodiment of the present invention includes a computer program, which, when executed by a processor, implements the method for processing document data provided in an embodiment of the present invention.

[0095] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device of the present invention. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0096] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0097] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0098] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0099] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a partitioning module, a creation module, and a processing module. The names of these modules do not necessarily limit the module itself; for example, a partitioning module can also be described as "used to partition unstructured data into structured document data and textual document data based on file structure and document content type."

[0102] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: Unstructured data is obtained from the document database based on the identifier of unstructured data, and the unstructured data is divided into structured document data and text document data according to the file structure and document content type. A structured information format file is constructed using the identifier and content of the structured document data, and a text format file is established using the identifier and content of the textual document data. After embedding the structured information format file into the text format file, a text format file of the unstructured data is created using a document template, and the text format file is stored in the data warehouse according to the version identifier of the text format file. The data warehouse stores structured data for data retrieval.

[0103] According to the technical solution of this invention, unstructured data is obtained from a document database based on the identifier of unstructured data, and the unstructured data is divided into structured document data and textual document data according to the file structure and document content type. A structured information format file is constructed using the identifier and content of the structured document data, and a text format file is established using the identifier and content of the textual document data. After embedding the structured information format file into the text format file, a text format file of the unstructured data is created using a document template, and the text format file is stored in a data warehouse according to its version identifier. The data warehouse stores structured data for data retrieval. Storing unstructured data according to version identifiers in the data warehouse improves the speed of unstructured data retrieval.

[0104] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0105] It should be noted that the collection, analysis, use, transmission, and storage of user personal information involved in the technical solution of this invention all comply with relevant laws and regulations, are used for legitimate and reasonable purposes, and are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once this user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0106] When applicable, including in certain relevant applications, data deidentification is used to protect user privacy, such as by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the specific address level), controlling how data is stored, and / or other methods of deidentification.

Claims

1. A method for processing document data, characterized in that, include: Unstructured data is obtained from the document database based on the identifier of unstructured data, and the unstructured data is divided into structured document data and text document data according to the file structure and document content type. A structured information format file is constructed using the identifier and content of the structured document data, and a text format file is established using the identifier and content of the textual document data. After embedding the structured information format file into the text format file, a text format file of the unstructured data is created using a document template, and the text format file is stored in the data warehouse according to the version identifier of the unstructured data text format file. The data warehouse stores structured data for data retrieval. The step of embedding the structured information format file into the text format file and then creating the unstructured data text format file using a document template includes: using a file parser to parse the text format file to obtain a parsed text format file, and using a structure parser to parse the parsed text format file to obtain a parsed structured information format file. The parsed structured information format file is embedded into the text format file using a document renderer to obtain a rendered file; the rendered file is then filled according to the document template to create a text format file of the unstructured data.

2. The method for processing document data according to claim 1, characterized in that, The process of obtaining unstructured data from a document database based on its identifier, and then classifying the unstructured data into structured document data and text-based document data according to file structure and document content type, includes: After obtaining unstructured data based on the identifier of unstructured data in the document database, the unstructured data is parsed to obtain the document structure of the unstructured data, and the unstructured data is split into multiple document structure data according to the document structure of the unstructured data. Based on the document content type corresponding to the unstructured data, the structured document data and the textual document data are obtained from the multiple document structure data.

3. The method for processing document data according to claim 2, characterized in that, The multiple document structure data include content titles, paragraphs, lists, image media, attribute groups, and attribute group lists; Constructing a structured information format file using the identifier and content of the structured document data, and establishing a text format file using the identifier and content of the textual document data, includes: The structured information format file is established using the identifiers of the attribute groups and the list of attribute groups, along with the contents of the attribute groups and the list of attribute groups. The text format file is constructed using the identifiers of the content title, the paragraphs, the list, and the image media, along with the content of the content title, the paragraphs, the list, and the image media.

4. The method for processing document data according to claim 1, characterized in that, The step of filling the rendered file according to the document template and creating a text format file of the unstructured data includes: Fill the parsed structured information format file in the rendered file according to the structured data identifier in the document template; Based on the unstructured data identifiers in the document template, populate the parsed text format file in the rendered file; The document template file after filling the file is used as the text format file of the unstructured data.

5. The method for processing document data according to claim 1, characterized in that, After storing the text format file in the data warehouse according to the version identifier of the text format file, the method further includes: The corresponding text format file is obtained based on multiple version identifiers of the text format file; Compare the text format files corresponding to different version identifiers to display the changes in the file formats corresponding to the different version identifiers.

6. The method for processing document data according to claim 1, characterized in that, The structured information format file includes a text format with structural attribute descriptions; the text format file includes a plain text file format.

7. An apparatus for processing document data, characterized in that, include: The segmentation module is used to retrieve unstructured data from the document database based on the identifier of unstructured data, and to segment the unstructured data into structured document data and text document data according to the file structure and document content type. A module is established to construct a structured information format file using the identifier and content of the structured document data, and to establish a text format file using the identifier and content of the textual document data. The processing module is used to embed the structured information format file into the text format file, create a text format file of the unstructured data through a document template, and store the text format file in the data warehouse according to the version identifier of the unstructured data text format file. The data warehouse stores structured data so that data can be retrieved in the data warehouse. The processing module is further configured to: use a file parser to parse the text format file to obtain a parsed text format file, and use a structure parser to parse the parsed text format file to obtain a parsed structured information format file; The parsed structured information format file is embedded into the text format file using a document renderer to obtain a rendered file; the rendered file is then filled according to the document template to create a text format file of the unstructured data.

8. An electronic device for processing document data, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images