Segmented data storage method, device, computer equipment and storage medium

Abstract

The application discloses a segmented data storage method and device, computer equipment and a storage medium. The method comprises the following steps: in the writing process, the basic structure and the extended structure of a file are constructed, and the attributes and the content of each object are written in sequence, the reference table and the root object information are recorded, and the file tail mark is added; in the reading process, the information of the reference table is obtained from the file tail part, the root object is positioned, each page object and the content contained are traversed according to the reference relationship, the text or the coordinate information is read and jumped according to the need until the whole file is processed. Through the method of the application, the data processing efficiency can be improved, the memory occupation can be reduced, and the diversified needs in different application scenarios can be met.

Description

Technical Field

[0001] This invention relates to data processing methods, and more specifically to segmented data storage methods, apparatus, computer equipment, and storage media. Background Technology

[0002] Currently widely used data storage formats, such as JSON, XML, and YAML, typically require loading the entire dataset into memory at once when performing data write and read operations. This means that whether converting a programming object to a data format or vice versa (reading), the entire amount of data must be processed. This is particularly problematic in smart contract applications, where the process involves parsing PDF contract files, including identifying elements such as titles, paragraphs, tables, and images, along with their coordinate information. Because these contract files may contain a large amount of image and coordinate data, the generated data volume is exceptionally large, placing enormous memory pressure on the program during write or read operations, and in severe cases, even causing service crashes.

[0003] Furthermore, different application scenarios have varying data requirements, but current technology requires each application to first load all relevant data into memory, even if only a portion of the data is actually used. This leads to several major problems: existing data formats lack effective data sharding mechanisms to support partial data writing and reading, easily causing excessive consumption of memory resources when processing large-scale data, affecting service stability. It also prevents selective data reading based on demand, requiring the loading of all data for each operation, wasting system resources and reducing processing speed. Additionally, it does not support filtering based on content element type, making it difficult to obtain specific types of data.

[0004] Therefore, it is necessary to design a new method to improve data processing efficiency, reduce memory usage, and meet the diverse needs of different application scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a segmented data storage method, apparatus, computer equipment and storage medium.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a segmented data storage method, comprising:

[0007] During the writing process, the basic and extended structures of the file are constructed, and the attributes and contents of each object are written in sequence. The reference table and root object information are recorded, and a file end marker is added.

[0008] During the reading process, the reference table information is obtained from the end of the file to locate the root object. Then, based on the reference relationship, each page object and its contents are traversed. If necessary, the file is jumped to read text or coordinate information until the entire file is processed.

[0009] The further technical solution is as follows: During the writing process, the basic structure and extended structure of the file are constructed, and the attributes and contents of each object are written sequentially. The reference table and root object information are recorded, and a file end marker is added, including:

[0010] Create a new EPG file;

[0011] Write header information into the EPG file;

[0012] Record the object offset and write the object attributes and specific object content in the EPG file;

[0013] After all objects are written to the EPG file, the object reference table is recorded and a file end marker is written.

[0014] The further technical solution is as follows: recording the object offset and writing the object attributes and specific object content in the EPG file includes:

[0015] The EPG file records object attributes and specific object content;

[0016] Write the information of the root object into the EPG file;

[0017] Record the starting position of the reference table relative to its contents in the EPG file.

[0018] The further technical solution is as follows: after all objects are written to the EPG file, the object reference table is recorded and a file end marker is written, including:

[0019] Write all the reference relationships between objects into the EPG file, and then write the end-of-file information into the EPG file to mark the end of the file writing process.

[0020] The further technical solution is as follows: During the reading process, information from the reference table is obtained from the end of the file to locate the root object, and then each page object and its contents are traversed according to the reference relationship. If necessary, text or coordinate information is read and the entire file is processed. This includes:

[0021] During the reading process, the EPG file is opened and loaded, and relevant information is read from the end of the file;

[0022] Move the file pointer to the beginning of the reference table and read all the information in the reference table;

[0023] Based on the information in the reference table, move the file pointer to the position of the root object and read the information of the root object;

[0024] Parse the properties of the root object to obtain reference information for all shard objects;

[0025] Process page objects one by one, jump to the beginning of the next page object, read all attributes and content information of the current page object, and for text content objects in the page object, determine whether to read coordinate information as needed, and adjust the file pointer position as needed for processing.

[0026] The further technical solution is as follows: Processing page objects one by one, jumping to the beginning of the next page object, reading all attributes and content information of the current page object, and for text content objects within the page object, determining whether to read coordinate information as needed, and adjusting the file pointer position as required, includes:

[0027] Based on the information in the reference table, move the file pointer to the beginning of the next page object to obtain the current page object;

[0028] Read all properties and content information of the current page object;

[0029] Move the file pointer to the beginning of the first text content object in the current page object;

[0030] Read information from the text content object;

[0031] Extract and read specific text content;

[0032] Determine whether the currently processed text content requires coordinate information;

[0033] If the text being processed requires coordinate information, move the file pointer to the corresponding coordinate object position and read the coordinate object's information;

[0034] Adjust the file pointer position based on the offset from the end of the text line, and continue processing the next text line or object;

[0035] If the text content being processed does not require coordinate information, then determine whether the text content object being processed is the last element in the page object;

[0036] If the currently processed text content object is not the last element in the page object, then the file pointer is moved to the starting position of the first text content object in the current page object;

[0037] If the currently processed text content object is the last element in the page object, then determine whether the currently processed page object is the last page in the EPG file;

[0038] If the currently processed page object is not the last page in the EPG file, then according to the information in the reference table, move the file pointer to the beginning of the next page object;

[0039] If the currently processed page object is the last page in the EPG file, proceed to the end step.

[0040] The further technical solution is as follows: the reference table records the offset of all objects, represented in hexadecimal, with one data entry per row, the row number being the object number, and the index starting from 1.

[0041] The present invention also provides a segmented data storage device, comprising:

[0042] The write unit is used to construct the basic and extended structures of the file during the writing process, sequentially write the attributes and contents of each object, record the reference table and root object information, and add a file end marker.

[0043] The reading unit is used to retrieve the reference table information from the end of the file to locate the root object during the reading process, and then traverse each page object and its contents according to the reference relationship, jumping to read text or coordinate information as needed until the entire file is processed.

[0044] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0045] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0046] The advantages of this invention compared to existing technologies are as follows: By constructing and sequentially recording the basic structure, extended structure, object attributes, and content of a file during writing, while maintaining a reference table and root object information, and adding a marker at the end of the file, this invention enables rapid data access and processing by quickly locating the root object from the end of the file and efficiently traversing each page object and its content through reference relationships during reading. This method reduces unnecessary memory usage because only specific required objects or pages need to be loaded, rather than the entire file. Furthermore, due to its flexible data structure design, it can adapt to different application scenarios, providing excellent support for both managing a large number of small files and efficiently reading and writing large files, thereby improving overall data processing efficiency and system response speed.

[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram illustrating an application scenario of the segmented data storage method provided in this embodiment of the invention;

[0050] Figure 2 This is a flowchart illustrating the segmented data storage method provided in an embodiment of the present invention.

[0051] Figure 3 A schematic diagram of the basic file structure provided for embodiments of the present invention;

[0052] Figure 4 This is an illustration of the PDF information file structure provided in the embodiments of the present invention;

[0053] Figure 5 A schematic block diagram of a segmented data storage device provided in an embodiment of the present invention;

[0054] Figure 6 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0057] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0058] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0059] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the segmented data storage method provided in an embodiment of the present invention. Figure 2 This is a schematic flowchart illustrating the segmented data storage method provided in this embodiment of the invention. This segmented data storage method is applied to a server that interacts with a terminal. By constructing the basic and extended structures of a file, it records object attributes, content, and reference relationships during the writing process and adds markers to the end of the file. This allows for rapid location of the root object during reading and efficient traversal of each page object and its content based on reference relationships. This method not only optimizes the data organization and reduces unnecessary memory usage, but also supports diverse needs in different application scenarios by allowing on-demand navigation to read text or coordinate information. Specifically, it manages the dependencies between objects through an effective reference table, allowing programs to directly access the required data without loading the entire file, thereby improving processing efficiency. Simultaneously, it provides flexible solutions for content requiring special processing (such as coordinate information), ensuring the accuracy and efficiency of data processing, and is suitable for various complex data processing scenarios.

[0060] Figure 2 This is a flowchart illustrating the segmented data storage method provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S120.

[0061] S110. During the writing process, construct the basic structure and extended structure of the file and write the attributes and contents of each object in sequence, record the reference table and root object information, and add a file end marker.

[0062] In this embodiment, the basic structure refers to the basic framework of the file, including the file header, object definitions, reference table, and file footer. The file header begins with a specific identifier (such as %epi), followed by a version number. Each object is preceded by a line specifying its object number and the keyword obj, and is marked with endobj at the end of the object. The reference table records the offsets of all objects to help quickly locate their positions. The file footer contains an offset pointing to the start of the reference table and an end marker.

[0063] The extended structure further designed different types of object formats, such as text objects, table objects, and location objects. Each object has its own specific attributes and content organization methods to facilitate more efficient storage and retrieval of information.

[0064] In one embodiment, step S110 described above may include steps S111 to S114.

[0065] S111. Create a new EPG file.

[0066] In this embodiment, the first step is to create a new Electronic Program Guide (EPG) file. This file will serve as the underlying container for storing data. In practice, this may involve opening a new file stream or creating a new file object.

[0067] S112. Write header information into the EPG file.

[0068] In this embodiment, header information is then written at the beginning of the EPG file. This header information typically includes a file format identifier (such as %epi), a version number, and other information crucial for identifying the file type and version.

[0069] S113. Record the object offset and write the object attributes and specific object content in the EPG file.

[0070] In one embodiment, step S113 described above may include steps S1131 to S1133.

[0071] S1131. Record the object attributes and specific object content in the EPG file.

[0072] In this embodiment, for each object, its attributes and specific content are first recorded. Attributes describe the object's characteristics, such as type and size, while the specific content contains the object's concrete data. For example, a text object may contain specific text content and its coordinate offset.

[0073] S1132. Write the information of the root object into the EPG file.

[0074] In this embodiment, the root object is the top-level node of the entire data structure, containing references to other objects. By writing the root object's information to a file, it ensures that subsequent traversal can start from the root object and continue through all related objects. For example, the root object may point to multiple page objects.

[0075] S1133. Record the starting position of the reference table relative to its contents in the EPG file.

[0076] In this embodiment, the reference table records the location information of all objects in the file, which is crucial for quickly locating specific objects. In this step, the starting position of the reference table relative to the file content needs to be recorded so that the reference table can be accurately located during subsequent reading.

[0077] S114. After all objects are written to the EPG file, record the object reference table and write the file end marker.

[0078] In this embodiment, the reference table records the offsets of all objects in hexadecimal, with one data entry per line. The line number is the object number, and the index starts from 1.

[0079] Write all the reference relationships between objects into the EPG file, and then write the end-of-file information into the EPG file to mark the end of the file writing process.

[0080] Once all objects have been written to the EPG file, the next step is to record the reference relationships between all objects and write them to the reference table. Additionally, end-of-file information needs to be added to the end of the file to indicate the end of the file writing process.

[0081] The above steps complete the entire process from creating a new EPG file to writing the necessary header information, object attributes and content, reference table, and finally, file footer information. This systematic approach not only ensures effective data storage but also improves the efficiency of data retrieval and processing.

[0082] In this embodiment, please refer to Figure 3 This document presents the structure and description of the basic Episode format. The file header begins with %epi, followed by a space and a version number, for example: %epi 1.0. A reference table is used to mark the offsets of all objects, represented in hexadecimal. Each line represents one data entry, and the line number is the object number, starting from 1. The file footer consists of two lines: the first line is start_xref [offset], where [offset] is the starting offset of the trailer section, represented in hexadecimal; the second line is %%EOF, indicating the end of the file. The file content consists of a series of objects, each preceded by a separate description containing the object number and the keyword obj, followed by the keyword endobj. Object attributes are placed between << and >>, each attribute occupying one line, in the format <attribute name><attribute content>. Attribute names are fixed to name types, while attribute content can be any basic or composite type. The data inside the object is located between >> and endobj, not bound by any specific data type, and can even contain zip data streams.

[0083] The epi basic format defines five basic types: integers and real numbers (such as 42 and 3.1415), strings (enclosed in double quotes and escaped using JSON format), names (starting with a forward slash / , used for keys in dictionaries and various other purposes), boolean values ​​(represented by the keywords true and false), and null objects (represented by the keyword null). In addition, three composite types are defined: arrays (ordered collections containing other objects), dictionaries (consisting of unordered key-value pairs, where keys are strings and values ​​can be any basic or composite type), and indirect references (forming links from one object to another, identified by an 'R' at the end).

[0084] Please see Figure 4 This presents the epi-pdfx format, which stores PDF content based on the epi base format extension. This format further defines the specific format of the file content objects. It includes the following types of objects:

[0085] Root object: The number is always 1, the property is / Pages, and the content is an array that identifies references to each page number object.

[0086] The Page object records PDF page information, with attributes including / Type page (identifying the page type), / LineCount (number of lines of text), and / CharCount (number of characters of text). The content consists of a multi-line dictionary, with each line recording one page element. The dictionary attributes are as follows: ReadOrder (reading order), Type (element type), Chars (number of characters in the element), Line (number of lines in the element), Content (object reference to the element's content), and Position (object reference to the element's coordinates).

[0087] The Text object records text. Its properties include ` / Type text` (identifying the text type). The content is arranged line by line using an array structure. The first data element is a string, and the second data element is the offset of the text in that line within the Position (based on `>>`). Empty lines are represented by `null`.

[0088] The `Table` object records table content. Its properties include ` / Type table` (identifying the table type), ` / Position` (table block coordinates), and ` / Headers` (table headers; null indicates a headerless table). The content consists of multiple dictionaries and arrays. The first line identifies a cell's information in dictionary format, containing `Row` (row number), `RowSpan` (number of rows occupied), `Col` (column number), `ColSpan` (number of columns occupied), `TextRows` (number of lines of text within the cell), and `CellPosOffset` (cell coordinate offset). This is followed by multiple array lines representing the various lines of text within the cell, with the same format as the `text` object.

[0089] The Position object records text coordinate information and has the attribute / Type position. Its content consists of multiple arrays, each containing two numeric arrays: the first array contains the coordinates of each individual character, and the second array contains the coordinates of the entire line of text. The origin of the coordinate system is located at the bottom left corner, and the unit is pixels, with the numbers representing the bottom left (x, y) and top right (x, y) corners.

[0090] The `TablePosition` object records table coordinate information and has the attribute ` / Type tablePosition`. The content consists of groups of multiple lines, each group representing the coordinate information of a single cell. The first line contains the cell's coordinates, and the remaining lines contain text coordinates (formatted the same as within the `Position` line). This detailed design ensures that EPG files can effectively store and organize various information from PDF documents.

[0091] In addition, the process of writing to a file is as follows:

[0092] First, during the initialization phase, a new EPG file needs to be generated to prepare for data writing. Next, the file header information is written, laying the foundation for subsequent data entry. During the writing of object attributes and content, each object has its specific writing order and method. For example, for a text object, the offset of the first line of text (relative to the object content) is recorded first, then the offset of the starting position of each line is recorded sequentially, and this recording continues until all relevant coordinate information is completely entered.

[0093] For other types such as location objects, table location objects, table objects, and page objects, the process is similar: first, record the starting position offset of the object relative to its content, and then write the corresponding content according to the specific object type. For example, location objects need to record and write specific coordinate information; while table objects need to include detailed information about the cells, such as the number of rows, columns, text rows, and the corresponding text object number, and record this information cell by cell until all are completed.

[0094] After all object information is entered, the next step is to write the root object. This step ensures that the top-level node of the entire data structure is established. Next, to ensure data consistency and integrity, the starting position of the reference table relative to its content needs to be recorded, and the reference relationships between all objects are written to the file. The final step is to write the end-of-file information, explicitly marking the end of the file writing process.

[0095] The detailed writing steps for specific object types, such as text objects, table objects, and page objects, also follow the same logic: start by recording the relevant information of the first row or the first element, and then gradually enter the information until all information is recorded. This systematic process ensures that various objects, their attributes, content, and references are written to the EPG file in an orderly manner, thereby guaranteeing data integrity and readability.

[0096] S120. During the reading process, the reference table information is obtained from the end of the file to locate the root object. Then, based on the reference relationship, each page object and its contents are traversed. If necessary, jump to read text or coordinate information until the entire file is processed.

[0097] In one embodiment, step S120 described above may include steps S121 to S125.

[0098] S121. During the reading process, open and load the EPG file, and read relevant information from the end of the file.

[0099] In this embodiment, the program first needs to locate the location where the EPG file is stored and open the file using a suitable API or method.

[0100] After opening the file, immediately read the relevant information from the end of the file. This is because reference tables are usually located at the end of the file, allowing for quick access without having to traverse the entire file.

[0101] S122. Move the file pointer to the beginning of the reference table and read all the information in the reference table.

[0102] In this embodiment, the file pointer is moved to the beginning of the reference table based on the file structure information obtained in the previous step.

[0103] Read all the information in the reference table, which includes references to the root object and other important data structures. The reference table is key to understanding the entire file structure.

[0104] S123. Move the file pointer to the position of the root object according to the information in the reference table, and read the information of the root object.

[0105] In this embodiment, the information provided by the reference table is used to locate the root object and read its contents.

[0106] The root object contains references to other shard objects, which are crucial for building the logical structure of the entire file.

[0107] S124. Parse the attributes of the root object to obtain reference information for all shard objects.

[0108] In this embodiment, the root object is parsed to extract reference information for all the shard objects contained therein.

[0109] This step helps to build a complete object network graph, so that each page object and its content can be accessed accurately.

[0110] S125. Process page objects one by one, jump to the beginning of the next page object, read all attributes and content information of the current page object, and for text content objects in the page object, determine whether to read coordinate information as needed, and adjust the file pointer position as needed for processing.

[0111] In one embodiment, step S125 described above may include steps S1251 to S12510.

[0112] S1251. Based on the information in the reference table, move the file pointer to the beginning of the next page object to obtain the current page object.

[0113] In this embodiment, the current page object refers to the page object where the file pointer is located.

[0114] Based on the reference table information, move the file pointer to the beginning of the next page object.

[0115] Ensure that each page object can be correctly located to prepare for subsequent data retrieval.

[0116] S1252. Read all attributes and content information of the current page object.

[0117] In this embodiment, all attributes and content information of the current page object are read.

[0118] This information includes the basic properties and style information of page objects, laying the foundation for more in-depth text content analysis.

[0119] S1253. Move the file pointer to the starting position of the first text content object in the current page object.

[0120] In this embodiment, the file pointer is moved to the starting position of the first text content object in the current page object.

[0121] Ensure that the information of the text content object can be read accurately.

[0122] S1254. Read information from the text content object;

[0123] S1255. Extract and read the specific text content;

[0124] S1256. Determine whether the currently processed text content requires coordinate information;

[0125] S1257. If the currently processed text content requires coordinate information, move the file pointer to the corresponding coordinate object position and read the coordinate object information.

[0126] S1258. Adjust the file pointer position according to the offset at the end of the text line, and continue processing the next text line or object.

[0127] In this embodiment, the text content is read sequentially, it is determined whether coordinate information is needed, and the file pointer position is adjusted as needed.

[0128] Specifically, extract and read the specific text content; determine if coordinate information is needed; if so, move the file pointer to the corresponding coordinate object position and read it; adjust the file pointer position according to the offset at the end of the text line, and continue processing the next text line or object.

[0129] S1259~S12510: Loop and Termination Conditions

[0130] S1259. If the text content being processed does not require coordinate information, then determine whether the text content object being processed is the last element in the page object.

[0131] If the currently processed text content object is not the last element in the page object, then execute S1253;

[0132] S12510. If the currently processed text content object is the last element in the page object, then determine whether the currently processed page object is the last page in the EPG file.

[0133] If the currently processed page object is not the last page in the EPG file, execute step S1251;

[0134] If the currently processed page object is the last page in the EPG file, proceed to the end step.

[0135] Check if the currently processed text content object is the last element in the page object. If not, repeat step S1253. If it is, determine if the current page is the last page. If it is not the last page, return to step S1251. If it is the last page, proceed to the end step.

[0136] Ensure all page and text content is processed correctly to avoid missing any critical information.

[0137] As described in detail above, this process involves not only basic file reading and information extraction, but also complex logical judgments and pointer operations to ensure efficient and accurate processing of various information within the EPG file. This meticulous workflow helps improve the accuracy of data processing while also guaranteeing the system's stability and efficiency.

[0138] In this embodiment, the initialization phase is performed first during the reading process. This phase includes opening and loading the EPG file, and reading relevant information starting from the end of the file. This information is crucial for subsequent reading operations. By reading from the end of the file, the location and structure information of the reference table can be quickly obtained, preparing for the following operations.

[0139] The next step is to locate the reference table. Based on the information obtained during the initialization phase, the file pointer is moved to the beginning of the reference table, and all information in the reference table is read. The reference table records the position of each object in the file and their mutual reference relationships, which is key to understanding the entire file structure. After completing this step, the root object, i.e., object number 1, can be located based on the data provided by the reference table. By moving the file pointer to the position of the root object and reading its information, it serves as the top-level node for constructing the entire data structure.

[0140] Next, the root object properties are parsed to read reference information for all fragment objects. This reference information points to other important data structures within the file, helping to establish a complete logical framework. Based on this framework, the Page objects are traversed. During each traversal, based on the information in the reference table, the file pointer is moved to the beginning of the next Page object, and then all properties and content information of that Page object are read.

[0141] When processing each Page object, the process first jumps to the beginning of the page element's text content object and reads its information. Next, the specific text content is extracted and read. During this process, it's necessary to determine if the currently processed text content requires coordinate information. If so, the file pointer is moved to the corresponding coordinate object position, and its information is read. Then, the file pointer position is adjusted based on the offset from the end of the text line to continue processing the next text line or object. If coordinate information is not needed, the coordinate processing step is skipped, and the process proceeds to the next step.

[0142] After processing each text content object, it's necessary to determine if it's the last element in the current Page object. If not, return to the step of "jumping to the beginning of the page element's text content object" to continue processing the next text content object; if so, further determine if the current Page object is the last page in the EPG file. If it's not the last page, return to the step of "jumping to the beginning of the next Page object" to continue processing the next page; if it is the last page, end the entire reading process.

[0143] Through the above series of systematic steps, all Page objects and their contained text content and coordinate information can be fully read and parsed from the EPG file, ensuring the integrity and accuracy of the data and providing a solid foundation for subsequent data analysis and application.

[0144] For example, the structure and organization of the EPI-PDFX sample file demonstrate the unique advantages of this format. First, at the beginning of the file, the file format and version information are specified by the mark "%epi-PDFX-1.0," where "epi" represents the file format, "PDFX" indicates that it is a format used to describe PDF information, and "1.0" indicates the version number. This explicit identification method facilitates quick identification of the file type and version, simplifying subsequent processing.

[0145] At the object level, taking object number 2 as an example, "2 obj" declares the object and indicates through the attribute / Type "Position" that it is an object containing coordinate information. Similarly, object number 3 contains the text string "["This is a piece of text" 0]" in its content section. The trailing 0 indicates that the coordinate information of this line of text in the PDF coordinate system is located relative to the content of the corresponding position object (i.e., object number 3). This design closely links the text with its coordinate information, facilitating accurate extraction and display.

[0146] For page information object 4, it records a detailed list of elements on a single page of a PDF. In this example, / Type “text” indicates that the element type is a text paragraph; / ReadOrder 0 indicates that this element is listed first in reading order; / Lines 1 / Chars 6 indicates that this text consists of one line containing 6 characters. Furthermore, / Content 3 R and / Position 2 R point to the specific locations of the text content and coordinate information, respectively; that is, the text content is recorded in object number 3, and the coordinate information is recorded in object number 2.

[0147] The root object (object number 1) plays a central role. It has only one property, `Pages`, which records the information object links for each page in page number order. For example, `[4 R]` indicates that the document contains only one page, and the information for the first page is recorded in object number 4. This structure not only simplifies the management of page information but also improves data access efficiency.

[0148] The EPI-PDFX format is designed with memory usage optimized. Small data items are stored directly as attributes, while large data items are stored as independent objects and linked by references. This enables object-by-object processing during reading and writing, ensuring that the memory required for each operation is controllable. Furthermore, the format is highly extensible; based on its fundamental data format framework, extended formats that can support different data types can be flexibly designed. Finally, thanks to its unique data structure, when retrieving the coordinates of a specific text line from a PDF file, the offset marked at the end of the text line allows for rapid location and data retrieval, significantly improving processing speed and accuracy.

[0149] For another example: The user first uploads two PDF contract documents. The system parses and converts these two documents into Episode Format (epi) files, generating two corresponding epi files. Next, the system uses a text comparison algorithm to compare and analyze these two epi files.

[0150] During the text comparison process, the system extracts all text content from each epidural file and calculates the feature values ​​for each text. By comparing these feature values, the system can identify all the differing text between the two documents. For each piece of differing text, the system uses its offset value (i.e., its position offset in the original PDF file) to obtain the specific line number and page number of that text in the PDF file.

[0151] After completing the above steps, the system will return all differing text content and their corresponding location coordinates. Subsequently, the front-end display module will load the original PDF file and highlight all differences based on the comparison results, allowing users to intuitively see the differences between the two contract documents.

[0152] The entire process ensures that users can efficiently and accurately compare and view the differences between two PDF contract documents, improving work efficiency and accuracy.

[0153] In contract comparison scenarios, by integrating with an SDK that reads epipi files, all text is read first, followed by the coordinate information of the lines showing differences. This fulfills the requirements for contract comparison functionality and page display. Only the coordinate data of the differing content is loaded, allowing only the necessary information to be read, while significantly reducing memory usage.

[0154] This embodiment addresses the issue of large data volume memory consumption and the need for data storage by type and on-demand retrieval. The Epi format borrows the advantages of the PDF file structure, employing a cross-reference table mechanism to mark the offsets of each object, supporting fragmented data writing and random access. In actual data storage, text information is separated from coordinate information, and the offset of the coordinate information is appended to the end of each line of text information, allowing users to read only the text information or quickly locate specific coordinate information when needed. This design not only optimizes memory usage and reduces unnecessary data loading, but also achieves fast data access and processing through efficient cross-reference tables, meeting diverse needs in different application scenarios. This characteristic of the Epi format makes it an ideal choice for processing large amounts of data and improving data retrieval efficiency.

[0155] The segmented data storage method described above constructs and sequentially records the file's basic structure, extended structure, object attributes, and content during writing, while maintaining a reference table and root object information. A marker is added to the end of the file, allowing for rapid traversal of the root object from the end of the file during reading. This facilitates efficient data access and processing by efficiently traversing each page object and its content through reference relationships. This method reduces unnecessary memory usage because only specific objects or pages need to be loaded, rather than the entire file. Furthermore, its flexible data structure design adapts to different application scenarios, providing excellent support for managing a large number of small files as well as efficiently reading and writing large files, thereby improving overall data processing efficiency and system response speed.

[0156] Figure 5 This is a schematic block diagram of a segmented data storage device 300 provided in an embodiment of the present invention. Figure 5As shown, corresponding to the above-described segmented data storage method, the present invention also provides a segmented data storage device 300. This segmented data storage device 300 includes a unit for executing the above-described segmented data storage method, and the device can be configured in a server. Specifically, please refer to... Figure 5 The segmented data storage device 300 includes a writing unit 301 and a reading unit 302.

[0157] The writing unit 301 is used to construct the basic structure and extended structure of the file during the writing process, and sequentially write the attributes and contents of each object, record the reference table and root object information, and add a file end marker.

[0158] The reading unit 302 is used to obtain the reference table information from the end of the file to locate the root object during the reading process, and then traverse each page object and its contents according to the reference relationship, and jump to read text or coordinate information as needed until the entire file is processed.

[0159] In one embodiment, the writing unit 301 includes:

[0160] The first write subunit is used to create a new EPG file; the second write subunit is used to write header information into the EPG file; the third write subunit is used to record object offsets and write object attributes and specific object content into the EPG file; and the fourth write subunit is used to record the object reference table and write a file end marker after all objects have been written into the EPG file.

[0161] In one embodiment, the second write subunit includes:

[0162] The recording module is used to record object attributes and specific object content in the EPG file; the root object writing module is used to write the root object information into the EPG file; and the reference table recording module is used to record the starting position of the reference table relative to its content in the EPG file.

[0163] In one embodiment, the third writing subunit is used to write the reference relationships between all objects into the EPG file and write file end information into the EPG file to mark the end of the file writing process.

[0164] In one embodiment, the reading unit 302 includes:

[0165] The following subunits are used: a tail reading subunit, which opens and loads the EPG file during the reading process and reads relevant information from the end of the file; a reference table reading subunit, which moves the file pointer to the beginning of the reference table and reads all information in the reference table; a root object subunit, which moves the file pointer to the root object based on the information in the reference table and reads the root object's information; a parsing subunit, which parses the attributes of the root object and obtains reference information for all fragment objects; and a sequential processing subunit, which processes page objects one by one, jumps to the beginning of the next page object, reads all attributes and content information of the current page object, and for text content objects in the page object, determines whether to read coordinate information as needed and adjusts the file pointer position as required.

[0166] In one embodiment, the sequential processing subunit includes:

[0167] The first movement module moves the file pointer to the beginning of the next page object based on information in the reference table to obtain the current page object. The first reading module reads all attributes and content information of the current page object. The second movement module moves the file pointer to the beginning of the first text content object in the current page object. The second reading module reads the information of the text content object. The third reading module extracts and reads the specific text content. The first judgment module determines whether the currently processed text content requires coordinate information. The third movement module, if the currently processed text content requires coordinate information, moves the file pointer to the corresponding coordinate object position and reads the coordinate object's information. The fourth movement module adjusts the file pointer position based on the offset at the end of the text line and continues processing the next... The first module is a text line or object; the second module is a judgment module, used to determine whether the currently processed text content object is the last element in the page object if the currently processed text content does not require coordinate information; if the currently processed text content object is not the last element in the page object, then the file pointer is moved to the starting position of the first text content object in the current page object; the third module is a judgment module, used to determine whether the currently processed page object is the last page in the EPG file if the currently processed text content object is the last element in the page object; if the currently processed page object is not the last page in the EPG file, then the file pointer is moved to the starting position of the next page object according to the information in the reference table; if the currently processed page object is the last page in the EPG file, then the end step is entered.

[0168] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned segmented data storage device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0169] The aforementioned segmented data storage device 300 can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0170] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0171] See Figure 6 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0172] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a segmented data storage method.

[0173] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0174] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a segmented data storage method.

[0175] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0176] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0177] During the writing process, the basic and extended structures of the file are constructed, and the attributes and contents of each object are written sequentially. The reference table and root object information are recorded, and a file end marker is added. During the reading process, the reference table information is obtained from the end of the file to locate the root object. Then, based on the reference relationship, each page object and its contents are traversed. If necessary, the file is jumped to read text or coordinate information until the entire file is processed.

[0178] The reference table records the offsets of all objects, represented in hexadecimal, with one data entry per line. The line number is the object number, and the index starts from 1.

[0179] In one embodiment, when the processor 502 implements the steps of constructing the basic and extended structures of the file and sequentially writing the attributes and contents of each object, recording the reference table and root object information, and adding a file end marker during the writing process, the specific implementation is as follows:

[0180] Create a new EPG file; write header information into the EPG file; record object offsets and write object attributes and specific object content into the EPG file; after all objects are written into the EPG file, record the object reference table and write the file end marker.

[0181] In one embodiment, when the processor 502 implements the step of recording the object offset and writing the object attributes and specific object content in the EPG file, it specifically implements the following steps:

[0182] Record object attributes and specific object content in the EPG file; write the root object information into the EPG file; record the starting position of the reference table relative to its content in the EPG file.

[0183] In one embodiment, when processor 502 implements the step of recording the object reference table and writing the file end marker after all objects have been written to the EPG file, it specifically implements the following steps:

[0184] Write all the reference relationships between objects into the EPG file, and then write the end-of-file information into the EPG file to mark the end of the file writing process.

[0185] In one embodiment, when the processor 502 performs the step of retrieving the reference table information from the end of the file to locate the root object during the reading process, and then traversing each page object and its contents according to the reference relationship, and jumping to read text or coordinate information as needed until the entire file is processed, the specific implementation is as follows:

[0186] During the reading process, the EPG file is opened and loaded, and relevant information is read from the end of the file; the file pointer is moved to the beginning of the reference table, and all information in the reference table is read; the file pointer is moved to the root object's position according to the information in the reference table, and the root object's information is read; the root object's attributes are parsed to obtain the reference information of all fragment objects; page objects are processed one by one, jumping to the beginning of the next page object, and all attributes and content information of the current page object are read. For text content objects in the page object, it is determined whether to read coordinate information as needed, and the file pointer position is adjusted as needed for processing.

[0187] In one embodiment, when the processor 502 performs the steps of processing page objects one by one, jumping to the beginning of the next page object, reading all attributes and content information of the current page object, and determining whether to read coordinate information and adjusting the file pointer position as needed for text content objects in the page object, the specific implementation is as follows:

[0188] Based on the information in the reference table, move the file pointer to the beginning of the next page object to obtain the current page object; read all attributes and content information of the current page object; move the file pointer to the beginning of the first text content object in the current page object; read the information of the text content object; extract and read the specific text content; determine whether the currently processed text content needs coordinate information; if the currently processed text content needs coordinate information, move the file pointer to the corresponding coordinate object position and read the coordinate object information; adjust the file pointer position according to the offset at the end of the text line and continue processing the next text line or object; if the currently processed text content does not need coordinate information, determine whether the currently processed text content object is the last element in the page object; if the currently processed text content object is not the last element in the page object, execute the step of moving the file pointer to the beginning of the first text content object in the current page object; if the currently processed text content object is the last element in the page object, determine whether the currently processed page object is the last page in the EPG file; if the currently processed page object is not the last page in the EPG file, execute the step of moving the file pointer to the beginning of the next page object based on the information in the reference table; if the currently processed page object is the last page in the EPG file, proceed to the end step.

[0189] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0190] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0191] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps:

[0192] During the writing process, the basic and extended structures of the file are constructed, and the attributes and contents of each object are written sequentially. The reference table and root object information are recorded, and a file end marker is added. During the reading process, the reference table information is obtained from the end of the file to locate the root object. Then, based on the reference relationship, each page object and its contents are traversed. If necessary, the file is jumped to read text or coordinate information until the entire file is processed.

[0193] The reference table records the offsets of all objects, represented in hexadecimal, with one data entry per line. The line number is the object number, and the index starts from 1.

[0194] In one embodiment, when the processor executes the computer program to implement the steps of constructing the basic and extended structures of the file and sequentially writing the attributes and contents of each object, recording the reference table and root object information, and adding a file end marker during the writing process, the specific implementation is as follows:

[0195] Create a new EPG file; write header information into the EPG file; record object offsets and write object attributes and specific object content into the EPG file; after all objects are written into the EPG file, record the object reference table and write the file end marker.

[0196] In one embodiment, when the processor executes the computer program to implement the steps of recording the object offset and writing the object attributes and specific object content in the EPG file, it specifically implements the following steps:

[0197] Record object attributes and specific object content in the EPG file; write the root object information into the EPG file; record the starting position of the reference table relative to its content in the EPG file.

[0198] In one embodiment, when the processor executes the computer program to implement the step of recording the object reference table and writing the file end marker after all objects have been written to the EPG file, the specific implementation is as follows:

[0199] Write all the reference relationships between objects into the EPG file, and then write the end-of-file information into the EPG file to mark the end of the file writing process.

[0200] In one embodiment, when the processor executes the computer program to implement the step of retrieving reference table information from the end of the file to locate the root object during the reading process, and then traversing each page object and its contents according to the reference relationship, and jumping to read text or coordinate information as needed until the entire file is processed, the specific implementation is as follows:

[0201] During the reading process, the EPG file is opened and loaded, and relevant information is read from the end of the file; the file pointer is moved to the beginning of the reference table, and all information in the reference table is read; the file pointer is moved to the root object's position according to the information in the reference table, and the root object's information is read; the root object's attributes are parsed to obtain the reference information of all fragment objects; page objects are processed one by one, jumping to the beginning of the next page object, and all attributes and content information of the current page object are read. For text content objects in the page object, it is determined whether to read coordinate information as needed, and the file pointer position is adjusted as needed for processing.

[0202] In one embodiment, when the processor executes the computer program to process page objects one by one, jumps to the beginning of the next page object, reads all attributes and content information of the current page object, and, for text content objects in the page object, determines whether to read coordinate information and adjusts the file pointer position as needed for processing, the specific steps are as follows:

[0203] Based on the information in the reference table, move the file pointer to the beginning of the next page object to obtain the current page object; read all attributes and content information of the current page object; move the file pointer to the beginning of the first text content object in the current page object; read the information of the text content object; extract and read the specific text content; determine whether the currently processed text content needs coordinate information; if the currently processed text content needs coordinate information, move the file pointer to the corresponding coordinate object position and read the coordinate object information; adjust the file pointer position according to the offset at the end of the text line and continue processing the next text line or object; if the currently processed text content does not need coordinate information, determine whether the currently processed text content object is the last element in the page object; if the currently processed text content object is not the last element in the page object, execute the step of moving the file pointer to the beginning of the first text content object in the current page object; if the currently processed text content object is the last element in the page object, determine whether the currently processed page object is the last page in the EPG file; if the currently processed page object is not the last page in the EPG file, execute the step of moving the file pointer to the beginning of the next page object based on the information in the reference table; if the currently processed page object is the last page in the EPG file, proceed to the end step.

[0204] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0205] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0206] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0207] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0208] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0209] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of segmented data storage, characterized by, The application relates to a method for writing and reading an EPG file, and a device for writing and reading an EPG file. In the writing process, the basic structure and extension structure of the file are constructed and the attributes and contents of various objects are written in sequence, the reference table and root object information are recorded, and a file tail mark is added; In the reading process, the information of the reference table is obtained from the file tail part, the root object is positioned, and then various page objects and the contents contained in the page objects are traversed according to the reference relationship; text or coordinate information is read as required until the whole file is processed; The method for writing an EPG file in the writing process, the basic structure and extension structure of the file are constructed and the attributes and contents of various objects are written in sequence, the reference table and root object information are recorded, and a file tail mark is added, comprising: A new EPG file is created; Header information is written in the EPG file; Object offset is recorded in the EPG file, and object attributes and specific object contents are written; After all objects are written in the EPG file, the reference table of the objects is recorded, and a file tail mark is written; The method for recording object offset in the EPG file and writing object attributes and specific object contents comprises: Object attributes and specific object contents are recorded in the EPG file; The information of the root object is written in the EPG file; The starting position of the reference table relative to the contents is recorded in the EPG file.

2. The method of claim 1, wherein, The method for recording the reference table of the objects after all objects are written in the EPG file and writing a file tail mark comprises: The reference relationship among all objects is written in the EPG file, and file tail information is written in the EPG file, marking the end of the file writing process.

3. The method of claim 1, wherein, The method for reading an EPG file in the reading process, the information of the reference table is obtained from the file tail part, the root object is positioned, and then various page objects and the contents contained in the page objects are traversed according to the reference relationship; text or coordinate information is read as required until the whole file is processed, comprising: In the reading process, the EPG file is opened and loaded, and relevant information is read from the file tail part; The file pointer is moved to the starting position of the reference table, and all information in the reference table is read; According to the information in the reference table, the file pointer is moved to the position of the root object, and the information of the root object is read; The attributes of the root object are parsed, and the reference information of all fragment objects is obtained; Page objects are processed one by one, the starting position of the next page object is jumped to, all attributes and content information of the current page object are read, and for the text content object in the page object, whether the coordinate information is read is judged according to the need, and the file pointer position is adjusted for processing as required.

4. The method of claim 3, wherein, The method for processing page objects one by one, jumping to the starting position of the next page object, reading all attributes and content information of the current page object, and for the text content object in the page object, judging whether the coordinate information is read according to the need and adjusting the file pointer position for processing as required, comprising: According to the information in the reference table, the file pointer is moved to the starting position of the next page object to obtain the current page object; All attributes and content information of the current page object are read; The file pointer is moved to the starting position of the first text content object in the current page object; The information of the text content object is read; Specific text content is extracted and read; determining whether the current text content needs coordinate information; if the current text content needs coordinate information, moving the file pointer to the corresponding coordinate object position and reading the coordinate object information; adjusting the file pointer position according to the offset at the end of the text line, and continuing to process the next text line or object; if the current text content does not need coordinate information, determining whether the current text content object is the last element in the page object; if the current text content object is not the last element in the page object, moving the file pointer to the start position of the first text content object in the current page object; if the current text content object is the last element in the page object, determining whether the current page object is the last page in the EPG file; if the current page object is not the last page in the EPG file, moving the file pointer to the start position of the next page object according to the information in the reference table; if the current page object is the last page in the EPG file, entering the end step.

5. The method of claim 1, wherein, The reference table records the offsets of all objects in hexadecimal, with one data line per row, and the row number is the object number, with the index starting from 1.

6. A segmented data storage device, characterized by, The device uses the segmented data storage method according to any one of claims 1 to 5, comprising: a writing unit for constructing the basic structure and extension structure of the file and writing the attributes and contents of each object in turn, recording the reference table and root object information, and adding a file tail marker during the writing process; a reading unit for obtaining the reference table information from the file tail during the reading process, positioning to the root object, and then traversing each page object and the contents contained therein according to the reference relationship, and jumping to read the text or coordinate information as needed until the entire file is processed.

7. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1 to 5.

Images