Data storage method and device

Abstract

The invention provides a data storage method and device, and relates to the field of storage. The method comprises the following steps: acquiring a PDF file; analyzing the PDF file to obtain an analysis result; the analysis result comprises text information obtained by performing text extraction on the PDF file or outline information of the PDF file; associating the analysis result with the PDF file to obtain a to-be-stored file; the to-be-stored file comprises a PDF file and an analysis result; and storing the to-be-stored file. Therefore, when the unstructured data such as the PDF file is stored, the original file of the PDF file is stored, and the analysis result of the PDF file and the PDF file are associated and then are together stored in a disk. Therefore, the user can access the analysis result corresponding to the PDF file at the same time when accessing the PDF file subsequently, so that the situation that the user performs manual algorithm analysis on the PDF file to obtain required data when accessing the PDF file each time is avoided, and the workload of secondary file processing of the user is reduced.

Description

Technical Field

[0001] This application relates to the field of storage, and more particularly to a data storage method and apparatus. Background Technology

[0002] Generally speaking, unstructured data has an irregular or incomplete data structure, lacks a predefined data model, and is not convenient to represent using a two-dimensional logical table in a database. For example, unstructured data includes office documents of all formats, text, images, Extensible Markup Language (XML) data, images, and audio / video information.

[0003] In the current solution, after unstructured data is stored, if a user wants to access the unstructured data to extract the data they need, the user needs to manually configure the corresponding algorithm to parse the unstructured data according to the different data content extraction requirements. This results in a large workload for users to process the files twice. Summary of the Invention

[0004] This application provides a data storage method and apparatus to solve the problem of users having a large workload in secondary file processing.

[0005] Firstly, a data storage method is provided, applied to a computing device. The method includes: acquiring a PDF file; parsing the PDF file to obtain a parsing result; the parsing result includes text information extracted from the PDF file, or an outline of the PDF file; associating the parsing result and the PDF file to obtain a file to be stored; the file to be stored includes the PDF file and the parsing result; and storing the file to be stored.

[0006] Based on the data storage method described above, when storing unstructured data such as PDF files, in addition to storing the original PDF file, the parsed results of the PDF file are also associated with the PDF file and stored together on disk. In this way, when users subsequently access a PDF file, they can simultaneously access the corresponding parsed results, avoiding the need for users to manually parse the PDF file using algorithms each time they access it, thus reducing the workload of secondary file processing for users.

[0007] Building upon the data storage method provided in the first aspect, one possible implementation is to establish a relationship between the PDF file and its corresponding parsing result. For example, a mapping table can be used to map the file identifier of the PDF file to the file identifier of the parsing result. In this way, by associating the PDF file and the parsing result based on their file identifiers, users can simultaneously obtain the parsing result of the PDF file whenever they access it after both are stored on disk, thus improving the efficiency of accessing PDF files.

[0008] Combining the data storage method provided in the first aspect, as a possible implementation, the parsing results of PDF files can be matched with the hardware infrastructure of the user's device. For example, it can detect whether the computing device includes an AI chip; if the computing device does not include an AI chip, text extraction is performed on the PDF file to obtain the parsing result; the parsing result includes the extracted text information; the file to be stored includes the extracted text information and the PDF file; if the computing device includes an AI chip, the AI ​​chip is used to parse the PDF file to obtain the parsing result; the parsing result includes outline information.

[0009] Thus, when storing unstructured data in PDF format, the storage method can be determined based on the hardware capabilities of the computing device, ensuring that the storage of unstructured data is compatible with the user's computing device hardware. This improves the resource utilization of the AI ​​chip or conventional central processing unit (CPU) of the computing device, thereby increasing the flexibility of storing unstructured data.

[0010] Building upon the data storage method provided in the first aspect, one possible implementation is for the computing device to input a PDF file into a layout analysis model run by an AI chip, obtaining the layout analysis results output by the model. These results include elements of various categories within the PDF file and their coordinate frames. The computing device then determines the outline information based on the positional relationships between the coordinate frames of each category of elements. In this way, by using an AI chip to perform layout analysis on the PDF and output outline information, users are spared the need for subsequent manual algorithmic analysis based on the PDF file's analysis results, thus reducing the workload of secondary file processing.

[0011] Optionally, the categories include headings and body text, with headings including at least a first-level heading.

[0012] Combining the data storage method provided in the first aspect, as one possible implementation, the computing device determines the association between each level of heading and the main text within at least one level of heading based on the positional relationship between the coordinate frames of elements of each category; the association relationship is used to indicate the main text associated with any level of heading. The computing device extracts the content information of the main text associated with each level of heading and determines the outline information based on the association relationship and the content information of the main text associated with each level of heading. In this way, the outline information is generated based on the association relationship between each level of heading and the main text in the PDF file, ensuring the accuracy of the outline information.

[0013] In conjunction with the data storage method provided in the first aspect, as one possible implementation, the main body includes text.

[0014] Optionally, the computing device inputs the headings at each level and the text associated with each heading into the optical character recognition model running on the AI ​​chip, and obtains the text content of the headings at each level and the text associated with each heading output by the optical character recognition model; and matches the text content of the headings at each level and the text associated with each heading with the coordinate frames of elements of each category.

[0015] In conjunction with the data storage method provided in the first aspect, as one possible implementation, the text includes tables.

[0016] Optionally, the headings at each level are input into the optical character recognition model running on the AI ​​chip to obtain the text content of the headings at each level output by the optical character recognition model; the tables associated with the headings at each level are input into the structure detection model running on the AI ​​chip to obtain at least one cell of the table associated with the headings at each level output by the structure detection model; at least one cell is input into the optical character recognition model running on the AI ​​chip to obtain the text content contained in at least one cell output by the optical character recognition model; the text content contained in at least one cell is clustered based on the row and column division of the cell to obtain the index type of the row and column of at least one cell output by the table recognition model; and the index types of the headings at each level and the tables associated with the headings at each level are matched with the coordinate boxes of the elements of each category.

[0017] Thus, for different types of data such as text and tables in the main text, the corresponding content extraction methods can be used to extract content information, thereby improving the applicability of the data storage method provided in the embodiments of this application.

[0018] In conjunction with the data storage method provided in the first aspect, as one possible implementation, the computing device extracts semantic information from headings at all levels and the associated body text, and corrects the relationships between headings and body text in the outline information based on this semantic information. Thus, by correcting the relationships between headings and body text in the outline information using semantic information, the accuracy of the outline information is improved.

[0019] In conjunction with the data storage method provided in the first aspect, as a possible implementation, after storing the file to be stored, the computing device can also return the PDF file and its corresponding parsing result to the user when the PDF file is accessed. The computing device receives a data access request; the data access request includes the file identifier of the PDF file; it queries the mapping table for the file identifier of the parsing result based on the file identifier of the PDF file; it queries metadata based on the file identifier of the PDF file and the file identifier of the parsing result to obtain the storage address of the PDF file and the storage address of the parsing result; and it returns a data access response; the data access response includes data stored at the storage address of the PDF file and data stored at the storage address of the parsing result.

[0020] Secondly, a data storage device is provided. This data storage device includes an acquisition module, a parsing module, and a storage module. The acquisition module is used to acquire PDF files. The parsing module is used to parse the PDF files to obtain parsing results; the parsing results include text information extracted from the PDF files, or outline information of the PDF files. The parsing module is also used to associate the parsing results and the PDF files to obtain a file to be stored; the file to be stored includes the PDF files and the parsing results. The storage module is used to store the file to be stored.

[0021] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the parsing module is specifically used to: map the file identifier of the PDF file to the file identifier of the parsing result in the mapping table.

[0022] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the parsing module is specifically used to: detect whether the computing device includes an AI chip; if the computing device does not include an AI chip, perform text extraction on the PDF file to obtain the parsing result; the parsing result includes the text information obtained from the text extraction; if the computing device includes an AI chip, use the AI ​​chip to parse the PDF file to obtain the parsing result; the parsing result includes outline information.

[0023] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the parsing module is specifically used to: input the PDF file into the layout parsing model running on the AI ​​chip, and obtain the layout parsing result output by the layout parsing model; the layout parsing result includes elements of various categories in the PDF file and the coordinate frames of each category of elements; and determine the outline information based on the positional relationship between the coordinate frames of each category of elements.

[0024] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the categories include headings and body text, with the headings including at least a first-level heading.

[0025] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the parsing module is specifically used to: determine the relationship between the headings at all levels and the main text in at least one level heading based on the positional relationship between the coordinate frames of elements of each category; the relationship is used to indicate the main text associated with any level heading; extract the content information of the main text associated with each level heading; and determine the outline information based on the relationship and the content information of the main text associated with each level heading.

[0026] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the main body includes text. The parsing module is specifically used to: input the text associated with each level of headings into the optical character recognition model running on the AI ​​chip, and obtain the text content of each level of headings and the text associated with each level of headings output by the optical character recognition model; and match the text content of each level of headings and the text associated with each level of headings with the coordinate frames of elements in each category.

[0027] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the main text includes tables. The parsing module is specifically used for: inputting the headings at each level into the optical character recognition model running on the AI ​​chip, obtaining the text content of the headings at each level output by the optical character recognition model; inputting the tables associated with the headings at each level into the structure detection model running on the AI ​​chip, obtaining at least one cell of the tables associated with the headings at each level output by the structure detection model; inputting at least one cell into the optical character recognition model running on the AI ​​chip, obtaining the text content contained in at least one cell output by the optical character recognition model; clustering the text content contained in at least one cell based on the row and column division of the cell, obtaining the row and column index types of at least one cell output by the table recognition model; and matching the index types of the headings at each level and the tables associated with the headings at each level with the coordinate frames of elements in each category.

[0028] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the main text includes tables. The parsing module is specifically used to: extract semantic information from headings at all levels and the associated main text; and correct the relationships between headings and main text in the outline information based on the semantic information.

[0029] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the data storage device further includes a query module and a response module. The acquisition module is further configured to: receive a data access request; the data access request includes the file identifier of the PDF file. The query module is configured to: query the mapping table for the file identifier of the parsed result based on the file identifier of the PDF file; query metadata based on the file identifier of the PDF file and the file identifier of the parsed result to obtain the storage address of the PDF file and the storage address of the parsed result. The response module is configured to: return a data access response; the data access response includes data stored at the storage address of the PDF file and data stored at the storage address of the parsed result.

[0030] In conjunction with the data storage device provided in the second aspect, as one possible implementation, the aforementioned data storage device may further include other modules that perform the operational steps of the data storage method described in the first aspect.

[0031] Regarding the technical principles and beneficial effects of the second aspect, please refer to the relevant description of the first aspect mentioned above, which will not be repeated here.

[0032] Thirdly, a computing device is provided, including a memory and a processor, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the data storage method described in any possible implementation of the first aspect above.

[0033] Fourthly, a computer program (product) is provided, the computer program (product) comprising: computer program code, which, when executed by a computer, causes the computing device to perform the data storage method described in any possible implementation of the first aspect above.

[0034] Fifthly, a chip is provided, including a processor for retrieving and executing instructions stored in a memory, such that a computing device on which the chip is mounted performs the data storage method described in any possible implementation of the first aspect above.

[0035] In a sixth aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, and the processor is used to execute code in the memory, wherein when the code is executed, the processor is used to execute the data storage method described in any possible implementation of the first aspect above.

[0036] In a seventh aspect, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the instruction being loaded and executed by a processor to implement the data storage method described in any possible implementation of the first aspect above. Attached Figure Description

[0037] Figure 1 A schematic diagram of the structure of a data storage system provided in this application;

[0038] Figure 2 A schematic diagram of the structure of a controller provided in this application;

[0039] Figure 3a A flowchart illustrating a data storage method provided in this application;

[0040] Figure 3b A flowchart illustrating the steps involved in parsing a PDF file, as provided in this application;

[0041] Figure 4 A schematic diagram illustrating a category of elements provided in this application;

[0042] Figure 5 A schematic diagram of an outline information provided for this application;

[0043] Figure 6 A flowchart illustrating the steps for parsing outline information provided in this application;

[0044] Figure 7 A schematic diagram of the structure of a data storage device provided in this application;

[0045] Figure 8 A schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0046] Unstructured data analysis refers to the analysis of unstructured data to extract the useful data needed by users. For example, technical documents in the original portable document format (PDF) produced by a company or individual in a certain technical field may contain technical processes or statistical data that the company or individual needs. In this case, the company or individual needs to perform unstructured data analysis on the unstructured data, i.e., the technical documents.

[0047] This application provides a data storage method and apparatus. The data storage method is executed by a computing device. The computing device acquires a PDF file, parses the PDF file, and obtains the parsing result. The parsing result includes text information extracted from the PDF file, or an outline of the PDF file. Then, the computing device associates the parsing result with the PDF file to obtain the file to be stored, and stores the file to be stored. Thus, when storing unstructured data such as PDF files, in addition to storing the original PDF file, the parsing result is also associated with the PDF file and written to disk. This allows users to access the corresponding parsing result simultaneously when accessing the PDF file later, avoiding the need for manual algorithm parsing to obtain the required data each time the user accesses the PDF file, reducing the workload of secondary file processing for the user.

[0048] The technical solutions involved in this application may be applied not only to current storage fields or storage devices, but also to future storage fields or storage devices, or storage systems including storage devices. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. A brief introduction to some concepts that may be involved in this application is given below.

[0049] Unstructured data refers to data that cannot be represented and stored using traditional matrix structures (tables, databases), lacking a clear pattern and format. In contrast, structured data has a clear data structure and format, making it easy to organize and analyze. Unstructured data includes, but is not limited to, data in the form of text, images, audio, and video, such as PDF files.

[0050] AI chips, also known as AI accelerators or computing cards, are modules specifically designed to handle the massive computational tasks in artificial intelligence applications. Examples include graphics processing units (GPUs) and neural network processing units (NPUs) with AI processing capabilities.

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the application will now be described in further detail with reference to the accompanying drawings.

[0052] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0053] Furthermore, in this application, directional terms such as "upper" and "lower" are defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms are relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0054] For unstructured data, such as PDF files, text extraction can be achieved using existing methods. For example, open-source tools like the Pyton library can extract text from PDF files. However, after the PDF file is saved to disk, users still need to manually configure corresponding algorithms to parse the unstructured data when accessing it to obtain the required data. This results in a significant workload for users due to the secondary processing of the file.

[0055] To address the aforementioned issues, the application scenarios of the embodiments of this application will be described below with reference to the accompanying drawings.

[0056] Figure 1 This application provides a schematic diagram of a data storage system. The data storage system includes a data access device 100 and a storage device 120. Figure 1 In the application scenario shown, the user accesses the application in device 100 via data. Figure 2 (Not shown in the image) to access data. The computer running these applications (such as data access device 100) may be referred to as a "computing device".

[0057] Data access device 100 can be a physical machine, a virtual machine, or a container. The physical machine can include, but is not limited to, one or both a client and a smart NIC. For example, data access device 100 includes a client, such as a host, desktop computer, server, laptop, or mobile device. Another example is that data access device 100 includes a smart NIC. This smart NIC, also known as a smart network adapter, not only performs the network transmission functions of a standard NIC but also provides a built-in programmable and configurable hardware acceleration engine. This improves application performance and significantly reduces CPU consumption in the host connected to the smart NIC, providing more CPU resources for the application. For example, in a highly virtualized environment, the host CPU needs to run open virtual switch (OVS) related tasks. Simultaneously, the host CPU also needs to handle storage, online or offline encryption / decryption of data packets, deep packet inspection, firewalls, complex routing, and other operations. These operations not only consume significant CPU resources but also, due to competition for CPU resources between different services, prevent the services from achieving optimal performance. As a hub connecting various services, smart network interface cards (NICs) accelerate these services.

[0058] In one possible example, data access device 100 accesses storage device 120 via a network to access data; for example, the network may include switch 110.

[0059] In another possible example, data access device 100 may also communicate with storage device 120 via a wired connection, such as a universal serial bus (USB) or a peripheral component interconnect express (PCIe) bus.

[0060] Figure 1 The storage device 120 shown can be a centralized storage system. A key feature of a centralized storage system is a unified entry point through which all data from external devices passes; this entry point is the engine 121 of the centralized storage system. The engine 121 has management functions, and many advanced functions of the storage system are implemented within it.

[0061] like Figure 1 As shown, engine 121 may have one or more controllers. Figure 1The following example illustrates the concept of engine 121 containing one controller. In one possible example, if engine 121 has multiple controllers, any two controllers can have a mirror channel, enabling any two controllers to serve as backups for each other, thereby preventing hardware failures from causing the entire storage device 120 to become unavailable. It should be understood that if engine 121 includes multiple controllers, then engine 121 can also be referred to as the array controller of storage device 120.

[0062] Engine 121 also includes a front-end interface 1211 and a back-end interface 1214. The front-end interface 1211 is used to communicate with the data access device 100 to provide data access services to the data access device 100. The back-end interface 1214 is used to communicate with hard drives to expand the capacity of the storage device 120. Through the back-end interface 1214, engine 121 can connect to more hard drives, thereby forming a very large storage resource pool.

[0063] In terms of hardware, such as Figure 1 As shown, the controller includes at least a processor 1212 and memory 1213. The processor 1212 is a central processing unit (CPU) used to process data access requests from outside the storage device 120 (servers or other storage systems), and also to process requests generated internally by the storage device 120. For example, when the processor 1212 receives write data requests from the data access device 100 through the front-end interface 1211, it temporarily stores the data in these write data requests in memory 1213. When the total amount of data in memory 1213 reaches a certain threshold, the processor 1212 sends the data stored in memory 1213 to at least one of the following hard drives for persistent storage: a mechanical hard drive 1221, a solid-state drive (SSD) 1222, or another hard drive 1223, through a back-end port.

[0064] Memory 1213 refers to internal memory that directly exchanges data with the processor. It can read and write data at any time and at high speed, serving as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory, such as random access memory (RAM) or read-only memory (ROM). For example, RAM can be DRAM or SCM. DRAM is a semiconductor memory and, like most RAM, is a type of volatile memory device. However, DRAM and SCM are merely illustrative examples in this embodiment; memory can also include other types of RAM, such as static random access memory (SRAM). For read-only memory, examples include programmable read-only memory (PROM) and erasable programmable read-only memory (EPROM). Additionally, memory 1213 can also be a dual in-line memory module (DIMM), i.e., a module composed of dynamic random access memory (DRAM), or an SSD. In practical applications, the controller can be configured with multiple memory modules 1213, and different types of memory modules 1213. This embodiment does not limit the number or type of memory modules 1213. Furthermore, memory modules 1213 can be configured to have a power-saving function. The power-saving function means that when the system loses power and then regains power, the data stored in memory modules 1213 will not be lost. Memory with a power-saving function is called non-volatile memory. Memory modules 1213 store software programs, and processor 1212 can run the software programs in memory modules 1213 to manage the hard disk. For example, the hard disk can be abstracted as a storage resource pool, and the storage resource pool can be provided to the server in the form of logical unit numbers (LUNs). Here, the LUN is actually the hard disk seen on the server. Of course, some centralized storage systems are also file servers themselves, and can provide shared file services to the server.

[0065] like Figure 1As shown, in this system, engine 121 may not have a hard drive slot; the hard drive needs to be placed in hard drive enclosure 122, and the back-end interface 1214 communicates with the hard drive enclosure 122. The back-end interface 1214 exists in the form of an adapter card within engine 121, and two or more back-end interfaces 1214 can be used simultaneously on one engine 121 to connect multiple hard drive enclosures. Alternatively, the adapter card can be integrated onto the motherboard, in which case the adapter card can communicate with processor 1212 via the PCIe bus.

[0066] It should be noted that, Figure 1 Only one engine 121 is shown in the figure. However, in actual applications, the storage system may contain two or more engines 121, and redundancy or load balancing may be performed between multiple engines 121.

[0067] The hard drive enclosure 122 includes a control unit 1225 and several hard drives. The control unit 1225 can have various forms. In one case, the hard drive enclosure 122 is a smart enclosure, such as... Figure 1 As shown, the control unit 1225 includes a CPU and memory. The CPU performs operations such as address translation and reading / writing data. The memory is used to temporarily store data to be written to the hard disk or to read data from the hard disk to be sent to the controller. Alternatively, the control unit 1225 can be a programmable electronic component, such as a data processing unit (DPU). A DPU has the versatility and programmability of a CPU, but is more specialized, capable of efficiently operating on network packets, storage requests, or analysis requests. A DPU differs from a CPU by its high degree of parallelism (the ability to handle a large number of requests). Optionally, the DPU can be replaced by a graphics processing unit (GPU), an embedded neural network processing unit (NPU), or other processing chips. Typically, there can be one, two, or more control units 1225. The functions of the control unit 1225 can be offloaded to the network interface card (NIC) 1226. In other words, in this embodiment, the hard disk enclosure 122 does not contain a control unit 1225; instead, the NIC 1226 performs data reading / writing, address translation, and other computational functions. At this point, network interface card 1226 is a smart network interface card. It can contain a CPU and memory. The CPU is used to perform address translation and data reading / writing operations. Memory is used to temporarily store data to be written to the hard drive or to read data from the hard drive to be sent to the controller. It can also be a programmable electronic component, such as a DPU. There is no ownership relationship between network interface card 1226 and the hard drives in hard drive enclosure 122; network interface card 1226 can access any hard drive in hard drive enclosure 122 (e.g., ...). Figure 1The illustrated hard disk drive 1221, solid-state drive 1222, and other hard disk drives 1223 make it relatively convenient to expand the hard disk when storage space is insufficient. In this embodiment, the other hard disk drive 1223 can be any type of storage device, such as a magnetic tape storage device.

[0068] Depending on the type of communication protocol between engine 121 and disk enclosure 122, disk enclosure 122 may be a serially attached small computer system interface (SAS) disk enclosure, an NVMe (Non-Volatile Memory Express) disk enclosure, or other types of disk enclosures. SAS disk enclosures use the SAS 3.0 protocol, and each enclosure supports 25 SAS disks. Engine 121 connects to disk enclosure 122 via an onboard SAS interface or a SAS interface module. NVMe disk enclosures function more like a complete computer system, with NVMe disks inserted into them. The NVMe disk enclosure then connects to engine 121 via an RDMA port. In some cases, engine 121 may also be referred to as a disk management device or storage controller.

[0069] In terms of hardware implementation, the hard disk enclosure 122 can be installed in the storage system, or the hard disk enclosure 122 can be encapsulated and set up independently. When the hard disk enclosure 122 exists independently, it can also be called a storage device or a storage system. This application does not limit this.

[0070] In one alternative implementation, storage device 120 is a centralized storage system integrating disk and controller. Storage device 120 does not have the aforementioned hard disk enclosure 122, and engine 121 is used to manage multiple hard drives connected via hard disk slots. The functionality of the hard disk slots can be implemented by backend interface 1214.

[0071] In some alternative implementations, storage device 120 is a distributed storage system. The distributed storage system includes a cluster of compute nodes and a cluster of storage nodes. The compute node cluster includes one or more compute nodes that can communicate with each other. Compute nodes can be servers, desktop computers, or controllers of storage arrays, etc. Hardware-wise, compute nodes can include processors, memory, and network interface cards (NICs), etc. The processor is a CPU used to process data access requests from outside the compute node or requests generated internally within the compute node. For example, when the processor receives a write data request from a user, it temporarily stores the data in the write data request in memory. When the total amount of data in memory reaches a certain threshold, the processor sends the data stored in memory to the storage node for persistent storage. In addition, the processor is also used for data computation or processing, such as metadata management, deduplication, data compression, virtualization of storage space, and address translation. In the embodiments provided in this application, the storage node can be a magnetic tape storage medium or other types of hard disks, etc. It is understood that the storage system described in the embodiments of this application can be a distributed storage system integrating storage and computing, or a distributed storage system with separate storage and computing; this application does not limit this.

[0072] For example, a distributed storage system can be implemented using network attached storage (NAS) technology. NAS refers to a network storage architecture that provides storage resources through file-level data access and sharing over an Internet Protocol (IP) network. In a NAS scenario, the NAS is an external device for the server / host, used to provide file-level storage space for the server / host in the distributed storage system.

[0073] It is worth noting that the above examples are merely possible implementations of the data access system provided in this embodiment and should not be construed as limiting this application. For example, Figure 1In the storage device 120 shown, data is stored as files on various hard drives. The files stored on each hard drive constitute a file storage system, which could be a distributed file system, such as a network file system (NFS). NFS is both a distributed file system and a network protocol used for accessing and sharing files between devices on the same local area network. For example, a NAS system can be implemented using the NFS protocol. A network file system is a low-cost network file-sharing option that allows users and applications to access, store, and update files on remote computers, just like using direct-attached storage. Network file systems use the Remote Procedure Call (RPC) protocol to route requests between clients and servers. While participating devices need to support network file systems, they do not need to know the details of the network. It is worth noting that RPC can be insecure, so network file systems should only be deployed on trusted networks behind firewalls. Although Windows supports this protocol, it is primarily used in Linux environments.

[0074] Regarding the controller described above, this application provides an optional example, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of a controller provided in this application. The controller 200 can be used to implement the functions of the controller in the engine 121 of the storage device 120 described above.

[0075] The following is combined Figure 2 The controller 200 is described by way of example. It includes a computing module 201, an I / O interface 202 and a preprocessing module 203. Figure 2 In addition to describing the controller 200, the document also describes related devices that may be communicatively connected to the controller 200. Figure 1 Examples of such devices (not shown in the diagram) include training device 204, database 205, data storage system 206, data acquisition device 207, etc.

[0076] The training device 204 can be an AI chip, a terminal, or other computing devices that support integer calculations, such as servers or cloud devices.

[0077] As one possible embodiment, the controller 200 and training device 204 are deployed on different physical devices (e.g., servers in a server or cluster), or the controller 200 and training device 204 are different physical devices. Exemplarily, the controller 200 and training device 204 are AI chips deployed on different physical devices. For example, the execution device 110 may be a GPU, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The training device 204 may be a GPU, NPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program in this application.

[0078] In another possible embodiment, the controller 200 and the training device 204 are deployed on the same physical device, or the controller 200 and the training device 204 are on the same physical device.

[0079] The data acquisition device 207 is used to collect training data and store the training data in the database 205. The data acquisition device 207 and the controller 200 can be the same or different devices.

[0080] The training device 204 is used to train the neural network using training data until the loss function in the neural network converges and its value is less than a certain threshold, at which point the neural network training is complete, thus enabling the neural network to achieve a certain level of accuracy. For example, the training device 204 can train the neural network using unstructured data such as PDF files as input and the parsed outline information as output. Alternatively, the training device 204 can train the neural network using tables as input and the predicted row and column index types as output. Yet another example is using a page from a PDF file as input and the recognized text content as output.

[0081] During training, the training device 204 performs backpropagation based on the loss function to update the parameters of the neural network with gradients until the loss function value is less than a specific threshold. Alternatively, all training data in the database 205 can be used for training, thus completing the neural network training and enabling it to perform target functions such as layout parsing, character recognition, and table structure recognition. Furthermore, the training device 204 configures the trained neural network 208 to the controller 200. The controller 200 is used to parse and store unstructured data such as PDF files based on the trained neural network 208.

[0082] The aforementioned neural network 208 can be a single-function model or a multi-function model composed of multiple models. For example, neural network 208 may include a layout parsing model, an optical character recognition model, and / or a structure detection model.

[0083] This application does not limit the specific implementation of the aforementioned layout parsing model, optical character recognition model, and / or structure detection model. For example, the layout parsing model can be based on a Swin transformer as the backbone, with deformable attention as the encoder and decoder. The optical character recognition model can be an optical character recognition (OCR) network based on a Convolutional Recurrent Neural Network (CRNN) with an added transformer. The structure detection model can be a structure detection network based on YOLOv9.

[0084] In some embodiments, the controller 200 and the training device 204 are the same computing device. The computing device can configure the trained neural network 208 to itself and use the trained neural network 208 to achieve the above-mentioned target function.

[0085] In other embodiments, the training device 204 can configure the trained neural network 208 to multiple controllers 200. Each controller 200 uses the trained neural network 208 to achieve the aforementioned target function.

[0086] It should be noted that in practical applications, the training data maintained in database 205 may not all come from data acquisition device 207; it may also be received from other devices. Furthermore, training device 204 may not necessarily train the neural network entirely based on the training data maintained in database 205; it may also obtain training data from the cloud or other sources. The above description should not be construed as limiting the embodiments of this application.

[0087] Furthermore, based on the functions performed by the controller 200, the following will describe the sub-architecture of the controller 200, such as the computing module 201, the I / O interface 202, and the preprocessing module 203.

[0088] Taking controller 200 as a storage controller as an example, computing module 201 can be a GPU, CPU, other general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., with AI functionality on the storage controller. Computing module 201 can include one or more AI chips with AI functionality, or it can be a combination of an AI chip and a CPU without AI functionality; this is not limited here. For example, computing module 201 is... Figure 1 The functional module corresponding to processor 1212 in the controller shown.

[0089] For example, the calculation module 201 is used to run the neural network 208 to realize the layout parsing function of the neural network 208, so as to obtain the parsed outline information.

[0090] For example, the computing module 201 is used to run the neural network 208 to realize the character recognition function of the neural network 208, so as to obtain the text content.

[0091] For example, the calculation module 201 is used to run the neural network 208 to realize the table structure recognition function of the neural network 208, so as to obtain the index type of the table's rows and columns.

[0092] I / O interface 202 is used for data interaction with external devices or apparatuses. Users can input data into I / O interface 202 through external devices or apparatuses, such as instructions to instruct controller 200 to initiate the execution of data storage methods. Alternatively, the input data can also come from database 205.

[0093] The preprocessing module 203 is used to preprocess the input data received from the I / O interface 202. In this embodiment, the preprocessing module 203 can be used to generate training data, such as a training set, a validation set, and a test set, based on the input data received from the I / O interface 202.

[0094] During the preprocessing of input data by the controller 200, or during the calculation process performed by the computing module 201 of the controller 200, the controller 200 can call data, code, etc., in the data storage system 206 for corresponding processing, and can also store the processed data and instructions into the data storage system 206. For example, the data storage system 206 can be... Figure 1 The memory module 1213 or hard disk enclosure 122 is shown.

[0095] Finally, I / O interface 202 can return the processing result to the user, so that the user can view the processing result.

[0096] Figure 2 This is merely a schematic diagram provided in an embodiment of this application. Figure 2 The positional relationships between the devices, equipment, components, modules, etc., shown do not constitute any limitation; for example, in Figure 2 In this context, the data storage system 206 is an external memory relative to the controller 200. In other cases, the data storage system 206 may also be placed within the controller 200.

[0097] The following example illustrates how the controller 200 executes the data storage method provided in the embodiments of this application.

[0098] Figure 3a This application provides a flowchart illustrating a data storage method that can be applied to the above-mentioned... Figure 1 or Figure 2 The controller is shown. In possible embodiments of this application, the subject executing the data storage method is not limited. For example, the data storage method can also be applied to any device with computing power and data processing functions, such as a processor of a computing device with storage capabilities.

[0099] In this embodiment, the controller is equipped with an AI chip and corresponding layout analysis model, optical character recognition model, and / or structure detection model. Please refer to... Figure 3a The data storage method provided in this application embodiment may include the following steps 301-304.

[0100] Step 301: The controller obtains the PDF file.

[0101] The controller obtains the PDF file that needs to be stored. For example, the PDF file may be generated by an internal application of the computing device to which the controller belongs, or it may be sent to the controller by the data access device 100.

[0102] Step 302: The controller parses the PDF file and obtains the parsing result.

[0103] The controller selects the method for parsing the PDF file based on the hardware of its computing device, and then parses the PDF file using the selected method to obtain the parsing result.

[0104] The hardware of the computing device can refer to whether the computing device is equipped with an AI chip. If it is equipped with an AI chip, the controller calls the AI ​​chip to parse the PDF file. If it is not equipped with an AI chip, the controller calls a conventional processor (such as a CPU) to parse the PDF file.

[0105] like Figure 3b As shown, the steps for parsing a PDF file may include steps 3021-3023.

[0106] Step 3021: The controller detects whether the computing device includes an AI chip.

[0107] The controller detects whether its own computing device is equipped with an AI chip.

[0108] As one possible implementation, the controller queries the hardware information of its own computing device (e.g., the controller can query via command) and determines whether an AI chip is configured based on the hardware information.

[0109] Optionally, the controller uses different commands to query hardware information in different operating systems or development environments. If the hardware information queried by the controller includes information about the AI ​​chip, it determines that its own computing device is equipped with an AI chip.

[0110] For example, the corresponding command is grep UUID (universally unique identifier) ​​when the operating system is Linux, the corresponding command is ManagementObjectSearcher when the operating system is Windows, and the corresponding command is psutil or platform when the development environment is Python.

[0111] Optionally, the computing device to which the controller belongs can refer to the controller itself, in which case the AI ​​chip can be the controller's computing module 201. The computing device to which the controller belongs can also be the storage device 120, in which case the AI ​​chip can be a chip with AI inference capabilities, such as a GPU or NPU, that is connected to the controller in the storage device 120.

[0112] Step 3022: If the computing device does not include an AI chip, the controller extracts text from the PDF file and obtains the parsing results.

[0113] Even without an AI chip in its own computing device, the controller extracts text from a PDF file and obtains the parsed results. These parsed results include the text information extracted from the PDF file.

[0114] As one possible implementation, the controller uses a processor without AI inference capabilities, such as a CPU, to extract text from PDF files.

[0115] Optionally, the above text extraction can be implemented based on a Python library with text extraction functionality, and this application does not limit this.

[0116] The above analysis results contain only simple text information within a simple PDF file, which consumes less computing resources of the controller or computing device and is suitable for computing devices without AI chips.

[0117] Step 3023: If the computing device includes an AI chip, the controller uses the AI ​​chip to parse the PDF file and obtain the parsing result.

[0118] With its own computing device equipped with an AI chip, the controller parses the PDF file and obtains the parsing results. These results include outline information extracted from the PDF file.

[0119] As one possible implementation, the controller uses an AI chip with AI inference capabilities to intelligently parse PDF files.

[0120] Optionally, the parsing of the above outline information can be achieved based on a layout parsing model, an optical character recognition model, and a structure detection model.

[0121] For example, the controller uses an AI chip to run a layout parsing model to extract elements of various categories and their coordinate frames from a PDF file, and outputs outline information based on the positional relationships between the coordinate frames of each category of elements. The controller also uses an AI chip to run a structure recognition model to output at least one cell of a table-type element. Finally, the controller uses an AI chip to run an optical character recognition model to perform text recognition on each category of elements and outputs the text content contained within each category of elements.

[0122] Among them, such as Figure 4 As shown, element categories can include headers, footers, headings, text, tables, images, etc. Figure 5 As shown, the outline information includes headings such as Heading 1, Heading 2, and Heading 3, as well as the content information of the main text corresponding to Heading 1-3 respectively.

[0123] The above outline parsing process is only an introduction to the principle; for specific outline parsing steps, please refer to [link / reference needed]. Figure 6Steps 601-604 shown will not be repeated here.

[0124] The above parsing results contain the outline information of the PDF file, which places high demands on the computing resources of the controller or computing device. It requires the use of AI chips for model inference and is suitable for computing devices equipped with AI chips. It has a high utilization rate of the user's hardware resources and avoids the need for users to perform real-time parsing of PDF files and subsequent manual algorithm analysis when they need to access PDF files to obtain useful information, thus reducing the workload of users in secondary file processing.

[0125] Step 303: The controller associates the parsing results and the PDF file to obtain the file to be stored.

[0126] The controller associates the parsing results with the PDF file's file identifier and stores them to obtain the file to be stored.

[0127] As one possible implementation, the controller stores a mapping table that represents the association between the parsing results and the PDF file.

[0128] Optionally, the PDF file has a unique file identifier in the file system, and the parsing result also has a unique file identifier in the file system. For example, the file identifier could be the filename, index, etc.

[0129] Optionally, the mapping table includes multiple key-value pairs. A PDF file and its corresponding one or more parsing results are associated with a key-value pair in the mapping table. For example, if the file identifier of the PDF file is "A", and the file identifiers of the two parsing results corresponding to the PDF file are "B" and "C" respectively, the key data of the corresponding key-value pair in the mapping table is "A", and the data of the value data corresponding to the key includes "B" and "C".

[0130] Step 304: The controller stores the file to be stored.

[0131] The controller writes the files to be stored to disk. Corresponding to step 3022, the files written to disk include a PDF file and text information extracted from the PDF file. Corresponding to step 3023, the files written to disk include a PDF file and an outline extracted from the PDF file.

[0132] As one possible implementation, the controller sends the file to be stored in memory to the disk frame 122 for persistent storage.

[0133] As one possible implementation, when the controller writes a file to disk, it generates metadata for that file. This metadata allows users to retrieve the corresponding parsing results when accessing a PDF file. For example, the PDF file's metadata includes its file identifier and storage address (e.g., physical address, logical address), while the metadata for the parsing results includes the file identifier and storage address of the parsing results. The metadata can also include other types of information such as data size and access history, which will not be elaborated upon here.

[0134] Based on steps 301-304 above, when storing unstructured data such as PDF files, in addition to storing the original PDF file, the parsing results of the PDF file are also associated with the PDF file and stored together on disk. In this way, when users subsequently access the PDF file, they can simultaneously access the corresponding parsing results, avoiding the need for users to manually parse the PDF file using algorithms each time they access it, thus reducing the workload of secondary file processing for users.

[0135] When storing unstructured data such as PDF files, the storage method can be determined based on the hardware of the computing device. This ensures that the storage of unstructured data is compatible with the hardware of the user's computing device, improves the resource utilization of the AI ​​chip or conventional CPU of the computing device, and thus increases the flexibility of storing unstructured data.

[0136] Furthermore, when a large number of file streams are simultaneously accessed and stored in the data storage system, the high concurrency efficiency of AI chips such as NPU and GPU means that only a small number of AI chips are needed to complete the parsing and storage of unstructured data, reducing reliance on processor resources and lowering hardware costs for users.

[0137] After the controller writes the file to be stored to disk in step 304 above, the data access device 100 can access the PDF file in the storage device 120. In order to avoid the user accessing the PDF file and parsing the PDF file again, the controller of the storage device 120 can return the PDF file and its corresponding parsing result to the data access device 100.

[0138] For example, the specific steps by which the controller returns a PDF file and its corresponding parsing result based on the data access request from the data access device 100 may include the following steps 305-308. Figure 3a (Not shown in the image).

[0139] Step 305: The controller receives a data access request.

[0140] The data access request includes the file identifier of the PDF file.

[0141] Step 306: The controller queries the mapping table for the file identifier of the parsed result based on the file identifier of the PDF file.

[0142] Please refer to the file identifiers of the PDF files and parsing results in the mapping table. Figure 3a The relevant description of step 303 will not be repeated here.

[0143] Step 307: The controller queries the metadata based on the file identifier of the PDF file and the file identifier of the parsing result to obtain the storage address of the PDF file and the storage address of the parsing result.

[0144] The metadata is generated when the file to be stored is written to disk; please refer to [link / reference needed] for details. Figure 3a The relevant description of step 304 will not be repeated here.

[0145] Step 308: The controller returns a data access response.

[0146] The controller returns a data access response to the data access device 100. The data access response includes the data stored at the storage address of the PDF file and the data stored at the storage address of the parsing result.

[0147] In this way, after the PDF file and its corresponding parsing results are saved to the disk at the same time, users can obtain the corresponding parsing results of the PDF file at the same time when accessing the PDF file. This avoids users having to manually parse the PDF file with algorithms to obtain the required data every time they access it, thus reducing the workload of users processing the file twice.

[0148] The above text combined Figure 3a The data storage method provided in the embodiments of this application has been described in general. The following will be combined with... Figure 6 The steps for parsing the outline information are explained in detail.

[0149] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating the steps involved in parsing outline information provided in this application. Figure 6 As shown, taking the parsing steps of the outline information as an example, which are performed by the AI ​​chip, the parsing steps of the outline information may include the following steps 601-604.

[0150] Step 601: Use the layout parsing model to parse the PDF file and obtain the layout parsing result output by the layout parsing model.

[0151] The layout analysis model takes a PDF file as input and outputs the layout analysis results of the PDF file.

[0152] As one possible implementation, the layout parsing model performs layout parsing on the PDF file to obtain layout parsing results, which include elements of various categories in the PDF file and the coordinate frames of each category of elements.

[0153] Optionally, the categories of elements in the aforementioned PDF file may include headings and body text. Headings may include at least one level heading, such as first-level headings, second-level headings, and third-level headings (this embodiment uses third-level headings as an example, but is not limited to only including third-level headings).

[0154] Optionally, the categories of elements in the aforementioned PDF file may also include headers, footers, etc., which will not be elaborated here.

[0155] For example, such as Figure 4 As shown, a single page in a PDF file includes header 1, first-level heading 1, body text 1, second-level heading 1, body text 2, second-level heading 2, body text 3, body text 4, body text 5, body text 6, body text 7, and footer 1.

[0156] The main body includes text, tables, and images. For example, main body 1, main body 2, main body 3, main body 5, main body 6, and main body 7 are text, while main body 4 is an image.

[0157] Step 602: Based on the positional relationship between the coordinate frames of elements of each category, determine the relationship between the headings at each level and the body text in at least the first-level heading.

[0158] As one possible implementation, the positional relationships between the coordinate boxes of elements of various categories include the positional relationships between headings at all levels and the body text, as well as the positional relationships between headings at all levels.

[0159] Optionally, the relationship between headings at each level and the main text can be determined based on the positional relationship between headings at each level and the main text, as well as the positional relationship between headings at each level.

[0160] For example, if the position of Heading 1 in a PDF file precedes Heading 1 and Heading 2, and the position of Body Text 2 in a PDF file is between Heading 1 and Heading 2, then a relationship is determined between Heading 1 and Body Text 1. Similarly, if the AI ​​chip identifies that Heading 1 in a PDF file is between Heading 1 and Heading 2, and Body Text 3 in a PDF file is between Heading 1 and Heading 2, then a relationship is determined between Heading 2 and Heading 2.

[0161] In the embodiments of this application, the determination of the above-mentioned association relationship can be achieved by the AI ​​chip using a layout parsing model.

[0162] As one possible implementation, AI chips can also combine the semantics of each level of headings with the semantics of the corresponding body text to verify the relationship between each level of headings and the body text, thereby improving the accuracy of the relationship.

[0163] Step 603: Extract the content information of the main text associated with each level of heading.

[0164] As one possible implementation, the body of the text includes the main body.

[0165] Optionally, the AI ​​chip runs an optical character recognition (OCR) model, inputting the headings at each level and the text associated with each level into the OCR model, and obtaining the text content of the headings at each level and the text associated with each level of heading from the OCR model. Then, the text content of each level of heading and the text associated with each level of heading is matched with the coordinate boxes of elements in each category.

[0166] For example, the text content of Heading 1 is matched with its coordinate frame in the PDF file layout; the text content of Heading 2 is matched with its coordinate frame in the PDF file layout; the text content of Heading 2 is matched with its coordinate frame in the PDF file layout; and the text content of Body Text 2 is matched with its coordinate frame in the PDF file layout. In this way, the positions of the text content of each level of heading and body text are determined in the PDF file layout, thereby mapping each level of heading to its corresponding body text content information, facilitating the accuracy of subsequent outline output.

[0167] The text content can be a summary, outline, or other key information.

[0168] As one possible implementation, the text includes tables.

[0169] Optionally, the step of extracting the content information of the table can be implemented by the AI ​​chip running an optical character recognition model, and may include the following sub-steps 1-5.

[0170] Sub-step 1: Input the headings at each level into the optical character recognition model running on the AI ​​chip to obtain the text content of the headings at each level output by the optical character recognition model.

[0171] Sub-step 2: Input the tables associated with each level of headings into the structure detection model running on the AI ​​chip, and obtain at least one cell of the tables associated with each level of headings output by the structure detection model.

[0172] Sub-step 3: Input at least one cell into the optical character recognition model running on the AI ​​chip to obtain the text content contained in at least one cell output by the optical character recognition model.

[0173] Sub-step 4: Based on cell row and column division, cluster the text content contained in at least one cell to obtain the index type of the row and column of at least one cell output by the table recognition model.

[0174] Clustering refers to dividing a collection of physical or abstract objects into multiple classes composed of similar objects. This application does not limit the clustering algorithm; for example, clustering algorithms can be hierarchical methods, partition-based methods, density-based methods, etc.

[0175] For example, regarding the content shown in Table 1, Table 1 is divided into 12 cells based on rows and columns. Following the top-to-bottom order of each column, the contents of each cell are "Name A", "Name B", "Name C", "Male", "Male", "Female", "15", "17", and "16". The clustering result of Table 1 could be that the first column's indicator type is "Name", the second column's indicator type is "Gender", and the third column's indicator type is "Age".

[0176] Table 1

[0177] NameA male 15 Name B male 17 Name C female 16

[0178] Sub-step 5: Match the index types of each level of headings and the tables associated with each level of headings with the coordinate frames of each category of elements.

[0179] Step 604: Determine the outline information based on the relationships and the content information of the main text associated with the headings at each level.

[0180] Based on the format of the outline information, the body text associated with each level of heading is set in the corresponding position of each level of heading to obtain the outline information.

[0181] For example, such as Figure 5 As shown, the first-level heading 1 includes the second-level headings 1 and 2. The body text corresponding to second-level heading 1 is the content information of body text 2, and the body text corresponding to second-level heading 2 is the content information of body text 3 to body text 7. Among them, body text 4 is an image, which can be directly displayed in the outline information.

[0182] Based on steps 601-604 above, if the user's computing device has an AI chip or AI processing capability, the AI ​​chip can be called to intelligently parse the PDF file and obtain the outline information of the PDF file. This avoids the user having to manually select the corresponding algorithm to analyze the PDF file, reducing the user's workload of secondary file processing.

[0183] To complement the data storage method provided in the embodiments of this application, the embodiments of this application also provide a data storage device 700, which is used to execute the above-described data storage method. For example... Figure 7 As shown, the data storage device 700 includes:

[0184] Module 710 is used to acquire PDF files.

[0185] The parsing module 720 is used to parse PDF files and obtain parsing results; the parsing results include text information obtained by extracting text from the PDF file, or outline information of the PDF file.

[0186] The parsing module 720 is also used to associate the parsing results with the PDF file to obtain the file to be stored.

[0187] Storage module 730 is used to store files to be stored.

[0188] As one possible implementation, the parsing module 720 is specifically used to map the file identifier of the PDF file to the file identifier of the parsing result in the mapping table.

[0189] As one possible implementation, the parsing module 720 is specifically used to: detect whether the computing device includes an AI chip; if the computing device does not include an AI chip, perform text extraction on the PDF file to obtain the parsing result; the parsing result includes the text information obtained from the text extraction; if the computing device includes an AI chip, use the AI ​​chip to parse the PDF file to obtain the parsing result; the parsing result includes outline information.

[0190] As one possible implementation, the parsing module 720 is specifically used to: input the PDF file into the layout parsing model running on the AI ​​chip, and obtain the layout parsing result output by the layout parsing model; the layout parsing result includes elements of various categories in the PDF file and the coordinate frames of each category of elements; and determine the outline information based on the positional relationship between the coordinate frames of each category of elements.

[0191] As one possible implementation, the categories include headings and body text, with headings including at least a first-level heading.

[0192] As one possible implementation, the parsing module 720 is specifically used to: determine the relationship between the headings at all levels and the body text in at least one level of heading based on the positional relationship between the coordinate frames of elements of each category; the relationship is used to indicate the body text associated with any level of heading; extract the content information of the body text associated with each level of heading; and determine the outline information based on the relationship and the content information of the body text associated with each level of heading.

[0193] As one possible implementation, the main body includes text. The parsing module 720 is specifically used to: input the headings at each level and the text associated with each level of heading into the optical character recognition model running on the AI ​​chip, and obtain the text content of the headings at each level and the text associated with each level of heading output by the optical character recognition model; and match the text content of the headings at each level and the text associated with each level of heading with the coordinate boxes of elements in each category.

[0194] As one possible implementation, the main text includes tables. The parsing module 720 is specifically used for: inputting headings at all levels into the optical character recognition model running on the AI ​​chip, obtaining the text content of the headings at all levels output by the optical character recognition model; inputting tables associated with headings at all levels into the structure detection model running on the AI ​​chip, obtaining at least one cell of the tables associated with headings at all levels output by the structure detection model; inputting at least one cell into the optical character recognition model running on the AI ​​chip, obtaining the text content contained in at least one cell output by the optical character recognition model; clustering the text content contained in at least one cell based on the row and column division of the cells, obtaining the row and column index types of at least one cell output by the table recognition model; and matching the index types of headings at all levels and the tables associated with headings at all levels with the coordinate frames of elements in each category.

[0195] As one possible implementation, the main text includes tables. The parsing module 720 is specifically used to: extract semantic information from headings at all levels and the associated main text; and correct the relationships between headings and main text in the outline information based on the semantic information.

[0196] As one possible implementation, the data storage device further includes a query module and a response module. The acquisition module 710 is further configured to: receive a data access request; the data access request includes the file identifier of the PDF file. The query module is configured to: query the mapping table for the file identifier of the parsed result based on the file identifier of the PDF file; and query metadata based on the file identifier of the PDF file and the file identifier of the parsed result to obtain the storage address of the PDF file and the storage address of the parsed result. The response module is configured to: return a data access response; the data access response includes data stored at the storage address of the PDF file and data stored at the storage address of the parsed result.

[0197] It should be understood that the above Figure 7The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0198] Figure 8 This is a schematic diagram of the structure of a computing device provided in this embodiment. Figure 8 As shown, the computing device 800 includes a processor 810, a bus 820, a memory 830, a communication interface 840, and a memory unit 850 (also referred to as a main memory unit). The processor 810, memory 830, memory unit 850, and communication interface 840 are connected via the bus 820.

[0199] It should be understood that in this embodiment, the processor 810 may be a CPU, but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0200] The processor may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, or one or more integrated circuits used to control the execution of the program in this application.

[0201] The communication interface 840 is used to enable communication between the computing device 800 and external devices or components. In this embodiment, the computing device 800 is used to implement... Figure 1 When the storage device 120 is in its function, the communication interface 840 is used as a physical port for sending and receiving data.

[0202] Bus 820 may include a pathway for transferring information between the aforementioned components (such as processor 810, memory unit 850, and memory 830). In addition to a data bus, bus 820 may also include a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 820 in the figure. Bus 820 may be a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Bus 820 can be divided into address bus, data bus, control bus, etc.

[0203] As an example, computing device 800 may include multiple processors. A processor may be a multi-core (multi-CPU) processor. Here, "processor" can refer to one or more devices, circuits, and / or computing units used to process data (e.g., computer program instructions).

[0204] It is worth noting that, Figure 8 Taking the computing device 800, which includes one processor 810 and one memory 830, as an example, the processor 810 and the memory 830 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0205] Memory cell 850 may be volatile memory or non-volatile memory, or may include both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0206] The memory 830 can correspond to the storage medium used to store computer instructions and other information in the above method embodiments, such as a disk, like a mechanical hard disk or a solid-state hard disk.

[0207] The computing device 800 described above can be a general-purpose device or a special-purpose device. For example, the computing device 800 can be an edge device (e.g., a box carrying a chip with processing capabilities). Optionally, the computing device 800 can also be a network device, a server, or other device with computing or storage capabilities.

[0208] It should be understood that the computing device 800 according to this embodiment may correspond to the data storage device 700 in this embodiment, and may correspond to the execution device according to... Figure 3a The corresponding entities in the method, and the above and other operations and / or functions of each module in the data storage device 700, are respectively implemented to achieve Figure 3a For the sake of brevity, the corresponding process of the Chinese method will not be elaborated here.

[0209] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in an electronic device (such as a network device). Of course, the processor and storage medium can also exist as discrete components in an electronic device.

[0210] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data storage method, characterized in that, Applied to a computing device, the method includes: Get PDF files; The PDF file is parsed to obtain the parsing result; the parsing result includes text information obtained by extracting text from the PDF file, or outline information of the PDF file; By associating the parsing result with the PDF file, a file to be stored is obtained; the file to be stored includes the PDF file and the parsing result. Store the file to be stored.

2. The method according to claim 1, characterized in that, The association between the parsing result and the PDF file includes: The file identifier of the PDF file is mapped to the file identifier of the parsing result in the mapping table.

3. The method according to claim 1 or 2, characterized in that, The process of parsing the PDF file to obtain the parsing result includes: Detect whether the computing device includes an AI artificial intelligence chip; If the computing device does not include the AI ​​chip, text extraction is performed on the PDF file to obtain a parsing result; the parsing result includes the text information obtained from the text extraction. If the computing device includes the AI ​​chip, the AI ​​chip is used to parse the PDF file to obtain the parsing result; the parsing result includes outline information.

4. The method according to claim 3, characterized in that, The process of using the AI ​​chip to parse the original PDF file includes: The PDF file is input into the layout parsing model running on the AI ​​chip to obtain the layout parsing result output by the layout parsing model; the layout parsing result includes elements of various categories in the PDF file and the coordinate boxes of the elements of various categories; The outline information is determined based on the positional relationship between the coordinate frames of the elements in each category.

5. The method according to claim 4, characterized in that, The categories include headings and body text, and the headings include at least one level heading.

6. The method according to claim 5, characterized in that, Determining the outline information based on the positional relationship between the coordinate frames of the elements in each category includes: Based on the positional relationship between the coordinate frames of the elements in each category, the association relationship between each level of heading and the main text in the at least one level heading is determined; the association relationship is used to indicate the main text associated with any level heading. Extract the content information of the main text associated with each level of heading; The outline information is determined based on the aforementioned relationships and the content information of the main text associated with the various levels of headings.

7. The method according to claim 6, characterized in that, The main text includes text, and the extraction of content information of the main text associated with the various levels of headings includes: The text associated with each level of headings is input into the optical character recognition model running on the AI ​​chip to obtain the text content of each level of headings and the text associated with each level of headings output by the optical character recognition model. Match the text content of each level of heading and the text associated with each level of heading with the coordinate boxes of each category of elements.

8. The method according to claim 6, characterized in that, The main text includes tables, and the extraction of content information of the main text associated with the various levels of headings includes: The various levels of headings are input into the optical character recognition model running on the AI ​​chip to obtain the text content of the headings at each level output by the optical character recognition model; The table associated with the various levels of headings is input into the structure detection model running on the AI ​​chip to obtain at least one cell of the table associated with the various levels of headings output by the structure detection model; The at least one cell is input into the optical character recognition model running on the AI ​​chip to obtain the text content contained in the at least one cell output by the optical character recognition model; Based on the row and column division of the cells, the text content contained in the at least one cell is clustered to obtain the index type of the row and column of the at least one cell output by the table recognition model; Match the index types of the headings at each level and the tables associated with the headings at each level with the coordinate frames of the elements in each category.

9. The method according to any one of claims 6-8, characterized in that, The method further includes: Extract the semantic information of the headings at each level and the associated text at each level; The semantic information is used to correct the relationship between the headings at each level and the main text in the outline information.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Receive a data access request; the data access request includes the file identifier of the PDF file; The storage address of the PDF file and the storage address of the parsing result are obtained based on the file identifier of the PDF file; A data access response is returned based on the storage address of the PDF file and the storage address of the parsing result; the data access response includes either the PDF file or the parsing result.

11. A data storage device, characterized in that, include: The acquisition module is used to acquire PDF files; The parsing module is used to parse the PDF file and obtain the parsing result; the parsing result includes text information obtained by extracting text from the PDF file, or outline information of the PDF file; The parsing module is further configured to associate the parsing result with the PDF file to obtain a file to be stored; the file to be stored includes the PDF file and the parsing result. A storage module is used to store the file to be stored.

12. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-10.

13. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-10.

Images