Tree file streaming method, device, equipment and storage medium

Abstract

Embodiments of the present disclosure provide a tree file streaming processing method, device and equipment and a storage medium. The method comprises: dividing a start tag and its corresponding end tag in a tree file to be processed, and text content between adjacent start tags and end tags into data blocks; sending the data blocks to a processor in sequence according to the order of the divided data blocks to form a data stream; and performing streaming processing on the data stream. In this way, the processor can continuously perform streaming processing on small batches of data, thereby reducing the pressure on the processor while improving the file processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for streaming processing of tree-structured files. Background Technology

[0002] In traditional data processing, data is first collected uniformly, then stored in a database, and then all data in the table is processed in batches before the results are output. That is, batch processing is completed at once, which does not have high requirements for the timeliness of the data, but has high requirements for processor performance.

[0003] With the development of information systems, the number of systems and the amount of data are constantly increasing, leading to the creation of XML, JSON, and other files to store large-capacity tree-structured data. Meanwhile, HTML files used in web pages and YAML files used in configuration files also employ tree structures. Although processor performance is continuously improving, batch processing methods for extremely large tree-structured files still place a significant burden on the processor, resulting in slow processing efficiency and high latency. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for streaming tree-structured files, which can improve the efficiency of tree-structured file processing.

[0005] In a first aspect, embodiments of this disclosure provide a method for streaming processing of tree-structured files, the method comprising:

[0006] Divide the start label and its corresponding end label in the tree file to be processed, as well as the text content between adjacent start and end labels, into data blocks;

[0007] Data blocks are sent to the processor in the order they are divided, forming a data stream;

[0008] Perform streaming processing on the data stream.

[0009] Among some possible implementations of the first aspect, the method also includes:

[0010] If the tree file meets the preset processing conditions based on its file size and timeliness level, then the tree file will be processed in a streaming manner.

[0011] Among the possible implementations of the first aspect, streaming processing of the data stream includes:

[0012] The data blocks in the data stream are processed sequentially as follows:

[0013] If the current data block is the start label and the current data block is the first data block, then generate the simulated path of the current data block and its corresponding nodes based on the root path and the current data block;

[0014] If the current data block is the start label and the current data block is not the first data block, then the simulation path of the current data block and its corresponding nodes are generated based on the simulation path of the previous data block and the current data block.

[0015] If the current data block is text content, then the current data block is saved to the node of the previous data block;

[0016] If the current data block is the end label and the current data block is not the last data block, then the simulated path of the previous data block of the data block to which the corresponding start label belongs is determined as the simulated path of the current data block.

[0017] If the current data block is the end label and is the last data block, then the simulated path of the current data block is determined to be the root path.

[0018] Among some possible implementations of the first aspect, the method also includes:

[0019] If the current data block is the start label and is not the first data block, and it contains a label attribute, then the label attribute will be saved to the node of the current data block.

[0020] Among some possible implementations of the first aspect, the method also includes:

[0021] Save the data in each node to memory;

[0022] When performing data statistics, the data corresponding to each node is extracted from memory and a statistics file is generated.

[0023] Among some possible implementations of the first aspect, the method also includes:

[0024] Based on the input simulation path, query the corresponding node and retrieve the data in the node.

[0025] In some possible implementations of the first aspect, the tree file is an XML file, a JSON file, an HTML file, or a YAML file.

[0026] Secondly, embodiments of this disclosure provide a streaming processing apparatus for tree-structured files, the apparatus comprising:

[0027] The partitioning module is used to divide the start label and its corresponding end label in the tree file to be processed, as well as the text content between adjacent start and end labels, into data blocks;

[0028] The sending module is used to send data blocks to the processor in the order they are divided, forming a data stream;

[0029] The processing module is used to perform streaming processing on the data stream.

[0030] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0031] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0032] In this disclosure, tree files can be divided into blocks and sent to the processor in the form of data streams, enabling the processor to continuously stream small batches of data, thereby reducing processor load while improving tree file processing efficiency.

[0033] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0034] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0035] Figure 1 A flowchart of a streaming method for tree-structured files provided in an embodiment of this disclosure is shown;

[0036] Figure 2 This illustration shows a schematic diagram of a data block partitioning method provided in an embodiment of the present disclosure;

[0037] Figure 3 This illustration shows a schematic diagram of simulated path changes and node generation provided by an embodiment of the present disclosure;

[0038] Figure 4 A structural diagram of a streaming device for tree-structured files provided in an embodiment of this disclosure is shown;

[0039] Figure 5 A structural diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0041] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0042] To address the problems in the background art, this disclosure provides a method, apparatus, device, and storage medium for streaming processing of tree-structured files.

[0043] Specifically, the start tag and its corresponding end tag in the tree file to be processed, as well as the text content between adjacent start tags and end tags, can be divided into data blocks. The data blocks are then sent to the processor in the order they are divided to form a data stream. This allows the processor to perform streaming processing on the data stream, thereby reducing processor load while improving the efficiency of tree file processing.

[0044] The following detailed description, with reference to the accompanying drawings and specific embodiments, illustrates the streaming processing method, apparatus, device, and storage medium for tree-shaped files provided in this disclosure.

[0045] This streaming processing method can be applied to mobile electronic devices or non-mobile electronic devices. For example, mobile electronic devices can be tablets, laptops, handheld computers, or ultra-mobile personal computers (UMPCs), while non-mobile electronic devices can be personal computers (PCs), supercomputers, or servers.

[0046] Figure 1 A flowchart of a streaming method for tree-structured files provided in an embodiment of this disclosure is shown, such as... Figure 1 As shown, the streaming processing method 100 may include the following steps:

[0047] S110, divide the start label and its corresponding end label in the tree file to be processed, as well as the text content between adjacent start and end labels, into data blocks.

[0048] Here, data blocks are defined as labels and text content. That is, when dividing a tree-structured file into data blocks, the start label is divided into a data block, the end label is divided into a data block, and the text content between adjacent start and end labels is also divided into data blocks.

[0049] For example, the tags and text content in the tree file can be divided into data blocks in sequence, thereby dividing the start tag and its corresponding end tag, as well as the text content between adjacent start tags and end tags in the tree file to be processed, into data blocks.

[0050] It should be noted that the streaming processing method provided in this disclosure is highly versatile, and the tree file can be a representative tree structure file such as XML, JSON, HTML, or YAML. Taking XML as an example, <*> represents a start tag.< / *> This indicates the corresponding closing tag, <*> and< / *> The portion between them is the tag body, and the data blocks can be divided as follows: Figure 2 As shown.

[0051] The left side shows the XML file, and the right side shows the data blocks obtained after dividing the XML file into data blocks, totaling 13 (note that "Part" here is only for comments and does not actually exist). For example... <bookstore> and< / bookstore> If the labels are not adjacent, they are divided into two groups during the partitioning process. <bookstore> 、< / bookstore> Two data blocks, <author> and< / author> For a pair of adjacent labels, they are divided into segments during partitioning. <author> 、< / author> Two data blocks, JK.Rowing is <author> and< / author> The text content between them is also divided into a data block during the partitioning process.

[0052] S120: Data blocks are sent to the processor in the order they are divided, forming a data stream.

[0053] See Figure 2 From part 1: <bookstore> The data is initially sent to the processor sequentially, forming a data stream, until part 13:< / bookstore> Finish.

[0054] S130 performs streaming processing on the data stream.

[0055] In other words, the processor performs streaming processing on the data stream.

[0056] For example, the processor can process the data blocks in the data stream sequentially as follows:

[0057] If the current data block is the start label and the current data block is the first data block, then generate the simulated path of the current data block and its corresponding nodes based on the root path and the current data block;

[0058] If the current data block is the start label and the current data block is not the first data block, then the simulation path of the current data block and its corresponding nodes are generated based on the simulation path of the previous data block and the current data block. Furthermore, if the current data block contains a label attribute, then the label attribute is saved to the node of the current data block.

[0059] If the current data block is text content, then the current data block is saved to the node of the previous data block;

[0060] If the current data block is the end label and the current data block is not the last data block, then the simulated path of the previous data block of the data block to which the corresponding start label belongs is determined as the simulated path of the current data block.

[0061] If the current data block is the end label and is the last data block, then the simulated path of the current data block is determined to be the root path.

[0062] In this way, the processor can process data blocks continuously based on the data stream, quickly determine the simulation path of each data block, and save the corresponding data to the corresponding node for subsequent statistics and queries.

[0063] The following section, in conjunction with Table 1, details the process... Figure 2 The streaming processing of the data stream formed by sending the data blocks shown is as follows:

[0064] Table 1

[0065]

[0066]

[0067] As shown in Table 1, initially, the processor has not yet received the data block, and the simulated path is the root path ".".

[0068] Step 1: The processor receives and processes data blocks. <bookstore>When it is the start tag and the first data block, it is directly determined by the root path "." and the data block. <bookstore>Generate data blocks <bookstore>The simulated path .bookstore and the corresponding node 1. Here, .bookstore is used as the name of node 1.

[0069] Step 2: The processor receives and processes data blocks. <book>When it is the start tag and not the first data block, it is determined according to the previous data block. <bookstore>Simulated path .bookstore and data blocks <book>Generate data blocks <book>The simulated path is .bookstore.book and the corresponding node 2. Here, .bookstore.book is the name of node 2.

[0070] Step 3: The processor receives and processes data blocks. <title lang＝"en"> At that time, since it is the start tag and not the first data block, and it contains tag attributes, it is determined according to the previous data block.<book> The simulated path .bookstore.book and data blocks<title lang="en"> The processor generates the simulated path of the data block `<title lang="en">` as `.bookstore.book.title` and its corresponding node 3, and saves the tag attribute `lang` to node 3. Here, `.bookstore.book.title` serves as the name of node 3.

[0071] Step 4: When the processor receives and processes the data block `Harry Potter`, since it is text content, it saves the data block `Harry Potter` to the node of the previous data block, i.e., node 3.

[0072] Step 5: The processor receives and processes the simulated path of the previous data block of data block ``, i.e., the data block `<book>`, as its simulated path. `.bookstore.book`. <b class="d_n">

[0073] Step 6, when the processor receives and processes the data block <author>, since it is the start tag and not the first data block, it is based on the previous data block< / title> When this is the end tag and not the last data block, the data block to which the corresponding start tag belongs is determined. <title lang＝"en"> . The previous data block is also the data block< / title> The simulated path .bookstore.book and data blocks <author>Generate data blocks <author>The simulated path is .bookstore.book.author and the corresponding node 4. .bookstore.book.author is used as the name of node 4.

[0074] Step 7: When the processor receives and processes the data block J K.Rowing, since it is text content, it saves the data block J K.Rowing to the node of the previous data block, namely node 4.

[0075] Step 8: The processor receives and processes data blocks.< / author> When this is the end tag and not the last data block, the data block to which the corresponding start tag belongs is determined. <author>The previous data block, i.e., the simulated path of the data block, is .bookstore.book.

[0076] Step 9: The processor receives and processes data blocks. <year> Figure 3< / year> < / author> The simulated path .bookstore.book and data blocks <year>Generate data blocks <year>The simulated path is .bookstore.book.yea and the corresponding node 5. Here, .bookstore.book.yea is the name of node 5.

[0077] Step 10: When the processor receives and processes data block 2005, since it is text content, it saves data block 2005 to the node of the previous data block, namely node 5.

[0078] Step 11: The processor receives and processes the data block.< / year> When this is the end tag and not the last data block, the data block to which the corresponding start tag belongs is determined. <year> Figure 4< / year> < / year> < / author> The simulated path is .bookstore.book.

[0079] Step 12: The processor receives and processes the data block.< / book> When this is the end tag and not the last data block, the data block to which the corresponding start tag belongs is determined. <book>The previous data block, i.e., the data block <bookstore>The simulated path is .bookstore.

[0080] Step 13: The processor receives and processes the data block.< / bookstore> When this happens, since it is the end tag and the current data block is the last data block, the data block is determined.< / book> < / book> < / bookstore> The simulation path is the root path.

[0081] To facilitate a more convenient and intuitive analysis of the simulated path changes and node generation process, a model was constructed as follows: Figure 4 The diagram shows the path changes and node generation.

[0082] According to embodiments of this disclosure, large-capacity tree files can be divided into blocks and sent to the processor in the form of data streams, enabling the processor to continuously stream small batches of data, thereby reducing processor load while improving the processing efficiency of large-capacity tree files.

[0083] In some embodiments, the file size and timeliness level of the tree file to be processed can be obtained, and it can be determined whether the tree file meets the preset processing conditions based on the file size and timeliness level. If the preset processing conditions are met, that is, the tree file can be processed by the process processing method 100, then the process processing method 100 is executed on the tree file.

[0084] For example, it can be determined whether the file size of the tree-shaped file is greater than a preset threshold and whether the timeliness level is higher than a preset threshold. If the file size is greater than the preset threshold and the timeliness level is higher than the preset threshold, then process method 100 is executed on the tree-shaped file.

[0085] In this way, large-capacity, high-time-sensitivity tree files can be prioritized for timely streaming processing, further improving the processing efficiency of large-capacity tree files.

[0086] In some embodiments, the processor can save the data in each node to memory during the streaming processing of the data stream, and then extract the data corresponding to each node from memory when performing data statistics, and generate a statistical file (e.g., a CSV file) for easy viewing and editing.

[0087] In some embodiments, since there is a one-to-one correspondence between the simulation path and the node, the corresponding node can be queried and the data in the node can be obtained based on the input simulation path.

[0088] At the same time, you can also query the corresponding simulated path based on the input node data. For example, entering HarryPotter will show that the simulated path is .bookstore.book.title.

[0089] In this way, node data or simulated paths can be quickly queried based on simulated paths or node data, making it convenient for users.

[0090] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0091] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0092] Figure 4 A structural diagram of a streaming apparatus for tree-structured files provided according to embodiments of the present disclosure is shown, such as... Figure 5 As shown, the streaming processing apparatus 400 may include:

[0093] The partitioning module 410 is used to divide the start label and its corresponding end label in the tree file to be processed, as well as the text content between adjacent start labels and end labels, into data blocks.

[0094] The sending module 420 is used to send data blocks to the processor in the order of the divided data blocks to form a data stream.

[0095] Processing module 430 is used to perform streaming processing on the data stream.

[0096] In some embodiments, the streaming processing apparatus 400 further includes:

[0097] The determination module is used to perform streaming processing on the tree file if the tree file meets the preset processing conditions based on its file size and timeliness level.

[0098] In some embodiments, the processing module 430 is specifically used for:

[0099] The data blocks in the data stream are processed sequentially as follows:

[0100] If the current data block is the start label and the current data block is the first data block, then generate the simulated path of the current data block and its corresponding nodes based on the root path and the current data block;

[0101] If the current data block is the start label and the current data block is not the first data block, then the simulation path of the current data block and its corresponding nodes are generated based on the simulation path of the previous data block and the current data block.

[0102] If the current data block is text content, then the current data block is saved to the node of the previous data block;

[0103] If the current data block is the end label and the current data block is not the last data block, then the simulated path of the previous data block of the data block to which the corresponding start label belongs is determined as the simulated path of the current data block.

[0104] If the current data block is the end label and is the last data block, then the simulated path of the current data block is determined to be the root path.

[0105] In some embodiments, the streaming processing apparatus 400 further includes:

[0106] The save module is used to save the tag attributes to the node of the current data block if the current data block is the start tag and the current data block is not the first data block.

[0107] In some embodiments, the storage module is also used to save the data in each node to memory.

[0108] The stream processing unit 400 also includes:

[0109] The extraction module is used to extract the data corresponding to each node from memory and generate a statistical file when performing data statistics.

[0110] In some embodiments, the streaming processing apparatus 400 further includes:

[0111] The query module is used to query the corresponding nodes based on the input simulated path and retrieve the data in the nodes.

[0112] In some embodiments, the tree file is an XML file, a JSON file, an HTML file, or a YAML file.

[0113] Understandable, Figure 5 Each module / unit in the streaming processing apparatus 400 shown has the function of implementing each step in the streaming processing method 100 provided in the embodiments of this disclosure, and can achieve its corresponding technical effect. For the sake of brevity, it will not be described in detail here.

[0114] ​ A structural diagram of an electronic device that can be used to implement embodiments of the present disclosure is shown. Electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 500 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0115] like ​ As shown, the electronic device 500 may include a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0116] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0117] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer program product, including a computer program tangibly contained in a computer-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0118] The various embodiments described above can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), payload programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] It should be noted that this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute method 100 and achieve the corresponding technical effects achieved by executing the method in the embodiments of this disclosure. For the sake of brevity, they will not be described in detail here.

[0122] In addition, this disclosure also provides a computer program product including a computer program that implements method 100 when executed by a processor.

[0123] To provide interaction with a user, the embodiments described above can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0124] The embodiments described above can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0125] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0126] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / book> < / bookstore> < / bookstore> < / bookstore>

Claims

1. A method for streaming processing of tree-structured files, characterized in that, The method includes: Divide the start label and its corresponding end label in the tree file to be processed, as well as the text content between adjacent start and end labels, into data blocks; The data blocks are sent to the processor in the order they are divided, forming a data stream; Perform streaming processing on the data stream; The streaming processing of the data stream includes: The data blocks in the data stream are processed sequentially as follows: If the current data block is the start label and the current data block is the first data block, then generate the simulated path of the current data block and its corresponding nodes based on the root path and the current data block; If the current data block is the start label and the current data block is not the first data block, then the simulation path of the current data block and its corresponding nodes are generated based on the simulation path of the previous data block and the current data block. Furthermore, if the current data block contains a label attribute, then the label attribute is saved to the node of the current data block. If the current data block is text content, then the current data block is saved to the node of the previous data block; If the current data block is the end label and the current data block is not the last data block, then the simulated path of the previous data block of the data block to which the corresponding start label belongs is determined as the simulated path of the current data block. If the current data block is the end label and is the last data block, then the simulated path of the current data block is determined to be the root path.

2. The method according to claim 1, characterized in that, The method further includes: If the tree file meets the preset processing conditions based on its file size and timeliness level, then the tree file is streamed.

3. The method according to claim 1, characterized in that, The method further includes: Save the data in each node to memory; In the case of data statistics, the data corresponding to each node is extracted from the memory and a statistics file is generated.

4. The method according to claim 1, characterized in that, The method further includes: Based on the input simulation path, query the corresponding node and retrieve the data in the node.

5. The method according to any one of claims 1-4, characterized in that, The tree file can be an XML file, a JSON file, an HTML file, or a YAML file.

6. A streaming processing device for tree-structured files, characterized in that, The device includes: The partitioning module is used to divide the start label and its corresponding end label in the tree file to be processed, as well as the text content between adjacent start and end labels, into data blocks; The sending module is used to send the data blocks to the processor in the order of the divided data blocks to form a data stream; The processing module is used to perform streaming processing on the data stream; The processing module is specifically used for: The data blocks in the data stream are processed sequentially as follows: If the current data block is the start label and the current data block is the first data block, then generate the simulated path of the current data block and its corresponding nodes based on the root path and the current data block; If the current data block is the start label and the current data block is not the first data block, then the simulation path of the current data block and its corresponding nodes are generated based on the simulation path of the previous data block and the current data block. Furthermore, if the current data block contains a label attribute, then the label attribute is saved to the node of the current data block. If the current data block is text content, then the current data block is saved to the node of the previous data block; If the current data block is the end label and the current data block is not the last data block, then the simulated path of the previous data block of the data block to which the corresponding start label belongs is determined as the simulated path of the current data block. If the current data block is the end label and is the last data block, then the simulated path of the current data block is determined to be the root path.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

Images