File analysis method and device
By dynamically expanding the memory pipeline during file parsing, the problem of insufficient memory during RDB file parsing is solved, the analysis efficiency and stability are improved, and resource waste is avoided.
Patent Information
- Application Number
- CN202411170642.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-06
AI Technical Summary
Existing RDB file parsing methods require the entire file to be loaded into memory, resulting in waste of memory space and low parsing efficiency, especially when there is insufficient memory, it is easy to cause file parsing failure.
By dynamically expanding the memory pipeline according to the size of the target file during file parsing, gradually reading and parsing data, avoiding the problem of insufficient memory and improving parsing efficiency and success rate.
It realizes the effective parsing of files under limited memory space, avoids resource waste and parsing failures, and improves the efficiency and stability of file parsing.
Smart Images

Figure CN119088594A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a file parsing method and device. Background Art
[0002] At present, for the Remote Dictionary Server (Redis) database, the local persistence processing of data mainly depends on the generation of Redis database (RDB) files. The corresponding RDB file is generated based on the data to be stored. During the persistence processing, the analysis of the RDB file can help locate the problem of Redis data storage and optimize the use of the Redis database. The existing RDB file analysis method needs to load the entire RDB file into the memory and then parse it. This requires that the memory space needs to be large enough to meet the complete reading of all RDB files, but too large a memory space often causes a waste of resources and affects the parsing efficiency of the RDB file. Summary of the invention
[0003] In view of this, an embodiment of the present invention provides a file parsing method, which can flexibly expand the memory space size during the file parsing process according to the size of the target file, thereby ensuring the progress of the file parsing process, avoiding file parsing failure due to insufficient memory, and improving the efficiency and success rate of file parsing.
[0004] To achieve the above object, according to a first aspect of an embodiment of the present invention, a file parsing method is provided, comprising:
[0005] According to the size of the memory pipeline, data not exceeding the size of the memory pipeline is read from the target file and stored in the memory pipeline;
[0006] According to the length information of the data, the data is taken out from the memory pipeline in sequence for parsing, wherein each data to be taken out is judged based on the length of the data;
[0007] If it is determined that the data is not entirely in the memory pipeline and the length of the data is greater than the size of the memory pipeline, deleting the parsed data in the memory pipeline and expanding the memory pipeline according to the length of the data;
[0008] The expanded memory pipeline is used as the updated memory pipeline, and the above steps are repeated until all the data in the target file are completely parsed to obtain the parsed data of the target file.
[0009] Optionally, the method further comprises:
[0010] If it is determined that the data is not all located in the memory pipeline and the length of the data is not greater than the size of the memory pipeline, deleting the parsed data in the memory pipeline;
[0011] Data is read from the target file, stored in the memory pipeline, and parsed to obtain parsed data of the target file.
[0012] Optionally, taking out the data from the memory pipeline in sequence for parsing according to the length information of the data includes:
[0013] Read the data from the memory pipeline in sequence according to the length information of the data;
[0014] The data is parsed according to a preset encoding method to obtain corresponding parsed data.
[0015] Optionally, the parsed data of the target file includes a target indicator and corresponding data, and the method further includes:
[0016] Statistical analysis is performed based on the target indicators and corresponding data to obtain target parameter data corresponding to the target indicators.
[0017] Optionally, before reading data not exceeding the size of the memory pipe from the target file and storing the data in the memory pipe according to the size of the memory pipe, the method further includes:
[0018] Verify the acquired target file and obtain the verification result;
[0019] When the verification result meets the preset condition, the memory pipeline size is initialized to obtain the memory pipeline.
[0020] Optionally, the expanding the memory pipeline according to the length of the data includes:
[0021] Determining a target memory size according to the length of the data and a preset buffer length;
[0022] The memory pipeline is expanded to the target memory size.
[0023] According to a second aspect of an embodiment of the present invention, there is provided a file parsing device, comprising:
[0024] A reading module, used for reading data not exceeding the size of the memory pipeline from a target file according to the size of the memory pipeline and storing the data in the memory pipeline;
[0025] A parsing module, used for taking out the data from the memory pipeline in sequence for parsing according to the length information of the data, wherein each data to be taken out is judged based on the length of the data;
[0026] an expansion module, configured to, when it is determined that the data is not entirely in the memory pipeline and the length of the data is greater than the size of the memory pipeline, delete the parsed data in the memory pipeline and expand the memory pipeline according to the length of the data;
[0027] The parsing module is used to use the expanded memory pipeline as the updated memory pipeline, and repeatedly execute the above steps until all data in the target file are completely parsed to obtain the parsed data of the target file.
[0028] Optionally, the device further comprises:
[0029] a deleting module, configured to delete the parsed data in the memory pipeline if it is determined that the data is not entirely in the memory pipeline and the length of the data is not greater than the size of the memory pipeline;
[0030] The parsing module is used to read and parse the unread data in the target file based on the memory pipeline to obtain the parsed data of the target file.
[0031] Optionally, the parsing module is used to sequentially fetch the data from the memory pipeline for parsing according to the length information of the data, including:
[0032] The reading module is used to read the data from the memory pipe in sequence according to the length information of the data;
[0033] The parsing module is used to parse the data according to a preset encoding method to obtain corresponding parsed data.
[0034] Optionally, the parsed data of the target file includes a target indicator and corresponding data, and the device further includes:
[0035] The analysis module is used to perform statistical analysis based on the target indicator and the corresponding data to obtain the target parameter data corresponding to the target indicator.
[0036] Optionally, the device further comprises:
[0037] A verification module is used to verify the acquired target file and obtain a verification result;
[0038] An initialization module is used to initialize the memory pipeline size and obtain the memory pipeline when the verification result meets a preset condition.
[0039] Optionally, the expansion module is used to expand the memory pipeline according to the length of the data, including:
[0040] A determination module, used to determine a target memory size according to the length of the data and a preset buffer length;
[0041] The expansion module is used to expand the memory pipeline to the target memory size.
[0042] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:
[0043] One or more processors; a storage device for storing one or more programs,
[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiment of the present invention.
[0045] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.
[0046] An embodiment of the above invention has the following advantages or beneficial effects: after obtaining the target file, a judgment is made based on the length of the data in the target file, thereby realizing dynamic expansion of the memory pipeline to ensure the parsing process of the data in the target file and improve the parsing efficiency and stability.
[0047] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a schematic diagram of the main process of the file parsing method according to an embodiment of the present invention;
[0049] Figure 2 It is a schematic diagram of the process of file parsing provided by an embodiment of the present invention;
[0050] Figure 3 is a schematic diagram of main modules of a file parsing device according to an embodiment of the present invention;
[0051] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0052] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0054] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.
[0055] Figure 1 1 is a schematic diagram of the main flow of the file parsing method according to an embodiment of the present invention, including:
[0056] Step S101, according to the size of the memory pipe, read data not exceeding the size of the memory pipe from the target file and store it in the memory pipe.
[0057] After obtaining the target file in RDB format that needs to be parsed, read the data in the target file into the memory pipe for subsequent parsing. When reading the data in the target file, the maximum amount of data read shall not exceed the size T of the memory pipe, that is, if the size of all data in the target file is not larger than the size of the memory pipe, all the data in the target file will be read at once; if the size of all data in the target file is larger than the size of the memory pipe, only part of the data in the target file will be read, and the size of this part of the data is equal to the size T of the memory pipe. It should be noted here that when reading data from the target file and storing it in the memory pipe, if there is data in the memory pipe, the amount of data read at this time should not exceed the difference between the size of the memory pipe and the amount of data already in the memory pipe, so as to ensure that the data read from the target file can be stored in the memory pipe.
[0058] Step S102, according to the length information of the data, the data is taken out from the memory pipeline in sequence for parsing, wherein each data to be taken out is judged based on the length of the data.
[0059] When parsing data in the memory pipeline, it is necessary to read the character string of each data in turn according to the length information of each data for data parsing. Among them, each data is stored in the form of a key-value data pair. Before taking out the data from the memory pipeline and parsing it, it is first necessary to judge based on the data length whether the data to be taken out is all in the memory pipeline. Reading and parsing will only be performed on the data that is all in the memory pipeline. Specifically, in order to accurately judge whether the data to be taken out is all in the memory pipeline, the present invention sets a start pointer start and an end pointer end for the memory pipeline. When data is not initially taken out from the memory pipeline, the values of start and end are both 0. Each time a data is taken out from the memory pipeline, end=end+M and start=end+1 will be updated according to the length byte number M of the taken data. When judging whether all the data are in the memory pipe, the data length M, the pointer position end at the end of the data and the position T-1 at the end of the memory pipe can be used to judge whether all the data are in the memory pipe. For example, the start pointer of the data is start, the number of bytes of the read data length is M, and the end pointer of the data is set to end=end+M. Judging whether all the data are in the memory pipe can be achieved by judging whether end is greater than T-1. When end≤T-1, the data stored in the memory pipe has not overflowed, and all the data are in the memory pipe. The content of the data can be obtained by reading the [start, end] bytes for parsing; when end>T-1, part of the data content is not in the memory pipe.
[0060] Optionally, the length information of the data is, for example, information stored in a position before the key-value data pair, and is used to indicate the length of the read data.
[0061] Step S103, when it is determined that the data is not entirely in the memory pipeline and the length of the data is greater than the size of the memory pipeline, the parsed data in the memory pipeline is deleted, and the memory pipeline is expanded according to the length of the data.
[0062] When it is determined that the data is not completely in the memory pipeline and the length of the data is greater than the size of the memory pipeline, that is, end>T-1, and M>T, keep the data in the memory pipeline, delete only the data that has been read and parsed in the memory pipeline, and expand the memory pipeline. When expanding, it is necessary to ensure that the expanded memory pipeline can store the data completely. After expansion, star=0, end=end-start.
[0063] Optionally, the size of the expanded memory pipeline may be a multiple of the size of the original memory pipeline, and the data may be completely placed therein.
[0064] Step S104, using the expanded memory pipe as the updated memory pipe, and repeating the above steps until all data in the target file are completely parsed, thereby obtaining the parsed data of the target file.
[0065] The size of the memory pipeline is updated according to the size of the expansion, and the data reading, data length judgment, and data parsing operations are repeated based on the updated memory pipeline size until all the data in the target file have completed the reading and parsing operations, and the complete parsed data of the target file is obtained. Among them, when the operation is performed based on the updated memory pipeline, the memory pipeline still stores part of the data that is not completely located in the memory pipeline, that is, retains part of the data of a data. When performing subsequent data reading operations in the target file, it is necessary to continue reading this part of the data. After reading all the data of the data, the next data reading operation can be performed.
[0066] Optionally, if there is still unread data remaining in the target file after reading the data in the target file based on the updated memory pipe, the data length is continued to determine whether capacity expansion is required, and the data reading and parsing operations are repeated again.
[0067] According to the file parsing method provided by the embodiment of the present invention, the data of the target file can be read according to the size of the memory pipe, and judgment can be made based on the data length. When the data length is greater than the memory pipe size, the memory pipe size can be flexibly expanded, and data parsing can be performed based on the expanded memory pipe size, thereby realizing dynamic expansion of the memory pipe to ensure the reading and parsing of the target file data, thereby improving the efficiency and stability of file parsing.
[0068] Optionally, the method further includes: deleting the parsed data in the memory pipe when it is determined that the data is not entirely located in the memory pipe and the length of the data is not greater than the size of the memory pipe; reading data from the target file, storing it in the memory pipe and parsing it to obtain the parsed data of the target file.
[0069] When it is determined that the data is not entirely in the memory pipeline and the length of the data is not greater than the size of the memory pipeline, that is, end>T-1, and M≤T, the size of the current memory pipeline can fully store the complete data of the data. Therefore, there is no need to expand the capacity, and only the parsed data in the memory pipeline needs to be deleted. At this time, start=0, end=end-start. And continue to read the data in the target file and store it in the memory pipeline for parsing to obtain the corresponding parsed data. Among them, the deletion operation of the parsed data can be achieved by pushing the parsed data out of the memory pipeline.
[0070] Optionally, according to the length information of the data, the data is sequentially taken out from the memory pipe for parsing, including: according to the length information of the data, the data is sequentially read from the memory pipe; according to a preset encoding method, the data is parsed to obtain corresponding parsed data. When parsing the data, the data length information stored in the front end of the data is first obtained, and according to the data length information, the complete data of the data is read from the memory pipe. The obtained data is parsed by ASCII code to obtain parsed data.
[0071] Optionally, before the data is parsed, a preset number of data may be acquired, and then the preset number of data may be parsed.
[0072] Optionally, the parsed data of the target file includes a target indicator and corresponding data, and the method further includes: performing statistical analysis based on the target indicator and the corresponding data to obtain target parameter data corresponding to the target indicator.
[0073] After obtaining the parsed data of the target file, further statistical analysis operations can be performed to analyze the parsed data and determine the problematic data contained therein. Specifically, the parsed data contains preset target indicators and corresponding data, such as the number of keys in the file, the size of the value content corresponding to the key, the number of elements in the set corresponding to the set type, etc. Statistical analysis of the above data includes, but is not limited to: using a max-heap to obtain the key data of the top N values; when the value is a set type, obtaining the number of values.
[0074] Optionally, based on the size of the memory pipe, before reading data not exceeding the size of the memory pipe from the target file and storing it in the memory pipe, the method further includes: verifying the acquired target file to obtain a verification result; when the verification result meets a preset condition, initializing the memory pipe size to obtain the memory pipe.
[0075] Before parsing the target file data, the target file needs to be verified, including the verification of the file size and content. Specifically, when downloading the target file, a task is started to determine whether the size of the target file has changed, that is, to determine whether the target file has been downloaded. When the size of the target file no longer changes within a period of time, it is determined that the file has been downloaded. The specific value of a period of time can be set according to actual conditions and is not limited. After the download is completed, an initialization operation is performed, including initializing the size of the memory pipe to T, and initializing the start and end pointers to 0 to indicate the beginning and end positions of each data.
[0076] Optionally, before obtaining the target file, you first need to install the Redis application and download the RDB file through the application. Based on this, the target file is downloaded through thread A, and the target file is parsed through thread B. The two threads can be executed concurrently to improve the processing efficiency of the RDB file. Among them, thread B can perform statistical analysis operations.
[0077] Optionally, the memory pipeline is expanded according to the length of the data, including: determining a target memory size according to the length of the data and a preset buffer length; and expanding the memory pipeline to the target memory size.
[0078] When expanding the memory pipeline, the size of the expansion can be determined according to the data length. Specifically, the size of the memory pipeline after expansion can be set equal to the sum of the data length and the preset buffer length. The preset buffer length can be set according to actual conditions and is not limited to ensure the normal data reading operation.
[0079] Optionally, the target memory size for expansion may also be a preset fixed value size.
[0080] According to the file parsing method provided by the embodiment of the present invention, the data of the target file can be read according to the size of the memory pipeline, and when the memory pipeline is small, the size of the memory pipeline can be flexibly expanded based on the data length, and the data can be parsed based on the size of the expanded memory pipeline, so as to realize the dynamic expansion of the memory pipeline, so as to ensure the reading and parsing of the target file data and improve the stability during parsing. Among them, the expansion of the memory pipeline based on the data length can make the expanded pipeline more adaptable to the size of the data in the target file, avoid the waste of resources and the reduction of efficiency caused by the excessive expansion of the memory pipeline, and at the same time, the efficiency of file parsing can be effectively improved by downloading files and parsing the data in the files through concurrent threads.
[0081] like Figure 2 The figure shows a flowchart of file parsing. After the process starts, the pipeline size is initialized first, the RDB file is read into the pipeline, and then the data size is read, and it is determined whether the data is in the pipeline. When the data is in the pipeline, the data is parsed to determine whether the file has been parsed. If the parsing is completed, the process ends, otherwise, the RDB file data is continued to be read into the pipeline. When the data is not all in the pipeline, it is determined whether the pipeline is large enough to accommodate the unparsed data, that is, whether the data length exceeds the pipeline length. If so, the parsed data in the pipeline is pushed out of the pipeline, the pipeline is expanded, and the unparsed data is continued to be read into the pipeline based on the expanded pipeline, and the operation of determining whether the data is all in the pipeline is repeated; if not, the parsed data in the pipeline is pushed out of the pipeline, and the unparsed data is continued to be read into the pipeline, and the operation of determining whether the data is in the pipeline is repeated.
[0082] Figure 3 3 is a schematic diagram of the main process of the file parsing device 300 according to an embodiment of the present invention. Figure 3 As shown, including:
[0083] A reading module 301 is used to read data not exceeding the size of the memory pipeline from a target file according to the size of the memory pipeline and store the data into the memory pipeline;
[0084] The parsing module 302 is used to sequentially fetch the data from the memory pipeline for parsing according to the length information of the data, wherein each data to be fetched is judged based on the length of the data;
[0085] The expansion module 303 is used to delete the parsed data in the memory pipeline and expand the memory pipeline according to the length of the data when it is determined that the data is not all in the memory pipeline and the length of the data is greater than the size of the memory pipeline;
[0086] The parsing module 302 is used to use the expanded memory pipe as the updated memory pipe, and repeatedly execute the above steps until all data in the target file are completely parsed, thereby obtaining the parsed data of the target file.
[0087] The file parsing device provided according to the embodiment of the present invention can read the data of the target file according to the size of the memory pipe, and make a judgment based on the data length. When the data length is greater than the memory pipe size, the memory pipe size is flexibly expanded, and data parsing is performed based on the expanded memory pipe size, thereby realizing dynamic expansion of the memory pipe to ensure the reading and parsing of the target file data, thereby improving the efficiency and stability of file parsing.
[0088] Optionally, the device further comprises:
[0089] A deleting module 304, configured to delete the parsed data in the memory pipeline if it is determined that the data is not entirely in the memory pipeline and the length of the data is not greater than the size of the memory pipeline;
[0090] The parsing module 302 is used to read and parse the unread data in the target file based on the memory pipeline to obtain the parsed data of the target file.
[0091] Optionally, the parsing module 302 is used to sequentially fetch the data from the memory pipeline for parsing according to the length information of the data, including:
[0092] The reading module 301 is used to read the data from the memory pipe in sequence according to the length information of the data;
[0093] The parsing module 302 is used to parse the data according to a preset encoding method to obtain corresponding parsed data.
[0094] Optionally, the parsed data of the target file includes a target indicator and corresponding data, and the device further includes:
[0095] The analysis module 305 is used to perform statistical analysis based on the target indicator and the corresponding data to obtain the target parameter data corresponding to the target indicator.
[0096] Optionally, the device further comprises:
[0097] Verification module 306, used to verify the acquired target file and obtain a verification result;
[0098] The initialization module 307 is used to initialize the memory pipeline size to obtain the memory pipeline when the verification result meets the preset condition.
[0099] Optionally, the expansion module 303 is used to expand the memory pipeline according to the length of the data, including:
[0100] A determination module 308, configured to determine a target memory size according to the length of the data and a preset buffer length;
[0101] The expansion module 303 is used to expand the memory pipeline to the target memory size.
[0102] According to the file parsing method provided by the embodiment of the present invention, the data of the target file can be read according to the size of the memory pipeline, and when the memory pipeline is small, the size of the memory pipeline can be flexibly expanded based on the data length, and the data can be parsed based on the size of the expanded memory pipeline, so as to realize the dynamic expansion of the memory pipeline, so as to ensure the reading and parsing of the target file data and improve the stability during parsing. Among them, the expansion of the memory pipeline based on the data length can make the expanded pipeline more adaptable to the size of the data in the target file, avoid the waste of resources and the reduction of efficiency caused by the excessive expansion of the memory pipeline, and at the same time, the efficiency of file parsing can be effectively improved by downloading files and parsing the data in the files through concurrent threads.
[0103] Figure 4 An exemplary system architecture 400 is shown to which the file parsing method or the file parsing device according to the embodiment of the present invention can be applied.
[0104] like Figure 4As shown, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0105] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).
[0106] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0107] The server 405 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 401, 402, and 403. The backend management server may analyze and process the received parsing request and other data, and feed back the processing results (such as target files, parsed data - only an example) to the terminal device.
[0108] It should be noted that the file parsing method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the file parsing device is generally disposed in the server 405 .
[0109] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0110] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0111] like Figure 5As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0112] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.
[0113] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.
[0114] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0116] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, they may be described as: a processor includes a reading module, a parsing module and a capacity expansion module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the reading module may also be described as "a module that reads data not exceeding the size of the memory pipe from a target file and stores it in the memory pipe according to the size of the memory pipe".
[0117] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:
[0118] According to the size of the memory pipeline, data not exceeding the size of the memory pipeline is read from the target file and stored in the memory pipeline;
[0119] According to the length information of the data, the data is taken out from the memory pipeline in sequence for parsing, wherein each data to be taken out is judged based on the length of the data;
[0120] If it is determined that the data is not entirely in the memory pipeline and the length of the data is greater than the size of the memory pipeline, deleting the parsed data in the memory pipeline and expanding the memory pipeline according to the length of the data;
[0121] The expanded memory pipeline is used as the updated memory pipeline, and the above steps are repeated until all the data in the target file are completely parsed to obtain the parsed data of the target file.
[0122] According to the technical solution of the embodiment of the present invention, the memory space size can be flexibly expanded during the file parsing process according to the size of the target file, thereby ensuring the progress of the file parsing process, avoiding file parsing failure due to insufficient memory, and improving the efficiency and success rate of file parsing.
[0123] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A file parsing method, characterized in that: include: According to the size of the memory pipeline, data not exceeding the size of the memory pipeline is read from the target file and stored in the memory pipeline; According to the length information of the data, the data is taken out from the memory pipeline in sequence for parsing, wherein each data to be taken out is judged based on the length of the data; If it is determined that the data is not entirely in the memory pipeline and the length of the data is greater than the size of the memory pipeline, deleting the parsed data in the memory pipeline and expanding the memory pipeline according to the length of the data; The expanded memory pipeline is used as the updated memory pipeline, and the above steps are repeated until all the data in the target file are completely parsed to obtain the parsed data of the target file.
2. The method according to claim 1, characterized in that: The method further comprises: If it is determined that the data is not all located in the memory pipeline and the length of the data is not greater than the size of the memory pipeline, deleting the parsed data in the memory pipeline; Data is read from the target file, stored in the memory pipeline, and parsed to obtain parsed data of the target file.
3. The method according to claim 1 or 2, characterized in that: The step of taking out the data from the memory pipeline in sequence for parsing according to the length information of the data includes: Read the data from the memory pipeline in sequence according to the length information of the data; The data is parsed according to a preset encoding method to obtain corresponding parsed data.
4. The method according to claim 3, characterized in that The parsed data of the target file includes target indicators and corresponding data, and the method further includes: Statistical analysis is performed based on the target indicators and corresponding data to obtain target parameter data corresponding to the target indicators.
5. The method according to claim 1, characterized in that Before reading data not exceeding the size of the memory pipeline from the target file according to the size of the memory pipeline and storing the data in the memory pipeline, the method further comprises: Verify the acquired target file and obtain the verification result; When the verification result meets the preset condition, the memory pipeline size is initialized to obtain the memory pipeline.
6. The method according to claim 5, characterized in that The expanding the memory pipeline according to the length of the data includes: Determining a target memory size according to the length of the data and a preset buffer length; The memory pipeline is expanded to the target memory size.
7. A file parsing device, characterized in that: include: A reading module, used for reading data not exceeding the size of the memory pipeline from a target file according to the size of the memory pipeline and storing the data in the memory pipeline; A parsing module, used for taking out the data from the memory pipeline in sequence for parsing according to the length information of the data, wherein each data to be taken out is judged based on the length of the data; an expansion module, configured to, when it is determined that the data is not entirely in the memory pipeline and the length of the data is greater than the size of the memory pipeline, delete the parsed data in the memory pipeline and expand the memory pipeline according to the length of the data; The parsing module is used to use the expanded memory pipeline as the updated memory pipeline, and repeatedly execute the above steps until all data in the target file are completely parsed to obtain the parsed data of the target file.
8. The device according to claim 7, characterized in that The device also includes: a deleting module, configured to delete the parsed data in the memory pipeline if it is determined that the data is not entirely in the memory pipeline and the length of the data is not greater than the size of the memory pipeline; The parsing module is used to read and parse the unread data in the target file based on the memory pipeline to obtain the parsed data of the target file.
9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.
10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
