A CSV large file fast retrieval method, device, equipment and storage medium

Abstract

The application discloses a CSV large file fast retrieval method and device, equipment and storage medium, and relates to the technical field of computers, which comprises the following steps: determining a first address, a second address and an end address from a CSV file stream based on a retrieval data request; wherein the first address is a title address of the CSV file stream, the second address is a starting address of the first data in the CSV file stream in the memory, and the end address is an end address of the last data in the CSV file stream in the memory; taking the number of bytes between the first address and the second address as a target step; determining a center address of the CSV file stream according to the second address and the end address; and starting from the center address, traversing the CSV file stream between the to-be-traversed addresses in the forward direction of the center address and the backward direction of the center address by using the target step, until the retrieval data request is met. The retrieval algorithm is optimized, and the effect of fast retrieval and low consumption in the CSV large file is achieved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for fast retrieval of large CSV files. Background Technology

[0002] Currently, stock trading often involves a large amount of archived data, such as market data, transaction records, and transaction history. This type of data is stored on the hard drive as CSV files. If a specific piece of data needs to be retrieved, all the data must be loaded first, and then all the data must be traversed and retrieved in memory, which consumes a lot of computing resources and time. Alternatively, searching directly by scanning the files on the disk is also very disk-intensive and slow.

[0003] In summary, optimizing retrieval algorithms to achieve fast and low-cost results, and quickly retrieving target data from large CSV files, is a technical problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for fast retrieval of large CSV files, which can achieve fast and low-power results through optimization of the retrieval algorithm, enabling rapid retrieval of target data in large CSV files. The specific solution is as follows:

[0005] Firstly, this application discloses a fast retrieval method for large CSV files, including:

[0006] Based on the data retrieval request, a first address, a second address, and an end address are determined from the CSV file stream; wherein, the first address is the header address of the CSV file stream, the second address is the starting address of the first data in the CSV file stream in memory, and the end address is the ending address of the last data in the CSV file stream in memory;

[0007] Use the number of bytes between the first address and the second address as the target step size;

[0008] The center address of the CSV file stream is determined based on the second address and the end address;

[0009] Starting from the center address, the CSV file stream between the addresses to be traversed is traversed in the direction before and after the center address using the target step size, until the data retrieval request is satisfied.

[0010] Optionally, before determining the first address from the CSV file stream based on the data retrieval request, the method further includes:

[0011] Obtain the CSV file stream and the retrieval data request sent by the user, which includes the index field and the identifier of the target data content in the index field.

[0012] Optionally, obtaining the CSV file stream includes:

[0013] Sort all fields of the CSV file to obtain a sorted CSV file;

[0014] The CSV file is converted to obtain a CSV file stream.

[0015] Optionally, the process of traversing the CSV file stream between addresses to be traversed, starting from the center address and using the target step size, in both the forward and backward directions from the center address, further includes:

[0016] Based on the target step size and according to the center address in an address offset manner, determine the third address located in the direction before the center address and the fourth address located in the direction after the center address; wherein, the third address and the fourth address are addresses to be traversed.

[0017] Optionally, determining the third address located before the center address and the fourth address located after the center address based on the target step size and according to the center address in an address offset manner includes:

[0018] Using the center address as the origin, offset by a target step size in the direction before and after the center address respectively, traverse the corresponding CSV file stream, and determine whether there is a corresponding first target address and second target address in the direction before and after the center address respectively; if they exist, determine the address after the first target address as the third address, and determine the target address as the fourth address.

[0019] Optionally, after determining whether there are corresponding first target addresses and second target addresses in the directions before and after the center address, the method further includes:

[0020] If it does not exist, the operation of determining a new center address is triggered, and the process jumps to execute the steps of offsetting by a target step size in the direction before and after the center address, respectively.

[0021] Optionally, the step of traversing the CSV file stream between the addresses to be traversed using the target step size in both the direction before and after the center address until the data retrieval request is satisfied includes:

[0022] The data between the third address and the fourth address is used as the target data, and the target data is retrieved from memory and converted into a target string;

[0023] Compare the target index field with the index field in the target string, and the target content identifier with the identifier in the target string;

[0024] If the comparison is consistent, the search ends and the target string is returned to the human-computer interaction interface.

[0025] Secondly, this application discloses a fast CSV large file retrieval device, comprising:

[0026] The address determination module is used to determine a first address, a second address, and an end address from a CSV file stream based on a retrieved data request; wherein, the first address is the header address of the CSV file stream, the second address is the starting address of the first data in the CSV file stream in memory, and the end address is the ending address of the last data in the CSV file stream in memory;

[0027] The step size determination module is used to take the number of bytes between the first address and the second address as the target step size;

[0028] A center address determination module is used to determine the center address of the CSV file stream based on the second address and the end address;

[0029] The data retrieval module is used to traverse the CSV file stream between the addresses to be traversed, starting from the center address and using the target step size, in the direction before and after the center address, until the data retrieval request is satisfied.

[0030] Thirdly, this application discloses an electronic device, including:

[0031] Memory, used to store computer programs;

[0032] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed method for fast retrieval of large CSV files.

[0033] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed fast CSV large file retrieval method.

[0034] Therefore, this application discloses a fast retrieval method for large CSV files, comprising: determining a first address, a second address, and an end address from a CSV file stream based on a retrieval data request; wherein the first address is the header address of the CSV file stream, the second address is the starting address of the first data entry in memory, and the end address is the ending address of the last data entry in memory; using the number of bytes between the first address and the second address as a target step size; determining the center address of the CSV file stream based on the second address and the end address; and traversing the CSV file stream between the addresses to be traversed, starting from the center address and using the target step size, in both directions before and after the center address, until the retrieval data request is satisfied. Therefore, by determining the first address, second address, and end address from the ordered CSV file stream, and then determining the target step size based on the number of bytes between the first and second addresses, this target step size serves as a standard for the number of bytes per row. After determining the center position of the CSV file stream, this target step size is used to determine the addresses to be traversed in the direction before and after the center position. This process is repeated to traverse the CSV file stream between the addresses to be traversed until the position of the target data in the data request is determined, at which point the process of determining the addresses to be traversed stops, and the data retrieval ends. This optimized retrieval algorithm enables fast retrieval of target data in large CSV files, achieving both speed and low resource consumption. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 This is a flowchart of a fast CSV file retrieval method disclosed in this application;

[0037] Figure 2 This application discloses a flowchart of a specific method for fast retrieval of large CSV files.

[0038] Figure 3 The flowchart below shows a specific method for fast retrieval of target data in a large CSV file arranged in ascending order, as disclosed in this application.

[0039] Figure 4 This is a schematic diagram of the structure of a fast CSV large file retrieval device disclosed in this application;

[0040] Figure 5This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0042] Currently, stock trading often involves a large amount of archived data, such as market data, transaction records, and transaction history. This type of data is stored on the hard drive as CSV files. If a specific piece of data needs to be retrieved, all the data must be loaded first, and then all the data must be traversed and retrieved in memory, which consumes a lot of computing resources and time. Alternatively, searching directly by scanning the files on the disk is also very disk-intensive and slow.

[0043] To address this, this application provides a fast retrieval scheme for large CSV files, which optimizes the retrieval algorithm to achieve fast and low-cost results, enabling rapid retrieval of target data within large CSV files.

[0044] Reference Figure 1 As shown, this embodiment of the invention discloses a fast retrieval method for large CSV files, including:

[0045] Step S11: Determine the first address, second address, and end address from the CSV file stream based on the retrieved data request; wherein, the first address is the header address of the CSV file stream, the second address is the starting address of the first data in the CSV file stream in memory, and the end address is the ending address of the last data in the CSV file stream in memory.

[0046] In this embodiment, the first address, second address, and end address are determined from the CSV file stream sorted in ascending order based on the data retrieval request. It's understood that the system considers data between two addresses as a complete data entry. The first data entry of the CSV file stream is obtained using this rule. Note that the first row is considered the header by default and is not included in the content calculation; therefore, the first address of the file is the header address (head). Then, the second address is set as the start address (begin), which is the starting address in memory for recording the first data entry. The total size of the CSV file stream is obtained, and the end address in memory for the entire data is calculated and denoted as end.

[0047] In this embodiment, before determining the first address from the CSV file stream based on the retrieval data request, the method further includes: acquiring the CSV file stream and acquiring a retrieval data request sent by the user terminal, which includes an index field and an identifier of the target data content in the index field. It can be understood that before acquiring the first address, the CSV file stream is first acquired, and the retrieval data request sent by the user terminal, which includes an index field and an identifier of the target data content in the index field, is acquired. Specifically, the index field key sent by the user terminal and the content identifier ID of the target data are acquired.

[0048] In this embodiment, obtaining the CSV file stream includes: sorting all fields of the CSV file to obtain a sorted CSV file; and converting the CSV file to obtain a CSV file stream. It is understood that in stock trading scenarios, stock transaction details are large and disorganized, specifically including but not limited to: transaction time, transaction price, and transaction volume. Therefore, sorting the target data in each field of the CSV file is generally performed in ascending order to obtain the arrangement of the content identifier IDs of the target content in each field, forming an ordered CSV file stream.

[0049] Step S12: Use the number of bytes between the first address and the second address as the target step size.

[0050] In this embodiment, after obtaining the first address and the second address, the number of bytes between the second address and the first address is calculated, and this number of bytes is used as the estimated number of bytes step. That is, the data between the header address and the start address is regarded as a complete data line. Therefore, the number of bytes of this complete data line is used as the target step size for each record.

[0051] Step S13: Determine the center address of the CSV file stream based on the second address and the end address.

[0052] In this embodiment, the center address of the CSV file stream is determined based on the second address and the end address. That is, by using the start address and end address of the actual content data, the total number of memory addresses of the complete data of the entire CSV file stream is obtained by subtracting the start address from the end address (end) from the start address (begin). Then, the result is divided by 2, and the target address is the center address of the entire CSV file stream.

[0053] Step S14: Starting from the center address, use the target step size to traverse the CSV file stream between the addresses to be traversed in the direction before and after the center address, until the data retrieval request is satisfied.

[0054] In this embodiment, starting from the center address, the target step size is used to simultaneously traverse in both directions before and after the center address to find the address to be traversed. Specifically, since the data length represented by the pre-set target step size is a complete data line, the address before and after the center address can be determined based on the center address and the target step size. However, in reality, the pre-set target step size may not be consistent with the actual data length of a complete data line; the target step size may be too long or too short. Therefore, the following steps are required to determine the address to be traversed:

[0055] Step 1: Using the center address as the starting point for the addresses to be traversed, offset the address forward and backward within one step.

[0056] Step 2: Determine if there is an address to be traversed within the address range of one step in both the forward and backward directions;

[0057] Step 3: If it does not exist, add another step based on the previous step, perform address offset, and search within that range for the address to be traversed. If it still does not exist, add another step until the address to be traversed is found.

[0058] Step 4: If they exist, locate the previous address to be traversed and the next address to be traversed, and extract the string between the address of the previous address to be traversed + 1 and the next address to be traversed, that is, extract the string between the center address and the next address to be traversed.

[0059] Step 5: Based on the string, further obtain the content identifier corresponding to the index field key, compare the size relationship between the content identifier and the target content identifier ID, and output the target content identifier ID and the target content that match the size comparison, thus completing the retrieval request.

[0060] Therefore, this application discloses a fast retrieval method for large CSV files, comprising: determining a first address, a second address, and an end address from a CSV file stream based on a retrieval data request; wherein the first address is the header address of the CSV file stream, the second address is the starting address of the first data entry in memory, and the end address is the ending address of the last data entry in memory; using the number of bytes between the first address and the second address as a target step size; determining the center address of the CSV file stream based on the second address and the end address; and traversing the CSV file stream between the addresses to be traversed, starting from the center address and using the target step size, in both directions before and after the center address, until the retrieval data request is satisfied. As can be seen, by determining the first address, second address, and end address from the ordered CSV file stream, and then determining the target step size based on the number of bytes between the first and second addresses, this target step size is used as a standard for the number of bytes per row. After determining the center position of the CSV file stream, this target step size is used to determine the addresses to be traversed in the direction before and after the center position. This process is repeated to traverse the CSV file stream between the addresses to be traversed until the position of the target data in the data request is determined in the CSV file stream. At this point, the process of determining the addresses to be traversed stops, and the data retrieval ends. This optimized retrieval algorithm enables fast retrieval of target data in large CSV files, achieving both speed and low resource consumption.

[0061] Reference Figure 2 As shown, this embodiment of the invention discloses a specific method for fast retrieval of large CSV files. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0062] Step S21: Determine the first address, the second address, and the end address from the CSV file stream based on the retrieved data request; wherein, the first address is the header address of the CSV file stream, the second address is the starting address of the first data in the CSV file stream in memory, and the end address is the ending address of the last data in the CSV file stream in memory.

[0063] Step S22: Use the number of bytes between the first address and the second address as the target step size.

[0064] Step S23: Determine the center address of the CSV file stream based on the second address and the end address.

[0065] For more detailed processing procedures in steps S21, S22, and S23, please refer to the aforementioned disclosed embodiments; they will not be repeated here.

[0066] Step S24: Based on the target step size and according to the center address in the address offset manner, determine the third address located in the direction before the center address and the fourth address located in the direction after the center address; wherein, the third address and the fourth address are both addresses to be traversed.

[0067] In this embodiment, taking the center address as the origin, the corresponding CSV file stream is traversed by offsetting by a target step size in the direction before and after the center address, respectively. It is then determined whether a first target address and a second target address exist in the direction before and after the center address. If they exist, the address following the first target address is determined as the third address, and the target address is determined as the fourth address. It can be understood that after calculating the middle address in the file memory using begin and end, the address is offset by step to obtain the two nearest \n characters, i.e., the addresses to be traversed, thereby quickly locating a complete data entry.

[0068] In this embodiment, after determining whether a corresponding first target address and second target address exist in the direction before and after the center address, the method further includes: if they do not exist, then an operation to determine a new center address will be triggered, and the process will jump to execute steps to offset by one target step size in the direction before and after the center address, respectively. It can be understood that if no target address or second target address is found within one step (i.e., if no target address or second target address is found within one step), the offset will be incremented by one step, and the address incremented by one step will be determined as the new center address. The search will continue until the first target address is found.

[0069] Step S25: Using the target step size, traverse the CSV file stream between the addresses to be traversed in the direction before the center address and the direction after the center address, until the data retrieval request is satisfied.

[0070] In this embodiment, the data between the third address and the fourth address is used as the target data, and the target data is retrieved from memory and converted into a target string; the target index field and the target content identifier in the target string are compared; if they match, the retrieval ends, and the target string is returned to the human-computer interaction interface. Understandably, after determining the third and fourth addresses, the data between them is considered a complete line of data to be retrieved. This data is then converted from a file stream to a string, and the corresponding field value is retrieved based on the index field in the retrieval request. The retrieved field value is compared with the target content identifier ID. If the comparison results are the same, the target content is the retrieved content, and it is directly returned to the human-computer interaction interface for the user. If the comparison results are inconsistent and the field value is greater than the target content identifier ID, the fourth address is set as the starting address according to the ascending order of the CSV file stream, and the process jumps to the step of determining the center address based on the starting and ending addresses. If the comparison results are inconsistent and the field value is less than the target content identifier ID, the third address is set as the ending address, and the process jumps to the step of determining the center address based on the starting and ending addresses. In this way, based on the method of comparing field values ​​and the target content identifier, if they are not equal, the values ​​of begin or end are modified according to the relationship between the target content identifier and the compared field values. The process then returns to the step of determining the center position to perform a new range of retrieval. Based on the size relationship between the comparison field values ​​recorded in the sorted CSV file stream and the target content identifier, the positional relationship between the comparison field content and the target content can be further determined, facilitating the determination of a new central location and the retrieval of a new range.

[0071] Reference Figure 3 As shown, taking a CSV file sorted in ascending order as an example:

[0072] 1. First, the user needs to specify a CSV data file, which field's value should be used as the comparison field, and the content identifier of the target data;

[0073] 2. The system considers data between two \n characters as a complete data entry. Based on this rule, the first data entry is retrieved. The first line is considered the title by default and is not included in the content calculation. Its byte count is calculated and used as the estimated byte count step for each record. The starting address of the first data entry in memory is set as begin.

[0074] 3. Next, obtain the total size of the file and calculate the end address of the entire data in memory, denoted as end;

[0075] 4. Calculate the middle address in the file memory using begin and end, and use step to offset the address to get the two nearest \n characters before and after it, so as to quickly locate the complete data (if there is no \n within one step, increment the offset by one step each time and continue searching until the first \n is found).

[0076] Specifically, with the center address as Central, we simultaneously search for \n in the directions before and after the center address using the step method. We then set the directions before and after the center address as C1 and C2, respectively, where C1 = Central and C2 = Central. Correspondingly, the addresses at one step in the directions before and after the center address are: B1 = C1 - step and E2 = C2 + step. We iterate through the CSV file stream from C1 to B1. If \n exists within this range, we record the address A0 of the first \n as the third address. If \n does not exist within this range, we set C1 = B1 and jump to the step of obtaining the new B1. Similarly, we iterate through the CSV file stream from C2 to E2. If \n exists within this range, we record the address A1 of the first \n as the fourth address. If \n does not exist within this range, we set C2 = E2 and jump to the step of obtaining the new E2.

[0077] 5. Retrieve the data from memory, convert it to a string format, and then extract the value of the comparison field from the string and compare it with the target content identifier;

[0078] Specifically, after obtaining the third address A0 and the fourth address A1, the file stream from A0+1 to A1 is extracted, and this part of the file stream is converted into a string. The content T corresponding to the index field is extracted from the string, and the size relationship between T and the target content identifier ID is compared.

[0079] a. If the results are the same, the search ends and the data is returned;

[0080] b. If they are not equal, modify the values ​​of begin or end based on the relationship between the target content identifier and the comparison field. Return 4 to perform a new range search;

[0081] Specifically, if they are not equal and ID > T, then the starting address is set to A1, and the process jumps to the step of determining the center address based on the starting address and the ending address, and the retrieval of the new address range is redefined;

[0082] If they are not equal and ID < T, then set the last address to A0 and jump to the step of determining the center address based on the start address and the last address to redetermine the search for the new address range.

[0083] Therefore, it can be seen that the retrieval process incorporates an estimated number of bytes per data record (step). When confirming the start and end positions of the data to be compared in memory, the method of reading data from the file is optimized, reducing the number of comparisons per step and thus retrieving the target data faster. Furthermore, by using the newline character \n as the calculation basis and continuously determining new center addresses, the amount of data retrieved from large CSV files is significantly reduced. In the method of comparing the extracted key-corresponding content T with the target content identifier, since the content of each field in the CSV file stream is arranged in ascending order, directly comparing the ID can determine the positional relationship between the currently obtained string content and the target content, making the process of confirming the target address simpler and suitable for fast retrieval of all ordered large CSV files.

[0084] Reference Figure 4 As shown, this embodiment of the invention also discloses a fast CSV large file retrieval device, comprising:

[0085] Address determination module 11 is used to determine a first address, a second address, and an end address from a CSV file stream based on a retrieved data request; wherein, the first address is the header address of the CSV file stream, the second address is the starting address of the first data in the CSV file stream in memory, and the end address is the ending address of the last data in the CSV file stream in memory;

[0086] Step size determination module 12 is used to take the number of bytes between the first address and the second address as the target step size;

[0087] The center address determination module 13 is used to determine the center address of the CSV file stream based on the second address and the end address;

[0088] The data retrieval module 14 is used to traverse the CSV file stream between the addresses to be traversed, starting from the center address and using the target step size, in the direction before and after the center address, until the data retrieval request is satisfied.

[0089] Therefore, this application discloses a method for determining a first address, a second address, and an end address from a CSV file stream based on a data retrieval request; wherein the first address is the header address of the CSV file stream, the second address is the starting address of the first data entry in memory, and the end address is the ending address of the last data entry in memory; the number of bytes between the first address and the second address is used as a target step size; the center address of the CSV file stream is determined based on the second address and the end address; starting from the center address, the CSV file stream between the addresses to be traversed is traversed in both the forward and backward directions of the center address using the target step size until the data retrieval request is satisfied. As can be seen, by determining the first address, second address, and end address from the ordered CSV file stream, and then determining the target step size based on the number of bytes between the first and second addresses, this target step size is used as a standard for the number of bytes per row. After determining the center position of the CSV file stream, this target step size is used to determine the addresses to be traversed in the direction before and after the center position. This process is repeated to traverse the CSV file stream between the addresses to be traversed until the position of the target data in the data request is determined in the CSV file stream. At this point, the process of determining the addresses to be traversed stops, and the data retrieval ends. This optimized retrieval algorithm enables fast retrieval of target data in large CSV files, achieving both speed and low resource consumption.

[0090] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0091] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the CSV large file fast retrieval method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0092] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0093] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0094] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0095] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the CSV large file fast retrieval method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0096] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for fast retrieval of large CSV files. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0098] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. Software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0099] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The foregoing has provided a detailed description of the fast CSV large file retrieval method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for fast retrieval of large CSV files, characterized in that, include: Based on the data retrieval request, a first address, a second address, and an end address are determined from the CSV file stream; wherein, the first address is the header address of the CSV file stream, the second address is the starting address of the first data in the CSV file stream in memory, and the end address is the ending address of the last data in the CSV file stream in memory; Use the number of bytes between the first address and the second address as the target step size; The center address of the CSV file stream is determined based on the second address and the end address; Starting from the center address, the CSV file stream between the addresses to be traversed is traversed in the direction before and after the center address using the target step size, until the data retrieval request is satisfied; Before determining the first address from the CSV file stream based on the retrieved data request, the method further includes: Obtain the CSV file stream and the retrieval data request sent by the user, which includes the index field and the identifier of the target data content in the index field; The process of obtaining the CSV file stream includes: Sort all fields of the CSV file to obtain a sorted CSV file; The CSV file is converted to obtain a CSV file stream.

2. The fast retrieval method for large CSV files according to claim 1, characterized in that, The process of traversing the CSV file stream between addresses to be traversed, starting from the center address and using the target step size, in both the forward and backward directions from the center address, further includes: Based on the target step size and according to the center address in an address offset manner, determine the third address located in the direction before the center address and the fourth address located in the direction after the center address; wherein, the third address and the fourth address are addresses to be traversed.

3. The fast retrieval method for large CSV files according to claim 2, characterized in that, The step of determining the third address located before the center address and the fourth address located after the center address based on the target step size and according to the center address in an address offset manner includes: Using the center address as the origin, offset by a target step size in the direction before and after the center address respectively, traverse the corresponding CSV file stream, and determine whether there is a corresponding first target address and second target address in the direction before and after the center address respectively; if they exist, determine the address after the first target address as the third address, and determine the target address as the fourth address.

4. The fast retrieval method for large CSV files according to claim 2, characterized in that, After determining whether there are corresponding first and second target addresses in the directions before and after the center address, respectively, the method further includes: If it does not exist, the operation of determining a new center address is triggered, and the process jumps to execute the step of offsetting by a target step size in the direction before and after the center address, respectively.

5. The method for fast retrieval of large CSV files according to claim 2, characterized in that, The step of traversing the CSV file stream between the addresses to be traversed using the target step size, respectively, in the direction before and after the center address, until the data retrieval request is satisfied, includes: The data between the third address and the fourth address is used as the target data, and the target data is retrieved from memory and converted into a target string; Compare the target index field with the index field in the target string, and the target content identifier with the identifier in the target string; If the comparison is consistent, the search ends and the target string is returned to the human-computer interaction interface.

6. A fast retrieval device for large CSV files, characterized in that, include: The address determination module is used to determine a first address, a second address, and an end address from a CSV file stream based on a retrieved data request; wherein, the first address is the header address of the CSV file stream, the second address is the starting address of the first data in the CSV file stream in memory, and the end address is the ending address of the last data in the CSV file stream in memory; The step size determination module is used to take the number of bytes between the first address and the second address as the target step size; A center address determination module is used to determine the center address of the CSV file stream based on the second address and the end address; The data retrieval module is used to traverse the CSV file stream between the addresses to be traversed, starting from the center address and using the target step size, in the direction before and after the center address, until the data retrieval request is satisfied. The CSV large file fast retrieval device is also used to acquire CSV file streams and acquire retrieval data requests sent by the user terminal, which include index fields and identifiers of target data content in the index fields. The CSV large file fast retrieval device is also used to sort all fields of the CSV file to obtain a sorted CSV file; and to convert the CSV file to obtain a CSV file stream.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the fast CSV large file retrieval method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the steps of the fast retrieval method for large CSV files as described in any one of claims 1 to 5.

Images