File reading and writing method and system
By writing and reading data by columns in a distributed file system, combining template functions and line number offsets, the problem of reduced read and write efficiency caused by increasing data volume is solved, and efficient data processing is achieved.
Patent Information
- Application Number
- CN202411324498.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-27
AI Technical Summary
With the increase in the amount of data, the read and write efficiency of data in distributed file systems gradually decreases, making it difficult to meet the needs of efficient data storage and management in large distributed environments.
By writing target data with the same processing logic into the distributed file system according to columns, and determining the line number offset based on the data type and batch volume when reading, combining template function instantiation to obtain the calling code, in the distributed file system Read data in batches according to columns to reduce the number of calls to virtual functions.
When data is written, it is stored in columns, and when reading, it is read in batches, which effectively reduces the number of calls of data and virtual functions and improves the efficiency of data processing.
Smart Images
Figure CN119200992A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of distributed file system data reading and writing, and specifically relates to a file reading and writing method and system. Background Art
[0002] Battery data is stored in the distributed file system (Hadoop Distributed File System, HDFS) in the distributed system infrastructure Hadoop to store and manage data in a large distributed environment. With the increase in business volume, the data that needs to be stored and managed continues to increase, resulting in a decrease in data reading and writing efficiency. Summary of the invention
[0003] Purpose of the invention: This application develops a file reading and writing method and system, aiming to solve the technical problem in the prior art that as the amount of data increases, the efficiency of data reading and writing decreases.
[0004] Technical solution: In a first aspect, an embodiment of the present application provides a file reading and writing method, which is applied to a distributed file system. The file reading and writing method includes:
[0005] Determine target data, and write the target data with the same processing logic into the distributed file system in columns;
[0006] Determine a row number offset based on the length of the data type of the target data and the batch processing amount of the target data;
[0007] Determine the calling position based on the line number offset and the line number starting position;
[0008] Instantiate the template function based on the data type and obtain the calling code;
[0009] Based on the calling code and the calling position, the target data having the same processing logic is read in batches by columns in the distributed file system.
[0010] In some embodiments, before writing the target data with the same processing logic into the distributed file system by column, the file reading and writing method further includes:
[0011] Convert the string type data in the target data into the integer value type data.
[0012] In some embodiments, before writing the target data with the same processing logic into the distributed file system by column, the file reading and writing method further includes:
[0013] Determine a primary classification parameter and a first storage area, and classify the target data into a plurality of first data based on the primary classification parameter, and store the first data in a plurality of first storage areas respectively;
[0014] A two-pole classification parameter and a second storage area are determined, and the first data are classified into a plurality of second data based on the two-pole classification parameter, and are respectively stored in a plurality of second storage areas, wherein the plurality of second storage areas are arranged in the same first storage area.
[0015] In some embodiments, the step of reading the target data in the distributed file system based on the calling code and the calling location includes:
[0016] Inserting the calling code into the calling location;
[0017] determining a reader of the target data;
[0018] The reader is controlled to read the target data of the calling location by calling the code.
[0019] In some embodiments, before the step of controlling the reader to read the target data at the calling location by calling the code, the file reading and writing method includes:
[0020] Determine a thread for reading the target data;
[0021] Obtaining the file handle when reading the target data;
[0022] The file handle is stored in the thread, so that the thread obtains corresponding target data through the file handle stored in the thread.
[0023] In some embodiments, the step of storing the file handle in the thread comprises:
[0024] Storing the file handle in a variable of a thread local storage keyword;
[0025] The thread local storage key is set in the thread.
[0026] In some embodiments, the target data is compressed using a data compression library when stored, and before controlling the reader to read the target data at the calling location through the calling code, the step of reading the target data based on the calling code and the calling location further includes:
[0027] Acquire a first decompressor for the target data, wherein the first decompressor is used to decompress data compressed by a data compression library;
[0028] Acquire a pseudo-random algorithm library, and reconstruct the first decompressor based on the pseudo-random algorithm library to acquire a second decompressor;
[0029] Before reading the target data, the target data is decompressed by the second decompressor.
[0030] In some embodiments, the target data is stored in a columnar format using an optimized row-column storage format, and the reader is a reader for reading a data warehouse row format file.
[0031] In some embodiments, the step of determining a reader of the target data comprises:
[0032] Acquire run-length-encoded decoded operation data, and transplant the decoded operation data into the reader;
[0033] The step of controlling the reader to read the target data of the calling position by calling the code comprises:
[0034] An instruction set of a processor for bit operations is obtained, and the decoding bit operation process of the variable-length integer encoding in the decoding operation of the run-length encoding in the reader is accelerated by the instruction set.
[0035] In a second aspect, an embodiment of the present application further provides a file reading system, which is applied to a distributed file system, and the file reading system includes:
[0036] A data writing module, the data writing module is used to determine target data, and write the target data with the same processing logic into the distributed file system in columns;
[0037] An offset determination module, the offset determination module being used to determine a row number offset based on a length of a data type of the target data and a batch processing amount of the target data;
[0038] A calling position determination module, the calling position determination module is used to determine the calling position based on the line number offset and the line number starting position;
[0039] A calling code determination module, the calling code determination module is used to instantiate the template function based on the data type and obtain the calling code;
[0040] A data reading module is used to batch read the target data with the same processing logic in columns in the distributed file system based on the calling code and the calling position.
[0041] Beneficial effect: Compared with the prior art, a file reading and writing method provided by an embodiment of the present application is applied to the storage and management of data in a distributed file system. The file reading and writing method provided by an embodiment of the present application writes the target data with the same processing logic into the distributed file system by column; when reading data, the row number offset of the batched target data is determined based on the length of the data type of the target data and the batch processing amount of the target data, and the row number offset is combined with the row number initial position to determine the calling position of the batched target data; the template function is instantiated according to the data type of the target data, and the calling code of the target data is obtained from the instantiated template function; finally, the target data with the same processing logic is read in batches by column in the distributed file system in combination with the calling code and the calling position. The present application stores the target data with the same processing logic by column when writing data, and reads the target data with the same processing logic by column in batches when reading. In the process of a batch reading, only one virtual function needs to be called to read multiple data, which effectively reduces the number of calls to data and virtual functions, calls more data with fewer calls, and improves the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0043] Figure 1 A flowchart of the steps of the file reading and writing method provided in the embodiment of the present application;
[0044] Figure 2 A specific flow chart of step 100 in the file reading and writing method provided in an embodiment of the present application;
[0045] Figure 3 A specific flow chart of step 500 in the file reading method provided in an embodiment of the present application;
[0046] Figure 4 A specific flow chart of setting a file handle in a thread provided in an embodiment of the present application;
[0047] Figure 5 A specific flow chart of step c in the file reading and writing method provided in an embodiment of the present application;
[0048] Figure 6 A module connection diagram of a file reading and writing system provided in an embodiment of the present application;
[0049] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0050] Figure 8 This is a schematic diagram of the HDFS structure;
[0051] Reference numerals: 10, data writing module; 20, offset determination module; 30, calling position determination module; 40, calling code determination module; 50, data reading module; 100, memory; 200, processor. DETAILED DESCRIPTION
[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0053] As a big data platform, battery data will be stored in the distributed file system (Hadoop Distributed File System, HDFS) in the Hadoop distributed system infrastructure. The distributed file system is one of the core components of Hadoop and is used to store and manage data in a large distributed environment. The distributed file system consists of three components: NameNode, DataNode, and Secondary NameNode. For details, please refer to Figure 8 , Figure 8 This is a schematic diagram of the HDFS structure, where:
[0054] NameNode is a node used to command other nodes to store data. Any file system needs to be able to map files according to file paths. NameNode is a computer used to store this mapping information and provide mapping services. It plays the role of "administrator" in the entire HDFS system. Therefore, there is only one NameNode in an HDFS cluster.
[0055] DataNode is a node used to store data blocks. When a file is recognized and divided into blocks by a named node, it will be stored in the assigned data node. Data nodes have the functions of storing data, reading and writing data. The stored data blocks are similar to the concept of "sectors" in hard disks and are the basic unit of HDFS storage.
[0056] Secondary NameNode is also known as "Secondary NameNode", which is the "secretary" of the NameNode. This description is very appropriate because it cannot replace the work of the NameNode, regardless of whether the NameNode is able to continue working. It is mainly responsible for sharing the pressure of the NameNode, backing up the status of the NameNode and performing some management work if the NameNode requires it to do so. If the NameNode is broken, it can also provide backup data to restore the NameNode. There can be multiple secondary NameNodes.
[0057] ORC, Optimized Row Columnar, is a columnar storage format widely used in the Hadoop ecosystem. It was originally introduced by the Apache Hive project to reduce Hadoop data storage space and speed up Hive queries.
[0058] As business volume increases, the amount of data that needs to be stored and managed continues to increase, resulting in lower data reading and writing efficiency.
[0059] In view of this, an embodiment of the present application provides a file reading and writing method, which is applied to the storage and management of data in a distributed file system. The file reading and writing method provided by the embodiment of the present application writes the target data with the same processing logic into the distributed file system by column; when reading data, the row number offset of the batched target data is determined based on the length of the data type of the target data and the batch processing amount of the target data, and the row number offset is combined with the row number initial position to determine the calling position of the batched target data; the template function is instantiated according to the data type of the target data, and the calling code of the target data is obtained from the instantiated template function; finally, the calling code and the calling position are combined to batch read the target data with the same processing logic by column in the distributed file system. The present application stores the target data with the same processing logic by column when writing data, and reads the target data with the same processing logic by column in batches when reading. In the process of a batch reading, only one virtual function needs to be called to read multiple data, which effectively reduces the number of calls to data and virtual functions, calls more data with fewer calls, and improves the efficiency of data processing.
[0060] In some embodiments, see Figure 1 , Figure 1 This is a step flow chart of the file reading and writing method provided in the embodiment of the present application. The file reading and writing method provided in the embodiment of the present application is specifically implemented through steps 100 to 500:
[0061] Step 100: Determine target data, and write the target data with the same processing logic into the distributed file system by column.
[0062] Specifically, when writing to the target data, the target data with the same processing logic is stored in columns in the data node of HDFS in ORC format. It can be understood that when the memory needs to query and read data, it is convenient to arrange multiple rows of target data with the same processing logic together and read them into the memory in batch mode, so that the CPU (Central Processing Unit) can load the target data to be processed into the cache at one time when fetching the target data, reducing the memory pause time and improving the reading efficiency of the target file; at the same time, the memory addresses of the target data in the same column are adjacent, and the read pointer only moves within a specific range and will not be affected by other columns, ensuring the relative isolation of the target data.
[0063] In some embodiments, see Figure 2 , Figure 2 The specific flow chart of step 100 in the file reading and writing method provided in the embodiment of the present application, the method of classifying and storing files when writing target data with the same processing logic into a distributed file system by column is specifically implemented through steps 110 to 120:
[0064] Step 110: Determine a primary classification parameter and a first storage area, and classify the target data into a plurality of first data based on the primary classification parameter, and store the data in a plurality of first storage areas respectively.
[0065] Step 120: Determine a binary classification parameter and a second storage area, and classify the first data into a plurality of second data based on the binary classification parameter, and store the first data in a plurality of second storage areas respectively, wherein the plurality of second storage areas are arranged in the same first storage area.
[0066] It can be understood that depending on the different application objects of the secondary battery, the specific attributes of the primary classification parameters and the secondary classification parameters can be determined according to needs. For example, when the secondary battery is used in the vehicle field, the vehicle enterprise can be defined as the primary classification, and the vehicle model can be defined as the secondary classification. The target data of the same vehicle enterprise can be stored in an enterprise folder, and then multiple vehicle model folders can be separated in the enterprise folder. Vehicles of the same model under the name of the enterprise can be placed in the same vehicle model folder. Isolation can be achieved at the file level, and when reading a vehicle model with a small amount of data, it will not be affected by the vehicle model with a large amount of data.
[0067] Step 200: Determine the row number offset based on the length of the data type of the target data and the batch processing amount of the target data.
[0068] Step 300: Determine the call location based on the line number offset and the line number start position.
[0069] In some embodiments, after acquiring the target data, before writing the target data with the same processing logic into HDFS by column, the data to be written is further processed, and the string type data in the target data is converted into integer value type data. It can be understood that the string type field is a variable length field, and different data may have different lengths when stored, so it is impossible to locate the data in the column to be found by converting the row number offset. Since the data of the secondary battery is basically a numerical type signal collected by the sensor, there are very few string type signals, such as the frame number, battery pack number, etc., so the string type signal can be converted into an integer value type signal through hexadecimal conversion and then stored, which can effectively ensure the consistency of the data length and facilitate the positioning of the target data by the row number offset.
[0070] Step 400: Instantiate the template function based on the data type and obtain the calling code.
[0071] As you can understand, template functions allow you to write functions that can handle multiple data types without having to write multiple versions for each data type. The definition of a template function is as follows:
[0072] template<typename T> ;
[0073] void myFunction(T arg){
[0074] / / Function implementation
[0075] };
[0076] Among them, T is a placeholder type that can be replaced by a specific type when calling.
[0077] Template function instantiation means that the compiler generates corresponding function code according to the actual data type passed in. For example, there is the following template function:
[0078] template<typename T> ;
[0079] void print(T value){
[0080] std::cout< <value<<std::endl;
[0081] }.
[0082] Exemplarily, when the above template function is called using a specific data type, the compiler generates a specific function instance, i.e., a calling code, for each type; for example, the calling code obtained when the template function is called using the integer (int) data type data 5 and the double-precision floating point (double) data type data 3.14 is as follows:
[0083] print(5); / / Instantiate to print <int>(int value);
[0084] print(3.14); / / Instantiate as print <double>(double value).
[0085] Step 500: Based on the calling code and the calling location, target data with the same processing logic is read in batches by column in the distributed file system.
[0086] It is understandable that the call of the target data will use virtual functions, and the call of the virtual function will be accompanied by a table lookup operation. Frequent table lookups in scenarios with a large amount of data will affect the performance of the system. When using the file reading and writing method provided by this application, the processing logic of the same column of data in a loop is the same, and the batch iteration method is used to reduce the number of virtual function calls. This application uses a template function. Since the data type length is fixed above, the data that needs to be processed in batches is retrieved through the row number offset, and the batch data that needs to be processed has the data type obtained in advance, the template function can be expanded inline, and the calling code of the template function is directly inserted into the calling point without the use of a function call mechanism, eliminating the overhead of function calls and making program execution more efficient.
[0087] In some embodiments, see Figure 3 , Figure 3 The specific flow chart of step 500 in the file reading method provided in the embodiment of the present application, the method of batch reading target data with the same processing logic by row in HDFS is specifically implemented through steps 510 to 530:
[0088] Step 510: insert the calling code into the calling location;
[0089] Step 520: Determine a reader of target data.
[0090] It can be understood that since the target data is stored in an optimized row-column storage format, a reader that reads the data warehouse row format file is used when reading the target data. Specifically, the present application uses the velox vectorized execution engine plug-in to process the target data. The velox plug-in native module supports the DWRF data warehouse row format data format. The DWRF format is a branch version of the ORC format, and its file structure is similar, so the reader that reads the DWRF format file can be reused to read the stored target data.
[0091] In some embodiments, after the reader is determined, the decoded operation data of RLEv2 (Run-Length Encoding version 2) is obtained, and the decoded operation data is transplanted into the reader.
[0092] In some embodiments, the target data is compressed using the zlib data compression library when it is stored, and the compressed stored data needs to be decompressed before the control reader reads the target data at the calling location by calling the code. Figure 5 , Figure 5 This is a specific flow chart for decompressing target data in the file reading and writing method provided in the embodiment of the present application. The method for decompressing the compressed target data is specifically implemented through steps d to f:
[0093] Step d: Obtain a first decompressor for target data, where the first decompressor is used to decompress data compressed by a data compression library.
[0094] It can be understood that since the target data is compressed using the zlib data compression library when stored, a Zlib decompressor (ZlibDecompressor) is selected to decompress the target data.
[0095] Step e: Obtain a pseudo-random algorithm library, and reconstruct the first decompressor based on the pseudo-random algorithm library to obtain a second decompressor.
[0096] Understandably, after analyzing the ORC file reading time consumption records, it is found that the decompression time of the zlib decompressor accounts for 60% of the total file reading time, the file decoding time accounts for 30% of the total file reading time, and other operations account for only 10%. It can be seen that the decompression efficiency has a great impact on the ORC file reading performance. To this end, Intel's decompression vectorization library ISA-L pseudo-random algorithm library is introduced to reconstruct the zlib decompressor and generate a second decompressor to speed up the decompression process.
[0097] Step f: Before reading the target data, decompress the target data by a second decompressor.
[0098] Step 530: Control the reader to read the target data of the calling location by calling the code.
[0099] It can be understood that since the run-length encoded decoding operation data is introduced in the reader, when reading the target data, BMI2 (Bit Manipulation Instructions 2) - the instruction set of the processor used for bit operations can be obtained. The instruction set can accelerate the decoding bit operation process of the variable-length integer encoding in the decoding operation of RLEv2 in the reader, and at the same time, unnecessary data can be pushed down and filtered during the decoding process.
[0100] In some embodiments, multiple DataNodes are set up, and the load of each DataNode is monitored in real time. The slow node is determined based on the P99.9 percentile request latency, and the read request is routed to the DataNode with lower load to reduce waiting time. At the same time, in a random scenario with small data volume, the DataNode does not need to pre-fetch when returning data, and only returns the required data, which can reduce pre-reading occupancy and reduce pre-amplification behavior.
[0101] In some embodiments, before the step of controlling the reader to read the target data of the calling location by calling the code, it is necessary to store the file handle in the thread, see Figure 4 , Figure 4 The specific flow chart of setting a file handle in a thread provided by the embodiment of the present application, the method of storing a file handle in a thread is specifically implemented through steps a to c:
[0102] Step a: Determine the thread that reads the target data.
[0103] Step b: Get the file handle for reading target data.
[0104] Step c: Store the file handle in the thread, so that the thread obtains the corresponding target data through the file handle stored in the thread.
[0105] Understandably, every time HDFS reads a file, it will open a new file handle for addressing and reading operations. The client will send a large number of file opening requests to the NameNode, increasing the pressure on HDFS. This application places the file reading handle in a thread to achieve internal multiplexing, reduce the number of times the client sends file opening requests to the NameNode, and reduce the pressure on HDFS.
[0106] In some embodiments, see Figure 5 , Figure 5 This is a specific flow chart of step c in the file reading and writing method provided in the embodiment of the present application. The method of storing the file handle in the thread is specifically implemented through steps c1 to c2:
[0107] Step c1: Store the file handle in a variable of thread local storage keyword.
[0108] Step c2: Set the thread local storage keyword in the thread.
[0109] It can be understood that the file reading and writing method provided by the embodiment of the present application is applied to the storage and management of data in a distributed file system. The file reading and writing method provided by the embodiment of the present application writes the target data with the same processing logic into the distributed file system by column; when reading data, the row number offset of the target data of the batch is determined based on the length of the data type of the target data and the batch processing amount of the target data, and the row number offset is combined with the row number initial position to determine the calling position of the target data of the batch; the template function is instantiated according to the data type of the target data, and the calling code of the target data is obtained from the instantiated template function; finally, the calling code and the calling position are combined to batch read the target data with the same processing logic by column in the distributed file system. The present application stores the target data with the same processing logic by column when writing data, and reads the target data with the same processing logic by column in batches when reading. In the process of a batch reading, only one virtual function needs to be called to read multiple data, which effectively reduces the number of calls to data and virtual functions, calls more data with fewer calls, and improves the efficiency of data processing.
[0110] Accordingly, the embodiment of the present application also provides a file reading and writing system, please refer to Figure 6 , Figure 6 This is a module connection diagram of a file reading and writing system provided in an embodiment of the present application. The file reading and writing system provided in an embodiment of the present application includes:
[0111] The data writing module 10 is used to determine the target data and write the target data with the same processing logic into the distributed file system in columns;
[0112] An offset determination module 20, the offset determination module 20 is used to determine a row number offset based on the length of a data type of the target data and a batch processing amount of the target data;
[0113] A calling position determination module 30, the calling position determination module 30 is used to determine the calling position based on the line number offset and the line number starting position;
[0114] A calling code determination module 40, the calling code determination module 40 is used to instantiate the template function based on the data type and obtain the calling code;
[0115] The data reading module 50 is used to batch read target data with the same processing logic in columns in the distributed file system based on the calling code and the calling position.
[0116] In some embodiments, the file reading and writing system is specifically used for:
[0117] Before writing the target data with the same processing logic into the distributed file system by column, the string type data in the target data is converted into the integer value type data.
[0118] In some embodiments, the data writing module 10 is specifically used for:
[0119] Determine a primary classification parameter and a first storage area, and classify the target data into a plurality of first data based on the primary classification parameter, and store the first data in a plurality of first storage areas respectively;
[0120] A binary classification parameter and a second storage area are determined, and the first data are classified into a plurality of second data based on the binary classification parameter, and are respectively stored in a plurality of second storage areas, wherein the plurality of second storage areas are arranged in the same first storage area.
[0121] In some embodiments, the data reading module 50 is specifically used for:
[0122] Insert the calling code at the calling location.
[0123] Determines the reader of the target data.
[0124] The control reader reads the target data of the calling location by calling the code.
[0125] In some embodiments, the file reading and writing system is specifically used for:
[0126] Determine the thread that reads the target data;
[0127] Get the file handle when reading the target data;
[0128] The file handle is stored in the thread, so that the thread obtains the corresponding target data through the file handle stored by itself.
[0129] In some embodiments, the file reading and writing system is specifically used for:
[0130] Store the file handle in a variable with the thread-local storage keyword;
[0131] Sets the thread-local storage keyword in the thread.
[0132] In some embodiments, the data reading module 50 is specifically used for:
[0133] Obtaining a first decompressor for target data, the first decompressor being used to decompress data compressed by a data compression library;
[0134] Obtain a pseudo-random algorithm library, and reconstruct a first decompressor based on the pseudo-random algorithm library to obtain a second decompressor;
[0135] Before reading the target data, the target data is decompressed by a second decompressor.
[0136] In some embodiments, the data reading module 50 is specifically used for:
[0137] Obtaining run-length-encoded decoded operation data, and transplanting the decoded operation data into a reader;
[0138] An instruction set of the processor 200 for bit operations is obtained, and the decoding bit operation process of the variable-length integer encoding in the decoding operation of the run-length encoding in the reader is accelerated by the instruction set.
[0139] Accordingly, an embodiment of the present application further provides an electronic device, see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device provided in an embodiment of the present application includes:
[0140] The memory 100 and the processor 200 are connected to each other in communication, the memory 100 stores computer instructions, and the processor 200 executes the file reading and writing method provided in the embodiment of the present application by executing the computer instructions. Specifically, the memory 100 and the processor 200 are connected via a communication bus.
[0141] In some embodiments, the processor 200 may be a central processing unit (CPU). The processor 200 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.
[0142] In some embodiments, the memory 100 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the methods in the method embodiments of the present application. The processor 200 executes various functional applications and data processing of the processor 200 by running the non-transitory software programs, instructions and modules stored in the memory 100, that is, implementing the methods in the above method embodiments.
[0143] In some embodiments, the memory 100 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device or at least one function; the data storage area may store data created by the processor 200, etc. In addition, the memory 100 may include a high-speed random access memory 100, and may also include a non-volatile memory 100, such as at least one disk memory 100, a flash memory device, or other non-volatile solid-state memory 100. In some embodiments, the memory 100 may optionally include a memory 100 remotely arranged relative to the processor 200, and these remote memories 100 may be connected to the processor 200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0144] In some embodiments, one or more modules are stored in the memory 100 and when executed by the processor 200, the methods in the above method embodiments are performed.
[0145] In some embodiments, the specific details of the above-mentioned electronic device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned method embodiments, and will not be repeated here.
[0146] The present application has introduced in detail a file reading and writing method and system provided by the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.< / double> < / int>
Claims
1. A file reading and writing method, characterized in that: Applied to a distributed file system, the file reading and writing method includes: Determine target data, and write the target data with the same processing logic into the distributed file system in columns; Determine a row number offset based on the length of the data type of the target data and the batch processing amount of the target data; Determine the calling position based on the line number offset and the line number starting position; Instantiate the template function based on the data type and obtain the calling code; Based on the calling code and the calling position, the target data having the same processing logic is read in batches by columns in the distributed file system.
2. The file reading and writing method according to claim 1, characterized in that: Before writing the target data with the same processing logic into the distributed file system in columns, the file reading and writing method further includes: Convert the string type data in the target data into the integer value type data.
3. The file reading and writing method according to claim 1, characterized in that: The method of writing the target data having the same processing logic into the distributed file system by column also includes: Determine a primary classification parameter and a first storage area, and classify the target data into a plurality of first data based on the primary classification parameter, and store the first data in a plurality of first storage areas respectively; A two-pole classification parameter and a second storage area are determined, and the first data are classified into a plurality of second data based on the two-pole classification parameter, and are respectively stored in a plurality of second storage areas, wherein the plurality of second storage areas are arranged in the same first storage area.
4. The file reading and writing method according to claim 1, characterized in that: The step of batch reading the target data having the same processing logic by column in the distributed file system based on the calling code and the calling position comprises: Inserting the calling code into the calling location; determining a reader of the target data; The reader is controlled to read the target data of the calling location by calling the code.
5. The file reading and writing method according to claim 4, characterized in that: Before the step of controlling the reader to read the target data at the calling position by calling the code, the file reading and writing method includes: Determine a thread for reading the target data; Obtaining the file handle when reading the target data; The file handle is stored in the thread, so that the thread obtains corresponding target data through the file handle stored in the thread.
6. The file reading and writing method according to claim 5, characterized in that: The step of storing the file handle in the thread comprises: Storing the file handle in a variable of a thread local storage keyword; The thread local storage key is set in the thread.
7. The file reading and writing method according to claim 4, characterized in that: The target data is compressed by a data compression library when stored. Before controlling the reader to read the target data at the calling position by calling the code, the step of reading the target data based on the calling code and the calling position further includes: Acquire a first decompressor for the target data, wherein the first decompressor is used to decompress data compressed by a data compression library; Acquire a pseudo-random algorithm library, and reconstruct the first decompressor based on the pseudo-random algorithm library to acquire a second decompressor; Before reading the target data, the target data is decompressed by the second decompressor.
8. The file reading and writing method according to claim 4, characterized in that: The target data is stored in columnar format using an optimized row-column storage format, and the reader is a reader for reading data warehouse row format files.
9. The file reading and writing method according to claim 4, characterized in that: The step of determining the reader of the target data comprises: Acquire run-length-encoded decoded operation data, and transplant the decoded operation data into the reader; The step of controlling the reader to read the target data of the calling position by calling the code comprises: An instruction set of a processor for bit operations is obtained, and the decoding bit operation process of the variable-length integer encoding in the decoding operation of the run-length encoding in the reader is accelerated by the instruction set.
10. A file reading and writing system, characterized in that: Applied to a distributed file system, the file reading system comprises: A data writing module, the data writing module is used to determine target data, and write the target data with the same processing logic into the distributed file system in columns; An offset determination module, the offset determination module being used to determine a row number offset based on a length of a data type of the target data and a batch processing amount of the target data; A calling position determination module, the calling position determination module is used to determine the calling position based on the line number offset and the line number starting position; A calling code determination module, the calling code determination module is used to instantiate the template function based on the data type and obtain the calling code; A data reading module is used to batch read the target data with the same processing logic in columns in the distributed file system based on the calling code and the calling position.