Data reading method and related device

Abstract

The invention provides a data reading method and a related device. The method comprises the steps of determining a first storage length according to a first attribute and a second attribute of a first file; wherein the first storage length is the storage length of each data record in a data area of the first file; reading a target data record of the first file from the data area according to the first storage length; and under the condition that the data type of target data in the target data record is a character string pointer, determining a first storage position in a character string area of the first file according to the target data, and reading the character string data based on the first storage position. According to the method and the device, the character string data can be efficiently and accurately read, the calculation cost during data reading is saved, and the data reading efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data reading method and related devices. Background Art

[0002] With the rapid development of information technology, the storage and processing of data files have become increasingly important. Data files consisting of file headers and data areas are increasingly becoming the mainstream way of data storage and exchange. Their wide application scope covers almost every corner of the information technology field. In emerging technology fields such as cloud computing, big data, and the Internet of Things, data files consisting of file headers and data areas play a vital role. For example, on a cloud computing platform, business data is stored in a distributed file system in the form of files. The file header contains information such as the data version and access rights to ensure the accuracy and privacy of the data; while the data area stores specific business data to support various types of data analysis and mining operations.

[0003] However, due to the inconsistent length of string data in data files, the reading method for string data is inconsistent with the reading method for other types of data. Therefore, how to read string data efficiently and accurately has become a hot topic in current research. Summary of the invention

[0004] The embodiment of the present application provides a data reading method and related devices. The embodiment of the present application can efficiently and accurately read string data, save the calculation cost when reading data, and improve the efficiency of data reading.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a data reading method, which includes: determining a first storage length according to a first attribute and a second attribute of a first file; wherein the first storage length is the storage length of each data record in a data area of ​​the first file; reading a target data record of the first file from the data area according to the first storage length; when the data type of the target data in the target data record is a string pointer, determining a first storage position in the string area of ​​the first file according to the target data, and reading the string data based on the first storage position.

[0007] An embodiment of the present application provides a data reading device, including: a determination module, used to determine a first storage length according to a first attribute and a second attribute of a first file; wherein the first storage length is the storage length of each data record in a data area of ​​the first file; a reading module, used to read a target data record of the first file from the data area according to the first storage length; the reading module is also used to determine a first storage position in a string area of ​​the first file according to the target data when the data type of the target data in the target data record is a string pointer, and to read the string data based on the first storage position.

[0008] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions; and a processor for implementing the data reading method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0009] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing a data reading method provided in an embodiment of the present application when executed by a processor.

[0010] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data reading method provided in the embodiment of the present application is implemented.

[0011] The embodiments of the present application have the following beneficial effects:

[0012] On the one hand, in the embodiment of the present application, when the target data is a string pointer, the string data corresponding to the target data is read from the storage position pointed to by the string pointer in the string area, so that when the string data itself has different lengths, each data record in the data area can also maintain a consistent storage length, and the storage length (first storage length) of each data record can be clearly determined according to the first attribute and the second attribute in the file header, so when reading the target data record, the storage position of the target data record can be directly determined according to the first storage length, thereby saving the calculation cost when reading data and improving the efficiency of data reading. On the other hand, the embodiment of the present application stores the string data in an orderly and continuous manner in the string area, rather than in the data area, which can avoid the waste of storage space caused by the difference in the storage length of the string data, so that storage resources are more efficiently and reasonably used. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A It is a flowchart of a data reading method provided in an embodiment of the present application;

[0014] Figure 1B is a schematic diagram of a process for determining a first storage length provided in an embodiment of the present application;

[0015] Figure 1C is a schematic diagram of a process of traversing a first attribute provided in an embodiment of the present application;

[0016] Figure 1D is a schematic diagram of a process for determining target data records provided by an embodiment of the present application;

[0017] Figure 1E It is a schematic diagram of a process for determining a storage starting point position and a storage end point position provided in an embodiment of the present application;

[0018] Figure 1F is a schematic diagram of a process for reading target data provided by an embodiment of the present application;

[0019] Figure 1G It is a schematic diagram of the process of verifying the storage length of a data file provided in an embodiment of the present application;

[0020] Figure 1H is a schematic diagram of a process for determining a first file to be read provided by an embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of the storage structure of the data file provided in the embodiment of the present application;

[0022] Figure 3 is a structural schematic diagram of a data reading device provided in an embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0025] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] In the following description, the terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0027] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0028] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0029] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0030] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0031] 1) Storage Length: It is used to describe the amount of space occupied by a data item or data structure on a storage medium (such as a hard disk, memory, or flash memory). It is usually measured in bytes or bits. Storage length covers the size of data at the physical level and is a basic concept in data management and file operations. For example, an integer (int) data occupies 4 bytes (32 bits) in modern computer architectures, while a character (char) data occupies 1 byte (8 bits).

[0032] 2) Magic Number refers to a fixed sequence of bytes contained in the file header or a specific location to identify the file type or format. The magic number is unique to a specific file format, and even if the file extension is changed or lost, the magic number can be used to quickly identify the true type of the file.

[0033] 3) Offset is the distance or quantity from a certain starting point (such as the starting position of an array, the beginning of a file, the starting point of a memory address, etc.) to the target position. This distance or quantity is usually measured in bytes, bits, or other data units.

[0034] In the related art, when faced with string data of variable length, data files often adopt the following two strategies: 1) All string data in the data file, regardless of the actual content length, are allocated storage space equal to the longest string; although this strategy can ensure the consistency of the data structure, it inevitably causes a large amount of storage resources to be idle and wasted, especially when most strings are much shorter than the maximum length, the computational cost consumed during reading is relatively high. 2) Allocate storage space to string data according to the actual content length; although this strategy can save a certain amount of storage space, when reading data files, it is difficult to determine the exact length of each data record in advance, especially when the record contains a string of uncertain length, and tedious length detection and boundary determination must be performed, which undoubtedly increases the computational complexity during the reading process and reduces the reading efficiency.

[0035] Based on the above-mentioned problems existing in the related technology, an embodiment of the present application provides a data reading method. On the one hand, the embodiment of the present application stores a string pointer pointing to the string data. When the length of the string data itself is different, the storage length of each data record in the data area can also be kept consistent, and the storage length (first storage length) of each data record can be clearly determined according to the first attribute and the second attribute in the file header. Therefore, when reading the target data record, the first storage position of the target data record can be directly determined according to the first storage length, and there is no need to determine the different storage lengths of all data records one by one in order to read the target data record, thereby saving the computing cost consumed when reading data and improving the efficiency of data reading. On the other hand, in order to ensure that the storage length of each data record is consistent, the related technology requires that the storage space in each data record must be sufficient to accommodate the storage length of the longest string data. For shorter string data, the storage length is not enough to fill the storage length reserved for the longest string data in the data record. Therefore, directly storing string data of different storage lengths in the data area will result in a waste of storage space. The embodiment of the present application stores the string data in an orderly and continuous manner in the string area rather than in the data area, which can avoid the waste of storage space caused by differences in the storage length of the string data, and make storage resources more efficiently and reasonably utilized.

[0036] The data reading method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0037] See also Figure 1A , Figure 1A This is an optional flow chart of the data reading method provided in the embodiment of the present application. Figure 1A The steps shown are explained as Figure 1A As shown, the data reading method is described by taking the execution subject of the electronic device as an example, and the method includes the following steps 101 to 103:

[0038] Step 101: determine a first storage length according to a first attribute and a second attribute of a first file.

[0039] In some embodiments, the first file refers to a data file from which a user or program needs to read data, and may also be referred to as a to-be-read file. The data file may be a database file, a log file, a configuration file, or any other file containing structured data. The data file of the embodiment of the present application is composed of a file header, a data area, and a string area, which are arranged in sequence in the data file, the file header being located at the beginning of the data file, the data area being immediately after the file header, and the string area being located behind the data area.

[0040] The file header can contain information about the data file itself, such as magic number, version, number of data columns and other attributes. The file header can be regarded as the directory of the data file. Parsing the data in the file header can tell you how to interpret and access the data in the data file. The data area is where the data is actually stored in the data file. The data area contains all the data records of the data file. Each data record consists of one or more fields (columns). Different fields can be of different data types, such as integer, floating point, string or character, etc. The string area is an area used to store string data separately. The string data is stored continuously in the string area. The field corresponding to the string type in the data area stores a pointer to the string data in the string area. Even if the storage length of the string data is different, the storage length of each data record in the data area can remain consistent, because the data record stores a pointer of a fixed storage length instead of the entire string data.

[0041] In some embodiments, the first attribute is an attribute in the file header used to describe the data type of each column of data in the data area. The first attribute can also be called a column type attribute; for example, a column in the data area can be an integer type, a floating point type, a date and time type, or a string type. The second attribute is an attribute in the file header used to specify the storage length of the string pointer in the data area. The second attribute can also be called a string data length attribute. The first storage length is the storage length required for each data record in the data area, and the first storage length is calculated based on the first attribute and the second attribute.

[0042] For example, the first attribute can be "[int, float, char, string]", to indicate that the data types of each column of data in the data area are integer, floating point, character and string respectively; the second attribute can be 8 (bit), to indicate that the storage length of the data in the column with string data type in the data area is 8 bits; the first storage length can be 32 (storage length corresponding to integer type) + 32 (storage length corresponding to floating point type) + 8 (storage length corresponding to character type) + 8 (the value stored in the string data length attribute) = 80 (unit: bit).

[0043] Through step 101, accurate data record storage information can be provided for subsequent data reading and processing, which helps to efficiently access and parse the data in the data area without the need to perform dynamic storage length calculation when reading each record; it is also possible to allocate an appropriate buffer or data packet size for data reading or transmission based on the first storage length, thereby reducing memory usage and network transmission overhead.

[0044] In the embodiment of the present application, the first attribute is used to characterize the data type of each data included in each data record in the data area, and the second attribute is used to characterize the storage length of the string pointer in the data area, see Figure 1B , Figure 1A The illustrated step 101 can be implemented through steps 1011 to 1014, which are described in detail below.

[0045] Step 1011, traverse the data types in the first attribute.

[0046] In some embodiments, the data type in the first attribute is determined when the data file is designed, and the data type in the first attribute describes the data type of each column of data in the data area, such as integer type, floating point type, string type, etc.; the data type in the first attribute can be traversed in the order of the data types described by the first attribute.

[0047] In some embodiments, each data type can be represented by N attribute elements, and the first attribute stores M attribute elements; N and M are both integers greater than 1, and N is less than M; see Figure 1C , Figure 1B The illustrated step 1011 can be implemented through steps 10111 to 10112, which are described in detail below.

[0048] Step 10111, taking N as the reading step length, read the attribute elements in the first attribute.

[0049] In some embodiments, the attribute element can be a descriptor of each data type in the first attribute. In the embodiment of the present application, N can be used as the second storage length to store the descriptor of each data type. In this way, the second storage length is used to determine the read step length to be read for each read operation when traversing the data types in the first attribute. The second storage length N can be the size of each data type descriptor in the first attribute, and the second storage length N also determines the step length of each advance of the reading program when traversing the first attribute. The read step length is the number of bytes or bits read by the reading program from the data file in a read operation. The stored data is the actual data in the first attribute, and the actual data can be composed of binary data.

[0050] For example, the first attribute is "0011011000010011", where the descriptors "0" and "1" are the attribute elements in the first attribute. The second storage length N can be equal to 4 bits, and "0011", "0110", "0001" and "0011" in the first attribute are read in sequence.

[0051] Step 10112: When each of N attribute elements is read, the data type represented by the N attribute elements read is determined as the traversed data type.

[0052] In some embodiments, the attribute elements in the first attribute may be binary data, and these binary data represent different data types. The mapping process is to determine these binary data as specific data types.

[0053] For example, the first attribute element of the second storage length N is "0011", and the preset key-value pair table can be queried to know that the data type mapped by "0011" is an integer type. The second attribute element of the second storage length N is "0110", and the preset key-value pair table can be queried to know that the data type mapped by "0110" is a string type. The third attribute element of the second storage length N is "0001", and the preset key-value pair table can be queried to know that the data type mapped by "0001" is a character type. The fourth attribute element of the second storage length N is "0011", and the preset key-value pair table can be queried to know that the data type mapped by "0011" is an integer type, and so on.

[0054] Through steps 10111 to 10112, using the second storage length N as a fixed reading step length can reduce frequent disk I / O operations or memory accesses that may be caused by uncertain reading step lengths, thereby improving the overall efficiency of data processing and efficiently traversing the attribute elements in the first attribute of the column; and through a mapping mechanism, the read storage data is accurately mapped to a preset data type. This mapping process based on a preset key-value pair table or other mapping rules can achieve accurate parsing and processing of different data types and structures, avoid storing data types with different storage lengths, and reduce unnecessary storage space occupancy and waste.

[0055] Next, see Figure 1B , and the description continues with step 1011 above.

[0056] Step 1012: When the traversed data type is a non-string type, obtain the first data length of the non-string type data.

[0057] In some embodiments, non-string data types are all other data types in the data file except string data types, such as integer data types, floating point data types, Boolean data types, and date and time data types. The non-string data types are characterized by having a fixed storage length, which is opposite to the non-fixed storage length of string data. The storage length of the non-string data type is the fixed storage length required for the non-string data type when storing. For example, the integer data type corresponds to 32 bits, and the character data type corresponds to 8 bits. Each time a non-string data type is traversed, the storage length corresponding to the non-string data type is a target addend.

[0058] Step 1013: When the traversed data type is a string type, obtain the second data length of the string pointer.

[0059] In some embodiments, the second data length of the string pointer is also stored in the file header. The second data length of the string pointer is a fixed size. For example, if the second data length of the string pointer is 8, then all string pointers have a fixed storage length of 8 bits. Every time a string type data type is traversed, the length of the string pointer is the fixed second data length.

[0060] Step 1014: perform a preset operation on the first data length and the second data length to obtain a first storage length.

[0061] In some embodiments, all first data lengths and second data lengths acquired in the previous steps may be accumulated to obtain the first storage length.

[0062] Through steps 1011 to 1014, an efficient and accurate first storage length calculation process is implemented, so that the starting storage position of any data record can be directly calculated through the first storage length without checking or parsing the boundaries of the data records one by one, which greatly speeds up the retrieval and access speed of data records and reduces the time delay of read and write operations.

[0063] Next, see Figure 1A , and the description will continue with step 101 above.

[0064] Step 102: Read a target data record of a first file from the data area according to a first storage length.

[0065] In some embodiments, the target data record is at least one specific data record that a user or program needs to retrieve and read from a data file. A data file may contain multiple data records, each of which may contain a series of data that may correspond to different data types and values. When a user is interested in certain specific data records, these specific data records that the user needs to analyze, process, or display are target data records.

[0066] In some embodiments, see Figure 1D , Figure 1A The illustrated step 102 can be implemented through steps 1021 to 1022, which are described in detail below.

[0067] Step 1021: Determine a storage start position and a storage end position in the data area according to the first storage length.

[0068] In some embodiments, the storage start position is the initial storage position of the target data record located in the data area, and the storage end position is the final storage position of the target data record located in the data area.

[0069] In some embodiments, see Figure 1E , Figure 1D The illustrated step 1021 may be implemented through steps 10211 to 10212, which are described in detail below.

[0070] Step 10211, obtaining the data record identifier of the target data record carried in the data read request.

[0071] In an embodiment of the present application, the data read request may carry a data record identifier of a target data record to be read. When determining the storage start position and the storage end position in the data area, the data read request may be parsed first, and the data record identifier may be obtained by parsing the data read request.

[0072] Step 10212, determining the storage start position and the storage end position according to the data record identifier and the first storage length.

[0073] In some embodiments, since the data records in the data area of ​​the first file are arranged in sequence, the data record identifier can be the number of data records in the data area of ​​the first file, and the data record identifier of the target data record can be the ranking value of the target data record among all data records in the entire data area.

[0074] For example, the data record identifier can be Q, that is, Q is the sorting value of the target data record in all data records, P=Q-1, then the storage position determined by the product of P and the first storage length can be determined as the storage starting position; accordingly, the previous adjacent storage position of the storage position determined by the product of Q and the first storage length in the data area is determined as the storage end position.

[0075] For example, if the first storage length is 100 bits, with the 0th bit in the data area as the starting point, and the target data record is the 10th data record in the data file (Q is 10, P is 9), then the storage starting point position can be calculated based on the first storage length as 100*9=900, and the starting bit position of the target data record in the data area is the storage position of the 900th bit; the storage ending point position can also be calculated based on the first storage length as 100*10-1=999, and the ending bit position of the target data record in the data area is the storage position of the 999th bit.

[0076] Through step 10211 to step 10212, the storage starting point position and the storage end point position are directly calculated without traversing the entire data area to find the target data record, ensuring that only the target data record is read or processed without misreading or interfering with adjacent data records. This can significantly reduce data access delays and reduce unnecessary memory and storage resource consumption, thereby improving data processing efficiency and computing resource utilization efficiency.

[0077] Next, see Figure 1D , and the description continues with step 1021 above.

[0078] Step 1022: determine the data record consisting of data in the range from the storage start position to the storage end position as the target data record.

[0079] In some embodiments, starting from the storage start position of the data area, all data from the storage start position to the storage end position can be read according to the storage order of the data, and these data together constitute a complete target data record.

[0080] Through step 1021 to step 1022, an efficient data positioning and data extraction process is formed, which directly locates the target data record and reads it, which helps to reduce unnecessary memory and storage resource consumption. There is no need to load or process data unrelated to the target data record, which can improve resource utilization efficiency.

[0081] Step 103 : when the data type of the target data in the target data record is a string pointer, determine a first storage position in the string area of ​​the first file according to the target data, and read the string data based on the first storage position.

[0082] In an embodiment of the present application, when reading target data in a target data record, if the data type of the read target data is a string pointer, the string data corresponding to the target data is read from the first storage location pointed to by the string pointer in the string area.

[0083] In some embodiments, target data refers to specific data values ​​that a user or program needs to extract from a target data record in a data file. In a data record, multiple columns of data may be included, and the target data is the data of certain specific columns that the user is interested in. A string pointer is an offset in a string area, pointing to the actual storage location of the string data in the string area; in the data record of the embodiment of the present application, the string data (i.e., string type data) is not directly stored in a column of the data type of the string type in the data area, but a string pointer pointing to the string data in the string area is stored in this column of the data type of the string type.

[0084] The storage location pointed to by the string pointer is the memory address or string area offset represented by the string pointer value. When the reading program encounters a string pointer, it uses this string pointer to find the exact location of the string data in the string area. The string data corresponding to the target data refers to the actual string data pointed to by the string pointer. After the reading program finds the first storage location of the string data according to the string pointer, it can start reading the character data from this first storage location in sequence until it encounters the termination character of the string (for example: "\0"), and then determine the string composed of all the read character data as the string data corresponding to the string pointer.

[0085] In some embodiments, see Figure 1F , Figure 1A The illustrated step 103 may be implemented through steps 1031 to 1033, which are described in detail below.

[0086] Step 1031, determining a starting character from the character string area according to the first storage location pointed to by the target data.

[0087] In some embodiments, the storage address information carried by the string pointer can be identified first, and this address information directly points to a specific storage location in the string area; then, based on this address information, positioning can be performed within the string area, and the character corresponding to the specific storage location can be used as the starting character of the string data.

[0088] For example, if the string pointer is 4 and the data in the string area is "This is an, exemplary data.", then the subscripts of the data in the string area are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 in sequence, and the character corresponding to the string pointer is "exemplary", so this "exemplary" is the starting character of the string data.

[0089] Step 1032, starting from the starting character, traverse the characters stored in the string area in sequence.

[0090] In some embodiments, starting from the starting character, the character data can be accessed and read one by one according to the storage order of the character data in the string area. The character data in the string area can be characters encoded in UTF-8 or binary encoded data.

[0091] Step 1033, in the case of traversing to the termination character, stop the traversal, and determine the starting character and the characters between the starting character and the termination character as the string data.

[0092] In some embodiments, during the traversal process, it can be detected whether the currently traversed character data is a termination character; the termination character is character data with a character termination identifier, and the termination character usually has one or more predefined features, such as specific ASCII code values, Unicode encodings, or other forms of identifiers, used to indicate the end of the string; when it is detected that the current character matches these predefined features, for example, when it is detected that the current character is the same as these predefined features, it is determined that the character is a termination character, and then the traversal process is immediately stopped, and the subsequent character data in the string area is no longer accessed.

[0093] For example, if the data in the string area is "This is an, exemplary data.", starting from the starting character "exemplary" to traverse the string area, if it is stipulated that "." is the termination character, then stop the traversal when "." is traversed.

[0094] In some embodiments, based on the starting character position where the starting character is located and the termination character position where the termination character is located determined in the previous steps, a range including multiple character data can be determined. This range includes the starting character itself and all the character data after the starting character and before the termination character. All the character data within this range can be regarded as a whole, that is, a complete string data.

[0095] For example, the data in the string area is "This is an exemplary piece of data.", starting from the starting character "exemplary" to traverse the string area, and stopping the traversal when reaching ".", then the string data is "exemplary data".

[0096] Through steps 1031 to 1033, directly locate the starting position of the string and only traverse to the termination character, avoiding unnecessary memory access and data processing, thereby improving the efficiency of string data processing, and precisely defining the boundary of the string, avoiding data reading errors or truncation.

[0097] In some embodiments, refer to Figure 1G , before step 102 shown in Figure 1A (where A in Figure 1G represents the above-mentioned step 101), the method further includes steps 201 to 203, which will be specifically described below.

[0098] Step 201, determine the storage length of the data area according to the first storage length and the third attribute.

[0099] Here, the third attribute is used to characterize the number of data records in the data area, and the third attribute can also be called the data record statistical attribute. In some embodiments, the third attribute in the file header stores the total number of data records in the data file. By multiplying the first storage length by the value in the third attribute, the storage length of the data area can be accurately calculated; since the third attribute is dynamically updated in the data file, that is, every time a new data record is added, the value of the third attribute is incremented by one, so the storage length of the data area calculated in step 201 represents the theoretical value of the storage length of the data area of the data file.

[0100] Step 202, determine the storage length to be verified of the first file according to the storage length of the data area, the fourth attribute, and the actual storage length of the file header of the first file.

[0101] Here, the fourth attribute is used to characterize the storage length of the string area, and the fourth attribute can also be called the string area length attribute. In some embodiments, the fourth attribute is dynamically updated in the data file, that is, every time a new string data is added, the value in the fourth attribute is correspondingly increased by the value of the storage length of the string data. The actual storage length of the file header is determined by reading the file header area in the data file. By adding the storage length of the data area, the value in the fourth attribute, and the actual storage length of the file header, the storage length to be verified of the first file to be read can be calculated, and this storage length to be verified is the theoretical value of the total storage length of the first file, which is used for subsequent comparison with the actual storage length of the file.

[0102] Step 203: When the storage length to be verified is not equal to the actual storage length of the first file, it is prohibited to determine the target data record of the first file from the data area.

[0103] In some embodiments, after calculating the storage length to be verified in step 202, the storage length to be verified can be compared with the actual storage length of the first file to be read; if the two are found to be unequal, it means that the first file has been tampered with or damaged during transmission or storage, and then security measures can be taken to prohibit further reading of target data records from the data area to prevent potential data errors or security risks.

[0104] Through steps 201 to 203, the storage length is verified before reading the data, so that potential problem files can be identified and eliminated in advance, avoiding unnecessary consumption of computing resources (such as CPU, memory and I / O operations) on invalid or damaged data.

[0105] In some embodiments, see Figure 1H ,exist Figure 1A Before step 101 shown, the method further includes steps 301 to 302, which are described in detail below.

[0106] Step 301, determining a serial number range of data records of each data file in a preset data file set; the serial numbers in the serial number range are data record identifiers used to distinguish different data records.

[0107] In some embodiments, the sequence number range is the range covered by the sequence number of the data record in each data file in the preset data file set; the data record sequence number range of each data file defines the range of the unique identifier of the data record in the file, so that different data records can be distinguished. The sequence number is an identifier used to uniquely identify each data record, and can be a continuous integer sequence.

[0108] For example, the serial number range of each data file can be determined based on the file name of each data file in the preset data file set. If the file names of all data files in the preset data file set are "0", "100", "200", "300" and "400", respectively, then the serial number ranges of the data records corresponding to these five data files can be determined to be [0,99], [100,199], [200,299], [300,399], [400,499] respectively.

[0109] Step 302, determining a data file whose sequence number range includes a preset target sequence number as a first file; wherein the target sequence number is the sequence number of the target data record.

[0110] In some embodiments, the target sequence number is the sequence number of a data record in a preset data file set that a user or program needs to read.

[0111] For example, the target sequence number is 150, and the sequence number ranges of the data records corresponding to the five data files are [0,99], [100,199], [200,299], [300,399], and [400,499], respectively. The second data file with a sequence number range of [100,199] can be determined as the first file to be read.

[0112] By determining the data record serial number range of each data file through steps 301 to 302, it is possible to quickly identify which data files contain data records with a specific serial number (i.e., the target serial number), so that when it is necessary to access a specific data record, the data file containing the data record can be directly located, avoiding unnecessary data file traversal and search, thereby significantly improving the efficiency of data access; and by matching the serial number range with the target serial number to determine the first file to be read, it is possible to ensure that only the file containing the target data record is accessed, which helps prevent unauthorized access and data leakage, enhances data security, and reduces the risk of data damage or loss due to misoperation or malicious attacks; it is also possible to reduce unnecessary resource consumption by only loading and processing files related to the target serial number, especially when processing a large number of data files, it is no longer necessary to use a large amount of computing resources (such as CPU, memory, and I / O bandwidth), which helps to optimize resource utilization and improve the overall performance and response speed of data reading.

[0113] Next, a data file proposed in an embodiment of the present application will be described.

[0114] The data file proposed in the embodiment of the present application can be stored in a binary storage mode. For example, the content in the data file is: 011110100110…011101, the first 32 bits of data represent the magic number attribute in the file header, the next 32 bits of data represent the version attribute in the file header, and so on. The encoding format of the string area of ​​the data file can be UTF-8.

[0115] In some embodiments, the storage structure of the data file is composed as follows: Figure 2As shown, the data file 500 includes a file header 510 (header), a data area 520 (data) and a string area 530 (stringTable). The data header 510 includes a magic number attribute 511 (Signature), a version attribute 512 (Version), a data record statistics attribute 513 (RecordSize) (i.e., a third attribute), a column number attribute 514 (FieldSize), a metadata attribute 515 (Metadata), a column type attribute 516 (ColumnMeta) (i.e., a first attribute) and a string area length attribute 517 (StringTableSize). The metadata attribute 515 includes a string data length attribute 5151 (StringPointer) (i.e., a second attribute) and a character compression type attribute 5152 (ZipType). The string data length attribute 5151 and the character compression type attribute 5152 can also be directly stored in the file header 510; the data area 520 can include multiple data records, each of which has the same storage length, and each data record can include multiple data of different data types. Figure 2 exemplarily shows data record 521, data record 522 and data record 523, and also exemplarily shows that data record 521 includes long integer data 521-1, integer data 521-2, integer data 521-3, string pointer 540-1 (pointing to string data 521-4), floating point data 521-5, Boolean data 521-6 and character data 521-7; string area 530 includes all string data, Figure 2 String data 521 - 4 , string data 522 - 4 , and string data 523 - 4 are exemplarily given, and each string data may end with “\0” (ie, a string termination mark). Figure 2 It is also exemplified that data record 522 includes long integer data 522-1, integer data 522-2, integer data 522-3, string pointer 540-2 (pointing to string data 522-4), floating point data 522-5, Boolean data 522-6 and character data 522-7; data record 523 includes long integer data 523-1, integer data 523-2, integer data 523-3, string pointer 540-3 (pointing to string data 523-4), floating point data 523-5, Boolean data 523-6 and character data 523-7.

[0116] The magic number attribute 511 is used to identify the type or format of the data file to ensure that the program that reads the file can correctly identify and process the file; the version attribute 512 is used to indicate the version of the data file, allowing the program to adjust the parsing logic based on the version information to be compatible with different versions of data files. The data record statistics attribute 513 records the total number of data records in the data file. This helps to determine the total size of the data area and locate a specific data record when reading the file. The column number attribute 514 indicates the number of columns (fields) contained in each data record. The metadata attribute 515 contains additional information about the data record, such as the string data length attribute 5151 and the character compression type attribute 5152. The storage length of the string data length attribute 5151 is 1 bit, indicating the storage length of the string pointer. For example, 0 represents that the storage length of the string pointer is 8 bits, and 1 represents that the storage length of the string pointer is 16 bits. The storage length of the character compression type attribute 5152 is 2 bits, indicating the character compression format of the string area. For example, 01 represents that the character compression format of the string area is gzip, 10 represents that the character compression format of the string area is snappy, 00 represents that the character compression format of the string area is lz4, and 11 represents that the character compression format of the string area is zstd.

[0117] The column type attribute 516 indicates the data type of each column (field) in each data record, and the data type can be byte type, integer type, long integer type, string type, single-precision floating point number, double-precision floating point number, etc. The string area length attribute 517 indicates the total space occupied by the string area, which can be used to verify the integrity of the string area.

[0118] The data file 500 of the embodiment of the present application adopts a self-defined data format, and the data file 500 does not rely on any complex third-party components, and can independently realize the reading and processing of data. All data records can be scattered and stored in multiple different data files to support the storage requirements of ultra-large data volumes. In addition, the data records are aligned when stored, so that the storage length of each data record is consistent, and arranged in a certain order, so that the data records can be quickly located to specific data records within the time complexity of O (logn) according to the binary search algorithm, greatly improving the efficiency of data search. And, because the structure and format of the data file have been solidified, the data file can be directly copied to complete the backup and transfer of all data records without complicated conversion or processing.

[0119] In some embodiments, when writing data to the data file 500 of the embodiment of the present application, the following optimization measures may be adopted:

[0120] (1) Before writing data to the data file 500 of this embodiment, a storage strategy for data records can be planned to store all data records in one or more data files 500. The data records in the same data file are arranged in ascending order by serial number, and the data file 500 is named using the smallest serial number of all data records in the current data file 500 for easy management and search.

[0121] (2) When writing data records to the data file 500, the data record statistical attribute 513 can be updated and the data records can be sorted (default ascending order) according to the data records to be stored to ensure the consistency and orderliness of the data. At the same time, the column number attribute 514 and the column type attribute 516 are initialized according to the column definition, and the storage length required for each data record (i.e., the first storage length) is calculated to optimize the use of storage space.

[0122] (3) The contents of the magic number attribute 511, the version attribute 512, and the metadata attribute 515 in the file header 510 may be reasonably set according to actual needs.

[0123] (4) According to the data type, write the data into the corresponding data record one by one. For string data, compression technology is used to reduce the storage space occupied. The compressed string data is written into the string area 530, and the "\0" character is used as the termination character of each string data. The string pointer in C++ automatically recognizes "\0" as the end of the string, which simplifies the storage and retrieval of the string.

[0124] (5) A string pointer pointing to the starting position of the string data is recorded in the data area 520, rather than directly storing the string data itself, so as to optimize the storage efficiency of the data area 520. At the same time, as the string data is written, the string area length attribute 517 is dynamically updated to ensure that the string area length attribute 517 reflects the latest storage length of the string area 530.

[0125] (6) The data in the string area 530 can be temporarily stored in a temporary file. After the writing of the data area 520 is completed, all the string data are uniformly written into the string area 530 to realize the complete construction of the data file 500. After the writing of the string area is completed, the string area length attribute 517 is rechecked and updated to ensure that the string area length attribute 517 accurately reflects the actual size of the string area.

[0126] Through the above-mentioned optimized data writing measures, the data file 500 of this embodiment not only improves the utilization efficiency of the storage space, but also simplifies the complexity of data retrieval and reading, and provides a solid foundation for subsequent data management and analysis.

[0127] In some embodiments, when reading data from the data file 500 of the embodiment of the present application, the following optimization measures may be adopted:

[0128] (1) When starting the reading process, first read the file header 510, and determine the storage length to be verified of the data file 500 based on the storage length of the data area 520, the string area length attribute 517 in the file header 510, and the actual storage length of the file header 510. When the storage length to be verified is equal to the actual storage length of the data file 500, proceed with the subsequent data reading process.

[0129] (2) For the data in each data record, a differentiated reading strategy can be adopted according to the data type: when the data type is a string type, the stored data itself is no longer read directly, but the stored string pointer is used to directly locate the exact storage location of the string data in the string area 530; then the string data is decompressed according to the previously defined compression method (the compression method defined in the character compression type attribute 5152) to restore the original content, thereby achieving efficient data record reading.

[0130] (3) When searching based on the serial number of a data record, first use the binary search method to quickly locate the location of the data record in the data file set; that is, when the target serial number is between the name serial number of the current data file and the name serial number of the next data file, it can be determined that the target data record exists in the current data file. The time complexity of this process is O(log n), which can significantly improve the efficiency of searching for specific data records.

[0131] (4) After determining the target data file to be read, due to the orderliness and alignment characteristics of the data records in the data area 520, the binary search method can be used again to directly locate the specific data record storage location according to the serial number of the data record, and the time complexity of O (log n) is also achieved, ensuring the high-speed response of the entire data retrieval process.

[0132] Through the above optimization measures, not only the accuracy of data reading is guaranteed, but also the efficiency of data retrieval is optimized. It is especially suitable for efficient management and query of large-scale data sets, providing users with a smoother data access experience.

[0133] The following is a description of an exemplary structure of the data reading device 445 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 3As shown, the software modules in the data reading device 445 may include: an acquisition module 4551, used to determine a first storage length according to a first attribute and a second attribute of the first file; wherein the first storage length is the storage length of each data record in the data area of ​​the first file; a reading module 4552, used to read a target data record of the first file from the data area according to the first storage length; the reading module is also used to determine a first storage position in the string area of ​​the first file according to the target data when the data type of the target data in the target data record is a string pointer, and read the string data based on the first storage position.

[0134] In some embodiments, the reading module 4552 is also used to determine a starting character from the string area according to the first storage position pointed to by the target data; starting from the starting character, traverse the characters stored in the string area in order; when traversing to the ending character, stop traversing, and determine the starting character and the characters between the starting character and the ending character as string data.

[0135] In some embodiments, the first attribute is used to characterize the data type of each data included in each data record in the data area, and the second attribute is used to characterize the storage length of the string pointer in the data area; the reading module 4552 is also used to traverse the data type in the first attribute; when the traversed data type is non-string type, obtain the first data length of the non-string data; when the traversed data type is string type, obtain the second data length of the string pointer; perform preset operations on the first data length and the second data length to obtain the first storage length.

[0136] In some embodiments, each data type is represented by N attribute elements, and M attribute elements are stored in the first attribute; N and M are both integers greater than 1, and N is less than M; the reading module 4552 is also used to read the attribute elements in the first attribute with N as the reading step; every time N attribute elements are read, the data type represented by the N attribute elements read is determined as the traversed data type.

[0137] In some embodiments, the reading module 4552 is also used to determine the storage starting position and the storage ending position in the data area according to the first storage length; and determine the data record consisting of data in the range from the storage starting position to the storage ending position as the target data record.

[0138] In some embodiments, the reading module 4552 is further used to obtain a data record identifier of a target data record carried in a data reading request; and determine a storage start position and a storage end position according to the data record identifier and the first storage length.

[0139] In some embodiments, the device also includes a data verification module, which is used to determine the storage length of the data area based on the first storage length and the third attribute; wherein the third attribute is used to characterize the number of data records in the data area; based on the storage length of the data area, the fourth attribute and the actual storage length of the file header of the first file, determine the storage length to be verified of the first file; wherein the fourth attribute is used to characterize the storage length of the string area; when the storage length to be verified is not equal to the actual storage length of the first file, it is prohibited to determine the target data record of the first file from the data area.

[0140] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0141] Correspondingly, an embodiment of the present application provides an electronic device for implementing the data reading method in the above embodiment. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the electronic device 1200 includes at least: a processor 1201 , a communication interface 1202 , and a storage medium 1203 configured to store executable instructions, wherein: the processor 1201 generally controls the overall operation of the electronic device 1200 .

[0142] The communication interface 1202 enables the electronic device to communicate with other terminals or servers through a network.

[0143] The storage medium 1203 is configured to store instructions and applications executable by the processor 1201, and can also cache data to be processed or processed by the processor 1201 and each module in the electronic device 1200, which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0144] An embodiment of the present application provides a data file structure, which includes a file header, a data area and a string area; the file header includes a first attribute and a second attribute; the data area includes a data record; the data record includes at least a string pointer; the string area includes string data; the string pointer points to the first storage position of the string data in the string area.

[0145] In some embodiments, the first attribute includes at least one preset data type.

[0146] In some embodiments, the file header also includes data record statistics attributes and string area length attributes.

[0147] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data reading method described in the embodiment of the present application.

[0148] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the data reading method provided by the present application embodiment, for example, Figure 1A The data reading method is shown.

[0149] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0150] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0151] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0152] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0153] In summary, on the one hand, in the embodiment of the present application, when the target data is a string pointer, the string data corresponding to the target data is read from the storage position pointed to by the string pointer in the string area, so that when the length of the string data itself is different, the data records in the data area can also maintain the same storage length, and the storage length (first storage length) of each data record can be clearly determined according to the first attribute and the second attribute in the file header, so when reading the target data record, the storage position of the target data record can be directly determined according to the first storage length, thereby saving the calculation cost when reading data and improving the efficiency of data reading. On the other hand, the embodiment of the present application stores the string data in an orderly and continuous manner in the string area, rather than in the data area, which can avoid the waste of storage space caused by the difference in the storage length of the string data, so that storage resources are more efficiently and reasonably used.

[0154] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data reading method, characterized in that: The method comprises: Determine a first storage length according to a first attribute and a second attribute of the first file; wherein the first storage length is the storage length of each data record in the data area of ​​the first file; Reading a target data record of the first file from the data area according to the first storage length; In the case where the data type of the target data in the target data record is a character string pointer, a first storage position in the character string area of ​​the first file is determined according to the target data, and the character string data is read based on the first storage position.

2. The method according to claim 1, characterized in that The determining a first storage location in a string area of ​​the first file according to the target data, and reading string data based on the first storage location, comprises: Determining a starting character from a character string area according to a first storage location pointed to by the target data; Starting from the starting character, traverse the characters stored in the string area in order; When the traversal reaches the termination character, the traversal is stopped, and the starting character and the characters between the starting character and the termination character are determined as character string data.

3. The method according to claim 1, characterized in that The first attribute is used to characterize the data type of each data included in each data record in the data area, and the second attribute is used to characterize the storage length of the character string pointer in the data area; The step of determining the first storage length according to the first attribute and the second attribute of the first file includes: Traversing the data types in the first attribute; When the traversed data type is a non-string type, obtain the first data length of the non-string type data; When the traversed data type is a string type, obtaining a second data length of the string pointer; A preset operation is performed on the first data length and the second data length to obtain the first storage length.

4. The method according to claim 3, characterized in that Each data type is represented by N attribute elements, and the first attribute stores M attribute elements; N and M are both integers greater than 1, and N is less than M; The traversing the data type in the first attribute includes: Taking N as the reading step length, read the attribute elements in the first attribute; Every time N attribute elements are read, the data types represented by the read N attribute elements are determined as the traversed data types.

5. The method according to claim 3, characterized in that: The step of reading the target data record of the first file from the data area according to the first storage length includes: Determining a storage start position and a storage end position in the data area according to the first storage length; A data record consisting of data within the range from the storage start position to the storage end position is determined as the target data record.

6. The method according to claim 5, characterized in that The step of determining a storage start position and a storage end position in the data area according to the first storage length includes: Obtaining a data record identifier of a target data record carried in a data read request; A storage start position and a storage end position are determined according to the data record identifier and the first storage length.

7. The method according to any one of claims 1 to 6, characterized in that: Before reading the target data record of the first file from the data area according to the first storage length, the method further includes: Determine the storage length of the data area according to the first storage length and a third attribute; wherein the third attribute is used to characterize the number of data records in the data area; Determine the storage length of the first file to be verified according to the storage length of the data area, the fourth attribute and the actual storage length of the file header of the first file; wherein the fourth attribute is used to characterize the storage length of the string area; In a case where the storage length to be verified is not equal to the actual storage length of the first file, determining the target data record of the first file from the data area is prohibited.

8. A data reading device, characterized in that: The device comprises: A determination module, configured to determine a first storage length according to a first attribute and a second attribute of a first file; wherein the first storage length is a storage length of each data record in a data area of ​​the first file; A reading module, configured to read a target data record of the first file from the data area according to the first storage length; The reading module is further configured to determine a first storage position in the string area of ​​the first file according to the target data when the data type of the target data in the target data record is a string pointer, and read the string data based on the first storage position.

9. An electronic device, characterized in that: include: A memory for storing computer executable instructions; A processor, configured to implement the data reading method according to any one of claims 1 to 7 when executing the computer executable instructions stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data reading method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data reading method according to any one of claims 1 to 7 is implemented.

Images