Method, device, electronic equipment and storage medium for data storage

Abstract

The application provides a data storage method, device, electronic equipment and storage medium. In the data storage method, the file number and the connection key of the data file to be stored are first obtained. For each data file, the data in the data file is classified according to each connection key to obtain the data corresponding to each connection key. Then, for each data file, the data corresponding to the same connection key in the data file is stored in the same data block to obtain each data block of the data file. Finally, based on the file number and the connection key of each data block, each data block is allocated to a data node for storage. Therefore, by using the method of the application, the data blocks associated with each other are stored in the same data node, so that the database query connection operation can be completed directly through local data transmission when the database query connection is performed.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] HDFS (Hadoop Distributed File System) is a core component of the Hadoop distributed system architecture, providing massive data storage capabilities by building a cluster of multiple machines. An HDFS cluster uses a master-slave architecture, typically consisting of one NameNode (management node) and several DataNodes. The management node is responsible for maintaining the entire file system structure and managing other data nodes, while the data nodes are responsible for the actual data storage.

[0003] In the HDFS distributed file system, data storage employs a "divide and conquer" approach to achieve the goal of "multiplying small amounts into larger ones." This involves prioritizing storage on data nodes with high idle rates and distributing them across these nodes. Because data blocks are scattered across different nodes, during database query joins, related data from different nodes needs to be transmitted over the network to the same data node to complete the database query join operation. This requires a significant amount of network transmission, consuming considerable time and network resources, thus greatly reducing the overall query efficiency of the database join. Summary of the Invention

[0004] In view of this, this application provides a data storage method, apparatus, electronic device, and storage medium to solve the problem in the prior art that a large amount of network transmission is required during the database query connection process, which consumes a lot of time and network resources, thus greatly reducing the query efficiency of the entire database query connection.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] The first aspect of this application discloses a data storage method applied to a distributed file system, wherein the data storage method includes:

[0007] Obtain the file number and join key of the data file to be stored; wherein, the file number is obtained by numbering the data files in advance according to a preset numbering rule, and related data files have the same file number; the join key is a field used to query and join data files;

[0008] For each data file, the data in the data file is classified according to each connection key to obtain the data corresponding to each connection key;

[0009] For each data file, the data corresponding to the same connection key in the data file is stored in the same data block to obtain the data blocks of the data file; wherein, each data block is identified by the connection key and the file number;

[0010] Based on the file number and the connection key of each data block, each data block is allocated to a data node for storage.

[0011] Optionally, in the above method, classifying the data in the data file according to each of the connection keys to obtain the data corresponding to each connection key includes:

[0012] The data file is partitioned according to the connection key to obtain the data corresponding to each connection key.

[0013] Optionally, in the above method, the step of allocating each data block to data nodes for storage based on the file number and the connection key of each data block includes:

[0014] Each data block is classified according to its file number and connection key, and data blocks with the same file number and connection key are selected.

[0015] Data blocks with the same file number and connection key are allocated to the same data node for storage.

[0016] Optionally, the above method, after allocating each data block to data nodes for storage based on the file number and the connection key of each data block, further includes:

[0017] Generate storage logs for data storage.

[0018] A second aspect of this application discloses a data storage apparatus applied to a distributed file system, wherein the data storage apparatus comprises:

[0019] The acquisition unit is used to acquire the file number and connection key of the data file to be stored; wherein, the file number is obtained by numbering the data files in advance according to a preset numbering rule, and related data files have the same file number; the connection key is a field used to query and join data files;

[0020] A classification unit is used to classify the data in each data file according to each connection key, so as to obtain the data corresponding to each connection key;

[0021] A storage unit is used to store the data corresponding to the same connection key in each data file into the same data block to obtain the data blocks of the data file; wherein each data block is identified by the connection key and the file number;

[0022] The allocation unit is used to allocate each data block to a data node for storage based on the file number and the connection key of each data block.

[0023] Optionally, in the above-described apparatus, the classification unit includes:

[0024] The partitioning subunit is used to partition the data file according to the connection key to obtain the data corresponding to each connection key.

[0025] Optionally, in the above-described apparatus, the distribution unit includes:

[0026] The filtering subunit is used to classify each data block according to the file number and the connection key, and filter out the data blocks that have the same file number and connection key;

[0027] The allocation subunit is used to allocate data blocks with the same file number and the same connection key to the same data node for storage.

[0028] Optionally, the above-mentioned apparatus further includes:

[0029] The log generation unit is used to generate storage logs for data storage.

[0030] A third aspect of this application discloses an electronic device, comprising:

[0031] One or more processors;

[0032] A storage device on which one or more programs are stored;

[0033] When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of the first aspects of this application.

[0034] The fourth aspect of this application discloses a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any one of the first aspects of this application.

[0035] As can be seen from the above technical solution, the data storage method provided in this application first obtains the file number and connection key of the data file to be stored. The file number is obtained by pre-numbering the data files according to a preset numbering rule, and related data files have the same file number. The connection key is a field used for querying and connecting data files. For each data file, the data in the data file is classified according to each connection key, obtaining the data corresponding to each connection key. Then, for each data file, the data corresponding to the same connection key is stored in the same data block, obtaining the data blocks of the data file. Each data block is identified by the connection key and the file number. Finally, based on the file number and connection key of each data block, each data block is allocated to data nodes for storage. Therefore, using the method of this application, when storing data, the data in the data file is classified according to the connection key to obtain the data corresponding to each connection key, and the data corresponding to the same connection key is stored in the same data block. Then, the data blocks are allocated to data nodes according to the file number and connection key, so that related data blocks are stored in the same data node. Therefore, when performing database query connections, the database query connection operation can be completed directly through local data transfer. This solves the problem in existing technologies where a large amount of network transmission is required during the database query connection process, consuming a lot of time and network resources, which greatly reduces the query efficiency of the entire database query connection. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating a data storage method disclosed in an embodiment of this application;

[0038] Figure 2 This is an example diagram of data storage disclosed in another embodiment of this application;

[0039] Figure 3 This is a schematic diagram of a data storage device disclosed in another embodiment of this application;

[0040] Figure 4 This is a schematic diagram of an electronic device disclosed in another embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0043] Furthermore, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0044] As the background technology indicates, the HDFS distributed file system uses a "divide and conquer" approach to data storage, aiming to achieve the goal of "multiplying small amounts into larger ones." This involves prioritizing storage on data nodes with high idle rates and distributing them across these nodes. Because the data blocks are scattered across different nodes, during database query joins, the related data from different nodes needs to be transmitted over the network to the same data node to complete the database query join operation. This requires a significant amount of network transmission, consuming considerable time and network resources, thus greatly reducing the overall query efficiency of the database join.

[0045] In view of this, this application provides a data storage method, apparatus, electronic device, and storage medium to solve the problem in the prior art that a large amount of network transmission is required during the database query connection process, which consumes a lot of time and network resources, thus greatly reducing the query efficiency of the entire database query connection.

[0046] The first aspect of this application provides a data storage method applied to a distributed file system, wherein the data storage method includes:

[0047] S101. Obtain the file number and connection key of the data file to be stored; wherein, the file number is obtained by numbering the data file in advance according to the preset numbering rules, and related data files have the same file number; the connection key is a field used to query and connect data files.

[0048] It's important to note that during data storage, the first step is to obtain the file number and join key of the data file to be stored. The file number is pre-assigned according to a preset numbering rule, and related data files share the same file number. For example, when uploading table files or creating new tables, a location number is assigned to these tables to identify several tables that need to be distributed along the same node. For instance, if table A (course table) and table B (course selection table) are related or may require a join operation, then both table A and table B can be numbered "001". The join key is the field used for querying and joining data files. For example, in the aforementioned course table and course selection table, the course table contains two fields: course number and course name, and the course selection table contains two fields: student name and course number. When it's necessary to count which courses each student has selected, a database query join operation is required. When performing a database query join operation, the join condition must be specified first. The course number in the course table is equal to the course number in the course selection table; therefore, the course numbers in tables A and B are the join key for the query join.

[0049] S102. For each data file, classify the data in the data file according to each connection key to obtain the data corresponding to each connection key.

[0050] It should be noted that, therefore, for each data file to be stored, the data in the data file is classified according to each connection key, thereby obtaining the data corresponding to each connection key.

[0051] Optionally, in another embodiment of this application, one implementation of step S102 may include:

[0052] The data file is partitioned according to the join key to obtain the data corresponding to each join key.

[0053] It's important to note that when using a Hive data warehouse, partitioned tables are typically used, meaning the table is partitioned based on a specific field such as "time". Therefore, the data file is partitioned according to the join key, resulting in data corresponding to each join key. The partitioning rules can be determined using hash values; this example uses hash values ​​based on the join key, ensuring that each individual data block stores the data corresponding to the join key.

[0054] S103. For each data file, store the data corresponding to the same connection key in the data file into the same data block to obtain the data blocks of the data file; wherein, each data block is identified by the connection key and the file number.

[0055] It's important to note that because data within HDFS (Hadoop Distributed File System) is organized and stored in data blocks, whether uploading data files or importing data into tables, the data is ultimately divided into blocks and stored on various data nodes. Therefore, for each data file, when storing data into data blocks, data corresponding to the same join key within the data file are grouped into the same data block, resulting in the various data blocks of the data file. In this way, each individual data block stores data corresponding to the same join key. Each data block is identified by its join key and file number for easy identification.

[0056] S104. Based on the file number and connection key of each data block, allocate each data block to a data node for storage.

[0057] It's important to note that after generating each data block, based on the file number and join key of each data block, the data blocks are allocated to different data nodes for storage. Data blocks with the same file number and join key are stored on the same data node. This way, when an analysis scenario requires a query join between tables A and B, since the data in tables A and B that meets the query join conditions are all on the same data node, this query join operation can be completed during the local node mapping phase. This avoids data transfer between different data nodes from the mapping phase to the reduction phase, saving network consumption in the cluster and improving the overall performance of HDFS (Hadoop Distributed File System) when performing data queries. For details, please refer to [link to relevant documentation]. Figure 2 A1 and B1 represent data blocks with the same file number and connection key.

[0058] Optionally, in another embodiment of this application, one implementation of step S104 may include:

[0059] The data blocks are categorized according to their file number and link key, and the data blocks with the same file number and link key are selected.

[0060] Data blocks with the same file number and link key are allocated to the same data node for storage.

[0061] It should be noted that when storing data blocks, the data blocks are first classified according to their file number and join key, and data blocks with the same file number and join key are selected. Then, data blocks with the same file number and join key are allocated to the same data node for storage.

[0062] Optionally, in another embodiment of this application, after performing step S104, the following may also be included:

[0063] Generate storage logs for data storage.

[0064] It should be noted that after allocating each data block to the data nodes for storage, a storage log for this data storage is generated, recording detailed information about the data storage for easy data retrieval later.

[0065] This application provides a data storage method that first obtains the file number and join key of the data file to be stored. The file number is pre-numbered according to a preset numbering rule, and related data files share the same file number. The join key is a field used for querying and joining data files. For each data file, the data is categorized according to each join key, obtaining the data corresponding to each join key. Then, for each data file, the data corresponding to the same join key is stored in the same data block, resulting in data blocks for the data file. Each data block is identified by its join key and file number. Finally, based on the file number and join key of each data block, the data blocks are allocated to data nodes for storage. Therefore, using this method, when storing data, the data in the data file is categorized according to the join key, obtaining the data corresponding to each join key, and the data corresponding to the same join key is stored in the same data block. Then, the data blocks are allocated to data nodes according to the file number and join key, ensuring that related data blocks are stored on the same data node. Thus, when performing database query connections, the database query connection operation can be completed directly through local data transfer. This solves the problem in existing technologies where a large amount of network transmission is required during the database query connection process, consuming a lot of time and network resources, which greatly reduces the query efficiency of the entire database query connection.

[0066] Another embodiment of this application also provides a data storage device applied to a distributed file system, such as... Figure 3 As shown, it includes:

[0067] The acquisition unit 301 is used to acquire the file number and connection key of the data file to be stored; wherein, the file number is obtained by numbering the data file in advance according to a preset numbering rule, and related data files have the same file number; the connection key is a field used to query and connect the data files.

[0068] The classification unit 302 is used to classify the data in each data file according to each connection key, so as to obtain the data corresponding to each connection key.

[0069] Storage unit 303 is used to store the data corresponding to the same connection key in each data file into the same data block to obtain the data blocks of the data file; wherein each data block is identified by the connection key and the file number.

[0070] The allocation unit 304 is used to allocate each data block to a data node for storage based on the file number and connection key of each data block.

[0071] In this embodiment, the specific execution process of the acquisition unit 301, classification unit 302, storage unit 303, and allocation unit 304 can be found in the corresponding... Figure 1 The specific implementation details of the method are not repeated here.

[0072] In a data storage apparatus provided in this application, the acquisition unit 301 first acquires the file number and link key of the data file to be stored; wherein, the file number is obtained by pre-numbering the data files according to a preset numbering rule, and related data files have the same file number; the link key is a field used for querying and joining data files. For each data file, the classification unit 302 classifies the data in the data file according to each link key, obtaining the data corresponding to each link key. Then, for each data file, the storage unit 303 stores the data corresponding to the same link key in the data file into the same data block, obtaining the data blocks of the data file; wherein, each data block is identified by the link key and the file number. Finally, the allocation unit 304 allocates each data block to data nodes for storage based on the file number and link key of each data block. Therefore, the method of this application, when storing data, classifies the data in the data file according to the join key, obtains the data corresponding to each join key, and stores the data corresponding to the same join key in the same data block. Then, the data blocks are allocated to data nodes according to the file number and join key, ensuring that related data blocks are stored on the same data node. Thus, when performing database query joins, the database query join operation can be completed directly through local data transfer. This solves the problem in existing technologies where a large amount of network transmission is required during the database query join process, consuming significant time and network resources, which greatly reduces the query efficiency of the entire database query join.

[0073] Optionally, in another embodiment of this application, one implementation of the above-mentioned classification unit 302 includes:

[0074] The partitioning sub-unit is used to partition the data file according to the connection key, so as to obtain the data corresponding to each connection key.

[0075] In this embodiment, the specific execution process of the partition subunit can be found in the corresponding method embodiment described above, and will not be repeated here.

[0076] Optionally, in another embodiment of this application, one implementation of the allocation unit 304 includes:

[0077] The filtering sub-unit is used to classify and process each data block according to its file number and connection key, and to filter out data blocks that have the same file number and connection key.

[0078] The allocation subunit is used to allocate data blocks with the same file number and connection key to the same data node for storage.

[0079] In this embodiment, the specific execution process of the filtering sub-unit and the allocation sub-unit can be found in the corresponding method embodiment above, and will not be repeated here.

[0080] Optionally, in another embodiment of this application, the above-mentioned data storage device may further include:

[0081] The log generation unit is used to generate storage logs for data storage.

[0082] In this embodiment, the specific execution process of the log generation unit can be found in the corresponding method embodiment described above, and will not be repeated here.

[0083] Another embodiment of this application also provides an electronic device, such as... Figure 4 As shown, it specifically includes:

[0084] One or more processors 401.

[0085] Storage device 402, on which one or more programs are stored.

[0086] When one or more programs are executed by one or more processors 401, the one or more processors 401 implement any of the methods described in the above embodiments.

[0087] Another embodiment of this application also provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the methods described in the above embodiments.

[0088] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0089] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0090] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for data storage, characterized in that, Applied to a distributed file system, wherein the data storage method includes: Obtain the file number and join key of the data file to be stored; wherein, the file number is obtained by numbering the data files in advance according to a preset numbering rule, and related data files have the same file number; the join key is a field used to query and join data files; For each data file, the data in the data file is classified according to each connection key to obtain the data corresponding to each connection key; For each data file, the data corresponding to the same connection key in the data file is stored in the same data block to obtain the data blocks of the data file; wherein, each data block is identified by the connection key and the file number; Based on the file number and the connection key of each data block, each data block is allocated to a data node for storage; The step of allocating each data block to a data node for storage based on the file number and the connection key of each data block includes: Each data block is classified according to its file number and connection key, and data blocks with the same file number and connection key are selected. Data blocks with the same file number and connection key are allocated to the same data node for storage.

2. The method according to claim 1, characterized in that, The step of classifying the data in the data file according to each of the connection keys to obtain the data corresponding to each connection key includes: The data file is partitioned according to the connection key to obtain the data corresponding to each connection key.

3. The method according to claim 1, characterized in that, After allocating each data block to a data node for storage based on the file number and the connection key of each data block, the method further includes: Generate storage logs for data storage.

4. A data storage device, characterized in that, An apparatus for storing data in a distributed file system includes: The acquisition unit is used to acquire the file number and connection key of the data file to be stored; wherein, the file number is obtained by numbering the data files in advance according to a preset numbering rule, and related data files have the same file number; the connection key is a field used to query and join data files; A classification unit is used to classify the data in each data file according to each connection key, so as to obtain the data corresponding to each connection key; A storage unit is used to store the data corresponding to the same connection key in each data file into the same data block to obtain the data blocks of the data file; wherein each data block is identified by the connection key and the file number; An allocation unit is used to allocate each data block to a data node for storage based on the file number and the connection key of each data block; The allocation unit includes: The filtering subunit is used to classify each data block according to the file number and the connection key, and filter out the data blocks that have the same file number and connection key; The allocation subunit is used to allocate data blocks with the same file number and the same connection key to the same data node for storage.

5. The apparatus according to claim 4, characterized in that, The classification unit includes: The partitioning subunit is used to partition the data file according to the connection key to obtain the data corresponding to each connection key.

6. The apparatus according to claim 4, characterized in that, Also includes: The log generation unit is used to generate storage logs for data storage.

7. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 3.

8. A computer storage medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 3.

Images