A data storage method, system, electronic device, and storage medium

Abstract

This invention provides a data storage method, system, electronic device, and storage medium. When a client receives a ticket from an authorization server, it establishes a connection with the HDFS system and sends a remote procedure access request to the HDFS system. When the HDFS system determines that a file matching the name of the file to be migrated exists in the remote access request, it obtains information on at least one data node corresponding to the file to be migrated and determines the shortest target link between the target data node and the client for each data node. The client communicates with the corresponding target data node according to the target link, obtains the data blocks corresponding to the file to be migrated from the target data node, and transmits the data blocks to a Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system. This invention can solve the problems of operational risks and increased operating costs that easily arise during data migration.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a data storage method, system, electronic device, and storage medium. Background Technology

[0002] With societal development and the accumulation of time, a vast amount of information has been generated, typically stored digitally on various devices. Due to the different application scenarios for different data, the required storage time, query speed, update speed, and other related requirements also vary significantly. Therefore, data that requires frequent querying or modification can be stored in OLTP (On-Line Transaction Processing) database systems that provide fast query and update operations, while data that is not modified and is used for statistical analysis can be stored in OLAP (On-Line Analytical Processing) data warehouses and big data systems.

[0003] However, most of the current data warehouses and big data systems use solid-state drives (SSDs) and hard disk drives (HDDs) as their underlying storage media. These have a relatively short lifespan and are not suitable for long-term data storage. The stored data needs to be migrated to new storage devices each time it is close to its expiration date. This data migration process can easily lead to operational risks and increase corresponding operating costs. Summary of the Invention

[0004] In view of this, the present invention provides a data storage method, system, electronic device and storage medium to solve the problem that existing storage devices have limited storage lifespans, requiring data to be migrated to new storage devices each time the expiration date approaches, which leads to operational risks and increased operating costs during the data migration process.

[0005] The first aspect of this invention discloses a data storage method applied to a client, the method comprising:

[0006] Upon receiving a ticket from the authorization server, a connection is established with the HDFS system, and a remote procedure access request is sent to the HDFS system. When the HDFS system determines that a file exists matching the name of the file to be migrated in the remote access request, it obtains at least one data node information corresponding to the file to be migrated, and determines the target link with the shortest distance between the target data node corresponding to each data node and the client. The ticket is returned by the authorization server when it confirms the client's legitimacy based on the verification information sent by the client; the verification information is generated by the client based on the authorization ticket request sent by the authentication server.

[0007] The system communicates with the corresponding target data node according to the target link, obtains the data block corresponding to the file to be migrated from the target data node, and transmits the data block to the Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system; wherein, the file to be migrated is data that does not need to be modified, has non-real-time query requirements, and has long storage time requirements.

[0008] A data storage system is disclosed in a second aspect of the present invention, the data storage system being applied to a client, the system comprising:

[0009] The remote procedure access request sending unit is used to establish a connection with the HDFS system when receiving a ticket from the authorization server, and send a remote procedure access request to the HDFS system. This allows the HDFS system to determine if a file exists that matches the name of the file to be migrated in the remote access request, obtain at least one data node information corresponding to the file to be migrated, and determine the target link with the shortest distance between the target data node corresponding to each data node and the client. The ticket is returned by the authorization server when it confirms the client's legitimacy based on the verification information sent by the client. The verification information is generated by the client through the verification information generation unit based on the authorization ticket request sent by the authentication server.

[0010] The transmission unit is used to communicate with the corresponding target data node according to the corresponding target link, obtain the data block corresponding to the file to be migrated from the target data node, and transmit the data block to the Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system; wherein, the file to be migrated is data that does not need to be modified, has a non-real-time query requirement, and has a long storage time requirement.

[0011] A third aspect of the present invention provides an electronic device, comprising: a processor and a memory, the processor and the memory being connected via a communication bus; wherein the processor is configured to call and execute a program stored in the memory; and the memory is configured to store the program, the program being configured to implement the data storage method disclosed in the first aspect of the present invention.

[0012] A fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the data storage method disclosed in the first aspect of the present invention.

[0013] This invention provides a data storage method, system, electronic device, and storage medium. When a client receives a ticket from an authorization server, it establishes a connection with the HDFS system and sends a remote procedure access request to the HDFS system. When the HDFS system determines that a file matching the name of the file to be migrated exists in the remote access request, it obtains at least one data node information corresponding to the file to be migrated and determines the target link with the shortest distance between the target data node and the client for each data node. The ticket is returned by the authorization server when it confirms the client's legitimacy based on the verification information sent by the client. The verification information is generated by the client based on the authorization ticket request sent by the client by the authentication server. For each target data node, the client communicates with the target data node according to the corresponding target link, obtains the data block corresponding to the file to be migrated from the target data node, and transmits the data block to a Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system. The file to be migrated is data that does not require modification, has non-real-time query requirements, and requires long storage time. The technical solution provided by this invention establishes a connection between the HDFS system and the Blu-ray storage system in advance. Then, through the client, files in the HDFS system that do not need to be modified, have non-real-time query requirements, and have long storage time requirements can be migrated to the Blu-ray storage system for storage. This solves the problem that existing storage devices have storage lifespans, which require data to be migrated to new storage devices every time the expiration date approaches. This can lead to operational risks and increased operating costs during the data migration process. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0015] Figure 1 This invention provides an architecture diagram of an HDFS system in a Hadoop cluster and a data warehouse interface with a Blu-ray storage system, as provided in an embodiment of the invention.

[0016] Figure 2 A flowchart illustrating a data storage method provided in an embodiment of the present invention;

[0017] Figure 3 A flowchart illustrating another data storage method provided in an embodiment of the present invention;

[0018] Figure 4A schematic diagram of a data storage system provided in an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0021] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0022] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0023] It should be noted that the terms "a" and "a plurality of" used in this invention disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0024] Hadoop: An open-source distributed computing and storage big data framework.

[0025] Greenplum: A data warehouse that provides data storage and retrieval capabilities.

[0026] S3: Simple Object Storage Service. Files are stored as objects, which are stored in buckets. Information about the storage is retrieved via URLs.

[0027] HDFS: A distributed file system, an application within a big data framework. The namenode is the master node, providing metadata information for file storage, while the datanodes are the worker nodes, storing the actual file data blocks.

[0028] access_key and secret_key: keys for a type of encryption algorithm.

[0029] Kerberos: An authentication algorithm where principal can be understood as username and keytab as password.

[0030] Blu-ray storage systems are highly reliable online storage devices that use Blu-ray discs as the storage medium. They consist of four parts: a robotic arm, an optical drive, a disc tray, and a management system. They are primarily used for online automatic recording, storage, retrieval, backup, and recovery of data. Using Blu-ray discs as the underlying medium, Blu-ray storage systems are characterized by long lifespan, low power consumption, and resistance to electromagnetic interference, making them ideal for long-term data retention.

[0031] See Figure 1 This paper illustrates an architecture diagram of an HDFS system in a Hadoop cluster and a data warehouse interface with a Blu-ray storage system, provided by an embodiment of the present invention.

[0032] based on Figure 1 The illustrated architecture shows the HDFS system in a Hadoop cluster and the data warehouse interface with the Blu-ray storage system. Correspondingly, this embodiment of the invention provides a data storage method, such as... Figure 2 As shown, this data storage method specifically includes the following steps:

[0033] S201: The client sends an authorization ticket request to the authentication server, so that the authentication server can obtain the corresponding authorization ticket based on the username, encrypt the authorization ticket with a password, and send the encrypted information to the client.

[0034] In this embodiment of the application, since the data in the HDFS system of the Hadoop cluster is diverse and the data-related business and usage are also different, data preprocessing is required to migrate the effective data to the Blu-ray storage system. The data is divided according to the update requirements, query requirements and retention time requirements. Data in the HDFS system that does not need to be updated or modified, has non-real-time query requirements and a long retention time is marked to generate files to be migrated. The marked files to be migrated can be used as objects to be migrated to the Blu-ray storage system.

[0035] In this embodiment of the application, after marking the files to be migrated in the HDFS system, the client can be further authenticated to ensure the legitimacy of the client connecting to the HDFS system, prevent access by unauthorized clients, and thus ensure the security of information in the HDFS system.

[0036] Specifically, host mappings can be added to the client's hosts file beforehand, and a Kerberos authentication component can be included in the client's request to the authentication server to generate an authorization ticket request, which is then sent to the authentication server. The Kerberos authentication component includes the username (principal) and password (keytab).

[0037] After receiving the authorization ticket request sent by the client, the authentication server can obtain the corresponding authorization ticket based on the username in the authorization ticket request, encrypt the obtained authorization ticket using the password in the authorization ticket request, obtain the corresponding encrypted information, and finally send the encrypted information back to the client.

[0038] S202: The client uses the password to decrypt the encrypted information, obtains the authorization ticket, and generates verification information based on the authorization ticket.

[0039] During the specific execution of step S202, when the client receives the encrypted information from the authentication server, it can use the password to decrypt the encrypted information, obtain the authorization ticket, and generate corresponding verification information based on the obtained authorization ticket.

[0040] In this embodiment, after generating the corresponding verification information based on the authorization ticket, the client can further send the generated verification information to the authorization server (Ticket Granting Server). After receiving the verification information, the authorization server can use the verification information to verify the legitimacy of the client. If the client is confirmed to be a legitimate client based on the verification information, the server can return the corresponding ticket to the client. If the client is confirmed to be an illegitimate client based on the verification information, the server can return the corresponding authentication failure information to the client.

[0041] Specifically, the authorization server can determine whether the client ID in the ticket in the verification information matches the client ID of the client; if they match, the client can be considered a legitimate client; if they do not match, the client can be considered an illegitimate client.

[0042] S203: When the client receives the ticket from the authorization server, it establishes a connection with the HDFS system and sends a remote procedure access request to the HDFS system. When the HDFS system determines that there is a file to be migrated whose name matches the file to be migrated in the remote access request, it obtains at least one data node information corresponding to the file to be migrated, and determines the target link with the shortest distance between the target data node corresponding to each data node information and the client.

[0043] In this embodiment, after the client sends the verification information back to the authorization server, it can detect in real time whether it has received a ticket from the authorization server. When the client detects a ticket from the authorization server, it can send a remote procedure access request to the master node namenode of the HDFS system through FileSystem based on the ticket from the authorization server. The remote procedure access request includes the name of the file to be migrated.

[0044] After receiving a remote procedure access request, the master node of the HDFS system can determine whether there is a file in the master node's metadata that matches the name of the file to be migrated. If there is a file to be migrated with a matching name, it can obtain at least one data node information corresponding to each data block related to the file to be migrated from the master node's metadata. For each data node information, it can find the target link with the shortest distance between the client and the target data node corresponding to that data node information, and return the target link corresponding to the target data node for each data node information to the client.

[0045] It should be noted that if the master node's metadata does not contain a file whose name matches the file to be migrated, the HDFS system can output a message indicating that no file with the name matching the file to be migrated currently exists.

[0046] S204: The client communicates with the corresponding target data node according to the corresponding target link, obtains the data block corresponding to the file to be migrated from the target data node, and transmits the data block to the Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system.

[0047] During the specific execution of step S204, when the client receives the target link fed back by each target data node, for each target data node, the client can communicate with the corresponding target data node through the corresponding target link in order to obtain the data blocks related to the file to be migrated from the target data node; after obtaining the data blocks related to the file to be migrated, the client can transmit the obtained data blocks to the Blu-ray storage system connected to the HDFS system through the target data node corresponding to the data block, until all the data blocks related to the file to be migrated on each target data node have been transmitted to the Blu-ray storage system.

[0048] Optionally, in this embodiment, after the client obtains the data block related to the file to be migrated from the target data node, it can create a storage bucket corresponding to the file to be migrated on the Blu-ray storage system connected to the HDFS system, obtain the target URL corresponding to the storage bucket, and finally transmit the obtained data block to the storage bucket created on the Blu-ray storage system through the target data node corresponding to the data block.

[0049] It should be noted that when the client successfully creates the storage bucket corresponding to the file to be migrated on the Blu-ray storage system, a target URL corresponding to that storage bucket will be generated accordingly.

[0050] It should be noted that each data block includes a corresponding check bit, which is used to characterize the integrity of the data block. That is, after the data block is transmitted to the Blu-ray storage system connected to the HDFS system, the checksum of the data block is calculated, and it is determined whether the checksum is consistent with the checksum value of the checksum bit in the data block. If the checksum is consistent with the checksum value of the checksum bit in the data block, it can be determined that the data block has been transmitted completely, that is, the data block has been transmitted intact; if the checksum is inconsistent with the checksum value of the checksum bit in the data block, it can be considered that the data block has not been transmitted completely, and the corresponding error message is output.

[0051] The error message indicates that the data block transmission failed.

[0052] In some embodiments, for each data block associated with the file to be migrated, if the client determines that the data block transmission has failed, it can reselect a data node from other data nodes on the HDFS system to communicate with, so that the data block can be retransmitted to the storage bucket created on the Blu-ray storage system through the reselected data node, so that the Blu-ray storage system can store the data block corresponding to the file to be migrated.

[0053] Furthermore, in this embodiment of the application, the client can initialize the S3 client object before creating the storage bucket corresponding to the file to be migrated on the Blu-ray storage system.

[0054] Furthermore, in this embodiment of the application, if the remote procedure access request includes the name of the directory of files to be migrated, then for each file name to be migrated in the directory of files to be migrated, steps S203 to S204 can be executed to transfer the data block related to the file to be migrated corresponding to the file name to be migrated to the Blu-ray storage system to complete the transfer of the file to be migrated, until all the files to be migrated related to the directory of files to be migrated have been transferred to the Blu-ray storage system.

[0055] S205: The client obtains the first total data volume information of the files to be migrated from the HDFS system, and the second total data volume information of the files to be migrated from the Blu-ray storage system.

[0056] During the specific execution step S205, after the client has transferred all data blocks related to the file to be migrated to the Blu-ray storage system, it can obtain the first total data volume information of the file to be migrated by calling the HDFS content query interface corresponding to the file to be migrated on the HDFS system; and query the second total data volume information of the file to be migrated from the Blu-ray storage system and the Greenplum database connected to it.

[0057] It should be noted that the second data total information of the files to be migrated retrieved from the Greenplum database refers to the amount of information of the files to be migrated stored on the Blu-ray storage system.

[0058] It should also be noted that the first total data information includes the number of rows and column names of the data tables of the files to be migrated stored on the HDFS system; the second total data information includes the number of rows and column names of the data tables of the files to be migrated stored on the Blu-ray storage system.

[0059] S206: The client compares the first total data information and the second total data information; if the first total data information and the second total data information are consistent, proceed to step S207; if the first total data information and the second total data information are inconsistent, proceed to step S208.

[0060] During the specific execution step S206, after obtaining the first total data information and the second total data information, the client can compare the number of data table rows in the first total data information with the number of data table rows in the second total data information, and compare the column names in the first total data information and the column names in the second total data information. If the number of data table rows in the first total data information is consistent with the number of data table rows in the second total data information, and the column names in the first total data information and the column names in the second total data information are consistent, then it can be determined that the migration file transferred to the Blu-ray storage system is complete.

[0061] If the number of rows in the data table in the first total data information is inconsistent with the number of rows in the data table in the second total data information, and / or, the column names in the first total data information and the column names in the second total data information are inconsistent, it can be determined that the migration file transferred to the Blu-ray storage system is incomplete, and an error message can be output to indicate that the migration file currently being transferred to the Blu-ray storage system is incomplete.

[0062] S207: The client confirms that the files to be migrated have been migrated completely.

[0063] S208: The client determines that the migration of the file to be migrated is incomplete and outputs the corresponding error message.

[0064] This invention provides a data storage method. When a client receives a ticket from an authorization server, it establishes a connection with the HDFS system and sends a remote procedure access request to the HDFS system. When the HDFS system determines that a file matching the name of the file to be migrated exists in the remote access request, it obtains at least one data node information corresponding to the file to be migrated and determines the target link with the shortest distance between the target data node and the client for each data node. The ticket is returned by the authorization server when it confirms the client's legitimacy based on the verification information sent by the client. The verification information is generated by the client based on the authorization ticket request sent by the client by the authentication server. For each target data node, the client communicates with the target data node according to the corresponding target link, obtains the data block corresponding to the file to be migrated from the target data node, and transmits the data block to a Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system. The file to be migrated is data that does not require modification, has non-real-time query requirements, and requires long storage time. The technical solution provided by this invention establishes a connection between the HDFS system and the Blu-ray storage system in advance. Then, through the client, files in the HDFS system that do not need to be modified, have non-real-time query requirements, and have long storage time requirements can be migrated to the Blu-ray storage system for storage. This solves the problem that existing storage devices have storage lifespans, which require data to be migrated to new storage devices every time the expiration date approaches. This can lead to operational risks and increased operating costs during the data migration process.

[0065] Furthermore, based on the data storage method provided in the above embodiments of this application, the embodiments of this application also include the following steps, such as... Figure 3 As shown, it specifically includes:

[0066] S301: When the client receives a reverse migration request sent by the user, it obtains the corresponding reverse migration information based on the reverse migration request.

[0067] The reverse migration information includes the object storage URL, access key, and access key.

[0068] In this embodiment of the application, after the client establishes a connection with the HDFS system based on the ticket returned by the authorization server, it can also detect in real time whether there is a reverse migration request sent by the user; when a reverse migration request sent by the user is detected, the client can receive the reverse migration request sent by the user and obtain the corresponding reverse migration information based on the reverse migration request.

[0069] It should be noted that the reverse migration information includes not only the object storage URL, access key, and access key, but also the reverse migration file name.

[0070] It should also be noted that the object storage URL is the URL of the storage bucket corresponding to the reverse migration file name.

[0071] S302: The client determines whether the corresponding reverse migration file exists in the Blu-ray storage system based on the reverse migration information; if the reverse migration file exists in the Blu-ray storage system, proceed to step S303.

[0072] During the specific execution of step 302, after obtaining the corresponding reverse migration information, the client can further call a pre-set SQL statement to determine whether there is a reverse migration file in the Blu-ray storage system that matches the reverse migration file name in the reverse migration information; if there is no reverse migration file in the Blu-ray storage system that matches the reverse migration file name in the reverse migration information, an error message can be output to prompt that there is no reverse migration file in the Blu-ray storage system that matches the reverse migration file name.

[0073] If a reverse migration file with a name matching the reverse migration file exists in the Blu-ray storage system, then step S303 can be executed.

[0074] S303: The client authenticates the user based on the access key and the access key; if the user's authentication is successful, proceed to step S304.

[0075] During the specific execution of step S303, if the client determines that there is a reverse migration file with a matching reverse migration file name in the Blu-ray storage system, it can further authenticate the user's identity based on the access key and the access key; if the user's identity authentication based on the access key and the access key is successful, then step S304 can be executed.

[0076] If user authentication fails based on the access key and access key, an error message can be output to indicate that authentication has failed.

[0077] S304: The client obtains the corresponding user file directory read / write permission information and determines whether the user file directory read / write permission information is allowed. If the user file directory read / write permission information is allowed, proceed to step S305.

[0078] During the specific execution of step S304, if the user's identity authentication is successful, the client can further obtain the user file directory read and write permission information related to the reverse migration file, and determine whether the user file directory read and write permission information is allowed to read and write; if the user file directory read and write permission information is allowed to read and write, that is, the reverse migration file is allowed to read and write, and then step S305 can be executed.

[0079] If the user's file directory read / write permissions are not allowed, that is, if the user's file directory read / write permissions are not allowed, then an error message can be output to indicate that the reverse migration file is not allowed to be read or written.

[0080] S305: The client migrates reverse migration files from the Blu-ray storage system to the HDFS system.

[0081] In this implementation, if the client determines that the reverse migration file is allowed to be read and written, it can create an external table based on the object storage URL in the Greenplum database according to the reverse migration information, and create a read function based on the S3 interface in the Greenplum database, so as to read the data block corresponding to the reverse migration file from the bucket corresponding to the object storage URL in the Blu-ray storage system according to the created read function.

[0082] After reading the corresponding reverse migration file, the client can connect to the HDFS system and check if the file directory corresponding to the reverse migration file exists in the HDFS system's metadata. If it does not exist, the client can output the corresponding error message. If it does exist, the client can create the corresponding new folder in the metadata and receive the data node information fed back by the master node based on the created new folder. This allows the client to transfer the data blocks of the read reverse migration file to the data node corresponding to the data node information, until the data blocks related to the reverse migration file are transferred to the data node corresponding to the data node information so that the data node can store the corresponding data blocks.

[0083] It should be noted that the master node can provide information on multiple data nodes based on the newly created folder. Correspondingly, if the master node provides information on multiple data nodes based on the newly created folder, the client can transfer one or more data blocks of the reverse-migrated file to the data node corresponding to any one of the data nodes provided, and then transfer another one or more data blocks of the reverse-migrated file to the data node corresponding to the other data node, until all data blocks related to the reverse-migrated file have been transferred to the HDFS system.

[0084] Furthermore, in this embodiment, the HDFS system can also back up the reverse migration file in other data nodes. That is, the HDFS system completes the copying of the reverse migration file between data nodes, and after the backup is completed, it sends the corresponding successful backup information to the client.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0087] The names of the messages or information exchanged between the multiple devices in the embodiments of this invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0088] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0089] It should be understood that the steps described in the method embodiments disclosed in this invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this invention is not limited in this respect.

[0090] Computer program code for performing the operations disclosed herein can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] The units described in the embodiments disclosed in this invention can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0092] The functions described above in this application can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0093] Based on the data storage method provided in the above embodiments of the present invention, correspondingly, the present invention also discloses a data storage system, such as... Figure 4 As shown, this data storage system is applied to the client side, and the data storage system includes:

[0094] The remote procedure access request sending unit 41 is used to establish a connection with the HDFS system when receiving a ticket from the authorization server, and to send a remote procedure access request to the HDFS system. This allows the HDFS system to obtain at least one data node information corresponding to the file to be migrated when it determines that there is a file to be migrated whose name matches the file to be migrated in the remote access request, and to determine the target link with the shortest distance between the target data node corresponding to each data node information and the client. The ticket is returned by the authorization server when it confirms that the client is a legitimate client based on the verification information sent by the client. The verification information is generated by the client based on the authorization ticket request sent by the authentication server.

[0095] The transmission unit 42 is used to communicate with the corresponding target data node according to the corresponding target link, obtain the data block corresponding to the file to be migrated from the target data node, and transmit the data block to the Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system; wherein, the file to be migrated is data that does not need to be modified, has non-real-time query requirements, and has long storage time requirements.

[0096] The specific principles and execution processes of each unit in the data storage system disclosed in the above embodiments of the present invention are similar to those in the above embodiments of the present invention. Figure 2 The publicly disclosed data storage method is the same, and can be found in the above embodiments of the present invention. Figure 2 The relevant parts of the publicly available data storage methods will not be elaborated here.

[0097] This invention provides a data storage system. When a client receives a ticket from an authorization server, it establishes a connection with the HDFS system and sends a remote procedure access request to the HDFS system. When the HDFS system determines that a file matching the name of the file to be migrated exists in the remote access request, it obtains at least one data node information corresponding to the file to be migrated and determines the target link with the shortest distance to the client for each data node. The ticket is returned by the authorization server when it confirms the client's legitimacy based on the verification information sent by the client. The verification information is generated by the client based on the authorization ticket request sent by the client by the authentication server. For each target data node, the client's corresponding target link communicates with the target data node, obtains the data block corresponding to the file to be migrated from the target data node, and transmits the data block to a Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system. The file to be migrated is data that does not require modification, has non-real-time query requirements, and requires long storage time. The technical solution provided by this invention establishes a connection between the HDFS system and the Blu-ray storage system in advance. Then, through the client, files in the HDFS system that do not need to be modified, have non-real-time query requirements, and have long storage time requirements can be migrated to the Blu-ray storage system for storage. This solves the problem that existing storage devices have storage lifespans, which require data to be migrated to new storage devices every time the expiration date approaches. This can lead to operational risks and increased operating costs during the data migration process.

[0098] Optionally, the verification information generation unit includes:

[0099] The authorization ticket request sending unit is used to send an authorization ticket request to the authentication server, so that the authentication server can obtain the corresponding authorization ticket based on the username, encrypt the authorization ticket using a password, and send the encrypted information to the client; wherein, the authorization ticket request includes the username and password;

[0100] The verification information generation subunit is used to decrypt encrypted information using a password to obtain an authorization ticket, and generate verification information based on the Ticket Granting Ticket.

[0101] Optional, the transmission unit includes:

[0102] The bucket creation unit is used to create a bucket corresponding to the file to be migrated on the Blu-ray storage system and obtain the target URL corresponding to the bucket.

[0103] The transmission subunit is used to transmit data blocks to a storage bucket on the Blu-ray storage system according to the target URL.

[0104] Optionally, the data storage system provided in this embodiment of the invention further includes:

[0105] The first judgment unit is used to calculate the check value of the data block after the data block is transmitted to the Blu-ray storage system connected to the HDFS system, and to determine whether the check value is consistent with the check value of the check bit in the data block.

[0106] The first determining unit is used to determine that the data block transmission is complete if the check value is consistent with the check value of the check bit in the data block.

[0107] The first output unit is used to determine that the data block transmission has failed if the check value is inconsistent with the check value of the check bit in the data block, and to output the corresponding error message.

[0108] Optionally, after the transmission unit is executed, the data storage system provided in this embodiment of the invention further includes:

[0109] The total data information acquisition unit is used to obtain the first total data information of the files to be migrated from the HDFS system, and to obtain the second total data information of the files to be migrated from the Blu-ray storage system.

[0110] The comparison unit is used to compare the first total data information and the second total data information;

[0111] The second determining unit is used to determine that the file to be migrated has been completely migrated if the first total data information and the second total data information are consistent.

[0112] The second output unit is used to determine that the migration of the file to be migrated is incomplete if the first total data information and the second total data information are inconsistent, and to output the corresponding error information.

[0113] Optionally, the data storage system provided in this embodiment of the invention further includes:

[0114] The reverse migration information acquisition unit is used to acquire the corresponding reverse migration information based on the reverse migration request when it receives a reverse migration request sent by the user; wherein, the reverse migration information includes the object storage URL, access key and access key;

[0115] The second judgment unit is used to determine whether there is a corresponding reverse migration file in the Blu-ray storage system based on the reverse migration information.

[0116] The migration unit is used to migrate the reverse migration files from the Blu-ray storage system to the HDFS system if reverse migration files exist in the Blu-ray storage system.

[0117] Optionally, before executing the migration unit, the data storage system provided in this embodiment of the invention further includes:

[0118] An authentication unit is used to authenticate users based on access keys and access keys;

[0119] The user file directory read / write permission information acquisition unit is used to acquire the corresponding user file directory read / write permission information if the user's authentication is successful.

[0120] The migration unit is also used to migrate reverse migration files from the Blu-ray storage system to the HDFS system if the user's file directory read / write permission information is allowed.

[0121] This application also provides an electronic device, which includes a processor and a memory connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; and the memory is used to store the program, which is used to implement a data storage method.

[0122] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing embodiments of the present invention. The electronic devices in the embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the disclosed embodiments of the present invention.

[0123] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0124] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0125] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the data storage method shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the data storage method of the embodiments disclosed in this invention.

[0126] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing a data storage method.

[0127] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes these programs, the electronic device: upon receiving a ticket from an authorization server, establishes a connection with the HDFS system and sends a remote procedure access request to the HDFS system. When the HDFS system determines that a file matching the name of the file to be migrated exists in the remote access request, it obtains at least one data node information corresponding to the file to be migrated and determines the target link with the shortest distance between the target data node corresponding to each data node information and the client. The ticket is returned by the authorization server when it confirms the client's legitimacy based on verification information sent by the client. The verification information is generated by the client based on the authorization ticket request sent by the authentication server. The electronic device communicates with the corresponding target data node according to the corresponding target link, obtains data blocks corresponding to the file to be migrated from the target data node, and transmits the data blocks to a Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system. The file to be migrated is data that does not require modification, has non-real-time query requirements, and requires long storage time.

[0128] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] It should be noted that the computer-readable medium disclosed in this invention may be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0130] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

Claims

1. A data storage method, characterized in that, Applied to a client, the method includes: Upon receiving a ticket from the authorization server, a connection is established with the HDFS system, and a remote procedure access request is sent to the HDFS system. When the HDFS system determines that a file exists matching the name of the file to be migrated in the remote procedure access request, it obtains at least one data node information corresponding to the file to be migrated, and determines the target link with the shortest distance between the target data node corresponding to each data node and the client. The ticket is returned by the authorization server when it confirms the client's legitimacy based on the verification information sent by the client; the verification information is generated by the client based on the authorization ticket request sent by the authentication server. According to the corresponding target link, the system communicates with the corresponding target data node, obtains the data block corresponding to the file to be migrated from the target data node, and transmits the data block to the Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system; the file to be migrated is data that does not need to be modified, has non-real-time query requirements, and has long storage time requirements. After transferring all data blocks related to the files to be migrated to the Blu-ray storage system, the method further includes: Obtain the first total data volume information of the files to be migrated from the HDFS system, and obtain the second total data volume information of the files to be migrated from the Blu-ray storage system; Compare the first total data information with the second total data information; If the first total data information and the second total data information are consistent, it is determined that the migration of the file to be migrated is complete; If the first total data information and the second total data information are inconsistent, it is determined that the migration of the file to be migrated is incomplete, and the corresponding error message is output. The process of generating the verification information includes: sending the authorization ticket request to the authentication server so that the authentication server obtains the corresponding authorization ticket based on the username; encrypting the authorization ticket using a password; and sending the encrypted information to the client; wherein the authorization ticket request includes the username and password; decrypting the encrypted information using the password to obtain the authorization ticket; and generating the verification information based on the Ticket Granting Ticket.

2. The method according to claim 1, characterized in that, The step of transferring the data block to the Blu-ray storage system connected to the HDFS system includes: Create a storage bucket corresponding to the file to be migrated on the Blu-ray storage system, and obtain the target URL corresponding to the storage bucket; The data block is transferred to the storage bucket on the Blu-ray storage system according to the target URL.

3. The method according to claim 1, characterized in that, The method further includes: After the data block is transmitted to the Blu-ray storage system connected to the HDFS system, the checksum of the data block is calculated, and it is determined whether the checksum is consistent with the checksum value of the check bit in the data block. If the check value matches the check value of the check bit in the data block, the data block transmission is considered complete. If the check value is inconsistent with the check value of the check bit in the data block, the data block transmission is determined to have failed, and the corresponding error message is output.

4. The method according to claim 1, characterized in that, The method further includes: Upon receiving a reverse migration request from a user, the system obtains the corresponding reverse migration information based on the request; wherein, the reverse migration information includes the object storage URL, access key, and access key. Determine whether a corresponding reverse migration file exists in the Blu-ray storage system based on the reverse migration information; If the reverse migration file exists in the Blu-ray storage system, the reverse migration file on the Blu-ray storage system is migrated to the HDFS system.

5. The method according to claim 4, characterized in that, The method further includes: The user is authenticated based on the access key and the access key; If the user's authentication is successful, obtain the corresponding user file directory read / write permission information; The process of migrating reverse-migration files from the Blu-ray storage system to the HDFS system includes: If the user file directory read / write permission information is set to allow read / write, the reverse migration files on the Blu-ray storage system will be migrated to the HDFS system.

6. A data storage system, characterized in that, The data storage system is applied to the client, and the system includes: The remote procedure access request sending unit is used to establish a connection with the HDFS system when receiving a ticket from the authorization server, and to send a remote procedure access request to the HDFS system. This allows the HDFS system to determine if a file exists that matches the name of the file to be migrated in the remote procedure access request. Then, the HDFS system obtains at least one data node information corresponding to the file to be migrated, and determines the target link with the shortest distance between the target data node corresponding to each data node and the client. The ticket is returned by the authorization server when it confirms the client's legitimacy based on verification information sent by the client. The verification information is generated by the client based on the authorization ticket request sent by the authentication server. The transmission unit is used to communicate with the corresponding target data node according to the corresponding target link, obtain the data block corresponding to the file to be migrated from the target data node, and transmit the data block to the Blu-ray storage system connected to the HDFS system until all data blocks related to the file to be migrated have been transmitted to the Blu-ray storage system; the file to be migrated is data that does not need to be modified, has non-real-time query requirements, and has long storage time requirements; After the transmission unit is executed, the data storage system further includes: The total data information acquisition unit is used to acquire the first total data information of the file to be migrated from the HDFS system, and to acquire the second total data information of the file to be migrated from the Blu-ray storage system. The comparison unit is used to compare the first total data information and the second total data information; The second determining unit is used to determine that the migration of the file to be migrated is complete if the first total data information and the second total data information are consistent. The second output unit is used to determine that the migration of the file to be migrated is incomplete if the first total data information and the second total data information are inconsistent, and to output the corresponding error information. The verification information generation unit includes: an authorization ticket request sending unit and a verification information generation subunit; The authorization ticket request sending unit is used to send the authorization ticket request to the authentication server, so that the authentication server obtains the corresponding authorization ticket based on the username, encrypts the authorization ticket using a password, and sends the encrypted information to the client; wherein, the authorization ticket request includes the username and password; The verification information generation subunit is used to decrypt the encrypted information using the password to obtain the authorization ticket, and generate the verification information based on the Ticket Granting Ticket.

7. An electronic device, characterized in that, include: A processor and a memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the data storage method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for performing the data storage method as described in any one of claims 1-5.

Images